Cancer incidence and competing mortality risk following 15 presenting symptoms in primary care: a population-based cohort study using electronic healthcare records

Study population

We used a cohort study design, based on medical records from English National Health Service general practices that contributed anonymised primary-care electronic health records to the Clinical Practice Research Datalink Gold (CPRD). CPRD covers approximately 6.9% of the UK population,9 and patients in CPRD are broadly representative of the UK general population with respect to age, sex and ethnicity.9 CPRD was linked to cancer diagnosis information from the English national cancer registry.10 We considered all cancers excluding non-melanoma skin cancer, as non-melanoma skin cancer is imperfectly registered and primarily managed in primary care.

A study flowchart is given in online supplemental appendix 1 figure 1. We first extracted a random sample of patients from CPRD for use as a reference group, choosing index dates randomly from ‘valid’ follow-ups from 1 January 2007 to 31 December 2017. Patients in this reference group were not necessarily symptom-free (online supplemental appendix 1 table 1). Coded symptom data are known to be incomplete,11 12 so it was not possible to create a truly symptom-free control group with the data available for this study. Thus, we chose to use a reference group that would represent the average risk for patients registered in primary care. We then created a symptomatic cohort of all patients in CPRD Gold who had consulted for any of 15 presenting symptoms and who were not in the reference group, choosing the index date as the date of their first ‘valid’ consultation for a symptom during 1 January 2007 to 31 December 2017.

For an individual patient, follow-up was judged to be ‘valid’ if they had been registered at their practice for at least 1 year; their practice was judged by CPRD to be providing data of a suitable standard for use in research (ie, after the practice’s ‘up-to-standard’ date); it was before the last data transfer to CPRD (ie, the ‘last collection’ date); the patient was registered at a CPRD practice (ie, before the patient’s ‘transfer out’ date and before their death); the patient was aged 30–99; and the patient had not yet had a recorded cancer diagnosis in the cancer registry (excluding non-melanoma skin cancer).

Outcomes

Both mortality and cancer diagnoses were considered. Mortality was identified from the primary care record; such information is highly concordant with the ‘gold standard’ official death registration records and is correct within 1 month 98% of the time.13 Cancers were split into seven groups for men and eight groups for women, summarised below and with a full ICD10 codelist in online supplemental appendix 1 table 2, guided by underlying body systems and corresponding major clinical specialties receiving urgent referrals for suspected cancer in England.14 Cancer diagnoses were sourced from linkages with the national cancer registry, and only the first cancer diagnosis was considered (excluding non-melanoma skin cancer); available cancer data covered diagnoses up until 31 December 2018.

The cancer groups considered were:

• Breast cancer (women only), including invasive breast and in-situ breast cancers.

• Gynaecological cancer (women only), including invasive cervical, in-situ cervical, ovarian, uterine and vulvar cancers.

• Lung, including lung cancer and mesothelioma.

• Upper gastrointestinal (GI), including liver, oesophageal, pancreatic and stomach cancers.

• Lower GI, including colon and rectal cancers.

• Urological, including bladder, in-situ bladder, kidney and other urinary tract cancers.

• Prostate cancer (men only).

• Haematological, including Hodgkin’s lymphoma, non-Hodgkin’s lymphoma, acute myeloid leukaemia, chronic lymphocytic leukaemia, other leukaemias, myeloma and other haematological cancers

• Other, including all other sites, specifically melanoma, unknown primary, thyroid and meningeal cancers, also including testicular cancer and male breast cancer.

The first outcome (of cancer diagnosis or non-cancer death) experienced by each patient was considered in the analysis. This means, for example, that in the analyses of cumulative incidence, a patient who died shortly following a cancer diagnosis would only be considered to have had a cancer diagnosis, and their death would not contribute to the estimation of mortality risk, irrespective of the cause of death. Patients with a cancer diagnosis on the same day as their death (including death certificate-only registrations of cancer) were treated as having had a cancer diagnosis rather than having died, noting that death certificate-only registrations remained at <0.4% through the study period.15

Symptoms

Patients were selected due to a primary care presentation with one (or more) of 15 cancer-relevant symptoms or due to being in the reference group. The index date of symptomatic patients was the date of their first recorded symptom during ‘valid’ follow-up (defined in Methods—study population).

The symptoms we considered were a subset of those known to be associated with the risk of specific types of cancer and are already included in referral guidelines for symptomatic cancer.3 16 The included symptoms form part of the presentation in 40% of all patients with cancer in England.1 We identified symptoms from coded primary care data using existing Read v2 phenotyping algorithms.16 The symptoms we considered were:

• Abdominal symptoms:

  • Abdominal pain.

  • Abdominal bloating (including abdominal distension).

  • Rectal bleeding

  • Change in bowel habits.

  • Dyspepsia

  • Dysphagia

  • Jaundice

• Respiratory symptoms

  • Dyspnoea

  • Haemoptysis

• Urological symptoms

  • Haematuria.

• Non-specific symptoms:

  • Fatigue

  • Night sweats.

  • Weight loss.

• Breast and reproductive organ symptoms:

  • Breast lump (including in men).

  • Post-menopausal bleeding.

Only the first presenting symptom for each patient was included, and each patient was included at most once in the analysis. For example, if a patient had a consultation for a breast lump in 2007 that did not result in a cancer diagnosis and a consultation for abdominal pain in 2010 that did result in a cancer diagnosis, only the risk after the 2007 consultation for a breast lump would be included in analysis. Symptoms were included in the model using one-hot encoding, with patients in the reference group having all symptom variables set to 0. Where patients in a symptomatic cohort presented with two or more symptoms on their index date, all were included as index symptoms (such occurrences were rare, see end of the Results section). Symptoms that were not consulted for on the same day as the index were not considered.

Smoking status, sex and age

Patients were categorised as ever-smokers or never-smokers. Ever-smokers included all patients with a record of being current or ex-smokers in their entire primary care record, including periods after a cancer diagnosis or before their record became eligible for use in this study; never-smokers included all other patients. Patients were grouped as male or female based on the recorded gender in their primary care record. Patients’ age was estimated as the number of years between the mid-point of their year of birth and their index date.

Statistical methods

The initial analysis described the distribution of patients in the sample and counts of cancer diagnoses and deaths within 12 months of any index symptom.

Hazards for specific cancers and non-cancer mortality were estimated using flexible parametric (Royston–Parmar) time-to-event models,17 using three degrees of freedom to model the baseline hazard. Follow-up for these analyses was censored at 18 months after the index symptom, at the first event (ie, cancer diagnosis or death) or the end of the available cancer registry follow-up on 31 December 2018 if earlier. Models were stratified by sex and included the following covariates:

• Age (restricted cubic spline with six knots).

• Smoking status (binary, ever record of smoking in primary care data vs never).

• Index symptom (15 binary variables indicating the symptom(s) each patient had on their index date (all zero for patients in the reference group)).

• An interaction with (the log of) follow-up time in months for each index symptom, allowing the association between symptom and cause-specific risk to decay over time. This was motivated by the fact that following many possible symptoms of cancer, excess risk is highest in the first months following presentation (eg,18).

The cumulative incidence of cancer group and non-cancer mortality was estimated by combining each of the cause-specific models into a multistate model using the latent failure time approach.18 We report cumulative incidence for combinations of age-, sex- and smoking-specific symptoms up to 12 months follow-up, with results focusing on estimated cumulative incidence at 12 months and age considered in 5 year intervals. To sense-check these model-based estimates, we additionally examined the crude cumulative incidence for each cancer group and non-cancer mortality within 12 months of each symptom by sex and smoking status using Aalen–Johansen non-parametric cumulative incidence curves.19 20

Concordant with the methods and evidence that informed the development of NICE guidelines, we have considered the modelled cumulative incidence at 12 months to represent the positive predictive value for the outcome for the symptom.3 Further, we calculated the (sex-/smoking-/symptom-specific) age at which the cancer risk exceeded the 3% risk threshold for referrals used in the UK. We additionally present similar estimates for each individual cancer group.

Statistical modelling used Stata 17 MP. Simulation of failure times was performed on a high-performance cluster using Stata 16 MP. Survival models were fit using the merlin package,21 and multistate modelling was facilitated by the multistate package.22 In principle, the cancer risk for any combination of symptoms can be estimated from the cause-specific models, but these have not been produced due to computational limitations and the very large number of potential combinations. Data extraction and analysis code are available at https://github.com/MattEBarclay/cprd_symptom_cancer_1.

Patient and public involvement

The study forms part of a programme of work examining the predictive value of symptoms for cancer diagnosis using electronic health records data. To support this programme, we ran three focus groups in August and September 2023 including a total of 15 patient and public involvement volunteers. Study reporting was informed by input from these volunteers, but no specific changes were made.

Age-adjusted cancer-specific HRs for smoking and each index symptom

Both male and female ever-smokers had far higher cancer-specific hazards of lung cancer than non-smokers (figure 1 and online supplemental appendix 3, HR 4.8, 95% CI 4.2 to 5.6, for women and HR 4.0, 95% CI 3.5 to 4.6, for men) and elevated hazards of urological (eg, for men: HR 1.4, 95%CI 1.2 to 1.5, online supplemental appendix 3 table 4) and upper GI cancers (eg, for men: HR 1.4, 95% CI 1.2 to 1.5, online supplemental appendix 3 table 1).

Request permission

Patients consulting for symptoms of possible cancer had similar or greater cause-specific hazards for almost every cancer site than the reference population (figure 1 and online supplemental appendix 3). Yet for 10 of the 15 studied symptoms, the symptom was associated with lower cause-specific hazards for death than the reference group (the exceptions being dysphagia, jaundice, dyspnoea, haemoptysis and weight loss).

Further, for many symptoms associated with a very high initial hazard of a specific cancer, while the hazard typically remained elevated at least 12 months after the index consultation, it tended to reduce over time. For example, online supplemental appendix 3 table 1 shows HRs for lung cancer. The HR for haemoptysis is 17.1, but there is a statistically significant interaction with (the natural log of) follow-up time in months with an HR of 0.7; by 12 months the HR for lung cancer is estimated to have decreased to around 7.3. This fits with the non-parametric results shown in the Aalen–Johansen plots, where after haemoptysis presentation diagnoses of lung cancer rapidly accrue until about 3 months follow-up, after which they continue growing but less rapidly (online supplemental appendix 2 figures 1–4).

Risk of specific cancer sites by age, sex and smoking status

After symptom presentation for patients with single index symptoms and based on simulations combining the cause-specific models, we present simulated cumulative incidence of each cancer site and of death without cancer at 3 months (online supplemental appendix 4 figures 1–4), 6 months (, online supplemental appendix 4 figures 5–8) and 12 months (figures 2–5, online supplemental appendix 5). Hereafter in this section, we discuss cumulative incidence at 12 months after symptom consultation. Unlike the HRs presented above, estimates of cumulative incidence varied substantially by sex, as women have lower baseline cancer risk.

Request permission

Request permission

Request permission

Request permission

Risk of non-cancer mortality

For most of the studied symptoms, symptomatic patients were less likely to die (without a cancer diagnosis) than similar patients in the reference group (figures 2–5). The three principal exceptions were jaundice, dysphagia and weight loss, for which post-presentation mortality exceeded that in the reference group, and also older patients with less-specific symptoms for whom the risk of non-cancer mortality was often higher than the risk of any cancer. For example, for male smokers presenting with dyspnoea, around 6% who presented at age 80 would develop cancer within 12 months, while 9% would die (figure 3, online supplemental appendix 5 table 1).

Presentation with multiple symptoms

Among symptomatic patients, 1.2% (10 360 of 835 995) consulted for more than one of the 15 studied symptoms on their index date, and a further 2.5% (21 167) consulted for an additional studied symptom within 30 days of an index symptom but before a cancer diagnosis (table 3). The proportion of patients with multiple index symptoms subsequently diagnosed with cancer within 12 months of the index (4.6%, 95% CI 4.2% to 5.1%) was higher than for patients with a single index symptom (3.5%, 95% CI 3.5% to 3.5%). This higher risk of cancer in patients with multiple index symptoms appeared applicable to many of the symptoms considered, but sample size limitations meant proportions developing cancer could often not be estimated precisely.

Strengths and weaknesses

Key strengths of the study are (a) the large representative dataset, allowing examination of a range of both common and rare symptoms and outcomes; (b) the joint estimation of the risks of different outcomes, including non-cancer mortality and risk of different types of cancer; and (c) the use of cancer registry data to ascertain the presence of cancer, as cancer may be under- or over-recorded in non-registry sources.23 While this study represents the most comprehensive and detailed description of the risk of cancer in symptomatic patients to date, there are various areas where future work could make further improvements.

Our study covers a period from 2007 to 2017, during which there have been many secular changes such as the introduction of public health education campaigns to raise awareness of symptoms of possible cancer among members of the public, alongside changes in clinical guidelines for referral for suspected cancer, and how NHS diagnostic services are configured, but we have not examined secular trends in estimated risks. Further, the study only considers deaths in patients without cancer, but it may be important to understand if patients die quickly after a cancer diagnosis. Our measure of the smoking status does not allow for a refined appreciation of smoking history and dose-response relationships. Additionally, our analytical approach only allowed each patient to be included once, not making full use of the longitudinal nature of EHR datasets.24 We did not consider interactions between symptoms and simulated outcomes for patients with a single symptom only, in part due to only a few patients having multiple symptoms. We did not have access to free-text data, despite evidence that coded data does not capture all symptoms.11 12 Finally, we only examined 15 symptoms, ignoring the many other symptoms and important health conditions that may be associated with the risk of cancer.1 17 25 A more detailed examination of potential limitations is given in online supplemental appendix 7.

Comparison with literature

A large and growing literature describes the risk of cancer following symptom presentations in primary care; Moore and colleagues summarised the literature pre-2020,17 and there are several recent papers.26–29 Existing literature (a) rarely considers competing non-cancer mortality risk, (b) rarely considers smoking status and (c) frequently provides no or only limited information on the age-dependent and sex-specific nature of the risk of different cancers. Much of the previous evidence additionally considers either the risk of all cancers combined or focuses on specific cancer sites judged to be of relevance to the specific examined symptoms a priori. We improve on previous descriptive studies by presenting a broad range of possible cancer diagnoses following presentation with a wider spectrum of index symptoms. Further research is needed to extend analyses similar to those reported here to a wider collection of symptoms.

Some existing evidence on the so-called red-flag symptoms such as rectal bleeding and haemoptysis suggests the risk of cancer exceeds 3% for all ages but did not examine the risk in different age groups17; our findings indicate that the risk of cancer following these symptoms only exceeds 3% beyond certain age cut-offs. Furthermore, we show that for non-specific symptoms, the risk of any cancer exceeds 3% at a considerably earlier age than the risk of a specific cancer type, underscoring the need for studies that comprehensively examine all major cancer types. Weight loss provides a cardinal example, where the risk of any cancer exceeded 3% in male non-smokers from age 55, but the risk of any individual site only reached 3% at age 85.

Other studies have aimed to develop risk prediction tools for cancer intended for use in a primary care setting (see, eg,30–32), and in particular the QCancer risk prediction tool33 34 already considers a range of symptoms and a risk of a diagnosis of different types of cancer. For decisions about the management of an individual patient, a risk prediction tool including multiple potential predictors may be more suitable than the results presented in this paper. We view our results as complementary; by describing what is effectively the average risk in patients presenting with these symptoms (by age, sex and smoking status), we can inform high-level policy decisions around the symptomatic diagnosis of cancer such as clinical guideline recommendations and help developers of more detailed risk prediction models by highlighting symptoms they may wish to consider. Further, our consideration of mortality risk provides relevant information that is frequently missing from current risk prediction tools (including QCancer) and that is especially important in frail and elderly populations.

Implications

Symptoms recorded in primary care data can be highly informative about both cancer risk and short-term mortality risk. In some cases, for example, lung cancer, smoking status is very strongly associated with the risk of cancer following a certain symptom. The risk of cancer and non-cancer mortality varies considerably by age; describing the ‘overall’ risk of cancer following a symptom may be misleading if non-cancer mortality is not considered. Some (non-cancer) deaths will relate to as-yet undiagnosed diseases which, like a cancer diagnosis, necessitates specialist assessment in secondary care, though this should be the subject of future inquiries.

For researchers, our results underline the methodological importance of accounting for the fact that symptoms may be associated with multiple different disease outcomes. Advanced statistical modelling strategies are helpful in assessing diagnostic outcomes using EHR data, and current statistical packages allow for relatively straightforward handling of competing risks either by directly modelling cumulative incidence (eg, the Fine-Grey model35) or, as here, by combining several cause-specific models.36 Diagnostic research should adopt strategies that allow consideration of the risk of several potentially related diseases (eg, multiple types of cancer, as in this study), which can be done even with simple analytical approaches such as the appropriate use of logistic regression.29

For clinicians and policymakers, our systematic assessment of the risk of cancer (and of non-cancer mortality) in symptomatic patients in primary care raises two key questions.

First, whether all age-sex-smoking status groups presenting with each of the studied symptoms and with an estimated any-cancer risk of above 3% should explicitly be added to NICE referral guidelines. This may indeed be justified, though given the high mortality rates in the oldest patients, there might also be a risk of over-testing in older men in particular. However, the degree to which the risk of over-testing is a concern relates to the exact causes of non-cancer mortality and the extent to which it relates to pre-diagnosed or new non-neoplastic diseases which could benefit from specialist diagnostic assessment and earlier diagnosis. As the components of non-cancer mortality due to pre-existing or new conditions are unclear, this should be addressed in future research. The current approach to cancer referral uses a normative threshold applicable to patients of any age and with any symptoms, and the results highlight the importance of considering whether patients are likely to benefit from prompt diagnosis.

Second, whether current referral pathways are necessarily ideal. For example, many abdominal symptoms were strongly associated with lower GI, upper GI and gynaecological cancers, and some form of referral pathway offering combined multi-specialty assessment may be justified for patients with these symptoms. Further, symptoms were often strongly associated with less common cancers such as haematological neoplasms; however, due to the low incidence of these conditions, the absolute risk rarely or never reached 3%; optimal diagnostic management of these patients is clearly challenging. Our findings may be helpful in clarifying referral criteria for new non-specific cancer pathways.