Development of scoring models to predict early pancreatic cancer risk in new-onset diabetes mellitus patients
Highlight box
Key findings
• By including routine clinical variables, we developed and validated new risk models to predict 1- and 3-year pancreatic cancer (PC) risk among new-onset diabetes mellitus (NODM) patients.
• Our new risk models had high accuracy in predicting 1- and 3-year PC risk with an area under receiver operating curve of 0.90 and 0.81, respectively.
What is known and what is new?
• Although NODM is a risk factor of PC, screening for all NODM patients is not cost-effective. However, there are few risk models to predict PC risk at three years after diagnosis of NODM, and no models to predict PC risk at one year
• We developed new risk models to predict 1- and 3-year PC risk with high accuracy.
What is the implication, and what should change now?
• Our new risk models could be utilized to inform prioritization of high-risk NODM patients for PC screening, in particular Asian populations.
Introduction
Pancreatic cancer (PC) is the sixth leading cause of cancer deaths worldwide (1) with a poor prognosis and an overall 5-year survival rate of 5–15% worldwide (2), which is largely due to delay in diagnosis with 85% being unresectable at diagnosis (3). The incidence and mortality of PC however has been increasing over time (4), and it is projected that PC cases and deaths would increase by 77.7% and 79.9% from 2018 to 2040, respectively (1).
While the majority of early-stage PC patients are asymptomatic, one of the early clues is new-onset diabetes mellitus (NODM) development. PC could lead to diabetes mellitus (DM) development due to disruption of endocrine functions of the pancreas, a para-neoplastic phenomenon caused by tumor-secreted products, as well as pancreatic ductal obstruction leading to glandular atrophy (5,6). Approximately 40% of PC patients developed DM within 36 months preceding cancer diagnosis (7). Among patients who develop DM at or after age 50 years, the 3-year PC incidence rate is 0.8% to 1.0%, which is 6- to 10-fold higher than that of general population (8). Another study showed that nearly 60% of PC in NODM occurred within 12 months of its onset (9,10). Therefore, targeting patients with NODM may provide a golden opportunity to screen for PC at its early stage.
However, given the large number of DM patients and low incidence of PC cases, universal PC screening in all DM patients is neither feasible nor cost-effective, particularly when there are no simple screening methods other than cross-sectional imaging or endoscopic ultrasonography. A risk stratification tool for patients with NODM may therefore offer benefit by informing prioritization of screening for high-risk groups. Although several risk models to predict PC have been developed (11), few were designated for NODM patients (Table S1) (12-18). For instance, an Enriching New-onset Diabetes for Pancreatic Cancer (END-PAC) model score of ≥3 has sensitivity of 78% and specificity of 85% for predicting PC within three years of NODM (14). However, the sensitivity of END-PAC drops to 73% for predicting PC at 6–12 months. Moreover, END-PAC model was developed on predominantly Caucasian population. There are currently no other risk models that specifically predict PC risk as early as one year after NODM, particularly in the Asian populations.
This study aimed to develop risk models based on Asian patients in Hong Kong, China for incident PC within 1- and 3-year after NODM, and to compare our risk models with other existing risk models. We present this article in accordance with the TRIPOD reporting checklist (available at https://hbsn.amegroups.com/article/view/10.21037/hbsn-2024-743/rc).
Methods
Data source
This study extracted data of patients in Hong Kong, China from the territory-wide electronic health database of the Clinical Data Analysis and Reporting System (CDARS) of the Hospital Authority (HA). The CDARS is a constantly updated database that captures clinical data from the healthcare system of all public hospitals and clinics in Hong Kong since its inception in 1993. Prior local studies have validated the coding accuracy in the CDARS and reported positive predictive values (PPVs) of >85% for clinical diagnoses (19,20). Further details of CDARS could be found in the supplementary material. Approval from the Institutional Review Board of the University of Hong Kong and the West Cluster of Hospital Authority was obtained for this study (UW 23-102). Patients’ consent was waived as it was a retrospective study of deidentified patient’s data. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments.
Study population
All adult patients with biochemically confirmed NODM, diagnosed between January 2008 and December 2017, in Hong Kong were identified from the CDARS. DM was diagnosed by a fasting glucose level of ≥7 mmol/L and/or a hemoglobin A1c (HbA1c) level of ≥6.5% for two consecutive times, which were measured within 3 months. The initial occurrence that fulfilled the criteria was considered, and date of the second measurement (i.e., DM diagnosis) was regarded as the index date. Patients were followed for three years until last recorded date, PC development, pancreatectomy, death, or 31 December 2020, whichever was earliest. Exclusion criteria included (I) type 1 DM [determined by International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) code of 250.*1 and 250.*3], (II) DM (including use of anti-diabetics) before the index date, (III) prior history of PC, pancreatic cyst, chronic pancreatitis or pancreatectomy (dated back to 1993 at the inception of the data registry). Presence of PC, pancreatic cyst, chronic pancreatitis or pancreatectomy was identified based on ICD-9-CM codes and/or histology data (Table S2). Figure 1 shows the patient selection flow diagram for development of risk models.
Outcomes of interest
The outcome of interest was incidence of PC at 1- and 3-year after index date. Diagnosis of PC was based on ICD-9-CM code of 157 and/or histologically confirmed diagnosis. One- and 3-year risk prediction models for PC among DM group were developed.
Predictive variables
Patients’ demographics and lifestyle information were collected, including age, sex, and smoking status. Smoking status was defined if the patient reported himself as a current smoker or ex-smoker, or if the patient had been diagnosed with chronic obstructive pulmonary disease (COPD), based on the evidence that smoking accounts for most of COPD cases (21). Clinical variables included body mass index (BMI) (22), body weight, blood glucose, lipid profile [including total cholesterol, low-density lipoprotein cholesterol (LDL-C), high-density lipoprotein cholesterol (HDL-C) and triglyceride], estimated glomerular filtration rate (eGFR), alanine transaminase (ALT), alkaline phosphatase (ALP) and hemoglobin (23). Blood glucose was measured by fasting glucose and/or estimated average glucose (EAG), in which EAG was calculated as (28.7 × HbA1c − 46.7) × 0.0555. The eGFR was converted from creatinine through the CKD-EPI equations (24). All clinical variables were measured at index date and one year prior to baseline. Clinical variables at index date were defined as the closest value to the index date from measurements taken 3 months before or on the index date; while BMI and weight at index date derived from measurements taken 3 months before or after the index date, as they are less likely to have rapid changes. Clinical variables at 1 year prior were defined as the closest value to 1 year before the index date among measurements taken between 36 and 3 months prior to the index date, so as to capture more data. Changes in values of clinical variables were also calculated by subtracting values of 1 year prior to index date from values at index date. Use of metformin, insulin and other anti-diabetic drugs up to 1 month after the index date were recorded (12). Other medications included aspirin, non-steroidal anti-inflammatory drugs (NSAIDs), statins and gastroprotective agents [proton pump inhibitors (PPIs) and histamine-2 receptor antagonists (H2RAs)] before or at index date. Use of gastroprotective agents may serve as a surrogate marker of early PC as epigastric pain or dyspepsia may be initial presentation of early PC (12). History of acute pancreatitis was also recorded.
Model development and statistical analysis
Data were extracted and analyzed using R version 4.3.1. We compared characteristics using Chi-squared test for categorical variables, and t-test for continuous variables. The NODM cohort was randomly split in 7:3 ratio into a training set (to develop the prediction model) and a testing set, respectively. Assuming data were missing at random (MAR), we first conducted multiple imputation for factors with <60% missing values, including outcome and candidate predictors in the imputation model to improve the MAR assumption, and created five imputed datasets (25). Detailed description of multiple imputation was in Supplementary file (Appendix 1).
For model development, we performed logistic regression analysis with forward selection for all five imputed training datasets, using the function “psfmi_lr” in R. Predictors were added to the model if the P value was <0.05. Model parameters were pooled based on Rubin’s rules. Risk prediction nomogram was developed to visualize the results. Stepwise Cox regression with forward selection (using the function “psfmi_coxer” in R) was also conducted, however, due to the violation of proportional hazards assumption and the relatively short follow-up period, logistic regression models were chosen for the current analysis (Table S3).
Model performance against testing set was assessed by examining discrimination and calibration and using 1,000 bootstrap samplings. The bootstrapping method evaluates the model’s stability and generalizability across various sample subsets. Discrimination was measured by area under the receiver operating characteristics curve (AUROC) and area under the precision-recall curve (AUPRC) pooled by Rubin’s rules (26). Calibration was assessed by Hosmer-Lemeshow goodness-of-fit test and graphically depicted in calibration plot, which examined agreement between mean predicted and observed risks over imputations. We also reported sensitivity, specificity, PPV, negative predictive value (NPV), F1 score using different specificity cut-offs in imputed testing set, to provide more comprehensive and intuitive measures for imbalanced datasets and to minimize the false positives in clinical practice. Number-needed-to-screen (NNS), which estimates number of high-risk NODM patients the model flags to identify one case, was reported. NNS was defined as ratio of inverse of prevalence of PC among high-risk NODM patients (i.e., inverse of PPV) to sensitivity.
We further compared our models with END-PAC model (using their scoring system) (14) and THIN model (12). Though existing models were developed for 3-year PC risk, we additionally validated them for 1-year PC risk for comparison with our 1-year model. The AUROCs were compared using Delong test, and the AUPRCs compared using bootstrap method.
Two sensitivity analyses were conducted: (I) only considering those who had both ICD and histological diagnosis of PC as outcome; (II) excluding people who died within 1 or 3 years of follow-up without PC. A two-sided P value of <0.05 was used to define statistical significance.
Results
Patient characteristics
A total of 117,121 NODM patients were identified. The mean age of whole cohort was 61.63 years [standard deviation (SD) 12.45], and 51.3% were males. Data completion rates of all predictors were at least 40%, except for HbA1c 1 year prior to index date (21%) (Table S4). Table 1 and Table S5 summarize patient characteristics after and before multiple imputation, respectively. In this cohort, 189 (0.16%) patients developed PC within three years, of whom 82 (43.4% of all PC cases) were diagnosed in the first year (Figure S1). Based on imputed dataset, NODM patients who developed PC within 3 years were more likely to be male, older, smoker, taking anti-diabetic drugs and gastroprotective agents, had prior acute pancreatitis, lower baseline BMI/weight, larger drop in BMI/weight 1 year prior to baseline, lower baseline total cholesterol, triglyceride and hemoglobin level, higher baseline eGFR/ALP/ALT with larger increase in eGFR/ALP/ALT from one year prior to baseline, but less likely to take aspirin.
Table 1
| Variables | Overall (n=117,121) | 1-year model | 3-year model | |||||
|---|---|---|---|---|---|---|---|---|
| No PC (n=117,039) | PC (n=82) | P | No PC (n=116,932) | PC (n=189) | P | |||
| Age (years) | 61.63 (12.45) | 61.62 (12.45) | 67.40 (10.59) | <0.001 | 61.61 (12.45) | 68.80 (10.61) | <0.001 | |
| Male | 60,118 (51.3) | 60,073 (51.3) | 45 (54.9) | 0.12 | 60,005 (51.3) | 113 (59.8) | <0.001 | |
| Acute pancreatitis | 590 (0.5) | 587 (0.5) | 3 (3.7) | <0.001 | 584 (0.5) | 6 (3.2) | <0.001 | |
| Smoker | 41,505 (35.4) | 41,468 (35.4) | 37 (45.1) | <0.001 | 41,426 (35.4) | 79 (41.8) | <0.001 | |
| Insulin | 1,856 (1.6) | 1,845 (1.6) | 11 (13.4) | <0.001 | 1,842 (1.6) | 14 (7.4) | <0.001 | |
| Other antidiabetic drug | 5,153 (4.4) | 5,136 (4.4) | 17 (20.7) | <0.001 | 5,130 (4.4) | 23 (12.2) | <0.001 | |
| Metformin | 22,056 (18.8) | 22,033 (18.8) | 23 (28.0) | <0.001 | 22,010 (18.8) | 46 (24.3) | <0.001 | |
| Statin | 37,734 (32.2) | 37,710 (32.2) | 24 (29.3) | 0.16 | 37,663 (32.2) | 71 (37.6) | <0.001 | |
| Aspirin | 27,345 (23.3) | 27,332 (23.4) | 13 (15.9) | <0.001 | 27,297 (23.3) | 48 (25.4) | 0.10 | |
| Non-steroidal anti-inflammatory drugs | 9,741 (8.3) | 9,733 (8.3) | 8 (9.8) | 0.25 | 9,730 (8.3) | 11 (5.8) | 0.002 | |
| Gastrointestinal drug | 36,648 (31.3) | 36,601 (31.3) | 47 (57.3) | <0.001 | 36,563 (31.3) | 85 (45.0) | <0.001 | |
| Histamine 2 receptor antagonists | 28,909 (24.7) | 28,874 (24.7) | 35 (42.7) | <0.001 | 28,844 (24.7) | 65 (34.4) | <0.001 | |
| Proton pump inhibitors | 10,724 (9.2) | 10,702 (9.1) | 22 (26.8) | <0.001 | 10,692 (9.1) | 32 (16.9) | <0.001 | |
| BMI at baseline (kg/m2) | 26.92 (3.72) | 26.93 (3.72) | 25.10 (3.54) | <0.001 | 26.93 (3.72) | 25.65 (3.49) | <0.001 | |
| ΔBMI from 1 year prior to baseline (kg/m2) | −0.02 (1.20) | −0.02 (1.20) | −0.69 (1.22) | <0.001 | −0.02 (1.20) | −0.41 (1.22) | <0.001 | |
| Glucose at baseline (mmol/L) | 7.19 (2.19) | 7.19 (2.19) | 8.93 (3.29) | <0.001 | 7.19 (2.19) | 8.03 (2.75) | <0.001 | |
| Δ glucose from 1 year prior to baseline (mmol/L) | 1.01 (2.11) | 1.01 (2.10) | 2.69 (3.45) | <0.001 | 1.01 (2.10) | 1.85 (2.79) | <0.001 | |
| Weight at baseline (kg) | 68.70 (12.20) | 68.70 (12.20) | 63.54 (11.52) | <0.001 | 68.70 (12.20) | 65.20 (11.54) | <0.001 | |
| Δ weight from 1 year prior to baseline (kg) | −0.11 (3.30) | −0.11 (3.30) | −1.43 (3.47) | <0.001 | −0.11 (3.30) | −1.08 (3.36) | <0.001 | |
| LDL-C at baseline (mmol/L) | 3.12 (1.02) | 3.12 (1.02) | 3.04 (1.13) | 0.08 | 3.12 (1.02) | 3.08 (0.99) | 0.20 | |
| Δ LDL-C from 1 year prior to baseline (mmol/L) | −0.04 (1.17) | −0.04 (1.17) | −0.03 (1.17) | 0.94 | −0.04 (1.17) | −0.08 (1.08) | 0.22 | |
| HDL-C at baseline (mmol/L) | 1.24 (0.35) | 1.24 (0.35) | 1.21 (0.39) | 0.08 | 1.24 (0.35) | 1.22 (0.35) | 0.044 | |
| Δ HDL-C from 1 year prior to baseline (mmol/L) | 0.00 (0.22) | 0.00 (0.22) | −0.08 (0.29) | <0.001 | 0.00 (0.22) | −0.03 (0.26) | <0.001 | |
| Total cholesterol at baseline (mmol/L) | 5.17 (1.17) | 5.17 (1.17) | 4.96 (1.18) | <0.001 | 5.17 (1.17) | 5.01 (1.09) | <0.001 | |
| Δ Total cholesterol from 1 year prior to baseline (mmol/L) | −0.04 (1.30) | −0.04 (1.30) | −0.08 (1.24) | 0.32 | −0.04 (1.30) | −0.09 (1.19) | 0.16 | |
| Triglycerides at baseline (mmol/L) | 1.82 (1.25) | 1.82 (1.25) | 1.58 (0.85) | <0.001 | 1.82 (1.25) | 1.59 (0.83) | <0.001 | |
| Δ Triglyceride from 1 year prior to baseline (mmol/L) | −0.01 (1.13) | −0.01 (1.13) | 0.05 (0.96) | 0.33 | −0.01 (1.13) | 0.03 (0.94) | 0.29 | |
| Haemoglobin at baseline (g/dL) | 13.71 (1.72) | 13.71 (1.72) | 12.96 (1.71) | <0.001 | 13.71 (1.72) | 13.33 (1.65) | <0.001 | |
| Δ Haemoglobin from 1 year prior to baseline (g/dL) | −0.03 (1.31) | −0.03 (1.31) | −0.69 (1.60) | <0.001 | −0.03 (1.31) | −0.30 (1.52) | <0.001 | |
| eGFR at baseline (mL/min/1.73 m2) | 82.29 (20.74) | 82.29 (20.74) | 86.82 (16.92) | <0.001 | 82.29 (20.74) | 81.39 (17.86) | 0.13 | |
| ΔeGFR from 1 year prior to baseline (mL/min/1.73 m2) | −0.02 (10.55) | −0.03 (10.55) | 2.62 (9.96) | <0.001 | −0.03 (10.55) | 1.64 (9.36) | <0.001 | |
| ALP at baseline (U/L) | 80.82 (35.84) | 80.73 (35.12) | 211.08 (241.82) | <0.001 | 80.73 (35.12) | 137.85 (172.71) | <0.001 | |
| Δ ALP from 1 year prior to baseline (U/L) | 1.91 (32.79) | 1.84 (32.05) | 112.71 (240.25) | <0.001 | 1.83 (32.05) | 52.52 (167.31) | <0.001 | |
| ALT at baseline (U/L) | 33.97 (50.20) | 33.94 (50.14) | 75.09 (97.63) | <0.001 | 33.95 (50.16) | 49.92 (70.33) | <0.001 | |
| Δ ALT from 1 year prior to baseline (U/L) | 1.51 (56.73) | 1.48 (56.68) | 42.04 (98.74) | <0.001 | 1.48 (56.70) | 17.59 (71.92) | <0.001 | |
Data are presented as mean (standard deviation) or n (%). ALP, alkaline phosphatase; ALT, alanine aminotransferase; BMI, body mass index; eGFR, estimated glomerular filtration rate; HDL-C, high-density lipoprotein cholesterol; LDL-C, low-density lipoprotein cholesterol; PC, pancreatic cancer.
There was a trend favoring a higher rate of pancreatectomy (a surrogate marker of less advanced cancer stage) among patients diagnosed with PC within 1 year of NODM compared with those diagnosed with PC between 1 and 3 years of NODM [20.7% (17 of 82) vs. 11.2% (12 of 107); P=0.11]. As for 1-year survival, 54 (65.8%) and 78 (72.9%) in the former and latter group died within 1 year after PC diagnosis, respectively (P=0.38); the median time of survival within 1 year was 5.53 and 3.26 months, respectively (P=0.23).
Development of risk models for PC after diagnosis of NODM
Logistic regression with forward stepwise selection was conducted in the training set to build prediction models for PC among NODM patients (Table 2). 1-year prediction model included age, use of insulin, aspirin and gastroprotective agents, baseline glucose, eGFR, ALP and ALT, and BMI change from 1 year (prior to baseline). NODM patients who were older [odds ratio (OR): 1.08; 95% confidence interval (CI): 1.05–1.12], used insulin (OR: 2.91; 95% CI: 1.12–7.61) and gastroprotective agents (OR: 3.40; 95% CI: 1.90–6.06), with higher baseline glucose (OR: 1.12; 95% CI: 1.05–1.20), eGFR (OR: 1.04; 95% CI: 1.01–1.06), ALP (OR: 1.005; 95% CI: 1.004–1.007), and ALT (OR: 1.002; 95% CI: 1.001–1.003) were more likely to develop PC within 1 year following NODM. On the other hand, NODM patients who had aspirin use (OR: 0.33; 95% CI: 0.16–0.71) and larger increase of BMI from 1 year prior to baseline (OR: 0.81; 95% CI: 0.74–0.90) were less likely to develop PC within 1 year. Risk algorithm of 1-year model was as follows:
Table 2
| Parameters | 1-year model | 3-year model | |||||
|---|---|---|---|---|---|---|---|
| Estimate | OR (95% CI) | P value | Estimate | OR (95% CI) | P value | ||
| (Intercept) | −17.429252 | <0.001 | −11.735245 | <0.001 | |||
| Age | 0.079446 | 1.083 (1.049, 1.117) | <0.001 | 0.069605 | 1.072 (1.051, 1.093) | <0.001 | |
| Female | −0.469378 | 0.625 (0.437, 0.894) | 0.01 | ||||
| History of acute pancreatitis | 1.256417 | 3.513 (1.101, 11.208) | 0.03 | ||||
| Insulin | 1.069337 | 2.913 (1.115, 7.612) | 0.03 | ||||
| Aspirin | −1.098602 | 0.333 (0.156, 0.714) | 0.005 | ||||
| Gastroprotective agents | 1.222557 | 3.396 (1.902, 6.062) | <0.001 | ||||
| Glucose at baseline | 0.112956 | 1.120 (1.046, 1.199) | 0.001 | 0.084842 | 1.089 (1.034, 1.146) | 0.001 | |
| BMI at baseline | −0.080945 | 0.922 (0.855, 0.994) | 0.04 | ||||
| ΔBMI from 1 year prior to baseline | −0.206535 | 0.813 (0.735, 0.901) | <0.001 | −0.148898 | 0.862 (0.787, 0.944) | 0.002 | |
| eGFR at baseline | 0.037461 | 1.038 (1.013, 1.064) | 0.004 | 0.023994 | 1.024 (1.010, 1.039) | 0.002 | |
| ALP at baseline | 0.005154 | 1.005 (1.004, 1.007) | <0.001 | 0.004456 | 1.004 (1.003, 1.006) | <0.001 | |
| ALT at baseline | 0.001996 | 1.002 (1.001, 1.003) | <0.001 | 0.001619 | 1.002 (1.001, 1.003) | 0.002 | |
The formula of the resulting logistic model is: P = e(Xβ)/1+ e(Xβ). The risk algorithm of the resulting Cox model is: predicted risk (as a percentage) = 100 × [1-Sexp(P)]. For 1 year: Xβ1-year = 0.079446 × Age + 1.069337 × Insulin − 1.098602 × Aspirin + 1.222557 × Gastroprotective agents + 0.112956 × Glucose at baseline − 0.206535 × ΔBMI from 1 year prior to baseline + 0.037461 × eGFR at baseline + 0.005154 × ALP at baseline + 0.001996 × ALT at baseline − 17.429252. For 3 years: Xβ3-year = 0.069605 × Age − 0.469378 × Female + 1.256417 × History of acute pancreatitis + 0.084842 × Glucose at baseline − 0.080945 × BMI at baseline − 0.148898 × ΔBMI from 1 year prior to baseline + 0.023994 × eGFR at baseline + 0.004456 × ALP at baseline + 0.001619 × ALT at baseline − 11.735245. ALP, alkaline phosphatase; ALT, alanine aminotransferase; BMI, body mass index; CI, confidence interval; eGFR, estimated glomerular filtration rate; OR, odds ratio.
As for 3-year model, predictive variables included age, sex, history of acute pancreatitis, baseline glucose, BMI, eGFR, ALP and ALT, and BMI change from 1 year prior to baseline (Table 2). Similar to the 1-year model, NODM patients who were older, had higher baseline glucose, eGFR, ALP and ALT were more likely develop PC within 1 year. Additionally, NODM patients who had acute pancreatitis (OR: 3.51; 95% CI: 1.10–11.2) were more likely to develop PC, while females (OR: 0.63; 95% CI: 0.44–0.89), and patients with higher baseline BMI (OR: 0.92; 95% CI: 0.86–0.99) had lower risk. The risk algorithm of 3-year model was as follow:
Corresponding risk nomograms for the 1- and 3-year models were constructed (Figure S2).
Risk model performance
Figure 2 shows performance of 1- and 3-year PC models against imputed testing sets. 1-year model (AUROC: 0.90, 95% CI: 0.84–0.97; AUPRC: 0.097, 95% CI: 0.045–0.241) and 3-year model (AUROC: 0.81, 95% CI: 0.75–0.86; AUPRC: 0.040, 95% CI: 0.019–0.103) both showed satisfactory discriminative ability. In Figure S3, calibration plot revealed satisfactory agreement between observed and predicted risks of PC at 1-year (P=0.30) and 3-year (P=0.37).
Table 3 shows sensitivity, specificity and NNS at different specificity thresholds, i.e., 0.80, 0.90, 0.95, 0.99 and 0.999, in imputed testing dataset. If we set specificity to be 99.9% and classified NODM patients with >1.68% predicted risk as high-risk, 14.63% of high-risk patients developed PC within 1 year, which was higher than overall prevalence of PC among all NODM patients [82/117,121=0.07%)]; and NNS to identify one PC was 27. If specificity threshold was set to 99.9% over 3 years, 14.6% of high-risk patients developed PC within 3 years [higher than overall prevalence among all NODM patients (0.16%)]; and NNS to identify one PC was 63.
Table 3
| Specificity threshold | Sensitivity | Risk threshold (%) | PPV (%) | NPV (%) | F1 (%) | Number needed to screen |
|---|---|---|---|---|---|---|
| 1-year PC | ||||||
| 0.80 | 0.875 | 0.07 | 0.298 | 99.989 | 0.593 | 384 |
| 0.90 | 0.750 | 0.12 | 0.509 | 99.981 | 1.012 | 262 |
| 0.95 | 0.625 | 0.17 | 0.846 | 99.973 | 1.669 | 190 |
| 0.99 | 0.458 | 0.45 | 3.039 | 99.963 | 5.699 | 72 |
| 0.999 | 0.250 | 1.68 | 14.634 | 99.949 | 18.462 | 27 |
| 3-year PC | ||||||
| 0.80 | 0.636 | 0.21 | 0.496 | 99.929 | 0.984 | 317 |
| 0.90 | 0.473 | 0.30 | 0.735 | 99.908 | 1.447 | 288 |
| 0.95 | 0.364 | 0.40 | 1.127 | 99.895 | 2.186 | 244 |
| 0.99 | 0.182 | 0.74 | 2.770 | 99.871 | 4.808 | 199 |
| 0.999 | 0.109 | 1.74 | 14.634 | 99.861 | 12.500 | 63 |
NPV, negative predictive value; PC, pancreatic cancer; PPV, positive predictive value.
Comparison with other models
END-PAC model was used to predict 3-year risk of PC in our cohort with an AUROC of 0.72 (95% CI: 0.65–0.78) and AUPRC of 0.017 (95% CI: 0.010–0.030) (Figure 2). We further used END-PAC score to predict 1-year PC risk, and AUROC was 0.74 (95% CI: 0.67–0.82) and AUPRC 0.023 (95% CI: 0.018–0.064). END-PAC score could not be used to create calibration plots as it had already defined patients with score of ≥3 as high-risk. THIN model showed satisfactory discriminative power for predicting 1-year (AUROC: 0.83, 95% CI: 0.74–0.93; AUPRC: 0.096, 95% CI: 0.033–0.242) and 3-year (AUROC: 0.73, 95% CI: 0.66–0.80; AUPRC: 0.049, 95% CI: 0.018–0.117) risk of PC in our cohort. In terms of AUROC among the three models, our 1-year model had a better discriminatory power than END-PAC model, and our 3-year model was better than END-PAC and THIN models, while the P values for AUPRCs were not significant (Table S6). However, the THIN model was poorly calibrated with significant deviations between the predicted and observed probabilities (P<0.001) in our data (Figure S4).
Sensitivity analysis
In sensitivity analysis, we censored patients without histological confirmation (patient characteristics shown in Table S7), and found that only 16 and 30 PC patients had both ICD and histological diagnosis at one and three years after NODM, respectively. Both 1- and 3-year models included history of acute pancreatitis, baseline BMI and ALP, and glucose, eGFR and BMI change from 1 year prior to baseline (Table S8). Our models showed similar performance with existing models, but all had wide 95% CIs (Figures S5,S6 and Table S9).
Moreover, 1,391 (1.19%) and 4,189 (3.58%) patients who died within 1 and 3 years of follow-up without PC were identified, respectively. We additionally conducted a sensitivity analysis excluding these patients (patient characteristics shown in Table S10). The predictive algorithms included similar sets of predictors (Table S11) and showed similar model performance compared with the main analysis, which further indicated that our models were robust in predicting 1- and 3-year PC following NODM (Figures S7,S8 and Table S12).
Discussion
In this territory-wide cohort study from Hong Kong, the cumulative incidence of PC in first three years following NODM was 0.16% in patients aged 18 years or older. Prediction models with risk nomograms were developed to estimate probability of developing PC within 1- and 3-year after NODM with good predictive performance. Given the relatively low incidence of PC, screening all NODM patients is not cost-effective and our models could serve as risk stratification tools to inform resource prioritization.
In our study, 3-year cumulative incidence of PC among NODM was 0.16%, which was slightly lower than the rates reported in a national cohort of Veterans in the US (0.25%) (27), a population representative database in the UK (0.4%) (12), and a nationwide population-cohort study in Denmark (0.6%) (28) as well as those reported by Maitra et al. from an assemble of NODM patients in clinical centers (0.85–1%) (29). This is in keeping with the observation that highest incidence of PC occurred in North America, Europe and Australia due to concurrent metabolic risk factors (such as smoking, obesity, diabetes and alcohol intake) (30); while Hong Kong had a relatively low prevalence of obesity and low smoking and alcohol consumption (31). Our lower incidence of PC could also be attributed to the fact that we included all adult patients aged over 18 years, while other studies included only patients aged over 40 or 50 years. Among these NODM patients with PC, we observed that 43% of PC cases were diagnosed within first year of NODM, consistent with previous findings (9). This supports our rationale of using 1-year period, in addition to 3-year period, in establishing prediction models, to facilitate early identification of early-stage PC. Previous studies showed that growth rate of tumor in untreated PC varied widely, from less than a month to more than 4 years (32); and progression from T1 to T4 could occur in an average of 14 months (33). Therefore, early detection of PC is critical for improving prognosis, and our 1-year model could achieve this goal (34).
Currently, there are few existing models to predict PC in NODM (12-18), with END-PAC being most commonly quoted. However, the END-PAC model was based on a cohort of predominantly Caucasian. Our risk models were developed from a territory-wide cohort of Asian patients with NODM and were based on routinely available clinical data. The AUROC of was 0.90 and 0.81, and AUPRC 0.01 and 0.04 for 1- and 3-year PC risk, respectively. Unlike END-PAC score that divided predictors into categories, our model used continuous variables, which provided more exact and precise risk of PC. Performance of END-PAC model was less satisfactory with AUROC of 0.74 and 0.72, respectively, in our local population, when compared with AUROC of 0.82–0.87 in their Western cohort (14). It is noteworthy that END-PAC model was derived from a relatively small cohort (n=1,561), and therefore the results would need further validation in other cohorts. As for THIN model, AUROC was 0.83 and 0.73 for 1- and 3-year PC risk, respectively, which are still lower than that of our models; particularly, the models were poorly calibrated in our dataset. Although the statistical tests for comparing model performance showed no significant differences between discriminative powers in terms of AUPRC, it may be because our outcome events were too rare, which also caused their wide 95% CIs. Further studies to capture more outcome events, are warranted to validate both our models and the existing models. Two recent studies (16,18) also utilized clinical parameters which showed an AUROC of 0.73–0.74 for predicting PC among NODM patients. Remarkably, another study incorporating five clinical predictors and 24 single nucleotide polymorphisms achieved an AUROC of 0.90 (17). These models were not validated in our dataset due to data unavailability.
The advantage of our model is allowing flexible risk stratification in real-world application. We might further fine tune risk threshold at different levels. By using 99.9% as specificity cutoff to minimize false positives, NNS to identify one PC was 27 and 63 at 1- and 3-year, respectively.
Our 1- and 3-year models included age, sex, history of acute pancreatitis, use of insulin, aspirin and gastroprotective agents, baseline glucose, eGFR, ALP and ALT, as well as BMI (both baseline and BMI drop). Notably, age, weight loss and glucose level have received particular attention as potential early indicators of PC, and were included in previous PC risk models (12,35,36). Their risk levels observed in our studies were in line with previous findings, that PC patients tended to be older, had higher blood glucose, rapid weight loss and hence lower BMI (37). As compared to established positive association of BMI with DM and PC risk (38), paradoxical decrease of BMI in patients with PC and NODM might be due to cancer related cachexia (39).
Baseline ALT level may be a surrogate marker of metabolic dysfunction-associated steatotic liver disease (MASLD), which additionally reflects underlying insulin resistance. Hyperinsulinemia and increased circulating insulin growth factor-1 (IGF-1) may promote cancer growth (40,41). A previous study also showed PC-associated DM cases had a higher ALT level than conventional DM (42). Similarly, ALP increase also confers a higher PC risk, in concordance with THIN model (12). ALP has been proposed to have a pro-proliferative role at cell level (43), and recognized as prognostic factor of lymph node involvement and poor PC outcomes (44,45). Higher eGFR level was also demonstrated in PC-associated DM as compared to conventional DM (12,42). Acute pancreatitis is likely due to tumor-induced obstructive pancreatitis (46), with diagnosis <1 and >2 years being associated with odds ratio of 21.4 and 2.7, respectively (47). In addition, short-term (<5 years) use of insulin was linked with higher risk, but not for longer duration of use (≥15 years) (48), which could be related to large fluctuation in glucose levels in those with concurrent PC. A meta-analysis showed that aspirin use was associated with 18% lower risk of PC (49). Use of gastroprotective agents was risk factor of PC, as shown in THIN model (12). A meta-analysis showed that PPI use was associated with 1.8-fold higher PC risk (50). PPIs induced hypergastrinemia, correlating with increase in pancreatic intraepithelial neoplasia (PanIN) grade and development of microinvasive cancer in p48-Cre/LSL-KrasG12D mice that developed PanIN (51). While reverse causality (undiagnosed PC presenting with epigastric pain/dyspepsia leading to drug use) and residual confounding could be a reason, use of gastroprotective agents as a surrogate marker of undiagnosed early PC was useful predictive factor.
Our study has limitations. First, family history, genetic susceptibility and other lifestyle factors (such as alcohol consumption) were not considered due to data unavailability. However, our prediction models were based on electronically available clinical variables. Particularly, although smoking status was considered, there were multiple missing values. The data on smoking status was mainly based on patients’ self-reports and supplemented by COPD diagnosis, which may introduce potential bias on estimating its effect on PC development. Second, we also had missing data among several predictive variables. Assuming data were missing at random, we addressed missing values by multiple imputation, based on the evidence that multiple imputation can provide accurate estimation of a logistic regression model even when the proportion of missingness is high, particularly for data with a large sample size (52). In view of the large population-based cohort of NODM patients in this study, the absolute count of available values of clinical factors remained substantial despite missing values. Some studies also suggested that the proportion of missingness should not be used to guide decisions on multiple imputation (53). Further studies with less missing values are warranted to validate our models. Third, our prediction models were derived from Chinese, and therefore generalizability and applicability to other ethnic groups needs to be validated in future studies. Fourth, this model was based on clinical diagnosis of PC and could still miss early or slow-growing PC. Lastly, cancer staging data was not available. We could only use pancreatectomy rate and 1-year survival as surrogate markers to compare the potential difference in detecting PC at 1- and 3-year.
Conclusions
Of all new PC cases that were detected within three years of NODM, 43% were diagnosed within the first year. We developed and validated prediction models with risk nomograms based on routine clinical variables for PC development within 1- and 3-year of NODM. The prediction models could be utilized to inform prioritization of high-risk groups for PC screening.
Acknowledgments
None.
Footnote
Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://hbsn.amegroups.com/article/view/10.21037/hbsn-2024-743/rc
Data Sharing Statement: Available at https://hbsn.amegroups.com/article/view/10.21037/hbsn-2024-743/dss
Peer Review File: Available at https://hbsn.amegroups.com/article/view/10.21037/hbsn-2024-743/prf
Funding: None.
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://hbsn.amegroups.com/article/view/10.21037/hbsn-2024-743/coif). E.Y.F.W. reports grants from Health Bureau, HKSAR, China; Hong Kong Research Grant Council, HKSAR, China; Social Welfare Department, Labour and Welfare Bureau, HKSAR, China; National Natural Science Foundation of China, China; and Narcotics Division, Security Bureau, HKSAR, China (all payments were made to his institution). W.K.L. reports honoraria from AbbVie, Janssen, and Ferring. The other authors have no conflicts of interest to declare.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Approval from the Institutional Review Board of the University of Hong Kong and the West Cluster of Hospital Authority was obtained for this study (UW 23-102). Patients’ consent was waived as it was a retrospective study of deidentified patient’s data.
Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.
References
- Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018;68:394-424. [Crossref] [PubMed]
- Allemani C, Matsuda T, Di Carlo V, et al. Global surveillance of trends in cancer survival 2000-14 (CONCORD-3): analysis of individual records for 37 513 025 patients diagnosed with one of 18 cancers from 322 population-based registries in 71 countries. Lancet 2018;391:1023-75. [Crossref] [PubMed]
- van Veldhuisen E, van den Oord C, Brada LJ, et al. Locally Advanced Pancreatic Cancer: Work-Up, Staging, and Local Intervention Strategies. Cancers (Basel) 2019;11:976. [Crossref] [PubMed]
- Kavaliauskas P, Dulskas A, Kildusiene I, et al. Trends in Pancreatic Cancer Incidence and Mortality in Lithuania, 1998-2015. Int J Environ Res Public Health 2022;19:949. [Crossref] [PubMed]
- Salvatore T, Marfella R, Rizzo MR, et al. Pancreatic cancer and diabetes: A two-way relationship in the perspective of diabetologist. Int J Surg 2015;21:S72-7. [Crossref] [PubMed]
- Sah RP, Nagpal SJ, Mukhopadhyay D, et al. New insights into pancreatic cancer-induced paraneoplastic diabetes. Nat Rev Gastroenterol Hepatol 2013;10:423-33. [Crossref] [PubMed]
- Aggarwal G, Kamada P, Chari ST. Prevalence of diabetes mellitus in pancreatic cancer compared to common cancers. Pancreas 2013;42:198-201. [Crossref] [PubMed]
- Andersen DK, Korc M, Petersen GM, et al. Diabetes, Pancreatogenic Diabetes, and Pancreatic Cancer. Diabetes 2017;66:1103-10. [Crossref] [PubMed]
- Chari ST, Leibson CL, Rabe KG, et al. Probability of pancreatic cancer following diabetes: a population-based study. Gastroenterology 2005;129:504-11. [Crossref] [PubMed]
- Pannala R, Basu A, Petersen GM, et al. New-onset diabetes: a potential clue to the early diagnosis of pancreatic cancer. Lancet Oncol 2009;10:88-95. [Crossref] [PubMed]
- Santos R, Coleman HG, Cairnduff V, et al. Clinical Prediction Models for Pancreatic Cancer in General and At-Risk Populations: A Systematic Review. Am J Gastroenterol 2023;118:26-40. [Crossref] [PubMed]
- Boursi B, Finkelman B, Giantonio BJ, et al. A Clinical Prediction Model to Assess Risk for Pancreatic Cancer Among Patients With New-Onset Diabetes. Gastroenterology 2017;152:840-850.e3. [Crossref] [PubMed]
- Dong X, Lou YB, Mu YC, et al. Predictive Factors for Differentiating Pancreatic Cancer-Associated Diabetes Mellitus from Common Type 2 Diabetes Mellitus for the Early Detection of Pancreatic Cancer. Digestion 2018;98:209-16. [Crossref] [PubMed]
- Sharma A, Kandlakunta H, Nagpal SJS, et al. Model to Determine Risk of Pancreatic Cancer in Patients With New-Onset Diabetes. Gastroenterology 2018;155:730-739.e3. [Crossref] [PubMed]
- Baecker A, Kim S, Risch HA, et al. Do changes in health reveal the possibility of undiagnosed pancreatic cancer? Development of a risk-prediction model based on healthcare claims data. PLoS One 2019;14:e0218580. [Crossref] [PubMed]
- Cichosz SL, Jensen MH, Hejlesen O, et al. Prediction of pancreatic cancer risk in patients with new-onset diabetes using a machine learning approach based on routine biochemical parameters. Comput Methods Programs Biomed 2024;244:107965. [Crossref] [PubMed]
- Sun Y, Hu C, Hu S, et al. Predicting Pancreatic Cancer in New-Onset Diabetes Cohort Using a Novel Model With Integrated Clinical and Genetic Indicators: A Large-Scale Prospective Cohort Study. Cancer Med 2024;13:e70388. [Crossref] [PubMed]
- Ali S, Coory M, Donovan P, et al. Predicting the risk of pancreatic cancer in women with new-onset diabetes mellitus. J Gastroenterol Hepatol 2024;39:1057-64. [Crossref] [PubMed]
- Cheung KS, Chen L, Chan EW, et al. Statins reduce the progression of non-advanced adenomas to colorectal cancer: a postcolonoscopy study in 187 897 patients. Gut 2019;68:1979-85. [Crossref] [PubMed]
- Chan EW, Lau WC, Leung WK, et al. Prevention of Dabigatran-Related Gastrointestinal Bleeding With Gastroprotective Agents: A Population-Based Study. Gastroenterology 2015;149:586-95.e3. [Crossref] [PubMed]
- Global, regional, and national deaths, prevalence, disability-adjusted life years, and years lived with disability for chronic obstructive pulmonary disease and asthma, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet Respir Med 2017;5:691-706. [Crossref] [PubMed]
- Xu M, Jung X, Hines OJ, et al. Obesity and Pancreatic Cancer: Overview of Epidemiology and Potential Prevention by Weight Loss. Pancreas 2018;47:158-62. [Crossref] [PubMed]
- Sah RP, Sharma A, Nagpal S, et al. Phases of Metabolic and Soft Tissue Changes in Months Preceding a Diagnosis of Pancreatic Ductal Adenocarcinoma. Gastroenterology 2019;156:1742-52. [Crossref] [PubMed]
- Inker LA, Eneanya ND, Coresh J, et al. New Creatinine- and Cystatin C-Based Equations to Estimate GFR without Race. N Engl J Med 2021;385:1737-49. [Crossref] [PubMed]
- Moons KG, Donders RA, Stijnen T, et al. Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 2006;59:1092-101. [Crossref] [PubMed]
- Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One 2015;10:e0118432. [Crossref] [PubMed]
- Munigala S, Singh A, Gelrud A, et al. Predictors for Pancreatic Cancer Diagnosis Following New-Onset Diabetes Mellitus. Clin Transl Gastroenterol 2015;6:e118. [Crossref] [PubMed]
- Jensen MH, Cichosz SL, Hejlesen O, et al. Risk of pancreatic cancer in people with new-onset diabetes: A Danish nationwide population-based cohort study. Pancreatology 2023;23:642-9. [Crossref] [PubMed]
- Maitra A, Sharma A, Brand RE, et al. A Prospective Study to Establish a New-Onset Diabetes Cohort: From the Consortium for the Study of Chronic Pancreatitis, Diabetes, and Pancreatic Cancer. Pancreas 2018;47:1244-8. [Crossref] [PubMed]
- Klein AP. Pancreatic cancer epidemiology: understanding the role of lifestyle and inherited risk factors. Nat Rev Gastroenterol Hepatol 2021;18:493-502. [Crossref] [PubMed]
- Ni MY, Canudas-Romo V, Shi J, et al. Understanding longevity in Hong Kong: a comparative study with long-living, high-income countries. Lancet Public Health 2021;6:e919-31. [Crossref] [PubMed]
- Ahn SJ, Choi SJ, Kim HS. Time to Progression of Pancreatic Cancer: Evaluation with Multi-Detector Computed Tomography. J Gastrointest Cancer 2017;48:164-9. [Crossref] [PubMed]
- Yu J, Blackford AL, Dal Molin M, et al. Time to progression of pancreatic ductal adenocarcinoma from low-to-high tumour stages. Gut 2015;64:1783-9. [Crossref] [PubMed]
- Chari ST, Kelly K, Hollingsworth MA, et al. Early detection of sporadic pancreatic cancer: summative review. Pancreas 2015;44:693-712. [Crossref] [PubMed]
- Setiawan VW, Stram DO, Porcel J, et al. Pancreatic Cancer Following Incident Diabetes in African Americans and Latinos: The Multiethnic Cohort. J Natl Cancer Inst 2019;111:27-33. [Crossref] [PubMed]
- Huang BZ, Pandol SJ, Jeon CY, et al. New-Onset Diabetes, Longitudinal Trends in Metabolic Markers, and Risk of Pancreatic Cancer in a Heterogeneous Population. Clin Gastroenterol Hepatol 2020;18:1812-1821.e7. [Crossref] [PubMed]
- Lee JH, Kim SA, Park HY, et al. New-onset diabetes patients need pancreatic cancer screening? J Clin Gastroenterol 2012;46:e58-61. [Crossref] [PubMed]
- Incio J, Liu H, Suboj P, et al. Obesity-Induced Inflammation and Desmoplasia Promote Pancreatic Cancer Progression and Resistance to Chemotherapy. Cancer Discov 2016;6:852-69. [Crossref] [PubMed]
- Yoo W, Choi H, Son YH, et al. Pancreatic cancer induces muscle wasting by promoting the release of pancreatic adenocarcinoma upregulated factor. Exp Mol Med 2021;53:432-45. [Crossref] [PubMed]
- Hutchison AL, Tavaglione F, Romeo S, et al. Endocrine aspects of metabolic dysfunction-associated steatotic liver disease (MASLD): Beyond insulin resistance. J Hepatol 2023;79:1524-41. [Crossref] [PubMed]
- Yanai H, Adachi H, Hakoshima M, et al. Metabolic-Dysfunction-Associated Steatotic Liver Disease-Its Pathophysiology, Association with Atherosclerosis and Cardiovascular Disease, and Treatments. Int J Mol Sci 2023;24:15473. [Crossref] [PubMed]
- Yoon BH, Ang SM, Alabd A, et al. Pancreatic Cancer-Associated Diabetes is Clinically Distinguishable From Conventional Diabetes. J Surg Res 2021;261:215-25. [Crossref] [PubMed]
- Yamamoto K, Awogi T, Okuyama K, et al. Nuclear localization of alkaline phosphatase in cultured human cancer cells. Med Electron Microsc 2003;36:47-51. [Crossref] [PubMed]
- Xiao Y, Lu J, Chang W, et al. Dynamic serum alkaline phosphatase is an indicator of overall survival in pancreatic cancer. BMC Cancer 2019;19:785. [Crossref] [PubMed]
- Pu N, Gao S, Xu Y, et al. Alkaline Phosphatase-To-Albumin Ratio as a Prognostic Indicator in Pancreatic Ductal Adenocarcinoma after Curative Resection. J Cancer 2017;8:3362-70. [Crossref] [PubMed]
- Liu J, Wang Y, Yu Y. Meta-analysis reveals an association between acute pancreatitis and the risk of pancreatic cancer. World J Clin Cases 2020;8:4416-30. [Crossref] [PubMed]
- Duell EJ, Lucenteforte E, Olson SH, et al. Pancreatitis and pancreatic cancer risk: a pooled analysis in the International Pancreatic Cancer Case-Control Consortium (PanC4). Ann Oncol 2012;23:2964-70. [Crossref] [PubMed]
- Bosetti C, Rosato V, Li D, et al. Diabetes, antidiabetic medications, and pancreatic cancer risk: an analysis from the International Pancreatic Cancer Case-Control Consortium. Ann Oncol 2014;25:2065-72. [Crossref] [PubMed]
- Sun J, Li Y, Liu L, et al. Aspirin use and pancreatic cancer risk: A systematic review of observational studies. Medicine (Baltimore) 2019;98:e18033. [Crossref] [PubMed]
- Alkhushaym N, Almutairi AR, Althagafi A, et al. Exposure to proton pump inhibitors and risk of pancreatic cancer: a meta-analysis. Expert Opin Drug Saf 2020;19:327-34. [Crossref] [PubMed]
- Huber MA, Nadella S, Cao H, et al. Does Chronic Use of High Dose Proton Pump Inhibitors Increase Risk for Pancreatic Cancer? Pancreas 2022;51:1118-27. [Crossref] [PubMed]
- Austin PC, van Buuren S. The effect of high prevalence of missing data on estimation of the coefficients of a logistic regression model when using multiple imputation. BMC Med Res Methodol 2022;22:196. [Crossref] [PubMed]
- Madley-Dowd P, Hughes R, Tilling K, et al. The proportion of missing data should not be used to guide decisions on multiple imputation. J Clin Epidemiol 2019;110:63-73. [Crossref] [PubMed]

