Data set characteristics

In the full set of samples collected from the ADEPTS cohort, age at the time of sample collection, 57.44 (range from 19.00 to 93.00) for controls and 69.72 (range from 43.00 to 91.00) for PDAC cases, emerged as a risk factor (OR = 1.06 (95% CI 1.04–1.09), p = 2.47×10⁻⁷) (Table 1). As a predictor in a logistic regression model age achieved a ROC AUC of 0.73 (95% CI 0.66–0.79), with a cut-off at 61.5 years (calculated using the Youden’s J statistic). This finding was also observed in both the discovery (Fig 1 and Table A in S1 Appendix, ROC AUC 0.74 (95% CI 0.64–0.83), cut-off at 70) and validation sets (Fig 1 and Table B in S1 Appendix, 0.74 (95% CI 0.64–0.82), cut-off at 60), which incorporated not only ADEPTS samples but also healthy control samples collected from UKCTOCS [36]. In our past research which was focused exclusively on UKCTOCS longitudinal samples, age similarly emerged as a risk factor for PDAC [37]. Furthermore, gender (OR = 2.72 (95% CI 1.46–5.27), p = 0.0015) and ethnicity taken as a one-hot encoded variable (OR = 2.02 (95% CI 1.34–3.03), p = 6.56×10⁻⁴) were also confirmed as significantly associated with an increased risk of PDAC (Table 1). In the whole set of samples collected from the ADEPTS cohort, men had a 2.72-fold risk of PDAC compared to their female counterparts. Individuals of Caucasian ethnicity demonstrated a decreased risk of PDAC in a one versus rest calculation (OR = 0.38 (95% CI 0.20–0.69), p = 0.0018) and no significant association was found between PDAC risk and Asian or Afro-Caribbean ethnicity in the ADEPTS dataset under the same modelling framework (Table 1). The association of gender and PDAC was also confirmed in the discovery (OR = 4.98 (95% CI 2.08–13.50), p = 0.00023, Fig 1 and Table A in S1 Appendix) and validation sets (OR = 2.65 (95% CI 1.11–6.58), p = 0.028 Fig 1 and Table B in S1 Appendix), but ethnicity, taken as a one-hot encoded variable, remained a significant predictor of PDAC only in the validation set (OR = 2.66 (95% CI 1.42–5.17), p = 0.0020) (Fig 1 and Table B in S1 Appendix), which as was highlighted above also includes healthy control UKCTOCS samples. Within the group of the clinical covariates only age and gender are significant predictors of PDAC in both the discovery and validation set (Fig 1), with only age achieving a significant AUC in the validation set between these two. However, this was concomitant with remarkably low sensitivity (Sens), positive predictive (PPV) value and negative predictive value (NPV) at 90% specificity (Spec): AUC 0.74 (95% CI 0.64–0.82), Sens 0.13 (95% CI 0–0.39), PPV 0.16 (95% CI 0–0.36), NPV 0.88 (95% CI 0.86–0.91).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Characteristics of the discovery and validation sets.

Number of controls across the discovery and validation sets (A), number of PDAC cases per stage (B), and association of BMI, Age, Diabetes, Ethnicity and Gender with PDAC status (C-F). In C, D, E and F dot sizes correspond to odds ratios and are colour coded according to their respective values, i.e., blue if OR<1 and red if OR>1. p values were calculated according to a logistic regression model with a bias reduction method. Purple dashed lines correspond to -Log[0.05]. G Receiver Operating Characteristic (ROC) Area Under the Curve (AUC), Sensitivity (Sens), Positive Predictive Value (PPV) and Negative Predictive Value (NPV) at 90% Specificity (Spec) performance of single marker models, i.e. BMI and Age, in the validation set. H Similar to A but for Gender, Ethnicity and Diabetes. Performances were calculated with the respective single feature models developed in the discovery set. The ROC AUC significance threshold is also represented by a purple dashed line at 0.5. Error bars in figures corresponding to the validation set are the 95% Confidence Intervals (CI), calculated by stratified bootstrapping 2000 times. See Statistical Analysis in Materials and methods for further details and Tables A, B and N in S1 Appendix.

https://doi.org/10.1371/journal.pcbi.1012408.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Cohort characteristics.

The data set used to develop and test the classifiers is a combination of samples collected from the ADEPTS cohort and selected controls from the UKCTOCS cohort. BMI: Body Mass Index. See Study Design in Materials and methods section for additional details. Odds ratio (OR) and respective 95% confidence intervals are also provided in the p value column.

https://doi.org/10.1371/journal.pcbi.1012408.t001

Development of a PDAC biomarker signature in the presence of confounding conditions

To aid the early detection of this cancer in individuals at risk, we aimed to develop a biomarker signature that could be used to differentiate between suspected PDACs and benign biliary conditions that often overlap in clinical presentation. We applied a uniquely developed ensemble learning model, with a logistic regression stacking layer (see Fig A in S1 Appendix and statistical analysis in the Materials and methods section), to a set of 539 serum samples (493 controls and 46 PDAC cases) which were analysed using the Olink Oncology II panel as well as four additional biomarkers we previously reported on [37]. These included IL6ST, VWF, THBS2 and CA19-9. The oncogenic and prognostic glycolytic enzyme PKM2 was additionally selected based on our past report of its diagnostic utility in biliary tract cancer patients [38–40].

The application of stacked ensemble modelling as presented herein bolsters the robustness of predictive outcomes, enhancing the performance of biomarker panels through the incorporation of serum biomarker levels and relevant clinical covariates for distinct diagnostic classes. Each base classifier within the ensemble is designed to provide a specialized distinction between confounding diagnoses and PDAC, thereby establishing a heterogeneous set of classifiers that facilitates the precise identification of PDAC (see statistical analysis section in Materials and methods). Previous studies have attested to the beneficial role of ensemble methods in augmenting early detection of PDAC against only healthy controls [37]. The implementation of stacked (Stack, Fig 2), specialized classifiers, developed within the discovery set, generated a biomarker signature capable of predicting PDAC with an AUC of 0.98 (95% CI 0.98–0.99), sensitivity of 0.99 (95% CI 0.98–1), PPV 0.92 (95% CI 0.91–0.92) and NPV 0.99 (95% CI 0.97–1) at 90% specificity. In contrast, the predictive efficacy of CA19-9 in the discovery set taken as a single predictor under a logistic regression model was 0.79 (95% CI 0.66–0.91) (p = 0.0016 under a one-sided bootstrap test comparing the two AUCs), sensitivity 0.67 (95% CI 0.50–0.83), PPV 0.32 (95% CI 0.26–0.38) and NPV of 0.97 (95% CI 0.96–0.99) at 90% specificity (see Table C in S1 Appendix). Amongst all biomarkers, CA19-9 demonstrated the most significant association (refer to Table C and Fig B in S1 Appendix for univariate trend associations across the discovery set), and one of the highest performances in the validation set (Fig B and Table D in S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Performance of individual base-learner classifiers, stack ensemble and state-of-the art algorithms.

A Base-learners performance in the discovery set. Each base-learner classifier was developed by training with a recursive feature elimination technique (RFE) and logistic regression (glm) in samples belonging to each specific diagnosis class against the same 24 PDACs in the discovery set. The performance reported in A is, nevertheless, of each classifier in the whole discovery set. The performances reported in B correspond to the base-learners developed in the discovery set but applied to the whole validation set. In C and D the performance of the ensemble stack based on the base-learners presented in A and B, as well as of state-of-the-art algorithms (xgbTree, RRF and RFE glm) is reported in the discovery and held-out validation sets, respectively. xgbTree, RRF and RFE glm were trained in the whole discovery set, which contrasts with the ensemble algorithm. Receiver Operating Characteristic (ROC) Area Under the Curve (AUC), Sensitivity (Sens), Positive Predictive Value (PPV) and Negative Predictive Value (NPV) at 90% Specificity (Spec).

https://doi.org/10.1371/journal.pcbi.1012408.g002

In the validation set, the ensemble signature predicted PDAC with an AUC of 0.95 (95% CI 0.91–0.99), sensitivity 0.86 (95% CI 0.68–1), PPV 0.54 (95% CI 0.48–0.58) and NPV of 0.98 (95% CI 0.95–1) at 90% specificity. Once again, this is an improvement with respect to CA19-9 (p = 0.0082, one-sided bootstrap test) taken as a univariate model developed in the discovery set; this CA19-9 model predicted PDAC status with an AUC of 0.80 (95% CI 0.66–0.93), sensitivity 0.65 (95% CI 0.48–0.56), PPV 0.49 (95% CI 0.41–0.56) and NPV of 0.95 (95% CI 0.92–0.98) at 90% specificity in the validation set. If we further validate only on the benign disease controls and PDACs collected from ADEPTS, the diagnostic-specific ensemble signature achieved an AUC of 0.96 (95% CI 0.92–0.99), sensitivity 0.82 (95% CI 0.64–0.95) at 90% specificity. This performance is also significantly higher than the performance of CA19-9 in a univariate model: AUC of 0.79 (95% CI 0.64–0.93) (p = 0.013 when compared with the full signature, one-sided test) and sensitivity of 0.18 (95% CI 0.03–0.69).

A closer examination of the individual performances of each base-learner classifier (Fig 2A and 2B) reveals that the logistic regression stacked ensemble approach has superior performance in both discovery and validation sets. Despite the best base-learner being trained on samples diagnosed as ’Gastritis/Reflux Disease’ (Fig 2A and 2B), its performance was also superseded by the AUC computed with the stack model, the logistic regression coefficients of which are delineated in Table E in S1 Appendix. The stack model significantly relies on the “Healthy”, “Chronic Pancreatitis”, “IgG4 Disease”, “Irritable Bowel Syndrome”, ‘Other Biliary Duct Disease”, “Sphincter of Oddi Dysfunction”, “No Relevant Diagnosis”, “Other Cancer” and “Pancreatic Cyst” base-learners. Even though the remaining diagnostic class base-learners, including "Gastritis/Reflux Disease", did not reach statistical significance (p<0.05), employing a stack that solely resorts to significant base-learners led to a reduction in generalization capacity: AUC 0.98 (95% CI 0.97–0.99), sensitivity 0.98 (95% CI 0.95–1), PPV 0.92 (95% CI 0.91–0.92), NPV 0.97 (95% CI 0.94–0.99) in the discovery set; AUC 0.93 (95% CI 0.87–0.99), sensitivity 0.82 (95% CI 0.64–0.95), PPV 0.53 (95% CI 0.47–0.57), NPV 0.97 (95% CI 0.95–0.99) in the validation set. Although the differences are not substantial, we retain the full set of base-learners to enhance the generalization capacity for predicting PDAC in unseen data sets and new samples. In fact, upon following recursive base-learner elimination the best ensemble was always proven to be the full set of 16 base classifiers.

The employment of stacked diagnosis-specialized classifiers surpassed the AUC performance of state-of-the-art algorithms such as random forests (RRF) and extreme gradient boosting methods (xgbTree), in terms of AUC, sensitivity, positive predictive value, and negative predictive value at 90% specificity (Fig 2C and 2D); although the performance AUC of the stacked classifier was only marginally significantly higher than that obtained with RRF (p = 0.040, one-sided) and not significant when compared with xgbTree (p = 0.26, one-sided), the sensitivity values at 90% specificity obtained with the alternative methods were, in fact, significantly lower, p = 0.028 and p = 0.045, respectively. The ensemble also outperformed a logistic regression model with recursive feature elimination (Fig 2C and 2D, and Materials and methods section) that did not rely on ensemble modelling (p = 0.0066, one-sided), further substantiating our choice of machine learning paradigm for facilitating the identification of PDAC cases in a clinical setting where confounding diagnoses may be present, and the prevalence is low.

The results with additional subsampling algorithms, i.e., under-sampling of the majority class and SMOTE (see Materials and methods), further reinforce our choice (Table F in S1 Appendix). Only xgbTree benefits from SMOTE but the result is not consistent between the discovery and the held-out validation sets when we evaluate the positive predictive value. Moreover, synthetic sample generation has been proven to be less efficient in high-dimensional datasets [41]. Further alternatives with adaptive synthetic data generation are necessary to evaluate the advantages of SMOTE in the current problem [42]. For the sake of simplicity, we generated all the subsequent results with the ensemble of base-learners derived with over-sampling of the minority class, which generalises better in the held-out validation set (Fig 2).

The comprehensive index signature, incorporating all diagnostic categories, was constituted by 49 features, of which 44 were proteins (see Fig 3 for the importance associated with each). Among these proteins, 21 demonstrated a significant association with PDAC in the discovery set; ICOSLG, GPNMB, ESM-1, DLL1, VWF, ERBB2, FCRLB, CEACAM5, EGF, CTSV, FASLG, Creatinine, CPE, CA9/CAIX, TBIL, CD207, CRP, CDKN1A, EPHA2, ITGAV, and MUC-16 (see Fig B and Table C in S1 Appendix). The remaining 23 proteins, namely CXCL13, ERBB3, FOLR1/FR-alpha, FADD, ERBB4, CD27, AREG/AR, ADAM-TS-15, ABL1, ANXA1, CXCL17, CD70, CEACAM1, CD48, IL6ST, CD160, PKM/PKM2, CYR61/CCN1, CRNN, ADAM-8, FOLR3/FRgamma, THBS2, GZMB, did not demonstrate a significant association with PDAC in univariate models (see Fig B and Tables C and D in S1 Appendix). Additionally, five clinical covariates—Gender, Age, Ethnicity, Diabetes, and Body Mass Index (BMI)—were identified as important predictors following comprehensive recursive feature elimination during cross-validation (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Features selected per diagnosis class (base-learner classifiers).

The scaled importance is calculated within each base-learner (Fig 2A). Selected features are ranked from left to right according to the average scaled importance across base learners. See Fig 1 and Tables B, C and D in S1 Appendix for the univariate predictive performances of each of the markers in the discovery and validation sets. See Materials and methods section for details on model-agnostic algorithm for feature importance calculation. See S1 Data file for the underlying data for the figure.

https://doi.org/10.1371/journal.pcbi.1012408.g003

Gene Ontology (GO) and biological pathway enrichment (Kyoto Encyclopaedia of Genes and Genomes; KEGG, Reactome Pathway Database; REAC and WikiPathways; WP) analysis was performed for the selected set of features using the gprofiler2 R package (Fig C in S1 Appendix). Top significant terms for biological processes (BP) included ‘circulatory system development’, ‘blood vessel morphogenesis’, ‘cell adhesion’, ‘angiogenesis’, ‘blood vessel development’, ‘regulation of cell adhesion’, ‘positive regulation of cell population proliferation’, ‘cell-cell adhesion’, and ‘regulation of developmental process’. Top relevant biological pathways included: ‘PI3K-KAT signalling pathway’, ‘ERBB signalling pathway’, ‘pathways in cancer’, ‘proteoglycans in cancer’, ‘platinum drug resistance’, ‘prostate cancer’, ‘type I diabetes mellitus’, ‘MAPK signalling pathway’ and ‘focal adhesion’.

The scaled importance of each feature and diagnostic class/classifier is depicted in Fig 3. It is of significance to note that not every biomarker was selected by each individualized classifier, highlighting the requirement for an array of diverse predictors, each tailored to specific underlying conditions, to effectively identify PDAC. This is consistent with the idea that heterogeneous ensembles are fundamental for predictive capacity in blind datasets [37,43].

Of the five selected clinical covariates, only Age, Ethnicity, and Gender manifested as significant predictors of PDAC in the validation set, as illustrated in Fig 1 and explained in the data set characteristics subsection (see also Tables A and B in S1 Appendix). It is worth emphasizing that the lack of significant association between certain markers and PDAC in the discovery set does not preclude their inclusion in the signature. These variables were selected due to their contribution to the enhanced robustness and generalization capacity in predicting PDAC during cross-validation with a recursive feature elimination routine (see Materials and methods). A similar trend was verified in prior work focussed on ensemble models for PDAC early detection against healthy controls and further substantiates the need for extensive discovery analysis in the data sets collected for the present project [37].

In our comprehensive analysis, we identified eight features that exhibited relatively elevated scaled importance in distinguishing controls from patients diagnosed with PDAC. These features, as detailed in Fig 3, include the biomarkers CA19-9, VWF, CPE, CTSV, CEACAM1, and CD160. To rigorously assess the diagnostic utility of these selected features, we devised a reduced index employing the diagnosis-specific ensemble strategy previously delineated. Diabetes and Age were incorporated as clinicodemographic variables. This strategy facilitates a targeted evaluation of the features’ collective performance in a clinical context. Detailed performance metrics and analytical outcomes of this reduced index are presented in S1 Appendix and Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance summary for selected models in symptomatic patients.

The probability values used to calculate the performance metrics were generated with each model developed in the training set and reported in the main text. Probability values for symptomatic patients belonging to the training set and validation set were concatenated to generate the ROC curves. Only ADEPTS samples had symptoms information. A. L. Derang.: Asymptomatic LFT Derangement. B. Pain: Back Pain. C. B. Habit: Change in Bowel Habit. W. Loss: Weight Loss. 95% confidence intervals are provided in parentheses. See also Table H in S1 Appendix for the explicit performance ranks according to model, symptom and metric and Fig 5. Receiver Operating Characteristic (ROC) Area Under the Curve (AUC), Sensitivity (Sens), Positive Predictive Value (PPV) and Negative Predictive Value (NPV) at 90% Specificity (Spec).

https://doi.org/10.1371/journal.pcbi.1012408.t002

In evaluating the efficacy of our ensemble modelling approach and data analysis protocol, we benchmarked it against biomarker combinations reported in existing literature [40,44–48]. Our comparison pitted our panels and protocol against a recursive feature elimination routine presented above that did not resort to ensemble modelling (RFE glm in Fig 2); the latter selected Diabetes, ABL1, ERBB3, ESM1, EGF, SYND1, PPY, TGFA, VEGFA as the best feature combination. From this set the first five markers were involved in the diagnosis-specific ensemble index (Fig 3), and from the remaining only TGFA and VEGFA are reported in the literature as part of best performing panels [40,44–48]. Despite this overlap, given that the starting point for RFE glm was the same as in our ensemble approach, a greater portion of single markers reported in the literature should have arisen as the most competitive when applied to our data set. This is not verified, performance wise, since the diagnosis-specific ensemble modelling still outperformed any other alternative, thus confirming our protocol’s added value in distinguishing diagnosed PDACs from benign and healthy controls (Fig 2).

Furthermore, it is critical to acknowledge that the best-performing index detailed in this study significantly deviates from our earlier work [37]. In the previous study, the goal was to develop a combined index that could distinguish undiagnosed PDAC cases from healthy controls years before a cancer diagnosis was made. In contrast, the current study focuses on identifying an analysis strategy or pipeline leading to effective biomarker panels for use in secondary care among at-risk populations. This shift in focus reflects our ongoing efforts to refine diagnostic tools tailored to the specific needs of different clinical contexts.

Application of the full PDAC ensemble signature in symptomatic patients

Our subsequent aim was to explore whether specific clinical manifestations were correlated with PDAC status in our ADEPTS patient cohort, for which such information was available (refer to Fig D and Table G in S1 Appendix). As a similar type of data was not available for the UKCTOCS subset (healthy controls) used in this work, we focused this section on the ADEPTS cohort.

In our prior research, we analysed 12 "red-flag" symptoms reported by patients up to 22 months before the diagnosis of pancreatic cancer was established [17]. In this work, ‘Vomiting’ (p = 0.17), ‘Asymptomatic LFT Derangement’ (p = 0.28), ‘Back pain’ (p = 0.54), ‘Change in Bowel Habit’ (p = 0.67) and ‘Rectal Bleeding’ (p = 0.76) were selected for PDAC (versus benign disease controls), yet only ‘Jaundice’ (p = 3.22×10⁻¹⁵), and ‘Weight Loss’ (p = 1.44×10⁻⁶) were significantly associated with PC cancer cases in the set of samples randomly selected from the ADEPTS cohort, in which the biomarker panel was tested (Fig 4B and Table G in S1 Appendix). Unsurprisingly, ‘Reflux’ (p = 0.022) and ‘Bloating’ (p = 0.048) were significantly associated with benign controls. Interestingly, ’Abdominal Pain’, ’Heartburn’, ’Anaemia’, and ’Dysphagia’ upon presentation were aligned more with the benign control cohort, albeit not significantly (refer to Fig 4 and Table G in S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Association between symptoms and PDAC.

A Number of subjects with each symptom according to PDAC status, case or control. B Association of symptoms with PDAC status, p values were calculated according to a logistic regression model with a bias reduction method. Purple dashed lines correspond to -Log [0.05]. In B dot sizes correspond to odds ratios and are colour coded according to their respective values, i.e., blue if OR<1 and red if OR>1. See also Table I in S1 Appendix. Only samples belonging to the ADEPTS cohort were used as no information about symptoms was available for the UKCTOCS set of samples.

https://doi.org/10.1371/journal.pcbi.1012408.g004

Within the framework presented in preceding sections, our ensemble of classifiers was developed independently of symptomatic data. To assess the overall efficacy of our signature and its predictive capacity for PDAC, we scrutinized its performance on a subset of ADEPTS patients, belonging to both discovery and validation cohorts, manifesting with ’Weight Loss’ (n = 56) and ’Jaundice’ (n = 40) (refer to Fig 4 and Table G in S1 Appendix). For each sample within these cohorts, where symptom data was accessible, probability scores were derived based on the ensemble model formulated above. We should emphasize that no additional model refinement was pursued and a simple concatenations of probability outputs was done. The decision to aggregate these probability scores is further rationalized by the relatively limited patient count exhibiting ’Weight Loss’ and ’Jaundice’ within the individual discovery and validation datasets (Table G in S1 Appendix). In the ADEPTS subset of samples presenting with ’Weight Loss’, an AUC of 0.95 (95% CI 0.90–0.1), a sensitivity of 0.94 (95% CI 0.29–1), a PPV of 0.80 (95% CI 0.56–0.81), and a NPV of 0.97 (95% CI 0.74–1) at 90% specificity were achieved (Fig 5 and Table 2). In patients presenting with ’Jaundice’, an AUC of 0.89 (95% CI 0.79–0.99), a sensitivity of 0.73 (95% CI 0.36–0.91), a PPV of 0.90 (95% CI 0.82–0.92), and a NPV of 0.73 (95% CI 0.54–0.89), at 90% specificity, were observed (Fig 5 and Table 2). Compared with the AUC obtained with a simple CA19-9 logistic regression model developed in the discovery set and by concatenating the probability scores in the discovery and validation as done above, a significantly lower AUC of 0.74 (95% CI 0.58–0.90) is achieved (p = 1.29×10⁻¹¹, one-sided bootstrap test), with a sensitivity of 0.53 (95% CI 0.24–0.76), a PPV of 0.70 (95% CI 0.50–0.77), and a NPV of 0.81 (95% CI 0.73–0.90), at 90% specificity, for patients presenting with ’Weight Loss’. For patients presenting with ‘Jaundice’ an AUC of 0.70 (95% CI 0.53–0.86) is reached, also significantly inferior (p = 1.94×10⁻⁷), with a sensitivity of 0.41 (95% CI 0.14–0.73), a PPV of 0.83 (95% CI 0.64–0.90), and a NPV of 0.55 (95% CI 0.46–0.73), at 90% specificity, for CA19-9 as the single predictor.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Receiver operating characteristic curves for selected models in symptomatic patients.

A Only CA19-9. B Full index signature. C Reduced index signature. The probability values used to calculate the performance metrics were generated with each model developed in the discovery set and reported in the main text. Probability values for symptomatic patients belonging to the discovery set and validation set were concatenated to generate the ROC curves. Only ADEPTS samples had symptoms information. A. L. Derang.: Asymptomatic LFT Derangement. B. Pain: Back Pain. C. B. Habit: Change in Bowel Habit. W. Loss: Weight Loss. See also Table 2 for numerical values for area under the curve and other metrics.

https://doi.org/10.1371/journal.pcbi.1012408.g005

With respect to other non-localising symptoms of note (Fig 5 and Tables 2 and H in S1 Appendix), the best predictive performance was noted for the full index signature where it was able to differentiate patients presenting with ‘abdominal pain’ due to benign conditions vs. PDAC with an AUC of 0.98 (95% CI 0.97–1), sensitivity of 0.94 (95% CI 0.81–1), PPV 0.43 (95% CI 0.40–0.45) and a NPV of 0.99 (95% CI 0.98–1), at 90% specificity. In those presenting with ‘change in bowel habit’, an AUC of 0.97 (95% CI 0.92–1), sensitivity 0.86 (95% CI 0.81–1), PPV of 0.51 (95% CI 0.41–0.55) and NPV of 0.98 (95% CI 0.95–1) was obtained. Both the full ensemble index and the 8-marker signature showed superior predictive performance to CA19-9 as a single marker (see Table I for the respective p-values, in S1 Appendix).

Correlation of the full PDAC ensemble signature with QCancer pancreatic score

In our final analysis, we juxtaposed the performance of our full ensemble classifier PDAC index against the QCancer risk prediction index, a clinical decision support tool available for primary care physicians, that integrates a myriad of individual-specific risk factors including age, sex, ethnicity, clinical measurements, diagnoses, and patient-reported symptoms into a risk stratifying point of care questionnaire [21]. The ‘Today’s QCancer’ index evaluates an individual’s current risk of having an undiagnosed cancer as well as the specific risk for 9 distinct underlying cancer types, including pancreatic (‘pancreatic’ score) [49,50]. The aim was to determine whether in combination, the QCancer eCDST and our biomarker index signature would be able to better discriminate PDAC patients in a symptomatic (ADEPTS) cohort or whether it would be redundant. As the current risk threshold set by the NICE is at 3% for triggering specialist referrals [51], we opted to assess the combined performance of our index signature and the eCDST at a same or lower cut-off values.

The number of samples for which a QCancer score was computed is illustrated in Fig 6C. Using the diagnostic-specific ensemble model delineated previously, probability scores for samples in both discovery and validation cohorts were used to ascertain the combined ROC AUC for those samples possessing a QCancer score. This amalgamation was imperative, considering the reduced number of samples with an associated QCancer score (Fig 6C). It should be emphasized that no subsequent refinements or training of the algorithm were conducted. The ensemble stack index demonstrated a remarkable performance, achieving an AUC of 0.98 (95% CI 0.97–0.99), a sensitivity of 0.99 (95% CI 0.97–1), a PPV of 0.91 (95% CI 0.90–0.91), and a NPV of 0.99 (95% CI 0.96–1), at 90% specificity. Interestingly, when considering only samples with a QCancer risk above 2 or 2.5, the biomarker and clinical covariate ensemble index exhibited comparatively lower performance (Fig 6A). For a QCancer risk above 3.0, the performance of the index decreases minimally once again, which is expected as the difficulty of correctly singling out cases from confounding controls is increased (Fig 6A). However, the QCancer pancreatic score did exhibit a correlation with the odds of PDAC as determined by the ensemble classifier (R = 0.36, p = 3.4×10⁻⁸, Fig 6D) which highlights an important link between the purely clinical variables recorded for this cohort and the PDAC signature. Most importantly, the stacked index succeeded in attributing higher odds ratios above 1 to several PDAC cases that would have otherwise escaped detection had a QCancer score above 3 been taken as the risk predictor (Fig 6D). Contrarily, when depending exclusively on the QCancer score, and using it to calculate the ROC AUC, the predictive capacity for PDAC in the ADEPTS samples is noticeably diminished in comparison to the performance of the ensemble index (Fig 6B); this was verified in all samples with a Qcancer score (p = 1.56×10⁻¹⁸, one-sided bootstrap test comparing AUCs), with a score above 2 (p = 1.24×10⁻¹⁰), above 2.5 (p = 3.68×10⁻¹³) and above 3 (p = 2.33×10⁻⁸). This justifies the PDAC signature as a useful complementary resource for enhanced and accelerated diagnosis in the clinic.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Prediction of PDAC in patients with specific symptoms and according to QCancer score values.

The ensemble stack was selected as the best model according to Fig 2. A Performance of the stack in participants for which a Qscore had been calculated or above a specific threshold, bigger than 2, 2.5 or 3.0. B Performance of the Qscore taken as the predictor of PDAC risk in participants for which a Qscore had been calculated or above a specific threshold, bigger than 2, 2.5 or 3.0. C Number of subjects that had a calculated Qscore or are above a specific threshold, bigger than 2, 2.5 or 3.0. D Correlation between QCancer score and odds ratio of PDAC according to the stacked ensemble. D is in log scale and R stands for the Person correlation coefficient and p for the p-value calculated with a t-test. The QCancer score is identified as Qscore in the figure panels. Receiver Operating Characteristic (ROC) Area Under the Curve (AUC), Sensitivity (Sens), Positive Predictive Value (PPV) and Negative Predictive Value (NPV) at 90% Specificity (Spec).

https://doi.org/10.1371/journal.pcbi.1012408.g006

Ethics statement

For the Accelerated Diagnosis of neuro Endocrine and Pancreatic TumourS (ADEPTS) study, University College London (UCL)/ University College London Hospital (UCLH) Research Ethics Committee reference 06/Q0512/106, IRAS Number 234637, NIHR portfolio no. 7343, patients were recruited at gastroenterology/hepatobiliary and surgical clinics at UCLH and the Royal Free Hospitals (RFH), London, UK. All patients recruited to the ADEPTS study provided written informed consent and no data allowing identification of patients was used.

For the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS), Joint UCL/UCLH Research Ethics Committee A (Ref. 05/Q0505/57), written informed consent for the use of samples in the trial and secondary ethically approved studies was obtained from donors and no data allowing identification of patients was used.

Study design

As our cohort, we used serum samples from the ADEPTS study [44] aimed at detecting pancreatic cancer in patients at an earlier stage. As part of the Early Diagnosis Research Alliance, the ADEPTS study (previously referred to as TRANSlational research in BILiary tract and pancreatic diseases study), commenced in 2018 and included a multicentre prospective blood sample collection from patients with non-specific but concerning symptoms associated with PDAC. Patients were recruited at gastroenterology/hepatobiliary and surgical clinics at UCLH and the RFH, London, UK. Blood samples were collected from subjects with benign hepatobiliary conditions as well as those with PDAC (stages I-IV).

For PDAC patients, tumour staging was performed according to the AJCC 8^th edition (TNM) based on cross-sectional imaging and for those undergoing surgery, based on multi-disciplinary team recordings. All included PDAC cases were histologically confirmed by UCLH and RFH local pathologists based on tissue analysis obtained by endoscopic ultrasound guided fine needle biopsies or specimens obtained during surgical resection.

For benign disease controls, patients were selected to include the following diagnoses: chronic pancreatitis, intraductal papillary mucinous neoplasms (IPMN), or benign pancreatic diseases (e.g., serous cystadenomas and pancreatic heterotopia). Patients with acute and chronic pancreatitis, pancreatic cysts, benign biliary duct diseases (e.g., IgG4 disease), liver disease, gastritis/reflux disease, gallstones as well as those with familial history of pancreatic cancer, were also used. Samples also included those collected from patients presenting with non-specific symptoms which were not otherwise explained by an underlying gastrointestinal pathology (such as non-specific abdominal pain and irritable bowel syndrome) as well as other malignancies. Medical history and confirmation of diagnosis was obtained from hospital medical records and included GP and secondary clinic referral letters. For 45 patients, a QCancer score was available at time of specialist centre consultations. QCancer calculates the probability of an individual as harbouring an existing, yet undiagnosed cancer, by considering their specific risk factors and presenting symptoms. These are digitally available for primary care physicians through patient record and data management portals such as EMIS Web and INPS and designed as clinical decision support tools to aid in assessment of need for specialist referrals [21–24].

To further represent the healthy population we also used samples from 72 healthy control UKCTOCS [36] samples that were collected from a nested case control discovery study part of UKCTOCS reported before [37]. The original UKCTOCS dataset from which data was used was derived from serum samples collected from post-menopausal women, aged between 50 and 74 years, who were recruited between the years 2001 and 2005 [36]. The collection of these samples was conducted in accordance with a specific Standard Operating Procedure [59,60]. For the current work our interest lies only with the UKCTOCS matched non-cancer controls, i.e., with no cancer registry code, from individual women selected based on collection date, age, and centre to minimize variation due to handling and storage. Comprehensive information regarding diabetes status for the selected UKCTOCS participants was either unavailable or incomplete. In addition, data on disease duration was not accessible. Consequently, it was not feasible to stratify samples to discovery and validation sets based on the type of diabetes they may have had. For the purposes of this study, only healthy controls that were matched to PDAC cases, with less than one year to diagnosis, were utilized.

A total of 539 serum samples (493 controls and 46 PDAC cases, see Table 1) were analysed using the Olink multiplex immunoassay Oncology II panel in addition to five in-house markers: Carbohydrate antigen 19–9 (CA19-9), Interleukin 6 Cytokine Family Signal Transducer (IL6ST/IL6RB), von Willebrand factor (VWF), Pyruvate kinase isozymes M1/M2 (PKM/PKM2) and Thrombospondin 2 (THBS2/TSP2). The selection of additional markers, beyond CA19-9, was informed by our preceding research in early detection of PDAC [37,48]. In those studies, a panel of markers was identified due to its demonstrated ability to facilitate the early detection of pancreatic cancer, with a lead time of up to two years prior to diagnosis.

Serum analyte measurements

All ADEPTS [44] samples were randomized for testing. Table J in S1 Appendix summarizes dilution factors and coefficients of variation. CA19-9 was measured using the Mucin PC/CA19-9 ELISA Kit (Alpha Diagnostic International) according to the manufacturer, using a 1:4 serum dilution. For VWF, we resorted to the Von Willebrand Factor Human ELISA Kit (abcam) at a 1:100 serum dilution. IL6ST/IL6RB by Quantikine human soluble gp130 (R&D Systems), according to manufacturer recommendations, at a 1:100 serum dilution. THBS2/TSP2 was measured using the Quantikine Human Thrombospondin-2 Immunoassay (R&D Systems) at a 1:10 serum dilution. Pyruvate kinase M2 (PKM2) was measured with an ELISA (Cloud-Clone Corp) at a 1:10 dilution.

We outsourced tests using the multiplex immunoassay Oncology II panel from Olink on all samples. This Olink panel measured known cancer antigens, growth factors, receptors, angiogenic factors, and adhesion regulators (as detailed in Table K in S1 Appendix). Identical assays were performed on a subset of samples derived from the UKCTOCS study [59,60].

To bridge the normalized protein expression values from Olink between the UKCTOCS and ADEPTS datasets, we selected a representative sample set of 16 from each cohort and plated them together. Subsequently, a correction was applied to the datasets using the statistical algorithms recommended in the Olink data normalization white paper [61]. This method ensured that the data from different batches and studies were comparable, thereby enhancing the robustness and validity of the findings.

Statistical analysis

The selected set of ADEPTS samples used in this work was partitioned into two distinct sets: a discovery subset, comprising two-thirds of the total sample size, and a held-out validation subset, encompassing the remaining one-third. All algorithms were trained in the discovery subset. Allocation into each subset was performed by stratifying for specific age ranges, diabetes status, PDAC status and control diagnosis class. For the PDAC cases, tumour stage was also used. The age stratification ranges were the following: 18<Age≤28; 29<Age≤38; 39<Age≤48; 49<Age≤58; 59<Age≤68; 69<Age≤78; Age≥79. The samples assigned to the control class were made of benign conditions such as: Sphincter of Oddi dysfunction, Pancreatic Cyst, Other Cancer, Other Biliary Duct Disease, No Relevant Diagnosis, Liver Disease, Irritable Bowel Syndrome, IgG4 Disease, Gastritis/Reflux Disease, Gallstone Disease, Familial Pancreatic Cancer, Chronic Pancreatitis, Acute Pancreatitis, Isolated LFT Derangement and Non-specific Abdominal Pain. We also added an additional set of healthy control samples collected from a nested study done in UKCTOCS samples used in a previous paper [62]. The controls matched by age to the PDAC cases in the UKCTOCS cohort that had a time to diagnosis below up to one year were selected. The allocation of these controls to the discovery or held-out validation sets was done according to the division used in our previous work [62]. The number of controls and cases collected for this study can be visualized in Fig 1. UKCTOCS controls are identified as ‘Healthy’.

The discovery held-out validation final split, i.e., with ADEPTS and UKCTOCS samples, put the prevalence of PDAC in the discovery set at close to 8%. The prevalence of PDAC in the resulting validation was approximately 14%. The held-out validation set was isolated and not used in any stage of model and biomarker signature development. This is akin to having a blinded dataset.

Receiver operating characteristic (ROC) curves were constructed for each model to assess diagnostic performance. The area under the curve (AUC) for the ROC curves was used as the metric. ROC curves were generated with the pROC R package (version 1.18.0, https://cran.r-project.org/web/packages/pROC/index.html). 95% CI for AUCs were determined by stratified bootstrapping. All AUC confidence intervals crossing 0.5 were considered to be non-significant. P values comparing ROC curves were also calculated using the pROC package, under a one-sided bootstrap approach with 10000 runs.

In order to evaluate the association between each of the single markers available for this work, including clinical covariates (see Fig 1 and Table 1), and PDAC status, we created univariate models using a logistic regression model implemented in the logistf R package (https://cran.r-project.org/web/packages/logistf/index.html, version 1.24.1). This approach fits a logistic regression model using Firth’s bias reduction method. The reported confidence intervals for odds ratios and tests were based on the profile penalized log likelihood and incorporate the ability to perform tests where contingency tables are asymmetric or contain zeros. The performance of single marker models was also verified in the discovery and held-out validation sets (see Fig B and Tables C and D in S1 Appendix). The same package was also used to verify the association of the presence of symptoms and PDAC status (see Fig 4).

A comprehensive multi-dimensional examination of the collated data was conducted by employing two distinct analytical frameworks. The first was a stacked ensemble algorithm where base-learners were developed according to the same algorithm but in subsets of the discovery set where samples belonging to a specific control diagnosis class were contrasted against the same 24 PDAC cases (see the proportions in Fig 1). The resulting base-learners were then stacked by a logistic regression model, (see Table E for the resulting coefficients and Fig A for the stacking procedures, in S1 Appendix). This approach aimed to leverage the predictive power of multiple models, thereby enhancing the robustness and potentially leading to more precise predictive outcomes [37,43,54]. For each base-learner classifier we resorted to a Recursive Feature Elimination (RFE) routine with logistic regression as the fitting algorithm available through caret (version 6.0–93, https://cran.r-project.org/web/packages/caret/index.html). Due to the prevalence of PDAC cases in the whole dataset being low, random under sampling of the majority class, here benign and healthy controls, if the PDACs are pitted against the whole set of controls, would pose a challenge for most algorithms. Therefore, creating an ensemble of classifiers specialised in contrasting a specific diagnostic class against PDAC allowed us more balanced subsets leading to increased performance (Fig 2). For the samples collected from UKCTOCS no symptoms information was available and, therefore, we created a separate classifier associated with this subset of individuals.

Selection of type and number of base-learners has been studied before in other areas [37,43]. Different approaches to this problem have been put forward that either focus on a greedy search for the best ensemble, or rely on diversity based metrics to ensure robustness in external datasets [43]. Here, we have chosen to enforce the diagnosis-specific design to ensure that we relied on clinically relevant features and respected the underlying question of ADEPTS [44]. Nevertheless, we also tested ensemble selection by recursively eliminating base-learners and the best ensemble performer was always the full set of 16 reported above, each contrasting the same PDACs against diagnosis-specific controls.

The second analytical framework followed a more traditional application of state-of-the-art algorithms to the whole discovery set. We tested 3 different algorithms under this framework: random forests (RRF, version 1.9.4, https://cran.r-project.org/web/packages/RRF/index.html); extreme gradient boosting trees (xgbTree, version 1.6.0.1, https://cran.r-project.org/web/packages/xgboost/index.html); and a generalized linear model with RFE (RFE glm). It is important to clearly stress the differences between this RFE glm model and the stacked ensemble model reported above. The additional RFE glm model, despite using similar techniques to each base-learner in the ensemble approach, was applied to the whole discovery data set, without division of the control samples into diagnosis classes. This resulted in one model only as opposed to 16, and therefore there is no need for stacking under this additional approach.

All ensemble base-models as well as all the additional state-of-the-art algorithms mentioned above, i.e., xgbTree, RRF and RFE glm, were trained in the discovery subset with leave-one-out cross validation in order to find the optimal set of input features or the optimum hyperparameters (see Table M in S1 Appendix). 1000 random parameter combinations were tested to achieved optimum performance.

We tested 3 subsampling algorithms combined with each of the models: oversampling of the minority class, the Synthetic Minority Oversampling Technique (SMOTE) [41,63] algorithm and under sampling of the majority class (see Table F in S1 Appendix). The sub-sampling routines were performed within the cross-validation procedure to avoid overfitting [64–66].

The RFE associated with 2 of the algorithms mentioned above was also performed within the cross-validation folds. This reduces data leakage and overfitting due to the fact that feature selection is performed for each training fold and a rank of potential feature groups is created based on their cross-validation performance [64–67], thus leading to the most robust option.

To verify if the PDAC index developed with the ensemble stacked approach had any association with metrics used in the clinic but not taken into account in any stage of algorithm training, we also gathered the QCancer pancreatic score [21] for individuals in the ADEPTS study (see Fig 6). This allowed us further validation of the diagnosis-based ensemble index and a view of its potential as a complementary measure.

The procedure for assessing feature importance in each base learner was a model-agnostic method based on a simple feature importance ranking measure [68], implemented in the R package vip (version 0.3.2, https://cran.r-project.org/web/packages/vip/index.html). The model-agnostic interpretability, by decoupling the interpretation from the model itself, introduces a level of flexibility that enables its application across any supervised learning algorithm. Despite the algorithm used for each diagnosis-specific classifier being the same, the model-agnostic approach allows us to be able to generalise the computed importances to other work in the literature.

Enrichment analysis for each of the signatures developed was performed with the gprofiler2 R package (version 0.2.1, https://cran.r-project.org/web/packages/gprofiler2/index.html). A threshold for multiple comparison correction under the framework of false discovery rate was instituted at 0.05.