2.2.1. Knee Osteoarthritis (OA).

Diagnosis: Classification based on Kellgren-Lawrence Grade (KLG) [30] – A severity scale (0–4) assigned by radiologists based on radiographic features including joint space narrowing, bone spurs, and sclerosis.

Prognosis: Prediction of disease worsening over a 4-year horizon [31], defined as:

• Structural Incidence for early-stage OA (KLG 0–1): Progression to KLG ≥ 2 or undergoing a Total Knee Replacement (TKR).
• Structural Progression for radiographic OA (KLG ≥ 2): An increase in KLG or undergoing a TKR.

2.2.2. Alzheimer’s Disease (AD).

Diagnosis: Classification of cognitive status using structural brain MRI, based on the diagnostic criteria established by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) study protocol:

• Cognitive Normal (CN): Participants with no significant impairment in cognitive functions or activities of daily living.
• Mild Cognitive Impairment (MCI): Participants with a subjective memory concern, objective memory loss, and preserved general cognition and functional performance.
• Alzheimer’s Disease (AD): Participants meeting the NINCDS/ADRDA criteria for probable AD.

Prognosis: Prediction of cognitive decline over a 2-year horizon:

• CN → MCI: Onset of cognitive impairment.
• MCI → AD or CN → AD: Progression to confirmed disease.

2.2.3. Breast Cancer (BC).

BI-RADS (Diagnosis): Classification system for breast imaging findings:

• Grade 0: Incomplete assessment; additional imaging evaluation is needed.
• Grade 1: Negative; no abnormalities found.
• Grade 2: Benign finding; no malignant characteristics present.

Prognosis: Prediction of a patient having a confirmed breast cancer diagnosis within 5 years of the scan date.

2.3.1. Knee osteoarthritis.

Data were sourced from the Osteoarthritis Initiative (OAI) [32] for model development and the Multicenter Osteoarthritis Study (MOST) [33] for external validation. The OAI is a long-term, multicenter observational study of 4,796 participants focused on knee OA biomarkers. We utilized bilateral posterior-anterior fixed flexion knee radiographs.

• Diagnosis Cohort (OAI): Comprised 47,041 radiographs from 4,508 subjects with KLG assessments.
• Prognosis Cohort (OAI): To predict structural worsening, a 4 year follow up from the baseline visit was used. This timeframe was selected to balance observing meaningful change and maximizing cohort size, as longer horizons led to significant participant drop-off. Further, knees with KLG 4 or a TKR at baseline were excluded. Two distinct prognosis tasks and their corresponding cohorts were defined based on baseline disease severity:
  • Structural Incidence: This task focused on predicting the onset of radiographic OA in knees with baseline KLG 0–1. The cohort included 420 patients who progressed (defined as reaching KLG ≥ 2 or undergoing a TKR) and 2,439 non-progressing controls.
  • Structural Progression: This task focused on predicting the worsening of established OA in knees with baseline KLG 2–3. The cohort included 571 patients who progressed (defined as an increase in KLG or undergoing a TKR) and 1,677 non-progressing controls.
• MOST Validation Cohort: Corresponding cohorts were created from the MOST dataset using a 5-year follow-up due to data availability. The incidence cohort included 505 progressors and 1,395 controls. The progression cohort included 562 progressors and 617 controls.

2.3.2. Alzheimer’s disease.

Data were obtained from the ADNI database [25], a longitudinal multicenter study aimed at validating biomarkers for AD progression. We utilized T1-weighted brain MRI scans.

• Diagnosis Cohort: Consisted of 2,723 scans from 662 patients classified as CN, MCI or AD.
• Prognosis Cohort: To predict cognitive decline within a 24-month follow-up, 914 scans from 365 unique patients were used. Progression was defined as a change from CN to MCI/AD or MCI to AD. To augment the dataset size, MRI scans from both baseline and available intermediate follow-up visits were utilized as distinct input time points; however, scans from patients already classified as AD were excluded. To prevent data leakage and ensure the model learned generalizable prognostic features rather than patient-specific anatomy, all scans from a single patient were strictly kept within the same data partition (training, validation, or test) during all experiments. The cohort included 148 progressing patients (contributing 343 scans) and 217 stable controls (contributing 571 scans).

2.3.3. Breast cancer.

We utilized a curated dataset of digital screening mammograms from NYU Langone Health [34], enhanced with longitudinal follow-up cancer labels up to five years post-imaging.

• Diagnosis Cohort: Contained 56,733 examinations from 33,681 patients with radiologist-assigned BI-RADS grades 0, 1, or 2.
• Prognosis Cohort: To facilitate robust model development, a balanced cohort was constructed. This included 3,000 examinations from 2,319 patients who were diagnosed with breast cancer within 5 years (but more than 130 days after the index scan) and 3,000 control examinations from 2,837 patients who remained cancer-free for the 5-year follow-up period. The mean age for patients contributing case examinations was 61.5 ± 11.4 years, while the mean age for controls was 56.4 ± 10.8 years.

To prevent data leakage, strict patient-level splits were used to ensure disjoint sets of individuals across all training, validation, and test sets, and no information from future visits was used as input features. Detailed demographic and clinical characteristics for all patient cohorts are provided in S1-S4 Tables.

2.4.1. Diagnosis pretraining.

We compare diagnosis-based initialization against modality-specific baselines (Fig 1). We use modality-appropriate initializations: OA is 2D and moderate-resolution, where ImageNet pretraining is standard; AD uses 3D brain MRI (no ImageNet analogue), and BC requires very high-resolution mammography where running large ImageNet backbones at native resolution is not tractable, so we use random initialization similar to previous works [8,12]. Concretely, prognosis models are initialized from either (i) diagnosis-pretrained weights learned on the corresponding diagnostic task, (ii) ImageNet weights (OA only), or (iii) random weights (AD, BC).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Illustrates the proposed approaches (a) single task prognosis with diagnostic pretraining and (b) Sequential learning with experience replay.

https://doi.org/10.1371/journal.pone.0344600.g001

2.4.2. Sequential learning with experience replay.

To mitigate catastrophic forgetting [28,29] in multitask settings, we used a two-phase approach (Fig 1).

1. Phase 1 (Diagnosis Pretraining): A model is trained on the large diagnosis cohort.
2. Phase 2 (Prognosis training with Diagnosis Replay): The model is fine-tuned using a multitask objective. At each step, a batch is randomly sampled with equal probability from either the prognosis cohort (to update on the prognosis task) or the diagnosis cohort (to “replay” and refresh diagnosis knowledge). This allows the model to learn the new task while preserving its original capabilities.

2.4.3. Comparative multitask approaches.

For comparison, we evaluated three alternative strategies:

• Single-cohort MT: An approach similar to that used by Tiulpin et al. [14] and Leung et al. [16], where the model is trained on both diagnosis and prognosis tasks using only the data available in the smaller prognosis cohort.
• Diagnosis-pretrained MT: Initializes with diagnostic pretraining but then fine-tunes solely on the prognosis cohort’s diagnosis and prognosis labels, without experience replay
• Concurrent MT: Trains simultaneously on both diagnosis and prognosis cohorts from a baseline initialization, without any pretraining.

2.5.1. Model architectures and preprocessing.

• OA: Input radiographs were normalized, and knee joints were extracted prior to analysis. ResNet-34 with attention [35,36] architecture was used on extracted knee joints.
• AD: Input MRI scans underwent standard bias correction and spatial normalization. A custom 3D-CNN [8] was used on processed T1 MRIs.
• BC: The four standard mammographic views were individually processed and cropped to a uniform resolution. A custom multi-view CNN [12] was used on the processed mammogram views.

2.5.2. Training protocol.

All models were trained using the Adam optimizer [37] coupled with a cosine annealing learning rate scheduler to ensure stable convergence. For the highly imbalanced breast cancer BI-RADS task, a weighted cross-entropy loss was employed to give appropriate importance to rare classes. Condition-specific data augmentation techniques (including random cropping, flips, and rotations) were applied during training to enhance model robustness.

For tasks with smaller sample sizes (all prognosis cohorts and the AD diagnosis cohort), we employed a 5-fold cross-validation scheme with strict, non-overlapping patient-level splits. For each of the 5 iterations, one fold was designated as the held-out test set. Of the remaining four folds, three were used for training the model, and one was used as the validation set for hyperparameter tuning and model checkpointing via early stopping. This process was repeated five times, ensuring that every patient served as part of a test set exactly once. The final reported performance metrics are the mean and standard deviation across the results from the five test folds.

For multitask learning with experience replay, tasks were sampled with equal probability (p = 0.5). Crucially, to prioritize the more challenging prognostic task, model checkpoints were saved based on the epoch achieving the highest prognosis AUROC on the validation set, rather than on the total loss

For each disease, the diagnosis and prognosis cohorts were drawn from the same underlying patient populations (OAI, ADNI, and NYU dataset). As such, patients in the longitudinal prognosis cohort were also represented in the larger cross-sectional diagnosis cohort. To prevent any data leakage between the pretraining and fine-tuning stages, a strict patient-level splitting procedure was enforced. For each of the 5 cross-validation folds of the prognosis task, the patient IDs assigned to the test set were identified first. These testset patients were then completely excluded from the training set of the large diagnosis model used for pretraining. This ‘test set holdout’ strategy ensures that our prognosis models were always evaluated on patients that the entire training pipeline, including the pretraining stage, had never seen, providing an unbiased estimate of performance.

2.5.3. Evaluation and analysis.

This study was prepared and reported in accordance with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) guideline wherever applicable. The pre-specified primary endpoint for all prognostic tasks was the Area Under the Receiver Operating Characteristic curve (AUROC), which measures the model’s overall ability to discriminate between patients who experience disease progression and those who do not.

Secondary endpoints for prognosis included the Area Under the Precision-Recall Curve (AUPRC), which provides additional insight into model performance, especially in cohorts with class imbalance. Sensitivity and specificity were computed additionally for breast cancer.

For the threshold-dependent metrics of sensitivity and specificity, the following procedure was used for each of the 5 cross-validation folds: first, an optimal decision threshold was determined by maximizing the Youden Index [38] on the validation set for that fold. This fold-specific threshold was then applied to the predictions on the corresponding test set to calculate sensitivity and specificity. The final reported metrics are the mean and standard deviation of the five resulting sensitivity and specificity values.

For diagnosis, we report balanced accuracy for the five-class knee OA task, following [16] to mitigate class imbalance. For the multi-class AD and BC tasks, we additionally report macro-AUROC and micro-AUROC, consistent with prior work. Macro-AUROC is the unweighted mean of one-vs-rest AUROCs across classes, giving each class equal weight. Micro-AUROC pools predictions and labels across all classes to compute a single AUROC, effectively weighting classes by their prevalence. Balanced accuracy is the mean of per-class recall (sensitivity).

To formally compare the primary approaches, ROC curves of ensembled cross-validation predictions were compared using the DeLong test [39]. For every patient in the test set, the predictive scores (probabilities) from the five respective cross-validation models were averaged. This resulted in a single set of ensembled predictions for the entire test cohort, from which a single, more stable ROC curve was generated for the statistical comparison. This practice of averaging predictions is a standard ensembling technique to reduce variance and provide a more robust estimate of model performance.

Stratified subgroup analyses were performed on pre-defined clinical subgroups: structural incidence (KL 0–1) vs. progression (KL 2–3) for Knee OA, and cognitive transition groups (e.g., CN vs. MCI) for Alzheimer’s Disease. These subgroup analyses were conducted to verify that the models were learning genuine prognostic signals rather than simply using baseline disease severity as a proxy for progression risk. This analysis was not powered for formal statistical tests of subgroup interaction. Statistical significance testing was focused on our primary hypothesis comparing the baseline and diagnosis-pretrained models.