Study design and participants

The current study used publicly available, de-identified data from the Osteoarthritis Initiative (OAI), an ongoing, multicenter, prospective cohort study (Clinical Trials.gov identifier: NCT00080171). As such, no additional ethical approval was required for secondary analysis. The original OAI study protocol was approved by the institutional review boards of all participating centers, including the coordinating center at the University of California, San Francisco (IRB number: 10-00532). The study was conducted in accordance with the Health Insurance Portability and Accountability Act (HIPAA), and all participants provided informed consent.

Six-hundred participants were selected based on frequent knee pain and a Kellgren-Lawrence Grade (KLG) of 1, 2, or 3 on knee radiographs at baseline [18]. They were required to have the baseline and 24-month radiographic data on medial tibiofemoral joint space width (JSW) [19], knee MRI, stored serum and urine specimens, and clinical data. Further details can be found in our study’s flowchart (S1 Fig).

Inclusion and exclusion criteria

FNIH OA Biomarkers Consortium undertook a nested case-control study (194 JSN and pain progression cases and 406 OA comparators) of progressive KOA within the OAI, a unique longitudinal cohort with a publicly available repository of joint images, biologic specimens, and clinical data obtained at annual clinic visits. Details of the study design have been published previously [18]. Briefly, participants eligible for the present study were those who had at least 1 knee with a KLG of 1–3 at baseline determined at a central reading site and for whom knee radiographs, knee MRIs, stored serum and urine specimens, and clinical data were available for the baseline and 24-month visits. One index knee was selected for each participant.

Knees were excluded from analysis under the following circumstances: if progression criteria were met by 12 months to enable the study of change in biomarkers before the progression definition was met, if radiographic lateral joint space narrowing (JSN) grade 2 or 3 was present at baseline, or if total knee replacement or total hip replacement had occurred prior to 24 months due to possible effects on biochemical markers.

Clinical outcome

A predetermined number of index knees were categorized into four groups based on outcome assessment at 24 months: (1) knees with both JSN and pain progression (n = 194), (2) knees with JSN progression but not pain progression (n = 103), (3) knees with pain but not JSN progression (n = 103), and (4) knees with neither JSN nor pain progression (n = 200). The main analysis focused on comparing knees with both JSN and pain progression to all other knees [12,20]. JSN progression was defined as a minimum JSW loss of ≥0.7 mm, and pain progression as a sustained (≥2 time points) increase of ≥9 points on the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) pain subscale (0–100 scale) [18].

After enrollment, participants underwent routine clinical examination, knee MRI, and knee joint radiography every 12 months for 2 years.

Clinical examination and radiography

The FNIH OA Biomarkers Consortium cohort study used the following clinical examination: [1] Knee pain severity scale. [2] Participant global assessment. [3] WOMAC Osteoarthritis Index. [4] Knee Outcomes in Osteoarthritis Survey (KOOS). [5] Limitation in activity due to knee pain. [6] General health and functional status. [7] Walking ability and endurance. [8] Upper leg strength. [9] Serum and urine biochemical biomarkers assay.

The knee fixed-flexion, posterior-anterior weight-bearing radiographs obtained at baseline, and all annual follow-up visits were performed using a Plexiglas positioning frame (SynaFlexer; BioClinica, Newark, CA), with knees flexed to 5–15° and feet internally rotated 10°. All radiographs were centrally read to determine the KLG [21]. The KLG assessments for radiographs were performed by Dr. Piran Aliabadi, MD and Dr. Burt Sack, MD, under the direction of Dr. David Felson, MD from the Boston University Clinical Epidemiology Research and Training Unit for the baseline through 24-month visits. Specifically, Dr. Piran Aliabadi and Dr. Burt Sack are board-certified radiologists with extensive experience in musculoskeletal imaging, and the assessments were conducted under the supervision of Dr. David Felson, a senior rheumatologist and epidemiologist with decades of expertise in osteoarthritis research at the Boston University Clinical Epidemiology Research and Training Unit. The minimum JSW in the medial femorotibial compartment was measured using automated software [22].

Biochemical biomarkers

Eighteen serum biomarkers included Cartilage Oligomeric Matrix Protein Cartilage Oligomeric Matrix Protein (sCOMP), Hyaluronic Acid (sHA), Type IIA Procollagen Amino-terminal Propeptide (sPIIANP), Type I Collagen C-terminal Telopeptide (sCTX-I), Aggrecan Chondroitin Sulfate 846 Epitope (sCS846), Matrix Metalloproteinase-3 (sMMP-3), Cleavage neoepitope of type II collagen (sC2C), Type II Collagen Neoepitope (sC1, 2C), C-Propeptide of type II collagen (sCPⅡ), sNTX-Ⅰ, and Nitrated triple helix of type II collagen (sColl2−1NO2). Urine biomarkers included C-terminal cross-linked telopeptide of type Ⅰ collagen (α-isomer) (uCTX-Ⅰα), C-terminal cross-linked telopeptide of type I collagen (β-isomer) (uCTX-Ⅰβ), uNTX-Ⅰ, Cleavage neoepitope of type II collagen (uC2C), Type II collagen neoepitope (uC1, 2C), Nitrated triple helix of type II collagen (uColl2−1NO2), and uCTX-Ⅱ. These markers reflect processes such as cartilage degradation and synthesis, bone turnover, and joint inflammation [12]. Serum and/or urine biochemical markers were measured, and urinary markers were standardized based on urinary creatinine concentration. The interassay coefficients of variation for these markers ranged from 3% to 12% [12].

MRI protocol and assessment

This study is based on baseline SAGittal 3-Dimensional Double Echo Steady-State with selective Water Excitation (SAG-3D-DESS-WE) MRI data acquired by the FNIH OA Biomarkers Consortium cohort study using Siemens Trio 3.0 Tesla scanners (Magnetom Trio, Siemens Healthcare, Erlangen, Germany) [23]. The SAG-3D-DESS-WE series utilizes near anisotropic voxels (0.7 mm slice thickness × 0.37 mm × 0.46 mm) to maximize in-plane sagittal spatial resolution in a reasonable acquisition time (10.5 min). The protocol of SAG-3D-DESS-WE can be found in S1 Table.

Two experienced musculoskeletal radiologists (F. Roemer and A. Guermazi) independently assessed the MRIs according to Magnetic resonance imaging OsteoArthritis Knee Score (MOAKS). All scores showed substantial (0.61–0.8) or high (0.81–1.0) agreements regarding intra-observer and inter-observer reliabilities.

An automated MRI segmentation approach using CNN [15] was developed for six anatomical structures (femur, tibia, femoral and tibial cartilages [16], and both meniscus [17]. To compute features that might be suitable for being used as biomarkers for KOA progression, we employ methods of automatic image segmentation to specify the anatomical Volume of Interests (VOIs) using CNN. Ambellan and colleagues [16] used the method to segment the femur, tibia, femoral, and tibial cartilage. The method of Tack and colleagues [17] was used to segment the medial and lateral meniscus.

VOIs (n = 20 knees) defining the femur, tibia, femorotibial cartilages, and meniscus were manually adjusted by two independent authors (including S.F.L., with 6 years of experience in orthopedics), all of whom were blinded to the clinical outcome data. The segmentation and MRI were viewed and adjusted using itk-SNAP version 3.8.0 software (www.itksnap.org). The Dice Similarity Coefficient (DSC) was used to assess manual adjustment and automated segmentation agreement (S4 Table).

Double echo steady-state signal feature map of load-bearing tissues

We developed the feature maps of load-bearing tissues using their own Double Echo Steady-State (DESS) signal intensity’s (mean pixel value), which builds upon previous studies that visualized the texture of knee MRI [24–26]. We used this method to detect the MRI changes in KOA progression participants.

Model development and test

In the FNIH OA Biomarkers Consortium cohort study, excluding 47 knee MRIs with non-conforming MR images, there were 594 participants with 1,753 knee MRIs selected during a 2-year follow-up. The knee MRIs were randomly split into a development cohort and a test cohort with a ratio of 1:1 at each visit time point. Each time point (−100 samples per group) would yield only −20 samples with an 8:2 or 7:3 split. A 1:1 split ensured better representation, enhancing model reliability. Although the FNIH dataset was originally constructed as a nested case-control study with matched pairs, we disrupted the original matching during the cohort split. Specifically, participants were randomly assigned to the development and test cohorts, disregarding their original case-control pairings. This strategy allowed us to evaluate model performance in a setting more reflective of real-world clinical variability and supported the use of generalized learning approaches, rather than those relying on strict pairwise comparison. The ratio of JSN and pain progression, JSN progression, pain progression, and non progression groups was 2:1:1:2 in each cohort.

The predictive model was developed in a total development cohort (n = 877), which was integrated by the development cohort of three visits. This model was tested in a total test cohort (n = 876), which incorporated by test cohort of three visits, including test cohort 1 (n = 301) at baseline, test cohort 2 (n = 297) at 1-year follow-up, and test cohort 3 (n = 278) at 2-year follow-up (Fig 1). The predictive performance was validated using 10-fold cross-validation (Repeated 100 interactions). To assess model stability, we compared LBTRBC-M performance across 100, 300, 1,000, and 25,000 cross-validation iterations (S18 Table). Given the limited performance gain and the high computational cost, we chose 100 iterations for the final analysis as a practical balance between accuracy and efficiency. The specifications of our hardware specs were Intel Core i5-1135G7 Central Processing Unit 2.40 GHz, 16 GB Random access memory, and a 512 GB Solid-State Drive.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Participant flow and MRI timeline.

A total of 600 eligible knees were identified from the FNIH OA Biomarkers Consortium cohort. After excluding 47 non-conforming knee MRIs, 1,753 knee MRIs were included over the 2-year follow-up period: 594 at baseline, 575 at 1-year follow-up, and 584 at 2-year follow-up. The predictive model was developed by total development cohort (877 knee MRIs) and tested by total test cohort (876 knee MRIs) which both were incorporated by three visits: baseline, 1-year follow-up, and 2-year follow-up. The ratio between the development cohort and test cohort was 1/1 in each visit time point. FNIH OA Biomarkers Consortium cohort: Foundation of the National Institutes of Health Osteoarthritis Biomarkers Consortium cohort, MRI: Magnetic Resonance Image, JSN: Joint Space Narrowing.

https://doi.org/10.1371/journal.pmed.1004665.g001

Three-dimensional MRI radiomic feature analysis

LBTRBC-M was developed in the total development cohort by integrating MRI radiomic features from load-bearing tissues, biochemical biomarkers, and clinical high-risk variables (Fig 2). The MRI protocol (S1 Table) and automatic CNN-based segmentation scheme (S2 Fig) were shown in our study. Image Biomarker Standardisation Initiative (IBSI) standardized MRI radiomic features [27] were extracted using Standardized Environment for Radiomics Analysis (SERA) package [28] from baseline, 1-year follow-up, and 2-year follow-up sagittal SAG-3D-DESS-WE MRI, and predictive models were constructed using an XGBOOST algorithm after using Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression (Repeated 1,000,000 times) for feature selection (S4 and S5 Figs). We have included a comparison of the number of features before and after LASSO selection. Initially, 2,947 features were considered in the model: 2,922 MRI radiomic features, 17 biochemical biomarkers, and 7 clinical variables. After applying LASSO, 255 non-zero features were retained in the final model, including 236 MRI radiomic features, 13 biochemical biomarkers, and 6 clinical variables. In our analysis, we utilized Statistical Analysis System software (SAS, version 9.4), leveraging the generalized regression platform with the LASSO method. The selection of the optimal λ was guided by the adaptive weighting approach combined with the Akaike Information Criterion, corrected (AICc) validation method. Specifically, λ was chosen from the optimal solution path where the AICc value was minimized, ensuring the best balance between model fit and complexity. The assumptions underlying LASSO regression, including the distribution of errors and the linearity of relationships between predictors and the outcome, were assessed during model development. The linearity assumption was checked through exploratory data analysis and transformations applied to variables as necessary. Residual diagnostics were performed to confirm that the error terms were approximately normally distributed. In addition to LBTRBC-M, several other models, including the Load-Bearing Tissue Radiomic Model (LBT-RM), Femur Radiomic Model (FE-RM), Femoral Cartilage Radiomic Model (FC-RM), Tibia Radiomic Model (TI-RM), Tibial Cartilage Radiomic Model (TC-RM), Lateral Meniscal Radiomic Model (LM-RM), Medial Meniscal Radiomic Model (MM-RM), MOAKS models, Biochemical biomarker Model (BM), and Clinical Model (CM) were constructed for comparison.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Workflow of the study.

Knee MRIs of the FNIH OA Biomarkers Consortium cohort were automatically segmented by CNN for feature extraction. After feature evaluation and modeling using the LASSO and XGBOOST algorithm, respectively, eight types of features (biochemical biomarkers, clinical variables, FE, FC, TI, TC, LM, and MM MRI radiomic features) were generated and further used to develop the LBTRBC-M. The performance of LBTRBC-M in predicting KOA progression (i.e., JSN and pain progression vs. JSN progression vs. pain progression vs. non progression) was tested in three visits. FNIH OA Biomarkers Consortium cohort: Foundation of the National Institutes of Health Osteoarthritis Biomarkers Consortium cohort, SAG-3D-DESS-WE: Sagittal 3-Dimensional Double Echo Steady-State with selective Water Excitation, CNN: Convolutional Neural Network, XGBOOST: eXtreme Gradient BOOSTing, LASSO: Least Absolute Shrinkage and Selection Operator, FE-RM: Femur Radiomic Model, FC-RM: Femoral Cartilage Radiomic Model, TI-RM: Tibia Radiomic Model, TC-RM: Tibial Cartilage Radiomic Model, LM-RM: Lateral Meniscal Radiomic Model, MM-RM: Medial Meniscal Radiomic Model, LBTRBC-M: Load-Bearing Tissue Radiomic plus Biochemical biomarker and Clinical variable Model, JSN: Joint Space Narrowing, SERA: Standardized Environment for Radiomics Analysis, MRI: Magnetic Resonance Image.

https://doi.org/10.1371/journal.pmed.1004665.g002

Selection of hyperparameters, model fitting and optimization process

The hyperparameters for XGBOOST and LASSO were selected based on prior literature and initial experiments. Specifically, the values for XGBOOST (max_depth: 6, subsample: 1, colsample_bytree: 1, min_child_weight: 1, α: 0, λ: 1, learning_rate: 0.3, iterations: 100) were chosen in accordance with recommendations in previous studies and through empirical testing to optimize model performance (S16 Table). These settings were further refined using grid search (150 grid points) with cross-validation to identify the best-performing combination. For LASSO, the penalty parameter (λ) was optimized using cross-validation to minimize the mean squared error (MSE). The selection of hyperparameters was guided by Hastie and colleagues [29] and adapted to the specifics of the dataset and model requirements.

In this study, we used XGBOOST with 10-fold cross-validation to fit and optimize the model. Model performance was assessed using metrics such as AUC, accuracy, Logloss, and Root Average Squared Error (RASE). Key hyperparameters (S15 Table), including max_depth, learning_rate, subsample, and λ, were optimized via grid search with cross-validation, with practical constraints based on prior literature (e.g., max_depth between 3 and 10, learning_rate between 0.1 and 0.3).

Reader experiments

Seven resident physicians (Clinical practice years: 1–4 years, physicians list: Liu, Zhao, Cao, J Li, Chen, X Wang, Dang, M Zhang) participated in a study to predict the progression of KOA using knee MRI (SAG-3D-DESS-WE sequence), biochemical biomarkers (17 types of serum and urine biomarkers), and clinical variables (Age, sex, Body Mass Index (BMI), race, WOMAC pain score, WOMAC disability score, and pain medication use). The study also evaluated the use of LBTRBC-M in assisting the predictions. The LBTRBC-M system provided the probability of four clinical outcomes (JSN and pain progression, JSN progression, pain progression, and non-progression). We used color coding to differentiate prediction results within various probability ranges. Since the outcome variable was a four-class label, random prediction would assign approximately 25% probability to each class. Therefore, when the LBTRBC-M model output a probability above 25.0% for a given class, we considered this as indicative of stronger prediction confidence and displayed the results were displayed in red font; otherwise, they were shown in black font. This color-coding scheme was designed to help resident physicians better understand and utilize the model’s output.

Evaluation of model performance

We evaluated the accuracy of the XGBOOST MRI radiomic models in the prediction of KOA progression using ROC analysis. Evaluation metrics were computed, including the Area Under ROC Curve (AUC), sensitivity, specificity, and kappa value. In this study, a “successful prediction” is defined through a combination of predictive accuracy and clinical relevance. From a statistical perspective, a prediction is considered successful if the model achieves an AUC greater than 0.70, a widely accepted threshold indicating reliable discriminative performance for clinical decision-making [30]. Beyond statistical accuracy, clinical utility serves as a crucial measure of success. A prediction is clinically meaningful if it helps identify high-risk patients likely to experience JSN progression and/or pain progression within the next two years. Furthermore, the model’s ability to integrate multiple biomarkers and clinical variables enhances its capacity to provide a comprehensive risk assessment, ensuring its applicability in real-world clinical decision-making. The performance of the predictive model was validated using 10-fold cross-validation (S3 Fig).

Missing data

Missing data were addressed using multiple imputation (Predictive mean matching). Specifically, data were missing for 2 knees in the MOAKS femur lateral posterior bone marrow lesion percentage (lesion that is edema), 13 knees for serum biomarkers, and 10 knees for urine biomarkers. The percentages of missing data were 0.3%, 2.1%, and 1.7%, respectively. Given these low percentages, the impact on the results was deemed minimal.

Statistical analysis

Generalized Estimating Equation (GEE) was used to assess the risk of KOA progression adjusting for the confounders (age, sex, BMI, knee side, race, WOMAC knee pain score) [31–34]. These confounders were selected based on prior literature and their potential impact on KOA progression. The GEE model assumed an autoregressive correlation structure, as this was deemed most suitable for the nature of the repeated measures in our data. MRI radiomic feature extraction and DSC calculation were performed in Matlab R2021a (version 9.10.0). GEE, LASSO logistic regression, XGBOOST modeling, and the Delong test were performed in the SAS. Model performance was assessed using AUC, sensitivity, specificity, accuracy, and kappa value in all development and test cohort. We compared all cohorts using the unpaired t test/one-way ANOVA tests for continuous variables and the χ² test, Mann–Whitney test, or Kruskal–Wallis test for categorical variables, as appropriate. For continuous variables, parametric tests (e.g., unpaired t test, one-way ANOVA) were employed when data were normally distributed, as assessed using the Shapiro–Wilk test. Non-parametric tests (e.g., Mann–Whitney test, Kruskal–Wallis test) were used when the assumption of normality was not met. For categorical variables, χ² tests were used to compare proportions across groups. To control for type I error in our analysis, both Tukey’s honestly significant difference (HSD) test and Bonferroni correction were applied during one-way ANOVA tests for multiple comparisons. Area differences between two Receiver Operator Characteristic (ROC) curves were compared using the DeLong test. For Bonferroni correction, the adjusted P value threshold was calculated by dividing 0.05 by the number of comparisons. Accordingly, we considered results statistically significant if the adjusted p value was below this threshold. The exact adjusted significance levels are reported where relevant. A P value < 0.05 was considered statistically significant unless otherwise specified after correction. R (version 4.1.1) with the pROC package (version 1.18.0) was used for all analyses.

Baseline characteristics

At the baseline visit, 293 (178/293, 61% females) individuals were included in the development cohort 1, while 301 (171/301, 57% females) individuals were included in the test cohort 1 (S2 Table). Results of baseline characteristics showed no significant differences between the two cohorts (S2 Table).

Baseline characteristics among the four groups (JSN and pain progression, JSN progression, pain progression, and non-progression) were generally similar, except for age (p = 0.018), female (p = 0.016), and KLG (p = 0.003) in test cohort 1 (Table 1). In addition, the levels of baseline serum/urine biochemical biomarkers showed no significant differences between the four groups in both cohorts (S3 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline characteristics of participants in development cohort 1 and test cohort 1.

https://doi.org/10.1371/journal.pmed.1004665.t001

Reliability of automatic segmentation

The CNN segmentation dataset was utilized for extracting MRI radiomic features, and all DSCs were above 0.800 (S4 Table). The feature selection process using LASSO regression was displayed in S4 and S5 Figs, and the selected features can be found in S5 Table.

Feature maps of load-bearing tissues

Fig 3 illustrates the distinct DESS signal intensity changes observed among the four groups across load-bearing structures. The high values of the femur and tibia were detected in JSN and pain progression, and pain progression group. The high values of femoral cartilage, tibial cartilage, lateral meniscus, and medial meniscus were detected in JSN and pain progression, and JSN progression group. As shown in S6 Fig, in the femur, tibia, and medial meniscus, the JSN and pain progression group exhibited the highest MRI radiomic values, followed by the pain progression group, and then the JSN progression group. Conversely, in the femoral cartilage, tibial cartilage, and lateral meniscus, the JSN and pain progression group had the highest MRI radiomic values, followed by the JSN progression group, and then the pain progression group.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The DESS signal feature maps of load-bearing tissues in different groups and prediction performance of LBT-RM, LBT-MOM, BM, CM, BCM, and LBTRBC-M in the test cohorts.

DESS signal intensity maps of load-bearing tissues were developed in JSN and pain progression (A), JSN progression (B), pain progression (C), and non progression group (D). The performance of predicting JSN and pain progression (E–H), JSN progression (I–L), pain progression (M–P), and non progression (Q–T) in LBT-RM, LBT-MOM, BM, CM, BCM, and LBTRBC-M in the test cohort 1 to 3 and the total test cohort. The results of test cohort 1, test cohort 2, test cohort 3, and the total test cohort corresponded to baseline, 1-years follow-up, 2-year follow-up, and encompassed the aforementioned follow-up time points. LBT-RM: Load-Bearing Tissue Radiomic Model, LBT-MOM: Load-Bearing Tissue MOAKS Model, BM: Biochemical biomarker Model, CM: Clinical Model, BCM: Biochemical biomarker plus Clinical variable Model, LBTRBC-M: Load-Bearing Tissue Radiomic plus Biochemical biomarker and Clinical variable Model, AUC: Area Under receiver operating characteristic Curve, MOAKS: Magnetic resonance imaging OsteoArthritis Knee Score, DESS: Double Echo Steady-State.

https://doi.org/10.1371/journal.pmed.1004665.g003

Algorithm selection

In our study, the performance of the LBTRBC-M model was compared across several machine learning algorithms, including XGBOOST, bootstrap forest, shallow neural network, support vector machines, decision tree, nominal logistic, and naive bayes (S15 Table). Among these, XGBOOST demonstrated the best overall performance, achieving the highest AUC (0.897) and the lowest Root Average Squared Error (RASE) (0.508). Additionally, XGBOOST had a favorable Entropy R Square (ERS) of 0.409 and a low misclassification rate (0.299), outperforming all other algorithms, including bootstrap forest and shallow neural network. Given its superior discriminative ability and error minimization, XGBOOST was selected as the optimal algorithm for predicting KOA progression in this study.

Performance of single-structure MRI radiomic models: Comparison with MOAKS imaging marker

The Single-Structure Radiomic Models (SS-RMs), including FE-RM, FC-RM, TI-RM, TC-RM, LM-RM, and MM-RM, showed low kappa values ranged from 0.042 to 0.296 (S7 Fig), moderate AUCs ranged from 0.507 (95% confidence interval (CI) [0.424, 0.589]) to 0.747 (95% CI [0.685, 0.801]) (S8–S13 Figs and S6 Table), and low accuracy ranged from 30.9% to 51.5% (S7 Table) in the test cohorts.

In the total test cohort, comparisons with Single-Structure MOAKS Models (SS-MOMs), SS-RMs showed significant AUC improvement in predicting KOA progression, including JSN and pain progression, JSN progression, and pain progression, except for FC-RM versus FC-MOM in predicting JSN and/or pain progression (S8 Table).

Performance of LBTRBC-M: Comparison with SS-RMs

The integration of SS-RMs into the LBT-RM resulted in improved predictive accuracy. The LBT-RM achieved kappa values ranged from 0.355 to 0.450 (S14 Fig), AUCs ranged from 0.761 (95% CI [0.685, 0.823]) to 0.857 (95% CI [0.796, 0.903]) (Fig 3E–3T), and accuracy ranged from 54.0% to 61.6% (S7 Table) in the test cohorts. In comparison, the Load-Bearing Tissue MOAKS Model (LBT-MOM), BM, CM, and Biochemical biomarker plus Clinical variable Model (BCM) had lower AUC values in the test cohorts, ranging from 0.643 (95% CI [0.541, 0.733]) to 0.736 (95% CI [0.668, 0.794]), 0.694 (95% CI [0.597, 0.776]) to 0.777 (95% CI [0.717, 0.827]), 0.563 (95% CI [0.487, 0.636]) to 0.741 (95% CI [0.669, 0.802]), and 0.706 (95% CI [0.613, 0.784]) to 0.822 (95% CI [0.751, 0.877]), respectively (S6 Table). Furthermore, the kappa values, AUCs, and accuracy of LBTRBC-M ranged from 0.551 to 0.628, 0.869 (95% CI [0.819, 0.907]) to 0.919 (95% CI [0.883, 0.946]), and 68.0% to 74.1% in the test cohorts (Fig 3E–3T and S6 and S7 Tables).

When predicting KOA progression, LBTRBC-M outperformed LBT-RM, BM, CM, BCM, and Load-Bearing Tissue MOAKS Model (LBTMBC-M) with significant AUC differences (Tables 2 and S8, p < 0.050). To predict JSN and pain progression, the AUC differences between LBTRBC-M and LBT-RM, BM, CM, BCM, and LBTMBC-M were 0.072 (95% CI [0.044, 0.100]; p < 0.001), 0.122 (95% CI [0.088, 0.157]; p < 0.001), 0.224 (95% CI [0.183, 0.264]; p < 0.001), 0.100 (95% CI [0.067, 0.134]; p < 0.001), and 0.096 (95% CI [0.064, 0.128]; p < 0.001) in the total test cohort, respectively (Table 2). Additionally, As shown in S8 Table, when predicting KOA progression, there was no significant AUC difference between LBT-RM and BCM in the total test cohort, such as JSN and pain progression, the AUC difference was 0.028 (95% CI [−0.015, 0.072], p = 0.203), for JSN progression, 0.023 (95% CI [−0.029, 0.075], p = 0.379), and for pain progression, 0.030 (95% CI [−0.026, 0.086], p = 0.288).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparing the areas under two correlated ROC curves between predictive models in the test cohorts.

https://doi.org/10.1371/journal.pmed.1004665.t002

Association of the model output with KOA progression

S9 Table displays the Odds Ratio (OR) of KOA progression for predictive model output in the entire cohort. The output of LBTRBC-M showed the highest ORs of JSN and pain progression in the predictive models. The adjusted OR of JSN and pain progression for LBTRBC-M output was 30.906 (95% CI [22.470, 42.511]); the adjusted OR of JSN progression for LBTRBC-M output was 6.465 (95% CI [4.740, 8.820]); and the adjusted OR of pain progression for LBTRBC-M output was 3.307 (95% CI [2.457, 4.452]).

Performance of LBTRBC-M-supported resident physicians: Compared with no LBTRBC-M-supported resident physicians

Seven resident physicians predicted the KOA progression using knee MRI, biochemical biomarkers, and clinical data (Fig 4A). Based on the predictive performance of individual physicians (S10 and S11 Tables), we found the assistance of LBTRBC-M significantly improved the accuracy of resident physicians in predicting KOA progression from 46.9% (95% CI [44.7%, 49.0%]) to 65.4% (95% CI [64.4%, 66.5%]) in the total test cohort (S12 Table). In addition, with LBTRBC-M support, the performance of physicians in predicting JSN and pain progression were also improved, with increased sensitivity and specificity of 68.1% (95% CI [66.2%, 70.0%]; p < 0.050) and 80.4% (95% CI [78.9%, 81.8%]; p < 0.050) in the total test cohort, respectively {compared to 57.5% (95% CI [51.7%, 63.2%]) and 51.8% (95% CI [48.8%, 54.9%]) without LBTRBC-M support, respectively}. Similarly, the prediction performance, including sensitivity and specificity of JSN progression, pain progression, and non progression in physicians, showed improvement with LBTRBC-M assistance. These improvements were validated in different test cohorts of visits (Fig 4B–4I and S15 and S12 Tables).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Performance of resident physicians and models in predicting the KOA progression in the total test cohort.

A schematic workflow of the assessment of KOA progression risk by the resident physicians with LBTRBC-M assistance (A). AUC for predicting JSN and pain progression (B), JSN progression (D), pain progression (F), and non progression (H) among the LBTRBC-M, LBTMBC-M, and average performance of all resident physicians without (blue dot) and with (red dot) the support of LBTRBC-M in the total test cohort was demonstrated. The performance of LBTRBC-M was superior to LBTMBC-M that of clinical experts given baseline MOAKS, serum biochemical level, and clinical data; however, as shown in B, D, F, and H, both the sensitivity and specificity of resident physicians were improved when assistance was provided by LBTRBC-M (black arrow). Individual performance of resident physicians was represented by open shapes (without LBTRBC-M support) and filled shapes (with LBTRBC-M support). The prognostic performance of resident physicians was greatly boosted by integrating LBTRBC-M into the loop, as shown in dashed connection lines in C, E, G, and I. The results of the total test cohort encompassed the baseline, 1-years follow-up, and 2-year follow-up points. Each colored line represents a different resident physician. AUC: Area Under receiver operating characteristic Curve, LBTRBC-M: Load-Bearing Tissue Radiomic plus Biochemical biomarker and Clinical variable Model, LBTMBC-M: Load-Bearing Tissue MOAKS plus Biochemical biomarker and Clinical variable Model, MOAKS: Magnetic resonance imaging OsteoArthritis Knee Score, AI: Artificial Intelligence.

https://doi.org/10.1371/journal.pmed.1004665.g004

Sensitivity analyses

To evaluate the robustness of our findings, sensitivity analyses were performed by re-running the GEE model using alternative correlation structures, including independent, exchangeable, and unstructured. The results were compared across these assumptions, and the analyses confirmed that our findings remained consistent and robust (S13 Table). To assess the impact of missing data on the study’s findings, sensitivity analyses were conducted (S14 Table). These included: comparing the results from the multiple imputation approach with those obtained from a complete case analysis (excluding cases with missing data). The results were consistent across both approaches, confirming that the handling of missing data did not significantly alter the study’s conclusions. To evaluate model robustness, we performed a sensitivity analysis by varying key hyperparameters (S17 Table) such as max_depth, learning_rate, subsample, and λ. We used Delong test to assess the effect of each parameter on model performance. This analysis aimed to identify the most influential parameters and quantify uncertainty, improving the model’s reliability and understanding its behavior in predicting KOA progression.

Testing of interactions

We tested the significance of interaction terms by comparing AUC differences between model combinations (LBTRBC-M versus LBTRB-M, LBTRBC-M versus LBTRC-M, LBTRC-M versus LBTRC-M) using the DeLong test. The results, presented in S8 Table, showed no significant AUC differences across test cohorts 1, 2, 3, and the total cohort. This indicates that adding different features (biochemical biomarkers, clinical variables) did not significantly improve model performance.

Probability distribution patterns in LBTRAB-M model predictions

In our contour plot analysis of the LBTRAB-M model (S3C Fig), probability distributions varied across progression categories. JSN and pain progression clustered between 5.0% and 99.0%, JSN progression between 15.0%−25.0% and 55.0%−75.0%, pain progression between 2.0% and 25.0%, and non-progression between 50.0%−99.0%. These patterns suggest distinct probability thresholds for each outcome, highlighting the need for ROC and precision-recall curve analyses to optimize sensitivity and specificity for clinical use.

Stratified cross-validation

To ensure a balanced distribution of outcome types across datasets, we performed a stratified split of the total development and test cohort for the LBTRBC-M model, maintaining an approximate 2:1:1:2 ratio of JSN and pain progression, JSN progression, pain progression, and non-progression (S16A and S16B Fig). Prior to stratification, the AUCs in the total test cohort were: 0.880 (95% CI [0.853, 0.903]) for JSN and pain progression, 0.913 (95% CI [0.881, 0.937]) for JSN progression, 0.886 (95% CI [0.856, 0.910]) for pain progression, and 0.909 (95% CI [0.888, 0.926]) for non-progression. Following stratification, the corresponding AUCs were: 0.853 (95% CI [0.826, 0.877]) for JSN and pain progression (ΔAUC = 0.027, 95% CI [−0.003, 0.058]; p = 0.080); 0.860 (95% CI [0.823, 0.891]) for JSN progression (ΔAUC = 0.053, 95% CI [0.016, 0.090]; p = 0.005), 0.878 (95% CI [0.843, 0.906]) for pain progression (ΔAUC = 0.008, 95% CI [−0.031, 0.047]; p = 0.697), and 0.853 (95% CI [0.824, 0.877]) for non-progression (ΔAUC = 0.056, 95% CI [0.030, 0.082]; p < 0.001) (S16 and S16D Fig and S19 Table).

Ethical approval

All deidentified patient level clinical data, outcome data, and MRI data in our study were obtained from the OAI, an ongoing, multicenter, prospective cohort study designed to identify biomarkers for KOA. The Health Insurance Portability and Accountability Act (HIPAA) compliant protocol of the OAI study had received institutional review board (IRB) approval. The original OAI study protocol was approved by the institutional review boards of all participating centers, including the coordinating center at the University of California, San Francisco (IRB number: 10-00532).