2.1.1. Data collection and preprocessing.

We collected the paired 58 sagittal MRIs and the corresponding multi-omic profiles (mRNA gene expression, DNA methylation, and copy number variation) of the patients from TCIA-BRCA and TCGA-BRCA projects. Since we would use a 15-fold cross-validation to train and test the models, each fold would have approximate 4 independent samples in the test set and 54 samples in the training set. The collected 3D sagittal MRIs comprised varying number of slices. To make it easy to interpret and perform visual inspections, these 3D MRIs were represented as 2D orthographic projections [25–26]. Then, to reduce computational complexity, these MRIs were resized to 128 × 128 by utilizing the nearest pixel method [27]. There were also some steps to process the multi-omic data. After collecting the raw multi-omic data, samples or features with all-zero values were removed to ensure data quality. Then, to focus on the most variable and informative features, the top 10 percent of the genes with the largest Coefficient of Variation (CV) were selected. This filtering process resulted in retaining 4515 genes for 754 patients in the final data matrices. The data matrices for each patient include a 2D mRNA gene expression matrix, a 2D DNA methylation matrix, and a 2D copy number variation matrix. These data matrices were then used to construct a 3D tensor, serving as the input for Bayesian tensor factorization (BTF) [28]. The inputted 3D tensor was finally decomposed into a 2D matrix with 17 latent factors for each patient. To further analyze the adaptive ability of the model, we also collected a set of gene expressions of 123 ER+/HER2+ BC patients from the TCIA-BRCA and the TCGA-BRCA projects to repeat the experiment.

2.1.2. Forward diffusion.

MRIs were degraded to pure noise images in the forward diffusion process. We presented the MRIs as x₀ and pure noise images as x_T, where T was the total steps assigned to degrade an MRI. The process of reasoning x_T was not instantaneous. Designing a Markov chain to obtain intermediate states x_t as transitions was the common strategy [16]. Specifically, due to the convenience of Gaussian noise in sampling data distributions, the model would add scheduled Gaussian noise into the sample to obtain a state at the deeper time step [29]. Mathematically, this could be represented as , where x_t was the distribution of the current noised image, x_t-1 was the distribution of the previous noised image, β_t was the variance schedule, 0<β_t<1, β₀ was the smallest number and β_T was the largest number and I was an identity matrix. Based on the definition of the Gaussian noise, q(x_t|x_t−1) could be further reparametrized as (1) , where (0,1). Let and . The Eq 1. could be substituted as (2) , which was able to be further merged as (3) , representing any intermediate states x_t in terms of the inputted MRI x₀. Fig 3A illustrated this process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Diffusion procedure.

A. Forward diffusion. B. Backward diffusion.

https://doi.org/10.1371/journal.pcbi.1012490.g003

2.1.3. Backward diffusion.

In the backward diffusion process, a new MRI would be synthesized from a pure Gaussian noise. In mathematical terms, the distribution of the synthesized MRI was represented as q(x_t−1|x_t, x₀). Due to the irreversible nature of the Markov chains, the strategy using to solve q(x_t−1|x_t, x₀) was to train a deep neural network model (U-Net based framework) to approximate the distribution of it. Before using U-Net framework to approximate the data distribution, employing Bayes’ rule to q(x_t−1|x_t, x₀), it would give (4) , where C(x_t, x₀) was a constant. Referred to the Eq 4., the mean and variance of q(x_t−1|x_t, x₀) could be parameterized as (5)

and (6) according to the standard Gaussian density function. Reformatting the Eq 3., x₀ could be expressed as Eq 7.

(7)

Referring to Eq 6. and Eq 7., could be further conceptualized as Eq 8.

(8)

According to the Eq 8. the mean μ_θ of the neural network-approximated distribution q(x_t−1|x_t, x₀) could be constructed as (9) , where ε_θ was the noise predicted by the deep neural network and τ was the condition. The loss function of the deep neural network would then be defined as Eq 10. Fig 3B illustrated the process of removing noise from noisy image in the backward diffusion.

(10)

Table 1 Algorithm 1 showed the pseudo-code of the training the CPDM and Table 1 Algorithm 2 showed the pseudo-code of synthesizing new MRIs according to the genomic information of patients.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Pseudo-codes for the CPDM.

https://doi.org/10.1371/journal.pcbi.1012490.t001

2.1.4. Conditioning.

2.1.4.1. Cross-attention. We strategically placed the cross-attention module between the downsampling and upsampling processes of the U-Net model to align the imaging information with the multi-omic information [30]. The cross-attention mechanism allowed the model to attend to relevant parts of the MRIs (say, modality B) based on the content of the multi-omics data (say, modality A), enabling the generation of MRI projections that match the multi-omic data. The essence of the cross-attention mechanism was interacting the query vectors from modality A with the key vectors from modality B to compute attention scores. These attention scores were then used to weight the value vectors from modality B, to capture the cross-modal interactions between these two modalities. Mathematically, the query vector for the modality A could be represented as , where X_A is the representation of the modality A and is a learnable weight matrix. Similarly, the key vector for modality B could be represented as , and the value vector for modality B could be represented as , where X_B is the representation of the modality B, and and are learnable weight matrices. Then, the attention scores (A) from modality A to modality B could be calculated by (11) , where is the dimension of the key vectors. The attention scores of modality A were used to calculate the weighted aggregation of the value vectors (WV) of modality B. The weighted value vectors could then be calculated as the dot product of the attention scores of modality A and the value vector for modality B, i.e. WV = A∙V_B. In the end, we computed the final output obtained from the cross-attention mechanism after aggregating the information from modality B using the attention scores computed from modality A, which could be used as input for subsequent layers. We call this a fused representation (FR) and (12) , where is an additional learnable weight matrix for the aggregation.

2.1.4.2. Integration of multi-omic and MRI data. To facilitate the synthesis of MRI projections that were both visually faithful and biologically significant, we used the inner product to integrate the compact multi-omic feature vectors, denoted as F, with detailed MRI projection images, denoted as P. In mathematics, the inner product X involved calculating the dot product between the genomic information vector and the MRI projection image [31]. So, we had (13) , where X_ij represents each element in the result of the inner product, F and P were feature matrix and image matrix both with size n×n, F_ik and P_kj represented the individual elements of the feature vector and image, respectively. This operation resulted in a matrix that encapsulates the correlations and interactions between the two data modalities. Each value in the resulting matrix represented the cumulative contribution of the corresponding elements from the multi-omics feature vector and the image. This matrix encoded the alignment and correlation between the genomic features and the MRIs. This matrix could then be used as a fusion mechanism to guide the synthesis of MRIs. We implemented the fusion mechanism by adding our computed inner product into the image matrix to update the Noisy Input (NI) of the model, i.e. (14) , where P was the noisy image and X was the inner product of the multi-omic feature vector and the image vector. This encapsulated both the molecular insights from the feature vectors and the visual information from the images [32]. Finally, both the noisy input and the feature vector were treated as the input of the CPDM.

2.1.5. CPDM evaluation.

The evaluation of CPDM was based on three performance metrics: Fréchet inception distance (FID), Mean Square Error (MSE) and Structural Similarity Index Measure (SSIM). FID was used to compare the similarity between the real image and the synthetic image by calculating the Fréchet distance. FID was articulated as (15) , where μ_x, μ_g represented the feature-wise mean of the real and synthetic images, while ∑_x, ∑_g were their covariance matrices [33]. A low FID score indicated a high degree of similarity between images. Achieving a score of 0.00 signifies that the two images were identical. We also calculated the Fréchet Inception Distance STandard Deviation (FID-STD) to evaluate the consistency and stability of the CPDM. This metric could assess the variation in the model’s performance across different runs or image batches. A lower FID-STD value indicated a higher degree of consistency in the quality of images generated by the CPDM, signifying its reliability. MSE quantified the average squared difference between estimated values and actual values and was extensively utilized in image reconstruction tasks. MSE emphasizes the accuracy of the reconstructed images, which was given by (16) , where I_i, were the pixel values of the real and generated images, respectively, and n was the number of pixels [34]. A lower MSE value was indicative of superior performance. SSIM was a metric designed to assess the perceived quality of images and their similarity. Differing from FID, SSIM specifically evaluates changes in luminance, contrast, and structure between two images, providing a score that ranges from -1 to 1. SSIM was defined as (17) , where μ_I, were the average intensities, σ_I², were the variances, was the covariance, and c₁, c₂ were constants [35]. A higher SSIM value indicates not only greater similarity between images but also an enhanced perceived quality.

2.2.1. Data collection and preprocessing.

The data collection process involved aggregating mutation status, ER status, survival, and ER+/HER2+ data from the TCGA database (https://www.cbioportal.org/). We prepared datasets for clinical analysis by matching labels to the synthesized MRIs based on the available paired clinical data. Table 2 presents the details of the datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The data distribution and details of datasets used to train XGBoost models.

https://doi.org/10.1371/journal.pcbi.1012490.t002

There are 754 patients with both multi-omic profiles and TP53 mutation status. Among them, 252 patients exhibited mutations in the TP53 gene, designated as 1, while the remaining 502 patients with normal genetic profiles were labeled as 0. For ER status, there are 708 patients with both ER status and multi-omics profiles. Among them, 544 are positive while 164 are negative. For the 66 ER+/HER2+ BCs with multi-omics profiles, 26 of them are from the Subgroup 1 and the other 40 are from the Subgroup 2. Among the 123 ER+/HER2+ BCs with gene expression data, 63 of them belong to the Subgroup 1 while the other 60 belong to the Subgroup 2. Subgroup 1 and subgroup 2 are the further categorizations for ER+/HER2+ BC [36]. The patients from these two subgroups usually have different treatment responses and clinical outcomes.

The approach for the survival dataset entailed compiling patients with recorded survival data, encompassing survival days and outcomes (survival or deceased) and matched MRIs. Considering all 754 patients who had generated MRIs, there were 740 patients with survival data. For the 123 patients who had ER+/HER2+ Subtype data, we could obtain 66 patients who had multi-omic profile-based MRIs and 123 patients with gene expression-based MRIs. This process ensured that the analyses were grounded in robust data intersections, offering a robust foundation for the subsequent analyses.

2.2.2. Classification and prediction.

For binary classification tasks, namely the prediction of mutation status, ER status and ER+/HER2+ subtypes, our approach entailed extracting image features using well-established tools such as PyRadiomics and pre-trained CNN models, including VGG16, ResNet50, and InceptionV3 [37–40]. Due to the complexity of the radiomics BC study, we identified the most proper extracting method in these commonly used tools via experiments. S1 Table shows the number of features extracted from MRIs generated according to the multi-omic profiles and gene expressions, respectively. These features were then employed as input to train XGBoost models, optimized through the RandomizedSearchCV tool [41–42].

Innovatively, we extended the utility of the extracted image features for survival analysis through the adoption of tools such as DeepSurv and CoxPHFilter [43–44]. By utilizing this strategy, we explored new dimensions of patient prognosis.

2.2.3. Evaluations.

In the classification task, the model evaluation results were based on the 10-fold cross-validation method. This process involved partitioning the training set into 10 equal batches. In each validation cycle, one batch was designated as the test set, and the remaining segments were amalgamated to form the training set. The model was trained and tested sequentially across all 10 folds. The overall performance of the model was then ascertained by averaging the outcomes from all 10 tests, ensuring a comprehensive assessment that leverages every data point for both training and validation.

Receiver operating characteristic (ROC) curves, area under the receiver operating characteristic curve (AUROC), precision-recall curves, area under the precision-recall curve (AUPRC), and the F1 score were used to evaluate the performance of XGBoost models. The ROC curve was a graphical representation of a classifier’s performance, plotting the true positive rate against the false positive rate across various decision thresholds. The AUROC (the full mark was 1.00) quantified the area under the ROC curve, with a higher value indicating better classification performance. On the other hand, the precision-recall curve plotted precision against the recall, focusing on the trade-off between accurately identified positive cases (precision) and the total actual positive cases captured (recall). The AUPRC (the full mark was 1.00) reflected the area under this curve, providing a measure of a model’s ability to balance precision and recall. The F1 score was the harmonic mean of precision and recall. The F1 score combined both precision and recall into a single metric, making it useful for evaluating models in scenarios where the class imbalance is present. A higher F1 score (the full mark was 1.00) indicates a better balance between precision and recall.

For the survival analysis tasks, the Concordance-index (C-index) and the p-value from the log-rank test were used to evaluate the performance of the survival models. The C-index stood as a fundamental metric widely employed in survival analysis and medical investigations to assess the effectiveness of predictive models in the context of time-to-event outcomes [45–46]. It served as a yardstick for measuring a model’s competence in correctly ordering pairs of observations with and without events according to their actual survival times. Spanning from 0.5 to 1.0, the C-index offered a concise yet informative gauge of a model’s capability to discriminate between patients exhibiting varying survival durations. By capturing the nuanced interplay between predictors and survival outcomes, the C-index assumed a pivotal role in appraising the predictive performance of models in the realm of medical research. Higher c-index values denoted stronger predictive abilities, indicating a model’s success in accurately ranking patients based on their actual survival times. The p-value from the log-rank test evaluates the other aspect of the survival models, such as the shape of the survival curves, providing information that the C-index may not capture [47]. The log-rank test p-value tests the null hypothesis that there is no difference between the survival curves of different groups. A lower log-rank test p-value (less than or equal to 0.05) indicates a statistically significant difference in survival distributions, thereby demonstrating the models’ ability to distinguish between patient groups with different survival patterns.

3.1.1. Data collection and preprocessing.

In each fold, the 54 samples from the training set consisting of paired multi-omic profiles and real MRI projections were used to train the CPDM, and 4 samples also with paired multi-omic profiles and real MRI projections in the test sets were used to test the performance of the trained model. Then, there were 726 patients with solely multi-omic profiles that could be collected from the TCIA-BRCA project, which could be used to guide MRI projection synthesis.

3.1.2. Model training.

To train a CPDM so it could generate MRI projections according to patients’ multi-omic profiles, we performed iteration steps on the training set consisting of paired MRI projections and multi-omic profiles until the loss was converged. S1A and S1B Fig showed the variation trend of the loss values of the model with the epoch increases. From both figures, along with the increment of the epoch, the losses of the models were decreasing and finally fluctuate between small intervals, reaching convergence. Based on the diagrams, it could be concluded that models have small and stable loss values after 1100 rounds of iteration, which could approximate the real sample distribution well during the denoising process. S2 and S3 Tables (the bold items were the finalized configurations) respectively showed 7 hyperparameters for CPDM optimization for the multi-omic profile version and gene expression version.

In addition, we also trained four classic deep generative models AEs, VAEs, Transformers, and GANs, using the same datasets. These comparisons were conducted to benchmark our model against these established standards, employing consistent performance metrics to critically evaluate and understand the unique strengths and limitations of each approach, thereby highlighting the advancements our models bring to the field.

3.1.3. Model performance.

Table 3 showed the training and testing performances of four baseline models and our CPDMs on the multi-omic dataset and the gene expression dataset. Each model was evaluated based on the four performance metrics. The calculations for FID, FID-STD, MSE, and SSIM were conducted following standardized protocols. From the table, three classical generative models trained on different datasets in this task all yielded results with lower performances than CPDM, especially low SSIM scores (all below 0.5). This reflects the advanced performance of CPDM compared to traditional generative models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance metrics.

https://doi.org/10.1371/journal.pcbi.1012490.t003

To further evaluate the quality of the synthetic images and the performance of the CPDM, we conducted a visual inspection of synthetic images on the test sets. Fig 4 demonstrated the visualized results of the multi-omic and gene expression versions, respectively. In each table, for each patient in the test set, the corresponding real MRI images from the database were displayed alongside a CPDM-synthesized MRI image. Displayed images allowed the comparison of the similarity between real and synthesized images. These comparisons revealed that the MRIs synthesized by the multi-omic data-based CPDM and gene expression data-based CPDM closely aligned with the real MRIs in content and quality, further corroborating the exceptional performance of the CPDMs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The results on the test set.

The figure presented results for four patients included in the test set. Each patient was shown with three images: the real clinical image (Real Img), the image synthesized from their multi-omic profiles (Multi-omic), and the image synthesized from their gene expressions (Gene expr).

https://doi.org/10.1371/journal.pcbi.1012490.g004

Due to the lack of real BC MRI images for patients in the unpaired dataset, we could not employ numerical evaluation for synthetic images. Instead, we resorted to visual inspection, which included inspecting the visual quality of the synthesized images, as well as the similarity between MRIs synthesized by multi-omic profile-based CPDM and gene expression-based CPDM. Fig 5 presented the synthetic results for 5 patients. The patients were selected according to the K-means clustering results of their multi-omic profiles and gene expressions, as shown in S2A and S2B Fig [48]. Through the visual inspections, it could be observed that MRIs synthesized based on the unpaired dataset exhibited granularity as close as real MRIs in the paired dataset. The synthesized MRIs always displayed distinct breast contours and clear tissue structures and could always bring out rich varieties based on the different input features. Moreover, MRIs for an identical patient synthesized by multi-omic data-based CPDM and gene expression data-based CPDM showed similarity in overall structure and detail. Additionally, we also used computational tools, including CharCPT-4 and x-ray interpreter, to further analyze the synthesized images and utilized the patients’ clinical data from TCGA-BRCA to validate the analysis results [49–50]. S4 and S5 Tables presented the results of using the computational tools to analyze the synthetic images. The findings from the computational tools for the synthetic images were consistent with the actual clinical data of the patients, suggesting the reliability of these images. The performance reflects the success of the CPDM in this task, especially its robust generalization capability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Synthetic images from unpaired data.

The figure showed synthetic images for five patients who had genomic data but no real clinical images. The three patients on the left were selected by clustering their multi-omic profiles, while the two on the right were selected by clustering their gene expressions. For each patient, an image synthesized from their multi-omic profiles and an image synthesized from their gene expressions were presented.

https://doi.org/10.1371/journal.pcbi.1012490.g005

3.2.1. Data collection and preprocessing.

The well-trained CPDM was then used to generate 754 (726 patients with unpaired data + 28 patients with paired data) MRIs according to patients’ multi-omic profiles. This step was repeated on the 123 HER2+/ER+ BCs’ gene expression dataset and got 123 synthetic MRIs. These synthetic images were used to predict the mutation status of BC driver genes, BC ER status, BC ER+/HER2+ subtypes, and survival information.

3.2.2. Models training.

S6 and S7 Tables showed the hyper-parameter tuning results of the XGBoost model for the TP53 mutation status prediction task using the MRI features extracted by the ResNet50 model and ER status prediction task using the MRI features extracted by the PyRadiomics tool, respectively. S8 Table showed the hyper-parameter tuning results of the XGBoost model for the ER+/HER2+ subtype prediction task using the MRI features extracted by the PyRadiomics tool.

S9 Table showed the hyper-parameter tuning results of the CoxPHfilter model for 740 patients with multi-omic-guided synthetic MRIs. The MRI features inputted into the model were extracted by the ResNet50 model. S10 Table showed the hyper-parameter tuning results of the CoxPHfilter model for 66 patients with ER+/HER2+ data. The data used to train this model was the features of multi-omic-guided synthetic MRIs, extracted by the ResNet50 model of these patients.

3.2.3. Model performances.

3.2.3.1. XGBoost model for TP53 mutation status and ER status prediction. The TP53 mutation status module in Table 4 showed the results of using different methods to predict the mutation status of the TP53 gene. Comparing these results with the baseline performance of TP53 mutation status prediction in S11 Table, it was observed that the synthetic MRI-based classification outcomes closely matched those derived from patients’ actual multi-omic profiles. This indicated that the synthetic MRI data encapsulated information that was almost consistent with the patients’ real multi-omic profiles. Furthermore, the AUPRCs for TP53 mutation status prediction in Table 4 significantly exceeded the baseline AUPRCs shown in S12 Table, suggesting that the model had learned to recognize the positive cases and made reliable predictions in practice. Additionally, Fig 6A and 6B illustrated the average ROC and precision-recall curves from cross-validation, respectively, further corroborating the classification capability of the model. These findings demonstrated the feasibility of predicting TP53 gene mutation status using synthetic MRIs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Average ROC and precision-recall curve for classification tasks from cross-validation.

A. Average ROC curves of the gene TP53 mutation status prediction based on the XGBoost model trained on the MRI (multi-omic version) features extracted by the PyRadiomics tool. B. Average precision-recall curves of the gene TP53 mutation status prediction based on the XGBoost model trained on the MRI (multi-omic version) features extracted by the PyRadiomics tool. C. Average ROC curves of the ER status prediction based on the XGBoost model trained on the MRI (multi-omic version) features extracted by the PyRadiomics tool. D. Average precision-recall curves of the ER status prediction based on the XGBoost model trained on the MRI (multi-omic version) features extracted by the PyRadiomics tool. E. Average ROC curves of the ER+/HER2+ subtypes prediction based on the XGBoost model trained on the MRI (gene expresssion version) features extracted by the PyRadiomics tool. F. Average precision-recall curves of the ER+/HER2+ subtypes prediction based on the XGBoost model trained on the MRI (gene expresssion version) features extracted by the PyRadiomics tool.

https://doi.org/10.1371/journal.pcbi.1012490.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. The results of the TP53 mutation status and ER status prediction.

https://doi.org/10.1371/journal.pcbi.1012490.t004

The ER status module in Table 4 presented the results of using different methods to predict the ER status of patients. The values in the table demonstrated that the AUPRC and F1 scores were comparable with the baseline results in S11 Table and the AUPRC notably exceeded the baseline AUROC in S12 Table. However, the test AUROC was substantially lower than the baselines, and there was also a nonnegligible drop in AUROC from the training set to the test set when using the synthetic images for classification. These discrepancies may be attributed to the unbalanced dataset and the potential inaccuracies of images synthesized by CPDM in some aspects caused by limited training images. Fig 6C and 6D depicted the average ROC and precision-recall curves from cross-validation for the model trained on the MRI features extracted by the PyRadiomics tool, providing additional context for the numerical results. Overall, these findings indicated that the model already had a certain classification ability, but it still had some limitations compared with using real multi-omics to make predictions.

3.2.3.2. XGBoost model for ER+/HER2+ subtype prediction. Table 5 showed the performance of different versions of the XGBoost model in predicting ER+/HER2+ subtypes. Fig 6E and 6F. illustrated the average ROC and precision-recall curves from cross-validation. These performance metrics indicated the model performed well in both discrimination and accuracy. Specifically, a high AUROC score suggested the model effectively differentiates ER+/HER2+ subtypes, while a high AUPRC score indicated the model could identify positive cases well. A good F1 score revealed that the model established a well-balanced trade-off between precision and recall. Moreover, by comparing the AUPRCs with the baseline AUPRCs in S12 Table, it could be observed that the obtained AUPRCs were markedly better than baseline AUPRCs. Likewise, the performance in Table 5 was competitive when compared to the baseline performance in S11 Table. Notably, the synthetic MRI-based classification results were superior to the baseline results based on patients’ actual multi-omic profiles presented in S11 Table. This improvement may be attributed to the richer information contained in images, enhancing the classification tasks. These observations collectively demonstrated the robustness of the model in predicting ER+/HER2+ subtypes. Additionally, we also plotted the SHapley Additive exPlanations (SHAP) diagrams in S3 Fig for the XGBoost models trained on MRI features extracted by the PyRadiomics tool [51]. From the diagram we could identify the features that were important for the subgroup prediction and enhance the interpretability of the model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. The results of the ER+/HER2+ subgroup prediction task.

https://doi.org/10.1371/journal.pcbi.1012490.t005

3.2.3.3. Survival analysis. Table 6 presented the performance (C-index score and log-rank test p-value) of the DeepSurv model and the CoxPHfilter models trained on the multi-omic profile-guided synthetic MRI features. Table 7 provided the performance of DeepSurv models and CoxPHfilter models trained for patients with ER+/HER2+ subtype data. The assessed C-index scores and log-rank test p-values from survival models on both training and test sets in the two tables, especially the models trained on MRI features extracted by ResNet50, demonstrated models’ potent capability in predicting patient prognosis. This could be further supported by their close alignment with the baseline performance shown in the S13 Table. Additionally, in Tables 6 and 7, compared to the CoxPHfilter model, DeepSurv showed slightly inferior performance. This may be attributed to the small dataset sizes available for deep learning in this task, hindering the ability of the model to effectively learn and generalize. What’s more, comparing the performance of the CoxPHfilter models on multi-omic and gene expression versions, it was observed that the performance in the multi-omic version was better than in the gene expression version. Nevertheless, the performance metrics of multi-omic data-based CPDM were lower than the CPDM based on gene expression. This indicated the image features influencing FID and SSIM scores may be independent from those used for predicting survival. This observation revealed the complex relationship between image performance metrics and the predictive capacity in survival analysis, suggesting differing roles of various features in survival prediction. Lastly, S4 Fig presented Kaplan-Meier plots to show the shapes of the survival curves, which the p-values of log-rank test may not fully illustrate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Performance of survival analysis for all patients with multi-omic profiles.

https://doi.org/10.1371/journal.pcbi.1012490.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Performance of survival analysis for patients with ER+/HER2+ subtype data.

https://doi.org/10.1371/journal.pcbi.1012490.t007