1. Introduction

The rapid advancements in deep learning have significantly impacted the medical field, especially in tasks such as cancer diagnosis, prognosis, and therapeutic response prediction. These tasks often necessitate a comprehensive characterization of a patient’s profile, which involves integrating heterogeneous data sources, including histopathology slides, molecular profiles, and clinical data [5,19–21,39,43,48,52].

Histopathology slides, or Whole-Slide Images (WSIs), are the gold standard in computational pathology for prediction tasks. These gigapixel images provide detailed visual information about tissue architecture and cellular morphology, making them invaluable for diagnosing various diseases, including cancer [17]. However, the high resolution of WSIs poses significant computational challenges, particularly in supervised learning tasks, which require efficient training and robust data representations. Current approaches, such as Multiple Instance Learning (MIL), extract small patches from WSIs and process them independently before aggregating their representations. Despite their utility, these methods often fail to capture the spatial context and heterogeneity of the tumor microenvironment [36,40].

Simultaneously, multi-omics data—including Copy Number Variations (CNV), RNA sequencing (RNAseq), and DNA methylation profiles—provide a molecular-level understanding of the biological processes underlying diseases. Integrating these diverse data types can reduce uncertainties arising from different experimental conditions and reveal complex biological interactions not apparent from single data sources [1]. Nevertheless, the high dimensionality and heterogeneity of multi-omics data present significant challenges. Each molecular source may encapsulate distinct biological functions, necessitating sophisticated methods to extract clinically relevant insights.

Integrating WSIs and multi-omics data is a promising yet challenging area in computational pathology. Combining the rich visual information from histopathological images with molecular profiles can enhance diagnostic accuracy, prognostic predictions, and therapeutic decisions. Despite notable efforts in this domain, such as those by Courtiol et al. [53] and Lu et al. [54], current methods often fail to leverage the full potential of multimodal data. For instance, Courtiol et al. combined deep features from histology images with transcriptomic data, while Lu et al. integrated genomic and histopathological data using a unified deep learning architecture. However, these approaches often rely on early integration techniques or standard feed-forward networks, which may not fully exploit the rich information provided by multi-omics data.

Moreover, a critical challenge in multimodal integration is dealing with incomplete datasets, a common issue in clinical settings where one or more modalities may be missing. Addressing this challenge is essential for the practical application of multimodal integration techniques.

This work introduces a novel method, Multimodal CustOmics, designed to integrate histology images with the patient’s molecular profile effectively. Our approach captures high correlations between and within different dimensions of the biological system, such as biological pathways or cellular regions. Utilizing a multiple-instance variational autoencoder with modality dropout, our method integrates multimodal data while remaining robust to missing data. We adapt the autoencoder to create a mixture-of-experts representation, similar to the approach by Jordan et al. [24], which enhances interpretability by aligning multiple relevant biological dimensions of the patient’s profile.

Our method provides three significant advantages:

• Enhanced Understanding: By dividing each modality into different functional groups, we achieve a more nuanced understanding of high-dimensional problems.
• Robustness to Missing Data: Implementing modality dropout within our framework creates an end-to-end pipeline resilient to missing modalities.
• Interpretable Interactions: The multidimensional representation facilitates easy interpretation of interactions between all dimensions from different modalities, thanks to the mixture-of-experts networks.

2. Materials and methods

To tackle the challenges of integrating multimodal data, we propose a deep-learning framework to create an interpretable image-omic representation, represented in Fig 1, that captures interactions at multiple levels of the biological system. This integration method is named Multimodal CustOmics and builds upon the strategy introduced in our previous work [3]. The original network optimally integrated heterogeneous data from different omics sources while preserving the specificity of each modality. The new version of our multimodal network can jointly integrate H&E slides and molecular profile features (mutation status, copy-number variation, RNA sequencing [RNA-seq] expression, DNA methylation, etc.). Additionally, it can interpret how the interaction between all those sources correlates with specific supervised tasks such as molecular subtype identification or survival prediction. Furthermore, this method facilitates the assessment of feature importance at multiple levels through ad-hoc scores computed within the framework: gene level, pathway level, and spatial level.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Modality encoding.

Each modality undergoes a specific methodology. For histopathology slides, spatial regions are extracted using a hypergraph encoder to obtain an embedding for each region. Genes are segregated into gene sets in multi-omic integration, resulting in a multi-omics embedding per set. Modality Dropout: Dropout Layer for modalities to deal with missing modalities by relaxing the constraint of needing all modalities at once. Latent Representation: The latent representation comprises multiple blocks, each representing the embedding of the interaction between a region and a pathway. Hierarchical Mixture of Experts: Prediction Model based on MoE architecture on the different block embeddings. Phase 1 learns the weights of each region with regards to a fixed pathway while phase 2 learns the weights of each pathway for the final prediction. Multi-Task Predictions: The latent representation is then utilized for supervised tasks such as classification or survival analysis or unsupervised tasks for tasks like clustering. Multi-Level Interpretations: Interpretation results are extracted at various levels: gene, gene-set, and spatial levels.

https://doi.org/10.1371/journal.pcbi.1013012.g001

At the gene level, the method outputs the importance score of each gene for each molecular source, both independently and in association with other modalities. At the pathway level, a Multimodal Pathway Enrichment Score (MPES) is computed to assess the importance of a specific pathway for a specific prediction task, such as molecular subtype classification or survival prediction. This score has also been extended to account for spatial correlations in the histopathology slide and reflects the importance of the interaction between spatial regions of the WSI and pathways.

Additionally, to further explain spatial interpretability results, we extract high-attention image patches and analyze them using the pre-trained Hover-Net model for cell instance segmentation and classification [13]. This analysis categorized cells into tumors, lymphocytes, stromal, necrosis, and epithelial cells. We quantitatively assessed the frequency of these cell types in high-attention patches for each patient. Additionally, we estimated the proportion of Tumor Infiltrating Lymphocytes (TILs) in these patches using the methodology developed by Saltz et al. [35], thus providing a deeper understanding of the tumor microenvironment.

2.1. Model construction

Within the scope of this study, we design, implement, and evaluate a multimodal integration network for integrating histopathology slides and multi-omics data. For , let us denote by W_i and O_i, respectively, the WSI and multi-omics bag for patient i. The goal of this study is to build and train a multi-task network that takes as input the two bags and creates an interpretable multimodal representation z_i for each patient such that .

2.1.1. Representation of histopathology images.

To effectively represent histopathology slides, we opted for a hypergraph approach to address the complex relationships within Whole Slide Images (WSIs) that cannot be effectively captured by traditional graph-based methods. Unlike standard graphs, which are limited to pairwise relationships, hypergraphs can model higher-order interactions among multiple entities, such as patches within a WSI. This is particularly relevant in histopathological analysis, where regions of interest often exhibit intricate morphological and spatial interdependencies. By leveraging a hypergraph structure, our method enables a more nuanced representation of these interactions, offering enhanced interpretability and adaptability in multimodal settings. Additionally, the hypergraph framework introduces computational efficiency compared to transformer-based architectures by maintaining a lower parameter count, aligning with the goal of dividing the multimodal representation learning task into smaller, interpretable strata as envisioned in Multimodal CustOmics.

To this end, an automated tissue segmentation is conducted using the method outlined in Lu et al. [30], which uses Otsu Binarization to separate the background from the tissue. Subsequently, non-overlapping patches of size 256x256 are extracted at a magnification of 20x. These patches are then given as input to a ResNet-18, trained with a contrastive strategy on histopathology data akin to Ciga et al. [10], generating a 1024-dimensional feature vector for each patch. The set of features associated with a set W_i comprising n_p patches is consolidated into a feature matrix . Each patch x_j is defined by its ResNet-18 feature representation h_j, encapsulating morphological properties, and a set of coordinates representing the spatial center of the patch.

Following the methodology of Benkirane et al. [4], a patch filtering process is executed to summarize information in the Whole Slide Image (WSI). This involves agglomerative clustering, where a similarity kernel is computed between all patch pairs. Two similarity matrices, and are computed, where and . Here, represents a morphological similarity metric, and signifies a spatial proximity metric. The overall similarity kernel is used to measure the similarity between patches by incorporating both histopathological () and genomic () feature similarities. To cluster highly similar patches, we apply agglomerative clustering, where patches x_i and x_j are assigned to the same cluster C_k if . This ensures that only patches with a similarity above the threshold z are merged, preserving structurally and functionally coherent regions. These clusters are then aggregated into a single representation , where and . This new patch representation approximates a localized region, effectively reducing the number of instances in WSIs while preserving a substantial amount of information.

To identify multiple regions of interest within the Whole Slide Image (WSI), we begin by dividing the extracted set of patches, denoted as , into K distinct clusters using the K-Means clustering algorithm. Each of these clusters represents a different category of regions within the WSI.

Once the clustering is completed, we define the neighborhood relationship between patches. Specifically, for each patch , we define its neighborhood as:

where:

• is a similarity function that quantifies the relationship between patches p_k and p_j.
• is a predefined threshold that determines whether two patches should be considered neighbors.

Using this neighborhood definition, we proceed to construct a hypergraph. The hypergraph structure is represented by an incidence matrix , where is the total number of patches, and |E_i| is the number of hyperedges. Each entry in the matrix is defined as follows:

Thus, the hyperedges in the hypergraph are formed based on patch similarity, and the incidence matrix encodes these relationships, indicating which patches belong to which hyperedges.

Next, the hypergraph is processed using a Graph Neural Network (GNN) that applies a series of hypergraph convolutional layers and attention mechanisms [2]. This GNN extracts meaningful representations for each patch, resulting in a feature matrix:

where each row corresponds to a learned feature vector of dimension z_dim for a patch in .

To obtain a region-level representation, we aggregate the patch-level features into region embeddings. Specifically, patches belonging to the same region (as defined by the initial K-Means clustering) are pooled together to form a compact feature representation for that region:

where each row of corresponds to the embedding of one of the K regions in the WSI. This final region-level representation serves as the condensed representation of the entire WSI, capturing spatial and contextual relationships across different regions.

2.1.2. Integration of multi-omics data.

We consider S omic sources in the set . The initial step in processing these omic sources involves partitioning each source into P gene sets, each possessing distinct functional properties, defined as . Subsequently, we engage in Variational Autoencoder-based representation learning for each of these P distinct gene sets, a concept introduced in [3]. The encoding networks are designed to accept inputs from all omic sources about each gene set, encapsulated by the operation . These representations are then concatenated to form a final matrix , serving as the comprehensive collection of multi-omics data.

2.1.3. Multimodal representation and prediction.

Upon acquiring the WSI bag and the multi-omics bag for a given patient i, the next step involves constructing a joint multimodal representation that effectively integrates spatial and molecular features. This representation is obtained through a bilinear operation applied to each pair of elements from both modalities, generating a three-dimensional tensor:

(1)

where each entry is computed as:

(2)

with denoting the bilinear fusion operator. This operator is adapted from the multimodal factorized bilinear pooling (MFB) method introduced by Yu et al. [46], which enables efficient feature fusion by capturing second-order interactions between modalities while reducing computational complexity compared to traditional cross-attention mechanisms. By leveraging this approach, the model preserves the informative relationships between the histopathological and molecular features in a structured manner. The resulting multimodal representation is then processed by a hierarchical mixture of experts (MoE) network [24], designed to model pathway-specific and global patient-level interactions. The first stage of this MoE network focuses on pathway-specific feature learning by computing a pathway representation for each pathway p through a weighted aggregation of the associated spatial regions:

(3)

where represents the learned importance weight assigned to spatial region k within pathway p, subject to the constraint to ensure a valid probability distribution. These trainable parameters allow the model to focus on the most informative regions relevant to each pathway, effectively prioritizing key spatial features for downstream analysis. Each pathway representation is subsequently passed through a linear transformation to generate pathway-specific predictions. In the second stage, the learned pathway representations are aggregated to form a unified patient-level representation , computed as:

(4)

where w_i,p denotes the learned importance weight assigned to pathway p, with ensuring a valid probability distribution. The parameters are optimized during training to emphasize the most relevant pathways for final decision-making. The resulting patient-level representation is then passed through a final linear layer, mapping it to the model’s final prediction. By structuring the multimodal fusion through hierarchical MoE networks, the proposed framework effectively models both local (pathway-specific) and global (patient-level) interactions, enabling robust integration of spatial and molecular features for improved predictive performance.

2.1.4. Downstream tasks.

Multimodal CustOmics accommodates training for two distinct tasks. Firstly, a supervised classification task aims to predict the probability of each class occurrence. This task undergoes training with a standard Categorical Cross-Entropy loss computed between the predicted classes and the ground truth labels.

The second task involves predicting survival outcomes, trained using the DeepSurv loss function outlined in Katzman et al. [25]. The model adopts the negative partial log-likelihood formula, expressed in our context as:

(5)

where E_i represents the event for patient i, denotes the risk function associated with the risk score estimated by the network’s output layer, and defines the risk set, signifying the patients still susceptible to failure after time t.

This method can also perform self-supervised clustering based on the DeepCluster framework [9], although this aspect will not be detailed in this paper.

2.1.5. Multimodal dropout.

To better enforce the robustness of our method to missing data, we implement multimodal dropout introduced in Cheerla et al. [7] to deal with missing modalities under the assumption of modalities missing at random. Instead of dropping single neurons, the idea is to drop entire feature vectors corresponding to specific modalities so that it scales up the weights of the others. This is applied to each sample data with a probability p for each modality. Dropout is implemented differently across modalities: for omics, entire modalities are randomly masked during training within the variational autoencoders’ input layer, while for WSI, dropout occurs before the multimodal fusion network. The dropout rate is a hyperparameter to tune.

3. Experimental setup

To evaluate the effectiveness of Multimodal CustOmics, we compare it against four established methods that integrate omics and histopathology data. For this comparison, we follow the study conducted by Chen et al. [8] and use the engineered baselines introduced for omics, histopathology, and multi-omics integration. In particular, we compare with:

• SNN architecture [27]: We compare against a feed-forward self-normalizing network (SNN) architecture [27] as a multi-omics baseline, where multi-omics data are concatenated before being fed into the network. This architecture, used in previous works [8,55], is considered state-of-the-art for histology-genomic integration.
• DeepSets [47]: We employ DeepSets, one of the first neural architectures designed for set-based deep-learning problems. DeepSets utilizes sum pooling over instance-level features. Its multimodal extension, presented by Chen et al. [8], processes omics data with an SNN and integrates them using bilinear pooling.
• Attention MIL [22]: We consider Attention MIL, a set-based network similar to DeepSets that replaces sum pooling with an attention pooling technique .
• DeepAttnMISL [45]: We include DeepAttnMISL, a set-based network that first applies K-Means clustering to instance-level features. Each cluster is processed using Siamese networks, and the cluster features are then aggregated using global attention pooling .
• MCAT [8]: We compare our method to MCAT, presented by Chen et al. , a transformer-based set-based network that represents the current state-of-the-art in multimodal histology-genomics integration. MCAT combines modalities using bilinear attention-based pooling.
• SurvPATH [56]: We include SurvPath, introduced by Jaume et al., a new transformer-based architecture that embeds transcriptomics information in the form of biological pathways. The omics layer in SurvPATH consists of an SNN architecture adapted to consider multi-omics data.
• HEALNet [26]: We also compare against HEALNet, introduced by Hemker et al., a multimodal fusion architecture designed for heterogeneous biomedical data integration. HEALNet preserves modality-specific information, captures cross-modal interactions, and demonstrates robustness to missing modalities. It achieves state-of-the-art performance in multimodal survival analysis across cancer datasets.

We follow each method’s experimental protocols outlined in their respective publications to ensure a fair comparison.

4. Results

4.1. Prediction results

We executed multiple test cases in a comprehensive evaluation of Multimodal CustOmics within a multitask framework encompassing both classification and survival analyses. The performances for the classification and survival tasks are presented in Tables 1 and 2. We assess model performance using two sets of metrics: classification metrics (balanced accuracy, F1-score, precision, recall, and Area Under ROC Curve (AUC)) and survival analysis metrics (concordance-index (C-index) [6]). We initially assessed cancer-type classification within the TCGA Pancancer cohort, revealing Multimodal CustOmics’ superior performance in terms of AUC compared to other benchmarked methods [16]. This notable performance can be attributed to the substantial patient cohort size and the rich information embedded within molecular data, a phenomenon well-documented across multiple studies [3,49,50].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Classification performances. Comparison of the classification performances for the 4 tasks concerning the area under ROC (AUC %) the standard deviation across the folds.

https://doi.org/10.1371/journal.pcbi.1013012.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Survival prediction performances. Comparison of the discrimination of the models for the 8 TCGA cohorts, determined by the Concordance Index (C-index %) the standard deviation across the folds.

https://doi.org/10.1371/journal.pcbi.1013012.t002

To underscore the method’s robustness concerning sample size, we evaluated smaller datasets, focusing on predicting molecular subtypes within three specific TCGA cohorts: TCGA-BRCA, TCGA-COAD, and TCGA-STAD. Across all classification tasks (as detailed in Table 1), Multimodal CustOmics consistently outperformed other comparable methods. Notably, in multi-omics integration, the mixed-integration VAE within Multimodal CustOmics demonstrated superior performance compared to the SNN utilized by other methods. An additional assessment showcased the significance of multi-omics integration, delineated in S4 Table, emphasizing the impact on performance when replacing the VAE encoder in Multimodal CustOmics with a standard SNN.

Further exploration into the integration strategies revealed the advantage of Multimodal CustOmics in exploiting diverse molecular data sources. Comparative analyses in S3 Table demonstrated that while SNN performed best with RNAseq alone, Multimodal CustOmics enhanced performance when integrating RNAseq with CNV and methylation data. This divergence in integration strategies suggests Multimodal CustOmics’ capacity to augment predictive power and unveil novel interactions among disparate data sources.

Regarding the survival outcome prediction across eight TCGA cohorts, Multimodal CustOmics’ displayed consistently encouraging predictive capabilities compared to the alternative methods. Notably, in survival analysis based solely on WSI, Multimodal CustOmics exhibited comparatively weaker performance in specific cohorts than the transformer architecture employed in MCAT. An ablation study (detailed in S4 Table) replaced Multimodal CustOmics’ hypergraph encoding for WSI embeddings with a visual transformer. While the transformer architecture demonstrated superior performance in analyzing individual WSI patches, the hypergraph-based approach achieved significantly better results when modeling region-wide tissue relationships that were then integrated with other modalities. This performance disparity highlights that although hypergraphs may not optimize patch-level feature extraction, their ability to encode relationships across broader tissue regions makes them particularly effective for capturing histological patterns at multiple scales. The limitations of the transformer’s pairwise attention mechanism become apparent when modeling extended spatial relationships, where hypergraph-encoded region-wide representations provide a more comprehensive view of tissue architecture. Furthermore, as shown in S7 Table, hypergraph architectures are inherently more parameter-efficient, requiring significantly fewer weights compared to transformer models while maintaining effective region-wide feature learning. These findings reinforce our architectural choice for histopathology encoding, suggesting that optimizing for region-wide tissue relationships with a lighter computational footprint, rather than local patch-level features alone, leads to more effective representation learning that better complements multimodal integration.

In survival tasks, particularly in multi-omics scenarios (without WSI), Multimodal CustOmics obtained better prediction performances, especially as other methods like SNN showed concordance indices approaching randomness (Table 2). Of note, Multimodal CustOmics had a lower standard deviation across folds than other state-of-the-art approaches.

While the primary analysis was conducted using Cancer Hallmarks, we also performed the same analysis using multiple gene sets (Oncologic Signatures [58] and ReacTOME [57]). The results displayed in S5 and S6 Tables show similar performance to those obtained using Hallmarks. The change, however, may occur in the interpretability results that may vary depending on which set of pathways is chosen, changing the biological focus given to the model.

4.2. Multi-level interpretability: classification

Multimodal CustOmics can conduct pathway enrichment analysis across multiple tasks. Fig 2 presents interpretability findings concerning the PAM50 subtype classification within the TCGA-BRCA dataset. The objective is to elucidate the determinants driving the discrimination of specific subtypes, notably the Her2 subtype, within a multimodal context.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. PAM50 interpretability analysis.

a. Pathway enrichment scores and associated p-values from the gene set variation analysis. The coloured contour on the slide on the left is here to show good tissue segmentation. b. SHAP values for the most influential genes affecting the stratification of the Her2 subtype and their impact on other subtypes. c. Spatial Enrichment Analysis for the top 2 pathways and their key genes. d. Gene importance within the considered pathways. e. Differential Analysis of cell distribution between high and low attention regions.

https://doi.org/10.1371/journal.pcbi.1013012.g002

The initial layer of interpretability operates at the gene level, utilizing a Multi-Omics Pathway Enrichment Score (MPES) and conducting Gene Set Variation analysis using normalized gene importance scores (details in the methods section). Fig 2B delineates essential pathways in Her2 subtype discrimination, notably highlighting the significance of estrogen response and KRAS signaling down hallmarks. The interrelation between the Her2 subtype and estrogen response has been extensively investigated [33], emphasizing their coexpression’s multifaceted impact on breast carcinogenesis, invasive behavior, and cellular growth.

Further exploration into gene-level importance is depicted in Fig 2C, spotlighting the predominant genes responsible for discriminating the Her2 subtype and contrasting their importance across other subtypes. Notably, the FOXA1 gene emerges with substantial importance, aligning with its suggested role as a transcription factor for Her2, as indicated in Cruz et al. [11].

Beyond multi-omics pathway enrichment analysis, Multimodal CustOmics extends interpretability to encompass multimodal enrichment, revealing spatial interactions within histopathology slides that correlate with specific pathways for discriminating Her2 subtypes. Fig 2D illustrates such interpretability outcomes for the estrogen response and KRAS pathways. Different cell populations within high-importance regions for each pathway are described to help biological interpretation. Notably, regions associated with the KRAS pathway exhibit increased proportions of stromal cells, suggesting potential regulation of tumor cell signaling via these stromal cells [42]. Conversely, the estrogen response demonstrates a strong interaction with regions featuring elevated densities of Tumor-Infiltrating Lymphocytes (TILs) and lymphocytes, corroborated by multiple sources [12,29]. This interaction holds particular significance when stratifying between ER- and ER+ patients.

4.3. Multi-level interpretability: survival

In a similar vein, survival analysis also lead to interpretable results. Fig 3 delineates the varying degrees of enrichment analysis for predicting survival outcomes within the TCGA Pancancer dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Interpretability analysis for pan-cancer survival outcome prediction task.

a. Pathway enrichment analysis. The coloured contour on the slide on the left is here to show good tissue segmentation. b. Kaplan Meier curve associated with the survival outcome prediction task, showing the high and low risk for death event stratification with a computed log-rank p-value. c. Spatial Enrichment Analysis for the top 2 pathways and their most important genes. d. Gene importance within the considered pathways. e. Differential Analysis of cell distribution between high and low attention regions.

https://doi.org/10.1371/journal.pcbi.1013012.g003

Specifically, Fig 3B underscores the predominant influence of the inflammatory response pathway on survival outputs, a finding consistent with existing literature [51].

Demonstrating the relevance of employing multimodal integration in pan-cancer survival analysis, Fig 3C illustrates the impact of incorporating multiple data sources on stratifying low and high-risk patients, as evidenced by Kaplan-Meier curves and their corresponding log-rank p-values.

Further investigation into the effect of essential pathways is portrayed in Fig 3D, showcasing the significance of interactions between the two most crucial pathways. Notably, heightened importance is observed within the inflammatory response pathway, characterized by increased lymphocyte densities and Tumor-Infiltrating Lymphocytes (TILs). In contrast, the epithelial-mesenchymal transition pathway manifests greater densities of stromal cells.

Delving deeper into the Epithelial-Mesenchymal Transition pathway, the primary influential gene appears to be FBN2, renowned for its inhibition of cancer cell invasion and migration, as stated in Mahdizadehi et al. [32], thereby explaining its inclination toward lower risk outcomes.

4.4. Multi-level interpretability validation

To ensure the reliability of interpretability findings by Multimodal CustOmics, we validated using the IALT data set, focusing on lung cancer. We undertook a comparative analysis of survival analysis interpretability results between the IALT dataset and a combined dataset of TCGA-LUAD/TCGA-LUSC for Lung Adenocarcinoma and Squamous Cell Carcinoma.

The pathway enrichment analysis, illustrated in Fig 4, revealed concordant outcomes between the TCGA and IALT datasets, identifying KRAS signaling down as the foremost pathway. This finding aligns with established literature [41], which underscores the prevalence of oncogenic KRAS mutations in approximately 25% of lung adenocarcinoma cases, thus representing a pivotal focus in current drug development.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of interpretations between the TCGA-LUAD/TCGA-LUSC datasets and IALT regarding the distinction between high and low survival risk.

a. Pathway Enrichment analysis highlights the task’s top pathways. b. Survival Outcome Prediction Performances using C-index on Omics, WSI and Multimodal setup. c. Differential Analysis of cell distribution between high and low attention regions.

https://doi.org/10.1371/journal.pcbi.1013012.g004

Subsequently, we explored the interactions between the KRAS signaling-down pathway and spatial regions within histopathology images. Fig 4B depicts that regions of high attention in both datasets exhibit similar distributions of cell types, indicating the robustness of our method across distinct datasets of the same cancer type. Notably, this distribution underscores the association between the KRAS pathway and tumor-infiltrating lymphocytes (TILs) heightened densities and marginally increased tumoral cell counts in predicting survival outcomes. This consistent association echoes previous findings in [28], highlighting a solid correlation between KRAS mutation status and tumor immunity-related characteristics, notably CD8+ TILs.

5. Discussion

The Multimodal CustOmics framework is a comprehensive toolbox to bridge prediction and interpretation within biological systems across multiple levels: genes, pathways, and spatial configurations. This multifaceted system generates three distinct interpretability scores concurrent with predictions, unraveling the biological knowledge underlying model outcomes. Empirical assessments underscore Multimodal CustOmics’ robust predictive capabilities, outperforming state-of-the-art methodologies in integrating multi-omics and histopathology data across eight diverse datasets. However, despite its efficiency with smaller datasets in this study, Multimodal CustOmics’ reliance on deep learning methodologies might restrict efficiency when confronted with limited training data availability.

Furthermore, Multimodal CustOmics stands out for its interpretability, facilitating a broader spectrum of analyses and enriching understanding across diverse biological modalities. Notably, although this study centered on three omics data types, Multimodal CustOmics exhibits versatility in seamlessly integrating varied omics data without necessitating framework alterations. This adaptability originates from an initial phase that independently trains on each source, serving as a normalization layer for heterogeneous sources.

The strategic segmentation of inputs into interpretable entities for spatial and molecular data mitigates challenges arising from the high dimensionality of whole slide images and multi-omics datasets. This partitioning augments interpretability and broadens the method’s applicability to uncharted pathways or spatial regions beyond this study’s scope. Multimodal CustOmics places significant emphasis on expansive interpretability functionalities. This aims to unveil predominant biological functions steering specific predictions across diverse data sources and scales. This comprehensive approach fosters collaboration between biologists and computational pathologists, offering a framework for in-depth analyses through enrichment analysis for omics and spatial data. The method extracts coherent insights from diverse data sources, unveiling a panoramic view of interconnected biological processes influencing the outcome of interest. By integrating omics and spatial data within enrichment analysis, Multimodal CustOmics enables a deeper understanding of the interplay between molecular information and spatial contexts, enriching investigative pathways for researchers in the field.

In addition, our Mixture-of-Experts (MoE) approach offers distinct advantages over traditional attention-based interpretability methods. By explicitly modeling the hierarchical nature of biological systems through specialized expert networks for different pathways and spatial regions, our approach achieves more stable and reliable interpretations of model decisions. As demonstrated in S1B Fig, our systematic stability analysis across the three TCGA subtype classification tasks reveals that the MoE-based architecture produces notably more consistent pathway importance scores compared to attention mechanisms. The rank correlation between weights from different training folds was substantially higher for MoE () than attention mechanisms (), with this difference becoming more pronounced in smaller datasets like STAD. This enhanced stability is particularly crucial in cancer genomics, where sample sizes are often limited and reliable feature importance is essential for downstream analysis and clinical interpretation. The superior consistency of our MoE-based interpretations suggests that the identified important pathways are more robust to variations in training data, providing a more dependable foundation for biological insights.

Regarding the missing modality strategy, our analysis revealed important insights into the model’s behavior under different missingness patterns. While our approach primarily operates under a Missing at Random (MAR) assumption, we investigated its robustness to Missing Not at Random (MNAR) scenarios through systematic experiments. By selectively removing RNAseq data from Lung Cancer samples during training while preserving CNV and methylation data, we discovered that the model can inadvertently learn to use missingness patterns as predictive shortcuts. This was evidenced by elevated Lung False Positive Classification Rates (LFPR) at low dropout rates during testing, as shown in S1A Fig. The figure illustrates the evolution of LFPR across different dropout rates, with optimal performance observed at moderate dropout rates (around 40%), suggesting that introducing controlled missingness during training forces the model to rely more heavily on genuine biological signals from available modalities rather than missingness patterns. However, the persistence of elevated LFPR compared to MAR baseline performance indicates that completely mitigating learned biases from systematic missingness remains challenging. These findings underscore the importance of careful consideration of missing data patterns in multimodal integration approaches and suggest that strategic data dropout during training may help improve model robustness.

Despite its potential and performance, our method has a few noteworthy limitations. Firstly, while this study successfully delineates interactions between omics and histopathology data, it lacks a method to effectively discern how important individual contributions of each omic source are to a patient’s molecular profile. It prevents a comprehensive understanding of each omic source’s distinct impact on the molecular landscape. Secondly, the link established between the generated representation and phenotype data, beyond mere predictive labels, solely relies on the conditioning of the latent space. While this conditioning methodology effectively incorporates phenotypical signals into the multimodal representation, it lacks a mechanism to unveil interactions between different modalities and diverse clinical variables explicitly. This omission presents an avenue for future development, potentially enhancing the interpretability of multimodal interactions and their associations with clinical factors.

6. Conclusion

In this study, we introduced Multimodal CustOmics, a novel deep-learning framework that effectively integrates histopathology images and multi-omics data to enhance cancer characterization and patient outcome predictions. Multimodal CustOmics demonstrates superior performance in both classification and survival tasks, outperforming existing state-of-the-art methods. Our model provides robust and interpretable results, enabling the extraction of meaningful insights at gene, pathway, and spatial levels. This multi-level interpretability is critical for understanding the complex biological interactions within the tumor microenvironment and advancing precision medicine. The robustness of Multimodal CustOmics to missing data, through a multiple-instance variational autoencoder with modality dropout, ensures its practical applicability in clinical settings. Our extensive evaluations using TCGA datasets and validation cohorts underscore the potential of Multimodal CustOmics to improve diagnostic accuracy, clinical outcome predictions and thus therapeutic decision-making, paving the way for more personalized and effective cancer treatments. This Multimodal CustOmics framework is publicly available in Github.

Supporting information