Fig 1.
Each modality undergoes a specific methodology. For histopathology slides, spatial regions are extracted using a hypergraph encoder to obtain an embedding for each region. Genes are segregated into gene sets in multi-omic integration, resulting in a multi-omics embedding per set. Modality Dropout: Dropout Layer for modalities to deal with missing modalities by relaxing the constraint of needing all modalities at once. Latent Representation: The latent representation comprises multiple blocks, each representing the embedding of the interaction between a region and a pathway. Hierarchical Mixture of Experts: Prediction Model based on MoE architecture on the different block embeddings. Phase 1 learns the weights of each region with regards to a fixed pathway while phase 2 learns the weights of each pathway for the final prediction. Multi-Task Predictions: The latent representation is then utilized for supervised tasks such as classification or survival analysis or unsupervised tasks for tasks like clustering. Multi-Level Interpretations: Interpretation results are extracted at various levels: gene, gene-set, and spatial levels.
Table 1.
Classification performances. Comparison of the classification performances for the 4 tasks concerning the area under ROC (AUC %) the standard deviation across the folds.
Table 2.
Survival prediction performances. Comparison of the discrimination of the models for the 8 TCGA cohorts, determined by the Concordance Index (C-index %) the standard deviation across the folds.
Fig 2.
PAM50 interpretability analysis.
a. Pathway enrichment scores and associated p-values from the gene set variation analysis. The coloured contour on the slide on the left is here to show good tissue segmentation. b. SHAP values for the most influential genes affecting the stratification of the Her2 subtype and their impact on other subtypes. c. Spatial Enrichment Analysis for the top 2 pathways and their key genes. d. Gene importance within the considered pathways. e. Differential Analysis of cell distribution between high and low attention regions.
Fig 3.
Interpretability analysis for pan-cancer survival outcome prediction task.
a. Pathway enrichment analysis. The coloured contour on the slide on the left is here to show good tissue segmentation. b. Kaplan Meier curve associated with the survival outcome prediction task, showing the high and low risk for death event stratification with a computed log-rank p-value. c. Spatial Enrichment Analysis for the top 2 pathways and their most important genes. d. Gene importance within the considered pathways. e. Differential Analysis of cell distribution between high and low attention regions.
Fig 4.
Comparison of interpretations between the TCGA-LUAD/TCGA-LUSC datasets and IALT regarding the distinction between high and low survival risk.
a. Pathway Enrichment analysis highlights the task’s top pathways. b. Survival Outcome Prediction Performances using C-index on Omics, WSI and Multimodal setup. c. Differential Analysis of cell distribution between high and low attention regions.