Multimodal CustOmics: A unified and interpretable multi-task deep learning framework for multimodal integrative data analysis in oncology
Fig 1
Each modality undergoes a specific methodology. For histopathology slides, spatial regions are extracted using a hypergraph encoder to obtain an embedding for each region. Genes are segregated into gene sets in multi-omic integration, resulting in a multi-omics embedding per set. Modality Dropout: Dropout Layer for modalities to deal with missing modalities by relaxing the constraint of needing all modalities at once. Latent Representation: The latent representation comprises multiple blocks, each representing the embedding of the interaction between a region and a pathway. Hierarchical Mixture of Experts: Prediction Model based on MoE architecture on the different block embeddings. Phase 1 learns the weights of each region with regards to a fixed pathway while phase 2 learns the weights of each pathway for the final prediction. Multi-Task Predictions: The latent representation is then utilized for supervised tasks such as classification or survival analysis or unsupervised tasks for tasks like clustering. Multi-Level Interpretations: Interpretation results are extracted at various levels: gene, gene-set, and spatial levels.