Study participants

All patient-related information in this study was reviewed and approved by the Institutional Ethics Committee at Sun Yat-sen University Cancer Center (Approval No. SL-B2024-229-01). As no patients were directly recruited for this study, informed consent was waived.

This retrospective study was conducted to develop SurvFinder, using data from four cohorts: three independent cohorts from separate medical centers and one cohort from the publicly available TCGA dataset. All cohorts contributed to both the development of the model and prognostic analyses. Each cohort consisted of patients with a confirmed diagnosis of stage II CRC through molecular and immunohistochemical testing of FFPE samples. For patients who received ACT in both internal and external cohorts, treatment regimens were administered according to standard protocols recommended by the NCCN guidelines [3,4]. The majority of patients were treated with fluoropyrimidine-based therapies, including: (1) monotherapy, primarily with capecitabine; (2) doublet regimens, mainly FOLFOX or XELOX; and (3) triplet regimens, typically FOLFOX or XELOX combined with monoclonal antibodies.

The Internal-CRCII cohort, used for model training and internal evaluation, included 842 cases diagnosed at Sun Yat-sen University Cancer Center between November 1, 2014, and December 31, 2018. Two external cohorts were utilized for validation: External-CRCII-1 cohort (438 cases diagnosed between August 1, 2012, and October 31, 2018, with accessible follow-up information from Tianjin Medical University Cancer Institute and Hospital) and External-CRCII-2 (405 patients diagnosed between January 1, 2015, and December 31, 2018, with available follow-up information from The Sixth Affiliated Hospital of Sun Yat-sen University). For each case in these three cohorts, clinicopathological parameters, including 14 conventional risk-related factors and the application of ACT, were collected. All tumor-related indicators (e.g., tumor budding, perineural invasion, vascular invasion, tumor differentiation, etc.) were assessed by the same experienced pathologist across all cohorts to ensure consistency.

Patients from all datasets underwent additional filtering before training based on the following exclusion criteria: (1) Patients who had received preoperative treatments, including neoadjuvant chemotherapy or radiotherapy; (2) Patients with a prior history of treatment for other cancers before the diagnosis of stage II CRC; (3) Patients with no observed relapse and a follow-up period of less than 24 months; (4) Patients lacking one or more of the 14 key clinicopathological variables used for comparison with MVNet (S7 Table); (5) Patients who died from causes unrelated to cancer; and (6) Patients with fewer than three available slides after quality control checks (e.g., slides with issues such as tissue folds, fractures, absence of tumor tissue, or those out of focus). Random sampling experiments show that the prognostic performance of the model improves as the number of included tumor slides per patient increases (S8 Table). Following the application of these exclusion criteria, the final numbers of cases in the Internal-CRCII, External-CRCII-1, and External-CRCII-2 datasets comprised 743, 352, and 331 cases, respectively.

Additionally, the study incorporated the TCGA-CRCII dataset, which comprised 229 cases from the TCGA-COAD and TCGA-READ projects. Due to lower data quality in the TCGA dataset, exclusion criteria were modified: patients without follow-up data, diagnostic slides, or slides meeting quality standards were excluded. To evaluate the generalizability of our model in cases with limited tissue sampling, no slide-number restriction to the TCGA-CRCII cohort was applied during validation. This resulted in 178 cases being used for validation.

Digitalization, immunohistochemistry, and WSI preprocessing

For each case, all available H&E-stained tumor slides were included for further analyses. All WSIs included in this study were scanned at a magnification of 40×, corresponding to a resolution of 0.25 μm/pixel, ensuring uniform pixel size across all datasets. Slides from Internal-CRCII and External-CRCII-1 cohorts were scanned with a PHILIPS Ultra Fast Scanner (Philips Electronics N.V., Amsterdam, the Netherlands) and saved in iSyntax format, while those from External-CRCII-2 dataset were scanned using an SQS-600P scanner (Shengqiang Technology, Shenzhen, China) and stored in SVS format. The tumor slides for the TCGA-CRCII dataset were downloaded from the Genomic Data Commons portal (https://portal.gdc.cancer.gov/).

WSINet was trained without any pixel-level or region-level annotations. Only slide-level labels (relapse versus non-relapse) were used for supervision. A total of 120 slides from both Internal-CRCII and External-CRCIIs were randomly selected to annotate normal, tumor, and tertiary lymphoid structure (TLS) regions for the development of SegNet. Immunohistochemistry (IHC) staining, using CD20, CD21, and CD10 markers, was employed to identify and define TLS phenotypes [18]. An experienced pathologist delineated TLS regions based on IHC results (S11 Fig). Specifically, TLS annotations on H&E slides were manually derived by referencing consecutive IHC-stained slides (CD20, CD21, CD10), based on morphological correspondence between serial sections.

To improve annotation consistency and accuracy, a meta-annotation method was applied to the procedure of labeling tumor and normal tissues [19]. For each WSI, at least four rectangular boxes containing representative normal and tumor tissue areas were extracted. These areas included various tissue types, such as vascular, smooth muscle, necrosis, normal, and tumor glands. Two pathologists with extensive experience in CRC diagnosis participated in tissue annotation and revision process. They used the open-source software QuPath (v0.2.3) and performed the annotations blinded to patient information and the study’s purpose to minimized bias. After annotation, the WSIs were further segmented into non-overlapping 256 × 256 tissue tiles, with the white background removed using Otsu’s method. The comparative visualization results demonstrate that the Otsu’s method effectively removes the white background (S12 Fig). To account for variations in tissue color from different scanning devices, all tiles from the External-CRCII-2 cohort were color-normalized to those from the Internal-CRCII cohort using the Vahadane method [20]. The predictive performance exhibited similar trends across both stained and unstained datasets, although the stained group consistently demonstrated superior accuracy (S13 Fig).

Deep learning model architecture

The SurvFinder framework consists of four principal neural networks: WSINet, SegNet, MVNet, and MMF (Fig 1), each designed to perform a specific task within the overall architecture.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The workflow of SurvFinder.

The SurvFinder framework consists of three main components: autonomous identification of risk-related tissue features, multi-class segmentation of prognostic biomarkers, and multi-view and multimodal risk prediction. WSINet is utilized to identify prognosis-associated tissue features, such as tertiary lymphoid structures (TLSs). SegNet is designed for automatic segmentation of TLSs and tumor tissue, as well as differentiation of TLS phenotypes. MVNet integrates multi-view characteristics of TLSs for prognosis evaluation and therapeutic decision-making, with multi-modal fusion also implemented. CRC, colorectal cancer; Attn, attention; TLS, tertiary lymphoid structure; SegNet, segmentation network; TNTM, TLS-normal-tumor segmentation model; TLSM, TLS subtyping model; Agg, aggregates; FL-1, primary follicles; FL-2, secondary follicles; MVNet, multi-view network; SpB, spatial feature branch; MpB, morphological feature branch; MMF, multi-modal fusion; ACT, adjuvant chemotherapy. Created in BioRender. Oa, A. (2025) https://BioRender.com/a45w458.

https://doi.org/10.1371/journal.pmed.1004614.g001

WSINet identifies prognostic tissue biomarkers from WSIs. SegNet then segments the regions of the biomarkers identified by WSINet, generating tissue heatmaps. MVNet further processes these segmented histological biomarkers by extracting and integrating multi-view features for risk stratification. MMF combines the multi-view features from MVNet with clinical data to produce a comprehensive fusion-based prediction.

WSINet, inspired by Lu and colleagues [21], predicts patient prognosis by identifying tissue patterns within WSIs. We utilized pretrained CTransPath model, developed by Wang and colleagues [22], which leverages the Swin Transformer architecture and was trained on the TCGA and PAIP datasets. Features were extracted from all image patches in each WSI. To comprehensively capture the tumor microenvironment, each case included at least three tumor slides, aggregating patch features across all WSIs per case. These case features are then processed through a simplified attention network, which assigns attention scores to each patch. Let denote the feature set for all N patches of a given case. The attention score for the -th patch is estimated by

where , , and are trainable weight matrices, and is the feature vector of the -th patch, obtained by passing through two fully connected layers. and represent the hyperbolic tangent and sigmoid activation functions, respectively. denotes element-wise multiplication. quantifies the relative contribution of the -th patch to the overall prognostic outcome for the case. Using these scores, we perform a weighted sum of all patch features to aggregate the case features. Finally, a linear layer is utilized to predict the patient’s prognostic outcome as

where denotes trainable weight matrices.

Tissue regions with high attention scores from WSINet were segmented using SegNet, which consists of two sequential patch-level classification models: TNTM and TLSM. TNTM initially classifies tissue tiles into TLS, normal, and tumor categories, while TLSM further categorizes the TLS regions identified by TNTM into specific TLS phenotypes. Both models utilize ResNet18 as the backbone architecture for patch classification. The results from TNTM are concatenated to form a tissue prediction heatmap for each WSI. After training TLSM, its model parameters were fixed, and TLS morphological features were extracted for further analysis.

MVNet integrates the multi-view prognostic features extracted by SegNet. It consists of two branches: the Spatial Branch (SpB) and the Morphological Branch (MpB). SpB utilizes the TLS tissue distribution heatmap for each WSI, which is generated by SegNet, as input to the model. Specifically, this heatmap provides a spatial representation of TLS tissue blocks within the WSI. Features capturing the spatial characteristics of these TLS tissue blocks—including their positions on the heatmap, spatial relationships with adjacent tumor tissues, as well as their area and subtype information—are extracted. These comprehensive spatial features are then processed using a multi-layer perceptron, resulting in the spatial feature vector . The MpB employs TLSM to convert the TLS images obtained from SegNet into features . These features are clustered for each individual case into 7 distinct categories. The cluster centers derived from this process serve as the representative lymphoid features for each patient [23]. Through clustering, the model’s training batch size is no longer restricted to 1. Increasing the batch size allows for more stable training of the model. These features are then processed using a simplified attention network, similar to the one employed in WSINet, to obtain the final features for each case. The visualization of the MpB architecture is illustrated in S3b Fig. The features from both branches are concatenated to form the multi-view feature representation for each patient. A feature fusion prediction module further integrates these features, i.e.,

where represents the deeply fused feature vector that integrates both spatial and morphological information. and are obtained by applying linear transformations to and using different weight matrices. and are the multi-view features, obtained by applying two separate linear transformations to , respectively. is the weight matrix used for the final fusion of multi-modal features.

The two fused feature vectors are concatenated into a single vector, which is then used for further prediction. Finally, a linear layer is employed to perform prognostic classification. The architecture of MVNet is depicted in S3a Fig.

The final network, Multi-Modal Fusion (MMF), integrates clinical data through a four-layer multi-layer perceptron. Fourteen clinicopathological variables were concatenated into a 14-dimensional vector and passed through four fully connected (FC) layers, each followed by a BatchNorm1d layer and a ReLU activation function, resulting in a 32-dimensional embedding (S3a Fig). Notably, the network was not trained with a separate target; its parameters were optimized end-to-end along with the multimodal model using the final prediction loss. This clinical embedding was concatenated with the 32-dimensional output from the penultimate FC layer of MVNet to form a 64-dimensional fused feature vector, which was processed by a final FC layer to produce the MMF prediction score. Clinical features are concatenated with , and a linear layer is employed for the final prognostic prediction. By integrating multi-view TLS features and clinical data, SurvFinder provides precise prognostic predictions for patients with stage II CRC.

Training and evaluation

During the model development phase, the prediction tasks in SurvFinder varied depending on the specific model. For WSINet and MVNet, predictive labels were based on relapse status, which included recurrence, metastasis, or death related to stage II CRC. In SegNet, the TNTM classifier was tasked with predicting tissue types (TLS, normal tissue, and tumor tissue), while the TLSM classifier focused on TLS subtypes (Agg, FL-1, FL-2). To ensure robust predictions, the Internal-CRCII dataset was split into five equal parts at the case level, and a 5-fold cross-validation strategy was applied. This consistent data division across the different models ensured the comparability of results. Notably, all models—including WSINet, SegNet, MVNet, and MMF—were exclusively developed and internally validated using the Internal-CRCII dataset. To avoid any form of information leakage or adaptation bias, the trained models were directly applied, without retraining or parameter adjustment, to three independent external validation cohorts. All prognostic models were trained for 500 epochs, while predictive models were trained for 100 epochs, with all model parameters trainable during the process [24]. The number of training epochs was determined empirically based on the performance and loss trends observed in the training and validation sets.

WSINet was initialized with random parameters and trained using the AdamW optimizer with weight decay. The cross-entropy loss function was used for optimization. The learning rate started at 0, followed by a linear warm-up for the first 10 epochs, peaking at 1e−3, and then decayed to 0 via a cosine annealing schedule by the end of training.

SegNet consists of two classifiers: TNTM (for tissue type classification) and TLSM (for TLS subtype classification). TNTM was trained on annotated tissue tiles (TLS, normal, and tumor) from WSI regions. Various augmentation techniques were applied, such as resizing, random rotations, vertical and horizontal flips, and color jittering (adjusting brightness, contrast, and saturation/hue). The model was initialized with pre-trained ResNet18 weights from ImageNet and optimized using the SGD optimizer with a momentum of 0.9, a weight decay of 1e−5, and a batch size of 128. The loss function used was cross-entropy. TLSM was trained on stitched TLS images (512 × 512 pixels) derived from TNTM’s prediction heatmaps. Similar to TNTM, TLSM was initialized with pre-trained ResNet18 weights and trained with the same setup.

MVNet comprises two branches: the MpB and the SpB. Both branches were initially trained with randomly initialized weights, while the entire MVNet model was trained with pre-trained weights from MpB and SpB. The Adam optimizer was used with an initial learning rate of 1e-3 and a cross-entropy loss function. Class weights were set at 0.5 for “No relapse” and 1.2 for “Relapse” to address class imbalance. An inverse time decay strategy (decay rate: 9e−3) was used to improve training stability and efficiency. Training was conducted with the full dataset in a single batch. The MMF model integrated clinical information with the MVNet features. The MMF training settings were kept consistent with MVNet for fair comparison, including epoch number, optimizer (Adam), loss function (cross-entropy), and class weights. A cosine learning rate decay with linear warm-up over 10 epochs and a batch size of 128 were employed. A batch size of 128 was used. For internal validation, each fold served as the test set once, with a model trained exclusively on the remaining folds. Predictions from all five folds were concatenated to form the complete internal test set. For external validation, the predicted probabilities from all five cross-validated models were first averaged to form an ensemble prediction, and only this ensemble prediction was binarized using the threshold to obtain the final categorical prediction. This ensemble approach was consistently applied to all external cohorts for both risk stratification and survival analyses. In this study, the stratification thresholds were calculated separately for each dataset. We further evaluated the stratification performance of the model under multiple fixed thresholds, and the results show that MVNet achieves highly significant stratification across all datasets, regardless of whether the thresholds are fixed or dataset-specific (S14 Fig and S3 Table). All patients with follow-up <24 months and no events were included in the survival analyses with their actual follow-up time and event status. All DL experiments were executed on four NVIDIA GeForce RTX 2080 Ti GPUs.

Feature construction for MVNet

To synthesize the multi-view information of TLSs automatically, we extracted both morphological and spatial TLS features using the developed SegNet. Based on the patch-level probabilities from the multi-tissue classifier TNTM, a prediction heatmap was generated for each WSI, indicating the predicted regions of TLSs, normal tissue, and tumor tissue to visualize tissue characteristics. Considering that isolated tiny tissue tiles resulting from mispredictions could cause statistical deviations in distance and area information, a series of image post-processing was adopted to mitigate the effects of noise while maintaining the original characteristics of the tissue heatmaps. The post-processing included removing normal tissue domains from the heatmap, removing connected domains consisting of a single pixel and tumor tissue domains smaller than the threshold, and filling gaps within connected domains. The threshold was set to 20 pixels in this study, based on relevant guidelines and previous literature [25–27]. To demonstrate whether this processing strategy altered the spatial distribution of various components, we randomly selected 50 slides from each dataset and calculated the similarity of their heatmaps before and after processing using the structural similarity index measure (SSIM) method (S15 Fig). This analysis was performed after model evaluation and was independent of any model training or prediction processes. Statistical results showed that the average similarity of the heatmaps before and after processing in each dataset exceeded 0.92, indicating that our processing strategy effectively reduced statistical errors while preserving the original structural information of the heatmaps. The post-processed heatmaps were then utilized for feature construction. For each closed TLS region on the heatmap, the morphological feature was extracted using the pretrained TLS subtype classifier TLSM, along with the predicted phenotype status. Next, the spatial features were constructed, including area, quantity, phenotypes, and relative distances to tumor regions of TLS structures. We analyzed the frequency distribution of the number of TLSs within different ranges of TLS areas, distances of TLSs to the tumor edge, and TLS subtypes across all slides in the four datasets (S16 Fig). The results showed similar distributions of these three metrics across the four datasets, thereby eliminating potential biases due to data collection and indirectly demonstrating the consistent performance of SegNet across datasets from various sources. This analysis was not involved in any stage of model training or evaluation. The continuous variables of distance and area are converted into categorical variables to facilitate subsequent feature construction. Specifically, the distances from the centers of TLSs to the tumor margin were categorized into seven distinct, non-overlapping intervals: ≤0 mm, (0, 0.4] mm, (0.4, 1.0] mm, (1.0, 2.2] mm, (2.2, 4.5] mm, (4.5, 6.4] mm, and >6.4 mm. For External-CRCII-1, the number of TLSs located outside the tumor area was evenly distributed across each distance interval. A uniform interval setting was applied to each dataset. Similarly, the continuous values of TLS area were divided into seven distinct, non-overlapping intervals: (0, 0.03] mm², (0.03, 0.05] mm², (0.05, 0.07] mm², (0.07, 0.10] mm², (0.10, 0.15] mm², (0.15, 0.28] mm², and >0.28 mm². According to the total area and quantity of the three TLS subtypes pertaining to each bin, a total of 226 features were designed to describe the spatial information. The number of segmentation bins was set to balance capturing spatial heterogeneity with maintaining computational stability, with reference to previous studies [28]. The detailed summary was shown in S9 Table.

Interpretability

Both attention- and attribution-based interpretability methods were performed for the explanation of SurvFinder. The best performing model in the cross-validation was selected for observation. For WSINet, the attention scores obtained from the attention layers were normalized, signifying the relative importance of all inputted WSI tiles. The top 50 tiles with the highest attention scores were selected as representative patches for each patient. These highlighted tiles from correctly predicted ‘Relapse’ and ‘No relapse’ patients were aggregated and clustered into five clusters separately. The adopted clustering method was k-means, with t-SNE used for visualization. For each cluster, specific morphological characteristics were summarized by an experienced pathologist, resulting in the identification of 10 prognosis-related morphological features.

Due to the differing structures of the model branches, disparate interpretability methods were applied to the two branches of MVNet. The interpretation of the MpB was similar to that of WSINet, where the feature with the highest attention for each case was extracted for subsequent statistical analysis. The Shapley Additive Explanation-based method (SHAP) was employed for the SpB [29–31]. SHAP-styled attribution decision plots were used to visualize the attribution weights and direction of spatial features for further analyses.

Statistical analysis

The study hypotheses and planned analyses were predefined. All planned analyses were conducted as described, and any deviations from the original plan, including exploratory or data-driven analyses, are clearly indicated. Statistical methods used are described below. SurvFinder was trained and validated using patient-level 5-fold cross-validation, ensuring consistent data splits across all models. Predictive model performance was evaluated using sensitivity, specificity, and the area under the receiver operating characteristic curve (AUROC). Prognostic models were assessed using the AUROC. High-risk and low-risk groups, stratified by prognostic models, were based on the predicted class (1 for high-risk and 0 for low-risk). The cutoff threshold of the DL model’s ROC curve was determined by the Equal Error Rate criterion to dichotomize the model’s probabilities into binary predictions. The primary endpoint of this study was relapse-free survival (RFS), defined as the time from the date of surgery to the date of first disease recurrence, metastasis or death from tumor. Statistical tests included chi-squared tests for confusion matrices, DeLong’s test for AUC comparisons, and log-rank tests for survival curves. Univariable and multivariable Cox regression analyses were used to identify prognostic factors, with significance defined as p < 0.05. Box plots displayed the 25th, 50th, and 75th quantiles. All statistical analyses were conducted in R version 4.2.1 (2022-06-23). Deep learning models were implemented in Python 3.8.12 using the PyTorch 1.10.1 framework. This study is reported as per the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline (S1 Checklist) and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guideline (S2 Checklist).

Study design and patient cohorts

The SurvFinder framework, as outlined in Fig 1, was developed and validated using data from four distinct patient cohorts. Following rigorous data filtering, we included a total of 6,950 H&E-stained pathology slides from 1,604 patients for model development. The detailed inclusion and exclusion criteria are provided in the Methods section. The Internal-CRCII cohort, which was used for model training and internal validation, comprised 3,494 slides from 743 stage II CRC patients from a single medical center. For external validation, we utilized the External-CRCII-1 and External-CRCII-2 cohorts, which consisted of 1,315 slides from 352 cases and 1,708 slides from 331 cases, respectively, collected from two independent medical centers. To ensure accurate representation of the tumor microenvironment, each case in both internal and external datasets included at least three formalin-fixed paraffin-embedded slides containing tumor tissue. Additionally, the TCGA-CRCII cohort, derived from the TCGA-COAD and TCGA-READ projects, included 433 slides from 178 stage II cases. A comprehensive description of these datasets is provided in S1 Fig.

WSINet identifies prognosis-related tissue biomarkers from WSIs

To identify tissue features relevant to prognosis from WSIs, we developed WSINet, an attention-based multiple-instance learning model that serves as the initial component of the SurvFinder framework. WSINet aggregates features from all WSIs of each case to generate a patient-level input for prognosis prediction. The model was trained and evaluated on the Internal-CRCII dataset using 5-fold cross-validation, achieving an AUROC of 0.695 (95% confidence interval [CI] [0.639,0.751]), indicating a statistically significant correlation between model predictions and actual relapse status (Fig 2a and 2b). The attention mechanism in WSINet was leveraged to identify high-attention tissue regions, which were then analyzed using k-means clustering to delineate five significant tissue biomarkers for both no-relapse and relapse groups (Fig 2c and 2d). Notably, the largest cluster within the no-relapse group was enriched for regions exhibiting dense lymphoid aggregates and follicle-like architecture, morphological hallmarks of TLSs (S2 Fig). Importantly, all clustering and feature identification analyses were restricted to the internal development cohort. The external datasets were used exclusively for independent validation, ensuring no data leakage and preserving the integrity of the evaluation process. Taken together, these results suggest that TLSs could serve as key favorable prognostic tissue biomarkers in stage II CRC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The performance and interpretation of WSINet for identifying prognosis-associated tissue biomarkers.

(a) Precision–recall and receiver operating characteristic (ROC) curves of WSINet, illustrating the mean performance across a 5-fold cross-validation. (b) Confusion matrix detailing the predictions. (c, d) Clustering results of highlighted tissue biomarkers for the ‘no relapse’ group (c) and ‘relapse’ group (d) using the k-means clustering method. This includes t-SNE plots (top), cluster characteristic descriptions, and patch counts for each cluster (bottom). The chi-squared test was used to measure predictive performance. Statistical significance is indicated as follows: ns, p > 0.05; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; ****p ≤ 0.0001. ns, not significant; RFS, relapse-free survival.

https://doi.org/10.1371/journal.pmed.1004614.g002

SegNet automatically segments multi-class tissue biomarkers

To further categorize and segment TLS regions identified by WSINet, as well as other tissue biomarkers such as tumor tissue, we developed SegNet, a patch-level multi-tissue classification network. SegNet comprises two models: the TLS-normal-tumor tissue classification (TNTM) and the TLS phenotype classification (TLSM). Both models were trained on Internal-CRCII and validated on both internal and external datasets. The TNTM model demonstrated excellent performance with AUROCs of 0.9997 (95% CI [0.9997,0.9997]) for TLS-like tissue, 0.9991 (95% CI [0.9991,0.9992]) for normal tissue, and 0.9997 (95% CI [0.9996,0.9997]) for tumor tissue on the Internal-CRCII dataset (Fig 3a and 3b), with similarly high performance on External-CRCII-1 and External-CRCII-2, confirming the model’s robustness across different datasets (Fig 3c–3f). Detailed performance metrics are provided in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Segmentation performance of SegNet for multi-class tissue biomarkers.

e ROC curve of TNTM, the first component of SegNet, for segmenting TLSs, normal, and tumor tissue on Internal-CRCII. (b) Confusion matrix of TNTM predictions on Internal-CRCII, where each cell represents the percentage of the total count of the true labels. (c, d) ROC curve (c) and confusion matrix (d) of TNTM on External-CRCII-1. (e, f) ROC curve (e) and confusion matrix (f) of TNTM on External-CRCII-2. (g) ROC curve of TLSM, the second component of SegNet, for identifying TLS phenotypes on Internal-CRCII. (h) Confusion matrix of TLSM performance on Internal-CRCII. (i, j) ROC curve (i) and confusion matrix (j) of TLSM on External-CRCII-1. (k, l) ROC curve (k) and confusion matrix (l) of TLSM on External-CRCII-2. The chi-squared test was used to measure predictive performance. Statistical significance is indicated as follows: ns, p > 0.05; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; ****p ≤ 0.0001. ns, not significant; TLS, tertiary lymphoid structure; TNTM, TLS-normal-tumor segmentation model; TLSM, TLS subtyping model; SegNet, segmentation network; Agg, aggregates; FL-1, primary follicles; FL-2, secondary follicles; Internal-CRCII, internal colorectal cancer stage II cohort; External-CRCII-1, external colorectal cancer stage II cohort 1; External-CRCII-2, external colorectal cancer stage II cohort 2.

https://doi.org/10.1371/journal.pmed.1004614.g003

The TLSM model was designed to further classify TLSs into three distinct phenotypes: aggregates (Agg), primary follicles (FL-1) and secondary follicles (FL-2) [32]. On Internal-CRCII dataset, the TLSM model achieved AUROCs of 0.9476 (95% CI [0.9366,0.9586]) for Agg, 0.91 (95% CI [0.8941,0.9259]) for FL-1, and 0.9851 (95% CI [0.9787,0.9916]) for FL-2. Similar performance was observed in the External-CRCII-1 and External-CRCII-2 datasets (Fig 3g–3l and S2 Table). These results validate SegNet’s ability to accurately segment and subtype tissue biomarkers, laying a histological foundation for further extraction and integration of multi-view prognostic features from WSIs.

MVNet predicts stage II CRC patients’ prognosis via multi-view combination

We then developed MVNet, a multi-view fusion model that integrates spatial and morphological characteristics of TLSs for prognosis prediction. MVNet consisted of three components: a SpB, a MpB, and a fusion predictor (S3 Fig). The predictive performance of these models was compared across all datasets (Fig 4a–4d). The MpB, trained on latent features derived from TLSM, achieved AUROCs of 0.688 (95% CI [0.637,0.739]), 0.65 (95% CI [0.581,0.719]), and 0.659 (95% CI [0.584,0.734]) on the Internal-CRCII, External-CRCII-1, and External-CRCII-2 datasets, respectively. The SpB, trained on spatial features such as distance, area, and phenotypes of TLS, achieved AUROCs of 0.746 (95% CI [0.699,0.793]), 0.737 (95% CI [0.667,0.808]), and 0.775 (95% CI [0.717,0.833]) on the same cohorts. Combining spatial and morphological features in the fusion predictor resulted in AUROCs of 0.827 (95% CI [0.789,0.864]), 0.805 (95% CI [0.749,0.860]), and 0.805 (95% CI [0.748,0.861]), demonstrating a statistically significant concordance between the MVNet predictions and the ground truth (S4 Fig). MVNet consistently outperformed both WSINet and the single-branch models, proving the effectiveness of multi-view integration for prognostic assessment (S5 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The predictive and prognostic performance of SurvFinder.

(a) ROC curves of WSINet, MpB, SpB, and MVNet on Internal-CRCII. (b–d) ROC curves of the MpB, SpB, and MVNet on External-CRCII-1 (b), External-CRCII-2 (c), and TCGA-CRCII (d), respectively. (e–h) Kaplan–Meier (K–M) survival curves for all cases stratified by the predicted risk from MVNet, on Internal-CRCII (e), External-CRCII-1 (f), External-CRCII-2 (g), and TCGA-CRCII (h), respectively. Censors are indicated with a ‘+’. (i–l) ROC curves comparing the prognostic capacity of MMF with MVNet and other significant clinicopathological factors on Internal-CRCII (i), External-CRCII-1 (j), External-CRCII-2 (k), and TCGA-CRCII (l), respectively. The log-rank test was used to calculate statistical significance. Statistical significance is indicated as follows: ns, p > 0.05; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; ****p ≤ 0.0001. ns, not significant; SpB, spatial feature branch; MpB, morphological feature branch; MVNet, multi-view network; MMF, multi-modal fusion; MMR, mismatch repair; LNS, lymph node sampling; SRCC, signet-ring cell carcinoma; MAC, mucinous adenocarcinoma; PNI, perineural invasion; Internal-CRCII, internal colorectal cancer stage II cohort; External-CRCII-1, external colorectal cancer stage II cohort 1; External-CRCII-2, external colorectal cancer stage II cohort 2; TCGA-CRCII, TCGA colorectal cancer stage II cohort.

https://doi.org/10.1371/journal.pmed.1004614.g004

MVNet surpasses and complements clinicopathological prognostic factors in stage II CRC

To assess MVNet performance in a real-life scenario, we first evaluated the RFS in the high- and low-risk subgroups predicted by MVNet using Kaplan–Meier (K–M) analysis. Significant survival differences (P < 0.0001) were observed across all datasets in log-rank tests (Fig 4e–4h). Next, we compared MVNet with 16 clinicopathological indicators frequently used for risk stratification as well as ACT status in stage II CRC. In univariate Cox regression analysis on Internal-CRCII, four factors were significant, including tumor budding (P = 0.044), MMR (P = 0.003), lymph node sampling (LNS, P < 0.001) and MVNet (P < 0.001) (S6 Fig). LNS was defined as the number of lymph nodes examined during surgery, with <12 nodes considered insufficient, in accordance with NCCN guidelines [3,4]. In multivariate Cox regression, MVNet demonstrated the highest hazard ratio (HR: 8.23, 95% CI: 5.43–12.47; p < 0.001), outperforming other significant factors (S6 Fig). These results confirmed that MVNet is a strong independent prognostic factor, significantly outperforming other established biomarkers. The correlation analysis also validated the independence of the model’s prediction (S7 Fig). Similar findings were observed in external datasets (S6c–S6h and S7b–S7d Figs). Additionally, the model’s performance was evaluated across multiple fixed thresholds (S3 Table).

MVNet consistently showed superior prognostic performance, establishing itself as an independent and robust predictor. To explore the potential for combining MVNet with clinicopathological data, we developed a MMF model. This fusion model exhibited consistently higher AUROCs than both MVNet and the clinical-only model across all datasets (Figs 4i–4l and S8), confirming the complementary nature of SurvFinder with established biomarkers.

Additionally, we integrated MVNet into nomograms alongside clinicopathological factors (S7e–S7h Fig), with calibration curves closely aligning with ideal predictive models (S7i–S7l Fig). These findings indicate that MVNet enhances the predictive power of established clinicopathological biomarkers, confirming its potential as both an independent prognostic tool and a complementary asset in the clinical management of in stage II CRC.

MVNet’s potential to inform ACT decisions in stage II CRC

We next explore MVNet’s potential to inform ACT decisions in stage II CRC. First, we assessed the impact of ACT on prognosis across different datasets (Fig 5a–5c). Although a significant association between ACT and improved prognosis was observed in the External-CRCII-2 cohort (p = 0.014), K–M analysis indicated no significant improvement in prognosis with ACT in the Internal-CRCII (p = 0.12) and External-CRCII-1 datasets (p = 0.53). Subsequently, we analyzed the MVNet-predicted high-risk subgroup (Fig 5d–5f). In this group, ACT was significantly associated with a favorable prognostic response in Internal-CRCII (p = 0.023) and External-CRCII-1 (p = 0.026) cohorts. Notably, in the External-CRCII-2 cohort, the high-risk group showed even stronger statistical significance and greater benefit in the K–M curve (p = 0.00081) compared to the full cohort. In contrast, MVNet-predicted low-risk patients derived no significant survival benefit from ACT in any cohort (Fig 5g–5i). In addition, MVNet consistently served as a significant prognostic factor across both the ACT and no-ACT subgroups (S4 Table and S9 Fig). The distribution of clinicopathological high-risk features across each subgroup is presented in S5 Table. We further performed Cox proportional hazards models including an interaction term between ACT and MVNet risk group. Although no significant interaction was observed in the internal dataset, significant or near-significant interactions were identified in two external cohorts (S6 Table). These results suggest that MVNet may serve as a clinically meaningful tool for guiding ACT decisions in stage II CRC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Performance of SurvFinder for decision-making of adjuvant chemotherapy.

(a–c) Kaplan–Meier (K–M) curves for all patients from Internal-CRCII (a), External-CRCII-1 (b), and External-CRCII-2 (c), stratified by the receipt status of adjuvant chemotherapy (ACT). Censors are indicated with a ‘+’. (d–f) Kaplan–Meier (K–M) survival curves for cases predicted as high risk by MVNet, stratified by the receipt status of ACT, on Internal-CRCII (d), External-CRCII-1 (e), and External-CRCII-2 (f), respectively. The log-rank test was used to calculate statistical significance. Statistical significance is indicated as follows: ns, p > 0.05; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; ****p ≤ 0.0001. ns, not significant; ACT, adjuvant chemotherapy; Internal-CRCII, internal colorectal cancer stage II cohort; External-CRCII-1, external colorectal cancer stage II cohort 1; External-CRCII-2, external colorectal cancer stage II cohort 2.

https://doi.org/10.1371/journal.pmed.1004614.g005

XAI on SurvFinder unveils prognostically relevant TLS characteristics

To enhance the interpretability of SurvFinder and better understand the prognostically relevant features learned by the model, we applied XAI techniques to the risk prediction models, as detailed in the Methods section. The top 10 spatial features contributing to prognosis were identified across internal and external datasets (Fig 6a–6c). Most of these spatial characteristics were associated with favorable prognosis, whereas the mean tumor area per slide correlated with poorer prognosis. These findings align with existing research in CRC and confirm that SurvFinder is grounded in established biological principles [33–35]. Interestingly, more than half of the identified features were located between 1 and 4.5 mm from the tumor margin, suggesting that TLSs in this region are critical for prognosis. Furthermore, TLS subtypes located 2.2–4.5 mm from the tumor margin consistently correlated with favorable prognosis, indicating that TLSs in this area may serve as important indicators of low-risk outcomes in stage II CRC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Interpretation of SurvFinder.

(a–c) Top 10 global spatial feature attributions to all predictions, sorted by distances to the tumor margin and subtypes for Internal-CRCII (a), External-CRCII-1 (b), and External-CRCII-2 (c), respectively. Each attribution feature represents the exact contribution of a TLS phenotype distributed within specific distances to the invasive margin of the tumor. (d–f) Confusion matrices on Internal-CRCII (d), External-CRCII-1 (e), and External-CRCII-2 (f), comparing the maturity differences of high-priority TLS morphological features extracted from ‘relapse’ and ‘no relapse’ groups. The chi-squared test was used to measure differences. Statistical significance is indicated as follows: ns, p > 0.05; *p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; ****p ≤ 0.0001. ns, not significant; Internal-CRCII, internal colorectal cancer stage II cohort; External-CRCII-1, external colorectal cancer stage II cohort 1; External-CRCII-2, external colorectal cancer stage II cohort 2.

https://doi.org/10.1371/journal.pmed.1004614.g006

We also investigated the morphological features of TLSs highlighted by the model. The attention maps revealed that mature TLSs (FL-2) were more prevalent in the non-relapse group, while immature TLSs (Agg and FL-1) were more frequent in the relapse group (Fig 6d–6f). This pattern was consistent across all datasets, suggesting that TLS maturity is positively correlated with better prognosis in stage II CRC. The WSI visualizations of the interpretable results are presented in S10 Fig. Overall, the interpretation analyses support the biological relevance of the features used by SurvFinder and provide new insights into the role of TLSs in stage II CRC prognosis.

SurvFinder’s predictive accuracy in TCGA dataset

To validate the generalizability of SurvFinder, we tested its predictive and prognostic performance on the TCGA dataset. Prognostic predictions in the TCGA-CRCII cohort yielded AUROCs of 0.575 (95% CI [0.455,0.694]) for the MpB, 0.606 (95% CI [0.508,0.703]) for the SpB, and 0.712 (95% CI [0.621,0.804]) for the MVNet (Fig 4d). These results demonstrate that SurvFinder maintains robust performance in the TCGA dataset (S4d and S5d Figs), with stable risk stratification as shown in the K–M curves (Fig 4h).

Due to incomplete clinical data in the TCGA cohort, Cox regression analyses were performed using only available clinicopathological factors and the MVNet score. In the univariate Cox analysis, five factors were statistically significant: tumor budding (p = 0.007), Grading (p = 0.037), T stage (p = 0.002), BD3 (p = 0.034) and MVNet (p = 0.002) (S6g Fig). Multivariate Cox regression confirmed MVNet as an independent prognostic factor (S6h Fig). MVNet continued to outperform other clinicopathological parameters and maintained stable performance in multimodal data integration (Figs 4l, S7d, S7h, and S7l). These results underscore SurvFinder’s ability to stratify risk and predict outcomes across diverse patient populations, supporting its potential generalizability across multiple datasets.