Table 1.
Comparison of three commonly used consistency-based semi-supervised methods.
Table 2.
Comparison of three commonly used consistency-based semi-supervised methods.
Fig 1.
Overview of our S2MMAM, including: (a) Supervised Multilevel Fusion Segmentation Network (SMF-SN). The inputs are CT images and pixel-level mask images, and the outputs are segmented lesion images, (b) Semi-supervised Multimodal Fusion Classification Network (S2MF-CN), and (c) processing of gene data. In the S2MMAM, the useful information of CT images is captured by SMF-SN and transferred to S2MF-CN to facilitate the execution of image prediction tasks. The S2MMAM utilizes the fusion of CT images and genetic data to accurately predict whether KRAS is mutated in NSCLC.
Fig 2.
Block diagram of the proposed SMF-SN architecture.
We adjust the dilation rates in ASPP in the bridge from 6,12,18 to 3,6,9 to better adapt SMF-SN to our segmentation task.
Fig 3.
The architecture of SE-ResNext.
SE-ResNeXt is improved from ResNeXt with SENet.
Fig 4.
Framework diagram of the proposed TAFA module.
Fig 5.
The overview of the Student Module, including (a) the specific implementation details of the Student Model, (b) Intra fusion component (IntraFC) aims to fuse classification and segmentation features at different levels, and (c) Inter fusion component (InterFC) aims to fuse CT image features and genetic features.
Table 3.
Patients’ medical record information in the dataset.
Table 4.
The initialization network configurations of model.
Table 5.
Comparison of classification performance of UNet, ResNet, ResNeXt, Inception-v3 and SE-ResNeXt on S2MMAM.
SE-ResNeXt(Ours) achieved the best results in all six comparative metrics.
Table 6.
Comparison of classification performance of TAFA on S2MMAM and four models with different fusion blocks.
TAFA(Ours) achieved the best results in all six comparative metrics.
Fig 6.
Comparison of the classification performance of IntraFC and three models using other fusion methods.
Fig 7.
Comparison of the classification performance of InterFC and two models using other fusion methods.
Fig 8.
AUC of our S2MMAM and five other medical image classification models on 30% labeled image dataset.
Table 7.
Comparison of the classification performance of S2MMAM and five other semi-supervised medical image classification models.
Table 8.
Six metrics were achieved on the test set by Baseline, Baseline+SMF-SN, Baseline+Gene, and our S2MMAM when using 30%, 40%, and 100% labeled training images.
Fig 9.
Comparison of the segmentation results obtained after training on Baseline strategy and Baseline+SMF-SN strategy: Baseline: Only classification task.
Baseline+SMF-SN: classification task and segmentation task. (a) and (b) are the wild type of NSCLC. (c) and (d) are the mutation of NSCLC. The region surrounded by the red line is the ground truth, and the region surrounded by the green line is the segmentation results.
Fig 10.
AUC were achieved on the test set by Baseline, Baseline+SMF-SN, Baseline+Gene and our S2MMAM, when using 30%, 40% and 100% labeled training images.
Fig 11.
F1 score were achieved on the test set by Baseline, Baseline+SMF-SN, Baseline+Gene and our S2MMAM, when using 30%, 40% and 100% labeled training images.
Table 9.
Comparison of the classification performance of S2MMAM and two other supervised medical image classification models.