https://orcid.org/0000-0002-0109-0998
Figures
Abstract
Many previous studies focused on differentiating between benign and malignant soft tissue tumors using radiomics model based on various magnetic resonance imaging (MRI) sequences, but it is still unclear how to set up the input radiomic features from multiple MRI sequences. Here, we evaluated two types of radiomics models generated using different feature incorporation strategies. In order to differentiate between benign and malignant soft tissue tumors (STTs), we compared the diagnostic performance of an ensemble of random forest (R) models with single-sequence MRI inputs to R models with pooled multi-sequence MRI inputs. One-hundred twenty-five STT patients with preoperative MRI were retrospectively included and consisted of training (n = 100) and test (n = 25) sets. MRI included T1-weighted (T1-WI), T2-weighted (T2-WI), contrast-enhanced (CE)-T1-WI, diffusion-weighted images (DWIs, b = 800 sec/mm2) and apparent diffusion coefficient (ADC) maps. After tumor segmentation on each sequence, 100 original radiomic features were extracted from each sequence image and divided into three-feature sets: T features from T1- and T2-WI, CE features from CE-T1-WI, and D features from DWI and ADC maps. Four radiomics models were built using Lasso and R with four combinations of three-feature sets as inputs: T features (R-T), T+CE features (R-C), T+D features (R-D), and T+CE+D features (R-A) (Type-1 model). An ensemble model was built by soft voting of five, single-sequence-based R models (Type-2 model). AUC, sensitivity, specificity, and accuracy of each model was calculated with five-fold cross validation. In Type-1 model, AUC, sensitivity, specificity, and accuracy were 0.752, 71.8%, 61.1%, and 67.2% in R-T; 0.756, 76.1%, 70.4%, and 73.6% in R-C; 0.750, 77.5%, 63.0%, and 71.2% in R-D; and 0.749, 74.6%, 61.1%, and 68.8% R-A models, respectively. AUC, sensitivity, specificity, and accuracy of Type-2 model were 0.774, 76.1%, 68.5%, and 72.8%. In conclusion, an ensemble method is beneficial to incorporate features from multi-sequence MRI and showed diagnostic robustness for differentiating malignant STTs.
Citation: Lee S, Lee S-Y, Jung J-Y, Nam Y, Jeon HJ, Jung C-K, et al. (2023) Ensemble learning-based radiomics with multi-sequence magnetic resonance imaging for benign and malignant soft tissue tumor differentiation. PLoS ONE 18(5): e0286417. https://doi.org/10.1371/journal.pone.0286417
Editor: Jose Gerardo Tamez-Peña, Tecnologico de Monterrey, MEXICO
Received: December 23, 2022; Accepted: May 15, 2023; Published: May 31, 2023
Copyright: © 2023 Lee et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was supported by the National Research Foundation of Korea (NRF, https://www.nrf.re.kr/eng/main/) grant funded by the Korea government (MSIT) 2022R1A2C4002395. Joon-Yong Jung received the funding. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: STT, Soft tissue tumor; MRI, Magnetic resonance imaging; CE, Contrast enhancement; DWI, Diffusion-weighted imaging; ADC, Apparent diffusion coefficient; R, Random forest; TSE, Turbo spin-echo; VOI, Volume of interest; Lasso, Least absolute shrinkage and selection operator; AUC, Area under the receiver operating curve
Introduction
Soft tissue tumors (STTs) are heterogeneous groups of tumors with different cellular origin and heterogeneous tissue components [1]. As there are more than 50 histologic subtypes in STTs, various extracellular components such as hemorrhage, mineralization, fat, and myxoid tissue are mixed in a mass with diverse distribution [2]. These components show distinctive signal intensity on a series of magnetic resonance images (MRI) and radiologists identify the diagnosis through the correlated signal intensity of certain components. Standard interpretation of STTs is based on a conventional MR sequence of T1- and T2-weighted images (T1-WI, T2-WI) and contrast-enhanced (CE)-T1-WI if contrast media is available. STTs with larger size, lobular contours, and heterogeneous components on conventional MRI are more likely to be malignant tumors, but there is no definite diagnostic qualitative criteria for diagnosis of soft tissue sarcoma because diverse subtype of soft tissue sarcoma can show various imaging findings [3–5]. Recently, advanced imaging techniques such as diffusion-weighted imaging (DWI), dynamic contrast-enhanced imaging, and MR spectroscopy are combined to acquire functional information including cellularity, vascularity, or metabolism [6]. This additional information could support radiologists’ assessment of conventional sequences when differentiating between benign and malignant tumors. However, it is still challenging to diagnose benign or malignant soft tissue tumor based on imaging alone, because multi-sequence MRI could lead to confusion by providing discrepant findings with benign or malignant histology of tumor itself, like low apparent diffusion coefficient (ADC) value of epidermal inclusion cyst, high ADC value of sarcoma with myxoid matrix or high-flow hemodynamic profile on dynamic contrast-enhanced imaging of vascular anomaly [7–9].
Radiomics is an approach utilizing high-dimensional quantitative features extracted from images to build a classification model. Recently, many radiomics studies have been published to distinguish the malignancy and histologic grades of STTs using conventional MRI [10–15]. In early studies, features from two MR sequences (T1-WI and T2-WI with or without fat suppression) were pooled in a single set of combined MR features and used as input for classification models [11, 12, 16]. More recent studies have developed separate models from each of the two MR sequences (fat-suppressed T2-WI and T1-WI with or without CE) and models based on combined features from two MR sequences [13, 14, 17]. They also pooled all MR features into a single set to construct a combined features set with the same method as in previous studies. When differentiating soft tissue sarcoma from benign tumors, diagnostic performance is higher when multiple MRI sequences are reviewed together [1, 18]. Therefore, radiomics models from multiple MR sequences including CE images and DWIs may be more useful than radiomics models from two conventional MR sequences. However, traditional pooling methods have possibility of loss of information from each MR sequences, which can result in decreased generality to histologically diverse tumors. Therefore, we assumed that the ensemble model was suitable for creating a model that reflects all information in each sequence and aimed to verify it.
The ensemble learning approach is a combination method of multiple classifiers to achieve better performance than single-model classification [19]. Predictions from several different classification models are used as input for the ensemble model and final predictions are yield by proportion of votes to favor one class. This process can resolve the reproducibility problem by eliminating selection bias of single-classifier algorithms [20]. We assumed that an ensemble learning of several different classification models which are respectively derived with input of a single MR sequence, would perform better by preserving information from different sequences. It can also be similar to the interpretation process by radiologists, by the point that they interpret the image finding by combining information from conventional and advanced MR sequences, and their decisions based on multi-sequences are usually more accurate than judging based on single-sequence alone. In field of radiomics study, it is not well understood if it is more accurate to combine the classification results from multiple MR sequences. In addition, no definite method to combine radiomic features has been established or evaluated.
In this study, we aimed to evaluate diagnostic performance of an ensemble of random forest (R) models with single-sequence inputs compared to R models with multi-sequence inputs for differentiating benign and malignant soft tissue tumors
Materials and methods
This retrospective study was approved by our institutional review board, and the requirement for informed consent was waived. This work was supported by the National Research Foundation of Korea grant funded by the Korea government (Ministry of Science and ICT) 2022R1A2C4002395.
Patient population
From January 2009 to August 2019, a total of 365 patients at our institution underwent 3.0T preoperative multi-sequence MRI including T1-WI, T2-WI, CE-T1-WI, DWIs, and ADC map for primary STT evaluation. The cohort was reduced to 125 patients according to the following exclusion criteria (Fig 1). First, multi-sequence MRI was performed after any neoadjuvant treatment was excluded (n = 47). Tumors showing distortion on MR images due to artifacts (n = 42) and lesions less than 1 cm in size (n = 31) were excluded. Second, all tumors required pathology confirmed by surgical excision with histological analysis on the excisional sample performed by one pathologist (C.K.J., 15 years of experience in musculoskeletal tumors) and patients without pathologic confirmation were excluded (n = 45). Finally, well-differentiated adipocytic tumors including lipomas and well-differentiated liposarcomas were excluded (n = 75) because of abnormal signal intensity on DWI and ADC map as a fat-suppressed sequence [21]. All included patients were previously involved in the patient cohort of our previous studies [22]. The previous studies utilized diffusion-weighted images for differentiation of benign and malignant STT and T2-weighted Dixon sequence for soft tissue sarcoma margin evaluation, respectively.
MRI protocols
MR images were acquired before surgery or neoadjuvant treatment in all patients. MRI was performed using 3.0T scanner (MAGNETOM Verio; Siemens Healthineers, Erlangen, Germany) with dedicated surface coils depending on tumor location. The standard MRI protocols included the longitudinal fat-suppressed T2-weighted turbo spin-echo (TSE) sequence, axial T1-weighted TSE sequence, axial T2-weighted TSE sequences with and without fat suppression, and longitudinal and axial fat-suppressed contrast-enhanced T1-weighted TSE sequence. MRI parameters are shown in S1 Table. Before contrast enhancement, a single-shot spin-echo echo-planar DWI sequence was also obtained on the axial plane. A parallel imaging technique using GeneRalized Autocalibrating Partially Parallel Acquisitions was combined with an acceleration factor of two. Sensitizing diffusion gradients were applied with b values of 0 and 800 sec/mm2 sequentially in the x, y, and z directions [23]. Pixel-based ADC maps were created from DWIs based on mono-exponential calculation using commercial software and a workstation (Leonardo MR Workplace; Siemens Medical Solution, Erlangen, Germany). For contrast-enhanced image, the contrast agent was injected with 0.1 mmol per kg body weight gadobutrol (Gadovist; Bayer, Berlin, Germany). To ensure a stable injection speed, the contrast agent was injected intravenously with an automatic injector (Spectris Solaris; Medrad, Indianola, Pennsylvania) at a rate of 2 mL/s. Subsequently, 20 mL of physiological saline was injected at the same rate.
Radiomics model acquisition
1. Tissue segmentation.
Segmentation was initially performed by one radiologist (S.L, 3 years of experience on musculoskeletal radiology) using the semiautomatic region intensity filter method, which was implemented using ITK-SNAP software, version 3.8.0 (open source, http://www.itksnap.org/) [24] on each MR sequence image and manually revised. Volume of interest (VOI) was drawn along the entire mass except for the most peripheral portions to avoid partial-volume effects. To review the reproducibility of VOI segmentation, final correction of the peripheral portion in VOI was edited in 20 tumor images by three readers: one student (H.J.J.) and two radiologists (S.L., S.Y.L., 3 and 13 years of experience in musculoskeletal radiology, respectively).
2. Image preprocessing and radiomic feature extraction.
The single VOI was selected by consensus of the three readers (H.J.J., S.L., and S.Y.L.) for further preprocessing steps. After VOI confirmation, normalization of the all 5 image sequences were conducted with the following equation: , with f (x) as normalized intensity, x as original intensity, μx as mean, and σx as standard deviation of image signal intensity [25]. As the average ranges of normalized signal intensity in all 5 image sequences extend from 5166 to 9704, gray-level discretization was done with a bin count of 200 to acquire the adequate contrast level and noise reduction in result image. Finally, the VOIs were resampled at a spatial resolution of 1 × 1 × 1 mm3 in T1-WI, T2-WI, and CE-T1WI and of 3 × 3 × 3 mm3 in DWI with a b value of 800 sec/mm2 and ADC maps with spline interpolation. Radiomic feature calculations were obtained using the PyRadiomics package (https://github.com/Radiomics/pyradiomics/) [25]. Within each VOI, (a) 18 first-order features, (b) 14 volume and shape features, and (c) 68 texture features were extracted as the original radiomic features. We only used original radiomic features acquired from each of the five sequences and divided them into three sets; T1 and T2 features (T features) from T1-WI and T2-WI, CE features from CE-T1WI, and diffusion and ADC features (D features) from DWIs and ADC maps. Volume and shape features were represented by those features extracted from DWI.
3. Feature selection and classification model building.
We used Python to construct the classification model with the least absolute shrinkage and selection operator (Lasso) regression and R algorithm. Lasso regression is frequently used for feature selection and overfitting prevention with L1 regularization [26]. The best lambda value which minimize the misclassification error is acquired through the setting of hyperparameter in Lasso regression, and the feature importance is determined by the numerical order of the regression coefficients. The parameters for Lasso regression are summarized in the S1 Appendix. The R classifier is a well-established classifier in radiomics and is used to produce multiple decision trees [27]. R integrates votes from each decision tree composing the forest and eventually acquires a percentage of votes that favors one class. The model development was performed on personal computer equipped with 11th Gen Intel Core i9-11900K 3.50GHz processor (Intel Corporation, Santa Clara, California). Multiple classification models were developed in two ways depending on how the information in each MR sequence was incorporated. The type-1 radiomics model is a traditional method of combining features. The four combinations of three input feature sets were as follows: 1) T features only (R-T model based on T1-WI and T2-WI), 2) T + CE features (R-C model based on T1-WI, T2-WI and CE-T1WI), 3) T + D features (R-D model based on T1-WI, T2-WI, DWI and ADC maps), and 4) T + CE + D features (R-A model based on T1-WI, T2-WI, CE-T1WI, DWI and ADC maps). The type-2 radiomics model is an ensemble model using five R models that were constructed from one of five sequences using T1, T2, CE, diffusion, and ADC features. In the ensemble model, the final classification was determined by soft voting using averaged probability from five, single-sequence-based R models. R algorithm itself is also based on ensemble learning method, but it generates the machine learning model based on bootstrapping of features to make subgroups for multiple decision trees, and this process can obscure the individual importance of features from each MR sequence for differentiation of benign and malignant STTs. However, as we applied the additional ensemble learning method for incorporation of information from each MR sequence in type-2 radiomics model, it can preserve the individual importance of features from each MR sequence. The hyperparameters were manually tuned in the type-1 radiomics models (See S2 Appendix for further details) and determined via the Grid Search algorithm in each of the five models consisting of the type-2 ensemble model (See S3 and S4 Appendices for further details). All steps for all radiomics workflows are demonstrated in Fig 2.
(a) Four alternative combinations of the three input feature sets were demonstrated in type-1 radiomics models with multi-sequence inputs. (b) Five R models built from one of five sequences containing including T1, T2, CE, diffusion, and ADC features generated type-2 radiomics model with ensemble learning approach.
Statistical analysis
Student’s t-test and chi-square tests were used to assess the difference between benign and malignant STTs. During image preprocessing, the Dice coefficient was calculated to measure similarity between the segmentations drawn by the three readers [28]. The Scikit-learn toolboxs were used to conduct the R classifier model (sklearn_ensemble.RandomForestClassifier) and Grid Search algorithm (sklearn model_selection.GridSearchCV) in Python. To assess performance of each radiomics model, five-fold cross-validation was performed, so 125 patients were divided into 5 parts including 25 patients in each part. Then, the training and test sets were consisted of 100 and 25 patients, respectively, and the model was validated by changing test set to one of 5 parts sequentially in each fold. The average of area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy were calculated with the results acquired from all five folds. Differences of AUCs among type-1 radiomics models and type-2 ensemble model were evaluated using the DeLong test. All statistical tests were performed using R version 4.0.0 (http://www.r-project.org/) and MedCalc for Windows, version 19.0 (MedCalc Software, Ostend, Belgium). A p-value <0.05 was considered statistically significant.
Results
In a total of 125 patients, the numbers of benign and malignant STTs were 71 (56.8%) and 54 (43.2%), respectively. The demographic data of all included tumors are shown in Table 1. There was a significant difference in age (p <0.001) and sex (p = 0.02) between benign and malignant STTs.
In 20 randomly selected tumor images for evaluation of segmentation reproducibility, 7 (35%) and 13 (65%) tumors were identified as benign and malignant STTs, respectively. The average and standard deviation of the Dice coefficient between the three segmentations by the three readers were 0.92 ± 0.08 (range, 0.57–1) on T1-WI, 0.93 ± 0.06 (range, 0.68–1) on T2-WI, 0.94 ± 0.06 (range, 0.71–1) on CE-T1WI, 0.92 ± 0.08 (range, 0.67–1) on b-800 DWI, and 0.90 ± 0.07 (range, 0.62–1) on ADC map.
In the type-1 radiomics models, averaged AUC, sensitivity, specificity, and accuracy from five-fold cross validation were 0.752 (95% confidence interval, 0.667 to 0.825), 71.8%, 61.1%, and 67.2% in R-T; 0.756 (0.671 to 0.828), 76.1%, 70.4%, and 73.6% in R-C; 0.750 (0.664 to 0.823), 77.5%, 63.0%, and 71.2% in R-D; and 0.749 (0.663 to 0.822), 74.6%, 61.1%, and 68.8% in R-A models, respectively. The number of selected radiomic features through Lasso regression in each model were 11, 11, 16, and 15 in the R-T, R-C, R-D, and R-A models, respectively. Considering the feature importance, top 10 features were included for model construction and are shown in Table 2 and Fig 3. In case of models that used input features including CE features (R-C and R-A models), the number of selected CE features were predominant and showed high importance for model construction. There was no predominant feature set contributing to R model construction among T and D features. However, no apparent difference in diagnostic performance of all type-1 radiomics models was observed, even though there were variations in the number and type of input feature sets in each model construction (p-value ranged from 0.63 to 0.97 in pairwise comparison of all type-1 radiomics models). Averaged AUC, sensitivity, specificity, and accuracy of classifier ensemble construction with type-2 radiomics models were 0.774 (0.691 to 0.844), 76.1%, 68.5%, and 72.8%, respectively. The results of type-1 and type-2 radiomics models are summarized in Table 3. The ROC curves and F1 scores of each five-fold cross validation are displayed in Fig 3.
(a) R-T, (b) R-C, (c) R-D, (d) R-A and (e) ensemble models showed average AUC values of 0.752, 0.756, 0.750, 0.749 and 0.774, respectively. The AUC of (e) type-2 radiomics model was superior to those of (a, b, c and d) all type-1 radiomics models, but the difference was not significant.
The AUC of a type-2 radiomics model was superior to those of all type-1 radiomics models, but the difference was not significant (p-value = 0.32, 0.42, 0.22, 0.24 compared to R-T, R-C, R-D and R-A model, respectively). In addition, the AUC of the final ensemble model in the type-2 radiomics model was higher than those of all individual models (T1WI model; AUC = 0.710 (0.622 to 0.787), T2WI model; AUC = 0.757 (0.672 to 0.829), CE-T1WI model; 0.716 (0.628 to 0.793), DWI model; AUC = 0.717 (0.629 to 0.794), and ADC model; AUC = 0.672 (0.582 to 0.753)). For further information, computational time for type-1 radiomics model (average training and test time = 34.65 and 3.40ms in R-A model) was shorter compared to type-2 radiomics model (average training and test time = 47.64 and 4.45ms in all individual models, average time for ensemble learning = 22.26ms).
Discussion
In this study, we evaluated the effectiveness of multi-sequence MRIs with a radiomics approach for distinguishing benign and malignant STTs. The ensemble of R classifiers with combination of single-sequence inputs showed the highest AUC with a slightly increased value compared to the other single R models with serial combination of multi-sequence features for diagnosing benign and malignant STTs.
There were a few studies regarding differentiation of benign and malignant STTs using radiomics from two conventional MR sequences [11, 12, 14, 15]. The research of Wang et al. [12] presented various AUCs according to the deployment of 12 different machine learning methods with the input of pooled features of T1- and T2-WI. They found that a combination of the Lasso feature selection method and R classification achieved the highest AUC of 0.82 in validation cohort. This result is comparable with that of the R-T model (0.752, 95% CI, 0.667–0.825) presented in our study, considering that all filtered radiomic features were included in model construction in the previous study. Also, the recent study of Yue et al. [14] showed that the AUC of combined MR features (0.892) showed no significant difference with than that of fat-suppressed T2-WI features (0.883). In our study, there was no difference of diagnostic performance of the R-A model, although the types of features selected through LASSO regression were distributed across all sequences. However, our study has an important difference with previous studies, by the point that we included all essential MR sequences that have been utilized for oncology imaging and compared individual influence of single-sequence feature inputs on radiomics by generating feature-stacking model and ensemble model.
Several previous studies demonstrated the benefit of CE and DWI MRIs for STT diagnosis. However, in our study, there was only slight improvement for diagnosing benign and malignant STTs with addition of sequence features in the radiomics model. This finding is consistent with one previous study regarding the diagnostic validity of ADC-derived radiomic features, which showed no significantly additive diagnostic value to conventional ADC measurement for differentiating benign and malignant STTs [22]. Also, another study that evaluated the predictive performance of radiomics models constructed with fat-saturated T2-WI, CE-T1-WI, and both combined images for soft tissue sarcoma grading had similar high performance between T2-based models and the combined model [15]. In our study, no difference in diagnostic performance of the R-A model was apparent, although the types of features selected through Lasso regression were distributed across all sequences. Therefore, we suspect that there is no prominent additional diagnostic value by adopting radiomic features from CE-T1-WI, DWI, or ADC maps in lieu of features from T1- and T2-WI for diagnosing malignant STTs.
Through ensemble learning, we can incorporate the individual information from each MR sequence in differentiating benign and malignant STTs. This is the first study regarding ensemble learning of radiomics on STTs, but a few studies have been published on other tumors such as brain, lung, and rectal cancers [29–31]. These studies generated ensemble models with a combination of outputs from different types of classifiers such as Nearest Neighbor or Support Vector Machine learning algorithms using a pooling set of imaging features. Unlike those studies, the ensemble model of this study was developed with a combination of outputs from multiple individual R models from each MR sequence. Through this method, we focused on integrating the information of each sequence rather than integrating the information of each machine learning method. This study found that diagnostic performance of the ensemble model was slightly higher than that of conventional feature-stacking models. Even though there was no significant difference in diagnostic performance between the two types of models, an ensemble model allow greater generality of radiomics models by preserving the individual importance of features from each MR sequence.
There are a few limitations in our study. First, this study was a retrospective single-center study with a single MR scanner. Although we validated model performances using averaged values from five-fold cross validation, external validation with data sets from different institutions is regarded as an optimal way to prove generalizability. Second, we did not compare the difference of diagnostic performance between classifier algorithms, which is typically referred to as the ensemble approach in other studies. R classifier is a representative ensemble learner, so it might be inappropriate to apply to other ensemble steps; however, R classifier is usually included in top ranking classifiers that are chosen for ensemble approaches in other studies [10, 32]. Our results showed additive diagnostic value of the additional ensemble step, so we suspect that there is no limitation for assembling R classifiers in the ensemble model. Third, we did not include clinical features like age and sex in the radiomics model construction. Based on previous radiomics research studying tumor grading and survival analysis of soft tissue sarcoma [13, 15, 33], age could be an influential feature for generating and ideal radiomics model classifying malignant soft tissue tumors. However, the focus of our study was comparing the performance of radiomics models with different feature combination methods, for which inclusion of clinical features was not crucial. Fourth, there are diverse histologic subtypes in STTs, and a certain type of tumor consists of more than 50% of the benign tumors in our study. It might be difficult to find a universally effective radiomics model for malignancy differentiation, and the diagnostic performance of generated radiomics models in STT differentiation is inferior to that of radiomics models in other tumors. However, this composition reflects the real incidence in clinical practice.
Conclusion
In this study, we compared the diagnostic performance of radiomics models with different feature assembling methods. The radiomics model constructed with pooled features extracted from multi-sequence MRI showed better performance compared to models constructed with single-sequence MRI. However, the diagnostic performance of radiomics models constructed with multi-sequence MRI showed no prominent improvement as more sequence features were assembled. Finally, when incorporating features from multiple sequences, the ensemble model showed better diagnostic performance than conventional model using pooling of combined features as a single set.
Supporting information
S2 Appendix. Hyperparameters of type-1 radiomics model using random forest classifier.
https://doi.org/10.1371/journal.pone.0286417.s003
(DOCX)
S3 Appendix. Parameters in GridSearchCV implementation in type-2 radiomics model.
https://doi.org/10.1371/journal.pone.0286417.s004
(DOCX)
S4 Appendix. Hyperparameter of type-2 ensemble model, determined by GridSearchCV implementation.
https://doi.org/10.1371/journal.pone.0286417.s005
(DOCX)
References
- 1. Wu JS, Hochman MG. Soft-Tissue Tumors and Tumorlike Lesions: A Systematic Imaging Approach. Radiology. 2009;253(2):297–316. pmid:19864525.
- 2. van Vliet M, Kliffen M, Krestin GP, van Dijke CF. Soft tissue sarcomas at a glance: clinical, histological, and MR imaging features of malignant extremity soft tissue tumors. European Radiology. 2009;19(6):1499–511. pmid:19266204
- 3. Boudabbous S, Hamard M, Saiji E, Gorican K, Poletti P-A, Becker M, et al. What morphological MRI features enable differentiation of low-grade from high-grade soft tissue sarcoma? BJR|Open. 2022;4(1):20210081. pmid:36105415
- 4. Crombé A, Marcellin PJ, Buy X, Stoeckle E, Brouste V, Italiano A, et al. Soft-Tissue Sarcomas: Assessment of MRI Features Correlating with Histologic Grade and Patient Outcome. Radiology. 2019;291(3):710–21. Epub 2019/04/10. pmid:30964422.
- 5. Kind M, Stock N, Coindre JM. Histology and imaging of soft tissue sarcomas. European Journal of Radiology. 2009;72(1):6–15. https://doi.org/10.1016/j.ejrad.2009.05.023. pmid:19835758
- 6. Ahlawat S, Fritz J, Morris CD, Fayad LM. Magnetic resonance imaging biomarkers in musculoskeletal soft tissue tumors: Review of conventional features and focus on nonmorphologic imaging. J Magn Reson Imaging. 2019;50(1):11–27. Epub 20190221. pmid:30793420.
- 7. De Schepper AM, De Beuckeleer L, Vandevenne J, Somville J. Magnetic resonance imaging of soft tissue tumors. European Radiology. 2000;10(2):213–23. pmid:10663750
- 8. Subhawong TK, Jacobs MA, Fayad LM. Diffusion-weighted MR Imaging for Characterizing Musculoskeletal Lesions. RadioGraphics. 2014;34(5):1163–77. pmid:25208274.
- 9. Suzuki C, Maeda M, Matsumine A, Matsubara T, Taki W, Maier SE, et al. Apparent diffusion coefficient of subcutaneous epidermal cysts in the head and neck comparison with intracranial epidermoid cysts. Acad Radiol. 2007;14(9):1020–8. pmid:17707308.
- 10. Xu W, Hao D, Hou F, Zhang D, Wang H. Soft Tissue Sarcoma: Preoperative MRI-Based Radiomics and Machine Learning May Be Accurate Predictors of Histopathologic Grade. American Journal of Roentgenology. 2020;215(4):963–9. pmid:32755226
- 11. Wang H, Chen H, Duan S, Hao D, Liu J. Radiomics and Machine Learning With Multiparametric Preoperative MRI May Accurately Predict the Histopathological Grades of Soft Tissue Sarcomas. Journal of Magnetic Resonance Imaging. 2020;51(3):791–7. https://doi.org/10.1002/jmri.26901. pmid:31486565
- 12. Wang H, Zhang J, Bao S, Liu J, Hou F, Huang Y, et al. Preoperative MRI-Based Radiomic Machine-Learning Nomogram May Accurately Distinguish Between Benign and Malignant Soft-Tissue Lesions: A Two-Center Study. J Magn Reson Imaging. 2020;52(3):873–82. Epub 20200229. pmid:32112598.
- 13. Yan R, Hao D, Li J, Liu J, Hou F, Chen H, et al. Magnetic Resonance Imaging-Based Radiomics Nomogram for Prediction of the Histopathological Grade of Soft Tissue Sarcomas: A Two-Center Study. J Magn Reson Imaging. 2021;53(6):1683–96. Epub 20210218. pmid:33604955.
- 14. Yue Z, Wang X, Yu T, Shang S, Liu G, Jing W, et al. Multi-parametric MRI-based radiomics for the diagnosis of malignant soft-tissue tumor. Magn Reson Imaging. 2022;91:91–9. Epub 20220504. pmid:35525523.
- 15. Peeken JC, Spraker MB, Knebel C, Dapper H, Pfeiffer D, Devecka M, et al. Tumor grading of soft tissue sarcomas using MRI-based radiomics. EBioMedicine. 2019;48:332–40. pmid:31522983
- 16. Fields BKK, Demirjian NL, Cen SY, Varghese BA, Hwang DH, Lei X, et al. Predicting Soft Tissue Sarcoma Response to Neoadjuvant Chemotherapy Using an MRI-Based Delta-Radiomics Approach. Molecular Imaging and Biology. 2023. pmid:36695966
- 17. Yang Y, Zhou Y, Zhou C, Zhang X, Ma X. MRI-Based Computer-Aided Diagnostic Model to Predict Tumor Grading and Clinical Outcomes in Patients With Soft Tissue Sarcoma. J Magn Reson Imaging. 2022. Epub 20220318. pmid:35303756.
- 18. Lee SY, Jee WH, Jung JY, Park MY, Kim SK, Jung CK, et al. Differentiation of malignant from benign soft tissue tumours: use of additive qualitative and quantitative diffusion-weighted MR imaging to standard MR imaging at 3.0 T. Eur Radiol. 2016;26(3):743–54. Epub 2015/06/18. pmid:26080796.
- 19. Polikar R. Ensemble based systems in decision making. IEEE Circuits and Systems Magazine. 2006;6(3):21–45.
- 20. Ju C, Bibaut A, van der Laan M. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics. 2018;45(15):2800–18. pmid:31631918
- 21. Nagata S, Nishimura H, Uchida M, Sakoda J, Tonan T, Hiraoka K, et al. Diffusion-weighted imaging of soft tissue tumors: usefulness of the apparent diffusion coefficient for differential diagnosis. Radiat Med. 2008;26(5):287–95. Epub 20080727. pmid:18661213.
- 22. Lee SE, Jung J-Y, Nam Y, Lee S-Y, Park H, Shin S-H, et al. Radiomics of diffusion-weighted MRI compared to conventional measurement of apparent diffusion-coefficient for differentiation between benign and malignant soft tissue tumors. Scientific Reports. 2021;11(1):15276. pmid:34315971
- 23. Khoo MM, Tyler PA, Saifuddin A, Padhani AR. Diffusion-weighted imaging (DWI) in musculoskeletal MRI: a critical review. Skeletal Radiol. 2011;40(6):665–81. Epub 20110212. pmid:21311884.
- 24. Yushkevich PA, Piven J, Hazlett HC, Smith RG, Ho S, Gee JC, et al. User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage. 2006;31(3):1116–28. Epub 20060320. pmid:16545965.
- 25. van Griethuysen JJM, Fedorov A, Parmar C, Hosny A, Aucoin N, Narayan V, et al. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017;77(21):e104–e7. pmid:29092951; PubMed Central PMCID: PMC5672828.
- 26. Tibshirani R. Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society Series B (Methodological). 1996;58(1):267–88.
- 27. Breiman L. Random Forests. Machine Learning. 2001;45(1):5–32. :1010933404324.
- 28. Zou KH, Warfield SK, Bharatha A, Tempany CM, Kaus MR, Haker SJ, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol. 2004;11(2):178–89. pmid:14974593; PubMed Central PMCID: PMC1415224.
- 29. Brunese L, Mercaldo F, Reginelli A, Santone A. An ensemble learning approach for brain cancer detection exploiting radiomic features. Computer Methods and Programs in Biomedicine. 2020;185:105134. https://doi.org/10.1016/j.cmpb.2019.105134. pmid:31675644
- 30. Farahani FV, Ahmadi A, Zarandi MHF, editors. Lung nodule diagnosis from CT images based on ensemble learning. 2015 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB); 2015 12–15 Aug. 2015.
- 31. Shayesteh SP, Alikhassi A, Fard Esfahani A, Miraie M, Geramifar P, Bitarafan-rajabi A, et al. Neo-adjuvant chemoradiotherapy response prediction using MRI based ensemble learning method in rectal cancer patients. Physica Medica. 2019;62:111–9. https://doi.org/10.1016/j.ejmp.2019.03.013. pmid:31153390
- 32. Singh A, Kumar R, Tiwari A-K. Prediction of Alzheimer’s Using Random Forest with Radiomic Features. Computer Systems Science and Engineering. 2023;45(1):513–30.
- 33. Peeken JC, Neumann J, Asadpour R, Leonhardt Y, Moreira JR, Hippe DS, et al. Prognostic Assessment in High-Grade Soft-Tissue Sarcoma Patients: A Comparison of Semantic Image Analysis and Radiomics. Cancers (Basel). 2021;13(8). Epub 20210416. pmid:33923697; PubMed Central PMCID: PMC8073388.