https://orcid.org/0000-0002-5878-5889
Figures
Abstract
Objectives
In this meta-analysis study, the main objective was to determine the accuracy of S-detect in effectively distinguishing malignant thyroid nodules from benign thyroid nodules.
Methods
We searched the PubMed, Cochrane Library, and CBM databases from inception to August 1, 2021. Meta-analysis was conducted using STATA version 14.0 and Meta-Disc version 1.4 softwares. We calculated summary statistics for sensitivity (Sen), specificity (Spe), positive and negative likelihood ratio (LR+/LR−), diagnostic odds ratio(DOR), and receiver operating characteristic (SROC) curves. Cochran’s Q-statistic and I2 test were used to evaluate potential heterogeneity between studies. A sensitivity analysis was performed to evaluate the influence of single studies on the overall estimate. We also performed meta-regression analyses to investigate the potential sources of heterogeneity.
Results
In this study, a total of 17 studies meeting the requirements of the standard were used. The number of benign and malignant nodules analyzed and evaluated in this paper was 1595 and 1118 respectively. This paper mainly completes the required histological confirmation through s-detect. The pooled Sen and pooled Spe were 0.87 and 0.74, respectively, (95%CI = 0.84–0.89) and (95%CI = 0.66–0.81). Furthermore, the pooled LR+ and negative LR− were determined to be 3.37 (95%CI = 2.53–4.50) and 0.18 (95%CI = 0.15–0.21), respectively. The experimental results showed that the pooled DOR of thyroid nodules was 18.83 (95% CI = 13.21–26.84). In addition, area under SROC curve was determined to be 0.89 (SE = 0.0124). It should be pointed out that there is no evidence of bias (i.e. t = 0.25, P = 0.80).
Citation: Zhong L, Wang C (2022) Diagnostic accuracy of S-Detect in distinguishing benign and malignant thyroid nodules: A meta-analysis. PLoS ONE 17(8): e0272149. https://doi.org/10.1371/journal.pone.0272149
Editor: Nguyen Quoc Khanh Le, Taipei Medical University, TAIWAN
Received: August 17, 2021; Accepted: July 13, 2022; Published: August 5, 2022
Copyright: © 2022 Zhong, Wang. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In the past period, the incidence rate of thyroid related diseases is increasing, which is closely related to the progress in biological characteristics and ultrasound diagnosis technology [1]. For the clinical diagnosis of this disease, ultrasonic examination method is usually selected, and its main advantage is that it has very high sensitivity [2]. In recent years, various new ultrasonic imaging technologies, such as ultrasound elastography, contrast-enhanced ultrasound, and superb micro-vascular imagine, have developed rapidly, which has greatly promoted their application in the field of medicine [3–5].
However, its accuracy is usually affected by the professional ability of doctors. Computer-aided diagnosis (CAD) technology is a popular topic in artificial intelligence and modern medical research [6]. Ultrasonic S-Detect technology is typically a new diagnosis method that uses a convolutional neural network deep learning algorithm to evaluate thyroid nodules according to the Ti-RADS dictionary, at present, it is the most widely used CAD diagnosis system in this field. The deep learning model is used to automatically detect and analyze the boundary, shape, internal echo, and other nodule information, overcome the interference of human factors, and objectively judge benign and malignant nodules [7]. A large number of research data show that if this technology is applied to the clinical diagnosis of thyroid nodules, it will obtain good accuracy [8–10]. The problem to be pointed out is that the application time of this technology is still relatively short, so there are some different views. In addition, there are some differences in the clinical results. For example, the Spe obtained by Yoo et al. is 88% [9], but the value obtained by other team is 41%, which is quite different [11].
According to the existing data, it is found that there are significant differences in the reported Spe. At present, there is no meta-analysis study on its application in the diagnosis of thyroid cancer. Therefore, this paper will focus on in-depth analysis and research in this aspect.
Methods
This study was conducted in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and MetaAnalyses) guidelines, the meta-analysis was not registered.
Literature search
We searched PubMed, Cochrane Library, and CBM databases from inception to August 1, 2021. The following keywords and MeSH terms were used: ["thyroid cancer" or "thyroid neoplasm" or "thyroid tumor" or "thyroid nodule "] and [“S-Detect” or “smart detect” or “artificial Intelligence” or “computer-aided diagnosis”]. We also performed a manual search to identify additional potential articles.
Selection criteria
The following four criteria were required for each study: (1) the study design must be a clinical cohort study or diagnostic test; (2) the study must relate to the accuracy of S-Detect for the differential diagnosis of benign and malignant thyroid nodules,and the final assessments from S-Detect were in dichotomized form: possibly benign and possibly malignant; (3) all thyroid nodules were histologically confirmed; and(4) published data in the fourfold (2×2) tables must be sufficient. If the study did not meet all the inclusion criteria, it was excluded. The most recent publication or publication with the largest sample size was included when the authors had published several studies using the same subjects.
Data extraction
Relevant data were systematically extracted from all included studies by two researchers using a standardized form. The researchers collected the following data: first author’s surname, publication year, language of publication, study design, sample size, number of lesions, source of the subjects, "gold standard". True positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) in the fourfold (2 × 2) tables were also collected.
Quality assessment
Methodological quality was independently assessed by two researchers using the Quality Assessment of Studies of Diagnostic Accuracy Studies (QUADAS) tool [12]. The QUADAS criteria include 14 assessment items. Each item was scored as "yes" (2), "no" (0), or "unclear"(1). The QUADAS score ranged from 0 to 28, and a score ≥ 22 indicated good quality.
Statistical analysis
STATA version 14.0 (Stata Corp, College Station, TX, USA) and Meta-Disc version 1.4 (Universidad Complutense, Madrid, Spain) software were used for meta-analysis. We calculated the pooled summary statistics for sensitivity (Sen), specificity (Spe), positive and negative likelihood ratio (LR+/LR−), and diagnostic odds ratio (DOR) with 95%confidence intervals (CIs). The post-test probabilities were calculated by the LR+and LR− and plotted on a Fagan nomogram. A summary receiver operating characteristic (SROC) curve and corresponding area under the curve (AUC) were obtained. The threshold effect was assessed using Spearman’s correlation coefficients. Cochran’s Q-statistic and I2 test were used to evaluate potential heterogeneity between studies. If significant heterogeneity was detected(Q test P<0.05, or I test>50%), a random-effects model or fixed-effects model was used. We also performed meta-regression analyses to investigate the potential sources of heterogeneity. A sensitivity analysis was performed to evaluate the influence of single studies on the overall estimate. We used Begger’s funnel plots and Egger’s linear regression tests to investigate publication bias.
Results
Characteristics of included studies
Initially, the search keywords were used to identify 40 articles. We reviewed the contents of the title and abstract of the article, and then listed 17 of its value. On this basis, we further reviewed the integrity of the content and data of the paper, and then excluded 6 papers. In the current meta-analysis, a total of 17 studies were used [7–10, 13–25]. In Fig 1 of this paper, the selection method and main steps of the article are described in detail. The numbers of benign and malignant thyroid nodules analyzed and evaluated in this paper were 1595 and 1118 respectively. In S1 Table, the research methods and basic characteristics adopted are shown in detail. We found that QUADAS scores were ≥ 24.
Quantitative data synthesis
We chose the random effect model, the main reason is that there is no very significant difference between different studies. We also conducted sensitivity analysis and found that the results were not affected (Fig 2). It is found that the pooled Sen and Spe were 0.87 (95%CI = 0.84–0.89) and 0.74 (95%CI = 0.66–0.81) respectively (Fig 3). The experimental results showed that there was no obvious correlation between the specificity and the sensitivity (r = 0.289, P = 0.260), which means that no threshold effect exists. It can be seen that the negative LR- and pooled LR+ were 0.18 (95%CI = 0.15–0.21) and 3.37 (95%CI = 2.53–4.50) respectively (Fig 4). The study found that after using S-Detect for this diagnosis, the DOR value was 18.83 (95% CI = 13.21–26.84) (Fig 5). In addition, the area below the SROC curve was equal to 0.89 (SE = 0.0124) (Fig 6). Meta-regression analysis results confirmed that no factor could explain the potential sources of heterogeneity (S2 Table). Through analysis, no evidence of publication bias was observed (in Fig 7). After egger’s test, there was also no evidence of publication bias (t = 0.25, P = 0.80). According to the research on Fagan diagram, when the pre-test probabilities were 25%, 50% and 75%, the negative and positive post-test probabilities were 6%, 15% and 35%, as well as 53%, 77% and 91%, respectively (Fig 8).
They did not significantly affect the results.
There is no publication bias observed.
The Fagan diagram analysis results obtained in this paper: a)—c) pre inspection probability is 25%, 50% and 75% respectively.
Discussion
As a very common disease, the study of thyroid nodule has attracted extensive attention. The related identification will affect the final clinical decision. At present, high-resolution ultrasound technology has been widely used in this field [25]. In recent years, the incidence rate of US has increased with the increasing incidence of the disease. In fact, this imaging method has obvious performance advantages. With the rapid development of artificial intelligence (AI) technology, it also drives the continuous improvement of this field. For example, "S-Detect" technology is a typical representative. This technology is helpful to better complete the morphological analysis of the disease, so as to promote the formulation of its clinical treatment plan [13–18]. However, only a few articles have reported its diagnostic performance for thyroid cancer, and they were mainly published by Korean researchers. To further study the diagnostic value of S-detect in thyroid ultrasound, more validation sets from different countries are required. Therefore, Our main purpose is to promote the better application of S-Detect in the field of clinical diagnosis of thyroid tumors.
This paper mainly analyzes the main performance and effect of s-detect in the analysis and diagnosis of thyroid nodules. We studied 17 subjects and evaluated 1595 benign and 1118 malignant nodules, respectively. The data obtained show that its comprehensive Sen value was 0.87, Spe value was 0.74 and DOR value was 18.83. The above results show that S-Detect has high accuracy in the field of clinical diagnosis of thyroid nodules, so it is a very good diagnostic tool. In general, we believe that threshold effect is a significant change that often occurs after a phenomenon has exceeded the relevant range. According to our analysis, we can find that there is no very obvious internal relationship between Sen and Spe, that is, there is no sufficient evidence related to the threshold effect. In the current study, we found no evidence of publication bias. Therefore, the current research confirms that S-Detect has good accuracy, and the result is consistent with the original report.
S-Detect has good accuracy in the field of clinical diagnosis of thyroid nodules, but there are still some limitations in this study. Firstly, the quality of the sample needs to be further improved, because its accuracy needs to be further improved. Secondly, the inclusion of studies with histological confirmation only and the retrospective nature of the meta-analysis could have led to subject selection bias. Thirdly, we did not find the information about registration and version of the used S-Detect software, which may be a potential source of heterogeneity. In particular, it should be pointed out that the current research objects are basically from Asian countries, which will affect the effectiveness of the current results.
According to the results of this meta-analysis, S-Detect can accurately distinguish malignant thyroid nodules from benign thyroid nodules. It is also an important auxiliary means of conventional ultrasonic examination technology. Because the research process is limited by a series of factors, we need to explore this problem in depth.
Supporting information
S1 Table. Baseline characteristics and methodological quality of all included studies.
https://doi.org/10.1371/journal.pone.0272149.s002
(DOCX)
S2 Table. Meta-regression analyses of potential source of heterogeneity.
https://doi.org/10.1371/journal.pone.0272149.s003
(DOCX)
References
- 1. Sapunar ZJ, Ferrer RP. Epidemiología del cáncer de tiroides en un Instituto Oncológico. Efecto de las nuevas recomendaciones clínicas [Incidence of thyroid cancer in an oncological clinic of Santiago, Chile]. Rev Med Chil. 2020; 148: 573–81. pmid:33399750
- 2. Kong W, Yue X, Ren J, et al. A comparative analysis of diffusion-weighted imaging and ultrasound in thyroid nodules. BMC Med Imaging. 2019; 19: 92. pmid:31752728
- 3. Zhu C, Zhong L, Lin M, Tian C, Wang C. The value of TI-RADS combined with superb micro-vascular imagine in distinguishing benign and malignant thyroid nodules: A meta-analysis. PLoS One. 2022;17(1):e0261521. pmid:35041691
- 4. Abdelgawad M, Shalaby H, Akkera M, Rashad Y, Grace L, Ibraheem K, et al. Ultrasound Elastography Predicts Thyroid Nodule Volume Reduction Rate After Percutaneous Ethanol Ablation. Am Surg. 2021;87(4):581–7. pmid:33131289
- 5. Sorrenti S, Dolcetti V, Fresilli D, Del Gaudio G, Pacini P, Huang P, et al. The Role of CEUS in the Evaluation of Thyroid Cancer: From Diagnosis to Local Staging. J Clin Med. 2021;10(19):4559. pmid:34640574
- 6. Ahmad OF, Soares AS, Mazomenos E, Brandao P, Vega R, Seward E, et al. Artificial intelligence and computer-aided diagnosis in colonoscopy: current evidence and future directions. Lancet Gastroenterol Hepatol. 2019; 4: 71–80. pmid:30527583
- 7. Choi YJ, Baek JH, Park HS, Shim WH, Kim TY, Shong YK, et al. A Computer-Aided Diagnosis System Using Artificial Intelligence for the Diagnosis and Characterization of Thyroid Nodules on Ultrasound: Initial Clinical Assessment. Thyroid. 2017; 27: 546–52. pmid:28071987
- 8. Han M, Ha EJ, Park JH. Computer-Aided Diagnostic System for Thyroid Nodules on Ultrasonography: Diagnostic Performance Based on the Thyroid Imaging Reporting and Data System Classification and Dichotomous Outcomes. AJNR Am J Neuroradiol. 2021; 42: 559–65. pmid:33361374
- 9. Yoo YJ, Ha EJ, Cho YJ, Kim HL, Han M, Kang SY. Computer-Aided Diagnosis of Thyroid Nodules via Ultrasonography: Initial Clinical Experience. Korean J Radiol. 2018; 19:665–72. pmid:29962872
- 10. Chung SR, Baek JH, Lee MK, Ahn Y, Choi YJ, Sung TY, et al. Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists. Korean J Radiol. 2020; 21: 369–76. pmid:32090529
- 11. Whiting PF, Weswood ME, Rutjes AW. Evaluation of quadas, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol. 2006; 6: 9. pmid:16519814
- 12. Boyuan Xing, Yun Zhao, Jie Ping. Diagnostic value of ultrasonic s-detect technique in benign and malignant thyroid TI-RADS class 4 nodules. Chinese Journal of ultrasound medicine. 2021; 37: 497–501.
- 13. Yu Bi, Mei Peng, Mingdi Fang. Value of ti-rads combined with s-detect in the diagnosis of thyroid nodules. Chinese Journal of ultrasound medicine. 2021; 37: 364–6.
- 14. Qian Li, Chunli Liu, Lanweil Guo. Application value of s-detect technology based on deep learning model in differential diagnosis of benign and malignant thyroid nodules. Journal of Zhengzhou University. 2021; 56: 285–9.
- 15. Mingdi Fang, Mei Peng, Yu Bi. Diagnostic value of artificial intelligence s-detect combined with calcification characteristics in thyroid nodules. Chinese Journal of medical ultrasound. 2021; 18: 177–81.
- 16. Chen , sun Pengfei, Guo Jun. Diagnostic value of artificial intelligence in benign and malignant thyroid nodules. Chinese Journal of ultrasound medicine. 2020; 36: 585–8.
- 17. Hong Han, Qing Yu, Lei Zhao. Preliminary application of ultrasound s-detect technology in the diagnosis of thyroid tumors. Chinese Journal of ultrasound imaging. 2018; 27: 28–30.
- 18. Szczepanek-Parulska E, Wolinski K, Dobruch-Sobczak K, Antosik P, Ostalowska A, Krauze A, et al. S-Detect Software vs. EU-TIRADS Classification: A Dual-Center Validation of Diagnostic Performance in Differentiation of Thyroid Nodules. J Clin Med. 2020; 9: 2495. pmid:32756510
- 19. Wei Q, Zeng SE, Wang LP, Yan YJ, Wang T, Xu JW, et al. The value of S-Detect in improving the diagnostic performance of radiologists for the differential diagnosis of thyroid nodules. Med Ultrason. 2020; 22: 415–23. pmid:32905560
- 20. Barczyński M, Stopa-Barczyńska M, Wojtczak B, Czarniecka A, Konturek A. Clinical validation of S-DetectTM mode in semi-automated ultrasound classification of thyroid lesions in surgical office. Gland Surg. 2020; 9: S77–S85. pmid:32175248
- 21. Kim HL, Ha EJ, Han M. Real-World Performance of Computer-Aided Diagnosis System for Thyroid Nodules Using Ultrasonography. Ultrasound Med Biol. 2019; 45: 2672–8. pmid:31262524
- 22. Xia S, Yao J, Zhou W, Dong Y, Xu S, Zhou J, et al. A computer-aided diagnosing system in the evaluation of thyroid nodules-experience in a specialized thyroid center. World J Surg Oncol. 2019; 17: 210. pmid:31810469
- 23. Molnár K, Kálmán E, Hári Z, Giyab O, Gáspár T, Rucz K, et al. False-Positive Malignant Diagnosis of Nodule Mimicking Lesions by Computer-Aided Thyroid Nodule Analysis in Clinical Ultrasonography Practice. Diagnostics (Basel). 2020; 10: 378. pmid:32517227
- 24. Fresilli D, Grani G, De Pascali ML, Alagna G, Tassone E, Ramundo V, et al. Computer-aided diagnostic system for thyroid nodule sonographic evaluation outperforms the specificity of less experienced examiners. J Ultrasound. 2020; 23: 169–74. pmid:32246401
- 25. Gharib H, Papini L, Paschke R, Duick DS, Valcavi R, Hegedüs L, et al. American association of clinical endocrinologists, association medici endocrinologi, and european thyroid association medical guidelines for clinical practice of thyroid nodules. Endocr Prac. 2010; 16: 468–75.