https://orcid.org/0000-0002-1529-5097
Figures
Abstract
Breast ultrasound provides a first-line evaluation for breast masses, but the majority of the world lacks access to any form of diagnostic imaging. In this pilot study, we assessed the combination of artificial intelligence (Samsung S-Detect for Breast) with volume sweep imaging (VSI) ultrasound scans to evaluate the possibility of inexpensive, fully automated breast ultrasound acquisition and preliminary interpretation without an experienced sonographer or radiologist. This study was conducted using examinations from a curated data set from a previously published clinical study of breast VSI. Examinations in this data set were obtained by medical students without prior ultrasound experience who performed VSI using a portable Butterfly iQ ultrasound probe. Standard of care ultrasound exams were performed concurrently by an experienced sonographer using a high-end ultrasound machine. Expert-selected VSI images and standard of care images were input into S-Detect which output mass features and classification as “possibly benign” and “possibly malignant.” Subsequent comparison of the S-Detect VSI report was made between 1) the standard of care ultrasound report by an expert radiologist, 2) the standard of care ultrasound S-Detect report, 3) the VSI report by an expert radiologist, and 4) the pathological diagnosis. There were 115 masses analyzed by S-Detect from the curated data set. There was substantial agreement of the S-Detect interpretation of VSI among cancers, cysts, fibroadenomas, and lipomas to the expert standard of care ultrasound report (Cohen’s κ = 0.73 (0.57–0.9 95% CI), p<0.0001), the standard of care ultrasound S-Detect interpretation (Cohen’s κ = 0.79 (0.65–0.94 95% CI), p<0.0001), the expert VSI ultrasound report (Cohen’s κ = 0.73 (0.57–0.9 95% CI), p<0.0001), and the pathological diagnosis (Cohen’s κ = 0.80 (0.64–0.95 95% CI), p<0.0001). All pathologically proven cancers (n = 20) were designated as “possibly malignant” by S-Detect with a sensitivity of 100% and specificity of 86%. Integration of artificial intelligence and VSI could allow both acquisition and interpretation of ultrasound images without a sonographer and radiologist. This approach holds potential for increasing access to ultrasound imaging and therefore improving outcomes related to breast cancer in low- and middle- income countries.
Author summary
Breast cancer poses an urgent crisis in low- and middle-income countries, but the majority of the world lacks access to medical imaging for assessment of the breast. The use of volume sweep imaging (VSI) allows image acquisition of palpable breast lumps without an experienced sonographer. Images of breast lumps from VSI acquisitions can be interpreted by existing artificial intelligence to allow automatic and rapid diagnosis in areas without sonographers and radiologists. In this study, we fed expert-selected VSI images obtained by medical students with no prior ultrasound experience into an artificial intelligence (S-Detect) to test the potential for automatic diagnosis of palpable breast lumps. We found high diagnostic accuracy demonstrating a strong proof of concept for the potential of automatic diagnosis of palpable breast lumps without a radiologist or sonographer. Integration of artificial intelligence and VSI could increase access to diagnostic imaging around the world with the goal of improving global health.
Citation: Marini TJ, Castaneda B, Parker K, Baran TM, Romero S, Iyer R, et al. (2022) No sonographer, no radiologist: Assessing accuracy of artificial intelligence on breast ultrasound volume sweep imaging scans. PLOS Digit Health 1(11): e0000148. https://doi.org/10.1371/journal.pdig.0000148
Editor: Ludwig Christian Giuseppe Hinske, Ludwig-Maximilians-Universität München Adolf-Butenandt-Institut: Ludwig-Maximilians-Universitat Munchen Biomedizinisches Centrum Munchen, GERMANY
Received: February 23, 2022; Accepted: October 21, 2022; Published: November 23, 2022
Copyright: © 2022 Marini et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available from the authors upon reasonable request. As individual data could identify patients this is not uploaded to a public repository as an additional safeguard on patient privacy.
Funding: This study was funded by the University of Rochester Fischer Fund. Funding was received by TM and AO. Samsung provided access to S-Detect. They provided salaries to their employees ZH and MHP. Butterfly provided access to their cloud service free of charge for the study. SR's time was supported by CONCYTEC (174-2020-FONDE-CYT-PUCP). Funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: BC has financial stake in Medical Innovation and Technology. This company seeks to bring ultrasound into rural areas. ZH is an employee of Samsung Medison. MHP is an employee of Samsung Electronics.
Introduction
Breast cancer is increasingly recognized as an urgent public health crisis in low- and middle-income countries (LMICs) secondary to a projected nearly 60% increase in both incidence and mortality of breast cancer over the last 20 years [1]. In contrast, in populations with access to screening mammography, there has been an observed decrease in breast cancer mortality [2–4]. It has been estimated that the majority of the nearly 2 million new cases of breast cancer every year will occur in LMICs [5]. The poor outcomes in LMICs owe, in part, to significant delays in breast cancer diagnosis (sometimes over 400 days) resulting in advanced disease at presentation [6]. Although diagnostic imaging is essential in the diagnosis of breast cancer, the majority of people in the world are thought to lack access to any diagnostic imaging [7,8]. Presumably, increased access to diagnostic imaging would be a first step in decreasing delays to diagnosis of breast cancer and improved outcomes.
Screening mammography is typically not an option in LIMCs for a variety of reasons [9,10]. Breast ultrasound, however, is a portable and low-cost imaging modality ideal for use in LMICs. In contradistinction to screening mammography which is performed on asymptomatic patients, ultrasound is typically used as a diagnostic technique to evaluate a patient presenting with a symptom, such as a palpable lump. Ultrasound and mammography are complementary imaging modalities each with their own strengths and weaknesses [11]. Since screening is often not available in LMICs, most patients with cancer will present with a palpable lump which is amenable to examination with ultrasonography [12]. In general, ultrasound is considered first-line evaluation for palpable breast abnormalities [13]. Despite its cost-effective and portable nature, ultrasound’s deployment has been limited by a lack of available ultrasound specialists for image acquisition and interpretation [8,14].
Volume sweep imaging (VSI) is an imaging technique that circumvents the problem of a lack of trained sonographers via the use of a standardized imaging protocol based on sweeps of the ultrasound probe over a target region in relation to external body landmarks [15]. VSI can be learned in a few hours by individuals without prior ultrasound training making ultrasound image acquisition readily accessible [16,17]. VSI has been combined with teleultrasound with encouraging clinical results for lung, obstetric, thyroid, and right upper quadrant scanning indications in Peru [15,18–21]. Clinical studies of breast and lung VSI have also shown promising results [22,23]. In the study of breast VSI, there was 97.6% agreement between VSI and standard of care ultrasound on mass presence and 87% agreement on BI-RADS assessments [22]. These results suggest that breast VSI could be deployed to rural areas to increase access to breast imaging. Nonetheless, a lack of interpreting providers and constraints on internet access bottleneck the impact of VSI as a radiologist is still needed to interpret the images.
Automatic interpretation of VSI with artificial intelligence would allow the performance of ultrasound imaging without any sonographer or radiologist. This has already been proposed and successfully tested for obstetric indications [24,25]. Such an automated system for breast scanning would offer a significant opportunity to increase access to breast imaging around the world. S-Detect (Samsung Medison, Seoul, Korea) is an artificial intelligence system developed as a tool for diagnostic analysis of breast masses [26]. It employs convolutional neural networks to arrive at a suggested diagnosis for a mass from a radiologist-detected lesion. S-Detect has shown excellent results in clinical testing (sensitivity of 85%, specificity of 95.4%, and diagnostic accuracy of 92.1%) highlighting its potential application in clinical practice [27]. Results consistently suggest S-Detect can perform at the same level or better than radiologists [27–32]. Therefore, this system would have significant potential to increase access to accurate diagnostic interpretations in the absence of a radiologist.
In this pilot study, we tested the performance of S-Detect on VSI ultrasound imaging performed by medical students without prior ultrasound experience with a portable ultrasound probe. The images tested were those obtained from the prior clinical study of breast VSI [22]. The goal of this study was to lay the theoretical groundwork for a fully automated system incorporating VSI and S-Detect. Comparison of the S-Detect VSI report was made to 1) the standard of care ultrasound report by an expert radiologist, 2) the standard of care ultrasound S-Detect report, 3) the VSI report by an expert radiologist, and 4) the pathological diagnosis. To our knowledge, S-Detect has never been previously tested on images obtained from a portable ultrasound device. Furthermore, previously, concerns had been raised regarding the use of S-Detect in cases where a sonographer is inexperienced or the ultrasound imaging was suboptimal, highlighting the importance of this investigation [26]. We hypothesized that S-Detect results between VSI and standard of care ultrasound would be similar given our experience with VSI and the literature surrounding S-Detect.
Results
There were 115 masses analyzed by S-Detect from the curated data set (n = 13 abscesses, n = 20 cancers, n = 23 cysts, n = 6 fat necrosis, n = 24 fibroadenomas, n = 6 lipomas, n = 5 miscellaneous diagnoses, n = 8 sebaceous cysts, and n = 10 unknown) [22]. Patient demographics are seen in Table 1. Example S-Detect interpretations of a fibroadenoma, a cyst, and a cancer are seen in Figs 1–3. Among cancers, cysts, fibroadenomas, and lipomas, there was 87.7% agreement on diagnosis to both expert standard of care interpretation and expert VSI interpretation (Cohen’s κ = 0.73 (0.57–0.9 95% CI), p<0.0001). Within these same mass categories, there was 90.4% agreement on diagnosis between S-Detect interpretation of VSI and S-Detect interpretation of standard of care imaging (Cohen’s κ = 0.79 (0.65–0.94 95% CI), p<0.0001) and 90.5% agreement between the S-Detect VSI interpretation and the pathological diagnosis (Cohen’s κ = 0.8 (0.64–0.95 95% CI), p<0.0001). Table 2 shows full diagnostic agreement across different categories.
B-mode ultrasound images of a fibroadenoma are shown with their respective mass segmentations and S-Detect produced reports. The standard of care image was obtained by a certified expert breast sonographer using a General Electric Logiq E10 ultrasound machine. The VSI image was obtained by a medical student without prior medical experience after less than 2 hours of training on a portable Butterfly iQ ultrasound probe. S-Detect results between standard of care and VSI are identical apart from standard of care echogenicity reported as isoechoic (a trivial difference). The S-Detect results for VSI were identical to expert interpretation of standard of care and VSI.
B-mode ultrasound images of a cyst are shown with their respective mass segmentations and S-Detect produced reports. The standard of care image was obtained by a certified expert breast sonographer using a Philips Epiq 7G ultrasound machine. The VSI image was obtained by a medical student without prior medical experience after less than 2 hours of training on a portable Butterfly iQ ultrasound probe. S-Detect results between standard of care and VSI are identical. The S-Detect results were also identical to expert interpretation of standard of care and VSI outside of mass shape which was called round on expert interpretation (a trivial difference).
B-mode ultrasound images of invasive ductal carcinoma are shown with their respective mass segmentations and S-Detect produced reports. The standard of care image was obtained by a certified expert breast sonographer using a Philips Epiq 7G ultrasound machine. The VSI image was obtained by a medical student without prior medical experience after less than 2 hours of training on a portable Butterfly iQ ultrasound probe. S-Detect results between standard of care and VSI are similar with ultimate agreement on the “possibly malignant” interpretation. The S-Detect results were also similar to expert interpretation of standard of care and VSI with ultimate agreement on the “possibly malignant” characterization. The original BI-RADS assessment on both VSI and standard of care imaging expert interpretation was 4 (biopsy warranted). There was complete agreement across expert interpretation and S-Detect on microlobulated margins.
Categorical variables are summarized as n (proportion (95% CI)), while continuous variables are summarized as mean±standard deviation.
Table 3 shows the sensitivity and specificity for malignancy in relation to the pathological diagnosis. Among cancer, cysts, fibroadenomas, and lipomas, S-Detect interpretations of VSI had 100% sensitivity and 86% specificity for cancer. Table 4 shows agreement of S-Detect interpretation of mass characteristics of VSI in relation to S-Detect interpretation of standard of care imaging, expert VSI interpretation, and expert standard of care interpretation among cancers, cysts, fibroadenomas, and lipomas. In this analysis, there was greater than 80% agreement across S-Detect interpretations of VSI and S-Detect interpretations of standard of care imaging, expert VSI interpretation, and expert standard of care interpretation for mass orientation, the combined analysis of mass shape, the combined analysis of mass margins, and the combined analysis of mass echogenicity. S1 Table shows the full analysis of mass characteristics agreement across different categories of masses.
SOC = standard of care.
The combined mass shape analysis is between “round/oval” or “irregular.” The combined margin analysis is between “circumscribed” or “non-circumscribed.” The combined echogenicity analysis is between “anechoic/hypoechoic/complex cystic or solid,” “isoechoic/hyperechoic,” or “heterogenous.” SOC = standard of care.
Among pathologically proven cancers (n = 20), S-Detect rated all as “probably malignant” on both standard of care imaging and VSI. Among pathologically proven fibroadenomas (n = 8), there was 88% agreement with S-Detect interpretation of standard of care imaging (n = 7/8) and 63% agreement with S-Detect interpretation of VSI (n = 5/8). Among the entire category of masses classified as fibroadenoma either by pathology or imaging stability there was 92% agreement with S-Detect interpretation of standard of care imaging (n = 22/24) and 88% agreement with S-Detect interpretation of VSI (n = 21/24). Among cysts, there was 78% agreement with S-Detect interpretation of standard of care imaging (n = 18/23) and 87% agreement with VSI S-Detect interpretation (n = 20/23).
Discussion
The results from our study showed high diagnostic accuracy of the S-Detect interpretation of VSI in relation to expert radiologist interpretation of standard of care imaging, S-Detect interpretation of standard of care imaging, expert radiologist interpretation of VSI, and the pathological diagnosis. The diagnostic accuracy of S-Detect was overall similar to prior published studies, performing on par with mammography [27,33]. The high agreement observed suggests the portable ultrasound image quality was not an obstacle to S-Detect interpretation. Therefore, with further work packaging the systems together, we believe VSI could be incorporated with S-Detect to produce an ultrasound system that would require neither an experienced sonographer nor interpreting radiologist to automatically and rapidly diagnose palpable breast masses (Fig 4). Such a system would simply require a few hours of training on the VSI protocol, a tablet equipped with the diagnostic software, and a portable ultrasound probe. Given the urgency and challenges posed by breast cancer in LMICs, such a system could assist in decreasing delays to diagnosis with the potential of improving outcomes. In many settings, the sobering alternative to automatic diagnosis may be no imaging at all highlighting the potential value of this approach.
Schematic demonstrating the integration of VSI and artificial intelligence. In this model, a patient presenting with a palpable breast mass would be scanned by an individual with a few hours of ultrasound training using the VSI protocol. Imaging acquisitions would be funneled into artificial intelligence software that outputs a diagnostic report. The diagnostic report would then be available for the health center and patient. This would result in rapid and automated diagnosis without a radiologist or a sonographer.
S-Detect showed excellent diagnostic performance across cysts, fibroadenomas, lipomas, and cancer. In all cases of malignancy, a rating of “possibly malignant” was produced which would result in a patient being referred for higher care and biopsy. There were three cases in which S-Detect referred to a cyst on VSI as “possibly malignant” which could be addressed in future updates of the program. Analysis of the VSI cyst results showed that the VSI images of those cysts likely did not have enough margin around the region of interest to allow accurate detection as S-Detect needs surrounding breast parenchyma to make an accurate diagnosis. After adding randomized noise around the cyst, benign results were able to be obtained in all cases. Similarly, a few cases of cysts on standard of care imaging were also reported as malignant. The standard of care cyst results were thought to relate to differences in the machines used to train S-Detect and would require further training and modification of S-Detect to produce benign results.
Apart from these few cases, expert analysis suggested that all results of S-Detect on the VSI exams were otherwise justifiable. In some cases, masses straddle the BI-RADS 3 or 4 categories making biopsy or follow-up reasonable. Similarly, though a mass may return pathologically benign, in many cases the suspicious features still warranted biopsy. The disagreements observed in mass characteristics likely related to the known accepted large inter-reader differences in interpretation of breast imaging that have been previously documented [32,34,35]. Many of the disagreements on mass characteristics were thought to be trivial in nature (i.e. calling a mass round or oval). Other disagreements were thought to relate to the S-Detect algorithm’s performance on images from different ultrasound machines and/or general error in expert interpretation.
These results suggest that this artificial intelligence is indeed ready to deploy in the near future. Even conceding a few benign lesions being considered “possibly malignant,” the authors believe the 100% detection of malignancy in this sample would easily negate any additional healthcare costs and anxiety associated with the few scattered false positive results which would be promptly reconciled on referral and/or biopsy. The false negative is the truly worst outcome possible in this dynamic as it would result in delay to diagnosis and potentially significant increased morbidity. In our sample, there were no cases of false negatives. Since S-Detect is already very high performing, future updates to the program algorithm should only increase its accuracy further.
The primary clinical advantage of this approach is facilitating increased access to ultrasound imaging which could decrease the often substantial delays to diagnosis of breast cancer associated with advanced disease along with increased morbidity and mortality [6,36]. While more than 70% of breast cancer cases present in stage 1 or 2 in high-income settings, it is estimated that only 20–50% of breast cancer cases present in stage 1 or 2 in LMICs highlighting the need to detect breast cancer earlier in these settings [37]. In cases of benign breast lesions, the automatic diagnostic framework would allow reassurance and potentially save the patient costs associated with workup including any associated travel expenses to a site with breast imaging capacity.
Despite the high incidence and mortality of breast cancer in LMICs, only 5% of global spending on cancer is currently allocated to those settings [38]. Automatic and rapid VSI breast ultrasound acquisition and diagnosis could be one way to help bridge the gap until advanced imaging services are more readily available. Advanced imaging services such as mammography are faced with substantial challenges to implementation in lower-resource settings [9,10]. Therefore, cost-effective and sustainable solutions to increase imaging access in LMICs should be considered [1,39–41]. Our proposed automated diagnostic framework (Fig 4) holds much potential in this regard, but from a public health perspective, studies will need to be conducted how best to market this technology to the community and integrate it into existing healthcare systems. Factors such as availability of biopsy services and surgical/oncological services may somewhat limit the initial impact of deployment at this present time.
There is further work to be done in order to create and integrate an automatic diagnostic framework for breast masses into clinical practice. S-Detect requires that a DICOM image be input into the program. In this study, all of the VSI frames selected for interpretation by S-Detect were chosen by an expert breast imager. This is a workflow and expertise restriction which may be eliminated in the near future by additional artificial intelligence systems trained to automatically detect the mass within the VSI frame and input it into S-Detect. Also, in general, artificial intelligence such as S-Detect could benefit from considering clinical parameters such as age and family history of breast cancer [31]. We noted in this study that abscesses and sebaceous cysts were suboptimally analyzed with S-Detect which is not surprising given that the clinical history is crucial for an experienced breast imager to make an accurate diagnosis. While it is true these cases are usually without clinical ambiguity negating the need for automated diagnosis, input of clinical history could favor a benign diagnosis if infectious symptom are present. Similarly, the artificial intelligence could be upgraded to expand from simply calling masses “possibly benign” or “possibly malignant.” It would be possible to have Breast Imaging-Reporting and Data System (BI-RADS) assessments or a suspected diagnosis (i.e. fibroadenoma or cyst) included in the final assessment.
Breast VSI in this form is currently limited to scanning of palpable abnormalities, not general screening. Automated approaches to whole breast ultrasound screening have been previously described but are not generally cost-effective or immediately deployable in rural areas [42]. As a theoretical possibility, S-Detect could also be integrated with whole breast screening ultrasound systems to produce automated diagnosis without any human operator, but again, costly systems would limit this approach’s relevance to LMICs. VSI could possibly be adjusted to completely scan the entire breast in future iterations, but this approach would be limited by many issues such as potentially suboptimal image quality, false positives, false negatives, and increased time to perform the exams. Palpable abnormalities are ideal for VSI examinations as they are often superficial and large in size allowing ease of diagnosis. Furthermore, the most common presenting feature of breast cancer is a palpable mass, and virtually all breast cancer in LMICs will present as a palpable lump [12,43]. Given the current reported delays to diagnosis for palpable lumps, a system for exclusive automatic diagnosis of palpable lumps is of significant value even though it will inherently not detect asymptomatic lesions which require mammographic examination or other advanced imaging.
One limitation of the study is that our analysis is limited to one single frame selected from the VSI cine clips for interpretation. Future expansions could be made to test multiple frames from a VSI clip to better characterize the performance of S-Detect, especially when a mass is only partially visualized on a VSI frame. Nonetheless, the goal of this study was to demonstrate a proof-of-concept minimizing the impact of this limitation. We approached the study with the interrelated goals of assessing S-Detect performance on both a portable ultrasound machine and on VSI scans. It is possible that S-Detect would perform better given an experienced operator acquiring the images on the portable ultrasound machine. Our study design precluded assessment of this possibility. However, as the ultimate goal of this study was to lay the groundwork for a system that could be used in rural areas, this limitation is of minimal significance to the conclusion that S-Detect functions well with scans obtained on a portable machine. In addition, the already high agreement with standard of care suggests that any increased performance of S-Detect on portable scans obtained by an expert sonographer could only be marginal at best.
Conclusion
The global community is facing an urgent public health crisis with increasing incidence and mortality of breast cancer in LMICs. The lack of readily available ultrasound imaging likely contributes to significant delays to diagnosis of palpable masses, presumably resulting in increased morbidity and mortality. New approaches are needed to increase access to medical imaging in cost-effective and sustainable ways. The integration of volume sweep imaging and artificial intelligence is a promising avenue to realize increased access to imaging. Our synthesis of an inexpensive, portable ultrasound scanner with VSI and artificial intelligence would allow for automatic rapid image acquisition and preliminary but actionable interpretation without a dedicated sonographer or radiologist. In this study, we demonstrated excellent diagnostic accuracy of S-Detect on VSI scans performed by medical students without prior ultrasound training after less than 2 hours of instruction. Further work is needed to synthesize the individual components of this system and automate the identification of a breast lump in a VSI clip for analysis by artificial intelligence. In the context of this urgent public health crisis, we propose expedited further study of this potentially life-saving innovation. Compared to current practice, this technology holds the potential to reduce cost by at least an order of magnitude while increasing accessibility and providing a preliminary, actionable diagnosis to millions of underserved patients.
Methods
Ethics statement
This study was approved by the University of Rochester Research Subjects and Review Board (Study 00005262). Formal written consent was obtained from every participant.
Breast volume sweep imaging
VSI was designed to increase access to imaging in areas with a lack of medical professionals [15]. In VSI, the ultrasound probe is swept over a target region in relation to easily recognized external body landmarks using presets of the ultrasound probe removing the need for anatomical knowledge or technical ultrasound ability. The sweeps are saved as video clips which are later interpreted by a specialist. VSI can be learned over the course of hours as opposed to the years of dedicated training often required for ultrasound specialists [16,17]. This approach has already been successfully piloted for lung, obstetric, right upper quadrant, and thyroid scanning [15,18–21,23]. Specifically, breast VSI has been successfully clinically tested with encouraging results [22]. At this time, breast VSI is only indicated for evaluation of palpable abnormalities. It is generally accepted that ultrasound is first-line evaluation for palpable breast abnormalities [13,44–46]. In breast VSI, the operator marks a palpable abnormality with an “X” and sweeps the probe over the abnormality in several orientations (Fig 5). The protocol can be performed within 5 minutes. It should be emphasized the individual acquiring the images is not adjusting the ultrasound machine or interpreting the images in any way. VSI operators are instead encouraged to focus on their probe technique maintaining good contact with the skin and holding the probe steady.
Illustration of how to perform breast VSI. Sweeps of the ultrasound probe are performed in the transverse, sagittal, radial, and anti-radial orientations.
S-Detect
S-Detect is a computer-aided diagnosis system originally designed as an assistance tool for radiologists. A PC-based research version of S-Detect was used for this study. DICOM images are input into S-Detect which outputs mass characteristics and a dichotomous assessment of “possibly benign” or “possibly malignant” (Fig 6). S-Detect always outputs a result for all possible categories regardless of the image quality. In the current version of S-Detect, a mass is required to be identified by the user. Once the mass has been identified, the contours of the mass can be subsequently edited if needed. The current version of S-Detect does not distinguish between images with a mass or without a mass. Therefore, future integration with VSI will require additional software to identify a mass within the breast tissue.
(A) Screenshot showing the S-Detect user interface. There are three possible methods to target the mass in S-Detect (bracket). In this screenshot, the “Target Point” method has been used. This method involves placing a single point within the target mass (green dot). This could also be accomplished using a “Target Area” or manually outlining the mass boundaries using the “Manual Contour” option. Once the mass has been targeted, S-Detect will produce an interpretation after pressing the “Calculate” button (arrow). (B) Screenshot showing the results after a mass has been targeted. S-Detect outputs a result of “possibly benign” or “possibly malignant” (arrow) as well as a list of mass characteristics (bracket). The mass segmentation is outlined in yellow and this can be adjusted or redone if necessary.
Study design
The masses in this study were obtained from a curated data set from a prior clinical study of breast VSI [22]. All breast VSI examinations were performed by medical students without prior ultrasound experience after less than two hours of training. The VSI scans were performed on the hand-held Butterfly iQ ultrasound probe at the University of Rochester (Butterfly Network, Guilford) with its small organ preset. VSI operators were blinded to the clinical history and were explicitly instructed not to look at the ultrasound images while scanning as an additional safeguard. Standard of care ultrasound examinations were performed in accordance with conventional clinical practice by experienced certified breast sonographers. Standard of care scans were performed on either a Logiq e10 (General Electric, Boston) or an Epiq 7G (Philips, Amsterdam) ultrasound machine with a high-frequency probe with settings adjusted across exams to obtain optimal imaging. The iQ costs approximately $2,000 while the state-of-the-art standard of care ultrasound machines are approximately 50–75 times more expensive.
Individuals over 18 years of age of either sex were eligible to enroll provided they felt a palpable breast lump. For the purposes of this investigation into the accuracy of S-Detect on VSI scans, subjects were eligible for analysis only if VSI and standard of care imaging both showed a sonographic mass as S-Detect does not provide interpretations of normal breast tissue. A VSI frame best representing the mass per expert analysis was chosen by a radiologist and converted to DICOM image format. This DICOM file along with a DICOM file from the standard of care exam were input into S-Detect. The S-Detect interpretation output included both mass characteristics and a final assessment. Mass characteristics included mass orientation (“parallel” or “not parallel”), mass shape (“round,” “oval,” or “irregular”), mass margins (“circumscribed,” “indistinct,” “microlobulated,” “angular,” or “spiculated”), mass echogenicity (“anechoic,” “hypoechoic,” “isoechoic,” “hyperechoic,” “complex cystic and solid,” or “heterogeneous”), and mass posterior acoustic features (“none,” “enhancement,” “shadowing,” or “combined”). The final assessment was “possibly benign” or “possibly malignant.”
The expert VSI exam interpretation procedure was previously described in the clinical study of breast VSI [22]. VSI expert interpretations were made from the best frames/sweeps available by a specialist in breast imaging with over 20 years of experience. The standard of care exam reports were produced per normal clinical practices by an unblinded breast imager. There was a similar reporting scheme employed for the expert interpretations to the one used by the artificial intelligence with two notable differences. The first is that S-Detect will always output a result even if the feature is suboptimally visualized. In the VSI scan expert readings, if one of these features was not visualized or non-diagnostically assessed, it was reported as non-visualized. In standard of care examinations, all features were always visualized. The second difference was the expert VSI and standard of care reports used discrete BI-RADS assessments as opposed to the “possibly benign” or “possibly malignant” categorization used by S-Detect.
BI-RADS is an important classification tool in breast imaging which provides a standard format for interpretation and reporting considered standard of care [47,48]. BI-RADS assessments for standard of care examinations apply to the entirety of the breast tissue examined whereas VSI is solely examining a single palpable location. Therefore, standard of care BI-RADS assessments were adjusted to account for only the palpable finding disregarding findings related to any other breast tissue routinely imaged as part of the standard of care exam. These adjusted assessments also disregarded other clinical history related to prior imaging. BI-RADS assessments were subsequently converted to “probably benign” (BI-RADS 2 or 3) or “probably malignant” (BI-RADS 4 or 5) for analysis. In clinical practice, BI-RADS 2 represents a finding with 100% chance of benignity such as a simple cyst and requires no follow-up. BI-RADS 3 represents a finding with less than a 2% chance of malignancy and is typically managed with a 6-month follow-up. Both BI-RADS 4 (2–95% chance of malignancy) and 5 (>95% chance of malignancy) are typically managed with biopsy.
Among the 170 lumps scanned in the prior clinical trial of breast VSI, n = 115 sonographic masses were identified on VSI and n = 119 sonographic masses were identified on standard of care. This discrepancy was due to 4 sonographic masses seen on standard of care imaging and not on VSI. This was further explored in the prior publication but related to subtle lesions likely of no clinical significance [22]. Since this is a pilot trial, we therefore only tested sonographic masses identified on VSI. The other lumps in the prior study corresponded to palpable areas without a corresponding sonographic mass which is a common occurrence. Therefore, these could not be tested in S-Detect since there was no sonographic mass to test. As explained earlier, this version of S-Detect requires the user to identify a mass which is subsequently fed to S-Detect for analysis. S-Detect does not distinguish background breast parenchymal tissue from sonographic breast masses.
We conducted several analyses of the of agreement including across all masses. However, abscess, sebaceous cysts, and fat necrosis often depend on the clinical history to be distinguished from malignancy. Therefore, we conducted a separate analysis removing these diagnoses from consideration. Additional analysis was performed solely for fibroadenomas, lipomas, cysts, and cancer individually and as a group. Mass characteristics were analyzed across the above groupings for agreement on the single descriptor along with combined analyses where similar descriptors were considered as agreement. These included “round/oval” or “irregular” for mass shape, “circumscribed” or “non-circumscribed” for mass margin, and “anechoic/hypoechoic/complex cystic or solid,” “isoechoic/hyperechoic,” or “heterogenous” for mass echogenicity. Agreement with the pathological diagnosis when available was also performed which was determined by biopsy result when possible. Due to the limited number of biopsies, a mass with a BI-RADS assessment of 2 (100% chance of benignity) was also noted as benign. BI-RADS 3 cases were excluded from the analysis as the pathology was likely benign but not definitively known. Finally, expert analysis of discrepancies in agreement was undertaken by an experienced radiologist.
Statistical analysis
Throughout, categorical variables are summarized by proportion (95% confidence interval (CI)), and continuous variables are summarized as mean ± standard deviation. For comparison of diagnostic accuracy, pathological diagnosis was used as the gold standard, and sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated. Agreement on both diagnosis and mass characteristics was quantified using overall agreement (%), as well as Cohen’s kappa. All statistical analyses were performed using MATLAB (R2019b, The Mathworks, Inc., Natick, MA).
Supporting information
S1 Table. Agreement on mass characteristics between S-Detect interpretation of VSI, compared to other interpretations, sub-divided by the masses included in the analysis.
https://doi.org/10.1371/journal.pdig.0000148.s001
(DOCX)
Acknowledgments
We thank Dr. David Waldman for his invaluable assistance in initiating the study.
We thank Dr. Thomas Foster and the Fischer Fund Committee at the University of Rochester for their support of the study.
We thank Samsung Medison for providing our access to S-Detect software and support.
We thank Averill Meadow, David Walton, Julia Benjamin, Amy Wilkinson, and Butterfly Network, Inc. for their invaluable support and assistance in facilitating our use of the iQ for this study.
We thank sonographers Vicki Crist, Karri Denehy, Vanessa Micciche, and Ashley Rideout for their sonographic expertise and tireless and invaluable assistance in the study.
We thank Laurie Christensen, Erica Longbine, Tammy Russell, and Nicole Underwood for their invaluable administrative and study assistance.
We thank Nadezhda Kiriyak, Sarah Klingenberger, Jane Lichorowic, and Gwen Mack for their invaluable support and assistance in the figures and illustrations.
We thank Lisa Bosivert, Daniel Colosi, Jannette Cong, Julie Giambrone, Susan Hobbs, Deborah Rubens, Matthew Smith, Michelle Snyder, and Courtney Strickland for their invaluable support and assistance in the study.
References
- 1. Anderson BO, Yip CH, Smith RA, Shyyan R, Sener SF, Eniu A, et al. Guideline implementation for breast healthcare in low-income and middle-income countries: overview of the Breast Health Global Initiative Global Summit 2007. Cancer. 2008;113(8 Suppl):2221–43. pmid:18816619.
- 2. Reduction in breast cancer mortality from organized service screening with mammography: 1. Further confirmation with extended data. Cancer epidemiology, biomarkers & prevention: a publication of the American Association for Cancer Research, cosponsored by the American Society of Preventive Oncology. 2006;15(1):45–51. Epub 2006/01/26. pmid:16434585.
- 3. Duffy SW, Tabár L, Yen AM-F, Dean PB, Smith RA, Jonsson H, et al. Mammography screening reduces rates of advanced and fatal breast cancers: Results in 549,091 women. 2020;126(13):2971–9. pmid:32390151
- 4. Tabár L, Vitak B, Chen TH-H, Yen AM-F, Cohen A, Tot T, et al. Swedish Two-County Trial: Impact of Mammographic Screening on Breast Cancer Mortality during 3 Decades. 2011;260(3):658–63. pmid:21712474.
- 5. Tfayli A, Temraz S, Abou Mrad R, Shamseddine A. Breast cancer in low- and middle-income countries: an emerging and challenging epidemic. J Oncol. 2010;2010:490631. pmid:21209708; PubMed Central PMCID: PMC3010663.
- 6. Romanoff A, Constant TH, Johnson KM, Guadiamos MC, Vega AMB, Zunt J, et al. Association of Previous Clinical Breast Examination With Reduced Delays and Earlier-Stage Breast Cancer Diagnosis Among Women in Peru. JAMA Oncol. 2017;3(11):1563–7. pmid:28542677; PubMed Central PMCID: PMC5824208.
- 7. AIUM Practice Guideline for the Performance of a Breast Ultrasound Examination. Journal of Ultrasound in Medicine. 2009;28(1):105–9. pmid:19106368
- 8. Ngoya PS, Muhogora WE, Pitcher RD. Defining the diagnostic divide: an analysis of registered radiological equipment resources in a low-income African country. Pan Afr Med J. 2016;25:99. pmid:28292062; PubMed Central PMCID: PMC5325496.
- 9. Corbex M, Burton R, Sancho-Garnier H. Breast cancer early detection methods for low and middle income countries, a review of the evidence. Breast. 2012;21(4):428–34. pmid:22289154.
- 10. Gutnik LA, Matanje-Mwagomba B, Msosa V, Mzumara S, Khondowe B, Moses A, et al. Breast Cancer Screening in Low- and Middle-Income Countries: A Perspective From Malawi. J Glob Oncol. 2016;2(1):4–8. pmid:28717676; PubMed Central PMCID: PMC5497737 manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO’s conflict of interest policy, please refer to www.asco.org/rwc or jco.ascopubs.org/site/ifc. Lily A. GutnikNo relationship to discloseBeatrice Matanje-MwagombaNo relationship to discloseVanessa MsosaNo relationship to discloseSuzgo MzumaraNo relationship to discloseBlandina KhondoweEmployment: Center for Medical Diagnostics, LilongweAgnes MosesNo relationship to discloseRacquel E. KohlerNo relationship to discloseLisa A. CareyResearch Funding: GlaxoSmithKline, Genentech/RocheClara N. LeeNo relationship to discloseSatish GopalNo relationship to disclose.
- 11. O’Connell AM, Marini TJ, Kawakyu-O’Connor DT. Cone-Beam Breast Computed Tomography: Time for a New Paradigm in Breast Imaging. J Clin Med. 2021;10(21). Epub 2021/11/14. pmid:34768656; PubMed Central PMCID: PMC8584471.
- 12. da Costa Vieira RA, Biller G, Uemura G, Ruiz CA, Curado MP. Breast cancer screening in developing countries. Clinics (Sao Paulo). 2017;72(4):244–53. pmid:28492725; PubMed Central PMCID: PMC5401614.
- 13. Harvey JA. Sonography of palpable breast masses. Semin Ultrasound CT MR. 2006;27(4):284–97. pmid:16915997.
- 14. Maru DS, Schwarz R, Jason A, Basu S, Sharma A, Moore C. Turning a blind eye: the mobilization of radiology services in resource-poor regions. Global Health. 2010;6:18. pmid:20946643; PubMed Central PMCID: PMC2964530.
- 15. Marini TJ, Oppenheimer DC, Baran TM, Rubens DJ, Toscano M, Drennan K, et al. New Ultrasound Telediagnostic System for Low-Resource Areas. Journal of Ultrasound in Medicine. 2021;40(3):583–95. pmid:32798267
- 16. Marini T, Castaneda B, Baran T, O’Connor T, Garra B, Tamayo L, et al. Lung Ultrasound Volume Sweep Imaging for Pneumonia Detection in Rural Areas: Piloting Training in Rural Peru. Journal of Clinical Imaging Science. 2019;9(35). pmid:31538033
- 17. Erlick M, Marini T, Drennan K, Dozier A, Castaneda B, Baran T, et al. Assessment of a Brief Standardized Obstetric Ultrasound Training Program for Individuals Without Prior Ultrasound Experience. Ultrasound Q. 2022. Epub 2022/10/13. pmid:36223486.
- 18. Marini TJ, Oppenheimer DC, Baran TM, Rubens DJ, Dozier A, Garra B, et al. Testing telediagnostic right upper quadrant abdominal ultrasound in Peru: A new horizon in expanding access to imaging in rural and underserved areas. PloS one. 2021;16(8):e0255919. Epub 2021/08/12. pmid:34379679; PubMed Central PMCID: PMC8357175 interests: ME, LT, and CC were paid employees of Medical Innovation and Technology during the study period. Medical Innovation and Technology also provided funding for the logistical costs associated with data collection. BC is the founder of and has a financial stake in Medical Innovation and Technology. This company seeks to bring ultrasound to rural areas through telemedicine. The authors would like to declare the following marketed product associated with this research: “Medical for Ultrasound” or “Med4US." This does not alter our adherence to PLOS ONE policies on sharing data and materials. There are no other patents, products in development, or marketed products associated with this research to declare.
- 19. Marini TJ, Weiss SL, Gupta A, Zhao YT, Baran TM, Garra B, et al. Testing telediagnostic thyroid ultrasound in Peru: a new horizon in expanding access to imaging in rural and underserved areas. Journal of Endocrinological Investigation. 2021. pmid:33970434
- 20. Toscano M, Marini TJ, Drennan K, Baran TM, Kan J, Garra B, et al. Testing telediagnostic obstetric ultrasound in Peru: a new horizon in expanding access to prenatal ultrasound. BMC Pregnancy and Childbirth. 2021;21(1):328. pmid:33902496
- 21. Marini TJ, Kaproth-Joslin K, Ambrosini R, Baran TM, Dozier AM, Zhao YT, et al. Volume sweep imaging lung teleultrasound for detection of COVID-19 in Peru: a multicentre pilot study. 2022;12(10):e061332. J BMJ Open. pmid:36192102
- 22. Marini TJ, Castaneda B, Iyer R, Baran TM, Nemer O, Dozier AM, et al. Breast Ultrasound Volume Sweep Imaging: A New Horizon in Expanding Imaging Access for Breast Cancer Detection. J Ultrasound Med. 2022. Epub 2022/07/09. pmid:35802491.
- 23. Marini TJ, Weis JM, Baran TM, Kan J, Meng S, Yeo A, et al. Lung ultrasound volume sweep imaging for respiratory illness: a new horizon in expanding imaging access. BMJ open respiratory research. 2021;8(1). Epub 2021/11/14. pmid:34772730; PubMed Central PMCID: PMC8593737.
- 24.
Saavedra AC, Arroyo J, Tamayo L, Egoavil M, Ramos B, Castaneda B, editors. Automatic ultrasound assessment of placenta previa during the third trimester for rural areas. 2020 IEEE International Ultrasonics Symposium (IUS); 2020 7–11 Sept. 2020.
- 25. Arroyo J, Saavedra A, Tamayo L, Egoavil M, Ramos B, Castaneda B. Automatic fetal presentation diagnosis from ultrasound images for rural zones: head location as an indicator for fetal presentation: SPIE; 2021.
- 26. Zhang D, Jiang F, Yin R, Wu GG, Wei Q, Cui XW, et al. A Review of the Role of the S-Detect Computer-Aided Diagnostic Ultrasound System in the Evaluation of Benign and Malignant Breast and Thyroid Masses. Medical science monitor: international medical journal of experimental and clinical research. 2021;27:e931957. Epub 2021/09/24. pmid:34552043; PubMed Central PMCID: PMC8477643.
- 27. Choi JS, Han BK, Ko ES, Bae JM, Ko EY, Song SH, et al. Effect of a Deep Learning Framework-Based Computer-Aided Diagnosis System on the Diagnostic Performance of Radiologists in Differentiating between Malignant and Benign Masses on Breast Ultrasonography. Korean journal of radiology. 2019;20(5):749–58. Epub 2019/04/18. pmid:30993926; PubMed Central PMCID: PMC6470083.
- 28. Bartolotta TV, Orlando AAM, Di Vittorio ML, Amato F, Dimarco M, Matranga D, et al. S-Detect characterization of focal solid breast lesions: a prospective analysis of inter-reader agreement for US BI-RADS descriptors. Journal of ultrasound. 2021;24(2):143–50. Epub 2020/05/25. pmid:32447631; PubMed Central PMCID: PMC8137795.
- 29. Cho E, Kim EK, Song MK, Yoon JH. Application of Computer-Aided Diagnosis on Breast Ultrasonography: Evaluation of Diagnostic Performances and Agreement of Radiologists According to Different Levels of Experience. J Ultrasound Med. 2018;37(1):209–16. Epub 2017/08/02. pmid:28762552.
- 30. Kim K, Song MK, Kim EK, Yoon JH. Clinical application of S-Detect to breast masses on ultrasonography: a study evaluating the diagnostic performance and agreement with a dedicated breast radiologist. Ultrasonography (Seoul, Korea). 2017;36(1):3–9. Epub 2016/05/18. pmid:27184656; PubMed Central PMCID: PMC5207353.
- 31. Kim SY, Choi Y, Kim EK, Han B-K, Yoon JH, Choi JS, et al. Deep learning-based computer-aided diagnosis in screening breast ultrasound to reduce false-positive diagnoses. Scientific Reports. 2021;11(1):395. pmid:33432076
- 32. O’Connell AM, Bartolotta TV, Orlando A, Jung S-H, Baek J, Parker KJ. Diagnostic Performance of an Artificial Intelligence System in Breast Ultrasound. 2022;41(1):97–105. pmid:33665833
- 33. Zeeshan M, Salam B, Khalid QSB, Alam S, Sayani R. Diagnostic Accuracy of Digital Mammography in the Detection of Breast Cancer. Cureus. 2018;10(4):e2448. Epub 2018/06/12. pmid:29888152; PubMed Central PMCID: PMC5991925.
- 34. Calas MJ, Almeida RM, Gutfilen B, Pereira WC. Interobserver concordance in the BI-RADS classification of breast ultrasound exams. Clinics (Sao Paulo). 2012;67(2):185–9. Epub 2012/02/24. pmid:22358246; PubMed Central PMCID: PMC3275126.
- 35. O’Connell AM, Bartolotta TV, Orlando A, Jung SH, Baek J, Parker KJ. Diagnostic Performance of An Artificial Intelligence System in Breast Ultrasound. J Ultrasound Med. 2021. Epub 2021/03/06. pmid:33665833.
- 36. Jassem J, Ozmen V, Bacanu F, Drobniene M, Eglitis J, Lakshmaiah KC, et al. Delays in diagnosis and treatment of breast cancer: a multinational analysis. Eur J Public Health. 2014;24(5):761–7. pmid:24029456.
- 37. Unger-Saldana K. Challenges to the early diagnosis and treatment of breast cancer in developing countries. World J Clin Oncol. 2014;5(3):465–77. pmid:25114860; PubMed Central PMCID: PMC4127616.
- 38. Breast cancer in developing countries. Lancet. 2009;374(9701):1567. pmid:19897110.
- 39. Denny L, de Sanjose S, Mutebi M, Anderson BO, Kim J, Jeronimo J, et al. Interventions to close the divide for women with breast and cervical cancer between low-income and middle-income countries and high-income countries. Lancet. 2017;389(10071):861–70. pmid:27814963.
- 40. Yip CH, Anderson BO. The Breast Health Global Initiative: clinical practice guidelines for management of breast cancer in low- and middle-income countries. Expert Rev Anticancer Ther. 2007;7(8):1095–104. pmid:18028018.
- 41. Yip CH, Buccimazza I, Hartman M, Deo SV, Cheung PS. Improving outcomes in breast cancer for low and middle income countries. World J Surg. 2015;39(3):686–92. pmid:25398564.
- 42. Kim SH, Kim HH, Moon WK. Automated Breast Ultrasound Screening for Dense Breasts. Korean journal of radiology. 2020;21(1):15–24. Epub 2020/01/11. pmid:31920025; PubMed Central PMCID: PMC6960307.
- 43. Barton MB, Elmore JG, Fletcher SW. Breast symptoms among women enrolled in a health maintenance organization: frequency, evaluation, and outcome. Ann Intern Med. 1999;130(8):651–7. pmid:10215561.
- 44. Expert Panel on Breast I, Moy L, Heller SL, Bailey L, D’Orsi C, DiFlorio RM, et al. ACR Appropriateness Criteria((R)) Palpable Breast Masses. J Am Coll Radiol. 2017;14(5S):S203–S24. pmid:28473077.
- 45. Giess CS, Smeglin LZ, Meyer JE, Ritner JA, Birdwell RL. Risk of malignancy in palpable solid breast masses considered probably benign or low suspicion: implications for management. J Ultrasound Med. 2012;31(12):1943–9. pmid:23197547.
- 46. Lehman CD, Lee AY, Lee CI. Imaging management of palpable breast abnormalities. AJR Am J Roentgenol. 2014;203(5):1142–53. pmid:25341156.
- 47. Burnside ES, Sickles EA, Bassett LW, Rubin DL, Lee CH, Ikeda DM, et al. The ACR BI-RADS experience: learning from history. J Am Coll Radiol. 2009;6(12):851–60. Epub 2009/12/01. pmid:19945040; PubMed Central PMCID: PMC3099247.
- 48.
Mendelson EB, Böhm-Vélez M, Berg WA. ACR BI-RADS Ultrasound. In: ACR BI-RADS Atlas, Breast Imaging Reporting and Data System. Reston, VA: American College of Radiology; 2013.