2.1. Search strategy

Two independent observers (YF, and JZ) searched databases (PubMed, EMBASE, and Web of Science) from inception to October 30, 2023. After duplicate removal, YF and JZ independently screened the title and abstract of studies to evaluate the eligibility. Then, the full text of the remaining studies was screened according to the study selection criteria. The reference list of included studies was searched to identify possible eligible studies. Any disagreement during the database search was discussed and resolved by consultation.

2.2. Inclusion criteria

We selected the studies for review if they met the following inclusion criteria based on PICO (Population, Index test, Comparator, and Outcome) elements:

1. Population: patients diagnosed with BC.
2. Index test: using AI-based methods to predict the status of the key molecular biomarkers (HER2, Ki67, PR, and ER) in patients.
3. Comparator: using biopsies or surgically excised samples to determine the status of the molecular biomarkers.
4. Outcome: Studies with enough data to predict the status of key molecular biomarkers in BC patients based on US images before surgical procedures (biopsy, excisional biopsy, surgery).

2.3. Exclusion criteria

We excluded the records that met the following exclusion criteria:

1. Studies incompatible with the PICO criteria mentioned above.
2. Reviews (narrative, systematic, and meta-analysis), case reports, conference abstracts, comments, and letters to editors.
3. Articles in a non-English language.
4. Studies not using AI-based methods.

2.4. Data extraction

We used standardized forms to review the included studies. Two independent reviewers (YF, and JZ) extracted the following data upon availability: The first author’s name, year of publication, type of study, country, name, and number of patient selection centers, aim of study, patients’ information (patient number, gender distribution, mean age), AI method, type of US, whether contrast agent is used or not. For radiomics studies, we extracted the following data specifically: segmentation method (automatic manual or semi-automatic), segmentation dimension and software, features number, type of extracted features, feature extraction software, feature selection methods, AI modeling algorithm, and whether the radiomics features are combined with other clinical data. Possible disagreements were discussed and resolved through a consensus.

2.5. Quality assessment

We used the “Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2)” tool to evaluate the presence of risk of bias and application concerns in different domains (“patient selection,” “index test,” “reference standard,” and “flow and timing”). Each domain assigned yes, no, or unclear for the assessment of risk of bias and low risk, high risk, or unclear for assessing applicability concerns. Also, we can tailor the QUADAS-2 signaling questions to our research questions to assess the risk of bias [13]. Moreover, we used the “Radiomics Quality Score (RQS)” system to measure the methodological rigor of the studies using the radiomics method. RQS has 16 items with corresponding scores under the following domains: data selection, medical diagnostic images, feature extraction, exploratory analysis, and model construction [14]. Two independent reviewers (JZ, and ZL) assessed the methodological quality and risk of bias using the QUADAS-2 and RQS tools, and disagreements were resolved by discussion and consultation.

2.6. Statistical analysis

We used STATA (version 15; MIDAS module) for the meta-analysis. Sensitivity and specificity with 95% confidence interval (CIs) were calculated for pooling the effect sizes. We plotted the summary receiver operating characteristic curve (SROC) to calculate the area under the curve (AUC). We indicated the pooled effect sizes using the forest plot. A random-effect model was utilized to combine the effect sizes and address the potential heterogeneity of studies. We designed a deeks funnel plot for evaluation of the publication bias. Some studies did not have enough data for meta-analysis. We manually calculated the needed parameters using sensitivity, specificity, and the number of positive and negative classes of each molecular biomarker. The Fagan plot was created to evaluate the clinical utility of the included studies with post-test probability when the pretest probabilities were available.

3.1. Literature search

We outline the study selection process details using the PRISMA flowchart in Fig 1. Our initial search identified 363 articles, of which 93 records were duplicates. The titles and abstracts of the 270 records were screened, and 235 articles were removed. After reading through the full text of the remaining 35 articles, twelve more records were removed due to the following reasons: 1) four studies were not about AI methods; 2) two studies did not predict the key molecular biomarkers in BC patients; 3) three studies were not in English; 4) three studies did not provide sufficient data for the review process. Then, we included 23 studies with a total number of 11005 patients for the qualitative review process [15–37].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flowchart for the included studies.

https://doi.org/10.1371/journal.pone.0303669.g001

Of these 23 studies, 15 studies were available for quantitative analysis, and eight studies were not included in the meta-analysis due to inadequate data.

3.2. Study characteristics

We showed the general characteristics of 23 included studies in Table 1, of which 14 were radiomics. The characteristics of the radiomics workflow are summarized in Table 2. Overall, twenty-one (21/23) studies were from China, and the other two were from Romania [33] and Canada [21]. Three studies [15, 25, 33] were prospective (3/23), while the remaining twenty were designed retrospectively (20/23). Regarding AI methods, fourteen studies used radiomics (14/23), six studies used deep learning (6/23), and three studies used machine learning methods (3/23). Three studies [15, 25, 33] retrieved patients from multiple centers (3/23), and the remaining twenty studies were from only one center (20/23). The overall number of patients was 11.005, with a mean age of 52.7 years. However, two studies [20, 23] did not provide the mean age of subjects. One study [36] included men BC; the remaining studies were from the female population. Regarding radiomics studies (14/23), we investigated different steps of radiomics workflow, including segmentation, feature processing, and modeling. Three studies [19, 32, 33] used the automatic/semiautomatic method to delineate the region of interest (ROI). The remaining 19 studies were segmented manually. Only one study [31] did not specify the segmentation method. The open-source ITK-SNAP software was the most commonly used tool for the segmentation process (5/14), and it was utilized in five studies [22, 28, 34, 36, 37]. Two studies used deep learning (2/14) for the segmentation process. Time-intensity curve software, GNU image manipulation program (GIMP), 3D-Slicer, and Darwin Research Platform were used once in the studies. Three studies [25, 31, 35] did not mention the segmentation software. Most studies extracted the shape, first-order, second-order, and wavelet radiomics features regarding the feature processing step. The PyRadiomics package was the most utilized tool to extract the features from ROIs. The extracted features in studies ranged from 86 to 5234. Among these features, the most robust ones should be selected, and different methods are utilized in studies to select them. The Least Absolute Shrinkage and Selection Operator (LASSO) regression was the most commonly used method among studies. Then, the selected features are fitted into a machine learning algorithm to train the data. The logistic regression (LR) algorithm was used in nine studies (9/14) for the model construction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. General information of the studies.

https://doi.org/10.1371/journal.pone.0303669.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. information of radiomics studies.

https://doi.org/10.1371/journal.pone.0303669.t002

3.3. Quality assessment

The 23 studies reached a mean RQS score of 12.64 with a minimum of 6 and a maximum of 19 out of 36 possible points. Table 1 shows the RQS score of each study in the corresponding column. The individual ratings for each RQS item and each study are provided in S1 Table. The following items were evaluated in none of the studies: phantom study, imaging at multiple points, cut-off analyses, potential clinical application, cost-effectiveness analyses, open science, and data.

Details of the assessment of included studies using QUADAS-2 are shown in S1 Fig and Fig 2 All studies assigned are “unclear” regarding the flow and timing item because none of the studies provided the timing of biopsy before US examination. Hao Cui et al. and Jia‑wei Li et al. assigned a “High risk of bias” because the imaging process approach and validation step were not specified.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. QUADAS-2 assessment of the included studies.

https://doi.org/10.1371/journal.pone.0303669.g002

3.4. Predictive performance of AI methods for HER2

Seventeen studies (17/23) predicted the HER2 status in BC patients, of which ten studies (10/17) had adequate data for quantitative analysis. Studies with insufficient data [15, 17, 18, 23, 27, 30, 37] were excluded from the quantitative analysis. For the prediction of HER2 status, pooled estimates were sensitivity, specificity, and summary receiver operating characteristic (SROC) with a 95% confidence interval. Table 3 shows the categorized extracted data for quantitative analysis. According to the generated forest plot (S2 Fig), the pooled sensitivity and specificity were 0.76 [0.65, 0.84] and 0.78 [0.70, 0.84], indicating a high rate of heterogeneity for both sensitivity (I2 = 67.16 [45.31, 89.01]) and specificity (I2 = 91.41 [87.42, 95.40]). Pooled results from the AUC estimates were 0.84 [0.15, 0.99].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Categorized extracted data for quantitative analysis.

https://doi.org/10.1371/journal.pone.0303669.t003

The funnel plot did not show publication bias (p: 0.46) (S3 Fig). Regarding clinical utility, the Fagan plot (S4 Fig) showed that AI models of HER2 status would increase the post-test probability to 46% from 20% with a PLR of 3 when the pretest was positive and would reduce the post-test probability to 7% with an NLR of 0.31 when the pretest was negative.

Bo-Yang Zhou et al. [15] designed a prospective study showing that the US-based convolutional neural network (CNN) model has a high accuracy (AUC: 0.89–0.96). This study differs from the other studies by the type of prediction model that distinguishes four classes of molecular biomarkers (HER2, ER, PR, Ki67). For this reason, we did not include this study in the meta-analysis.

Meng Jiang et al. [17] designed a similar study to Bo-Yang Zhou et al. They constructed a deep-learning model of US images that differentiates the four categories of BC (Luminal A, Lumina B, HER2+, and Triple-negative) based on key molecular biomarkers. In two test sets, the model yielded a good accuracy of 80.07% to 97.02% and 87.94% to 98.83%. Compared to the study of Bo-Yang Zhou et al., this study used two test sets with accuracy similar to the mentioned study. The high level of accuracy may be due to the overfitting since both studies are deep-learning models.

Xianyu Zhang et al. [23] developed a deep learning US model for three-class classification of BC, including HER2, hormone receptor, and triple-negative. The deep learning model achieved an accuracy of 85.6%, with a gold standard of pathological results in the test set. Also, the model’s accuracy was 90% in an external test set. Interestingly, this study and the two previous [15, 17] studies with deep learning models had a high accuracy in test sets.

Similar to the previous studies, Zilong Xu et al. [30] constructed a deep learning US model. The model yielded an AUC of 0.84 and an accuracy of 80.56% in the validation set for HER2 status prediction.

Linyong Wu et al. [37] developed a US-radiomics model in patients with breast ductal carcinoma in situ (DCIS) to predict HER2, ER, PR, and Ki67. Since this study did not specify the exact number of positive and negative classes of each biomarker in training and testing sets, we did not include it in the quantitative analysis. Evaluations indicated that the model had an AUC of 0.94 and 0.74 in the training and test set for HER2 status prediction.

Mengwei Ma et al. [18] fitted the US texture features to different machine learning algorithms. They established different models for different tasks. All machine learning models yielded a similar AUC for differentiating HER2 from other subtypes. However, the Random Forest model performed slightly better, with an AUC of 0.855 in the test set. On the other hand, the logistic regression model outperformed other models to differentiate HER2+ from HER2- with a slightly better AUC of 0.744.

Taken together, AI models determined the HER2 status in BC patients with moderate to high precision in the quantitative analysis with a pooled sensitivity, specificity, and AUC of 0.76, 0.78, and 0.84, respectively. Deep learning models showed that outperformed the other AI methods for the prediction of HER2 status.

3.5. Predictive performance of AI methods for Ki67

Thirteen studies (13/23) predicted the Ki67 status in BC patients, of which seven studies (7/13) had adequate data for quantitative analysis. Studies with insufficient data were excluded from the quantitative analysis [15, 17, 18, 31, 33, 37].

For the prediction of Ki67 status, pooled estimates were sensitivity, specificity, and SROC with a 95% confidence interval. Table 3 shows the categorized extracted data for quantitative analysis. According to the generated forest plot (S5 Fig), the pooled sensitivity and specificity were 0.80 [0.67, 0.89] and 0.76 [0.69, 0.81], indicating a high rate of heterogeneity for both sensitivity (I2 = 89.99 [84.49, 95.48]) and specificity (I2 = 62.76 [34.22, 91.30]). Pooled results from the AUC estimates were 0.81 [0.78, 0.84].

The funnel plot did not show publication bias (p: 0.09) (S6 Fig). Regarding clinical utility, the Fagan plot (S7 Fig) showed that AI models of Ki67 status would increase the post-test probability to 45% from 20% with a PLR of 3 when the pretest was positive and would reduce the post-test probability to 6% with an NLR of 0.26 when the pretest was negative.

Hao Cui et al. [31] examined the accuracy of a US radiomics model for Ki67 status prediction. Sensitivity and specificity are the two main metrics for evaluating diagnostic tools. This study did not report these two metrics and Ki67 classes in the dataset. However, the regression model achieved an AUC of 0.71.

In contrast to all other studies, Ioana Bene et al. [33] showed that no texture parameter was statistically significant in the univariate analysis for the Ki67 biomarker prediction in BC patients.

Also, Mengwei Ma et al. [18] developed different machine-learning models. In contrast to HER2 status prediction, the Naive Bayes (NB) model performed better than other algorithms. For differentiating high Ki67 from low Ki67 expression, the NB model had an AUC of 0.739 in the test set.

Altogether, AI models determined the Ki67 status in BC patients with moderate to high precision in the quantitative analysis with a pooled sensitivity, specificity, and AUC of 0.80, 0.76, and 0.81, respectively.

3.6. Predictive performance of AI methods for ER and PR

Six studies (6/23) predicted the ER and PR status in BC patients, of which only two studies (2/6) had adequate data [22, 33], which is not enough to pool the effect sizes for quantitative analysis. As mentioned, Bo-Yang Zhou et al. [15] designed a CNN model of US features with high accuracy (AUC: 0.89–0.96) in predicting BC molecular subtypes’ four-classification (HER2, ER, PR, Ki67). Also, Linyong Wu et al. [37] constructed a US-radiomics model to predict HER2, ER, PR, and Ki67 biomarkers. Analysis showed that the model had an AUC of 0.94 and 0.74 in the training and test set. We noted that Meng Jiang et al. [17] constructed a deep-learning model that could differentiate the four categories of BC (Luminal A, Lumina B, HER2+, and Triple-negative) based on key molecular biomarkers such as ER and PR. The model yielded a good accuracy of 80.07% to 97.02% and 87.94% to 98.83% in the two test sets.

Also, Mengwei Ma et al. [18] developed several machine-learning models. The LR algorithm showed the highest AUC of 0.878 among all machine learning algorithms for differentiating ER+ and ER- patients. Similarly, the LR algorithm outperformed others in differentiating PR+ and PR- patients with an AUC of 0.879.

We can conclude from the data that AI models of ER and PR status prediction have the potential to be applied in future clinical practice since models showed promising precision for differentiating ER and PR biomarkers in BC patients.

4.1. Limitations and future prospects

Although we reviewed the included studies comprehensively, this study encountered some inherent limitations. in this study, we included eligible studies regarding key molecular biomarkers in BC patients to pool the diagnostic effect sizes. However, several studies were short of enough data for quantitative analysis, which restricted us from reporting more reliable results, especially a single diagnostic metric for ER and PR markers. Therefore, we suggest future research provide data as far as possible to assist future meta-analyses in pooling the valuable findings to reach a single valuable finding. Only three studies retrieved the patients from multiple data centers. Since data collection from a single institute may not reflect the characteristics of the entire population, it is recommended to recruit patients from multiple data centers. Most of the included studies collected the data retrospectively. Since retrospective data collection has a high potential to cause bias, we need more prospective studies in the upcoming research. Since the assessment of the risk of bias and the quality of the literature showed some limitations, we are going to offer some recommendations: 1) Specify the selection criteria, study type, and patient characteristics; 2) Show the data acquisition, image processing, and validation process details; 3) apply the same reference standard to all patients as far as possible.