Data preparation.

This study used GI endoscopic biopsy specimens with confirmed diagnoses that were stored at Seegene Medical Foundation (SMF), Seoul (Seongdong -gu, Seoul). These specimens were selected from specimens that SMF had received from approximately 400 local clinics and hospitals throughout Korea for histopathologic diagnosis. To use these specimens for research purpose, we obtained approvals from the Institutional Review Board (IRB) of SMF (IRB Approval number: SMF-IRB-2020–007) and IRB of Korea Advanced Institute of Science and Technology (KAIST) (IRB Approval number: KAIST-IRB-20–379). All experiments were conducted in accordance with the relevant guidelines and regulations given by the two IRBs. All patient records were completely anonymized, and all data were stored and analyzed only in SMF servers. This study corresponds to the guidelines for performing laboratory quality control and test method evaluations, such as accuracy tests, using remaining human-derived specimens after treatment and diagnosis and received waiver of informed consent. Among the specimens stored at SMF after diagnosis between June 2018 and April 2021, 3,979 gastric biopsy and 1,848 colorectal biopsy hematoxylin-eosin stained slides were randomly selected and scanned (Panoramic 250 Flash III, 3DHISTECH, 20x). Of these, some were excluded for various reasons during the study (to be described later); ultimately, 1,762 gastric biopsy and 1,509 colorectal biopsy WSIs were used to develop a classifier model. The data were divided into training (80%), validation (10%), and test (10%) based on the WSIs. For accurate training and evaluation of the model, the training, validation, and test data were divided at patient level (i.e., whole slide) with no overlap. Table 1 shows in detail how the WSIs were specifically divided into training, validation, and test data. To develop AI classification models for daily QC in real-world practice, we used raw data, preserving routine artifacts and interobserver variability, to strictly reflect daily practice, rather than artificially establishing well refined WSI data sets for high model accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Details of the datasets used for the development of each AI model.

https://doi.org/10.1371/journal.pone.0278542.t001

Ensuring diagnostic reliability with usual interobserver variability.

In total, five pathologists with at least five years of clinical experience and specialization in pathology (MJK, YRC, SHL, YMC, and YSK) made annotations, and all WSIs were first reviewed by each annotator. If an annotator disagreed with an existing diagnosis, at least two pathologists reviewed the applicable slides together. If at least two pathologists disagreed with the existing diagnosis, the slides were excluded from the data collection set. Thus, cases of disagreement by only one pathologist were considered to reflect realistically acceptable interobserver variability, and annotations made by those who agreed with the existing diagnosis were included as training data. The annotation rules are provided in the S1 Methods.

Reflecting the qualitative environment of pathologic interpretation in the real world.

The quality of the slides was not adjusted for optimal model accuracy. In fact, routine artifacts (e.g., poor sectioning, poor staining, poor fixation, and air bubbles) were used in training because they lacked artificial improvement (re-preparation high-quality slides just for training). However, poor-quality slides that were difficult for pathologists to interpret were excluded.

Scan quality was also not adjusted artificially. Scanner manufacturers declare an average error rate of 1–3% for slides that require rescanning, while others are mostly minor problems that do not affect the interpretation; these include some small out-of-focus areas, overlapping tiles, and some scanned tissues with cropped edges. To reflect these routine minor errors, artificial improvement measures, such as rescanning to obtain high-quality WSIs, were not implemented for training only. Moreover, poor-quality WSIs that would be difficult for pathologists to interpret were excluded.

Definition of each class for training and internal validation.

Model development was attempted for daily QC in all GI endoscopic biopsy specimens at our institution; thus, initially we used four categories (malignant [M], dysplasia [D], negative for dysplasia [N], and uncategorized [U]) to classify all possible diagnoses. Subsequently, we started the development of a quaternary classifier model based on these classes (Table 2). However, the inclusion of class “U,” which by definition had the highest heterogeneity among diseases, degraded the performance of the model [39] and was determined to be ineffective in daily practice. Accordingly, a ternary classifier model that excluded class U was selected for the trial run; thus, cases corresponding to class U were excluded from the training dataset for development of the ternary classifier model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Definition of each class for training and internal validation.

https://doi.org/10.1371/journal.pone.0278542.t002

Training of AI models.

WSIs include giga pixel-level data; thus, they are not suitable to be used all at once for deep learning (DL). Accordingly, WSIs that include giga pixel-level data must be suitably converted to be processed by the models. Model construction was based on a method of converting and processing WSI into patches, similar to that used in various existing studies [40, 41]. As mentioned in numerous studies on convolution neural network (CNN)-based WSI processing [42, 43], the complexity of the CNN can cause a loss of data and resources from WSIs when they are processed all at once. Consequently, we conducted model training in three steps: the data preprocessing step, patch classifier training step, and WSI classifier training step (Fig 1). The same model training method was applied to gastric and colorectal models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the training of the AI model for daily QC.

An overview of the modeling method used, which includes three steps, is shown. The first step is the data preprocessing step; data are organized and converted in several patches that can be used to train the models. The next step is the patch classifier training step; training is carried out based on the labeled data for each patch generated from a single WSI. The final step is the training WSI classifier step; models that generate information for classification of WSIs are trained by combining the data regarding each WSI. The same model training method was applied to gastric and colorectal models. Abbreviations: AI (artificial intelligence), WSI (whole slide image), QC (quality control), GUI (graphical user interface), M (Malignant), D (Dysplasia), N (Negative for dysplasia).

https://doi.org/10.1371/journal.pone.0278542.g001

Data preprocessing.

The first step in AI model training was the data preprocessing step. Model training was conducted based on ground truth data generated by the method mentioned in the previous section and labeled data as shown in Table 1. Data were set up without annotation for class N WSIs and with annotations for class M and D WSIs. The WSIs were saved in mrsx format, and data preprocessing was conducted with Openslide’s library [44].

The method used in our study largely required two deep neural network (DNN) models for WSI learning. To train each model, data were prepared in two forms: WSI data and patch data. For WSI data, the scan configuration values of the collected images were checked and images with different values were excluded from the dataset. This was to prevent images scanned under different conditions from causing errors in the model. The WSI data set finally selected is shown in Table 1. The next step was patch data preprocessing. For class N, Openslide’s library was used to convert a whole slide to multiple patches, each of which contained a 256x256 pixel image (image with a width of 256 pixels and a height of 256 pixels) derived from the WSI. The patch size was determined in consideration of the trade-off between user convenience and performance. More specifically, using the same set of slides, we generated alternative patches of 128×128, 256×256, 512×512, and 1024×1024 pixel images, which are typical input image sizes for CNN, and then compared the actual classification performances, while seeking user feedback on the size appropriateness for the user interface (UI). For example, 1024×1024 and 512×512 size pixel images sometimes produced slighty better performance but were considered unsuitable for determining the lesion location information provided on the UI (refer to “Development of SeeDP”). In addition, with the 128x128pixel images, the classification model performance was noticeably deteriorated. Thus, the 256×256 pixel image patch size was considered adequate. For classes M and D, only patches within the annotations were selected and saved; patches outside the annotations were not included in the dataset.

For an accurate assessment of the model performance and feedback, training, validation, and test datasets were constructed based on the WSIs and their associated patches. If the training, validation, and test datasets were constructed without considering the association between WSIs and patches, patch images generated from the same WSI could be found simultaneously in the training dataset and test or validation dataset. Lastly, patch data were randomly sampled from the generated training pool to balance and minimize bias in the data classifiers (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Data preprocessing workflow.

The workflow for generating patch images during the data preprocessing step is shown. This workflow involves a series of processes for generating patch images based on the annotation data with WSIs loaded into Openslide’s library, which were processed differently according to class. Abbreviations: WSI (whole slide image).

https://doi.org/10.1371/journal.pone.0278542.g002

Model training (patch classifier).

In the second step of AI model training, i.e., the patch classifier training step, CNN-based DNN architecture was used to train the image classifiers. CNN has shown remarkable achievements since the development of AlexNet [45]. In particular, combining DL based on CNNs has led to its active use in the medical domain for fields dependent on human intervention, such as segmentation and detection [46–49]. For patch image classification, we used DenseNet201, which showed excellent performance in DNN architecture for model [50]. DenseNet is well recognized for its remarkable performance in competitive object recognition benchmark tasks, such as ImageNet and CIFAR-100 [51, 52]. DenseNet inputs the connect feature-maps generated in all previous layers to subsequent layers and achieves easy training and parametric efficiency by reusing the features. Using this method, deep layers of the network can reuse the features by accessing all connected feature-maps generated in previous layers [53]. We inputted training data that were setup in the pre-trained DenseNet201 model. Patch classifier model was trained to accurately infer three correct labels (classes M, D, and N) for each patch image (Fig 3). Moreover, in order to improve the effectiveness of the training process, noise data were removed based on the training loss value of the DenseNet201 model [39].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Concept of patch classifier training.

The conceptual diagram of the patch image training process is shown.

https://doi.org/10.1371/journal.pone.0278542.g003

The classification results generated from the patch classifiers had two purposes: 1) to be used as elements for inference by the WSI classifiers and 2) to provide explanation in the UI. In particular, WSIs were trained, based on the model trained in this step, for use as elements for WSI classifiers. Patch classifiers can generate classification results by distribution in each class, which can be used as core information.

Model training (WSI classifier).

WSI classifiers were trained with the patch classification results, which were fragmented data of WSIs, as elements for inference by correct WSI labels. In the WSI classifier training step, WSI classifiers were trained through a series of processes based on 1) the patch maker model used in the data preprocessing step, and 2) the patch classifiers from the model training step. This allowed for the efficient use of resources needed for WSI classification, which included giga pixel data, and each model was designed to operate dynamically like a single model.

In the WSI classifier training process (Fig 4), WSIs were converted to patch images by the patch maker, and converted patch images were reconstructed based on the locational information to train WSI classifiers. This WSI classifier training process largely included three steps: the patch maker step, patch classifier step, and WSI training step. In the patch maker step, WSI were converted to several patch images. To preserve the location information of each patch image during this conversion process, index and location information of the converted patch and the WSI from which the patch was generated were recorded in a database (DB). In the patch classifier step, classification information of each patch from a single WSI was inferred through the trained patch classifier. The inference information of each patch image was stored in the DB as per the index of the patch image. Accordingly, the index, location, and classification information of each patch image of a single WSI were stored in the DB. Ultimately, the DB of patch images was combined for WSI training, and reconstructed WSIs were generated. Reconstructed WSIs included a condensed version of the characteristic information of the WSI; thus, this information can be processed in CNNs, in a similar manner as that for a single image. We trained the slide classifier to classify the reconstructed WSIs using the same architecture as the patch classifier DenseNet201. In other words, the entire process followed a process that utilized the DenseNet architecture to produce patch classification results to generate a distribution of predicted confidence values for each patch image and to train another DenseNet to classify the reconstructed WSIs. This series of processes was applied to both gastric and colorectal models to generate two models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Training process for WSI classifiers.

A conceptual diagram of the training process for WSI classifiers is shown. A single WSI is converted into several patch images by the patch maker and distribution information is generated by the patch classifier, which is then converted to a reconstructed WSI. Reconstructed WSIs typically have images rectangular in shape. To classify the reconstructed WSIs, we used a trained CNN model as the WSI classifier. This series of processes was applied to both the gastric and colorectal models to generate two models. Abbreviations: WSI (whole slide image), CNN (convolution neural network).

https://doi.org/10.1371/journal.pone.0278542.g004

Redefining classes for the SeeDP daily QC system.

Between September 15, 2021 and December 14, 2021, four pathologists (YMC, JRK, BDL, and YSK) participated in the trial run. First, the classes of pathologic diagnoses were redefined as shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Definition of classes by the pathologic diagnosis for the SeeDP daily QC system.

https://doi.org/10.1371/journal.pone.0278542.t004

Classification for data training and validation in the AI model development phase (Table 2) and classification by the pathologic diagnosis in the trial run phase (Table 4) differed. Specifically, “tubular adenoma, high grade dysplasia (TA, HGD)” and “tubular adenoma, grade uncertain” were included in class D for training in the AI model development phase, but were redefined as class M in the SeeDP system for the daily QC trial run. Additionally, class D was redefined as low grade dysplasia (LGD), indefinite for dysplasia, and AUS. Each class was redefined in the trial run for clinical implementation to assist in the immediate correction of FNs by the rapid detection of potential human error by the pathologists, rather than to ensure a more accurate diagnosis by the AI model, as explained earlier. Therefore, specific words included in pathologic diagnoses were considered as keywords for classification by the pathologic diagnosis (Table 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Keywords for classification by the pathologic diagnosis.

https://doi.org/10.1371/journal.pone.0278542.t005

Based on the classification in Table 5, cases that included the corresponding keyword in the pathologic diagnosis (text information) were classified as the indicated class. Such classification results were compared (1:1 basis) against the classification by AI model prediction for determining concordance. Each keyword for classification was applied in the following order: M>U>D>N. However, “sessile serrated adenoma/polyp,” “sessile serrated adenoma,” and “sessile serrated lesion” were prioritized over the keywords for class D. For example, certain diagnoses, such as “Adenocarcinoma, moderately differentiated,” “Neoplastic lesion, suspicious for malignancy,” “Tubular adenoma, high grade dysplasia,” “Tubulovillous adenoma, low to focal high grade dysplasia,” “Adenoma with focal carcinomatous change,” “Tubular adenoma, grade uncertain,” “Malignant neoplasm,” "Atypical glandular proliferation, favor neoplastic,” and “Neuroendocrine carcinoma” involve italicized keywords, but are ultimately classified into class M based on the bolded keywords. Diagnoses such as “Tubular adenoma, low grade dysplasia,” “Atypical glandular proliferation, favor dysplasia,” “Atypical glandular proliferation, indefinite for dysplasia,” and “Atypical glandular proliferation, undetermined significance” are classified as class D based on the rule described above. “Small cell nests with neuroendocrine feature” is also classified into class U, along with “Neuroendocrine tumor, grade 1 (carcinoid tumor).”

Based on the above-mentioned classification rule, if “Tubulovillous adenoma, low to focal high grade dysplasia” is considered as class M by pathologic diagnosis classification, but class D by WSI AI prediction, then this case is labeled as a “discordant” case, requiring review by a pathologist. Some ambiguous cases such as "Atypical glandular proliferation, favor neoplastic,” “Atypical glandular proliferation, undetermined significance,” and “Atypical glandular proliferation, indefinite for dysplasia” could be classified into class M, D, and D, respectively, by pathologic diagnosis classification. However, the AI prediction result can be of class M, D, or even N, depending on the characteristics of each WSI. Such “non-typical” cases are also labeled as discordant with the AI prediction, requiring review by a pathologist.

For “non-typical” cases described above, additional slides (serial cut or recut slides) are often prepared. In such cases, the AI prediction results for each WSI may vary. For example, the prediction result for the original slide may be of class D, but one serial cut slide may be classified as class D, another slide may be classified as class D, and a deeper cut slide may be classified as class N, with disappearance of the lesion (D, D, M, and N, respectively). When AI predictions of serial slides are prepared from a single block and they vary like this, the final prediction was determined in the order of M>U>D>N. In the aforementioned example, the AI prediction would indicate “M” as the final prediction. If the pathologic diagnosis is “TA, LGD,” it would be marked as “D,” which would also be considered a discordant case. For simply improving concordance between the pathologic diagnosis and AI prediction (accuracy), “D,” which appeared most often (among D, D, M, and N) in the prediction results of each WSI slide, could be used as the final class. However, for post-analytic QC and the rapid detection of FN, the more clinically serious result was prioritized over the more frequent AI prediction result.

Performance in the laboratory (KAIST) validation.

We trained the gastric and colorectal models using the methods described above. More specifically, those models were both based on a novel patch-based framework in which a WSI was split into multiple patches, the latent features from the patches were extracted for a patch-level classifier, and the patch-level features were aggregated for a slide-level classifier, both of which were based on DenseNet201. In laboratory tests, the WSI classifier achieved accuracy rates of 96.0% and 95.8% in classifying histopathological WSIs of gastric and colorectal endoscopic biopsy specimens, respectively, into three classes (Negative for dysplasia, Dysplasia, and Malignant). However, six WSIs were excluded from the assessment of the colorectal model due to errors. When these six WSIs were included, the accuracy dropped to 92.0%. Both models properly classified class N WSIs, but misclassified a small number of class D and M WSIs as class N (see S2 Table).

Performance in the in-house (SMF) validation.

Before the real-world application of the finally selected two models, some of the daily signed-out endoscopic biopsy slides were used as a pre-test set for in-house validation. The pre-test was conducted using daily slides data generated in the real-world to verify the practicality of the application of the models in daily practice by indirectly testing the performance in clinical practice with unbalanced data distribution. We used 491 gastric biopsy WSIs and 319 colorectal biopsy WSIs in the pre-test. The percentage of class N is expected to be high in real-world practice; thus, this test was designed to verify whether the models can stably show high accuracy under similar conditions. Accordingly, while the lab validation data were evenly distributed by class, the pre-test dataset had a relatively higher number of class N WSIs. Consequently, gastric and colorectal models showed accuracies of 93.08% and 95.30%, respectively, which are suitable for the real-world implementation of daily QC (see S3 Table).

Distribution of the classification by AI prediction in the trial run for daily QC.

In total, 26,133 WSIs were signed out by four pathologists and scanned during the 3-month period, comprising 13,630 gastric biopsy WSIs and 12,503 colorectal biopsy WSIs from 12,734 patients. Of these, there were 13 gastric biopsy WSIs and 40 colorectal biopsy WSIs that were not reviewed by the SeeDP daily QC system owing to scan failure (error) despite two or three re-scans. The distribution of AI predictions for specific pathologic diagnoses are shown in Tables 6 and 7. (The dataset of the present study are presented as S1 File).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Distribution of the classification by AI prediction of gastric biopsy WSIs.

https://doi.org/10.1371/journal.pone.0278542.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Distribution of the classification by AI prediction of colorectal biopsy WSIs.

https://doi.org/10.1371/journal.pone.0278542.t007

Analysis of cases discordant between the AI prediction and pathologic diagnosis.

In both models, false positive (FP) rate was somewhat high. In the gastric model, the FP rate was higher for H. Pylori-associated chronic gastritis (HCG) than chronic gastritis among gastritis diagnoses, with an increased FP rate in cases with findings of a secondary reaction, related to erosion, ulcers, regenerative changes, and inflamed granulation tissue (+ ero/u/reg/inflamed) compared to that in cases without such findings (Table 6). In the colorectal model, cases of inflammatory polyps, active inflammation, or suspicious for inflammatory bowel disease showed a higher FP rate than nonspecific or inactive inflammation cases (Table 7). Moreover, such cases occasionally showed red heat in areas with inflammatory reactions or severe activity accompanied by erosion or ulceration in the patch-level heatmap (S8.1 Fig in S8 Fig). Such cases can be used as training data to improve the performance of future models when the pathologic diagnosis is found to be correct during the review of discordant cases by pathologists.

In the gastric model, there were some negative cases that were falsely predicted as class D. These cases showed blue heat in some areas, such as foveolar epithelium with some darker stained areas, poor quality sections with knife marks, (S8.2 Fig in S8 Fig) and intestinal metaplasia with atrophy. As shown as Table 6, two of these cases were xanthomatous lesions with no heat in the foamy histiocytes (S8.2 Fig in S8 Fig). However, in one case, some red heat was observed in an area accompanied by erosion; thus, it was incorrectly predicted as class M. In this area of the slide, the stroma was pressed by erosion, making it appear more cellular, and the cytoplasm was more pinkish than other areas; however, red heat was not observed in other areas with typical foamy histiocytes (S8.3 Fig in S8 Fig).

Ambiguous diagnoses, such as AUS, and “indefinite for dysplasia (IFD)”, were classified as class D by pathologic diagnosis and as class D by AI model prediction in 61.54% of such cases (8/13 cases). During model training, AUS and IFD initially belonged to class U in the quaternary model, but were excluded from the training dataset when the model was converted to a ternary model. However, it was very interesting to find that the accuracy was lower for AUS/IFD cases than for LGD despite the current model classifying most of them as class D. Moreover, "dysplasia, grade uncertain" and HGD belonged to class D when defining the training dataset (Table 2), but were redefined as class M by pathologic diagnosis when applied to the SeeDP system daily QC (Table 4). The gastric model tended to follow the redefined pathologic diagnosis classification, while the colorectal model tended to favor class D for focal HGD. However, for HGD cases, the colorectal model tended to follow the redefined classification, like the gastric model (Tables 6 and 7).

In the gastric model, one FN case, diagnosed as TA, LGD but classified as class N, had a small lesion, along with slight focal blue and red heat in portions of the WSI, which may have been neglected during spatial analysis in the WSI prediction step (S8.4 Fig in S8 Fig). This phenomenon was observed more frequently in the prediction of class D in the colorectal model than in the gastric model. The FN rate of LGD colorectal biopsies, predicted as class N, was 12.14%, which was higher than that in the gastric model (1.14%). However, owing to the difference in clinical significance between gastric LGD and colorectal LGD (gastric LGD requires more aggressive treatment), the class D prediction tendency of both models was considered to be practically ideal (Tables 6 and 7).

NET cases showed a tendency to be predicted as class N in both models, while the heatmap patterns mostly showed no heat (S8.5 Fig in S8 Fig). The four cases that were predicted as class M in the colorectal model were accompanied by unnecessary red heat associated with aforementioned inflammatory reaction. However, one specific case that was predicted as class M in the gastric model was somewhat different (S8.6 Fig in S8 Fig). While low-magnification findings seemed to show NET, high-magnification findings indicated that this case needed to be differentiated from “oxyntic gland adenoma exhibiting infiltrative growing pattern” and “gastric adenocarcinoma of the fundic-gland type”. This case was descriptively diagnosed as “favor NET,” for which the need for differentiation from these diagnoses, along with ancillary tests, were recommended. For such cases, it may be better to advise caution by classifying them as class M rather than as class N; thus, the response of this model was considered to be reasonable.

There was one case of s/f mucosa-associated lymphoid tissue (MALT) lymphoma within the daily QC set, which was predicted as class N with no heat in the WSI. The case involved HCG accompanied by lymphoid follicles (LFs), and localized lymphoepithelial body-like lesions were found near the LFs. The case was signed out with a descriptive diagnosis of Wotherspoon grade 3 (S8.7 Fig in S8 Fig). Although some FPs were occasionally found, which were predicated as class M with red heat in severe gastritis with LFs or lymphoid aggregate, in this case, a negative prediction was made.

Our system was designed so that AI models could be actively used without excess change to the conventional workflow of the diagnostic process.

We opted for a method that could quickly detect human errors that may occur during pathologic interpretation, allowing immediate corrections without affecting the actual interpretation process. In the daily practice of pathologists, GI endoscopic biopsy slides are provided without distinction between gastric and colorectal biopsy specimens; and are interpreted according to their serial number. In many cases, gastric and colorectal biopsy specimens from the same patient are received and processed consecutively; it is not suitable to process such slides by differentiating them according to organ. However, the SeeDP program is designed to selectively run the AI model corresponding to each organ based on the organ information listed in the heading of each specimen. Therefore, when pathologists review slides through the AI-assisted daily QC system, they can review them by serial number as well. Additionally, when pathologists perform diagnoses through WSIs and want to refer to the prediction results of the AI model, they can follow the receipt number.

However, if the heading information and the organ do not match, the model designed for that organ cannot be run without revising the heading. Moreover, the models cannot be run for other organs that may be present together in the GI endoscopic biopsy specimens, such as the esophagus, duodenum, and anus. Furthermore, for various diseases belonging to class U (e.g., NET, MALT, submucosal tumor, inflammatory bowel disease, and serrated lesions), which were initially excluded from the study, our current AI models do not yet provide a cautionary warning. Therefore, further research is needed to overcome this limitation (i.e. including other diseases). Therefore, we are currently investigating an upgrade to a quaternary classifier model that would include NET grade 1 or 2.

Our goal was 100% practical application.

The results of our study were obtained by scanning all microscopic slides signed out daily and running AI models, instead of simply selecting interesting cases for assistance with AI models. We aimed to develop AI models for all GI endoscopic biopsy specimens to achieve 100% practical application in daily QC. During the trial run period, all endoscopic biopsy slides signed out by four pathologists were 100% digitalized and almost all were double checked by the AI models. However, the system is still not applicable for non-endoscopic biopsy and cytology specimens. Additional research and development efforts are needed for other organs and all specimens received for testing.

We developed our own post-analytic daily QC system with an AI-assisted DP total solution (SeeDP) that integrates a WSI viewer and AI models, combined with a pathology LIS.

First, for the early detection of potential human errors in microscopic interpretations based on conventional methods, we designed a system to differentiate specimens based on the heading information in the pathologic report and automatically labelling and applying either the gastric or colorectal model, as suitable for that specimen. Next, we set major keywords in the text information of the pathologic diagnosis for automatic classification of cases into three classes (classes M, D, and N). Moreover, the prediction results from the AI model suitable for each specimen were exposed to the text classification of M, D, and N for comparison against classification by pathologic diagnosis to check for concordance. Discordant cases were listed first, so that pathologists could prioritize their review. When the pathologists reviewed these cases, the AI prediction results were not visualized by heatmaps provided separately from WSIs; rather, the heatmaps were masked on or off directly on top of WSIs to enable an intuitive comparison between the WSI and heatmap at all magnifications. Lastly, we included functions that enabled pathologists to assess the AI prediction results for reviewed cases and leave their feedback. By continuing to collect FN and FP cases and using them as an additional model training resources, we were able to establish a system with continuous improvement in model performance.

However, the system cannot detect cases in which both the AI prediction and pathologic diagnosis are FNs. Therefore, the models we developed and implemented were designed to maintain high NPV to minimize such risk.

We developed a powerful and clinically influential system that can ultimately help enhance patient safety.

Numerous researchers have contemplated and suggested improvements in, and the standardization of, test methods that can clinically enhance patient safety. In particular, in the field of gastroenterology, an endoscopy is performed first; if necessary, a tissue biopsy is performed to obtain a confirmative diagnosis through a histopathological examination. In this process, a discrepancy may exist between the endoscopic findings and the histopathologic diagnosis, and patient safety is not assured given this inconsistency. Therefore, studies to enhance the patient’s stability from the stage prior to biopsy have been continuously conducted, and recent studies using AI are noteworthy [54, 55]. Similarly, pathologic laboratory QC ensures high reliability in the quality of testing, which ultimately helps in enhancing patient safety [10]. The monthly review of random slides, which is the most popular method among conventional post-analytic QC methods, serves an important internal function for narrowing the discrepancies between pathologists by blinded reviews, communication of discordant cases, and regular sharing of feedback [9]. In many cases, however, reviewers tend to ignore or dismiss minor discrepancies in existing diagnoses and in their approach to the review process. This may be because the time gap between the original diagnosis and review is at least one month or longer. Therefore, although the random slide review may be a key internal function, it often does not have direct clinical influence on patients or the clinicians ordering the test. Additionally, 1–2 months may have already passed since the initial diagnosis of a severe discordance. Therefore, even if the corrected diagnosis is conveyed to the clinician or the patient, the previous FN may have deprived the patient of appropriate treatment.

Improvements in pathology laboratory QC achieved by combining our AI-assisted daily QC system with a monthly review of random slides can be easily inferred through the comparisons in Table 8. In summary, the currently used monthly random slide reviews, performed by an average of 14 pathologists during the 33 months prior to the SeeDP AI-assisted daily QC system trial run had relatively little practical influence on detecting or correcting hidden errors in pathologic diagnoses. In contrast, 11 hidden errors were corrected within an average of 40.2 days by slide reviews compared to the methods used during that 33-month period. Further, eight cases of errors were detected within 1.2 days by four pathologists during the 3-month SeeDP AI-assisted daily QC system trial run period, and corrections were made within an average of 3.4 days. As a comparison, when the duration and number of human pathologists were roughly adjusted based on these results, we found that approximately 7–10 times as many slides could be reviewed by pathologists, compared with conventional methods and nearly 100% of GI endoscopy slides can be double-checked by the AI models; approximately 17–30 times as many current potential human errors could be detected within an average of 1.2 days. Consequently, more active measures could be expected before the erroneous pathology report is conveyed to the patient. These findings indicate that the AI-assisted daily QC system is a direct and powerful tool for improving daily QC, in quantitative, qualitative, and time utility aspects, while also being clinically influential, from a patient safety aspect.

As seen in Table 9, errors detected and corrected by the AI models were mostly human errors, rather than those caused by the diagnostic ability of the pathologist or the difficulty of the diagnosis itself. Such potential human errors are often difficult to recognize by medical record officers or pathologists by simply reviewing the diagnostic information, especially when lesions are missed in the microscopic FOV or when an incorrect diagnosis is made without typographical errors. Many studies have shown that AI models can improve the diagnostic accuracy of cases that are difficult to diagnose, and have a positive impact on junior pathologists and trainees [29, 56]. However, they do not provide substantive assistance to pathologists with sufficient experience. Additionally, because the concept of ambiguity always exists in the field of pathology, many pathologists worldwide are continuously endeavoring to reduce discrepancies through periodic discussions and agreements; this problem may be an everlasting obstacle for pathologists to overcome. Further, most pathologists try to perform an accurate diagnosis (or minimize the possibility of a misdiagnosis) when they encounter cases that are considered difficult to diagnose. Some of the actions pathologists take for patients (and for themselves too) include preparing recut or serial cuts, ordering ancillary tests, referring to past slides, thoroughly reviewing medical records, seeking secondary opinions or advice from experts, reviewing similar cases, and looking up related literature. It is doubtful that the AI model could considerably replace the diagnostic capabilities of pathologists compared to the result of the previously mentioned accumulated efforts. The real problem may be when pathologists sign out in daily practice without realizing the possibility of making mistakes. As discussed above, unpredictable simple human errors ultimately result in errors that are difficult to detect in conventional monthly random reviews. A lot of such hidden human errors, like those mentioned above have likely persisted, and some of these hidden errors can baffle pathologists months or years later. Moreover, patients may have to face unnecessary burdens.