Evidence has shown that deep learning computer aided detection (CADe) system achieved high overall detection accuracy for polyp detection during colonoscopy.
The detection performance of CADe system on non-polypoid laterally spreading tumors (LSTs) and sessile serrated adenomas/polyps (SSA/Ps), with higher risk for malignancy transformation and miss rate, has not been exclusively investigated.
A previously validated deep learning CADe system for polyp detection was tested exclusively on LSTs and SSA/Ps. 1451 LST images from 184 patients were collected between July 2015 and January 2019, 82 SSA/Ps videos from 26 patients were collected between September 2018 and January 2019. The per-frame sensitivity and per-lesion sensitivity were calculated.
(1) For LSTs image dataset, the system achieved an overall per-image sensitivity and per-lesion sensitivity of 94.07% (1365/1451) and 98.99% (197/199) respectively. The per-frame sensitivity for LST-G(H), LST-G(M), LST-NG(F), LST-NG(PD) was 93.97% (343/365), 98.72% (692/701), 85.71% (324/378) and 85.71% (6/7) respectively. The per-lesion sensitivity of each subgroup was 100.00% (71/71), 100.00% (64/64), 98.31% (58/59) and 80.00% (4/5). (2) For SSA/Ps video dataset, the system achieved an overall per-frame sensitivity and per-lesion sensitivity of 84.10% (15883/18885) and 100.00% (42/42), respectively.
Citation: Zhou G, Xiao X, Tu M, Liu P, Yang D, Liu X, et al. (2020) Computer aided detection for laterally spreading tumors and sessile serrated adenomas during colonoscopy. PLoS ONE 15(4): e0231880. https://doi.org/10.1371/journal.pone.0231880
Editor: Wajid Mumtaz, National University of Sciences and Technology, PAKISTAN
Received: January 2, 2020; Accepted: April 2, 2020; Published: April 21, 2020
Copyright: © 2020 Zhou et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: The CADe system (EndoScreener) was developed by Shanghai Wision AI Co., Ltd. The system was provided free-of-charge for this study. Employees in the company were not involved in the study in any way, including in study design, statistical analysis or manuscript writing. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Colonoscopy is the gold-standard diagnostic tool for colorectal cancer (CRC) and precancerous lesions. However, the adenoma miss-rate (AMR) ranges from 6% to 27%. It is noteworthy that precancerous lesions with a flat morphology, smooth surface, indistinct boundaries or isochromatic with background are more susceptible to be missed[2, 3, 4]. Given these subtle features of morphology and color, nonpolypoid laterally spreading tumors (LSTs) and sessile serrated adenoma/polyps (SSA/Ps) are among the easist-to-miss lesions[5, 6]. LSTs extend laterally along the colon wall without a polypoid morphology[7, 8]. It could be categorized into 2 types as granular type (LST-G) and non-granular type (LST-NG). The LSTs have a remarkable high risk of malignancy transformation, studies have shown that 63.1% of the investigated LSTs are adenomas with villous structures 20.9%-36.0% of LSTs were found to have high grade intraepithelial neoplasia (HGIN), moreover, LSTs can also develop into a submucosal invasive cancer[6, 9, 10]. Serrated polyps (SPs) are the second most common type of colon polyps, among which, SSA/Ps take about 10–20% of SPs[11, 12] and 5–10% of lesions found during screening colonoscopy. SSA/Ps are a high-risk precursor for CRC via serrated pathway[5, 13, 14], the estimated 10-year risk for CRC transformation from SSA/Ps ranges from 2.56% to 4.43% according to the existence of cytological dysplasia and are dependent on the size. The detection of SSA/Ps via colonoscopy is always difficult due to the pale color and flat morphology.
Evidence has shown that with a second party observing the monitor synchronously during colonoscopy, polyp detection rate could be improved by means of addressing the unrecognized polyps within the visual field. However, in comparison with polypoid lesions, to detect LSTs and SSA/Ps is a more challenging task, it is likely that adding an additional human observer would not completely overcome the challenges. Thanks to the breakthrough of artificial intelligence (AI), ideally, real-time computer aided detection (CADe) system during colonoscopy with a comparable detection capability to expert endoscopist could process each frame of the endoscopic video stream and alert any detected targets synchronously, this is a more consistent and reliable set-up than a human assistant, because CADe system would consistently analyze every corner of the screen and not miss any single frame of a suspicious lesion. Our CADe system has been previously demonstrated to improve polyp and adenoma detection rate in real world colonoscopy setting. The CADe system was especially developed to capture and identify the local features of colon polyps of variant histology and thus to achieve a high sensitivity and specificity. However, the large-scaled specialized test of such CADe system on LSTs and SSA/Ps is lacking. Therefore, the aim of this study was to investigate the detection performance of the local-feature-prioritized CADe system on LSTs and SSA/Ps.
Materials and methods
CADe system (EndoScreener, Shanghai Wision AI Co., Ltd. China)
The real-time automatic polyp detection system (Fig 1) was developed on a deep learning architecture, which was previously validated to have a per-image sensitivity of 94.38%, per-image specificity of 95.92% and an area under the receiver operating characteristic curve of 0.984 on consecutive colonoscopy images. In addition, the algorithm has a per-polyp sensitivity of 100.00% (per-image sensitivity of 91.64%) and a per-image specificity of 95.40% in colonoscopy videos.
a. Original colonoscopy image frames generated during regular colonoscopy procedures. b. Deep convolutional neural network (CNN): SegNet architecture. (http://mi.eng.cam.ac.uk/projects/segnet/), which calculates the probability of. belonging to a polyp for each pixel in the input colonoscopy image frame. c. Probability map, which showed the probability of belonging to a polyp (color blue represents probability = 0, color red represents probability = 1, color in between represents 0<probability<1), for each pixel in the input image frame. d. Based on the probability map, hollow blue boxes are added to the original image, in order to highlight the polyp areas for the observing clinicians.
In the preliminary study, the algorithm was trained on 5545 colonoscopy images with variant polyp morphology and histology, among which there were 103 images of 69 LSTs, including 56 images of 34 LST-G and 47 images of 35 LST-NG.
To better distinguish the intra-class and inter-class variability and thus to achieve high sensitivity, specificity and to effectively detect partially occluded polyps, this network structure of the algorithm is characterized by its small receptive fields that extract the local visual features of the input image, as well as the high nonlinearity of its consecutive convolution layers without pooling.
The system provides a per-pixel labeling of its detected area on each processed frame.
This study was conducted in the endoscopy center of Sichuan provincial people’s hospital, China.
We retrospectively collected images of LSTs and video clips of SSA/Ps from the database of our center which were not included for the development of EndoScreener during 2017 from 2 endoscopy centers. 1451 images of 199 LSTs from 184 patients were obtained from July 2015 to January 2019. 82 videos of SSA/Ps based on pathological diagnosis from 26 patients were obtained from September 2018 to January 2019, each SSA/Ps video was cut into clip from the first frame of its appearance till the last frame it disappeared. The average length of the videos was 17.99-second-long. All the images and videos in the database were acquired from Olympus EVIS LUCERA CV260 (SL)/CV290 (SL) and Fujifilm VP-4450 HD and adapted high-quality and high definition colonoscopes.
Basic demographic characteristics including gender and age; the sizes, locations and histological results of each lesion were illustrated.
We used the CADe system to process each image of LSTs dataset and video clips of SSA/Ps datasets, the system provided a per-pixel labeling of its detected area. The evaluation of the algorithm is based on the consensus among a 3-person panel of senior endoscopists in our center who carefully re-checked each image of the labeled LSTs dataset and each frame of the labeled SSA/Ps video dataset. For image analysis on LSTs, the panel of experienced endoscopists recorded the number of correctly labeled LSTs missed LSTs. Then the panel labeled the border of each LST by hand. For SSA/Ps video assessment, the panel of experienced endoscopists carefully re-checked each labelled frame in the video and recorded the number of correctly labeled polyps and missed polyps. Authors had no access to information that could identify individual participants during or after data collection.
This study was reviewed and approved by institutional review board (Ethics committee) of Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital on May 17, 2017.
Written informed consents were obtained from all individuals who were detected LSTs and SSA/Ps during colonoscopy and approved to provide the lesions’ images and videos for this study.
All methods were carried out in accordance with relevant regulations of Sichuan Academy of Medical Sciences & Sichuan Provincial People’s Hospital.
The study protocol conforms to the ethical guidelines of the 1975 Declaration of Helsinki as reflected in a priori approval by the institution’s human research committee.
The main outcome is per-image or per-frame sensitivity and per-lesion sensitivity. The secondary outcome is ratio of correctly labeled area.
If the label of algorithm detection is on an actual lesion (LSTs or SSA/Ps according to the tested datasets), then it is considered a true positive (TP) and only one true positive will be counted for each actual lesion, regardless of how many algorithm labels fall on that lesion. The absence of the algorithm label on an actual lesion is counted as one false negative (FN). Therefore, the total number of true positives and false negatives is the total number of lesion appearance in the validation images or frames. Therefore, per-frame sensitivity was defined as TP divided by total number of polyp appearances = TP / (TP+FN). This is the commonly accepted statistical method for evaluating image detection.
Because in real clinical setting, intermittent detections could also lead to an effective notice to the endoscopist and thus to avoid a miss diagnosis, we used per-lesion sensitivity to show the pragmatic detection capability of the CADe system. The per-lesion sensitivity is defined as the number of lesions correctly detected by the algorithm in at least one frame divided by the total number of actual lesions.
To measure how well the algorithm can detect the full range of an LST, we also measured the coverage ratio of correctly labeled area, which is defined as the number of pixels of the correctly labeled area which were confirmed by the endoscopist-panel, divided by the number of pixels of the panel-labeled area of each lesion on each frame.
Baseline data of the tested datasets (Table 1)
In the LSTs image dataset, there were 1451 images containing LSTs of 199 LSTs lesions from 184 patients, among which 90 (50.00%) were female, and the mean age was 63.11±11.47(standard deviation, s.d.). There were 24 carcinomas (12.06%), 16 SSA/Ps (8.04%), 148 conventional adenomas (74.37%), 77 advanced adenomas (38.69%), and 11 hyperplastic or inflammatory component (5.53%). The average diameter of all the LSTs was 2.33±1.13cm.
In the SSA/Ps Video Dataset, there were 82 colonoscopy video clips of 42 SSA/Ps lesions with 18885 frames in total, each with an SSA/Ps appearing from the beginning until the end. The length of the dataset is 12.59 minutes in total and 17.99s per lesion on average. These videos were obtained from 26 patients, including 7 females (26.92%). The mean age was 50.81±10.07. All of the SSA/Ps were Is in morphology according to Paris classification system.
In the SSA/Ps video dataset, there are 29 (69.05%) diminutive lesions (≤0.5cm), 12 (28.57%) small lesions (<0.5cm, < = 1cm), and one (2.38%) large lesion. Other baseline information are presented in Table 1.
The detection sensitivity and ratio of correctly labeled area on LSTs (Table 2)
LSTs were categorized into 4 subgroups which are LST-G(H), LST-G(M), LST-NG(F), and LST-NG(PD), the per-image sensitivity of each subgroup was 93.97% (343/365), 98.72% (692/701), 85.71% (324/378) and 85.71% (6/7), while the per-lesion sensitivity of each subgroup was 100.00% (71/71), 100.00% (64/64), 98.31% (58/59) and 80.00% (4/5). The overall per-image sensitivity and per-lesion sensitivity of LSTs were 94.07% (1365/1451) and 98.99% (197/199) respectively. 2 lesions were missed by CADe system in LST-NG(F) and LST-NG(PD) subgroup separately.
The ratio of correctly labeled area for each subgroup was LST-G(H) 35.66%, LST-G(M) 41.49%, LST-NG(F) 35.6%, LST-NG(PD) 43.47%.
The detection sensitivity on SSA/Ps (Table 3)
According to the size, the SSA/Ps was classified into 3 subtypes: diminutive (≤0.5cm), small (>0.5cm&≤1cm) and large (>1cm), the CADe system had achieved a per-frame sensitivity of each subgroup as 80.29% (10356/12899), 92.90% (5080/5468), 86.29% (447/518). The per-lesion sensitivity for each subgroup was 100.00% (29/29), 100.00% (12/12) and 100.00% (1/1) respectively. The overall per-frame sensitivity and per-lesion sensitivity of SSA/Ps were 84.10% (15883/18885) and 100.00% (42/42).
CRC is one of the leading causes of cancer related death[21, 22] and is a major public health issue given its high incidence and mortality rate. Screening colonoscopies have allowed a significant decrease of mortality rate and incidence of CRC in adults (by 51% and 32%, respectively). colonoscopy has also allowed an increase in 5-year survival rate in CRC, mainly as a result of early detection as well as removal of precancerous lesions. Missed precancerous lesions, including conventional adenomas, SSA/Ps as well as LSTs with various pathology, during colonoscopy might lead to subsequent colorectal cancer. Miss diagnosis of any precancerous lesions should be decreased because evidence has shown that less interval cancer could be expected under endoscopists with high adenoma detection rate (ADR)[24, 25]. Although conventional adenomatous polyps with a protruding morphology and redder color contributes the majority of ADR, the detection of non-polypoid LSTs and isochromatic SSA/Ps is equally important because they are even more susceptible to be missedand some of them are more risky for malignant transformation. Given the importance of better identification of any precancerous lesions during colonoscopy, tremendous efforts from both hardware engineers and clinicians have been put to make lesions more visible to the endoscopists, However, endoscopists still miss a large portion of precancerous lesions within the screen, for both lesions with non-obvious visual features and lesions that briefly flashed across the screen. These conditions are very challenging even for experienced endoscopists, because humans are always susceptible to “inattentional blindness”, “change blindness”, visual gaze pattern, fatigue, inter-observer variabilities or any distractions [18, 27, 28, 29, 30, 31, 32]. Till now, only second-observer strategies seem helpful to decrease the miss rate of visible polyps for low ADR detectors[26, 33]. However, adding a second human observer might not completely overcome the challenges.
In comparison with human observer, high-performance CADe has high reproducibility, fidelity and uniformity. These advantages are especially important for detecting lesions that are non-obvious, briefly visible or partially occluded.
Therefore, it is crucially important to investigate the detection performance exclusively on LSTs and SSA/Ps of any CADe system which was developed for screening early cancers and pre-cancerous lesions, because if a CADe system is only capable to detect polypoid lesions, then its clinical value for decreasing interval cancer is inadequate. In the preliminary technical study, we have considered the importance of identifying the local features of a lesion, as pattern of pits and micro-vessels, and thus to build an algorithm that relies less on overall morphology This strategy is especially helpful for identifying flat lesions. Moreover, when a lesion flashed across the edge of the screen or partially behind a fold, an algorithm that is trained more on local features and does not rely on the full appearance may still detect it by capturing the detailed features. Therefore, such CADe system should be assumed to have a good performance in detecting LSTs and SSA/Ps.
In this study, it is easy for the CADe system to detect the LST-G due to multiple polypoid granules. As a result, the 93.97% per-frame sensitivity for LST-G-H (homogeneous type) and 98.72% per-frame sensitivity for LST-G-M (nodular mixed type) was comparable with the per-frame sensitivity for all kinds of polyps in our preliminary study. However, LST-NG is difficult to detect because of the absence of any polypoid protuberance. Nevertheless, the surface texture of LST-NG is distinguishable because most of the LSTs are conventional adenomas in pathology and should have the same micro-surface features. The CADe system achieved an 85.71% per-frame sensitivity for LST-NG-F (flat type), 85.71% for LST-NG-PD (pseudo-depressed type), the system missed one lesion in each subgroup.
The ratio of correctly labeled area for the 4 subgroups ranges from 35.6% to 43.47%. The CADe system detected well on the protruding part and the margin. Moreover, when lesion is observed at a close distance, detailed micro-surface structures are identifiable, the CADe system could also detect well on the smooth regions (Figs 2 and 3). However, the CADe system failed to completely label the smooth area when a lesion was photographed in far distance when micro-surface structures is visually indistinguishable, it was also challenging for the system to detect the smooth area when the color of the lesion was similar to the background. This is attributed to inadequate visual features. New technology and detection strategy are needed to be deployed to better identify flat and isochromatic lesions that lie far from the camera. Noteworthy, although partially tagged, the alerts generated by the CADe system may still effectively notice the endoscopist and avoid a miss during real clinical setting.
Green tags are per-pixel predictions of the system in 4 subgroups of LSTs.
Green tags are per-pixel predictions of the system on LSTs with different pathology, as hyperplastic, inflammatory, tubular adenoma, villous adenoma, serrated adenoma and adenocarcinoma.
SSA/Ps are usually flat with an unclear boundary, the color the SSA/Ps are commonly whitish or isochromatic to the background, the micro-surface structures are similar with hyperplastic polyps under white light colonoscopy[35, 36]. Hence, the overall per-frame sensitivity was no more than 90%, the 80.29% sensitivity (S1–S4 Videos) for small SSA/Ps subgroup was much lower than the 91.64% sensitivity for all kinds of polyps in the preliminary study. The per-lesion sensitivity was 100.0%, nevertheless, the system might miss a briefly visible lesion because it misses one fifth of the frames. Therefore, it is important to improve the detection performance by adding more training data for the CADe system to “learn” the unique II-O pit and varicose microvascular vessel (VMV) of SSA/Ps. Nevertheless, the performance was consistent with the previous technical study, the same CADe system with such a sensitivity for SSA/Ps has increased the deetction rate of diminutive polyps significantly.
However, although adding more training data might be helpful for the CADe system to distinguish subtle inter-class visual features, it will be extremely difficult to obtain a marginal benefit as well as to suppress false positive rate, because when a lesion lies far from the camera, it is challenging for any CADe system or human expert to be specific due to inadequate pixels in the image. Further improvement may be obtained by double-AI CADe system, of which one AI captures local feature of definite lesions which lie close, a second AI independently captures other features of suspected lesions which lie far. Therefore, such double-AI strategy should be more specific for definite lesions and won’t compromise the sensitivity for suspected lesion which lies far. Moreover, the distinguished display of definite and suspected lesions could be more comprehensible, and endoscopists can rely on the new system instead of being confused by false alarms or dissatisfied by the insufficient sensitivity for suspected lesions.
In addition, it is important to notice that the current CADe system should be used as a supplement quality assurance tool, because none of them has achieved 100% accuracy. The endoscopists should carefully inspect the colon firstly by themselves instead of fully relying on the detection of the system. The users should be aware of the system-undetected lesions with a basic attention. Therefore, it is crucially important for a CADe system to be well developed and rigorously validated. On one hand, if and only if a CADe system has a good performance to detect non-obvious lesions, it could be considered clinical helpful. On the other hand, the validation dataset of such system should be representative with a large sample size and various target lesions. Moreover, the validation dataset should be obtained from real clinical setting and should contains consecutive unselected unaltered images or videos after the development of the algorithm. Only this, the result, i.e sensitivity and specificity, of the system’s performance is meaningful and the endoscopist will accurately know how much he can rely on a CADe system. Last but not the least, a post-colonoscopy double check of the system labeled videos by senior endoscopists should be considered to make sure full benefit of the patients.
This study has several limitations, firstly, there were not sufficient samples of LST-NG-PD images and large size SSA/Ps videos, the detection result might not be generalizable for these subtypes, more data should be added to evaluating the detection performance of the CADe system on these subtypes. Secondly, as a limitation of current calculation power of GPU, the current system can only extract a small portion of pixels from these lesions and thus might miss some detailed characteristics after extraction. Therefore, the system might not be so sensitive for lesions lie far from the camera when the number of pixels is inadequate, this will be addressed along with the hardware development. Thirdly, we did not assess the specificity of the system in this study, because the algorithm was unchanged as in the preliminary study, in which the specificity has been tested on larger image and video datasets.
The local-feature-prioritized polyp detection CADe system could detect LSTs and SSA/Ps with a high sensitivity which indicates it might be a promising quality assurance during colonoscopy to reducing miss diagnosis of easy-to-miss precancerous lesions along with conventional polyps and thus has the potential to decrease the risk of interval cancer. The impact of such CADe system on the detection of LSTs and SSA/Ps in real clinical setting should be specifically investigated in large-scaled clinical trials.
Dr. Xiao Xiao, Zhiwei Zhang and Jingjia Liu from Shanghai Wision AI Co., Ltd. developed the CADe system according to the study design and provided technical description of the technology. We thank Dr. Wenfei Zhang for the advice on statistical analysis.
- 1. Mahmud N, Cohen J, Tsourides K, et al. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterol Rep 2015;3:179–84
- 2. Rex DK, Cutler CS, Lemmel GT, et al. Colonoscopic miss rates of adenomas determined by back-to-back colonoscopies. Gastroenterology 1997;112:24–28. pmid:8978338
- 3. Heresbach D, Barrioz T, Lapalus MG, et al. Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies. Endoscopy 2008; 40:284–290. pmid:18389446
- 4. Wang P, Liu X, Berzin TM et al. Effect of a deep-learning computer-aided detection system on adenoma detection during colonoscopy (CADe-DB trial): a double-blind randomised study. Lancet Gastroenterol Hepatol. 2020 Jan 22. [Epub ahead of print]
- 5. Fan C, Younis A, Bookhout CE, et al. Management of Serrated Polyps of the Colon. Curr Treat Options Gastroenterol 2018;16:182–202 pmid:29445907
- 6. Zhao X, Zhan Q, Xiang L, et al. Clinicopathological characteristics of laterally spreading colorectal tumor. PLoS One 2014;9:e94552. pmid:24751926
- 7. Hurlstone D, Sanders D, Cross S, et al. Colonoscopic resection of lateral spreading tumours: a prospective analysis of endoscopic mucosal resection. Gut 2004;53:1334–9. pmid:15306595
- 8. Kudo S. Endoscopic mucosal resection of flat and depressed types of early colorectal cancer. Endoscopy 1993;25:455–61 pmid:8261988
- 9. Rotondano G, Bianco MA, Buffoli F, et al. The Cooperative Italian FLIN Study Group: prevalence and clinico-pathological features of colorectal laterally spreading tumors. Endoscopy 2011;43:856–61. pmid:21826628
- 10. Uraoka T, Saito Y, Matsuda T, et al. Endoscopic indications for endoscopic mucosal resection of laterally spreading tumours in the colorectum. Gut 2006;55:1592–7. pmid:16682427
- 11. Hetzel JT, Huang CS, Coukos JA, et al. Variation in the detection of serrated polyps in an average risk colorectal cancer screening cohort. Am J Gastroenterol 2010;105:2656–64. pmid:20717107
- 12. Szylberg L, Janiczek M, Popiel A, Marszalek A. Serrated polyps and their alternative pathway to the colorectal cancer: a systematic review. Gastroenterol Res Pract 2015;573814. pmid:25945086
- 13. Kahi CJ. Screening Relevance of Sessile Serrated Polyps. Clin Endosc 2019; 52: 235–238. pmid:30625264
- 14. Cenaj O, Gibson J, Odze RD. Clinicopathologic and outcome study of sessile serrated adenomas/polyps with serrated versus intestinal dysplasia. Mod Pathol 2018;31:633–642. pmid:29271414
- 15. Erichsen R, Baron JA, Hamilton-Dutoit SJ, et al. Increased risk of colorectal cancer development among patients with serrated polyps. Gastroenterology 2016;150:895–902.e5 pmid:26677986
- 16. Burgess NG, Pellise M, Nanda KS, et al. Clinical and endoscopic predictors of cytological dysplasia or cancer in a prospective multicentre study of large sessile serrated adenomas/polyps. Gut 2016;65:437–46. pmid:25731869
- 17. Limketkai BN, Lam-Himlin D, Arnold CA, Arnold MA. The cutting edge of serrated polyps: a practical guide to approaching and managing serrated colon polyps. Gastrointest Endosc 2013;77:360–75. pmid:23410696
- 18. Wang P, Berzin TM, Glissen Brown JR, et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut 2019. 68:1813–1819. pmid:30814121
- 19. Wang P, Xiao X, Glissen Brown JR, et al. Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy. Nature Biomedical Engineering 2018;2:741–748 pmid:31015647
- 20. Bandos AI, Rockette HE, Song T, Gur D. Area under the free response ROC curve (FROC) and a related summary index. Biometrics 2009;65:247–256 pmid:18479482
- 21. American Cancer Society. Cancer Facts and Figures: 2017. Atlanta, Georgia. 2017. https://www.cancer.org/research/cancer-facts-statistics/all-cancer-facts-figures/cancer-facts-fgures-2017.html. Accessed August 15, 2017
- 22. Fang JY, Zheng S, Jiang B, et al. Consensus on the Prevention, Screening, Early Diagnosis and Treatment of Colorectal Tumors in China: Chinese Society of Gastroenterology, October 14–15, 2011, Shanghai, China. Gastrointest Tumors 2014;1:53–75 pmid:26672726
- 23. Burke C, Kaul V, Pohl H. Polyp resection and removal procedures: insights from the 2017 digestive disease week. Gastroenterol Hepatol 2017;13:1–24.
- 24. Corley DA, Jensen CD, Marks AR, et al. Adenoma detection rate and risk of colorectal cancer and death. N Engl J Med 2014; 370:1298–1306 pmid:24693890
- 25. Kaminski MF, Regula J, Kraszewska E, et al. Quality indicators for colonoscopy and the risk of interval cancer. N Engl J Med 2010;362:1795–1803 pmid:20463339
- 26. Aslanian HR, Shieh FK, Chan FW, et al. Nurse observation during colonoscopy increases polyp detection: a randomized prospective study. Am J Gastroenterol 2013;108:166–172 pmid:23381064
- 27. Rogart JN, Siddiqui UD, Jamidar PA, et al. Fellow involvement may increase adenoma detection rates during colonoscopy. Am J Gastroenterol 2008;103:2841–6 pmid:18759826
- 28. Wolfe JM, Reinecke A, Brawn P. Why don’t we see changes?: the role of attentional bottlenecks and limited visual memory. Vis cogn 2006;14:749–80 pmid:18974792
- 29. Memmert D, Unkelbach C, Ganns S. the impact of regulatory ft on performance in an inattentional blindness paradigm. J Gen Psychol 2010;137:129–39 pmid:20441129
- 30. Simons DJ, Chabris CF. Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 1999;28:1059–74 pmid:10694957
- 31. Simons DJ, Rensink RA. change blindness: past, present, and future. Trends Cogn Sci 2005;9:16–20 pmid:15639436
- 32. Lami M, Singh H, Dilley JH. Gaze patterns hold key to unlocking successful search strategies and increasing polyp detection rate in colonoscopy. Endoscopy 2018;50:701–707 pmid:29415286
- 33. Tziatzios G, Gkolfakis P, Triantafyllou K. Effect of fellow involvement on colonoscopy outcomes: A systematic review and meta-analysis. Dig Liver Dis 2019; 51:1079–1085 pmid:31272937
- 34. Alagappan M, Brown JRG, Mori Y, et al. Artificial intelligence in gastrointestinal endoscopy: The future is almost here. World J Gastrointest Endosc 2018;10:239–249. pmid:30364792
- 35. Hewett DG, Kaltenbach T, Sano T, et al. Validation of a simple classification system for endoscopic diagnosis of small colorectal polyps using narrow-band imaging. Gastroenterology 2012;143: 599–607 pmid:22609383
- 36. Sano Y, Tanaka S, Shigita K, et al. Narrow-band imaging (NBI) magnifying endoscopic classification of colorectal tumors proposed by the Japan NBI Expert Team. Dig Endosc 2016, 85: 822–828
- 37. Uraoka T, Higashi R, Horii J, et al. Prospective evaluation of endoscopic criteria characteristic of sessile serrated adenomas/polyps. J Gastroenterol 2015; 50:555–63 pmid:25270966