https://orcid.org/0000-0003-0128-4917
Figures
Abstract
Background
Intraoperative neurophysiological monitoring is essential in neurosurgical procedures. In this study, we built and evaluated the performance of a deep neural network in differentiating between the presence and absence of a lateral spread response, which provides critical information during microvascular decompression surgery for the treatment of hemifacial spasm using intraoperatively acquired electromyography images.
Methods and findings
A total of 3,674 image screenshots of monitoring devices from 50 patients were prepared, preprocessed, and then adopted into training and validation sets. A deep neural network was constructed using current-standard, off-the-shelf tools. The neural network correctly differentiated 50 test images (accuracy, 100%; area under the curve, 0.96) collected from 25 patients whose data were never exposed to the neural network during training or validation. The accuracy of the network was equivalent to that of the neuromonitoring technologists (p = 0.3013) and higher than that of neurosurgeons experienced in hemifacial spasm (p < 0.0001). Heatmaps obtained to highlight the key region of interest achieved a level similar to that of trained human professionals. Provisional clinical application showed that the neural network was preferable as an auxiliary tool.
Conclusions
A deep neural network trained on a dataset of intraoperatively collected electromyography data could classify the presence and absence of the lateral spread response with equivalent performance to human professionals. Well-designated applications based upon the neural network may provide useful auxiliary tools for surgical teams during operations.
Citation: Kim M, Park S-K, Kubota Y, Lee S, Park K, Kong D-S (2022) Applying a deep convolutional neural network to monitor the lateral spread response during microvascular surgery for hemifacial spasm. PLoS ONE 17(11): e0276378. https://doi.org/10.1371/journal.pone.0276378
Editor: Anwar P.P. Abdul Majeed, Universiti Malaysia Pahang, MALAYSIA
Received: August 10, 2021; Accepted: October 6, 2022; Published: November 2, 2022
Copyright: © 2022 Kim et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data are available from the Github repository. (https://github.com/kimminsoo-ns).
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Hemifacial spasm (HFS) is a neurological disorder characterized by irregular and involuntary contractions of the facial muscles innervated by the ipsilateral facial nerve. The pathogenesis of HFS may involve vascular compression of the facial nerve at its origin near the brainstem, resulting in demyelination and ephaptic transmission [1, 2]. Although underdiagnosis, misdiagnosis, and the absence of population-based data obscure the prevalence of HFS, its incidence is reportedly 9.8–11 per 100,000 individuals. Microvascular decompression that detaches the culprit vessel from the facial nerve is widely accepted as the first-line surgical treatment and provides relief from spasm in more than 90% of patients [3–7].
The lateral spread response (LSR), a pathological response observed in patients with HFS and can be detected by electromyography (EMG), is elicited by the stimulation of one branch of the facial nerves, resulting in the contraction of the facial muscles innervated by the other branch of the facial nerve. Although the exact mechanism of the LSR is not fully understood, cross-transmission of antidromic activity from the stimulated branch of the facial nerve may contribute to the phenomenon. Continuous monitoring of the LSR during surgery can identify the culprit vessels and confirm that adequate decompression has been carried out [8, 9]. Such continuous monitoring requires dedicated, experienced personnel and cannot be carried out without specialized electrophysiology knowledge.
In the current study, we constructed a dataset of intraoperative EMG with expert annotation to indicate the presence or absence of the LSR. We then developed a deep neural network to classify the LSR using current-standard, off-the-shelf tools. A computer-assisted program based on the neural network could easily be adapted into a commercially available intraoperative monitoring (IOM) system. To the best of our knowledge, no previous studies have used deep learning technology with a computer vision algorithm to automate the diagnosis of LSR. If a deep learning-based algorithm could interpret IOM findings as accurately as a human professional, concerning surgical findings could be detected earlier and improve clinical outcomes.
Methods
Standard protocol approval, registration, and patient consent
Data acquisition and processing in this prospective study were approved by the institutional review board of Samsung Medical Center (SMC2020-10-174). The study was conducted in accordance with the relevant guidelines and regulations. The requirement of written informed consent from the participants was waived because the study involved secondary use of data that were collected primarily for clinical reasons and were appropriately de-identified.
Image dataset
We collected LSR data from adult participants who had undergone microvascular decompression to treat HFS at our institution between August and October 2019. All data were obtained during surgery via screenshots of the IOM suite (Xltek Protektor32 IOM; Natus Neurology, Ontario, Canada). A total of 3,674 images from 50 patients were included for model training and validation: 1,869 with LSR and 1,805 without LSR. Fifty images collected from another 25 patients were used for model testing: 25 with LSR and 25 without LSR. We obtained multiple images from all participants by taking screenshots every minute during the surgical procedures.
To maintain data consistency, anesthetic techniques, surgical procedures, perioperative patient care, and IOM were performed according to the institution’s standard protocol, which has been described elsewhere [10]. All images were annotated as either “LSR presence” or “LSR absence” by an IOM technologist (S.K.P.), confirmed by a neurosurgeon (M.K.), and then anonymized. Representative images of each LSR class are presented in Fig 1.
The typical delayed electrical activity (A, LSR presence) in the mentalis muscle following stimulation of the temporal branch of the facial nerve disappeared (B, LSR absence) when the culprit vessel was detached from the facial nerve during the surgical procedure.
Image preparation
The goal of data preprocessing was to set the region of interest and remove unnecessary data other than the graph itself. The original screenshots measured 1,920 × 1,080 pixels (width × height) and had an RGB color profile. We cropped the images to 720 × 720 pixels, ensuring they contained all four graphical waves of the facial muscles from the moment of facial nerve stimulation to the disappearance of electrical activity. Next, a minimum filter was applied to thicken and enhance the line art information. After resizing the images to 256 × 256 pixels, converting to black and white, and removing the background information and grid, we resized the images to 64 × 64 pixels (Fig 2).
Setting the region of interest and removing unnecessary data other than the graph itself. The original screenshots were 1,920 × 1,080 pixels (width x height), with an RGB color profile. We cropped the images (B) to 720 × 720 pixels, ensuring they contained all four graphical waves of the facial muscles from the moment of facial nerve stimulation to the disappearance of electrical activity. Next, a minimum filter was applied to thicken and enhance the line art information. After resizing the image to 256 × 256 pixels, converting them to black and white, and removing the background information and grid (C), we resized the images to 64 × 64 pixels (D).
Deep neural network implementation
The neural networks were constructed by combining several convolutional, average pooling, and dense layers. We trained the neural networks using an ADAM optimizer [11] and binary cross-entropy as the loss function, which performs as well as the current-standard, off-the-shelf, open-source software and common hardware. To improve performance, we trained the neural networks using more than 2,500 patterns of hyper-parameter variable combinations, such as convolutional layer depth, the number of convolutional layer filters, dense layer size, batch size, optimizer learning rate, and regularizer settings for each layer, among others. We added early stopping and L2 regularization functions to avoid overfitting [12–14].
Evaluation of classification results
We prepared a completely new set of images to test the model, 25 with LSR and 25 without LSR, from another 25 patients whose data had not been used at all in training or validation. Therefore, the images in the test had never been exposed to the neural network.
Among the models built with combinations of hyper-parameter variables, those with the highest accuracy and area under the receiver operating characteristic curve (AUC) when classifying the test set were compared with the assessments of neurosurgeons and IOM technologists experienced in HFS and microvascular decompression. To ensure that performance was compared between the neural network and human participants under the same conditions, the experts did not have access to the patients’ clinical data, such as demographic characteristics, the patient’s preoperative condition, or stage of surgical procedure at the time of test image acquisition. The following metrics were used to evaluate the classification performance of the neural network and human professionals: primary outcome of classification accuracy, calculated as the proportion of images in the test set that were labeled correctly, and common metrics such as the sensitivity, specificity, positive and negative predictive values, and AUC.
Heatmaps
Gradient-weighted class activation maps (Grad-CAM++) [15] and saliency maps that applied the SmoothGrad method [16] were used to visualize the focus made by the neural network. This process was performed using the Python tf-keras-vis toolkit (https://github.com/keisen/tf-keras-vis), which was contributed by one of the authors (Y.K.). These methods use localization maps that visualize the spatial information of the network input in the convolutional layers and expand them to an input-sized saliency map.
Packaging into a computer-assisted interpreting program
The best performing neural network was packaged into a standalone application running on a 64-bit Windows system without Python installed. This process was performed using the Pyinstaller module (https://www.pyinstaller.org).
Statistical analysis
Predictive accuracy is expressed as mean ± standard deviation. Differences between groups were analyzed using one-sample t-tests. All p-values <0.05 were considered statistically significant. All data analyses were conducted using GraphPad Prism version 6.01 for Windows (GraphPad Software, La Jolla, CA, USA).
Results
The neural network could accurately classify images of intraoperative EMG with LSR presence
The neural network that we constructed was successful in correctly differentiating all 50 test images. The sensitivity, specificity, and positive and negative predictive values were calculated as 100%. The neural network with the best accuracy and AUC was constructed by combining the following hyper-parameter settings: depth of convolutional layers, 5; number of convolutional layer filters, 32; batch size for fitting, 8; learning rate of optimizer, 0.0003. The structure of the neural network is shown in Table 1.
The neural network and human professionals showed equivalent performance
The images were classified by the human participants under the same conditions as the neural network. The performance of the neural network was compared with that of 16 human professionals (11 neurosurgeons and 5 IOM technologists). All 11 board-certified neurosurgeons were currently active in the field of functional neurosurgery and were trained in microvascular decompression surgery. Six of them had more than 3 years of experience in their functional neurosurgical field. All 5 IOM technologists performed neurophysiological monitoring in the operation room during the neurosurgical or orthopedic procedures. Four of them had more than 3 years of experience in neurophysiological monitoring.
The proposed neural network obtained 100% accuracy, which was significantly higher (p = 0.001) than that of human professionals (average, 82.8% ± 13.6%). When we categorized the human professionals according to their experience and specialty, the accuracy of the technologist group (94.0% ± 11.3%) was significantly higher (p = 0.0200) than that of the neurosurgeon group (77.7% ± 11.7%). The performance of the neural network was equivalent to that of the technologist group (p = 0.3013) but better than that of the neurosurgeon group (p < 0.0001). A professional with more than 3 years of experience tended to be more accurate at classifying the LSR than a professional with less experience, but the difference in accuracy was not significant between the neural network and experience subgroups. In the binary classification of the 50 images of LSR presence or absence, the AUC of the neural network was 0.96 (Fig 3).
A professional with more than 3 years of experience tended to be more accurate than a less experienced professional in classifying lateral spread response (LSR), but the difference in accuracy was not statistically verifiable between the neural network and subgroups by experience due to the small number of participants included. To classify 50 images as LSR presence or absence, the neural network’s AUC was 0.96. AUC, area under the receiver operating characteristic curve; IOM, intraoperative monitoring.
The trained neural network identified key features differentiating LSR findings
The heatmap of the representative LSR images is shown in Fig 4. Increased intensity was observed around the lowermost wave, representing the response of the mentalis muscle; this was identified as the key feature in differentiating LSR presence and absence. In images with LSR presence, hot uptake was apparent near the peak of the lowermost wave, whereas hot uptake in images with LSR absence was spread all over the lowermost wave. This finding illustrates why our neural network could correctly classify the test images. The heatmap images highlighted the same area that humans read when interpreting the EMG graphs to classify the LSR. Among the two visualization techniques, the SmoothGrad images best displayed the region that our neural network might have utilized for its inference.
Increased intensity was observed around the lowermost wave, representing the response of the mentalis muscle in the images with LSR presence (A, B) and absence (C, D). (A, B) In those with LSR presence, the presence of the peak of the lowermost wave is the main criterion for judgment, as can be seen in the hot spots, especially in (B). (C, D) In the case of LSR absence, the deep neural network concentrates the entire lowermost wave because there is no peak in the corresponding region. This finding is very similar to how humans interpret graphs. Red indicates the focus of the neural network.
The neural network-based application was useful in real circumstances
In clinical practice, neural networks are used to assist clinicians in the classification of new observations into one of the labeled groups. In the present study, this meant recognizing whether the intraoperative EMG wave had an LSR or not. The neural network was packaged into an executable binary program available for 64-bit Windows; thus, it would return a prediction if exposed to images. The program does not require training, nor does it need a computer with a graphic processing unit. After exposure to the image, it rapidly displays the prediction on the screen within a second.
We tentatively tested the program in a clinical situation. Although our test could not be validated statistically or officially published in its current setting, it may be useful, especially when the LSR findings differed from those of the IOM technologist. In such cases, the IOM technologist notified the surgeon about the discrepancy of the finding and reported both their own and the neural network’s prediction. The surgeon was therefore informed and could interact better with the IOM technologist about the situation.
Discussion
This study showed that deep learning algorithms for classifying LSR presence and absence can perform well. The network provided accuracy that was equivalent to or better than that of the human professionals for classification (Fig 3). It could be used to provide clinical support during surgery when IOM findings are insufficient for concise classification by a human. Interestingly, the heatmap generated by the Grad-CAM++ and SmoothGrad algorithms highlighted the key region of interest within each image at a level similar to that of trained human professionals.
Our most important contribution was that we built a neural network to classify a line chart. Most previous publications have used deep learning algorithms to categorize medical photos [17–20], radiographic images [21–25], pathologic specimens [26–28], or other microscopic features [29, 30] that often require labor-intensive preprocessing before being input into a neural network. Conversely, our network could use screenshotted information immediately acquired from the IOM suite, without any time-consuming preprocessing of the data. Interpreting medical information via a line chart image, without the need for calculation, is more similar to what humans actually do, and the heatmap in Fig 4 supports that assumption. The neural network focused only on the peak of the lowermost wave, which represented an abnormal latent response in the mentalis muscle in the image with LSR presence (Fig 4A and 4B). Conversely, the neural network seemed to search for the peak on the whole lowermost wave of the image with LSR absence (Fig 4C and 4D).
Neural networks table IOMs can give prompt feedback to the surgical team during operations. Our algorithms can recognize the presence or absence of the LSR immediately after the EMG graph appears on the screen of the IOM suite. Thus, real-time feedback to the surgical team serves the purpose of the IOM itself. That is, surgeons can gain information both from the IOM technologist and our algorithm. Sometimes, mismatched information between interpreters and machine learning can prompt surgeons to reconsider the adequate localization for decompression of the target nerve. As shown in Fig 3, the accuracy of interpretation by an inexperienced interpreter was inferior to machine learning. Therefore, if the surgical team is inexperienced or if human error is possible due to fatigue, it would be preferable to use machine learning as an auxiliary device.
The present study has some limitations. First, only 3,600 images of 50 participants from a single center were included in the training and validation sets. However, the neural network showed good performance when tested using completely new data from 25 participants. This could also be interpreted as an advantage; a neural network can be created using a small sample size of data at each individual institute. Second, the screen layout for IOM monitoring varies greatly depending on user preference and manufacturer design. To overcome this limitation, we have developed a standalone application that runs on a 64-bit Windows-based system. The neural network should return the right answer as long as the captured position is adjusted to suit the screen layout of the equipment. The present study, therefore, overcame one major limitation of previous studies, namely that deep learning algorithms in prior clinical applications have rarely been applied outside the original study. Finally, essential clinical information that should be considered when reading an EMG was excluded from the present study. In further studies, we will add information such as time elapsed from the start of the operation and temporal changes from the initial image to ensure that the neural network has similar information as that when a human makes a judgment.
In conclusion, the present study showed that a deep neural network trained on a relatively small dataset of intraoperatively collected EMGs could detect LSR, a characteristic pathology in patients with HFS, and that the performance of the neural network was similar to that of human neurosurgeons and IOM technologists. These results were obtained from medical line chart information using standard, off-the-shelf, open-source software and common hardware. Therefore, our deep learning algorithm may be useful to surgical teams during operations.
Acknowledgments
The authors would like to thank the Young Functional Neurosurgeon colleagues of the Korean Society of Stereotactic and Functional Neurosurgery for their support with performing this study.
Portions of this work were presented as an oral presentation at the 12th Scientific Meeting for the Asian Australian Society of Stereotactic and Functional Neurosurgery (AASSFN 2020), Gyeongju, Republic of Korea, September 17–19, 2020.
References
- 1. Gardner WJ. Cross talk—the paradoxical transmission of a nerve impulse. Arch Neurol. 1966;14(2):149–56. Epub 1966/02/01. pmid:4378157.
- 2. Nielsen VK. Pathophysiology of hemifacial spasm: II. Lateral spread of the supraorbital nerve reflex. Neurology. 1984;34(4):427–31. Epub 1984/04/01. pmid:6322049.
- 3. Barker FG 2nd, Jannetta PJ, Bissonette DJ, Larkins MV, Jho HD. The long-term outcome of microvascular decompression for trigeminal neuralgia. N Engl J Med. 1996;334(17):1077–83. Epub 1996/04/25. pmid:8598865.
- 4. Hatem J, Sindou M, Vial C. Intraoperative monitoring of facial EMG responses during microvascular decompression for hemifacial spasm. Prognostic value for long-term outcome: a study in a 33-patient series. Br J Neurosurg. 2001;15:496–9. pmid:11814001
- 5. Hyun SJ, Kong DS, Park K. Microvascular decompression for treating hemifacial spasm: lessons learned from a prospective study of 1,174 operations. Neurosurg Rev. 2010;33:325–34. pmid:20349099
- 6. Kondo A, Date I, Endo S, Fujii K, Fujii Y, Fujimaki T. A proposal for standardized analysis of the results of microvascular decompression for trigeminal neuralgia and hemifacial spasm. Acta Neurochir (Wien). 2012;154:773–8. pmid:22327325
- 7. Lawrence JD, Frederickson AM, Chang YF, Weiss PM, Gerszten PC, Sekula RF. An investigation into quality of life improvement in patients undergoing microvascular decompression for hemifacial spasm. J Neurosurg. 2018;128(1):193–201. Epub 2017/02/12. pmid:28186450.
- 8. Kong DS, Park K, Shin BG, Lee JA, Eum DO. Prognostic value of the lateral spread response for intraoperative electromyography monitoring of the facial musculature during microvascular decompression for hemifacial spasm. Journal of neurosurgery. 2007;106:384–7. pmid:17367059
- 9. Ying TT, Li ST, Zhong J, Li XY, Wang XH, Zhu J. The value of abnormal muscle response monitoring during microvascular decompression surgery for hemifacial spasm. Int J Surg. 2011;9:347–51. pmid:21396488
- 10. Lee S, Park SK, Lee JA, Joo BE, Kong DS, Seo DW, et al. A new method for monitoring abnormal muscle response in hemifacial spasm: A prospective study. Clinical neurophysiology: official journal of the International Federation of Clinical Neurophysiology. 2018;129(7):1490–5. Epub 2018/04/08. pmid:29625882.
- 11. Kingma D, Ba J. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations. 2014.
- 12. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems—Volume 1; Lake Tahoe, Nevada: Curran Associates Inc.; 2012. p. 1097–105.
- 13.
Prechelt L. Early Stopping—But When? In: Orr GB, Müller K-R, editors. Neural Networks: Tricks of the Trade. Berlin, Heidelberg: Springer Berlin Heidelberg; 1998. p. 55–69.
- 14. Nowlan SJ, Hinton GE. Simplifying Neural Networks by Soft Weight-Sharing. Neural Computation. 1992;4(4):473–93.
- 15. Chattopadhay A, Sarkar A, Howlader P, Balasubramanian VN, editors. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV); 2018: IEEE.
- 16. Smilkov D, Thorat N, Kim B, Viégas F, Wattenberg M. Smoothgrad: Removing noise by adding noise. arXiv 2017. arXiv preprint arXiv:170603825.
- 17. Xu T, Zhang H, Xin C, Kim E, Long LR, Xue Z, et al. Multi-feature based Benchmark for Cervical Dysplasia Classification Evaluation. Pattern Recognit. 2017;63:468–75. Epub 2017/06/13. pmid:28603299; PubMed Central PMCID: PMC5464748.
- 18. Goyal M, Reeves ND, Rajbhandari S, Ahmad N, Wang C, Yap MH. Recognition of ischaemia and infection in diabetic foot ulcers: Dataset and techniques. Comput Biol Med. 2020;117:103616. Epub 2020/02/20. pmid:32072964.
- 19. Ryu D, Jo Y, Yoo J, Chang T, Ahn D, Kim YS, et al. Deep learning-based optical field screening for robust optical diffraction tomography. Sci Rep. 2019;9(1):15239. Epub 2019/10/28. pmid:31645595; PubMed Central PMCID: PMC6811526 Tomocube Inc., a company that commercializes ODT and is one of the sponsors of the work.
- 20. Liu G, Hua J, Wu Z, Meng T, Sun M, Huang P, et al. Automatic classification of esophageal lesions in endoscopic images using a convolutional neural network. Ann Transl Med. 2020;8(7):486. Epub 2020/05/13. pmid:32395530; PubMed Central PMCID: PMC7210177.
- 21. Sogawa T, Tabuchi H, Nagasato D, Masumoto H, Ikuno Y, Ohsugi H, et al. Accuracy of a deep convolutional neural network in the detection of myopic macular diseases using swept-source optical coherence tomography. PLoS One. 2020;15(4):e0227240. Epub 2020/04/17. pmid:32298265; PubMed Central PMCID: PMC7161961.
- 22. Rajkomar A, Lingam S, Taylor AG, Blum M, Mongan J. High-Throughput Classification of Radiographs Using Deep Convolutional Neural Networks. J Digit Imaging. 2017;30(1):95–101. Epub 2016/10/13. pmid:27730417; PubMed Central PMCID: PMC5267603.
- 23. Dunnmon JA, Yi D, Langlotz CP, Ré C, Rubin DL, Lungren MP. Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs. Radiology. 2019;290(2):537–44. Epub 2018/11/14. pmid:30422093; PubMed Central PMCID: PMC6358056.
- 24. Thüring J, Rippel O, Haarburger C, Merhof D, Schad P, Bruners P, et al. Multiphase CT-based prediction of Child-Pugh classification: a machine learning approach. Eur Radiol Exp. 2020;4(1):20. Epub 2020/04/07. pmid:32249336; PubMed Central PMCID: PMC7131973.
- 25. Sánchez Fernández I, Yang E, Calvachi P, Amengual-Gual M, Wu JY, Krueger D, et al. Deep learning in rare disease. Detection of tubers in tuberous sclerosis complex. PLoS One. 2020;15(4):e0232376. Epub 2020/04/30. PubMed Central PMCID: PMC7190137 form of computational credits for his project on “Identification and localization of tubers in Tuberous Sclerosis Complex with deep learning convolutional neural networks”. JYW, DK, HN, MEB, MS, and JP received funding to collect the data as a part of the TACERN collaborative. There are no patents, products in development or marketed products to declare. This does not alter our adherence to PLOS ONE policies on sharing data and materials. pmid:32348367
- 26. Du Y, Zhang R, Zargari A, Thai TC, Gunderson CC, Moxley KM, et al. Classification of Tumor Epithelium and Stroma by Exploiting Image Features Learned by Deep Convolutional Neural Networks. Ann Biomed Eng. 2018;46(12):1988–99. Epub 2018/07/28. pmid:30051247; PubMed Central PMCID: PMC6286645.
- 27. Zhang S, Han F, Liang Z, Tan J, Cao W, Gao Y, et al. An investigation of CNN models for differentiating malignant from benign lesions using small pathologically proven datasets. Comput Med Imaging Graph. 2019;77:101645. Epub 2019/08/28. pmid:31454710; PubMed Central PMCID: PMC6800808.
- 28. Xu J, Luo X, Wang G, Gilmore H, Madabhushi A. A Deep Convolutional Neural Network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing. 2016;191:214–23. Epub 2017/02/06. pmid:28154470; PubMed Central PMCID: PMC5283391.
- 29. Zhou L, Yu Q, Xu X, Gu Y, Yang J. Improving dense conditional random field for retinal vessel segmentation by discriminative feature learning and thin-vessel enhancement. Comput Methods Programs Biomed. 2017;148:13–25. Epub 2017/08/05. pmid:28774435.
- 30. Wang B, Xiao L, Liu Y, Wang J, Liu B, Li T, et al. Application of a deep convolutional neural network in the diagnosis of neonatal ocular fundus hemorrhage. Biosci Rep. 2018;38(6). Epub 2018/10/20. pmid:30333258; PubMed Central PMCID: PMC6435455.