3.1.1 Summary of architectures used and model purpose.

All the DLS studies that were included had a convolutional neural network (CNN) as their main architecture, although the architecture used and the layer modification differed depending on the task required and the dataset used. Commonly used CNN architectures were: Inception-V3, Inception-V4, ResNet101, DenseNet, SeResNe50, U-Net, GoogleNet, EfficientNet, and VGGNet, most of which were reported to be pre-trained on ImageNet datasets and modified based on model needs, with the exception of three studies [40,47,50]. Four out of five non-DLS studies that utilised support vector machines performed optic disc detection as a region of interest as part of the pre-processing step, followed by feature extraction of relevant features, and subsequent classification with SVM between normal and papilloedema images.

3.1.2 Dataset acquisition.

Datasets were retrospectively obtained by a majority of the studies, with the exception of one mixed linear model study that prospectively collected data from two Danish regional tertiary hospitals that specialised with idiopathic intracranial hypertension [46]. Publicly available datasets for papilloedema are scarce as reported in Table 2. A detailed summary of the number of normal retinal images and papilloedema images reported are provided in Table A for DLS and Table B for non-DLS ML models in S5 Appendix. Neither of the two largest datasets of papilloedema images used are publicly accessible [30,32]. Only four DLS studies use datasets that are publicly available such as STARE, DRIONS, ACRIMA, and Kim Ungsoo’s Kaggle dataset (Table 2) [33,38–40]. Classification performance of DLS depends on both the size and quality of the initial training datasets used, with performance generally improving with an increase in size of datasets [57]. Despite the emphasis on its importance in recent studies [58,59], estimating the required training dataset size was not performed by any of the DLS studies included in this review. Only one linear mixed model study estimated that they required a minimum of 16 participants based on their power analysis [46]. Though several sample-size estimating methods have been proposed, these are not commonly used due to the lack of standardised model-specific guidelines [57,59,60].

3.1.3 Participant eligibility criteria.

Only six DLS and two non-DLS explicitly stated their inclusion and exclusion criteria [30,32,48,53–55]. However, the studies may vary in the level of detail they provide and diagnostic criteria they use. For instance, Li et al. (2022) included images that had a normal fundus and those with a selection of 12 fundus diseases, each defined by a specific diagnostic criterion [30]. Milea et al. (2020) reported comprehensive eligibility criteria in their supplementary file with the main inclusion criteria being fundus photographs of the optic disks and definitive corresponding clinical diagnoses made by expert neuro-ophthalmologists. The main exclusion criteria in both studies were insufficient image quality or a failure to annotate images with consistent labels. There is significant potential for bias in image classification studies due to uncertainty in diagnostic criteria, variation in methods for data collection, presence of duplicates, variable quality of images, and subsequent feature selection bias. In the absence of clear reporting of eligibility criteria in a majority of the studies included in the review, it is difficult to assess report risk of bias in participant or image selection.

3.1.4 Reference standard and annotations.

The ‘ground truth’, also known as the ‘reference standard’, is ‘the best available method for establishing the presence or absence of the target condition’ [61]. The reference standard for twelve of the DLS studies was set by ophthalmologists, neuro-ophthalmologists, or ‘expert clinicians’ [30,32,40,41,43,44,47,49,51,53–55]. Only one study used neuroimaging or direct evidence (elevated lumbar puncture opening pressures) of raised intracranial pressure as a reference standard. Similar to DLS models, ophthalmologists or neuro-ophthalmologists were most commonly used for annotating and grading of all the papilloedema images based on the Friesen scale [34–37].

Three studies did not report reference standard setting [33,38,50], with one paper partially reporting how the reference standard was set for a local dataset through the use of a ‘combination of detailed clinical history, clinical examination, ophthalmic imaging, and/or neurological imaging’ [39]. One further study employed an ambiguous description of ground truth where the presence or absence of a condition was annotated by ‘one or multiple human readers for each image’ where a combination of ‘structured information’ and ‘manually-extracted textual information’ was used [31]. Several studies used more than one ophthalmologist for annotation of a single image, thereby aiming to account for intra-rater variability [30,32,47,51,53].

3.1.5 Discussion of diversity and patient demographics.

Seven studies used multi-ethnic datasets for training, though only two studies published in the last year provided a complete statistical summary of the ethnic groups in their dataset [32,39,41,44,53,54,62]. Through the use of heat maps, Milea et al. (2020) discussed the diversity in their dataset in terms of age, gender, country of origin, and ethnicity. They demonstrated instances in which the AI model perceived a ‘pathological area’ based on ethnicity.

3.2.1 Storage of data.

Datasets used by studies that are publicly available such as STARE, DRIONS, ACRIMA, RFMiD, RFMiD 2.0, and Kim Ungsoo’s Kaggle can be found online. Of those studies not using publicly available datasets, Milea et al.’s (2020) BONSAI dataset is stored in the Singapore Eye Research Institute, and Avramidis et al.’s (2020) paediatric fundus dataset is stored in a Health Insurance Portability and Accountability Act-compliant Research Electronic Data Capture Database at University of Southern California. The remainder of the studies did not report the location of their data storage. STARE was the most commonly used online dataset source of papilloedema and normal retinal images by four studies [34–36,45]. Locally available datasets were used by 6 studies although details of data location were not reported [34–37,45,46].

3.2.2 Pre-processing steps in deep learning systems.

Pre-processing includes any steps that transform raw data through cleaning, noise reduction, removal of missing or inconsistent data, clustering, feature engineering, filtering, and modelling for improved and more efficient performance of the model [63]. Though a majority of the studies (n = 13) reported that they used pre-processing techniques, very few provided detailed information on the steps involved [30–33,38,40,43,47–50,53,55]. Standardisation of size and definition of region of interest at the optic disc were the two most common reasons for pre-processing fundus images [31 33, 40, 47-50, 53, 62]. Ahn et al. (2019) also adjusted variations in lighting and brightness through Gaussian filtering prior to cropping the image with the region of interest [33].

Quellec et al. (2020) performed illumination variations using a YCrCb colour space and converted all images to an RGB image for interpretation by the model [31]. Milea et al. (2020) developed an automated segmentation algorithm that performed a quality check, analysed the optic disk size, shape, margins, colour balance, exposure, brightness, and sharpness [32]. They performed automatic cropping of the optic disc image before classification of papilloedema occurred [32]. They also used U-Net which is another neural network framework for a pixel-level image segmentation to localise the optic disk [32]. Avramidis et al. (2022) developed an unsupervised optic disc detection algorithm based on intensity thresholding using blood vessels in that region and morphological operations [48].

3.2.3 Pre-processing steps in non-deep learning systems.

Optic disc detection and vessel segmentation to extract the region of interest was the most commonly used pre-processing step [34–36,45,52]. Naing and Aimmanee automate image segmentation by using factorized gradient vector flow, a special gradient vector flow for texture segmentation in oedematous optic discs, to localise the optic disc boundary with high accuracy [52]. Fatima et al. (2017) and Akbar et al (2017) also used a 2D Gabor wavelet for enhancement of blood vessels. Images were then cropped out and resized to standardise size and region of interest [35–38]. Akbar et al. (2017) also transformed their images into greyscale [34,35]. Echegeray et al. (2011) used an ophthalmic medical technician for cropping and resizing of digital photos and converted it into a right eye orientation as part of their standardisation process [37]. Johnson et al. (2018) uniquely used U-Net to create a blood vessel probability map for vessel inpainting as part of their pre-processing step [42]. Hagen et al. (2023) used a deep learning algorithm for extraction of the visible optic disc [46]. They also used a deep learning algorithm and image software program for semi-automatic segmentation of peripapillary arterioles and paired venules [46].

3.2.4 Handling of missing data, outliers, or data inconsistencies.

Six studies described how missing data, duplicates, or poor-quality data were managed [29,31–33,40,54]. Visual inspection and manual removal of poor-quality images, missing data, or mislabelled data was the most common method by all five studies. Additionally, Milea et al. (2020) also used a quality control algorithm and quoted a labelling error of 0.9% in their testing dataset [32].

3.2.5 Data augmentation techniques.

Data augmentation techniques are used to increase the size and quality of the training dataset to reduce overfitting and increase the overall generalisability of a model and are particularly valuable in the case of limited datasets [64]. Geometric transformations, random erasing, mixing images, feature space augmentations, and data warping are all methods for data augmentation of image datasets. The most commonly used methods to augment data in the included DLS studies were rotation and flipping [32,33,38,43,47,49–51,55]. Random dropout, adjustments to brightness and contrast, and cropping certain areas of the image were also used [32,33,48,49]. Images were typically reported to have been increased by 3, 5 or 10 times the original sample size by studies that reported these numbers [33,40,48].

3.2.6 Handling of class imbalance.

Class imbalance refers to the issue of having a higher number of data points for one class such as normal retinal images compared to another class such as papilloedema, particularly in the training set [65]. Class imbalance can significantly reduce the overall performance and generalisability of model, particularly in unseen datasets. There are data level methods (oversampling or undersampling), algorithmic level methods (thresholding, cost sensitive learning, or one-class classification), and hybrid methods that have been proposed to overcome the issue of class imbalance [65]. Very few studies contained detailed reporting of how class imbalance was handled. Among those who did, Milea et al. (2020) performed data augmentation and set weights to their loss function to reduce any class imbalance [32] while Thiagarajan and Suguna (2023) trimmed the data to have a near equal number of images in each class [38]. Quellec et al. (2020) used a ‘balanced dataset’ such that all frequent conditions were equally represented, but did not apply this to rare conditions such as papilloedema [31].

3.3.1 Study and participant characteristics.

We provide the summary statistics of all reported data in Table 3; age and gender of participants, number of images of normal optic disc and papilloedema in training and testing, and number of participants in training and testing.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary statistics of reported data in included studies.

https://doi.org/10.1371/journal.pdig.0000783.t003

Reporting rates for each parameter were inconsistent; for example, while 14 and 11 of the 19 DLS studies reported the number of images of papilloedema in their training and testing sets respectively, only 3 of 19 reported the number of participants in their study (for both training and testing sets), limiting the ability to provide a comprehensive overview of the study characteristics. There was significant variability in the number of images used to train and test the DLS models. The median number of images of normal optic discs used in the training set was 2509 (IQR 580–9156) and in the testing set was 569 (IQR 119–1378). The number of papilloedema images in the training and testing sets was lower with a median of 1292 (IQR 201–2882) in training set and 201 (IQR 57–388) in the testing set. This represents a significant degree of heterogeneity and class imbalance between the datasets. For the studies using non-DLS machine learning algorithms, insufficient numbers reported the above parameters to generate median and IQRs. Six models reported using multi-ethnic datasets, yet none provided a complete quantitative breakdown of the ethnic groups within their cohorts of participants. The reported representation of participants from high-income countries (HICs) was 2.4 times more in the included datasets when compared to participants from low-income countries (LMICs) (Fig 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Origin of participants in datasets as stated by authors in the included studies. The Organization for Economic Cooperation and Development (OCED) 2024/2025 categorization was used to defined HICs and LMICs. NR represents data that was not reported by studies.

https://doi.org/10.1371/journal.pdig.0000783.g005

3.3.2 Model performance.

A range of parameters as defined in S4 Appendix were used to measure the overall model performance, with accuracy, area under the receiving operator curve (AUROC), sensitivity, and specificity being the most commonly used metrics. Accuracy was the most commonly used performance metric by 14 DLS and 5 non-DLS studies, followed by sensitivity and specificity, which were reported by 12 DLS and 5 non-DLS studies. AUROC was used by 11 DLS and 2 non-DLS studies. F1 score and precision recall were reported by 4 DLS studies. PPV and NPV values were provided by one DLS and two non-DLS studies. The category ‘others’ include Cohen’s weighted K coefficient provided by one non-DLS study, Spearman’s rank provided by one DLS study, and precision value given by one DLS study.

Accuracy rates were more than 80% in 17 of the 19 studies which reported this value. Indicators of good performance such as high AUROC, accuracy, sensitivity, specificity or precision cannot be interpreted in isolation in studies using AI models. For instance, a model with one of the highest levels of reported accuracy (99.17%) was Saba et al. (2021)’s DLS which only included 60 images of normal retina and 40 images of papilloedema in their testing set, without any external validation. In contrast, Milea et al. (2020)’s robust reporting of their study provides a better overview of their BONSAI model’s performance. BONSAI’s reported accuracy was 94.8% for distinguishing papilloedema from other pathologies and normal retina in their internal validation dataset. This reduced to 87.5% when validated with an external dataset.

However, AUROC is generally preferred over accuracy as a measure of performance for binary classification, especially in imbalanced datasets [66]. AUROC measures the area under a Receiving Operator Characteristics (ROC) curve. It provides a measure of aggregated classification performance, allows comparison of performance between different classifiers, and helps examine trade-offs between true positive and false positive rates. Ranging from 0.5 to 1.0, 0.5 suggests that the test is no better than chance at making the distinction. Among four studies that performed external testing of their models [30,32,33,39], AUROC values between internal and external testing sets were reported to be very similar (see Table C in S5 Appendix) in three out of four of the studies that performed external testing, which reflects positively on the model’s performance in being able to distinguish between normal retina and papiloedema images.

Of the 27 studies, ten provided the necessary data to construct contingency tables for a meta-analysis of diagnostic test accuracy using a random-effects model. Of these, seven studies used deep learning systems and three used non-DLS machine learning. Of these ten contingency tables, four were calculated from external test data, while the others did not specify if it was an external or internal test set. The sensitivity of the models among these studies ranged from 0.76-0.96 (mean = 0.89, SD = 0.07), while the specificity ranged from 0.69-0.98 (mean = 0.88, SD = 0.10). The total number of images in these testing sets ranged from 100-1886 (mean = 565, SD = 578). The I² value was calculated at 74%, indicating a large amount of heterogeneity between studies; however, it should be noted that in small meta-analyses the I² has been shown to be biased and should be interpreted with caution [67]. The pooled sensitivity and specificity were calculated to be 0.87 [95% CI 0.76-0.93] and 0.90 [95% CI 0.74-0.97], respectively (Fig 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Summary of receiving operator characteristics curve for seven studies included in the meta-analysis.

https://doi.org/10.1371/journal.pdig.0000783.g006

These results show a strong diagnostic performance overall, with a high pooled sensitivity and specificity. However, the validity of this interpretation is limited by the small number of studies, a high degree of variability in the size of datasets, and high level of heterogeneity. Additionally, not all models were externally validated. It is also worth noting that only two of the four model types from this systematic review were included in the meta-analysis; neither of the studies using a linear mixed model or random forest classifier included the necessary data to be included in the meta-analysis.

3.3.3 Performance of DLS compared against clinicians.

Chang et al. developed an AI model to aid differentiating paediatric pseudopapilloedema from true papilloedema [54]. Their AI model was found to be most helpful in identifying mild papilloedema which can be challenging to diagnose in children. The model’s sensitivity for diagnosing mild papilledeoma was significantly higher at 87.8% in external testing compared to the two human experts (53.1% and 49%) [53]. The performance of the BONSAI deep learning system was compared against both first line clinicians and expert neuro-ophthalmologists [28,32,41,51]. These studies included a relatively diverse cohort of 454 patients (800 images) where 60% were women with near equal proportions of Asian (48.7%) and Caucasian (50.9%) participants [28]. The mean age of participants was 44.5 (+/- 20.1) years [28]. The DLS outperformed 30 clinicians based in Singapore regardless of their ophthalmic training with an error rate of 15.3% [28]. By contrast, Emergency Department (ED) physicians had the highest error rate of 47.9% in detecting optic disc abnormalities including papilloedema [28]. The DLS was also found to be significantly faster (25 seconds) than 2 expert neuro-ophthalmologists (61 and 74 minutes) in classifying normal, papilloedema and other disc abnormality, though there was no statistically significant difference in terms of accuracy for papilloedema detection [41]. Accurate classification of the severity of the papilloedema (mild-moderate or severe) by the DLS (87.9%) was comparable to the ability of 3 neuro-ophthalmologists with an accuracy of 84.1%, 91.8%, and 73.9%, though these findings are not generalisable due to the very small sample cohort of neuro-ophthalmologists [44].

3.3.4 Model interpretability.

In black box methods such as DLS models, visualisation using class activation maps can provide some insight into model’s interpretation of abnormalities and help understand performance of the model, especially when images are misclassified. However, there are limitations to these methods for understanding decisions made by the model at an individual level and may not provide any ‘performance guarantees’ [68]. Only five deep learning studies used techniques to improve the interpretability of their model. Quellec et al. (2020) used heatmaps to show how much each pixel contributed to prediction of pathology, however not for papilloedema [31]. Cao et al. (2020) used feature maps to capture the regions on fundus photography and visual field tests where abnormality was detected. Milea et al. (2020),Branco et al. (2024), and Chang et al. (2024) used class activation mapping (CAM) to visualise the key features contributing to the neural networks’ performance ( [32,53,54]). Milea et al. (2020) also found that there were differences in what the AI model perceived as ‘pathological area’ based on ethnicity [32]. Avramidis et al (2022) used saliency maps to confirm signs of papilloedema as interpreted by the model and verify the presence of these findings clinically [48]. Liu et al. (2023) performed CAM analyses for all the images used in their study and found that there were areas with ‘strong activation’ outside the optic disc in 32% of the eyes where the images were incorrectly classified [40]. Those that were correctly classified showed strong activation at the site of the disc.