Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Development and Validation of a Diabetic Retinopathy Referral Algorithm Based on Single-Field Fundus Photography

Development and Validation of a Diabetic Retinopathy Referral Algorithm Based on Single-Field Fundus Photography

  • Sangeetha Srinivasan, 
  • Sharan Shetty, 
  • Viswanathan Natarajan, 
  • Tarun Sharma, 
  • Rajiv Raman
PLOS
x

Abstract

Purpose

To develop a simplified algorithm to identify and refer diabetic retinopathy (DR) from single-field retinal images specifically for sight-threatening diabetic retinopathy for appropriate care (ii) to determine the agreement and diagnostic accuracy of the algorithm as a pilot study among optometrists versus “gold standard” (retinal specialist grading).

Methods

The severity of DR was scored based on colour photo using a colour coded algorithm, which included the lesions of DR and number of quadrants involved. A total of 99 participants underwent training followed by evaluation. Data of the 99 participants were analyzed. Fifty posterior pole 45 degree retinal images with all stages of DR were presented. Kappa scores (κ), areas under the receiver operating characteristic curves (AUCs), sensitivity and specificity were determined, with further comparison between working optometrists and optometry students.

Results

Mean age of the participants was 22 years (range: 19–43 years), 87% being women. Participants correctly identified 91.5% images that required immediate referral (κ) = 0.696), 62.5% of images as requiring review after 6 months (κ = 0.462), and 51.2% of those requiring review after 1 year (κ = 0.532). The sensitivity and specificity of the optometrists were 91% and 78% for immediate referral, 62% and 84% for review after 6 months, and 51% and 95% for review after 1 year, respectively. The AUC was the highest (0.855) for immediate referral, second highest (0.824) for review after 1 year, and 0.727 for review after 6 months criteria. Optometry students performed better than the working optometrists for all grades of referral.

Conclusions

The diabetic retinopathy algorithm assessed in this work is a simple and a fairly accurate method for appropriate referral based on single-field 45 degree posterior pole retinal images.

Introduction

Single-field fundus photography with interpretation by trained readers serves as a screening tool to identify patients with diabetic retinopathy (DR) for referral for further ophthalmic evaluation and management [1,2]. The ease of use, convenience, and ability to detect retinopathy favors single-field funds photography; however, the reported sensitivity values are less than ideal when compared with 7-standard field photography [3]. Nevertheless, it continues to be a screening method, especially when more numbers need to be screened [4]. In India, although the overall prevalence of DR is lower than western population, the absolute numbers to be screened is very high [5,6].

The major challenge in this model of screening is the training of graders. The conventional grading of DR by Early Treatment Diabetic Retinopathy Study (ETDRS)[7] and the International Clinical Disease Severity Scale for DR[8] needs understanding of the ETDRS scales and is more pertinent when multiple fields of retinal images are obtained. The aim of this study was to develop a simplified algorithm for referral of DR, which can be used by graders (physician/ophthalmologist/optometrist/trained paramedical worker) to grade single-field retinal images and to refer sight-threatening DR (STDR) for appropriate care. The accuracy of the algorithm was determined by a pilot study among optometrists, with the gold standard being the grading by a retinal specialist.

Materials and Methods

The study was exempt from approval of Institutional Review Board of Vision Research Foundation, Chennai, India.(Letter attached)

Development of the Algorithm

The International Clinical Diabetic Retinopathy and Diabetic Macular Edema Severity Scale[8] incorporated evidence on disease progression from the ETDRS and classified DR into 5 levels of severity based on retinal changes. The main advantage of the international severity scale was that the levels of severity were relevant to the clinical management plan for the patient. This facilitated communication between ophthalmologists and primary health-care providers. The problem with this scoring method is the learning curve involved in remembering the grades of retinopathy such as those which need immediate referral, 4–6 months follow-up, and yearly follow-up. To simplify, a red–green–yellow triage algorithm was developed based on the referral guidelines prescribed by the American Academy of Ophthalmology Retina/Vitreous Panel. Preferred Practice Pattern Guidelines [9].

The grading of disease severity is based on the International Clinical Diabetic Retinopathy and Diabetic Macular Edema Severity Scale [8], while the referral guidelines are based on the American Academy of Ophthalmology guidelines [9]. The algorithm included the lesions of DR and number of quadrants involved for DR referral and location of lesions for diabetic macular edema referral (Fig 1). The single-field (posterior pole 45 degree) retinal image is divided into 4 quadrants by 2 imaginary lines. The horizontal line passes through the fovea and center of disc and the line tangent to the horizontal line passes through the center of the disc (Fig 1). For macular edema, 2 concentric imaginary circles are drawn at 1 disc diameter (1DD) and 2 disc diameter (2DD) from the center of macula.

thumbnail
Fig 1. The red–green–yellow triage algorithm for quadrant-wise and macular evaluation, by using single-field 45 degree posterior pole photograph (lower left retinal photograph).

https://doi.org/10.1371/journal.pone.0163108.g001

The algorithm consisted of various DR lesions (listed below) and the number of quadrants (1–4).

The following are the various types of DR lesions to be identified.

Small red dots/blots.

These include microaneurysms, dot-blot hemorrhages, and flame-shaped hemorrhages.

Yellow lesions.

These include the deeper yellow lesions with sharp margins, which are the hard exudates.

White lesions.

These include the more superficial, grayish white lesions with fluffy margins, and the cotton wool spots.

Venous abnormality or beading.

These include venous dilation, loops, and beading.

IRMA/NVD/NVE.

Intraretinal microvascular abnormality (IRMA) represent abnormal blood vessels, spider-web–like appearance (do not cross an artery or a vein and, therefore, appears only on one side of the vessel). Neovascularization at the disc (NVD) and neovascularization elsewhere (NVE) are new vessels from the surface of the retina or optic disc. Fibrous tissue accompanies the development of the NVD or NVE and sometimes may not have clearly evident new vessels.

Vitreous or preretinal hemorrhage.

Preretinal hemorrhage is a boat-shaped lesion anterior to the retina with the blood settling in the space between the retina and vitreoretinal membrane. The hemorrhage has a flat top owing to the blood settling under the force of gravity. The hemorrhage in the vitreous cavity is the “vitreous hemorrhage.”

Detached retina.

These include the tractional retinal detachment and combined tractional and rhegmatogenous retinal detachments.

As a preliminary step, the task involves noting down the presence or the absence of DR lesions and the number of quadrants having those lesions.

Similarly, the macular grading is done by noting the presence or absence of these lesions in (i) outside the outer circle, drawn at 2DD (ii) between the inner and the outer circles, drawn at 1DD and 2DD, respectively, and (iii) within the inner circle. Finally, an assessment of marked lesions is made to know which color boxes were marked.

If only green boxes are marked, for both eyes, a yearly follow-up is advised. This included no evidence of clinical signs of DR, mild non-proliferative DR (NPDR- with microaneurysms only) and no evidence of diabetic macular edema (DME). If any yellow boxes are marked, a review after 6 months is advised. This included lesions/signs of moderate NPDR- more than just microaneurysms but less than severe NPDR. However, if the visual acuity is worse than 20/40, an immediate referral is advised to look for other causes of reduced vision. If any red box is marked, an immediate referral is advised. This included findings of severe NPDR- more than 20 intraretinal hemorrhages in each of 4 quadrants, definite venous beading in 2+ quadrants, prominent IRMA in 1+ quadrant, PDR- neovascularization, vitreous/preretinal hemorrhage and DME.

Pilot Study for Validation

The details regarding the pilot study was mailed to all eligible optometrists (n = 150) at a tertiary eye care institution. Eligible optometrists included working optometrists and optometry students in their final years of their optometry training. Participants were given a choice whether they would like to participate in the pilot study. Their participation and attendance was voluntary and they provided verbal consent. Ninety-nine participants confirmed their participation in the study for the study date. The study involved 50 posterior pole 45 degree retinal images with a mix of normal and all stages of DR from image database of a population-based study on DR. In the initial population-based study, informed consent was obtained from the participants regarding the use of the images for further studies. The institutional review board approved the study. These images were graded by single retina specialist for yearly follow-up, 6 monthly follow-up, and immediate referral. We included one retina specialist in the study so as to avoid inter-observer bias. The retina specialist is experienced with diabetic retinopathy grading and diabetic retinopathy screening for more than two decades.

A baseline information was collected from all participants related to demographics, highest qualification, number of years of optometric education, years of work experience as an optometrist, and information on whether they have seen a retinal image before and prior experience of DR grading. A summary of the baseline information is presented in Table 1.

There were 50 practicing optometrists and 49 optometry students. A didactic lecture was taken for all the participants explaining the features of all the lesions of DR. The retinal images were projected on 2 wall-mounted white screens measuring 102 inches × 144 inches and on 2 Liquid Crystal Display (LCD) monitors measuring 22 inches × 41 inches (SANYO LCD-47XR7H; SANYO Electric, Co., Ltd., Japan), by a projector (SANYO PLC-XT35; SANYO Electric, Co., Ltd.) in an auditorium. The participants were seated such that the screens are clearly visible to them. A guided grading of 10 images was done to make them familiar with the grading process. Thereafter, 50 images were shown, and the participants graded the referral based on the algorithm on plain sheets. All participants graded 50 images; thus, there were 4950 expected referral responses (50 × 99). Ninety-nine participants actually gave 4780 responses (170 referral responses were not given). A comparison of these responses was done with the referral response by the retinal specialist.

Statistical Analysis

Data were analyzed using the Statistical Packages for the Social Sciences (SPSS, version 14.0). Baseline information was analyzed as mean, standard deviation, and range or as proportions. The diagnostic performance of the optometry group was assessed by means of kappa statistics, sensitivity, specificity, area under receiver-operating characteristic curves (AUCs), and 95% confidence intervals.

For an alpha level of 0.05, effect size of 0.3, [10] and a power of 0.85, we estimated that a sample size of 100 participants would be required.

A kappa statistic (κ) between 0.01 and 0.20 was designated as “slight agreement”; 0.21 and 0.40 as “fair agreement”; 0.41 and 0.60 as “moderate agreement”; 0.61 and 0.80 as “substantial agreement”; and 0.81 and 0.99 as “almost perfect agreement.” A subsequent analysis was done to determine the performance of the working optometrists versus the optometry students.

Results

Table 1 presents the demographic characteristics of the study participants. The mean age of the participants was 22 years (range: 19–43 years), 87% being women. Forty-five (90%) of those practicing had completed Bachelor’s degree in Optometry, having undergone an average of 4 years (range: 2–7 years) of optometry education. Of those who responded (n = 37) to the question about “years of experience,” about 57% (21/37) of optometrists reported as having less than 2 years of optometry experience. A majority (91%) reported of having seen a fundus photograph before, and 80% reported as not having prior experience in grading DR.

Table 2 presents the agreement between optometry participants versus retina specialist for DR for the 3 referral criteria. Optometry participants correctly identified 91.5% images that required immediate referral (κ = 0.696). Similarly, they correctly identified 62.5% of images as requiring review after 6 months (κ = 0.462) and 51.2% of those requiring review after 1 year (κ = 0.532).

thumbnail
Table 2. The Number of Correct and Misclassified Images in Each Referral Type and the Agreement Between Optometrists Versus Retina Specialists.

https://doi.org/10.1371/journal.pone.0163108.t002

Table 2 also presents the diagnostic performance among the graders. The sensitivity and specificity of the algorithm among optometrists was 91% and 78% for immediate referral, 62% and 84% for review after 6 months, and 51% and 95% for review after 1 year criteria. The AUC was highest (AUC = 0.855) for immediate referral, second highest (AUC = 0.824) for review after 1 year, and 0.727 for review after 6 months criteria.

Table 3 presents the performance of optometry students versus the working optometrists.

thumbnail
Table 3. Performance of Working Optometrists versus Optometry Students.

https://doi.org/10.1371/journal.pone.0163108.t003

The optometry students performed better for all grades of referral than the working optometrists (AUCs were 0.860, versus 0.842 for immediate referral, 0.748 versus 0.722 for review after 6 months and 0.753 versus 0.720 for review after 1 year, p <0.001 for all criteria).

We also compared the diagnostic performance of the participants who reported as having performed DR grading versus as not having performed prior DR grading. For those who had performed prior DR grading, the AUC was observed to be 0.860 (SE = 0.018) and the AUC for those who had not performed prior DR grading was 0.848 (SE = 0.007) and the differences by chi-square test was not significant, p = 0.534.

To assess the effect of learning curve, we divided the dataset into two sets, consisting of 25 images per set. We analyzed the diagnostic performance in each set; the AUC for the first 25 images was 0.835 (SE = 0.009) and AUC for the second set of 25 images was 0.858 (SE = 0.008). The difference in the diagnostic performance between the first set and second set was not statistically significant, p = 0.06.

Discussion and Conclusions

We report a novel pictorial method for identification of diabetic retinopathy lesions for appropriate time based referrals, namely—immediate, in 6 months, and in one year with a specific focus for referring sight-threatening DR (Or that which need immediate referral). The algorithm tested among optometrists demonstrated 91% sensitivity for images that needed immediate referral. Even though all participants received a uniform training before the commencement of grading, we were interested in determining whether differences exist in the ability of referral between working and student optometrists. We observed that the diagnostic performance of the optometry students was better than that of the working optometrists for all grades of referral. A likely explanation could be that those in training may be more open to learning new concepts compared with the practicing optometrists who may show some resistance or be biased towards concepts acquired with experience.

S1 Table shows the diabetic retinopathy referral guidelines in various nations.

The main differences in these guidelines are the differential referral of moderate and severe nonproliferative DR; few guidelines adopt less than yearly referral while others prefer an immediate referral. Similarly, the referral criteria for diabetic macular edema also differ across studies because of the differences in the definitions used. S1 Table also shows the expected performance of the algorithm in various guidelines. The performance in our study is similar to that used in Vision 2020 India, Scottish DR, and NHMRC Australia guidelines. It is partly similar to that of Pacific Island Nations, New Zealand DR screening, Diabetic Retinopathy Screening Service for Wales, American Academy of Ophthalmology, and the International Council of Ophthalmology guidelines but different from the DR screening program of Ireland.

Previous studies have compared the grading performance of nonophthalmologist versus ophthalmologist. Farley et al[11] observed that using single-field 45 degree nonmydriatic retinal camera, family physicians failed to refer only 10.2% of the 1040 patients who needed referral compared with the referral advice of the ophthalmologist. A sensitivity of 85% to 97% and specificity of 80% to 96% have been reported for trained graders for detecting STDR and referable DR using nonmydriatic retinal images [12,13]. Andonegui et al[14] examined the performance of primary-care physicians in comparison with that of the ophthalmologist in utilizing 5-field nonmydriatic photographs for detecting DR, in a randomized sample of 200 patients. The physicians received an online clinical preparatory education before grading the images. The agreement between primary-care physicians and the ophthalmologist was between 80% and 95% for the detection of DR. Similarly, our study showed sensitivity of 91% and specificity of 78% to detect immediate referrals. However, for yearly follow-up, the sensitivity was lower (51%) but with good specificity (95%). This can be probably owing to erring toward a more frequent follow-up rather a delayed one in less severe DR.

Like the grading algorithm described in the study, there were previous reports of similar simplified grading tools. Rudnisky et al[15] described a web based computer assisted ETDRS algorithm, and found a good agreement for levels of retinopathy and macular edema with standard slide-film stereoscopic 7-field fundus photography. Likewise, Lecleire-Collet et al[16] described DR classification based on visual comparison between three digital color fundus photos and standard retinal photographs and found the diagnosis of severe levels of DR with high sensitivity and specificity (100% and 58% respectively). Aldington et al[17] in the EURODIAB IDDM complication study reported a new system of 45 degree fundus photo field grading which was used by graders. They showed good repeatability and accuracy of the system. Gangaputra et al[18] in the ACCORD Eye study and FIND study described photographic grading scales which were condensed to correspond to clinical scales. The agreement between clinical and the new scale used by graders was 69% (ACCORD eye) and 74% (FIND study).

Over the last few years, the screening of DR has undergone a paradigm shift to digital retinal photography. As the graders are often nonmedical staff, there is a need to have simple algorithms to refer DR for appropriate management. Recently, Brady et al[19] have described an online tool to train non-experts (Amazon mechanical Turk workforce) to diagnose presence or absence of DR. By minimal training they could rapidly and correctly categorize normal versus abnormal. However, this training could not classify the disease to allow referral. The algorithm described in our study and validated among optometrists can be a simple and accurate method for referral. The algorithm can be tweaked appropriately to regional referral guideline. It may also be appropriate to test the algorithm among other care givers such as physicians, nurses, and photographers.

The optometrists are now increasingly involved in care of diabetic eye disease as often they are the first contact for people with diabetes. Currently their grading is influenced by their training during their optometry education and can be inconsistent. The new algorithm allows them to have uniform diagnostic and referral platform. A similar unified grading platform is currently available in few telemedicine networks like Joslin vision network, where all graders are licensed optometrists and use Joslin Vision Network (JVN) protocol for grading DR. However, use of these algorithms require validated standardized method of certification and training. The algorithm can also serve as a good teaching tool in both undergraduate and postgraduate curriculum for physicians, ophthalmologists, optometrist and other allied courses for people involved in DR care.

We observed no significant difference in the performance of those who reported as had done prior DR grading compared with those who had not. We observed no statistically significant learning effect, although the participants performed slightly better for the second set of images presented compared to first set of images. In addition, we did not conduct a re-grading of the images but that would have been interesting. We are unable to report the effect of re-grading, but we believe that conducting regular training would improve the performance of the participants.

Our study also shows that about 9% of the participants incorrectly classified the images that required immediate referral and this may be of concern. However, the 91% of participants who had correctly advised referral had no prior experience in grading DR images. In addition, only one didactic lecture and a guided grading of just 10 images may not be sufficient. The participants need more education and training. Therefore, regular frequent training may help to further increase the number of correct referrals. Although our algorithm utilizes single-field photography which is advantageous for a quick screening of large masses, it will not replace clinical examination using indirect ophthalmoscopy.

Over the last few years, the screening of DR has undergone a paradigm shift to digital retinal photography. As the graders are often nonmedical staff, there is a need to have simple algorithms to refer DR for appropriate management. The algorithm described in this work and validated among optometrists can be a simple and accurate method for referral. The algorithm can be tweaked appropriately to regional referral guideline. It may also be appropriate to test the algorithm among other care givers such as physicians, nurses, and photographers.

In conclusion, we describe a simple referral algorithm for DR, which has been validated among optometrists and can be used in DR screening services for accurate referral of sight-threatening DR for further management and care.

Supporting Information

S1 Table. Diabetic Retinopathy Referral Guidelines in Various Nations.

Abbreviations: DR, diabetic retinopathy; NPDR, nonproliferative diabetic retinopathy; DME, diabetic macular edema; CSME, clinically significant macular edema; NHMRC, National Health and Medical Research Council. Scottish: R0, no DR; R1, mild background DR; R2, referable background DR; R3, referable background; R4, proliferative; R5, enucleated; R6, inadequately visualized; M1, observable maculopathy; M2, referable maculopathy; Wales (DRSSW)- R1, mild background DR; R2, preproliferative DR; R3S, stable post treatment; M1, referable maculopathy; R3A,active proliferative DR. New Zealand and Pacific Island Nations: R0, no DR; R1, minimal DR; R2, mild DR; R3, moderate DR; R4, severe DR; R5, proliferative; RT, stable treated; M0, no maculopathy; M1, minimal; M2, mild; M3, mild; M4, moderate; M5, severe; MT, stable treated; UK-R0, none; R1, background; R2, preproliferative; R3, proliferative; R3S, stable treated proliferative; M0, nonreferable maculopathy; M1, maculopathy. Ireland: R0, no DR; R1, background DR; R3, referable background DR; P1, stable treated DR; P2, unstable treated DR; M0, no maculopathy; M1, referable maculopathy.

https://doi.org/10.1371/journal.pone.0163108.s002

(DOCX)

Author Contributions

  1. Conceptualization: RR TS S. Shetty S. Srinivasan.
  2. Data curation: RR S. Srinivasan.
  3. Formal analysis: S. Srinivasan RR VN.
  4. Investigation: RR S. Shetty TS.
  5. Methodology: RR TS S. Srinivasan.
  6. Project administration: RR.
  7. Supervision: RR TS.
  8. Validation: RR S. Srinivasan VN.
  9. Visualization: S. Srinivasan RR.
  10. Writing – original draft: S. Srinivasan RR TS S. Shetty VN.
  11. Writing – review & editing: S. Srinivasan RR TS.

References

  1. 1. Olson JA, Strachan FM, Hipwell JH, Goatman KA, McHardy KC, Forrester JV, et al. A comparative evaluation of digital imaging, retinal photography and optometrist examination in screening for diabetic retinopathy. Diabet Med. 2003; 20: 528–534. pmid:12823232
  2. 2. The Liverpool Declaration. Royal Liverpool University Hospital, Liverpool. Screening for diabetic retinopathy in Europe 15 years after the St. Vincent declaration. 2005. Available: http://reseauophdiat.aphp.fr/Document/Doc/confliverpool.pdf. Accessed 24 October 2015.
  3. 3. Gupta V, Bansal R, Gupta A, Bhansali A. Sensitivity and specificity of nonmydriatic digital imaging in screening diabetic retinopathy in Indian eyes. Indian J Ophthalmol. 2014; 62: 851–856. pmid:25230960
  4. 4. Rani PK, Raman R, Agarwal S, Paul PG, Uthra S, Margabandhu G, et al. Diabetic retinopathy screening model for rural population: awareness and screening methodology. Rural Remote Health. 2005; 5: 350. pmid:16212469
  5. 5. Connolly V, Unwin N, Sherriff P, Bilous R, Kelly W. Diabetes, prevalence and socioeconomic status: a population based study showing increased prevalence of Type 2 diabetes in the deprived areas. J Epidemiol Community Health. 2000; 54: 173–177. pmid:10746110
  6. 6. Rema M, Pradeepa R. Diabetic retinopathy: an Indian perspective. Indian J Med Res. 2007; 125: 297–310. pmid:17496357
  7. 7. ETDRS Research Group: Grading diabetic retinopathy from stereoscopic color fundus photographs: an extension of the modified early house classification (ETDRS report number 10). Ophthalmology. 1991; 98(Suppl): 786–806.
  8. 8. Wilkinson CP, Ferris FL III, Klein RE, Lee PP, Agardh CD, Davis M, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003; 110: 1677–1682. pmid:13129861
  9. 9. American Academy of Ophthalmology Retina/Vitreous Panel. Preferred Practice Pattern®Guidelines. Diabetic Retinopathy. San Francisco, CA: American Academy of Ophthalmology; 2016. Available at: www.aao.org/ppp.
  10. 10. Bhargava M, Cheung CY, Sabanayagam C, Kawasaki R, Harper CA, Lamoureux EL, et al. Accuracy of diabetic retinopathy screening by trained non-physician graders using non-mydriatic fundus camera. Singapore Med J. 2012; 53: 715–719. pmid:23192497
  11. 11. Farley TF, Mandava N, Prall FR, Carsky C. Accuracy of primary care clinicians in screening for diabetic retinopathy using single-image retinal photography. Ann Fam Med. 2008; 6:428–434. pmid:18779547
  12. 12. Pugh JA, Jacobson JM, Van Heuven WA, Watters JA, Tuley MR, Lairson DR, et al. Screening for diabetic retinopathy. The wide-angle retinal camera. Diabetes Care. 1993; 16: 889–895. pmid:8100761
  13. 13. Klein R, Klein BE, Neider MW, Hubbard LD, Meuer SM, Brothers RJ. Diabetic retinopathy as detected using ophthalmoscopy, a nonmydriatic camera and a standard fundus camera. Ophthalmology. 1985; 92: 485–491. pmid:4000642
  14. 14. Andonegui J, Berástegui L, Serrano L, Eguzkiza A, Gaminde I, Aliseda D. Agreement among ophthalmologists and primary care physicians in the evaluation of retinographies of diabetic patients. Arch Soc Esp Oftalmol. 2008; 83: 527–531. pmid:18803124
  15. 15. Rudnisky CJ, Tennant MT, Weis E, Ting A, Hinz BJ, Greve MD. Web-based grading of compressed stereoscopic digital photography versus standard slide film photography for the diagnosis of diabetic retinopathy. Ophthalmology. 2007; 114: 1748–1754. pmid:17368543
  16. 16. Lecleire-Collet A, Erginay A, Angioi-Duprez K, Deb-Joardar N, Gain P, Massin P. [A new grading system from color fundus photographs for screening for diabetic retinopathy]. J Fr Ophtalmol. 2007; 30: 674–687. pmid:17878820
  17. 17. Aldington SJ, Kohner EM, Meuer S, Klein R, Sjølie AK. Methodology for retinal photography and assessment of diabetic retinopathy: the EURODIAB IDDM complications study. Diabetologia. 1995; 38: 437–444. pmid:7796984
  18. 18. Gangaputra S, Lovato JF, Hubbard L, Davis MD, Esser BA, Ambrosius WT, et al; ACCORD Eye Research Group. Comparison of standardized clinical classification with fundus photograph grading for the assessment of diabetic retinopathy and diabetic macular edema severity. Retina. 2013; 33: 1393–1399. pmid:23615341
  19. 19. Brady CJ, Villanti AC, Pearson JL, Kirchner TR, Gupta OP, Shah CP. Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res. 2014; 16: e233. pmid:25356929