To create a standardized method for evaluating the video recordings of patients self-instilling eye drops and to determine the level of agreement of eye drop instillation efficacy, safety and efficiency ratings by three masked graders.
78 patients with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with the use of eye drop medications.
Participants were video recorded while self-instilling artificial tears sequentially to both eyes. Three masked observers graded these video recordings on three criteria: efficacy (the determination of whether an eye drop was instilled on the ocular surface), safety (assessment of whether the tip of the medication bottle made contact with the ocular surface or eyelids), and efficiency (the number of eye drops expressed from the bottle).
Main Outcome Measures
After grading the video recordings based on efficacy, safety, and efficiency, kappa statistics were used to estimate inter-rater agreement.
The mean kappa level of agreement for efficacy, safety, and efficiency was 0.64 (95% confidence interval (CI), 0.42–0.87), 0.73 (95% CI, 0.58–0.88), and 0.62 (95% CI, 0.42–0.81), respectively.
Citation: Park MS, Patel MM, Sarezky D, Rojas C, Choo C, Choi M, et al. (2016) Inter-Rater Agreement in the Assessment of Video Recordings of Eye Drop Instillation by Glaucoma Patients. PLoS ONE 11(1): e0145764. doi:10.1371/journal.pone.0145764
Editor: Fabio Rapallo, University of East Piedmont, ITALY
Received: October 1, 2015; Accepted: December 8, 2015; Published: January 5, 2016
Copyright: © 2016 Park et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: APT is supported by an unrestricted grant to Northwestern University from Research to Prevent Blindness, New York, NY. The Northwestern University Biostatistics Consultation Service received funding from the Department of Ophthalmology to carry out the statistical analysis. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: APT has the following competing interests: Alcon Laboratories, consultant; Watson Laboratories, consultant; Apotex, Inc., consultant; Aeon Astron, Inc., consultant; and Sandoz, Inc., consultant. This does not alter the authors’ adherence to PLOS ONE policies on sharing data and materials.
Current treatment for glaucoma relies heavily on the self-administration of ocular hypotensive eye drops. Several studies have examined the various reasons underlying improper instillation of eye drops by patients, including poor visual acuity , older age [2–4], and physical comorbidities . One of the main barriers to glaucoma eye drop compliance is the physical coordination required to, under ideal circumstances, express a single drop such that it lands on the ocular surface without the tip of the bottle making contact with the eye, eyelids or face. The ability to properly administer an eye drop requires visualizing  and squeezing the bottle [1, 3], and also detecting whether the drop has landed on the eye. Studies that have video recorded patients administering their eye drops have found that a large proportion of patients do not correctly instill their eye drops in that they miss the conjunctival sac  or use multiple drops when only one is needed [2–3, 5].
Furthermore, patients may contaminate the bottle tip or sustain trauma through touching the ocular surface or eyelids. Many glaucoma patients may have other co-morbid ocular conditions that may predispose them to eye infections such as the presence of a trabeculectomy bleb, for example. Geyer, et al  reported that the bottle tip was more contaminated than the eye drops themselves, allowing transfer of microorganisms if the bottle tip touches the patient’s ocular surface.
In using more than one drop, patients may run out of eye drops earlier, and thus may need a medication refill before the insurance company will cover it. If a patient cannot financially cover the cost of paying for a medication out-of-pocket , the patient may use the medication less frequently or run out and stop using a medication completely until the next refill is covered. This would lead to further problems of nonadherence due to improper dosing. Furthermore, there may be more long-term side effects such as increased lid pigmentation from a prostaglandin analog if an excessive number of drops are used or if the medication is inadvertently applied to the eyelid [9–10].
We propose a three-part classification system for the evaluation of eye drop self-instillation, in which determinations are made as to whether an eye drop landed on the ocular surface (efficacy), whether there was contact between the tip of the bottle and the ocular surface or eyelids (safety), and how many drops were expressed (efficiency). Although previous studies have variably assessed similar outcomes using video recordings, those studies did not attempt to validate the methodology. The goal of this study is to validate the use of video recordings to measure efficacy, safety and efficiency of eye drop self-instillation by assessing inter-rater agreement by three masked graders.
Materials and Methods
This study was approved by the Northwestern University Institutional Review Board. Written informed consent was obtained from each participant. The study followed the tenets of the Declaration of Helsinki, and is in compliance with the Health Insurance Portability and Accountability Act of 1996.
Patients aged 40–85 years old with open-angle glaucoma or ocular hypertension who had at least 6 months of experience with ocular hypotensive eye drops were sequentially recruited for the study at the time of their clinical office visits. To be included in the study, patients had to self-administer 1–3 eye drop medications daily at home, and have a best corrected visual acuity of 20/50 or better in each eye. Exclusion criteria included moderate to severe cognitive defects as defined by a mini mental status exam score < 20, or the presence of a clinically significant tremor. Once participants fulfilled the inclusion-exclusion criteria and gave informed consent, they were video recorded while self-instilling artificial tears. The video camera was a digital HD recorder, with 42X zoom. Participants were given an opened, sterile bottle of artificial tears (10 mL), and instructed to instill 1 drop onto each eye, in their typical manner, while being recorded. All participants were given access to a mirror and/or a reclining chair to mimic their usual practice of eye drop instillation if they desired. No practice sessions were allowed and the first attempt was recorded. The research assistant was allowed to film the patient in any view and zoom she deemed allowed her the best view of the eye bottle tip and ocular surface.
Three masked graders (MSP, MP, DS) independently scored the 156 video recordings of 78 participants based on efficacy, safety, and efficiency using a forced choice protocol. The videos were viewed on a 13 inch Macbook Pro at 0.5X speed. Graders were medical students who were familiar with the literature on glaucoma medication compliance and were trained using sample video recordings. Graders were allowed to replay the videos an unlimited number of times. Efficacy was graded in a binary fashion: the participant either successfully instilled at least one drop on the ocular surface or failed to do so. Safety was also graded in a binary fashion: the bottle tip either did or did not make contact with the ocular surface or periocular skin. Efficiency was graded as the number of drops expelled from the bottle (from 0 to a maximum of 5). Participants who instilled a continuous stream of eye drops were given the maximum score of 5.
Kappa statistics were used to estimate inter-rater agreement. Weighted kappa was used for inter-rater agreement of efficiency. Analyses were done separately for right and left eyes. Kappa levels and their confidence intervals were then averaged across the three raters and two eyes.
All raw data are included in a supporting information Excel spreadsheet (S1 Raw Data). Kappa statistics describing the level of agreement among the graders are summarized in Table 1. Fig 1 graphically demonstrates the proportion of eyes for which there was agreement among graders.
Graphical representation of the agreement among graders for efficacy, safety and efficiency, separated by right and left eyes.
Efficacy: The mean kappa level of agreement for efficacy was 0.60 (95% CI, 0.36–0.84) for the right eye and 0.68 (95% CI, 0.47–0.89) for the left eye. When combining the average of the right and left eyes, the mean kappa level of agreement for efficacy was 0.64 (95% CI, 0.42–0.87). The range of grader scores of percentages of patients with successful eye drop instillation onto the ocular surface was 79.5–87.2% and 74.4–87.2% for the right and left eye, respectively.
Safety: The mean kappa level of agreement for safety was 0.75 (65% CI, 0.60–0.89) for the right eye and 0.71 (95% CI, 0.56–0.87) for the left eye. When combining the average of the right and left eyes, the mean kappa level of agreement for safety was 0.73 (95% CI, 0.58–0.88). The range of grader scores of percentages of patients making no contact between the bottle tip and ocular or skin surface was 46.2–66.7% and 57.3–74.4% for the right and left eye, respectively.
Efficiency: The mean kappa level of agreement for number of drops instilled was 0.55 (95% CI, 0.37–0.73) for the right eye and 0.68 (95% CI, 0.47–0.88) for the left eye. When combining the average of the right and left eye, the mean kappa level of agreement for efficiency was 0.62 (95% CI, 0.42–0.81). The range of grader scores of percentages of patients using 1 drop per instillation was 64.1–76.9% and 64.1–89.7% for the right and left eye, respectively.
We propose a standardized system for evaluating eye drop self-instillation technique using three domains. Efficacy is a determination of whether an eye drop made contact with the ocular surface. Safety is a determination of whether the tip of the bottle made physical contact with the ocular surface or eyelid or facial skin. Efficiency is a determination of the number of eye drops expressed from the medication bottle. Using this evaluation system, we evaluated the eye drop instillation technique of both eyes of 78 patients. The kappa levels of agreement combined for both eyes for efficacy (0.64), safety (0.73), and efficiency (0.62) demonstrated substantial agreement among graders, suggesting the assessment of video recordings to evaluate eye drop self-instillation technique is valid and reasonably reproducible.
Prior studies have used video recording to assess whether patients successfully self-instill eye drops [9, 11–13]. Stone, et al  used a single grader observing video recordings of glaucoma patients and found that 14 of 64 (21.9%) and 36 of 117 (30.8%) patients successfully instilled a single drop into the eye without touching a 15 ml and 2.5 ml bottle tip to the eye, respectively. Hennessy, et al  also used a single grader and found that 39% of patients successfully instilled a single drop into the eye without touching the bottle tip to the eye. Since only one grader assessed the video recordings in these studies, the reproducibility of the assessments of the recordings is unknown.
The results of the current study are similar to those reported previously with respect to successful self-instillation of eye drops. Stone, et al  found that 96 of 116 (82.8%) with a 15 mL bottle and 48 of 64 (75.0%) with a 2.5 mL bottle successfully instilled an eye drop onto the ocular surface, compared to 60 of 78 (76.9%) and 59 of 78 (75.6%) for the right and left eye, respectively in the current study (Fig 2a and 2b). Hennessy, et al  similarly reported that 71% of glaucoma patients successfully instilled a drop onto the ocular surface.
Fig 2a and 2b show the efficacy ratings that were assigned to each patient for the right and left eyes, respectively, by each grader. All three graders agreed for 83.3% of right eyes and 82.1% of left eyes. A score of 1 indicates successful instillation of the eye drop. A score of 2 indicates an eye drop did not land on the ocular surface.
Of the three parameters, patients were least successful with safety: 23 of 78 (29.5%) and 22 of 78 (28.2%) patients touched the bottle tip to the ocular adnexae of the right and left eyes, respectively, in this study (Fig 3a and 3b). This compares to 34% of patients touching the bottle tip to the eye or eyelashes and 52% of patients touching the face with the bottle tip in a previous report by Sleath, et al . In Hennessy, et al, 33% of glaucoma patients video recorded self-administering eye drops  made contact between the eye and bottle tip. This highlights the importance in educating glaucoma patients of the need to avoid touching the bottle tip to the ocular surface, thus reducing the risk of trauma or infection.
Fig 3a and 3b show the safety ratings that were assigned to each patient for the right and left eyes, respectively, by each grader. All three graders agreed in 82.1% of right eyes and 79.5% of left eyes. A score of 1 indicates the bottle tip did not touch the ocular surface or eyelid. A score of 2 indicates the bottle tip touched the ocular surface, eyelids, eyelashes, or cheek.
Hennessy, et al reported 147 of 204 (72.1%) used one drop , compared to 49 of 78 (62.8%) and 56 of 78 (71.8%) in the right eye and left eye, respectively for the current study (Fig 4a and 4b; Table 2). 25 of 64 (39.1%) and 55 of 116 (47.4%) of patients in the Stone, et al  study used only eye drop per instillation using the 15 ml and 2.5 ml bottle respectively. They also reported 19 of 64 (29.7%) and 26 of 116 (22.4%) of patients instilling a continuous stream of fluid, which is much higher compared to our study.
Fig 4a and 4b show the graders’ assessments of the number of eye drops expressed from the bottle for the right and left eyes, respectively. All three graders gave a score of 1 (only 1 eye drop was instilled) for 62.8% of right eyes and 71.8% of left eyes. Scores of 0–5 were allowed, with 5 indicating a continuous stream was expressed.
A limitation of this study is the fact the video recordings were made using standard lighting conditions and amateur videographers. Therefore, the video recordings may not have been of the highest possible quality, although this may be more representative of the methodology and typical conditions utilized by most investigators. The artificial tear bottle used in the study may differ significantly from the glaucoma medication bottle used by the participants at home. Some video recordings may have been blurry or may not have clearly captured the entire process of eye drop instillation. Graders were required to make forced choice decisions, so it is possible that agreement could have been higher if recordings of poor quality were excluded from the analysis. Although we did not capture information on the quality of the recordings, we believe the vast majority were of high quality. Kappa statistics appropriately adjust for statistical chance agreement and therefore, are lower than the actual agreement observed in the data. In this study, the average kappa statistics were 30% lower than raw agreement for efficacy and efficiency, and 16% lower for safety. Therefore, kappa statistics, although statistically appropriate, underestimate the actual agreement.
This is the first study that analyzes the validity of using a three-component system for grading proper self-instillation of eye drops: efficacy, safety, and efficiency. As one of the main treatments for glaucoma is topical medications to control intraocular pressure, glaucoma patients should ideally know how to properly instill a single eye drop onto the ocular surface. This would not only allow for better control of intraocular pressure, but would minimize excessive costs of using more than one drop per instillation [2–3, 5–6, 8]. Furthermore, patients must not contaminate the bottle tip to avoid introducing pathogens that could lead to infections . Video recording is advantageous in that it guides both researchers and clinicians in determining whether patients are properly administering eye drops. Using the three parameters of efficacy, safety, and efficiency may facilitate better understanding of the barriers to proper eye drop instillation. Use of this validated, standardized method to grade eye drop self-instillation technique will allow investigators to compare results from different studies in a more meaningful way which will help advance research in this field.
S1 Raw Data. Excel spreadsheet with raw data for 78 participants.
Conceived and designed the experiments: MSP MMP MC APT. Performed the experiments: MSP MMP DS CR CC MC. Analyzed the data: MSP MMP DS DL AWR APT. Wrote the paper: MSP MMP DS AWR APT.
- 1. Aptel F, Masset H, Burillon C, Robin A, Denis P. The influence of disease severity on quality of eye-drop administration in patients with glaucoma or ocular hypertension. Br J Ophthalmol. 2009;93(5):700–701.
- 2. Brown MM, Brown GC, Spaeth GL. Improper topical self-administration of ocular medication among patients with glaucoma. Can J Ophthalmol. 1984;19(1):2–5. pmid:6608974
- 3. Kass MA, Hodapp E, Gordon M, Kolker AE, Goldberg I. Patient administration of eye drops: observation. Part II. Ann Ophthalmol. 1982;14(9):889–893. pmid:7181353
- 4. Dickinson C, Shim M. The influence of manual dexterity on reading speed with a hand-held magnifier. Invest Ophthalmol Vis Sci. 2007;48(9):4368–4374. pmid:17724229
- 5. Winfield AJ, Jessiman D, Williams A, Esakowitz L. A study of the causes of non-compliance by patients prescribed eyedrops. Br J Ophthalmol. 1990;74(8):477–480. pmid:2390523
- 6. Burns E, Mulley GP. Practical problems with eye-drops among elderly ophthalmology outpatients. Age Ageing. 1992;21(3):168–170. pmid:1615776
- 7. Geyer O, Bottone EJ, Podos SM, Schumer RA, Asbell PA. Microbial contamination of medications used to treat glaucoma. Br J Ophthalmol. 1995;79(4):376–379. pmid:7742288
- 8. Sleath B, Robin AL, Covert D, Byrd JE, Tudor G, Svarstad B. Patient-reported behavior and problems in using glaucoma medications. Ophthalmology. 2006;113(3):431–436. pmid:16458967
- 9. Stone JL, Robin AL, Novack GD, Covert DW, Cagle GD. An objective evaluation of eyedrop instillation in patients with glaucoma. Arch Ophthalmol. 2009;127(6):732–736. doi: 10.1001/archophthalmol.2009.96. pmid:19506189
- 10. Alm A, Grierson I, Shields MB. Side effects associated with prostaglandin analog therapy. Surv Ophthalmol. 2008;53(6):S93–S105.
- 11. Hennessy AL, Katz J, Covert D, Protzko C, Robin AL. Videotaped evaluation of eyedrop instillation in glaucoma patients with visual impairment or moderate to severe visual field loss. Ophthalmology. 2010;117(12):2345–2352. doi: 10.1016/j.ophtha.2010.03.040. pmid:20580092
- 12. Hennessy AL, Katz J, Covert D, Kelly CA, Suan EP, Speicher MA et al. A video study of drop instillation in both glaucoma and retina patients with visual impairment. Am J Ophthalmol. 2011;152(6):982–988. doi: 10.1016/j.ajo.2011.05.015. pmid:21821228
- 13. Sleath B, Blalock S, Covert D, Stone J, Skinner A, Muir K et al. The relationship between glaucoma medication adherence, eye drop technique, and visual field defect Severity. Ophthalmology. 2011;118(12):2398–2402. doi: 10.1016/j.ophtha.2011.05.013. pmid:21856009