Figures
Abstract
Optical coherence tomography (OCT) derived retinal measures, particularly peri-papillary retinal nerve fiber layer (RNFL) thickness, have been proposed as outcome measures in remyelinating and neuroprotective trials in multiple sclerosis (MS). With increasing utilization of multiple centers to improve power, elucidation of the impact of different OCT technologies is crucial to the design and interpretation of such studies. In this study, we assessed relation and agreement between RNFL thickness and total macular volume (in MS and healthy controls) derived from three commonly used OCT devices: Stratus time-domain OCT, and Cirrus HD-OCT and Spectralis, two spectral-domain (SD) OCT devices. OCT was performed on both Cirrus HD-OCT and Stratus in 229 participants and on both Cirrus HD-OCT and Spectralis in a separate cohort of 102 participants. Pearson correlation and Bland-Altman analyses were used to assess correlation and agreement between devices. All OCT retinal measures correlated highly between devices. The mean RNFL thickness was 7.4 µm lower on Cirrus HD-OCT than Stratus, indicating overall poor agreement for this measurement between these machines. Further, the limits of agreement (LOA) between Cirrus HD-OCT and Stratus were wide (−4.1 to 18.9 µm), indicating poor agreement at an individual subject level. The mean RNFL thickness was 1.94 µm (LOA: −5.74 to 9.62 µm) higher on Spectralis compared to Cirrus HD-OCT, indicating excellent agreement for this measurement across this cohort. Although these data indicate that these three devices agree poorly at an individual subject level (evidenced by wide LOA in both study cohorts) precluding their co-utilization in everyday practice, the small difference for mean measurements between Cirrus HD-OCT and Spectralis indicate pooled results from these two SD-devices could be used as outcome measures in clinical trials, provided patients are scanned on the same machine throughout the trial, similar to the utilization of multiple different MRI platforms in MS clinical trials.
Citation: Warner CV, Syc SB, Stankiewicz AM, Hiremath G, Farrell SK, Crainiceanu CM, et al. (2011) The Impact of Utilizing Different Optical Coherence Tomography Devices for Clinical Purposes and in Multiple Sclerosis Trials. PLoS ONE 6(8): e22947. https://doi.org/10.1371/journal.pone.0022947
Editor: Steven Jacobson, National Institutes of Health, United States of America
Received: April 14, 2011; Accepted: July 1, 2011; Published: August 11, 2011
Copyright: © 2011 Warner et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: National Multiple Sclerosis Society (TR 3760-A-3 to P.A.C. and RG 4212-A-4 to L.J.B.), National Eye Institute (R01 EY 014993 and R01 EY 019473 to L.J.B.), and Braxton Debbie Angela Dillon and Skip (DADS) Donor Advisor Fund (to P.A.C., L.J.B., and E.M.F.). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Dr. Laura Balcer has served as a consultant to Biogen Idec for development of visual outcome measures for MS trials. She is also on the advisory board for an MS trial sponsored by Biogen Idec. Dr. Elliot Frohman has received speaker honoraria and consulting fees from Biogen Idec, TEVA, and Athena. He has also received consulting fees from Abbott Laboratories. Dr. Shiv Saidha has received consulting fees from MedicalLogix for the development of continuing medical education programs. Dr. Peter Calabresi has provided consultation services to Novartis, Teva, Biogen-IDEC, and has received grant support from EMD-Serono, Teva, Biogen-IDEC, Genentech, Bayer, Abbott, and Vertex. There are no patents, products in development or marketed products to declare. This does not alter the authors' adherence to all the PLoS ONE policies on sharing data and materials.
Introduction
Although multiple sclerosis (MS) is conventionally regarded as an immune-mediated demyelinating disorder of the central nervous system (CNS), neuroaxonal degeneration represents a significant proportion of MS pathobiology, and has been shown to be the major correlate of MS related disability [1]–[5]. While the etiology of axonal degeneration in MS remains incompletely elucidated, it is primarily thought to occur either as a result of acute, immune-mediated axonal transection or as a result of chronic demyelination with associated loss of trophic support [6]–[10]. Neuronal atrophy or loss in MS is in turn thought to be the derivative of retrograde axonal degeneration or anterograde transynaptic degeneration [11]–[13]. Since a major pathologic substrate of neuroaxonopathy in MS may be chronic demyelination, and neuroaxonopathy is regarded as the major determinant of disability in MS, remyelinating and neuroprotective strategies have become important therapeutic goals in MS. With increasing research of potential remyelinating and neuroprotective drugs, and the imminent transition of such agents from laboratory to clinical trial testing, tools allowing objective quantification of the effects of such strategies on myelin, neurons and axons are required. One such potential tool proposed for this purpose is optical coherence tomography (OCT) [14], [15].
OCT is a rapid, non-invasive, office-based imaging technique allowing objective quantification of retinal structures with high resolution, including determination of peri-papillary retinal nerve fiber layer (RNFL) thickness and total macular volume (TMV) [16]–[19]. The anterior visual pathway is a frequent target of the MS disease process. Acute optic neuritis (AON) occurs in approximately 30–70% of MS patients during the course of their illness and 94–99% of MS patients have plaques in their optic nerves at post-mortem analysis, irrespective of AON history [20]–[23]. Optic nerve demyelination (due to clinical or subclinical optic neuropathy) results in retrograde degeneration of constituent optic nerve axons. Since these axons originate from the retinal nerve fibers, this process is reflected by RNFL thinning (which may be objectively quantified by OCT) [24]. Retinal nerve fiber degeneration may in turn lead to death of ganglion cells (from which they originate) and contribute to reductions in TMV as measured by OCT [12], [25]. RNFL thickness has been shown to correlate with disability, brain atrophy, visual function and visual quality of life in MS [26]–[32].
Given the predilection of MS to afflict the optic nerves and the lack of confounding related to myelin (the axonal projections of retinal ganglion cells to the optic nerves are uniquely unmyelinated) [33], [34], the eye has been proposed as a model within which to study the neurodegenerative processes associated with MS [14], [15]. Demyelinated yet intact axons, could potentially be remyelinated or protected, resulting in RNFL and ganglion cell preservation. Several characteristics of OCT, including pathological specificity, good structure-function correlation, good reproducibility and reliability and its potential to identify change over time, enable its use to detect and monitor the course of disease-related neurodegeneration in MS and to document the neuroprotective and potentially neurorestorative effects of novel therapeutic agents [25]–[28], [35]–[38].
Within the past decade, the greatest advancement in OCT technology has been the development of fourth-generation spectral domain (SD) OCT. SD-OCT has faster axial scan velocities, higher axial resolution and better reproducibility than older third-generation time domain (TD) OCT technology [37], [39]–[42]. In addition to different generations of OCT technologies (with differing measurement algorithms), there are multiple OCT machines of the same generation produced by different manufacturers, with known differences in the measurement algorithms between these machines [43], [44]. While comparisons have been previously performed between various OCT machines [43]–[45], such comparisons have not specifically assessed agreement or compatibility of conventional OCT measures derived from differing OCT analysis platforms. If multi-center trials utilizing OCT outcome measures are to be performed, this information is crucial to the design and interpretation of these studies. We examined the agreement between measures of RNFL thickness and TMV obtained with Stratus TD-OCT (Carl Zeiss Meditec, Dublin, California), Cirrus High Definition (HD) SD-OCT(Carl Zeiss Meditec, Dublin, California), and Spectralis SD-OCT (Heidelberg Engineering, Heidelberg, Germany) in MS and healthy controls.
Methods
The study protocol for this specific study was approved by the Institutional Review Boards of Johns Hopkins University and the University of Texas Southwestern, and the study was performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki. Written informed consent was obtained from all study participants, who represented an unselected convenience sample of subjects willing to undergo evaluation for research purposes. MS participants had their diagnosis confirmed by the treating neurologist (P.A.C, E.A.F), based on McDonald criteria [46], and MS classification was recorded based on recognized MS subtypes [47] – relapsing remitting MS (RRMS), secondary progressive MS, primary progressive MS and relapsing progressive MS. The latter three progressive MS variants were combined as one group (progressive MS) in analyses due to the small sample sizes of these individual groups. Healthy controls were recruited from among Johns Hopkins University and the University of Texas Southwestern staff and unaffected family members of MS patients. Exclusion criteria for study participation included a history of ocular surgery, retinal disease, glaucoma, hypertension, diabetes, and spherical refractive error of more than 6.0 diopters. All OCT scans were performed without pupillary dilatation by experienced technicians.
In the Cirrus-Stratus cohort, scans were performed in random order on both Stratus OCT model 3000, software version 4.0.2 and Cirrus HD-OCT model 4000, software version 3.0 during the same clinical visit. Using the Stratus OCT device, RNFL thickness was acquired using the fast retinal thickness protocol consisting of three consecutive 3.4 mm-diameter circular scans (256 A-scans/B-scan) centered on the optic disc. The fast macular thickness protocol, which uses six 6-mm long intersecting radial scan lines centered on the fovea (128 A-scans/B scan), was used to calculate TMV. Cirrus HD-OCT RNFL thickness measurements were acquired using the optic disk cube 200×200 protocol. This protocol consists of 200 horizontal scan lines (each composed of 200 A-scans) that form a 6×6×2 mm volume cube from which a circle of 1.73 mm radius is automatically centered on the middle of the optic disc. Cirrus HD-OCT macular data were obtained using the macular cube 512×128 protocol (512 A-scans/ B-scan; the central vertical and horizontal scans are composed of 1024 A-scans), which forms a 6×6×2 mm volume cube. This scan protocol provides a profile of TMV, average macular thickness (AMT) and average thickness for the 9 macular subfields as defined by the Early Treatment Diabetic Retinopathy Study (ETDRS) areas [48]. Scans with signal strengths less than 7 were not included in our analyses.
In the Cirrus-Spectralis cohort, scans were performed in random order on both Cirrus HD-OCT and Spectralis OCT software version 5.2.4 on the same day. The Cirrus HD-OCT model and scan protocol are described above. Spectralis OCT RNFL thickness measurements were acquired using the RNFL-N protocol with an Automatic Real Time (ART) of 16 and a signal quality of at least 20 dB. Spectralis macular data were obtained using 20×20 degree (or greater) raster scans consisting of at least 25 lines each. The macular scans had an ART of at least 11 and a signal quality of at least 20 db. The TMV was calculated within the ETDRS macular grid.
Calculations and statistical analyses were performed using Stata 11.0 (StataCorp, College Station, TX). Only the right eyes of participants were used for analyses to avoid bias due to inter-eye correlation. T-test was used to compare OCT measurements between MS and healthy controls, as the examined variables followed a normal distribution. Pearson correlation coefficients were used to assess the linear relation between OCT-derived retinal measures from different OCT machines. The agreement between retinal measures acquired from different machines was evaluated with the Bland-Altman method [49], [50]. The interscanner agreement index was calculated for both RNFL thickness and TMV measurements for each subject. The interscanner agreement index, as defined by Bland and Altman has previously been used in the quantification of interscanner variation between MRI machines [51], [52], as it is common practice for different MRI analysis platforms to be co-utilized in MS trials [53]–[55].
If xa is the measurement on machine a, and xb is the measurement on machine b, then the interscanner agreement is defined as follows:Stratus and Spectralis OCT calculate TMV from the sum of the 9 macular ETDRS subfields. Alternatively, Cirrus HD-OCT calculates TMV from a much larger area of the retina, the 6×6×2 mm volume cube. To more accurately compare TMV measures between the 3 OCT machines, we used the manufacturer's (Carl Zeiss Meditec) formula to calculate TMV from Cirrus HD-OCT over the 9 macular ETDRS subfields:
Results
A total of 229 participants (RRMS: n = 138, mean age = 40.8±10.7; Progressive MS: n = 22, mean age = 52.3±10.0; Healthy controls: n = 69, mean age = 35.2±9.0) were included in the Cirrus HD-OCT vs. Stratus OCT part of this study. Of these, 174 were women and 55 were men, and there were no significant differences in sex ratios between the groups. Summary of statistical analyses from this cohort are illustrated in Table 1.
Cirrus HD-OCT RNFL thickness and TMV were significantly lower in RRMS and progressive MS than healthy controls (p<0.001 for both measures in both comparisons), in keeping with prior studies [25], [26], [28]. While Cirrus HD-OCT and Stratus OCT measures of RNFL thickness correlated strongly (r = 0.93; P<0.001) (Figure 1) with one another, Cirrus HD-OCT consistently measured lower average RNFL thickness than Stratus, where the mean difference was 7.4 µm (95% CI: 6.7–8.2 µm), indicating poor agreement on average for this measurement between the two machines across the cohort. The mean difference was determined using the Bland-Altman method [49], [50], and the difference in RNFL thickness measurements (Stratus RNFL−Cirrus RNFL) was plotted over mean RNFL thickness (Cirrus RNFL+Stratus RNFL/2) (Figure 2). Moreover, as the mean RNFL thickness increased, the calculated difference between the two devices also increased, resulting in wide 95% limits of agreement (LOA: −4.1 to 18.9 µm) spanning 23 µm, indicating poor agreement at an individual subject level (subgroup analyses presented in Table 1 and Figure 2).
Scatter plot of average RNFL thickness (1a) and TMV (1b) measured by Stratus OCT plotted against average RNFL thickness and TMV measured by Cirrus HD-OCT (n = 229). There was a strong linear agreement between Cirrus HD-OCT and Stratus OCT measurements of RNFL thickness (solid line) (r = 0.93; p<0.001) and TMV (solid line) (r = 0.95; p<0.001). The line of equality (dotted line) demonstrates Cirrus HD-OCT measurements of RNFL thickness are generally lower than with Stratus OCT, and Cirrus HD-OCT measurements of TMV are consistently greater than with Stratus OCT.
Bland-Altman plots show poor agreement between Cirrus HD-OCT and Stratus OCT measures of RNFL thickness for RRMS (2a), progressive MS (2b) and healthy control (2c) eyes and suboptimal agreement between measures of TMV for RRMS (2d), progressive MS (2e) and healthy control (2f) eyes. Within each plot, the solid line indicates the mean RNFL thickness difference, while the dotted line denotes the 95% LOA. On average, Cirrus HD-OCT measured a lower RNFL thickness than Stratus OCT and the LOA were wide. On average, Cirrus HD-OCT measured a higher TMV than Stratus OCT, but with narrow LOA indicating the potential for agreement for this measurement at an individual subject level by extrapolation.
Across the entire cohort, there was a strong correlation between the two machines for measures of TMV (r = 0.95; P<0.001) (Figure 1). Cirrus HD-OCT consistently measured a greater TMV than Stratus OCT (Figure 1), where the mean difference was −1.30 mm3 indicating less than optimal agreement for this measurement between the two devices across the cohort. The 95% LOA ranged from −1.59 mm3 to −1.01 mm3, which spanned 0.58 mm3 (Figure 2), indicating that one can achieve good agreement at an individual subject level by extrapolation (by adding a constant of 1.33 mm3 to Stratus measurements) (subgroup analyses presented in Table 1 and Figure 2).
A separate cohort of 102 participants (RRMS: n = 66; healthy controls: n = 36) were included in the Spectralis vs. Cirrus HD-OCT part of the study, in order to limit strain on our study population. Of these, 84 were women and 18 were men, and there were no significant differences in sex ratios between the groups. The mean age was 41.5±9.7 years in the RRMS group and 34.6±9.3 years in the healthy control group.
RNFL thickness was measured on both Spectralis and Cirrus HD-OCT machines in 95 participants (61 RRMS and 34 healthy controls), while TMV was measured on both devices in 98 participants (63 RRMS and 35 healthy controls). Summary of statistical analyses from this cohort are illustrated in Table 2. Spectralis RNFL thickness and TMV were significantly lower in RRMS than healthy controls (p<0.001 for both), in keeping with prior studies [25], [26], [28]. Overall, there was a strong correlation between Spectralis and Cirrus HD-OCT measures of RNFL thickness (r = 0.96; P<0.001), however the correlation line slightly overlapped the line of equality (Figure 3). Bland-Altman analyses revealed that Spectralis consistently measured higher average RNFL thickness than Cirrus HD-OCT, where the mean difference was 1.94 µm (95% CI: 1.15–2.72 µm), indicating excellent agreement on average for this measurement between the two devices across the cohort (Figure 4). However, as the mean RNFL thickness increased, the calculated difference between the two devices also increased, resulting in wide 95% LOA (−5.74 to 9.62 µm) spanning 15.36 µm, indicating poor agreement at an individual subject level (subgroup analyses presented in Table 2 and Figure 4).
Scatter plot of average RNFL thickness (3a) and TMV (3b) measured by Spectralis OCT plotted against average RNFL thickness and TMV measured by Cirrus HD-OCT (n = 95). There was a strong linear agreement between Spectralis and Cirrus HD-OCT measurements of RNFL thickness (solid line) (r = 0.9622; P<0.001) and TMV (r = 0.9255; P<0.001). The line of equality (dotted line) demonstrates Spectralis measurements of RNFL thickness are slightly higher than with Cirrus HD-OCT, and that Spectralis measurements of TMV are also generally higher than with Cirrus HD-OCT.
Bland-Altman plots show low mean differences (solid lines) indicating good agreement on average across studied cohorts, but with wide LOA (dotted lines) indicating poor agreement at an individual patient/subject level between Spectralis and Cirrus HD-OCT derived measures of RNFL thickness in RRMS (4a) and healthy control (4b) eyes and measures of TMV in RRMS (4d) and healthy control (4e) eyes. On average, Spectralis measured higher values of RNFL thickness and TMV than Cirrus HD-OCT. Analysis of the mean differences between Spectralis and Cirrus HD-OCT by quartile revealed consistent variability between machines with respect to RNFL thickness (4c) and TMV (4f) measures.
Across the cohort (n = 98), there was a strong correlation between the two machines for measurements of TMV (r = 0.93; P<0.001) (Figure 3). Spectralis OCT generally measured a greater TMV than Cirrus HD-OCT (Figure 4), where the average mean difference for TMV between the two devices was 0.387 mm3, and the 95% LOA (0.013 mm3 to 0.761 mm3) spanned 0.748 mm3, indicating fair agreement on average for TMV across the cohort, as well as at an individual subject level (subgroup analyses presented in Table 2 and Figure 4).
The interscanner agreement between Cirrus HD-OCT and Stratus was 91.4±5.4% for RNFL thickness (n = 229) and 82.1±2.1% for TMV (n = 229) (Table 3). Meanwhile, the interscanner agreement between Spectralis and Cirrus HD-OCT was 96.1±3.0% for RNFL thickness (n = 95) and 95.1±2.0% for TMV (n = 98) (Table 3). Boxplots of the interscanner agreement indices for these comparisons are shown in Figure 5. Interscanner agreement remained virtually unchanged when the above analyses were repeated without inclusion of healthy controls.
Boxplots illustrating the inter-quartile range for interscanner agreement indices show stronger agreement between Cirrus HD-OCT and Spectralis OCT measurements than between Cirrus HD-OCT and Stratus OCT. The lower and upper boundaries of each rectangle represent the 25th and 75th percentiles respectively, while the horizontal lines in between these represent the 50th percentile.
Discussion
Results of this study have several important implications relevant to the utilization and interpretation of outcome measures from different OCT devices in clinical trials and at an individual patient level for clinical purposes. The Bland-Altman statistical method is useful for determining how much one method of measurement differs from an existing method of measurement [49], [50]. The mean difference between measurements indicates the level of agreement on average for the measurement of interest between the two devices across the cohort, while the limits of agreement (LOA) indicate the level of agreement for the measurement of interest between the devices at an individual patient or subject level. Clinical judgment must be applied to the interpretation and estimation of level of agreement of the mean differences or LOA generated by the Bland-Altman approach.
Compared to Stratus (TD-OCT), Cirrus HD-OCT (SD-OCT) averaged lower measures of RNFL thickness (mean difference: 7.4 µm) and higher measures of TMV (mean difference: −1.30 mm3), in keeping with previous studies of healthy controls, MS and glaucoma patients [45], [56], [57]. These results suggest poor agreement on average for these measures across the cohort. For this reason, we do not recommend using pooled results of RNFL thickness or TMV from these two devices as outcome measures in clinical trials. Although the LOA for TMV were narrow, the LOA for RNFL thickness were wide (23 µm) suggesting these two devices should also not be used inter-changeably to monitor RNFL thickness clinically. Further, given the correlation line for RNFL thickness between Cirrus and Stratus devices diverges from the line of equality, developing a formula allowing extrapolation of RNFL measures from one device to the other is likely to be difficult and potentially inaccurate.
Likewise, the correlation line for RNFL thickness between Cirrus HD-OCT and Spectralis (both SD-OCT devices) crosses the line of equality making the development of a formula allowing extrapolation of RNFL measures from one of these devices to the other difficult and also inaccurate. However, Spectralis averaged narrowly higher measures of RNFL thickness (mean difference: 1.94 µm) and TMV (mean difference: 0.387 mm3) than Cirrus HD-OCT, in keeping with prior studies of healthy controls, MS and glaucoma patients [58]–[60], suggesting excellent agreement on average for these measures across the cohort. Furthermore, the interscanner agreement indices between Spectralis and Cirrus HD-OCT were higher than those determined between Stratus and Cirrus HD-OCT, and had a median value of 96.8% for RNFL thicknesses and 95.4% for TMV (Table 3). For reference, a previous study found the overall median interscanner agreement was 91.1% for MRI measurements of T2-weighted lesion load, and 96.7% for measurements of T2-weighted lesion load on MRI scanners of the same field strength [51]. Thus, the agreement between SD-OCT machines examined in this study are comparable to the agreements noted between different MRI platforms that have been successfully co-utilized in previous multiple sclerosis trials [51]. It is unclear, however, if interscanner agreement for MRI measures representative of neurodegeneration including T1-weighted lesion load, T1-weighted lesion volume and cerebral atrophy are comparable to those for T2-weighted lesion load outlined above. Further studies assessing this are warranted. Nonetheless, the small mean difference and good interscanner agreement indices seen between the two SD-OCT devices included in this study imply pooled results of RNFL thickness and TMV from these devices could be used as outcome measures in clinical trials, provided patients were scanned on one device throughout the study.
In order to determine the acceptable mean difference between devices, two known factors were considered during the interpretation of the data. First, it is recognized from longitudinal analyses of OCT in MS, of predominantly the relapsing-remitting subtype, that on average each year of follow-up is associated with a 2 µm reduction in RNFL thickness, in the absence of clinical AON events [38]. Although the degree of RNFL change over time in active MS has not been specifically determined, we postulate it may be even greater in subsets of patients with inflammatory disease activity. This may be particularly relevant since MS patients with active MS tend to be targeted for recruitment to clinical trials and trials may be greater than one year in duration. Secondly, AON has been proposed as a disease model within which to study potential neuroprotective/neurorestorative agents, with OCT measures representing primary outcome measures. Following AON, 75% of patients will sustain a 10–40 µm reduction in RNFL thickness in the affected eye within 3–6 months (indicating significant and rapid axonal degeneration secondary to inflammation and demyelination) [27]. Given these two models for clinical trials, the expected difference between study groups is larger than the mean difference observed for the two spectral domain devices included in this study. As such, the mean difference in RNFL measurements between these spectral domain devices suggests excellent agreement on average across the cohort, and the two devices may be used together within a clinical trial. However, it is worth noting that based upon the two models for clinical trials outlined above, a larger sample size is likely to be required for longitudinal studies of MS patients in the absence of AON, given annualized reductions in RNFL thickness in these MS patients are less than those seen over short periods of time following AON. Based upon the wide LOA, however, if both Cirrus HD-OCT and Spectralis OCT were to be utilized to generate pooled measures of RNFL thickness or TMV in clinical trials, it would be imperative that a patient scanned on one device continue to be scanned on that same device for the duration of the study. Since the LOA for RNFL thickness were wide (15.41 µm) between the spectral domain devices analyzed, this indicates poor agreement at an individual subject level, suggesting these two devices should not be used inter-changeably to monitor RNFL thickness clinically. Clinicians should be aware of the poor agreement between Cirrus HD-OCT and Spectralis (as well as between Cirrus HD-OCT and Stratus) for RNFL thickness at an individual patient level and ideally perform scans on two machines for a period while switching from one OCT platform to another in order to allow equivalent comparison retrospectively and prospectively.
Differences in the segmentation algorithms between the OCT devices may explain some of the observed differences in measures of TMV between the devices [43], [44], [61]–[63]. For example, retinal thickness is measured from the inner limiting membrane (ILM) to the inner/outer photoreceptor junction with Stratus, from the ILM to the retinal pigment epithelium layer with Cirrus HD-OCT, and from the ILM to Bruch's membrane with Spectralis. These differences may explain why Stratus measures the lowest TMV measures and Spectralis the highest TMV measures.
Differences in resolution and image acquisition speeds between the investigated OCT devices may also explain some of the observed differences in retinal measures [39]–[42], [64]. Acquisition rates (axial scans per second) and resolution are 400 and 10 µm respectively with Stratus (TD-OCT), 27,000 and 5 µm respectively with Cirrus HD-OCT (SD-OCT), and 40,000 and 3.9 µm respectively with Spectralis (SD-OCT). The 70-fold increase in speed with SD-OCT technology allows greater sampling of retinal information and the generation of 3D images. Since less information is captured with TD-OCT, scan information must be averaged for retinal quantification, and only 2D images are obtained. There are additional differences in the processing techniques of these devices which may also contribute to observed discrepancies in retinal measurements. Cirrus HD-OCT software automatically centers on the optic nerve head and the fovea during RNFL and TMV measurement, while with Stratus OCT, the operator manually performs this task. There is also a time delay between data acquisition capture and the displayed fundus image with Stratus, while Cirrus-HD OCT and Spectralis capture data in real-time. Further, Spectralis utilizes a real-time eye tracking system that is able to adjust for eye movements during scanning. Despite these differences, our findings support suggestions that both TD-OCT and SD-OCT technologies capture RNFL thinning and TMV reduction in eyes of patients with MS [25], [45], [65]. However, it must be noted this was not the primary objective of this study, and as a result the healthy controls and MS participants in this study were not age-matched. It is important to emphasize that three commonly used OCT devices were assessed in terms of their agreement in this study; however, there are several other OCT devices which were not included in this study as they are not available at our institution. If other investigators are considering co-utilizing other OCT devices in clinical trials, it is imperative they first determine the agreement between these devices, rather than generalizing our findings as applicable to all OCT machines. Our finding that Cirrus HD-OCT and Spectralis (both SD-OCT) retinal measures could be potentially pooled as outcome measures may not necessarily imply acceptable agreement between other SD-OCT devices for this purpose. As data in this study were acquired from two separate cohorts (Stratus-Cirrus HD-OCT and Spectralis-Cirrus HD-OCT cohorts), it was not possible to assess agreement between Stratus and Spectralis by Bland-Altman analyses. However, prior studies assessing this demonstrate similarly poor agreement between these two devices, comparable to that observed in our Stratus-Cirrus HD-OCT cohort [66], [67]. Although not determined during this study, but relevant to the findings of this study, our group and others have previously demonstrated excellent intrascanner reproducibility for the OCT devices examined. Our group has found Stratus to have an interrater intraclass correlation (ICC) of 0.89, an intrarater ICC 0.98 and an intervisit ICC of 0.91 [36], while some other groups have reported even higher ICCs for Stratus [68]. Our group has also found Cirrus HD-OCT to have high ICCs, with an interater ICC of 0.97, an intrarater ICC of 0.99 and an intervisit ICC of 0.97 [37]. Similarly, Spectralis has also been shown to have a high intrarater ICC of 0.99 [69].
Trials of potentially remyelinating and neuroprotective drugs are forthcoming in MS. These studies are likely to be multi-center trials in order to enable the recruitment of appropriate numbers of patients to achieve adequate statistical power. The results of this study are relevant to the design and interpretation of studies utilizing OCT for outcome measures, particularly those in which there may be differences in the local availability of OCT technologies across centers in the study. We suggest Stratus TD-OCT should not be co-utilized with either Cirrus HD-OCT or Spectralis SD-OCT. Cirrus HD-OCT and Spectralis (both SD-OCT) may be co-utilized, provided patients scanned on one machine continue to be scanned on that same machine for the duration of the study, since this may improve data reliability consistent with the known excellent same-machine reliability of these devices. Thus for example, a center with only Cirrus HD-OCT and another center with only Spectralis may both participate in such a study. We suggest an equal number of centers with each device should be recruited for study enrolment and an equal number of participants should be recruited at each center, as is the manner in which commonly available MRI platforms have been successfully incorporated into prior MS clinical trials [53]–[55]. We hope our findings increase the ease of utilizing OCT for potential primary outcome measures in clinical trials as we embark upon the highly anticipated next chapter of MS research in remyelination and neuroprotection.
Author Contributions
Conceived and designed the experiments: CVW SBS AMS GH SKF CMC AC TCF ERB LJB EMF SS PAC. Performed the experiments: CVW SBS AMS GH SKF AC. Analyzed the data: CVW SBS AMS CMC SS PAC. Contributed reagents/materials/analysis tools: CVW SBS AMS GH SKF AC EMF PAC. Wrote the paper: CVW SBS AMS GH SKF CMC AC TCF ERB LJB EMF SS PAC.
References
- 1. van Waesberghe JH, Kamphorst W, De Groot CJ, van Walderveen MA, Castelijns JA, et al. (1999) Axonal loss in multiple sclerosis lesions: Magnetic resonance imaging insights into substrates of disability. Ann Neurol 46(5): 747–754.
- 2. De Stefano N, Narayanan S, Francis GS, Arnaoutelis R, Tartaglia MC, et al. (2001) Evidence of axonal damage in the early stages of multiple sclerosis and its relevance to disability. Arch Neurol 58(1): 65–70.
- 3. Minneboo A, Uitdehaag BM, Jongen P, Vrenken H, Knol D, et al. (2009) Association between MRI parameters and the MS severity scale: A 12 year follow-up study. Mult Scler 15(5): 632–637.
- 4. Calabrese M, De Stefano N, Atzori M, Bernardi V, Mattisi I, et al. (2007) Detection of cortical inflammatory lesions by double inversion recovery magnetic resonance imaging in patients with multiple sclerosis. Arch Neurol 64(10): 1416–1422.
- 5. Calabrese M, Agosta F, Rinaldi F, Mattisi I, Grossi P, et al. (2009) Cortical lesions and atrophy associated with cognitive impairment in relapsing-remitting multiple sclerosis. Arch Neurol 66(9): 1144–1150.
- 6. Kornek B, Lassmann H (1999) Axonal pathology in multiple sclerosis. A historical note. Brain Pathol (9): 651–656.
- 7. Kornek B, Storch MK, Weissert R, Wallstroem E, Stefferl A, et al. (2000) Multiple sclerosis and chronic autoimmune encephalomyelitis: A comparative quantitative study of axonal injury in active, inactive, and remyelinated lesions. Am J Pathol 157(1): 267–276.
- 8. Trapp BD, Peterson J, Ransohoff RM, Rudick R, Mork S, et al. (1998) Axonal transection in the lesions of multiple sclerosis. N Engl J Med 338(5): 278–285.
- 9. Bjartmar C, Trapp BD (2001) Axonal and neuronal degeneration in multiple sclerosis: Mechanisms and functional consequences. Curr Opin Neurol 14(3): 271–278.
- 10. Dutta R, Trapp BD (2007) Pathogenesis of axonal and neuronal damage in multiple sclerosis. Neurology 68(22): Suppl 3S22–31; discussion S43–54.
- 11. Rawes JA, Calabrese VP, Khan OA, DeVries GH (1997) Antibodies to the axolemma-enriched fraction in the cerebrospinal fluid and serum of patients with multiple sclerosis and other neurological diseases. Mult Scler 3(6): 363–369.
- 12. Shindler KS, Ventura E, Dutt M, Rostami A (2008) Inflammatory demyelination induces axonal injury and retinal ganglion cell apoptosis in experimental optic neuritis. Exp Eye Res 87(3): 208–213.
- 13. Madigan MC, Rao NS, Tenhula WN, Sadun AA (1996) Preliminary morphometric study of tumor necrosis factor-alpha (TNF alpha)-induced rabbit optic neuropathy. Neurol Res 18(3): 233–236.
- 14. Frohman EM, Fujimoto JG, Frohman TC, Calabresi PA, Cutter G, et al. (2008) Optical coherence tomography: A window into the mechanisms of multiple sclerosis. Nat Clin Pract Neurol 4(12): 664–675.
- 15. Saidha S, Eckstein C, Ratchford JN (2010) Optical coherence tomography as a marker of axonal damage in multiple sclerosis. CML - Multiple Sclerosis 2(2): 33–43.
- 16. Hsu IJ, Sun CW, Lu CW, Yang CC, Chiang CP, et al. (2003) Resolution improvement with dispersion manipulation and a retrieval algorithm in optical coherence tomography. Appl Opt 42(2): 227–234.
- 17. Huang D, Swanson EA, Lin CP, Schuman JS, Stinson WG, et al. (1991) Optical coherence tomography. Science 254(5035): 1178–1181.
- 18. Parisi V, Manni G, Spadaro M, Colacino G, Restuccia R, et al. (1999) Correlation between morphological and functional retinal impairment in multiple sclerosis patients. Invest Ophthalmol Vis Sci 40(11): 2520–2527.
- 19. Trip SA, Schlottmann PG, Jones SJ, Altmann DR, Garway-Heath DF, et al. (2005) Retinal nerve fiber layer axonal loss and visual dysfunction in optic neuritis. Ann Neurol 58(3): 383–391.
- 20. Balcer LJ (2006) Clinical practice. optic neuritis. N Engl J Med 354(12): 1273–1280.
- 21. Frohman EM, Frohman TC, Zee DS, McColl R, Galetta S (2005) The neuro-ophthalmology of multiple sclerosis. Lancet Neurol 4(2): 111–121.
- 22. Toussaint D, Perier O, Verstappen A, Bervoets S (1983) Clinicopathological study of the visual pathways, eyes, and cerebral hemispheres in 32 cases of disseminated sclerosis. J Clin Neuroophthalmol 3(3): 211–220.
- 23. Ikuta F, Zimmerman HM (1976) Distribution of plaques in seventy autopsy cases of multiple sclerosis in the united states. Neurology 26(6 PT 2): 26–28.
- 24. Naismith RT, Xu J, Tutlam NT, Trinkaus K, Cross AH, et al. (2010) Radial diffusivity in remote optic neuritis discriminates visual outcomes. Neurology 74(21): 1702–1710.
- 25. Burkholder BM, Osborne B, Loguidice MJ, Bisker E, Frohman TC, et al. (2009) Macular volume determined by optical coherence tomography as a measure of neuronal loss in multiple sclerosis. Arch Neurol 66(11): 1366–1372.
- 26. Fisher JB, Jacobs DA, Markowitz CE, Galetta SL, Volpe NJ, et al. (2006) Relation of visual function to retinal nerve fiber layer thickness in multiple sclerosis. Ophthalmology 113(2): 324–332.
- 27. Costello F, Coupland S, Hodge W, Lorello GR, Koroluk J, et al. (2006) Quantifying axonal loss after optic neuritis with optical coherence tomography. Ann Neurol 59(6): 963–969.
- 28. Pulicken M, Gordon-Lipkin E, Balcer LJ, Frohman E, Cutter G, et al. (2007) Optical coherence tomography and disease subtype in multiple sclerosis. Neurology 69(22): 2085–2092.
- 29. Gordon-Lipkin E, Chodkowski B, Reich DS, Smith SA, Pulicken M, et al. (2007) Retinal nerve fiber layer is associated with brain atrophy in multiple sclerosis. Neurology 69(16): 1603–1609.
- 30. Siger M, Dziegielewski K, Jasek L, Bieniek M, Nicpan A, et al. (2008) Optical coherence tomography in multiple sclerosis: Thickness of the retinal nerve fiber layer as a potential measure of axonal loss and brain atrophy. J Neurol 255(10): 1555–1560.
- 31. Villoslada P, Sepulcre J, Toledo J, Bejarano B (2008) Retinal nerve fiber layer is associated with brain atrophy in multiple sclerosis. Neurology 71(21): 1747; author reply 1747–8.
- 32. Grazioli E, Zivadinov R, Weinstock-Guttman B, Lincoff N, Baier M, et al. (2008) Retinal nerve fiber layer thickness is associated with brain MRI outcomes in multiple sclerosis. J Neurol Sci 268(1–2): 12–17.
- 33. Perry VH, Lund RD (1990) Evidence that the lamina cribrosa prevents intraretinal myelination of retinal ganglion cell axons. J Neurocytol 19(2): 265–272.
- 34.
Heckenlively AG JR, editor. (1991) Principles and practices of clinical electrophysiology of vision. Mosby Year Book Inc. pp. 25–52.
- 35. Schuman JS, Pedut-Kloizman T, Pakter H, Wang N, Guedes V, et al. (2007) Optical coherence tomography and histologic measurements of nerve fiber layer thickness in normal and glaucomatous monkey eyes. Invest Ophthalmol Vis Sci 48(8): 3645–3654.
- 36. Cettomai D, Pulicken M, Gordon-Lipkin E, Salter A, Frohman TC, et al. (2008) Reproducibility of optical coherence tomography in multiple sclerosis. Arch Neurol 65(9): 1218–1222.
- 37. Syc SB, Warner CV, Hiremath GS, Farrell SK, Ratchford JN, et al. (2010) Reproducibility of high-resolution optical coherence tomography in multiple sclerosis. Mult Scler 16(7): 829–839.
- 38. Talman LS, Bisker ER, Sackel DJ, Long DA Jr, Galetta KM, et al. (2010) Longitudinal study of vision and retinal nerve fiber layer thickness in multiple sclerosis. Ann Neurol 67(6): 749–760.
- 39. Choma M, Sarunic M, Yang C, Izatt J (2003) Sensitivity advantage of swept source and fourier domain optical coherence tomography. Opt Express 11(18): 2183–2189.
- 40. de Boer JF, Cense B, Park BH, Pierce MC, Tearney GJ, et al. (2003) Improved signal-to-noise ratio in spectral-domain compared with time-domain optical coherence tomography. Opt Lett 28(21): 2067–2069.
- 41. Wojtkowski M, Srinivasan V, Fujimoto JG, Ko T, Schuman JS, et al. (2005) Three-dimensional retinal imaging with high-speed ultrahigh-resolution optical coherence tomography. Ophthalmology 112(10): 1734–1746.
- 42. Keane PA, Mand PS, Liakopoulos S, Walsh AC, Sadda SR (2009) Accuracy of retinal thickness measurements obtained with cirrus optical coherence tomography. Br J Ophthalmol 93(11): 1461–1467.
- 43. Menke MN, Dabov S, Sturm V (2009) Comparison of three different optical coherence tomography models for total macular thickness measurements in healthy controls. Ophthalmologica 223(6): 352–356.
- 44. Grover S, Murthy RK, Brar VS, Chalam KV (2009) Comparison of retinal thickness in normal eyes using stratus and spectralis optical coherence tomography. Invest Ophthalmol Vis Sci.
- 45. Bock M, Brandt AU, Dorr J, Pfueller CF, Ohlraun S, et al. (2010) Time domain and spectral domain optical coherence tomography in multiple sclerosis: A comparative cross-sectional study. Mult Scler 16(7): 893–896.
- 46. Polman CH, Reingold SC, Edan G, Filippi M, Hartung HP, et al. (2005) Diagnostic criteria for multiple sclerosis: 2005 revisions to the “McDonald criteria”. Ann Neurol 58(6): 840–846.
- 47. Lublin FD, Reingold SC (1996) Defining the clinical course of multiple sclerosis: Results of an international survey. national multiple sclerosis society (USA) advisory committee on clinical trials of new agents in multiple sclerosis. Neurology 46(4): 907–911.
- 48. ETDRS Research Group. (1985) Early treatment diabetic retinopathy study research group. photocoagulation for diabetic macular edema. early treatment diabetic retinopathy study report number 1. Arch Ophthalmol 103(12): 1796–1806.
- 49. Altman DG, Bland JM (1986) Comparison of methods of measuring blood pressure. J Epidemiol Community Health 40(3): 274–277.
- 50. Bland JM, Altman DG (1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1(8476): 307–310.
- 51. Filippi M, van Waesberghe JH, Horsfield MA, Bressi S, Gasperini C, et al. (1997) Interscanner variation in brain MRI lesion load measurements in MS: Implications for clinical trials. Neurology 49(2): 371–377.
- 52. Filippi M, Rocca MA, Gasperini C, Sormani MP, Bastianello S, et al. (1999) Interscanner variation in brain MR lesion load measurements in multiple sclerosis using conventional spin-echo, rapid relaxation-enhanced, and fast-FLAIR sequences. AJNR Am J Neuroradiol 20(1): 133–137.
- 53. Comi G, Pulizzi A, Rovaris M, Abramsky O, Arbizu T, et al. (2008) Effect of laquinimod on MRI-monitored disease activity in patients with relapsing-remitting multiple sclerosis: A multicentre, randomised, double-blind, placebo-controlled phase IIb study. Lancet 371(9630): 2085–2092.
- 54. Comi G, Cohen JA, Arnold DL, Wynn D, Filippi M, et al. (2011) Phase III dose-comparison study of glatiramer acetate for multiple sclerosis. Ann Neurol 69(1): 75–82.
- 55. Radue EW, Stuart WH, Calabresi PA, Confavreux C, Galetta SL, et al. (2010) Natalizumab plus interferon beta-1a reduces lesion formation in relapsing multiple sclerosis. J Neurol Sci 292(1–2): 28–35.
- 56. Knight OJ, Chang RT, Feuer WJ, Budenz DL (2009) Comparison of retinal nerve fiber layer measurements using time domain and spectral domain optical coherent tomography. Ophthalmology 116(7): 1271–1277.
- 57. Vizzeri G, Weinreb RN, Gonzalez-Garcia AO, Bowd C, Medeiros FA, et al. (2009) Agreement between spectral-domain and time-domain OCT for measuring RNFL thickness. Br J Ophthalmol 93(6): 775–781.
- 58. Tan BB, Natividad M, Chua KC, Yip LW (2011) Comparison of retinal nerve fiber layer measurement between 2 spectral domain OCT instruments. J Glaucoma.
- 59. Garcia-Martin E, Pueyo V, Pinilla I, Ara JR, Martin J, et al. (2011) Fourier-domain OCT in multiple sclerosis patients: Reproducibility and ability to detect retinal nerve fiber layer atrophy. Invest Ophthalmol Vis Sci 52(7): 4124–4131.
- 60. Leite MT, Rao HL, Weinreb RN, Zangwill LM, Bowd C, et al. (2011) Agreement among spectral-domain optical coherence tomography instruments for assessing retinal nerve fiber layer thickness. Am J Ophthalmol 151(1): 85–92.e1.
- 61. Pons ME, Garcia-Valenzuela E (2005) Redefining the limit of the outer retina in optical coherence tomography scans. Ophthalmology 112(6): 1079–1085.
- 62. Costa RA, Calucci D, Skaf M, Cardillo JA, Castro JC, et al. (2004) Optical coherence tomography 3: Automatic delineation of the outer neural retinal boundary and its influence on retinal thickness measurements. Invest Ophthalmol Vis Sci 45(7): 2399–2406.
- 63. Costa RA, Skaf M, Melo LA Jr, Calucci D, Cardillo JA, et al. (2006) Retinal assessment using optical coherence tomography. Prog Retin Eye Res 25(3): 325–353.
- 64. Jeoung JW, Park KH (2010) Comparison of cirrus OCT and stratus OCT on the ability to detect localized retinal nerve fiber layer defects in preperimetric glaucoma. Invest Ophthalmol Vis Sci 51(2): 938–945.
- 65. Saidha S, Syc SB, Ibrahim MA, Eckstein C, Warner CV, et al. (2011) Primary retinal pathology in multiple sclerosis as detected by optical coherence tomography. Brain 134(Pt 2): 518–533.
- 66. Seibold LK, Mandava N, Kahook MY (2010) Comparison of retinal nerve fiber layer thickness in normal eyes using time-domain and spectral-domain optical coherence tomography. Am J Ophthalmol 150(6): 807–814.
- 67. Arthur SN, Smith SD, Wright MM, Grajewski AL, Wang Q, et al. (2011) Reproducibility and agreement in evaluating retinal nerve fibre layer thickness between stratus and spectralis OCT. Eye (Lond) 25(2): 192–200.
- 68. Budenz DL, Fredette MJ, Feuer WJ, Anderson DR (2008) Reproducibility of peripapillary retinal nerve fiber thickness measurements with stratus OCT in glaucomatous eyes. Ophthalmology 115(4): 661–666.e4.
- 69. Langenegger SJ, Funk J, Toeteberg-Harms M (2011) Reproducibility of retinal nerve fiber layer thickness measurements using the eye tracker and retest function of spectralis(R) SD-OCT in glaucomatous eyes and healthy controls. Invest Ophthalmol Vis Sci 52(6): 3338–3344.