Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Reproducibility of Resting State Connectivity in Patients with Stable Multiple Sclerosis

  • Daniela Pinter ,

    daniela.pinter@medunigraz.at

    Affiliation Department of Neurology, Medical University of Graz, Graz, Austria

  • Christian Beckmann,

    Affiliation Donders Institute, Cognitive Neuroscience Department and Centre for Cognitive Neuroimaging, Radboud University Nijmegen, Nijmegen, The Netherlands

  • Marisa Koini,

    Affiliation Department of Neurology, Medical University of Graz, Graz, Austria

  • Eva Pirker,

    Affiliation Department of Neurology, Medical University of Graz, Graz, Austria

  • Nicola Filippini,

    Affiliation The Oxford Centre for Functional MRI of the Brain, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom

  • Alexander Pichler,

    Affiliation Department of Neurology, Medical University of Graz, Graz, Austria

  • Siegrid Fuchs,

    Affiliation Department of Neurology, Medical University of Graz, Graz, Austria

  • Franz Fazekas,

    Affiliation Department of Neurology, Medical University of Graz, Graz, Austria

  • Christian Enzinger

    Affiliations Department of Neurology, Medical University of Graz, Graz, Austria, Division of Neuroradiology, Department of Radiology, Medical University of Graz, Graz, Austria

Reproducibility of Resting State Connectivity in Patients with Stable Multiple Sclerosis

  • Daniela Pinter, 
  • Christian Beckmann, 
  • Marisa Koini, 
  • Eva Pirker, 
  • Nicola Filippini, 
  • Alexander Pichler, 
  • Siegrid Fuchs, 
  • Franz Fazekas, 
  • Christian Enzinger
PLOS
x

Abstract

Given increasing efforts to use resting-state fMRI (rfMRI) as a biomarker of disease progression in multiple sclerosis (MS) we here explored the reproducibility of longitudinal rfMRI over three months in patients with clinically and radiologically stable MS. To pursue this aim, two approaches were applied in nine rfMRI networks: First, the intraclass correlation coefficient (ICC 3,1) was assessed for the mean functional connectivity maps across the entire network and a region of interest (ROI). Second, the ratio of overlap between Z-thresholded connectivity maps for each network was assessed. We quantified between-session functional reproducibility of rfMRI for 20 patients with stable MS and 14 healthy controls (HC). Nine rfMRI networks (RSNs) were examined at baseline and after 3 months of follow-up: three visual RSNs, the default-mode network, sensorimotor-, auditory-, executive control, and the left and right fronto-parietal RSN. ROI analyses were constrained to thresholded overlap masks for each individual (Z>0) at baseline and follow-up.In both stable MS and HC mean functional connectivity across the entire network did not reach acceptable ICCs for several networks (ICC<0.40) but we found a high reproducibility of ROI ICCs and of the ratio of overlap. ROI ICCs of all nine networks were between 0.98 and 0.99 for HC and ranged from 0.88 to 0.99 in patients with MS, respectively. The ratio of overlap for all networks was similar for both groups, ranging from 0.60 to 0.75.Our findings attest to a high reproducibility of rfMRI networks not only in HC but also in patients with stable MS when applying ROI analysis. This supports the utility of rfMRI to monitor functional changes related to disease progression or therapeutic interventions in MS.

Introduction

Multiple sclerosis (MS) is an inflammatory, neurodegenerative disease [13] and the major cause for non-traumatic disability in young adults [4]. Physical and cognitive deficits of MS have been related not only to structural damage but also to functional imbalance in and between brain networks [5]. Therefore, the study of functional changes of the brain by fMRI holds great promise to better understand the pathophysiologic mechanisms of the disease and their modification by therapeutic interventions [6]. Given recent increasing propositions to use resting-state fMRI (rfMRI) as a biomarker of disease progression and to monitor and/or predict motor and cognitive function in MS [79], we here explored the reproducibility of rfMRI over three months in patients with stable MS and compared findings to healthy controls.

RfMRI allows the investigation of changes within and across multiple functional networks without bias of task performance, adherence or subject effort and is increasingly used in patient cohorts [10,11]. Independent component analysis (ICA) has emerged as a powerful tool for exploring rfMRI data in both healthy and brain-diseased populations [12].

A high test-retest reproducibility of fMRI data is a pre-requisite for their application in clinical practice and clinical populations [1315]. Comparing group activation maps is not ideal for establishing reproducibility of fMRI signals [16], as the step of statistical thresholding of images can exaggerate very small differences between maps [17]. Hence, computing intraclass correlation coefficients (ICCs) is frequently recommended to assess fMRI reproducibility [18,19].

Recent studies tested reproducibility of fMRI activation in patients with MS using task-related fMRI for motor and cognitive functions [20,21]. However, advantages of rfMRI (e.g. no limitation due to a specific task, no ceiling and floor effects related to cognitive or behavioral constraints) lead to an increasing use of rfMRI to explore MS-related changes of brain integrity in cross sectional and longitudinal settings [10]. Although empirical evidence suggest high levels of converging findings, so far no study explicitly investigated reproducibility of longitudinal rfMRI in MS. Therefore, we quantified and compared between-session reproducibility of rfMRI derived network structure for patients with MS and healthy controls in terms of ICC values [18,19] and the ratio of overlap [22,23].

Materials and Methods

1.1. Participants

24 patients with MS and 15 age- and sex matched healthy controls (HC) underwent comprehensive 3T MRI (T1- and FLAIR-weighted imaging, rfMRI) at baseline (BL) and after three months of follow-up (FU). The study was approved by the ethics committee of the Medical University of Graz. All participants gave written informed consent.

Patients were selected from the MS outpatient clinic of the Department of Neurology, Medical University of Graz if they had a clinically and radiologically stable disease and were on continuous disease-modifying treatment (glatiramer-acetate, β-interferons or natalizumab). Only patients with a diagnosis of relapsing-remitting (RR) clinically-definite MS were considered [24]. Patients had to have no relapse within the previous two months, had not received corticosteroids eight weeks prior to inclusion, and had no history of serious psychiatric illness (e.g. depression) or other neurologic disorders. All patients underwent additional clinical and neuropsychological testing to control for disease activity.

MRI data of four patients and one healthy control finally had to be excluded (four due to scanner or movement artefacts, one patient did not attend the follow-up assessment), resulting in a final sample of 14 healthy controls and 20 patients (see Table 1 for characteristics).

thumbnail
Table 1. Demographics and morphological MRI characteristics of healthy controls (HC) and patients with multiple sclerosis (MS).

https://doi.org/10.1371/journal.pone.0152158.t001

1.2. Clinical and Neuropsychological Assessment

In patients with MS disability was measured using the Expanded Disability Status Scale (EDSS) [25] both at baseline and a 3-months follow-up. To capture possible changes in the cognitive subdomains known to be most sensitive to change in MS, we assessed the subtest for episodic memory of the Wechsler memory scale (WMS) [26], and two subtests of the Brief Repeatable Battery of Neuropsychological Tests (BRB-N) [27], namely the Symbol Digit Modalities Test (SDMT; information processing speed, sustained attention, and concentration) and Paced Auditory Serial Addition Test (3-second version; PASAT; sustained attention and concentration) at BL and FU, using parallel test forms if available.

1.3. MRI

MRI was performed on a 3 Tesla TimTrio scanner (Siemens Healthcare, Erlangen, Germany) using a 12-channel head coil. High-resolution structural 3D images were acquired by means of a T1-weighted MPRAGE sequence with 1 mm isotropic resolution (TR = 1900 ms, TE = 2.19 ms, 176 slices). A fluid-attenuated inversion recovery (FLAIR) sequence with 1x1x3 mm3 resolution served for the assessment of the hyperintense T2-lesion load in the patients (T2-LL; TR = 9000 ms; TE = 69 ms, 44 slices). RfMRI data were acquired with a 3x3x3.75 mm resolution (Sequence = 2D echo planar imaging (EPI), TR = 3000 ms; TE = 30; 150 volumes, field of view = 192 x 192 mm2, Matrix = 64 x 64, 36 slices, slice thickness = 3 mm, acquisition time = 7.5 minutes). Patients were asked to close their eyes during the rfMRI sequence. The total imaging time was approximately 20 minutes.

1.4. Image Analysis

1.4.1. Structural Analyses.

Normalized Brain Volume (NBV; in cm3) was estimated from the T1-weighted MPRAGE images using SIENAX (Structural Image Evaluation, using Normalisation, Single-Time-Point Estimation v 2.6), part of the FMRIB Software Library (FSL).

Percent brain volume change (PBVC) was assessed using SIENA (Structural Image Evaluation, using Normalisation; Two-Time-Point Estimation v 2.6). T2-lesion load (T2-LL) was assessed by a semi-automated region growing algorithm [28] subsequent to lesion identification by an experienced rater.

1.4.2. Functional Analyses—rfMRI.

In the first step, individual resting state data were preprocessed using FEAT (FMRIB's Expert Analysis Tool, v 6.0, part of FSL v 5.0.4 [29]. Individual pre-statistical processing included: motion correction using MCFLIRT, brain extraction, spatial smoothing using a Gaussian kernel of FWHM (full width at half maximum) of 6 mm [30], high pass temporal filtering using a cut-off of 150 s (0.007 Hz), linear registration to main structural image (BBR) and nonlinear registration warp resolution of 10 mm. High-resolution T1 scans served as targets for image registration. Next, Independent Component Analysis (ICA) was used for rfMRI data exploration (FSL-MELODIC, v 3.12), denoising the data and filtering out components based on high vs. low frequency content (fsl_regfilt command line tool). The resulting denoised functional images were resampled to standard space (MNI152 template 2 mm). Dual-regression analyses against ten resting-state templates from 36 healthy controls [31,32] on the denoised, registered functional images of each subject was performed in order to obtain individual spatial maps of the networks.

Group functional connectivity maps for BL and FU were computed for both groups and assessed for statistical significance (using FSL Randomise; see Fig 1). Furthermore, for each subject the longitudinal change of resting-state connectivity was individually computed and the difference maps were used to assess pre-post differences of rfMRI-connectivity networks at group level (using FSL Randomise).

thumbnail
Fig 1. Group functional connectivity maps.

Fig 1 illustrates the visual network (visual 00), the default-mode network (DMN) and sensorimotor network (motor) at baseline (BL) and follow-up (FU) for both groups. Tfce corrected p-values > 0.99. HC = Healthy controls, MS = patients with MS.

https://doi.org/10.1371/journal.pone.0152158.g001

1.4.3. Intraclass Correlation Coefficient (ICC).

The ICC assesses reproducibility by comparing the between-subject variance to total variance and it is therefore higher when within-subject variance is low and between-subject variance is high. The ICC for each network was assessed for the mean functional connectivity maps across the entire network of the group (EN) and a region of interest (ROI).

Regions of interest (ROI) were obtained by overlap masks for each individual (Z>0) at BL and FU and the ICC [19] were calculated as: where BMS is the between subject mean square error, EMS the error mean square and k the number of comparisons. This formula estimates the correlation of the subject signal intensities between sessions, modeled by a two-way ANOVA, with random subject effects and fixed session effects. An ICC value approaching 1 indicates high reproducibility, while a value close to 0 indicates very low reproducibility.

1.4.4. Ratio of overlapping connectivity (R12).

We assessed the ratio of overlapping functional connectivity by using the Dice coefficient [23]: Where V1 and V2 denote the number of suprathreshold voxels (Z>0) in the Z-volume at BL and FU, respectively, and V-overlap stands for the number of suprathreshold voxels in both Z-volumes. The R12 can range from 0 (no overlap) to 1 (perfect overlap) [13,33].

1.5. Statistical Analysis

Clinical, morphological and reproducibility scores were analyzed with the Statistical Package of Social Science (IBM SPSS Statistics 23). The level of significance was set at 0.05. As nine rfMRI networks were compared, for reproducibility scores a Bonferroni-adjusted level of significance of 0.0056 was applied. Baseline comparisons between groups and within-group comparisons were done using t-tests.

Results

2.1. Behavioral and Morphological findings

During the 3 months follow-up period patients with MS showed no clinical or morphological (MRI) evidence of disease activity (i.e. new lesions, enlarging lesions or new gadolinium-enhanced lesions), no change of T2-LL, EDSS and cognitive (memory, processing speed, attention) function (see Table 2). Slight improvements in memory performance were observed, most likely representing a learning effect, as no parallel version of the WMS was available. There were no measurable changes in global brain volume over three months for both groups (Table 1).

thumbnail
Table 2. Disease activity (T2-LL), motor and cognitive function at baseline (BL) and follow-up (FU) for patients with multiple sclerosis (MS).

https://doi.org/10.1371/journal.pone.0152158.t002

2.2. Identification and Comparison of resting-state networks at baseline and follow-up

Nine rfMRI networks were investigated in both groups: three visual networks (00, 01, 02), the default-mode network (DMN; 03), one sensorimotor network (05), one auditory network (06), one executive control network (07), and a left and right fronto-parietal network (08, 09) (individual networks in line with Smith et al., 2009; available at http://www.fmrib.ox.ac.uk/analysis/brainmap+rsns/). The cerebellar network (04) was not examined as the most inferior slices of the cerebellum had been cut off in some subjects to ensure full coverage of the cortex.

Pre-post comparison, including individual difference maps for each group revealed no significant changes in HC and MS. Fig 1 illustrates exemplarily three of the nine networks (the visual network (00), DMN and sensorimotor network) at BL and FU for both groups separately (all nine networks are illustrated in the supplement S1 Fig).

2.3. Reproducibility of resting-state networks assessed by ICC

Entire network (EN) ICC scores of the HC ranged from 0.73 (visual network 01) to 0.00 (auditory and left fronto-parietal network). For patients with MS, EN ICC’s were generally lower, ranging from 0.42 (auditory network) to 0.00 (visual 02 and sensorimotor network; Table 3).

thumbnail
Table 3. Intraclass correlation coefficients (ICCs) of separate networks for HC and patients with MS, for mean of the entire network connectivity (EN) and mean of the ROI connectivity.

ICC values of task-related fMRI for HC and patients with MS (hand tapping) as reported by Bosnell et al. (2008), are 0.91 and 0.76, respectively.

https://doi.org/10.1371/journal.pone.0152158.t003

ROI ICCs for the HC across all networks were between 0.97 and 0.99. The highest reproducibility was found for two visual networks and the executive control network. ROI ICCs for patients with MS were between 0.88 and 0.99 across all networks. The highest ROI reproducibility was found for the executive control network. The lowest ROI reproducibility for patients with MS was found in the DMN.

Compared to patients with MS, controls obtained higher EN and ROI ICC scores across all networks, which however were not significant after Bonferroni-adjustment (p = 0.0056).

2.4. Reproducibility of resting-state networks assessed by overlapping connectivity maps

Mean overlap of functional connectivity across all networks was between 0.61 and 0.75, for HC and between 0.60 and 0.75 in patients with MS. Highest overlap of Z-thresholded connectivity maps (Z>0) was found in the DMN (0.75), for HC and patients with MS. For HC, the lowest R12 reproducibility was observed in the executive control network. For patients with MS, the lowest R12 reproducibility was found in the visual 02 network.

Overlap of Z-thresholded functional connectivity maps (Z>0) did not differ significantly between groups across all nine networks (Table 4, Fig 2).

thumbnail
Table 4. Mean overlap (R12) of individual Z-thresholded connectivity maps (Z>0) for each network for HC and MS.

https://doi.org/10.1371/journal.pone.0152158.t004

thumbnail
Fig 2. Representative overlapping maps.

Representative connectivity maps for baseline (BL), follow-up (FU) and overlap of Z-thresholded (Z>0) connectivity maps for the DMN (X = 45, Y = 54, Z = 45) for a single healthy control and a single patient with MS.

https://doi.org/10.1371/journal.pone.0152158.g002

Discussion and Conclusions

This study showed high reproducibility of rfMRI over three months as assessed by ROI ICCs (0.97–0.99 and 0.88–0.99) and ratio of overlap (0.61–0.75 and 0.60–0.75) for HC and patients with stable MS, respectively. ICCs (of entire network and ROI) showed comparable reproducibility in HC and patients with MS and also the ratio of overlap did not differ between groups.

Mean connectivity across the entire network did not reach acceptable ICCs in both groups for several networks (e.g., visual 00, 02, DMN, executive control; ICC<0.40), highlighting the importance of choosing the proper ROI for extraction of signal change [17]. Various approaches to assess ICCs have been explored. Typically, a summary statistic for each subject is obtained for a ROI. This can be the mean or median contrast value within the region, or the value of the contrast at the peak of group activation [18]. The specific ROI frequently confines to an area most likely to be activated due to a task (e.g., sensorimotor cortex for a motor task, or insula for a cognitive task [20,21,34]). Given our data-driven analytic approach of rfMRI, using ICA with dual regression [12,35], we refrained from using anatomically defined ROIs, such as the precuneus (e.g., frequently used as a seed for the default mode network) and instead used the entire network maps at group level and individual overlap masks at BL and FU as shown in Fig 2. The latter proved to provide highly reproducible ICCs. The application of an entire network mask might thus be too over-inclusive to assess rfMRI reproducibility in terms of a single mean score. To assess reproducibility across whole-brain activation, a more sophisticated analytical approach, such as voxel-wise ICC analysis [18] would be more appropriate.

While several studies have investigated reproducibility of task-related fMRI [16,17,36,37], less is known about the reproducibility of rfMRI [13,38,39]. Kristo et al. (2014) compared the reproducibility of task-free fMRI and task motor activity in 16 healthy subjects with a test-retest interval of seven weeks. They found that although both approaches properly identified critical brain areas for motor task performance, rfMRI was less reliable compared to task-related fMRI. Nevertheless, Chou et al. (2012) reported the reproducibility of rfMRI measures to be outstanding and potentially suitable as biomarker for disease progression and treatment effects in clinical trials and individual patients [14]. Recent studies showed moderate to high test-retest reliability of rfMRI in healthy controls [38,40,41]. High reproducibility of longitudinal fMRI has also been reported across multiple sites for task-related and rfMRI [34,42]. However, these previous longitudinal assessments of variability in fMRI have been carried out predominantly using healthy controls [20].

Cross-sectional rfMRI studies in patients with MS suggest that prominent functional changes can be detected in many networks (e.g. default mode or sensorimotor network) and correlate with clinical and/or structural MRI measures [5,43]. Furthermore, the use of functional imaging techniques has highlighted that cortical reorganization might have a role in limiting the clinical consequences of tissue damage [5,44]. The application of rfMRI thus bears great potential to monitor disease progression or the impact of therapeutic intervention. It can be easily added to standard MRI and is not influenced by task-performance. RfMRI could complement structural conventional and quantitative MRI techniques, in order to better understand MS pathophysiology of, for instance, the correlates of cognitive dysfunctioning [45]. However, as Filippi et al. (2013) highlighted, rfMRI still requires a careful standardization of acquisition and analysis protocols, a careful assessment of scanner stability and intraindividual variability over time, and normative values as a reference [5].

In the presented study, we used a short (7.5 minutes) rfMRI-sequence, instructing the patients to close their eyes and explored intraindividual variability over time. We chose to assess two different measures of reproducibility (ICC and R12), that are frequently used and can be easily obtained. Both scores provide complementary information and for the given reasons might be useful as an outcome variable in clinical populations. Although the ICC might be less informative in clinical setting as clinicians most likely interpret fMRI results based on conventional P or Z-thresholding [13], some researchers underlined the poor test–retest reliability of suprathreshold voxel counts, given the fact that this approach strongly depends on the statistical threshold used [18,46]. In the present study, we used a threshold of Z>0. As the ratio of overlap is sensitive to the statistical threshold, altering this threshold to e.g. Z>1.5 would decrease the ratio of overlap (HC: 0.61–0.75 to 0.39–0.59; MS: 0.60–0.75 to 0.38–0.54), whereas the ICC is affected to a much lesser extent (HC: 0.97–0.99 to 0.80–0.99; MS: 0.88–0.99 to 0.80 to 0.96; data not shown). Hence, ICC and overlap ratios produce a different view of reproducibility [47], and both scores attested reasonable to high reproducibility of rfMRI for our sample of patients.

Reproducibility of fMRI activation in patients with MS has been assessed in task-related fMRI, showing high reproducibility of motor function and cognitive function in patients with MS [20,21]. In line with Bosnell et al. (2008), we also found greater variability for patients with MS compared to HC, regarding the ICCs (see also Table 3). For each individual, there are several sources of variability in fMRI (e.g. physiological, such as caffeine ingestion or fatigue; and psychological factors, such as attention and compliance) that can modulate brain response [20,38]. Patients with MS (in the context of widespread cerebral pathology) are certainly likely to bear even more sources of variability. Nevertheless, the fMRI signal in patients is still highly reliable and the ratio of overlap did not differ between HC and patients in our study. Loitfelder et al. (2014) used Bland-Altman plots to explore variability of the fMRI signal changes over time. The percent signal changes noted using this method were stable in HC and patients with MS, indicating a high level of repeatability [21]. Furthermore, patients did not provide more outliers than HC.

In our study, rfMRI networks were highly reproducible not only in HC but also in patients with MS, suggesting that the application of rfMRI to monitor functional changes related to disease progression in MS in clinical practice is reasonable and feasible. Further studies, also accounting for fatigue and anxiety levels of the patients might prove useful to assess the potential influence of behavioral variance on rfMRI.

Supporting Information

S1 Fig. Group functional connectivity maps of all networks.

S1 Fig illustrates all nine networks at baseline (BL) and follow-up (FU) for both groups. Tfce corrected p-values > 0.99. HC = Healthy controls, MS = patients with MS.

https://doi.org/10.1371/journal.pone.0152158.s001

(TIF)

Author Contributions

Conceived and designed the experiments: DP MK CE. Performed the experiments: DP MK EP. Analyzed the data: DP CB NF. Contributed reagents/materials/analysis tools: AP SF FF CE. Wrote the paper: DP CB MK EP NF AP SF FF CE.

References

  1. 1. Compston A, Coles A. Multiple sclerosis. Lancet. Elsevier Ltd; 2008;372: 1502–17.
  2. 2. Langdon DW. Cognition in multiple sclerosis. Curr Opin Neurol. 2011;24: 244–9. pmid:21519256
  3. 3. Chiaravalloti ND, DeLuca J. Cognitive impairment in multiple sclerosis. Lancet Neurol. Elsevier Ltd; 2008;7: 1139–51.
  4. 4. Pugliatti M, Rosati G, Carton H, Riise T, Drulovic J, Vécsei L, et al. The epidemiology of multiple sclerosis in Europe. Eur J Neurol. 2006;13: 700–722. pmid:16834700
  5. 5. Filippi M, Agosta F, Spinelli E, Rocca M. Imaging resting state brain function in multiple sclerosis. J Neurol. 2013;260: 1709–13. pmid:23052604
  6. 6. Filippi M, Rocca M a, Barkhof F, Brück W, Chen JT, Comi G, et al. Association between pathological and MRI findings in multiple sclerosis. Lancet Neurol. 2012;11: 349–360. pmid:22441196
  7. 7. Janssen AL, Boster A, Patterson B, Abduljalil A, Prakash RS. Resting-state functional connectivity in multiple sclerosis: An examination of group differences and individual differences. Neuropsychologia. Elsevier; 2013; 1–12.
  8. 8. Cruz-Gómez AJ, Ventura-Campos N, Belenguer A, Avila C, Forn C. The link between resting-state functional connectivity and cognition in MS patients. Mult Scler. 2013; 2013.
  9. 9. Faivre A, Rico A, Zaaraoui W, Crespy L, Reuter F, Wybrecht D, et al. Assessing brain connectivity at rest is clinically relevant in early multiple sclerosis. Mult Scler. 2012;18: 1251–1258. pmid:22307385
  10. 10. Enzinger C, Deluca J. Large-scale neuronal network dysfunction in multiple sclerosis?: Evidence from resting-state fMRI. Neurology. 2012; 1416–1417.
  11. 11. Cole DM, Smith SM, Beckmann CF. Advances and pitfalls in the analysis and interpretation of resting-state FMRI data. Front Syst Neurosci. 2010;4: 8. pmid:20407579
  12. 12. Zuo XN, Kelly C, Adelstein J, Klein D, Castellanos F, Milham M. Reliable Intrinsic Connectivity Networks: Test-Retest Evaluation Using ICA and Dual Regression Approach. Neuroimage. 2010;49: 2163–2177. pmid:19896537
  13. 13. Kristo G, Rutten GJ, Raemaekers M, de Gelder B, Rombouts S, Ramsey NF. Task and task-free FMRI reproducibility comparison for motor network identification. Hum Brain Mapp. 2014;35: 340–352. pmid:22987751
  14. 14. Chou YH, Panych LP, Dickey CC, Petrella JR, Chen NK. Investigation of long-term reproducibility of intrinsic connectivity network mapping: A resting-state fMRI study. Am J Neuroradiol. 2012;33: 833–838. pmid:22268094
  15. 15. Griffanti L, Rolinski M, Szewczyk-Krolikowski K, Menke RA, Filippini N, Zamboni G, et al. Challenges in the reproducibility of clinical studies with resting state fMRI: An example in early Parkinson’s disease. Neuroimage. Elsevier B.V.; 2016;124: 704–713.
  16. 16. McGonigle DJ, Howseman AM, Athwal BS, Friston KJ, Frackowiak RS, Holmes a P. Variability in fMRI: an examination of intersession differences. Neuroimage. 2000;11: 708–734. pmid:10860798
  17. 17. Aron AR, Gluck M, Poldrack R. Long-term test-retest reliability of functional MRI in a classification learning task. Neuroimage. 2006;29: 1000–6. pmid:16139527
  18. 18. Caceres A, Hall DL, Zelaya FO, Williams SCR, Mehta M. Measuring fMRI reliability with the intra-class correlation coefficient. Neuroimage. Elsevier Inc.; 2009;45: 758–768.
  19. 19. Shrout P, Fleiss J. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979;86: 420–428. pmid:18839484
  20. 20. Bosnell R, Wegner C, Kincses ZT, Korteweg T, Agosta F, Ciccarelli O, et al. Reproducibility of fMRI in the clinical setting: implications for trial designs. Neuroimage. 2008;42: 603–10. pmid:18579411
  21. 21. Loitfelder M, Fazekas F, Koschutnig K, Fuchs S, Petrovic K, Ropele S, et al. Brain activity changes in cognitive networks in relapsing-remitting multiple sclerosis—Insights from a longitudinal fMRI study. PLoS One. 2014;9.
  22. 22. Raemaekers M, Vink M, Zandbelt B, van Wezel RJ a, Kahn RS, Ramsey NF. Test-retest reliability of fMRI activation during prosaccades and antisaccades. Neuroimage. Elsevier Inc.; 2007;36: 532–542.
  23. 23. Dice LR. Measures of the Amount of Ecologic Association Between Species. Ecology. 1945;26: 297–302.
  24. 24. Polman CH, Reingold SC, Banwell B, Clanet M, Cohen J a, Filippi M, et al. Diagnostic criteria for multiple sclerosis: 2010 revisions to the McDonald criteria. Ann Neurol. 2011;69: 292–302. pmid:21387374
  25. 25. Kurtzke J. Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS). Neurology. 1983;33: 1444–1452. pmid:6685237
  26. 26. Härting C, Markowitsch H, Neufeld H, Calabrese P, Deisinger K, Kessler J. Wechsler Gedächtnistest—Revidierte Fassung. WMS-R. (Manual). Hans Huber.; 2000.
  27. 27. Rao S. A Manual for the Brief Repeatable Battery of Neuropsychological Tests in Multiple Sclerosis. Milwaukee: Medical College of Wisconsin.; 1990.
  28. 28. Plummer DL. DispImage: a display and analysis tool for medical images. Rev Neuroradiol. 1992;5: 489–495.
  29. 29. Jenkinson M, Beckmann CF, Behrens TEJ, Woolrich MW, Smith SM. Fsl. Neuroimage. 2012;62: 782–790. pmid:21979382
  30. 30. Poldrack RA, Mumford JA, Nichols TE. Handbook of functional MRI data analysis. Cambridge University Press; 2011.
  31. 31. Smith SM, Fox PT, Miller KL, Glahn DC, Fox PM, Mackay CE, et al. Correspondence of the brain’ s functional architecture during activation and rest. PNAS. 2009;106: 13040–13045. pmid:19620724
  32. 32. Filippini N, MacIntosh BJ, Hough MG, Goodwin GM, Frisoni GB, Smith SM, et al. Distinct patterns of brain activity in young carriers of the APOE-epsilon4 allele. Proc Natl Acad Sci U S A. 2009;106: 7209–14. pmid:19357304
  33. 33. Rombouts S, Barkhof F, Hoogenraad F, Sprenger M, Scheltens P. Within-subject reproducibility of visual activation patterns with functional magnetic resonance imaging using multislice echo planar imaging. Magn Reson Imaging. 1998;16: 105–113. pmid:9508267
  34. 34. Forsyth JK, McEwen SC, Gee DG, Bearden CE, Addington J, Goodyear B, et al. Reliability of functional magnetic resonance imaging activation during working memory in a multi-site study: Analysis from the North American Prodrome Longitudinal Study. Neuroimage. Elsevier Inc.; 2014;97: 41–52.
  35. 35. Beckmann CF. Modelling with independent components. Neuroimage. Elsevier Inc.; 2012;62: 891–901.
  36. 36. Eaton K, Szaflarski J, Altaye M, Ball A, Kissela B, Banks C, et al. Reliability of fMRI Studies of Language in Post-Stroke Aphasia Subjects. Neuroimage. 2008;41: 311–322. pmid:18411061
  37. 37. Clément F, Belleville S. Test-retest reliability of fMRI verbal episodic memory paradigms in healthy older adults and in persons with mild cognitive impairment. Hum Brain Mapp. 2009;30: 4033–4047. pmid:19492301
  38. 38. Zuo X, Xing X. Test-retest reliabilities of resting-state FMRI measurements in human brain functional connectomics: A systems neuroscience perspective. Neurosci Biobehav Rev. Elsevier Ltd; 2014;45: 100–118.
  39. 39. Damoiseaux JS, Rombouts S a RB, Barkhof F, Scheltens P, Stam CJ, Smith SM, et al. Consistent resting-state networks across healthy subjects. PNAS. 2006;103: 13848–53. pmid:16945915
  40. 40. Shehzad Z, Kelly a. MC, Reiss PT, Gee DG, Gotimer K, Uddin LQ, et al. The resting brain: Unconstrained yet reliable. Cereb Cortex. 2009;19: 2209–2229. pmid:19221144
  41. 41. Meindl T, Teipel S, Elmouden R, Mueller S, Koch W, Dietrich O, et al. Test-retest reproducibility of the default-mode network in healthy individuals. Hum Brain Mapp. 2010;31: 237–246. pmid:19621371
  42. 42. Jovicich J, Minati L, Marizzoni M, Marchitelli R, Sala-Llonch R, Bartrés-Faz D, et al. Longitudinal reproducibility of default-mode network connectivity in healthy elderly participants: A multicentric resting-state fMRI study. Neuroimage. Elsevier B.V.; 2015;
  43. 43. Sacco R, Bonavita S, Esposito F, Tedeschi G, Gallo A. The contribution of RS Networks to the study of cortical reorganization in MS. Mult Scler Int. 2013;
  44. 44. Enzinger C, Fazekas F. Measuring Gray Matter and White Matter Damage in MS: Why This is Not Enough. Front Neurol. 2015;6: 1–4.
  45. 45. Rocca M a, Amato MP, De Stefano N, Enzinger C, Geurts JJ, Penner I-K, et al. Clinical and imaging assessment of cognitive dysfunction in multiple sclerosis. Lancet Neurol. Elsevier Ltd; 2015;14: 302–317.
  46. 46. Marshall I, Simonotto E, Deary IJ, Maclullich A, Ebmeier KP, Rose EJ, et al. Repeatability of motor and working-memory tasks in healthy older volunteers: assessment at functional MR imaging. Radiology. 2004;233: 868–877. pmid:15498902
  47. 47. Lee J, Hsu E, Rashkin E, Thatcher J, Kreitschitz S, Gale P, et al. Reliability of fMRI motor tasks in structures of the corticostriatal circuitry: Implications for future studies and circuit function. Neuroimage. Elsevier Inc.; 2010;49: 1282–1288.