Peak oxygen uptake (VO2peak) via cardiopulmonary exercise testing is considered the gold standard for testing aerobic capacity in healthy participants and people with various medical conditions. The reliability and responsiveness of cardiopulmonary exercise testing outcomes in persons with MS (PwMS) have not been extensively studied.
(1) to investigate the reliability of cardiopulmonary exercise parameters in PwMS; (2) to determine the responsiveness, in terms of the smallest detectable change (SDC), for each parameter.
Two repeated measurements of cardiopulmonary exercise outcomes were obtained, with a median time interval of 16 days.
Thirty-two PwMS suffering from subjective fatigue performed cardiopulmonary exercise tests on a cycle ergometer, to voluntary exhaustion. We calculated the reliability, in terms of the intra-class correlation coefficient (ICC [2,k]; absolute agreement), and the measurement error, in terms of standard error of measurement (SEM) and SDC at individual (SDCindividual) and group level (SDCgroup).
The ICC for VO2peak was 0.951, with an SEM of 0.131 L∙min−1 and an SDCindividual of 0.364 L∙min−1. When corrected for bodyweight, the ICC of VO2peak was 0.933, with an SEM of 1.7 mL∙kg−1∙min−1 and in an SDCindividual of 4.6 mL∙kg−1∙min−1.
Generalization of our study results is restricted to fatigued PwMS with a low to mild level of disability.
Citation: Heine M, van den Akker LE, Verschuren O, Visser-Meily A, Kwakkel G, TREFAMS-ACE Study Group (2015) Reliability and Responsiveness of Cardiopulmonary Exercise Testing in Fatigued Persons with Multiple Sclerosis and Low to Mild Disability. PLoS ONE 10(3): e0122260. https://doi.org/10.1371/journal.pone.0122260
Academic Editor: Bart Rypma, University of Texas at Dallas, UNITED STATES
Received: July 11, 2014; Accepted: February 15, 2015; Published: March 19, 2015
Copyright: © 2015 Heine et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The TREFAMS-ACE study is funded by the Fonds NutsOhra (ZonMw 89000005). The funders had no role in study desgin, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Multiple sclerosis (MS) is a chronic demyelinating inflammatory disorder of the central nervous system (CNS) with unknown etiology. Progressive demyelination within the CNS results in a range of motor, sensory, cerebellar and cognitive dysfunctions, many of which may, directly or indirectly, limit physical fitness. The resulting impairments and limitations in daily activities may subsequently result in considerable physical deconditioning and even in comorbidities that reduce life expectancy.[2–4] Maintaining or improving physical fitness in persons with MS (PwMS) is paramount. In the past two decades, several intervention studies have investigated the feasibility and potential benefits of various types of exercise training in PwMS.[5–8] These studies have showed that training regimes like endurance, resistance or combined training can be used safely by PwMS. In addition, these training regimes may prevent deconditioning or improve physical fitness in these patients.[5–8] Measuring physical fitness is important in the context of clinical practice to evaluate people’s health status, and in the context of research to evaluate the effects of exercise regimes or other interventions. Cardiopulmonary exercise testing (CPET) using maximal cycling ergometry is considered the ‘gold standard’ for the assessment of exercise tolerance in both healthy people and people with various medical conditions. CPET is preferably accompanied by measurements of ventilation and gas exchange (i.e. VO2peak) for direct assessment of aerobic function. If a measure is to be used in clinical practice, it should be both content-valid and reliable. In this case, consecutive measurements from a given subject who has not changed, should be the same under several conditions (test-retest).
Recently, we showed that CPET is a valid measure of physical fitness in PwMS with a low to mild level of disability (Expanded Disability Status Scale; EDSS ≤ 4.0). In addition, we showed that the Oxygen Uptake Efficiency Slope (OUES), as a potential alternative to VO2peak in PwMS who are unable to attain maximal exercise intensities, is highly predictive of VO2peak. Even though the validity and the responsiveness of CPET in PwMS have not been studied extensively until recent[8, 12, 14], the CPET has been used as an outcome in clinical trials quite often already.[8, 15–20] For instance, in a study by Petajan et al., a 22% improvement in VO2peak was found following a 15 week aerobic training program. In contrast, Mostert and Kesseling were unable to replicate these results, in terms of VO2peak, in a shorter but more intensive training program. However, little is known about the extent to which maximal exercise performance in PwMS is hampered by day-to-day variation. In other words, how confident are we that the reported 22% change in VO2peak is not due to measurement error? To our knowledge, one study has suggested that CPET can be used reliably in PwMS under stringent laboratory conditions. This study, by Langeskov-Christensen and colleagues reported that a change of ~10% is required to consider this change significant. However, MS is an unstable medical condition with symptom experience changes even within days, yet little is known on the reliability and responsiveness of CPET in a less stringent laboratory setting which may, for instance, substantially affect our confidence in the 22% change in VO2peak as reported by Petajan et al. This question is important in the light of clinical research and practice, as the responsiveness of a test may provide valuable information on the required number of participants (i.e. statistical power) in clinical trials using CPET, on the ability of the test to determine individual changes in VO2peak in clinical practice, and to determine if the change in VO2peak following a clinical trial can be considered beyond measurement error. Our research question was therefore: what is the reliability and responsiveness of CPET, and in particular VO2peak, as assessed by two consecutive CPETs when applied to PwMS?
The objective of the present study was twofold: (1) to investigate the reliability of CPET, and in particular VO2peak, in PwMS; (2) to determine the responsiveness, in terms of the smallest detectable change.
Participants and Methods
A joint convenience sample was composed from patients treated at two study centers (VU Medical Center, Amsterdam, the Netherlands and St. Antonius Medical Center, Nieuwegein, the Netherlands). The sample consisted of 32 participants of the TREFAMS-ACE study (Treating Fatigue in Multiple Sclerosis: Aerobic Therapy, Cognitive Behavioral Therapy, Energy Conservation Management; ISRCTN69520623 and ISRCTN58583714) with definite MS. No a priori power analysis was conducted. The sample size of 32 was reached by requesting participation in this reliability study of all TREFAMS-ACE participants who attended the first or second follow-up measurement between august 2012 and October 2013. The TREFAMS-ACE study and the present reliability study were approved by the Medical Ethics Board of the VU University Medical Center Amsterdam. Each potential participant received both written and oral information about the TREFAMS-ACE trial before providing written informed consent. In addition, participants provided supplementary consent for participation in the present reliability study, which involved an additional exercise test. To be included in the TREFAMS-ACE study, participants had to experience substantial fatigue (≥ 35 on the fatigue subscale of the Checklist on Individual Strength), had to be ambulant (≤ 6.0 on the Expanded Disability Status Scale [EDSS]) and had to have no clinical depression (≤ 11 on the Depression subscale of the Hospital Anxiety and Depression Scale). Potential participants were excluded if they had participated in a professionally supervised therapy program to alleviate fatigue in the three months before inclusion. Potential participants were also excluded if they had comorbidities precluding maximal exercise participation, or MS relapse confirmed by a neurologist in the three months prior to study participation. Details of the TREFAMS-ACE study have been described elsewhere.
Prior to the cardiopulmonary exercise test, self-reported baseline characteristics and disease specifics were recorded: sex, age, height, weight, body mass index (BMI), time since diagnosis and type of MS (classified as relapsing-remitting, primary progressive, secondary progressive or not specified). Neurological disability was determined by a certified physician using the EDSS.
Cardiopulmonary exercise testing (CPET)
Two identical incremental exercise tests were performed on a programmable, electromagnetically braked cycle ergometer (Kettler X7, Heinz Kettler, Germany) to determine maximal exercise capacity. Within each study center, one assessor conducted the test and re-test. The two exercise tests were conducted during the follow-up phase of the TREFAMS-ACE study, but at least 10 weeks after the intervention phase. A period of one to three weeks was scheduled between the two consecutive tests, depending on the participants’ flexibility in making an appointment. Participants were asked not to change their exercise habits between the two tests. During the intervention phase of the TREFAMS-ACE study, participants had already performed at least one exercise test, and were thus familiar with the test protocol. To avoid bias, the two assessors in each center and the participants were not informed of the results of the first test prior to the second test.
Handlebars and saddle of the cycle ergometer were adjusted to match each participant’s anthropometrics. Following a 3-minute rest phase, participants started cycling at 25 Watt, with power output increasing by 10 Watt (women) or 15 Watt (men) each minute. This protocol was based on the rationale that small increments prolong exercise duration and enable participants with peripheral limitations to sustain longer exercise duration. Preferably, the exercise was terminated by voluntary exhaustion following 8–12 minutes of exercise. During the exercise testing, participants were asked to maintain a cadence of 60–80 rotations per minute (rpm). Participants were verbally encouraged, especially beyond a respiratory exchange ratio > 1.00. The exercise test was terminated by voluntary exhaustion, by a cadence < 45 rpm or for safety reasons, compliant with the American College of Sports Medicine’s guidelines for clinical exercise testing.
Each participant’s gas exchange was measured using a portable mixing-chamber monitoring system (Cortex MetaMax 3B; Cortex Medical; Germany). The Cortex Metamax is a valid and reliable system for measuring ventilation parameters during exercise.[27, 28] Volume, pressure, and gas analyzers were calibrated automatically by the system prior to each test using a 3L syringe, atmospheric pressure at the time (mmHg), and both ambient and reference gases (4.98% CO2, 17.05% O2), respectively. Raw data was averaged over 10-second intervals for analysis. Peak exercise values were defined as the highest recorded 10-second average. The following parameters derived from the CPET were used in the present study: peak oxygen consumption (VO2peak), maximal work output (Wmax), maximal ventilation (VEmax), maximal respiratory exchange ratio (RER), perceived exertion immediately following voluntary exhaustion, measured by the 20-point BORG scale, and maximal heart rate. In the current literature VO2peak is often, but not always, expressed relative to bodyweight. Hence, the present study used both the corrected and uncorrected VO2peak. In addition, we included the Oxygen Uptake Efficiency Slope (OUES). The OUES may be a valuable submaximal measure of physical fitness in those PwMS for whom VO2peak is a measure of performance, rather than aerobic capacity. The OUES was determined by measuring the slope of VO2 (mL∙kg−1∙min−1) and log10 VE (L∙min−1) until voluntary exhaustion. The OUES has been claimed to be independent of exercise duration, exercise protocol and assessor.[30, 31] It has also been shown that the OUES is highly predictive of VO2peak in PwMS.
From a psychometric point of view, reliability can be defined as the proportion of total variance in the measurements which indicates a ‘true’ difference between patients. In this concept, each observation is assumed to be composed of two components: a true score and an error associated with the association. Based on this definition, the proportion of between-patient variance compared to the total variance was calculated by the intra-class correlation coefficients (ICC [2, k]) model (i.e., a two-way random model with absolute agreement). An ICC greater than 0.80 reflects excellent reliability, whereas ICCs from 0.70 to 0.79 reflect good reliability. The recommended minimum for the lower limit of the 95% confidence interval (CI) is 0.85. Measurement error was defined as the systematic and random error in a patient’s score that is not attributed to true changes in CPET performance. The standard error of measurement (SEMagreement) was used for the measurement error, to determine the precision of the total score of both tests. The SEM describes the error in interpreting an individual’s test score. It allows the ‘true’ test performance to be estimated using a reliability coefficient and is computed by multiplying the standard deviation of the total score by the square root of 1 minus its reliability coefficient (SEM = SDpooled x √1—ICC). The smallest detectable change (SDCindividual) in each aerobic capacity outcome was computed as 1.96 x √2 x SEM to obtain a 95% CI.[32, 35] In addition we considered it valuable, especially for clinical research, to determine the smallest detectable difference at group level (SDCgroup) rather than at individual level. The SDCgroup was calculated as SDCindividual / √n. The Bland–Altman procedure was used to check for heteroscedasticity of the test and retest for each outcome measure. All analyses were conducted using the SPSS statistical package (version 19.0).
A total of 13 men and 19 women (Table 1) completed both tests without complications. The median disability level on the EDSS scale was 2.5 (Interquartile range (IQR) 2.0–3.0) Most participants had relapsing-remitting MS (N = 27). The second test was conducted after a median interval of 16 days (IQR 14–19 days) following the first.
Reliability and responsiveness
The reliability and responsiveness of the CPET outcomes are presented in Table 2. The reliability of both the %HRmax and the BORG score was based on 31 instead of 32 participants due to one missing value, as the heart rate monitor lost signal during the exercise test for one participant, and the BORG score was not acquired within one minute post voluntary exhaustion for another participant. The ICC of VO2peak was 0.951, with an SEM of 0.131 L∙min−1and an SDCindividual of 0.364 L∙min−1. After correcting for body weight, the ICC for VO2peak was 0.933, with an SEM of 1.7 mL∙kg−1∙min−1 and an SDCindividual of 4.6 mL∙kg−1∙min−1. The ICC for maximal power was 0.971, with an SEM of 9.1W and an SDC of 25.2W. As regards peak ventilation, the ICC was found to be 0.936, with an SEM of 6.4 L/min and an SDCindividual of 17.7 L/min. As can be seen from the Bland-Altman plots (Fig. 1) there were no signs of heteroscedasticity, except for the OUES. Potential outliers were included in the analysis.
BORG, rating of perceived exertion following voluntary exhaustion; HRmax, maximal heart rate expressed as percentage of the age-predicted maximal heart rate (%); OUES, oxygen uptake efficiency slope; RER, respiratory exchange ratio; VE, peak ventilation (L∙min−1); VO2peak, peak oxygen uptake (L∙min−1and mL∙kg−1∙min−1); Wmax, maximal power output (Watt).
The purpose of the present study was twofold: (1) to investigate the reliability of cardiopulmonary exercise parameters in PwMS; (2) to determine the responsiveness, in terms of the smallest detectable change, of each parameter. The aerobic capacity, in terms of VO2peak, was low (21.6 ± 6.4 mL∙kg−1∙min−1) when compared to reference values for sedentary healthy people (39.0 ± 6.8 mL∙kg−1∙min−1 for sedentary men; 30.0 ± 5.4 mL∙kg−1∙min−1 for sedentary women). The present study showed that the reliability of VO2peak, Wmax and VE could be considered excellent. However, we found that some individuals may need large changes in aerobic capacity (>21%) for them to be considered real changes beyond the 95%CI intervals of measurement error.
Reliability and responsiveness
The excellent reliability in two consecutive tests of VO2peak, as shown by the high ICCs, is in line with ICCs previously reported for CPET.[38, 39] Langeskov-Christensen and colleagues found a Pearson correlation coefficient of 0.98, which is of the same magnitude as the ICC found in the present study, for two consecutive CPETs in a sample of PwMS with similar disability (mean EDSS = 2.6). In contrast to the excellent test-retest reliability, the smallest detectable change for an individual PwMS was relatively large which questions the responsiveness of this measure. A change of at least 0.367 L∙min−1 or 4.6 mL∙kg−1∙min−1 (~21%) was required for it to be considered a statistically significant improvement or deterioration. This may indicate that the incremental exercise test at individual level may not be suitable for monitoring change in aerobic capacity in PwMS. However, the small SDCgroup found in the present study suggests that, at group level, small changes in the aerobic capacity of PwMS can be identified. The SDCindividual in the present study (~21%) was considerably larger than the required change found in the study by Langeskov-Christensen et al. (~10%). One possible explanation is that this may be related to the more stringent standardization in the study by Langeskov-Christensen et al. (in terms of food intake, time between tests and exercise behavior). A second possible explanation is that the present study used two different exercise labs, which may have introduced some additional variance. Thirdly, the present study recruited PwMS who experienced excessive fatigue. Since there was no restriction on various factors that may have influenced fatigue, perceived fatigue may have been different on each test occasion, which may explain some of the variance between the tests. However, on the contrary, the participants of the present study were familiar with the testing protocol due to the fact that this reliability study was done during the follow-up phase of an intervention study which also included CPET. Hence, this should have reduced the likelihood of a familiarization effect during the second test and reduced the between-test variance in comparison to for example Langeskov-Christensen et al.
Recently, it was shown that VO2peak is a valid outcome for physical fitness in fatigued PwMS with a low to mild level of disability. In the present study, the mean RER and %HRmax confirm that, in PwMS and low to mild disability the CPET can be considered a valid measure of aerobic capacity.[12, 14] However, the outcome of CPET in PwMS with a moderate level of disability may be more closely related to function than to physical fitness. We have suggested the oxygen uptake efficiency slope as an alternative measure of physical fitness, which can also be used in case of submaximal exercise duration. An ICC of 0.93 has been previously reported for the OUES in healthy people. However, the present study suggests a relatively high day-to-day variation in oxygen uptake efficiency in PwMS. A change of ~27% is needed, based on the results of the present study, for it to be considered a significant change. Alternatively, the OUES may complement the assessment of VO2peak, as it may capture some of the physiological mechanisms (e.g. respiratory muscle work) that can explain the variation in VO2peak between two tests.
Some limitations to this study need to be addressed, which may affect the interpretation of our results and can provide clues for ways to reduce measurement error and improve reliability. First, two different exercise labs were involved, with a different assessor, use of equipment and recruitment area. Research on the interrater reliability and interlab reliability of CPET in PwMS will provide additional insights into modifiable factors to reduce measurement error. Second, some of the day-to-day variation between the tests may be related to uncontrolled sources of variance, like food and beverage intake, but also to daily differences in subjective symptoms like fatigue. Third, participants of the present study had a low to mild (EDSS < 4.0) level of disability, and experienced excessive fatigue, which restricts the generalization of our study results.
At individual level, CPET can be used reliably to assess physical fitness status, but less so to determine significant changes, in terms of aerobic capacity unless the test circumstances are rigorously (e.g. timing, temperature, food and beverage intake etc.) controlled between tests. This may also include subjective symptoms like fatigue. At group level, CPET can be reliably used to determine physical fitness status and establish change. Further research is warranted on the cardiovascular working mechanisms of exercise therapy in MS, the trainability of PwMS, and clinical relevance of changes in aerobic capacity in PwMS.
S1 Dataset. Data underlying the presented results.
The authors would like to thank the staff of the St. Antonius Hospital Multiple Sclerosis Center, Nieuwegein and VU University Medical Center, Amsterdam for their efforts to find eligible participants for the present study. This study was performed on behalf of the Treating Fatigue in Multiple Sclerosis: Aerobic Training, Cognitive Behavioural Therapy, Energy Conservation management (TREFAMS-ACE) Study Group (S1 Appendix).
Conceived and designed the experiments: MH GK OV AV. Performed the experiments: MH LA. Analyzed the data: MH LA OV AV GK. Wrote the paper: MH LA OV AV GK.
- 1. Compston A, Coles A. Multiple sclerosis. Lancet. 2008;372:1502–1517. pmid:18970977
- 2. Marrie RA, Hanwell H. General health issues in multiple sclerosis: comorbidities, secondary conditions, and health behaviors. Continuum (Minneap Minn). 2013;19:1046–1057. pmid:23917100
- 3. Marrie RA, Horwitz RI. Emerging effects of comorbidities on multiple sclerosis. Lancet Neurol. 2010;9:820–828. pmid:20650403
- 4. Wens I, Dalgas U, Stenager E, Eijnde BO. Risk factors related to cardiovascular diseases and the metabolic syndrome in multiple sclerosis—a systematic review. Mult Scler. 2013;19:1556–1564. pmid:24048545
- 5. Latimer-Cheung AE, Pilutti LA, Hicks AL, Martin Ginis KA, Fenuta AM, MacKibbon KA, et al. Effects of exercise training on fitness, mobility, fatigue, and health-related quality of life among adults with multiple sclerosis: a systematic review to inform guideline development. Arch Phys Med Rehabil. 2013;94:1800–1828. pmid:23669008
- 6. Rietberg MB, Brooks D, Uitdehaag BM, Kwakkel G. Exercise therapy for multiple sclerosis. Cochrane Database Syst Rev. 2005:CD003980.
- 7. Motl RW, Pilutti LA. The benefits of exercise training in multiple sclerosis. Nat Rev Neurol. 2012;8:487–497. pmid:22825702
- 8. Langeskov-Christensen M, Heine M, Kwakkel G, Dalgas U. Aerobic Capacity in Persons with Multiple Sclerosis: A Systematic Review and Meta-Analysis. Sports Med. 2015:In press.
- 9. Stickland MK, Butcher SJ, Marciniuk DD, Bhutani M. Assessing exercise limitation using cardiopulmonary exercise testing. Pulm Med. 2012;2012:824091. pmid:23213518
- 10. Motl RW, Fernhall B. Accurate prediction of cardiorespiratory fitness using cycle ergometry in minimally disabled persons with relapsing-remitting multiple sclerosis. Arch Phys Med Rehabil. 2012;93:490–495. pmid:22225573
- 11. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63:737–745. pmid:20494804
- 12. Heine M, Hoogervorst EL, Hacking HG, Verschuren O, Kwakkel G. Validity of Maximal Exercise Testing in Patients With Low to Moderate Multiple Sclerosis. Phys Ther. 2014;94:1168–1175. pmid:24677255
- 13. Heine M, Verschuren O, Kwakkel G. Validity of Oxygen Uptake Efficiency Slope in Patients with Multiple Sclerosis. J Rehabil Med. 2014;46:656–661. pmid:24850225
- 14. Langeskov-Christensen M, Langeskov-Christensen D, Overgaard K, Møller AB, Dalgas U. Validity and Reliability of VO2−max Measurements in Persons with Multiple Sclerosis. J Neurol Sci. 2014;342:79–87. pmid:24825731
- 15. Golzari Z, Shabkhiz F, Soudi S, Kordi MR, Hashemi SM. Combined exercise training reduces IFN-gamma and IL-17 levels in the plasma and the supernatant of peripheral blood mononuclear cells in women with multiple sclerosis. International immunopharmacology. 2010;10:1415–1419. pmid:20797460
- 16. Rampello A, Franceschini M, Piepoli M, Antenucci R, Lenti G, Olivieri D, et al. Effect of aerobic training on walking capacity and maximal exercise tolerance in patients with multiple sclerosis: a randomized crossover controlled study. Phys Ther. 2007;87:545–555. pmid:17405806
- 17. Rasova K, Havrdova E, Brandejsky P, Zalisova M, Foubikova B, Martinkova P. Comparison of the influence of different rehabilitation programmes on clinical, spirometric and spiroergometric parameters in patients with multiple sclerosis. Mult Scler. 2006;12:227–234. pmid:16629428
- 18. Schulz KH, Gold SM, Witte J, Bartsch K, Lang UE, Hellweg R, et al. Impact of aerobic training on immune-endocrine parameters, neurotrophic factors, quality of life and coordinative function in multiple sclerosis. Journal of the neurological sciences. 2004;225:11–18. pmid:15465080
- 19. Mostert S, Kesselring J. Effects of a short-term exercise training program on aerobic fitness, fatigue, health perception and activity level of subjects with multiple sclerosis. Mult Scler. 2002;8:161–168. pmid:11990874
- 20. Petajan JH, Gappmaier E, White AT, Spencer MK, Mino L, Hicks RW. Impact of aerobic training on fitness and quality of life in multiple sclerosis. Annals of neurology. 1996;39:432–441. pmid:8619521
- 21. Beckerman H, Blikman LJ, Heine M, Malekzadeh A, Teunissen CE, Bussmann JB, et al. The effectiveness of aerobic training, cognitive behavioural therapy, and energy conservation management in treating MS-related fatigue: the design of the TREFAMS-ACE programme. Trials. 2013;14:250. pmid:23938046
- 22. Vercoulen JH, Hommes OR, Swanink CM, Jongen PJ, Fennis JF, Galama JM, et al. The measurement of fatigue in patients with multiple sclerosis. A multidimensional comparison with patients with chronic fatigue syndrome and healthy subjects. Arch Neurol. 1996;53:642–649. pmid:8929171
- 23. Kurtzke JF. Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS). Neurology. 1983;33:1444–1452. pmid:6685237
- 24. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67:361–370. pmid:6880820
- 25. Buchfuhrer MJ, Hansen JE, Robinson TE, Sue DY, Wasserman K, Whipp BJ. Optimizing the exercise protocol for cardiopulmonary assessment. J Appl Physiol Respir Environ Exerc Physiol. 1983;55:1558–1564. pmid:6643191
- 26. Medicine ACoS. Clinical Exercise Testing. In: Thompson WR, Gordon NF, Pescatello IS, editors. ACSM's Guidelines for Exercise Testing and Prescription. Philadelphia: Lippincott Williams & Wilkins; 2000. pp. 105–134.
- 27. Medbo JI, Mamen A, Welde B, von Heimburg E, Stokke R. Examination of the Metamax I and II oxygen analysers during exercise studies in the laboratory. Scand J Clin Lab Invest. 2002;62:585–598. pmid:12564617
- 28. Brehm MA, Harlaar J, Groepenhof H. Validation of the portable VmaxST system for oxygen-uptake measurement. Gait Posture. 2004;20:67–73. pmid:15196523
- 29. Borg G. [Physical training. 3. Perceived exertion in physical work]. Lakartidningen. 1970;67:4548–4557. pmid:5477775
- 30. Baba R, Nagashima M, Goto M, Nagano Y, Yokota M, Tauchi N, et al. Oxygen uptake efficiency slope: a new index of cardiorespiratory functional reserve derived from the relation between oxygen uptake and minute ventilation during incremental exercise. J Am Coll Cardiol. 1996;28:1567–1572. pmid:8917273
- 31. Akkerman M, van Brussel M, Hulzebos E, Vanhees L, Helders PJ, Takken T. The oxygen uptake efficiency slope: what do we know? J Cardiopulm Rehabil Prev. 2010;30:357–373. pmid:20724931
- 32. De Vet HC, Terwee CB, Knol DL, Bouter LM. When to use agreement versus reliability measures. J Clin Epidemiol. 2006;59:1033–1039. pmid:16980142
- 33. Portney L, Watkins M. Foundations of Clinical Research: Applications to Practice. East Norwalk, CT: Appleton & Lange; 2000.
- 34. Streiner D, Norman G. Health Measurement Scales. A Practical Guide to Their Development and Use. 2nd ed. Oxford, United Kingdom: Oxford University Press; 2000.
- 35. Beckerman H, Roebroeck ME, Lankhorst GJ, Becher JG, Bezemer PD, Verbeek AL. Smallest real difference, a link between reproducibility and responsiveness. Qual Life Res. 2001;10:571–578. pmid:11822790
- 36. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1:307–310. pmid:2868172
- 37. Herdy AH, Uhlendorf D. Reference values for cardiopulmonary exercise testing for sedentary and active men and women. Arq Bras Cardiol. 2011;96:54–59. pmid:21109909
- 38. Meyer T, Georg T, Becker C, Kindermann W. Reliability of gas exchange measurements from two different spiroergometry systems. Int J Sports Med. 2001;22:593–597. pmid:11719895
- 39. Barron A, Dhutia N, Mayet J, Hughes AD, Francis DP, Wensel R. Test-retest repeatability of cardiopulmonary exercise test variables in patients with cardiac or respiratory disease. Eur J Prev Cardiol. 2014;4:445–453. pmid:24398370
- 40. Van Laethem C, De Sutter J, Peersman W, Calders P. Intratest reliability and test-retest reproducibility of the oxygen uptake efficiency slope in healthy participants. Eur J Cardiovasc Prev Rehabil. 2009;16:493–498. pmid:19483619