Cochrane reviewers are strongly encouraged to evaluate the quality of evidence for the most important outcomes by using the GRADE approach and to report these results in a Summary of Findings (SoF) table. We aimed to assess whether outcomes reported in the SoF table of Cochrane reviews could be considered patient-important outcomes (PIOs) and the quality of the available evidence for these outcomes.
We performed a methodological review of Cochrane reviews published between March 2011 and September 2014. For a random sample of Cochrane reviews reporting a SoF table, we extracted all outcomes reported in this table and evaluated whether they could be considered PIOs (i.e., mortality, other clinical events, adverse events, function, pain, quality of life and therapeutic decisions). Then, we collected the quality of evidence for every outcome in these SoF tables.
We included 290 reviews issued by 47 of the 53 Cochrane Review Groups. Every SoF table included a median of 5 outcomes, for a total of 1414 outcomes; 1089 (77%) could be considered PIOs. Almost all reviews (n = 278, 96%) included at least one PIO in their SoF table. The quality of evidence for the outcomes was high for 12% (n = 168), moderate for 28% (n = 402) and low or very low for 45% (n = 640). Less than one quarter of reviews (n = 63) included at least one PIO with high-quality evidence that favoured a benefit of the experimental intervention evaluated in half of them (n = 34 reviews).
Citation: Yordanov Y, Dechartres A, Ravaud P (2018) Patient-important outcomes in systematic reviews: Poor quality of evidence. PLoS ONE 13(4): e0195460. https://doi.org/10.1371/journal.pone.0195460
Editor: Gianni Virgili, Universita degli Studi di Firenze, ITALY
Received: November 14, 2017; Accepted: March 22, 2018; Published: April 5, 2018
Copyright: © 2018 Yordanov et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data have been uploaded as Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
In the last decade, clinicians and researchers have been encouraged to recommend healthcare interventions based on their effect on patient-important outcomes (PIOs) such as death, other clinical events, quality of life or functional outcomes[1–6]. However, several methodological reviews reported that the use of PIOs in randomized controlled trials (RCTs) of various medical specialities is still far from optimal [7–10]. In a recent report assessing trials of critically ill patients, less than one quarter of primary outcomes (24%) were PIOs. This proportion was as low as 5% when exploring outcomes besides mortality after intensive care unit discharge, such as functional disability or quality of life. Similar results were previously found for diabetes or cardiovascular  trials, with 18% and 23% of these trials assessing PIOs. In contrast, some recent studies suggested that systematic reviews are more likely to evaluate PIOs than their individual trials[10, 11].
A key component of systematic reviews is the elaboration of a clear and focused clinical question by specifying the types of participants, interventions, comparisons, and outcomes that should be considered[12, 13]. When planning a Cochrane review, authors are strongly encouraged to consider all outcomes that are meaningful to patients, physicians, policy makers or any other health care stakeholders, regardless of their availability in individual trials [13, 14]. When reporting the review, they are asked to present a Summary of Findings (SoF) table summarizing the quality of evidence and treatment effect magnitude for the most essential outcomes for decision-making by using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach [13–18].
In this study, we aimed to evaluate: 1) whether outcomes reported in the SoF table could be considered PIOs and 2) the quality of evidence for these outcomes in a sample of recent Cochrane reviews.
This is a methodological review of recently published Cochrane reviews. For each Cochrane review reporting an SoF table, we extracted all outcomes reported in this table and evaluated whether they could be considered PIOs. Then, we evaluated the quality of evidence for these outcomes.
We previously obtained data from all systematic reviews published between March 2011 and September 2014 from the Cochrane Collaboration . Data were provided as XML files and contained all information reported by the review authors using RevMan, the review software developed by the Cochrane Collaboration to prepare and update systematic reviews.
Using R 3.2.2 and the XML package, we identified all reviews of RCTs including at least one SoF table. Reviews including observational studies were excluded. We also excluded reviews including only trials published before 2007 to focus on recent topics. Then, we manually examined all review titles and abstracts when necessary to select those evaluating a healthcare intervention (pharmacological or non-pharmacological). Reviews evaluating diagnostic test accuracy or economic evaluations were excluded. Among the set of eligible reviews, by using a random number generator, we drew a random sample of 300 reviews for comprehensive evaluation.
Identification of the SoF for the main comparison
Most reviews have a single SoF table, but some have several, corresponding to different comparisons. In this case, we manually identified the main comparison as reported by the review authors. If no main comparison was identified as such, we selected the one with the largest number of available outcomes and included the largest number of trials. When the same SoF table reported various comparisons, the review was excluded.
For every included systematic review, we collected the following characteristics by using a standardized data extraction sheet:
- Review general characteristics: title, digital object identifier (DOI), protocol publication date, review first publication date, Cochrane Review Group (e.g., Gynaecological Cancer Group), type of intervention (i.e., pharmacological or non-pharmacological), number of excluded and included trials.
- Characteristics of the main comparison: Patient or population description, setting, intervention and comparator
- Every outcome reported in the SoF table including outcome description and follow-up period.
Classification of outcomes
We categorized each outcome as follows: mortality, other clinical events (e.g., myocardial infarction or stroke), therapeutic decision (e.g., transfusion), function (e.g., anxiety, depression, disability and dyspnoea), pain, quality of life, adverse events or side effects (identified as such by the review authors), physiological parameters (e.g., blood pressure, weight), biological parameters (e.g., cholesterol levels), radiological parameters (e.g., measure of joint space), compliance (e.g., discontinuation for any reason), process (e.g., duration of surgical procedure), resource use (hospitalisations), cost-effectiveness and satisfaction with care. A single reviewer classified all outcomes. As a quality measure, 10% of the outcomes were classified independently in duplicate (YY and AD).
Definition of PIOs
Quality of evidence
We evaluated the quality of evidence of each outcome reported in the SoF table based on the review authors assessment. The evidence was classified as high quality if further research is very unlikely to change the authors confidence in the estimate of effect; moderate quality if further research is likely to have an important impact on the authors’ confidence in the estimate of effect and may change the estimate; low quality if further research is very likely to have an important impact on the confidence in the estimate of effect and likely to change the estimate; and very low quality if the estimate of the effect is very uncertain. We evaluated separately the quality of evidence for outcomes that could be considered PIOs. For those considered with high quality of evidence, we also evaluated whether this was evidence of benefit for the experimental intervention (results in favour of the experimental intervention: i.e., statistically significant results).
The analysis was mostly descriptive. Continuous data are presented as median (Q1–Q3) and qualitative data as frequency (percentage). All analyses involved use of R v3.1.1 (R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/).
Selection of relevant Cochrane systematic reviews
The selection process was previously described. Briefly, between March 2011 and September 2014, 2796 Cochrane systematic reviews were published, 1670 did not include an SoF table (60% of screened reviews) and 820 were eligible for inclusion. We identified a random sample of 300 reviews, but 10 were additionally excluded because their SoF presented different interventions within the same table, which left 290 reviews for further evaluation.
Characteristics of included reviews
The reviews were issued by 47 of the 53 Cochrane Review Groups. Every review included a median of 11 trials overall (Q1-Q3: 5–21), with a median of 5 trials per main comparison (Q1-Q3: 3–12). The experimental intervention was non-pharmacological in 40% of the reviews (n = 115). Every SoF table included a median of 5 outcomes (Q1-Q3: 3–7), for a total of 1414 outcomes. The corresponding meta-analyses included a median of 3 trials (Q1-Q3: 2–7).
Among the 1414 outcomes reported in the SoFs, the most common were functional outcomes (27%, n = 384), clinical events (14%, n = 198) and adverse events (12%, n = 174). Mortality represented 10% (n = 138) of the outcomes and quality of life 7% (n = 98). Among the outcomes, biological parameters represented 6% (n = 89), process and resource use 5% (n = 74), and physiological parameters and compliance 4% (n = 56) and 3% (n = 45). A total of 1089 outcomes (77%, 95% CI: 75–79) could be considered PIOs (Table 1). Almost all reviews included at least one PIO in the SoF table (n = 278, 96%; 95% CI: 93–98) (Fig 1). Per review, the median proportion of PIOs among the outcomes reported in the SoF was 86% (Q1-Q3: 67–100).
Quality of available evidence
The quality of evidence provided by the outcomes in the SoF tables was high for 12% (n = 168, 95% CI: 10–14) and moderate for 28% (n = 402, 95% CI: 26–31) (Table 2). For 45% of the outcomes (n = 640, 95% CI: 43–48), the quality of the evidence was low or very low. No GRADE assessment was available for 12% of the outcomes (n = 164, 95% CI: 10–13) because according to the review authors, these outcomes were not reported or measured in any individual trials. The quality of the available evidence was very close when focusing on only PIOs, with 41% (n = 449) having high and moderate evidence and 44% (n = 475) low to very low evidence.
Less than one-quarter of the reviews, 22% (n = 63; 95% CI: 17–27), included at least one PIO with evidence considered of high quality by the review authors. In half (n = 34) of these reviews, the results were in favour of the experimental intervention (i.e., statistically significant results) (Fig 1).
In this study, we evaluated whether outcomes reported in the SoF table of recent Cochrane reviews could be considered PIOs and the quality of evidence provided for these outcomes. The large variety of review groups represented allowed us to explore very different medical specialties. More than three-quarters of the outcomes reported in the SoF tables could be considered PIOs. However, for a large proportion of the available evidence (45%), the quality was considered low or very low. About one quarter of the reviews included at least one PIO with high-quality evidence and for half of them, this evidence showed a benefit of the experimental intervention.
The Cochrane Collaboration strongly encourages review authors to include SoF tables to present their main findings . These tables, first introduced in Cochrane reviews in 2008, aim to synthesize in a simple, transparent and accessible format, key information on the assessed interventions’ magnitude of effect, sum of available data for the main outcomes and quality of evidence . Therefore, SoF tables should include the most important outcomes whether they were measured in individual trials or not. Previous studies reported that as compared with reviews without SoF tables, the inclusion of SoF tables seemed to improve readers’ general understanding of the reviews and allowed them to better identify the critical information and find results for important outcomes (93% vs 44%). However, not all Cochrane reviews present a SoF table. In our study, we excluded 60% of Cochrane reviews because they had no SoF table, which is consistent with a previous study showing a proportion of inclusion of SoF tables evolving from 31% in 2008 to 57% in 2013 . An alternative version of the SoF table called the interactive SoF (iSoF) allowing review authors to choose alternative displays (absolute effects rather than relative effects) is being tested, which may help generalise their use.
We found that most of the outcomes reported in the SoF tables could be considered PIOs, with a median proportion of PIOs in the SoF tables of 86%. In a study that focused on recently published Cochrane and non-Cochrane reviews but also on registered systematic reviews protocols (PROSPERO), Ameur et al described that 95% of Cochrane reviews included at least one PIO among their primary outcomes. This proportion was higher than for non-Cochrane reviews. Two previous studies also found consistent results, with 68% to 71% of the reviews considering PIOs as primary outcomes[21, 22]. A large survey of the completeness of main outcomes mapped the availability of information for a major clinical outcome across randomized trials and systematic reviews in an entire discipline: 75% of reviews reported the major clinical outcome as compared with only 20% of primary trial reports . All these results highlight the gap in outcomes between systematic reviews and their individual trials, which may explain why the quality of evidence was frequently limited for the outcomes reported in the SoF table, as shown by our results. In our study, only 22% of the reviews had at least one PIO with a high level of evidence, with half showing a benefit of the experimental intervention. The failure to consider PIOs in clinical trials may lead to erroneous evaluation of benefits, with possible serious consequences for patients, and represents waste[28–31]. To improve this situation, core outcome sets (COSs) should be developed (ie, standardized sets of outcomes, that have been agreed upon, and that should be measured and reported in every trial for a specific healthcare condition) and their use promoted [32–34]. The number of COSs developed is still limited but is progressively increasing, such as their use in systematic reviews [11, 25, 33, 35–37].
Our study has limitations. Although we relied on detailed and previously published definitions for PIOs, considerable judgment remains when assessing the importance of the outcomes, and patients or experts in the assessed medical field were not involved in this evaluation. Our results cannot be generalized to all systematic reviews because Cochrane reviews have precise conduct and reporting guidelines[13–15] and were described as more transparently reported than all other types of systematic reviews. Our study confirms that many outcomes reported in the SoF tables of recent Cochrane reviews can be considered PIOs. However, the quality of evidence for these outcomes remains limited.
Our study confirms that many outcomes reported in the SoF tables of recent Cochrane reviews can be considered PIOs. However, the quality of evidence for these outcomes remains limited.
S1 Table. Quality of available evidence for outcomes reported in the SoF tables for the main comparison in 290 recent Cochrane reviews: Comparison of all outcomes, to PIOs and surrogate outcomes.
S1 File. Supporting data 1: Complete list of outcomes, classified and categorized.
- 1. Guyatt G, Montori V, Devereaux PJ, Schunemann H, Bhandari M. Patients at the center: in our practice, and in our use of language. ACP J Club. 2004;140(1):A11–2. pmid:14711297.
- 2. Richards T, Montori VM, Godlee F, Lapsley P, Paul D. Let the patient revolution begin. BMJ. 2013;346:f2614. Epub 2013/05/16. pmid:23674136.
- 3. Montori VM, Gandhi GY, Guyatt GH. Patient-important outcomes in diabetes—time for consensus. Lancet. 2007;370(9593):1104–6. pmid:17905147.
- 4. Frank L, Basch E, Selby JV, Patient-Centered Outcomes Research I. The PCORI perspective on patient-centered outcomes research. Jama. 2014;312(15):1513–4. pmid:25167382.
- 5. Methodology Committee of the Patient-Centered Outcomes Research I. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. Jama. 2012;307(15):1636–40. pmid:22511692.
- 6. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. Jama. 2012;307(15):1583–4. pmid:22511682.
- 7. Rahimi K, Malhotra A, Banning AP, Jenkinson C. Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review. BMJ. 2010;341:c5707. pmid:21041324.
- 8. Gandhi GY, Murad MH, Fujiyoshi A, Mullan RJ, Flynn DN, Elamin MB, et al. Patient-important outcomes in registered diabetes trials. Jama. 2008;299(21):2543–9. pmid:18523223.
- 9. Gaudry S, Messika J, Ricard JD, Guillo S, Pasquet B, Dubief E, et al. Patient-important outcomes in randomized controlled trials in critically ill patients: a systematic review. Ann Intensive Care. 2017;7(1):28. pmid:28271450.
- 10. Saldanha IJ, Li T, Yang C, Owczarzak J, Williamson PR, Dickersin K. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J Clin Epidemiol. 2017;84:85–94. pmid:28249722.
- 11. Ameur H, Ravaud P, Fayard F, Riveros C, Dechartres A. Systematic reviews of therapeutic interventions frequently consider patient-important outcomes. J Clin Epidemiol. 2017. pmid:28147235.
- 12. Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21. pmid:12612111.
- 13. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. www.cochrane-handbook.org.
- 14. O’Connor D, Green S, Higgins JPT. Chapter 5: Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated September 2011]. ed. www.cochrane-handbook.org.: The Cochrane Collaboration; 2011.
- 15. Schünemann HJ, Oxman AD, Higgins JPT, Vist GE, Glasziou P, Akl E, et al. Chapter 11: Completing ‘Summary of findings’ tables and grading the confidence in or quality of the evidence. In: Higgins JPT, Chandler J, (editors) CM , editors. Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). ed. Cochrane, 2017. http://training.cochrane.org/handbook: The Cochrane Collaboration; 2017.
- 16. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is "quality of evidence" and why is it important to clinicians? Bmj. 2008;336(7651):995–8. Epub 2008/05/06. pmid:18456631.
- 17. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Bmj. 2004;328(7454):1490. Epub 2004/06/19. pmid:15205295.
- 18. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6. pmid:18436948.
- 19. Yordanov Y, Dechartres A, Atal I, Tran V, Boutron I, Crequit P, et al. Avoidable waste of research related to outcome planning and reporting in clinical trials. BMC Medicine—under review. 2017.
- 20. Dechartres A, Ravaud P, Atal I, Riveros C, Boutron I. Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC medicine. 2016;14(1):100. pmid:27377062.
- 21. Alonso-Coello P, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, Akl EA, Vernooij RW, et al. Systematic reviews experience major limitations in reporting absolute effects. J Clin Epidemiol. 2016;72:16–26. pmid:26560992.
- 22. Agarwal A, Johnston BC, Vernooij RW, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, et al. Authors seldom report the most patient-important outcomes and absolute effect measures in systematic review abstracts. J Clin Epidemiol. 2016. pmid:27555080.
- 23. Cochrane Collaboration. "Cochrane Review Groups"[Internet] 2017 [Date Accessed: 19/09/2017] [cited 2017 19/09/2017]. www.cochrane.org/contact/review-groups].
- 24. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620–6. pmid:20434024.
- 25. Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5:11. pmid:26792080.
- 26. Alonso-Coello P, Vernooij RWM, Carrasco-Labra A, Vandvik P, Guyatt GH. RE: Which estimates comprise optimal reporting in systematic reviews? Journal of Clinical Epidemiology. 2017;81(Supplement C):140–1. https://doi.org/10.1016/j.jclinepi.2017.01.001.
- 27. Ioannidis JP, Horbar JD, Ovelman CM, Brosseau Y, Thorlund K, Buus-Frank ME, et al. Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants. BMJ. 2015;350:h72. Epub 2015/01/28. pmid:25623087.
- 28. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9. pmid:19525005
- 29. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75. pmid:24411645.
- 30. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. pmid:24411643.
- 31. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76. Epub 2014/01/15. pmid:24411647.
- 32. Williamson P, Altman D, Blazeby J, Clarke M, Gargon E. Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012;17(1):1–2. pmid:22294719
- 33. Williamson P, Clarke M. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative: Its Role in Improving Cochrane Reviews. Cochrane Database Syst Rev. 2012;5:ED000041. Epub 2012/05/18. pmid:22592744.
- 34. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132. Epub 2012/08/08. pmid:22867278
- 35. Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews?—a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials. 2013;14. pmid:23339751
- 36. Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5(1):11. pmid:26792080.
- 37. Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS ONE. 2016;11(1):e0146444. pmid:26785121.
- 38. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016;13(5):e1002028. pmid:27218655.