Figures
Abstract
Background
Core outcome sets (COS) comprise a minimum set of outcomes that should be measured and reported in all trials for a specific health condition. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative maintains an up to date, publicly accessible online database of published and ongoing COS. An annual systematic review update is an important part of this process.
Methods
This review employed the same, multifaceted approach that was used in the original review and the previous two updates. This approach has identified studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. This update includes an analysis of the inclusion of participants from low and middle income countries (LMICs) as identified by the OECD, in these COS.
Results
Eighteen publications, relating to 15 new studies describing the development of 15 COS, were eligible for inclusion in the review. Results show an increase in the use of mixed methods, including Delphi surveys. Clinical experts remain the most common stakeholder group involved. Overall, only 16% of the 259 COS studies published up to the end of 2016 have included participants from LMICs.
Citation: Davis K, Gorst SL, Harman N, Smith V, Gargon E, Altman DG, et al. (2018) Choosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countries. PLoS ONE 13(2): e0190695. https://doi.org/10.1371/journal.pone.0190695
Editor: Roberta W. Scherer, Johns Hopkins University Bloomberg School of Public Health, UNITED STATES
Received: September 12, 2017; Accepted: December 19, 2017; Published: February 13, 2018
Copyright: © 2018 Davis et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was funded by the NIHR (National Institute for Health Research, https://www.nihr.ac.uk/), grant number NF-SI_0513-10025. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: DGA, JMB, MC, ST and PRW are members of the COMET Management Group and coapplicants on grants to support COMET and related work. KD and EG are the COMET Project Co-ordinators. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
Introduction
Measuring appropriate outcomes in clinical trials enables the benefits and harms of specific treatments to be compared between trials and allows decision makers, such as patients and clinicians, to be best informed in their choice. This is known as comparative effectiveness research (CER) [1]. Heterogeneity of outcome reporting in trials is, however, common, even in trials exploring the effects of the same intervention on a specific disease [2]. The lack of comparability that results from this causes waste in research [3,4]. This waste could be avoided with the development and systematic use of core outcome sets (COS) in clinical trials research, as demonstrated by the uptake in trials of a COS developed in rheumatoid arthritis [5]. A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials for a specific health condition [2]. An element of the scope of the COS, to be defined, is whether the COS is to be applicable to all interventions or exclusively to specific intervention types. The use of COS would ensure all trials produce evidence that can be combined appropriately with the results of other trials, while not restricting the researchers’ ability to explore other, additional outcomes [2].
The purpose of the COMET (Core Outcome Measures in Effectiveness Trials) Initiative is to encourage the development, promotion and application of COS across all health areas. It does so by maintaining an up-to-date, online, publicly-accessible database of ongoing and published COS studies. Three systematic reviews have comprehensively searched for COS; the original review, conducted in 2013 [6], and two updates, one conducted in 2015 [7] and the second conducted in 2016 [8].
The 2016 update showed that COS had been developed for 13 of the world’s 25 most prevalent health conditions, leaving 12 of the 25 as being in need of COS development [8]. Eighty-two percent of the COS applicable to the world’s most prevalent health conditions involved participants from Europe and North America only. International participation in COS was highlighted as a key research gap. Representation of stakeholders from Africa and South America was particularly low in comparison to other continents, at only one and three studies (respectively). Further research is needed to establish how COS can be developed to have the greatest global relevance and applicability to the global burden of disease.
Aims
The aims of the current study were to (i) update the systematic review [6–8] in order to identify any further studies where a COS has been developed; (ii) to describe the methodological approaches taken in these studies, (iii) to identify countries in which COS development participants are located and (iv) to highlight areas for future COS development and improvement.
Methods
Systematic review update
The methods used in this updated review followed the same approach used in the original review and in the previous two updates [6–8].
Study selection.
Inclusion and exclusion criteria: As described in detail previously [6], studies were eligible for inclusion if they developed or applied methodology for determining which outcome domains or outcomes should be measured, or are important to measure, in clinical trials or other forms of health research. As noted in the 2016 update [8], by using the term `outcome' we are referring to something that occurs as a result of the specific health condition (e.g. diarrhoea) and by `outcome domain' we are referring to the grouping of individual outcomes (e.g. bowel function, which would include diarrhoea).
Types of participants and interventions: As described previously [6], studies were categorised as eligible if they reported the development of a COS, regardless of any restrictions by age, health condition or setting, which could be used to assess the effect of interventions for that condition.
Identification of relevant studies.
In March 2017, we searched MEDLINE via Ovid and SCOPUS (including EMBASE) without language restrictions. The search identified studies that had been published or indexed between the previous systematic review update [8] in January 2016 and the end of December 2016. The multifaceted search strategy, developed for the original review using a combination of text words and index terms and combining three concepts of search terms that cover ‘randomised trial’, ‘systematic review’, ‘methodology’ and ‘outcomes’ [9] was used in the current update, with adaptations appropriate for each database (S1 Table). The search was deferred until March 2017 due to indexing issues that had been identified with the previous reviews [8], namely that studies published at the end of a particular year of interest had not been indexed in the database until the beginning of the following year and hence were not captured by the database search. We searched in March for this update to try to overcome this limitation. In addition to this database searching, we also completed hand searching activities as previously described [8]. We identified any studies that had been submitted directly to the COMET database. We also examined references cited in eligible studies and in ineligible studies that referred to or used a COS.
Selecting studies for inclusion in the review.
Records from each database were combined and duplicates were removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Titles and abstracts were divided between reviewers (KD, NH, VS and PRW) who independently checked the title and abstract of each citation assigned to them and classified them as include, exclude or unsure. Where there was uncertainty in the classification of a record, a second reviewer was consulted. If agreement could not be achieved, the citation was referred to a fifth reviewer (SG). Full papers were divided between three reviewers (KD, NH and VS) for full-paper review. Reasons for exclusion at this stage were documented for articles judged to be ineligible.
Checking for agreement between the reviewers.
During each stage of the review process, agreement between reviewers was assessed. Prior to independently assessing records, the four reviewers (KD, NH, VS and PRW) independently checked batches of abstracts and full papers to confirm consistency. This was repeated in batches of 10 until complete agreement was reached in three consecutive batches.
Checking for correct exclusion.
Of the records that had been excluded on the basis of the title and abstract, full text papers were obtained for a 1% sample and a fifth reviewer (SG) assessed correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records. Of the records that had been excluded after reading their full text papers, 5% were assessed for correct exclusion at that stage. In this review, no studies were excluded incorrectly.
Data extraction.
As described in detail previously [6], data were extracted in relation to the study aim(s), health area, target population, interventions covered, methods of COS development and stakeholder groups involved. Data relating to the geographical locations of participants included in the development of COS were extracted for studies found in the current update, as was done in the previous reviews.
Data analysis and results.
As previously [6], we used a narrative analysis and the results are presented descriptively. The results were analysed for the scope of their aims, intended use, population characteristics and intervention characteristics. The methods used, and stakeholders involved in selecting outcomes were also analysed. This update also included an analysis of the inclusion of participants from low and middle income countries (LMICs), as identified by the OECD, in these COS.
Results
Description of studies
Following the removal of duplicates, 4406 citations were identified in the database search. A total of 3887 records were excluded during the title and abstract stage, and a further 503 were excluded following the assessment of full text papers (Fig 1). Table 1 provides a summary of the reasons for exclusion of papers at abstract and full text stage. Sixteen citations relating to 13 new studies met the inclusion criteria. In addition to the database search, two additional citations were identified as being eligible for inclusion in the review. In total, 18 reports relating to 15 new studies describing the development of 15 COS were added to the review during this update (S2 Table).
Three of the reports included in this updated are linked to COS studies that were reported in previous reviews [6,7]. One report is linked to a previously published preliminary COS, which has since been modified and endorsed [10]. Two further reports are linked to previously published COS, which have been revised to meet the current standards of outcome measure development [11,12]. All updates and revisions reported in the linked papers are reflected in the tables below.
Included studies
Year of publication.
Our analysis of the year of first publication of each COS study included in the previous reviews has been updated to include the 15 new studies identified in this updated review (Fig 2). Of the 15 studies discussed here, 13 studies were published in 2015 or 2016, and two studies were published in 2014. Two studies were identified through reference checking of a COS study. One of these studies had been excluded from a previous review at the title and abstract screening stage because of the lack of COS information in the abstract [13]. The other study was identified through reference checking of related COS studies and had not come up in any of the previous review searches [14]. As referred to by the study authors, the scope of this COS, specifically the health condition, overlaps with earlier COS.
Scope of core outcome sets.
The scope of published COS studies is summarised in Table 2 and includes 259 COS studies (describing the development of 309 COS) that were included in the three previous systematic reviews and the 15 new COS studies that have been added by this updated review. Please see S2 Table in supplementary material for the scope of the disease categories and names relevant to each of the 15 new COS. During this update, five studies that were judged to be COS in previous reviews were reviewed and excluded [15–19]. One was retrospectively deemed to refer to how to measure outcomes rather than what to measure [18] and four others were deemed to be composite end points rather than core outcome sets [15–17,19].
Methods used to select outcomes.
The methods used to develop the 15 new COS are presented in Table 3 alongside the methods used in the combined reviews [6–8]. The results show an increase in the use of mixed methods, including Delphi surveys.
Stakeholders involved in selecting outcomes.
Table 4 lists the stakeholders that were involved in selecting outcomes for inclusion in the COS identified in this update and the combined reviews. Regarding the 259 published COS studies, 225 have provided details about the stakeholders who participated in the development process. Of these 225 studies, clinical experts have been involved in selecting outcomes for inclusion in 222 (99%) studies; this contrasts with public representatives, who have been included in only 62 (28%) studies.
Public representatives include patients, carers, health and social care service users and people from organisations who represent these groups. The degree of public participation within the development of the COS studies included in this updated review is described in Table 5. Seven of the eight studies that reported including public participants, provided details about their participation. Among studies that report public participation, levels of participation range from 15% [20] to 66% [21]. Where patients are involved, they tend to make up a greater percentage of the participants than in COS reported in previous reviews.
Countries involved in the development of COS
Table 6 lists the geographical locations, by continent, of the participants involved in developing the COS in this update and in the combined reviews, as reported by the included studies.
Fig 3 displays a choropleth map of the world, which uses differences in shading to indicate the number of COS studies that have been developed with the inclusion of participants from each country. Of the 259 COS studies published by the end of 2016, the countries which have been most involved in the COS development process are the United States of America (USA) (n = 161) and the United Kingdom (UK) (n = 124). There are 93 countries which are displayed on the map in the lightest shading; these represent the locations that have not been included in the development of any COS.
Of the 259 published COS studies, 41 (16%) have included participants from low and middle income countries (LMICs). These LMICs were identified according to the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) list of Official Development Assistance (ODA) recipients [22]. Table 7 presents a breakdown of the LMICs that have been involved in the development of COS.
Discussion
This update to the previous three COMET Initiative systematic reviews has identified 18 publications relating to 15 new COS, suggesting that COS continue to be developed and published.
A range of resources are available on the COMET website to facilitate COS development. These include a core resource pack, plain language summaries and information about how to optimise the involvement of parents, young people and patient organisations in COS studies. This work has now been brought together in the COMET Handbook [23] to provide guidance on the development, application and implementation of COS.
Of the 259 COS studies published up to the end of 2016, the USA and the UK continue to be the most frequently represented countries by stakeholders involved. Use of the COMET website is continuing to increase, with the number of visitor locations reaching 175 countries in 2017 [24], however there are 96 countries that have not been included in the development of any COS. COS developed with globally rather than nationally located participants might improve the global applicability of clinical trials. To be comprehensive, a COS should be applicable in all relevant settings, including across different countries where appropriate. COS are more likely to be applicable within those countries that the participants in the development process are from. Therefore, if a COS is intended to have international applicability, this would have implications for how it is developed, who is involved, and the resources required for the development process. In particular, the developers should involve participants from countries where the prevalence or burden of the disease is high, but, with this in mind, it is worth noting that only 16% of COS developed to date have included participants from LMICs. Furthermore, COS developed with participants from multiple countries might improve the global applicability of the COS and, consequently, the global relevance and impact of the clinical trials that these are used in. A survey has recently been undertaken to identify the top priorities for trials methodological research in LMICs to inform further research and ultimately to improve clinical trials in these regions. The priority most commonly graded as critically important, amongst 400 participants from over 80 countries, was choosing appropriate outcomes to measure [25].
Implications
The 15 new COS studies have been added to the COMET database, ensuring that it will remain up to date and continue to assist trialists, researchers, clinicians and others to design clinical trials, guidelines and systematic reviews. It will also remain a key source of information for those looking to develop a COS. Although reporting has improved, use of the COS-STAR guideline [26] will help further.
The updated review reported here highlights a continuing need to involve participants from a greater range of geographical locations in COS development, particularly from LMICs, in order to increase the global relevance of COS.
Limitations
Gathering accurate data on the geographic location of participants is dependent on the reporting of this information, which was provided for over 70% of the COS studies in this review update. It is possible that there could be more COS involving participants from LMICs than reported here, although, there is no obvious reason why those reporting geographic information may differ with respect to LMIC involvement.
The abstracts reviewed in this study were reviewed by a single author only, in contrast to previous reviews where two reviewers reviewed each abstract. It is conceivable that some COS may have been missed at this stage of screening, however checks have been put in place to minimise incorrect exclusion, including batch checks and additional reviewers checking samples of excluded studies. The impact of this lack of identification on the usefulness of the COMET database is minimised, as the annual systematic review update is only one of a number of methods used to populate the COMET database, which also include database alerts and direct contact with COS developers throughout the year.
Conclusions
In conclusion, this study has demonstrated the continuing development of new COS. The results show an increase in the use of mixed methodologies for COS development but suggest a need to push for greater public participation and the involvement of stakeholders from a broader range of countries, in particular LMICs. These efforts will increase the applicability of COS to global health and tackling the global burden of disease.
Supporting information
S1 PRISMA Checklist. PRISMA checklist for content of a systematic review.
https://doi.org/10.1371/journal.pone.0190695.s001
(DOC)
S2 Table. Table of reports included in updated review (n = 18).
https://doi.org/10.1371/journal.pone.0190695.s003
(DOCX)
References
- 1. Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med. 2009;151(3):203–205. pmid:19567618
- 2. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132. pmid:22867278
- 3. Chalmers I, Glasziou P. Avoidable waster in the production and reporting of research evidence. Lancet. 2009;374(9683):86–89. pmid:19525005
- 4. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. pmid:24411643
- 5. Kirkham JJ, Boers M, Tugwell P, Clarke M, Williamson PR. Outcome measures in rheumatoid arthritis randomised trials over the last 50 years. Trials. 2013;14:324. pmid:24103529
- 6. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS ONE. 2014;9(6):e99111. pmid:24932522
- 7. Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and user survey. PLoS ONE. 2016;11(1):e0146444. pmid:26785121
- 8. Gorst SL, Garon E, Clarke M, Smith V, Williamson PR. Choosing important health outcomes for comparative effectiveness research: an updated review and identification of gaps. PLoS ONE. 2016;11(12):e0168403. pmid:27973622
- 9. Gargon E, Williamson PR, Clarke M. Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review. BMC Medical Research Methodology. 2015;15:26. pmid:25888523
- 10. Khanna D, Mittoo S, Aggarwal R, Proudman SM, Dalbeth N, Matteson EL, et al. Connective Tissue Disease-associated Interstitial Lung Diseases (CTD-ILD)—Report from OMERACT CTD-ILD Working Group. The Journal of rheumatology. 2015; 42(11):2168–71. pmid:25729034
- 11. Milman N, Boonen A, Tugwell P, Merkel PA; OMERACT Vasculitis Working Group. Clinicians’ perspective on key domains in ANCA-associated vasculitis: a Delphi exercise. Scandinavian Journal of Rheumatology. 2017;46(2):112–7. pmid:27438345
- 12. Orbai AM, de Wit M, Mease P, Shea JA, Gossec L, Leung YY, et al. International patient and physician consensus on a psoriatic arthritis core outcome set for clinical trials. Annals of the Rheumatic Diseases. 2017;76(4):673–80. pmid:27613807
- 13. Myatt L, Redman CW, Staff AC, Hansson S, Wilson ML, Laivuori H, et al. Strategy for Standardization of Preeclampsia Research. Hypertension. 2014;63(6):1293–301. pmid:24688121
- 14. Deyo R.A., et al., Report of the NIH Task Force on Research Standards for Chronic Low Back Pain. Journal of Pain. 2014;15(6):569–85. pmid:24787228
- 15. Billings FT, Shaw AD. Clinical trial endpoints in acute kidney injury. Nephron. 2014;127(1–4):89–93. pmid:25343828
- 16. Coolsen MM, Clermonts SH, van Dam RM, Winkens B, Malagó M, Fusai GK, et al. Development of a composite endpoint for randomized controlled trials in pancreaticoduodenectomy. World J Surg. 2014;38(6):1468–1475. pmid:24366279
- 17. Fong F, Rogozinska E, Allotey J, Kempley S, Shah DK, Thangaratinam S. Development of maternal and neonatal composite outcomes for trials evaluating management of late-onset pre-eclampsia. Hypertens Pregnancy. 2014;33(2):115–131. pmid:24303960
- 18. Scher HI, Eisenberger M, D'Amico AV, Halabi S, Small EJ, Morris M, et al. Eligibility and outcomes reporting guidelines for clinical trials for patients in the state of a rising prostate-specific antigen: recommendations from the Prostate-Specific Antigen Working Group. Journal of Clinical Oncology. 2004;22(3):537–556. pmid:14752077
- 19. Smaïl-Faugeron V, Fron Chabouis H, Durieux P, Attal JP, Muller-Bolla M, Courson F. Development of a core set of outcomes for randomized controlled trials with multiple outcomes—example of pulp treatments of primary teeth for extensive decay in children. PLoS ONE. 2013;8(1):e51908. pmid:23300955
- 20. Al Wattar BH, Tamilselvan K, Khan R, Kelso A, Sinha A, Pirie AM, et al. Development of a core outcome set for epilepsy in pregnancy (E-CORE): a national multi-stakeholder modified Delphi consensus study. BJOG: An International Journal of Obstetrics and Gynaecology. 2017;124(4):661–667.
- 21. Gerritsen A, Jacobs M, Henselmans I, van Hattum J, Efficace F, Creemers GJ, et al. Developing a core set of patient-reported outcomes in pancreatic cancer: A Delphi survey. European journal of cancer. 2016;57:68–77. pmid:26886181
- 22.
Organisation for Economic Co-operation and Development. Development Assistance Committee (DAC) list of Official Development Assistance (ODA) recipients. Available from http://www.oecd.org/dac/stats/daclist.htm. Accessed July 17, 2017.
- 23. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18.
- 24.
Google. Google Analytics. 2017. Available from: https://analytics.google.com. Accessed July 17, 2017.
- 25. Rosala-Hallas A, Bhangu A, Bowman L, Blazeby J, Clarke M, Lang T, et al. Global Health Trials Methodological Research Agenda: Results from a priority setting exercise. Trials. Under Review.
- 26. Kirkham JJ, Gorst S, Altman DG, Blazeby J, Clarke M, Devane D, et al. Core Outcome Set–STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016;13(10):e1002148. pmid:27755541