The Core Outcome Measures in Effectiveness Trials (COMET) database is a publically available, searchable repository of published and ongoing core outcome set (COS) studies. An annual systematic review update is carried out to maintain the currency of database content.
The methods used in the fourth update of the systematic review followed the same approach used in the original review and previous updates. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Searches were carried out in March 2018 to identify studies that had been published or indexed between January 2017 and the end of December 2017.
Forty-eight new studies, describing the development of 56 COS, were included. There has been an increase in the number of studies clearly specifying the scope of the COS in terms of the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics. Public participation has continued to rise with over half (n = 27, 56%) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased, in particular participation from non-clinical research experts has risen from 32% (mean average in previous reviews) to 62% (n = 29). Input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) have all increased since the previous reviews.
This update included a pronounced increase in the number of new COS identified compared to the previous three updates. There was an improvement in the reporting of the scope, stakeholder participants and methods used. Furthermore, there has been an increase in participation from Australasia, Asia, South America and Africa. These advancements are reflective of the efforts made in recent years to raise awareness about the need for COS development and uptake, as well as developments in COS methodology.
Citation: Gargon E, Gorst SL, Harman NL, Smith V, Matvienko-Sikar K, Williamson PR (2018) Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research. PLoS ONE 13(12): e0209869. https://doi.org/10.1371/journal.pone.0209869
Editor: Katie Gillies, University of Aberdeen Institute of Applied Health Sciences, UNITED KINGDOM
Received: October 2, 2018; Accepted: December 12, 2018; Published: December 28, 2018
Copyright: © 2018 Gargon et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This work was supported by the National Institute for Health Research Senior Investigator Award (NF-SI_0513-10025) and the Medical Research Council North West Hub for Trials Methodology Research grant (MR/K025635/1). Karen Matvienko-Sikar is funded by a Health Research Board Interdisciplinary Capacity Enhancement Award (2015-1026). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: EG and PW are members of the COMET Management Group. All other authors had no competing interests. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Insufficient consideration of the outcomes measured in clinical trials, and other studies, is increasingly being addressed through the development and use of core outcome sets (COS). A COS is an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all trials for a specific clinical area . The Core Outcome Measures in Effectiveness Trials (COMET) Initiative aims to collate and stimulate the development and application of COS, by maintaining a public repository of studies relevant to the development of COS (The COMET database, http://www.comet-initiative.org/studies/search). A survey of users of the database was undertaken to understand the reasons why people were searching the COMET database . The most frequent users were people thinking about developing a COS to see whether a COS already exists in their area of interest to avoid any duplication, therefore emphasising the importance of keeping the database current.
A systematic review was conducted to initially populate the COMET database , and it has been subsequently updated annually to include all published COS up to, and including, 2016 [2, 4, 5]. The previous update showed that COS developers are adopting a more structured approach towards COS development, and public participation in COS development has also increased . The review, however, highlighted a gap in the involvement of stakeholders from a wider range of geographical settings, in particular low and middle income countries (LMIC). This is important to increase the applicability of COS to global health and tackling the global burden of disease .
The database is therefore an integral resource to not only the development of COS, but also to the uptake of COS in research and in the avoidance of unnecessary duplication and waste of scarce resources. Eligible studies are added to the database as they are found; however, an additional annual update to the systematic review means that the database is kept up to date.
The aims of the current study were consistent with the previous update , specifically to: (i) update the systematic review in order to identify any further studies where a COS has been developed; (ii) to describe the methodological approaches taken in these studies, and (iii) to highlight areas for future COS development and improvement.
Systematic review update
The methods used in this updated review followed the same approach used in the original review  and previous updates [2, 4, 5]. The methods were described in detail previously [3, 5]; a summary of methods is provided below.
The inclusion and exclusion criteria were described in detail in the original systematic review . Studies were eligible for inclusion if they had applied methodology for determining which outcome domains or outcomes should be measured in clinical trials or other forms of health research. As described previously, studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Studies describing the development of a Patient Reported Outcome (PRO) COS (a core set of patient-reported symptoms and health related quality of life domains) or Core Event Set (a core set of adverse events or complications) were eligible for inclusion in the review update. Studies describing the update of an existing COS were included as linked papers to the original COS.
Identification of relevant studies
In March 2018, MEDLINE via Ovid and SCOPUS were searched without language restrictions. The search identified studies that had been published or indexed since the previous systematic review update, between January 2017 and the end of December 2017. The multifaceted search strategy, developed for the original review using a combination of text words and index terms, was used for the current update  (S1 Table). Ovid included a new search option entitled ‘MEDALL’ that incorporates all previous search options and provides the most extensive coverage, including the ‘in-process’ citations as well as E-pub ahead of print citations. This search option was selected for this update of the systematic review in order to identify eligible studies at the earliest possible opportunity.
Hand searching was completed, including any studies that had been submitted directly to the COMET database, references cited in eligible studies, as well as those in ineligible studies that referred to or used a COS.
Selecting studies for inclusion in the review
As described previously [3, 5], records from each database were combined and duplicates removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Two of five reviewers (EG, SG, NH, VS and KMS) independently checked the title and abstract of each citation. Citations were retained for further checking if agreement could not be reached. Two of the same five reviewers assessed each full paper for inclusion in the review. In cases of disagreement a third reviewer was consulted. The reasons for exclusion at this stage were documented for articles judged to be ineligible.
Checking for agreement between reviewers
One reviewer was new to this systematic review and had not been involved in previous updates (KMS). During each stage of the review process, agreement between KMS and the lead reviewer (EG) was assessed prior to independently assessing records. Agreement between all other reviewers had been checked in previous updates.
Checking for correct exclusion
At abstract stage.
A 1% sample of the records excluded on the basis of the title and abstract was checked by a sixth reviewer (Jamie Kirkham) and assessed for correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records.
Described in full previously , data were extracted in relation to the study aim(s), health area, target population, interventions covered, methods of COS development and stakeholder groups involved. Data relating to the geographical locations of participants included in the development of COS were also extracted.
Data analysis and presentation of results
We report the review in accordance with PRISMA guidelines  (S1 File). We describe the studies narratively, and present the findings in text and tables. As in previous updates, we did not anticipate conducting any statistical analyses to combine the findings.
Description of studies
Following the removal of duplicates, 5140 records were identified in the database search. A total of 4626 records were excluded during the title and abstract stage, and a further 444 were excluded following the assessment of full text (Fig 1). Table 1 provides a summary of the reasons for exclusion of records at full text stage. Seventy records related to 45 new studies that met the inclusion criteria. In addition to the review search, three additional records were identified through database search alerts as being eligible for inclusion in the review. These three studies were not identified during the review search, as although they were published in 2017, they had not been indexed in the databases at the time we ran our search. A further 15 reports were identified by hand searching references of included studies. In total, 88 reports relating to 48 new studies describing the development of 56 COS were included for the first time in this update (S2 Table).
Year of publication.
The year of first publication of COS has been updated to include the 48 new studies included in this update (Fig 2). Of the 48 studies identified in this update, 36 were published in 2017, 11 were published in 2016 and one study was published in 2015.
Scope of core outcome sets.
The scope of published COS studies is summarised in Table 2. This includes study aims, setting for intended use, population characteristics and intervention characteristics. Table 2 highlights that there has been an increase in the number of COS studies specifying the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics of the newly developed COS. Fig 3 displays the number of COS that have been developed in each disease category. The majority of COS have been developed within the areas of cancer, rheumatology, heart & circulation, neurology and orthopaedics & trauma. Disease categories and disease names are provided in S2 Table.
Methods used to select outcomes.
The methods used to develop the 48 new COS identified in the current review are presented in Table 3 alongside the methods used in the four previous systematic reviews. Table 3 highlights that the use of mixed methods to develop a COS has remained high (n = 35; 73%). The use of the Delphi technique has also remained high with 48% (n = 23) of studies using the technique in combination with other methods.
People involved in selecting outcomes.
The list of stakeholders included in COS development (Table 4) has been updated to include the COS identified in this update. Of the 307 published COS studies, 273 (89%) provided information about the stakeholders who participated in the COS development. Clinical experts were included in 268 (98%) studies; this is in direct contrast to public representatives who were included in 89 (33%) studies. Public participation has continued to rise in recent years, with 56% (n = 27) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased in comparison to the original review and previous review updates, with participation from non-clinical research experts, in particular, rising from 32% (n = 73) as the mean average in previous reviews to 62% (n = 29) in the current review.
As described previously [3, 5], public representatives include patients, carers, health and social care service users and people from organisations who represent these groups. The degree of public participation within the development of the COS studies included in this updated review is described in Table 5. All of the 27 studies that reported including public participants provided some detail about their participation. Among Delphi studies that reported participation of both clinical experts and the public, levels of participation for public participants ranged from 6%  to 69% .
Table 6 lists the geographical locations, by continent, of the participants involved in developing the COS in this update and in the combined reviews, as reported by the included studies. In the current review, participant locations were provided in 41 of the 48 studies, however for the remaining seven studies, location information was only provided for the authors. The majority of COS were developed with the inclusion of participants located in Europe (n = 38; 93%) and North America (n = 28; 68%). However, input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) has increased since the previous reviews. The number of countries that have been involved in the development of a COS ranges from 1 to 76 (a median of 6).
This systematic review update has identified 48 new COS articles that describe 56 new COS. This is a marked increase in the number of new COS identified in the previous three updates, which included 28, 20 and 15 new COS publications (describing the development of 29, 20 and 15 COS) respectively [2, 4, 5]. Similarly, there has been a continued increase in COMET website and database visits, new visitors, number of database searches and visits from around the world in 2017 (Source of data usage: Google Analytics), with 24,884 visits; 15,546 new visitors; 5142 database searches, and visitors from 139 countries. The sustained growth in use suggests that the COMET website and database are continuing to raise awareness about the existence of, and need for, COS; which could explain this increase in the number of COS published in 2017 and subsequently included in this review. Their inclusion in the database brings the overall total of COS publications to 307, relating to 366 COS.
This review has identified an improvement in the reporting of COS scope (population and intervention characteristics), stakeholder participants and methods used. Core Outcome Set-STAndards for Reporting (COS-STAR) were published in 2016 . This is the first update since the publication of the COS-STAR guidelines, and it is encouraging to see that ten of the 48 studies (21%) included in this update referenced these reporting guidelines. This provides a baseline with which to compare future updates, where we hope to see a continued increase in the number of studies reporting their work using these guidelines.
The previous update highlighted an increase in the use of mixed methods, including the Delphi method, for COS development . A recent survey of ongoing COS developers indicated that mixed methods, including Delphi, were commonly used in ongoing COS development, and there is an increased inclusion of public participants in ongoing studies . As such, the percentage of new studies including public participation is expected to rise in future updates of this systematic review as ongoing studies are published. Participation of non-clinical research experts has increased, and as one of the criteria included in the recently published minimum standards for COS development , we would expect that this should continue to increase in new COS studies included in any further updates to this review. The Core Outcome Set-STAndards for Development (COS-STAD) recommendations will help users to assess whether a particular COS has been developed using a reasonable approach.
The involvement of stakeholders from a range of countries, but in particular lower and middle income countries, has previously been identified as a gap in COS development.  This update has shown an increase in participation from Australasia, Asia, South America and Africa. A recent survey to identify the top priorities for trials methodology research in LMICs found that choosing appropriate outcomes to measure was one of the two most important topics . The involvement of LMIC countries in COS development and uptake is therefore a key area of importance and efforts should be made to improve this.
In conclusion, we have completed the fourth update to a systematic review of COS, identifying COS published and indexed in 2017. COS methodology, and consensus development methodology more generally, have developed over recent years. This is reflected in the improvements seen throughout these updates, not only in the reporting of important COS details (such as scope), but in the methods chosen for development and the stakeholders who participate in the process. With the recent development and publication of guidelines and minimum standards for COS development [1, 12], it is hoped that improvements in these areas will continue.
S1 File. PRISMA Checklist.
PRISMA checklist for content of a systematic review.
We would like to acknowledge Jane Blazeby (Bristol University) and Mike Clarke (Queen’s University Belfast) for their involvement in the conceptualisation and methodology in the original systematic review on which this update is based.
We would like to thank Yalda Ashraf Kharaz (University of Liverpool) for help with document translation of one paper. We also acknowledge Jamie Kirkham (University of Liverpool) for checking for correct exclusion in this review update.
- 1. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280. Epub 2017/07/07. pmid:28681707; PubMed Central PMCID: PMCPMC5499094.
- 2. Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS ONE. 2016;11(1):e0146444. Epub 2016/01/20. pmid:26785121.
- 3. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS ONE. 2014;9(6):e99111. Epub 2014/06/17. pmid:24932522; PubMed Central PMCID: PMC4059640.
- 4. Gorst SL, Gargon E, Clarke M, Smith V, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and Identification of Gaps. PLoS One. 2016;11(12):e0168403. Epub 2016/12/16. pmid:27973622; PubMed Central PMCID: PMC5156438 to support COMET and related work. EG is a member of the COMET Management Group and is the COMET Project Co-ordinator. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
- 5. Davis K, Gorst SL, Harman N, Smith V, Gargon E, Altman DG, et al. Choosing important health outcomes for comparative effectiveness research: An updated systematic review and involvement of low and middle income countries. PLoS One. 2018;13(2):e0190695. Epub 2018/02/14. pmid:29438429; PubMed Central PMCID: PMCPMC5810981.
- 6. Rosala-Hallas A, Bhangu A, Blazeby J, Bowman L, Clarke M, Lang T, et al. Global health trials methodological research agenda: results from a priority setting exercise. Trials. 2018;19(1):48. Epub 2018/02/06. pmid:29397795; PubMed Central PMCID: PMCPMC5798175.
- 7. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. Epub 2009/07/22. pmid:19621072; PubMed Central PMCID: PMCPMC2707599.
- 8. McNamara RL, Spatz ES, Kelley TA, Stowell CJ, Beltrame J, Heidenreich P, et al. Standardized Outcome Measurement for Patients With Coronary Artery Disease: Consensus From the International Consortium for Health Outcomes Measurement (ICHOM). Journal of the American Heart Association. 2015;4(5). Epub 2015/05/21. pmid:25991011; PubMed Central PMCID: PMC4599409.
- 9. MacLennan S, Williamson PR, Bekema H, Campbell M, Ramsay C, N'Dow J, et al. A core outcome set for localised prostate cancer effectiveness trials. BJU international. 2017;120(5b):E64–e79. Epub 2017/03/28. pmid:28346770.
- 10. Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Devane D, et al. Core Outcome Set-STAndards for Reporting: The COS-STAR Statement. PLoS Med. 2016;13(10):e1002148. Epub 2016/10/19. pmid:27755541; PubMed Central PMCID: PMCPMC5068732 EG, PRW, and ST are members of the COMET Management Group. DM is a member of the Editorial Board of PLOS Medicine. DD, SG, JJK, JS, and PT declare no competing interests.
- 11. Biggane AM, Brading L, Ravaud P, Young B, Williamson PR. Survey indicated that core outcome set development is increasingly including patients, being conducted internationally and using Delphi surveys. Trials. 2018;19(1):113. Epub 2018/02/20. pmid:29454368; PubMed Central PMCID: PMC5816387.
- 12. Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017;14(11):e1002447. Epub 2017/11/18. pmid:29145404; PubMed Central PMCID: PMCPMC5689835.