A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition. The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development and application of COS, by including data on relevant studies within a publically available internet-based resource. In recent years, there has been an interest in increasing the development of COS. Therefore, this study aimed to provide an update of a previous review, and examine the quality of development of COS. A further aim was to understand the reasons why individuals are searching the COMET database.
A multi-faceted search strategy was followed, in order to identify studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. Additionally, a pop up survey was added to the COMET website, to ascertain why people were searching the COMET database.
Thirty-two reports relating to 29 studies were eligible for inclusion in the review. There has been an improvement in the description of the scope of a COS and an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. Clinical experts continue to be the most common group involved in developing COS, however patient and public involvement has increased. The pop-up survey revealed the most common reasons for visiting the COMET website to be thinking about developing a COS and planning a clinical trial.
This update demonstrates that recent studies appear to have adopted a more structured approach towards COS development and public representation has increased. However, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database appears to be a useful resource for both COS developers and users of COS.
Citation: Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR (2016) Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS ONE 11(1): e0146444. doi:10.1371/journal.pone.0146444
Editor: Silvio Garattini, Mario Negri Institute for Pharmacology Research, ITALY
Received: September 30, 2015; Accepted: November 9, 2015; Published: January 19, 2016
Copyright: © 2016 Gorst et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was funded by the NIHR (National Institute for Health Research), grant number NF-SI_0513-10025. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: MC, JMB, DGA, and PRW are members of the COMET Management Group and co-applicants on grants to support COMET and related work. EG is a member of the COMET Management Group and is the COMET Project Co-ordinator. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
When designing clinical trials, it is important to measure appropriate outcomes, so that the results can be compared with other trials and will be as useful as possible to decision makers. At present, many studies which explore the effects of the same intervention on a specific health condition measure and report different outcomes, making it difficult to compare, contrast or combine their findings. This causes problems for people trying to use healthcare research. Inconsistency in outcome measurement and outcome reporting bias have led to avoidable waste in the production and reporting of research [1,2]. However, these issues could be reduced through the development and application of core outcome sets (COS). A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition . The existence of a COS allows the results of trials to be brought together as appropriate and ensures that all trials provide usable evidence. COS are not intended to restrict the number of outcomes in a particular trial, rather, the intention is that the outcomes in the COS will always be collected and reported, and it is fully expected that researchers will continue to explore additional outcomes .
The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development, application and promotion of COS, by including data on relevant individual studies within a publically available internet-based resource. The COMET database is a unique inventory that includes published accounts of COS development, as well as planned and ongoing work. Initially, the database included studies that had been identified by ad hoc means and a systematic review which addressed outcome selection in clinical trials in children . In order to bring the database content up-to-date and be comprehensive, a systematic approach was needed to identify relevant material. Gargon and colleagues  conducted a systematic review in 2013, the first comprehensive search for COS in health research, and identified 198 studies that determined which outcomes or domains should be measured in all clinical trials for a specific health condition. The review revealed wide variation in the methods used to develop COS, and highlighted the need for methodological guidance, including how to engage key stakeholder groups, particularly members of the public, in the development and implementation of COS. When using the term ‘public’ through this report we include patients, carers, health and social care service users and people from organisations who represent these groups . The involvement of the public in the development of COS is particularly relevant for comparative effectiveness research where long term patient centred outcomes are often the important endpoints.
Awareness of the need for COS and knowledge of the COMET Initiative has continued to grow, reflected in website and database usage figures . More than 16,500 visits were made to the website in 2014 (36% increase over 2013) and 9780 new visitors (43% increase). By December 2014, a total of 6588 searches of the COMET database had been completed; however no information had been collected about the reasons for searching the database.
To update the original systematic review , in order to identify any further studies where a COS has been developed, and to describe the methodological techniques used in these studies. In addition, to understand the reasons why individuals are searching the COMET database.
Systematic review update
The methods used in this updated review followed the same approach used in the original review .
Study selection. Inclusion and exclusion criteria.
As described in detail previously . Broadly speaking, studies were eligible for inclusion if they had applied methodology for determining which outcome domains or outcomes should be measured, or are important to measure, in clinical trials or other forms of health research.
Types of participants and interventions.
As previously , studies were categorised as eligible if they related to participants of any age, with any health condition, in any setting, and assessed the effect of any or all interventions for that condition.
Identification of relevant studies.
In January 2015, we searched MEDLINE via Ovid and SCOPUS (including EMBASE) without language restrictions. The search identified studies that had been published from January 2013 onwards (MEDLINE) and August 2013 onwards (SCOPUS). The multifaceted search strategy developed in the original review using a combination of text words and index terms  was used in the current review, with adaptations appropriate for each database.
Selecting studies for inclusion in the review.
Records from each database were combined and duplicates were removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Two of three reviewers (SG, JW and BG) independently checked the title and abstract of each citation. One reviewer (SG) assessed the title and abstract of all citations. A second reviewer (JW) assessed the title and abstract of the first half of the citations and a third reviewer (BG) assessed the title and abstract of the second half of the citations. Citations were retained for further checking if agreement could not be reached. One reviewer (SG) assessed each full paper for inclusion in the review and another reviewer (BG) assessed half of the full papers. Reasons for exclusion at this stage were documented for articles judged to be ineligible.
Checking for agreement between reviewers.
During each stage of the review process, agreement between reviewers was assessed. Prior to independently assessing records, the three reviewers (SG, JW and BG) independently checked batches of abstracts and full papers for agreement.
Checking for correct exclusion.
Of the records that had been excluded on the basis of the title and abstract, full text papers were obtained for a 1% sample and a fourth reviewer (EG) assessed correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records. Of the records that had been excluded after reading their full text papers, 5% were assessed for correct exclusion at that stage.
As described in detail previously , data were extracted in relation to the study aims, health area, target population, methods of COS development and stakeholder groups involved.
Data analysis and presentation of results.
As described previously , the results are presented descriptively.
During May-June 2015, a pop up survey consisting of one question was added to the COMET website search page, with the intention of finding out why people were searching in the COMET database. At the beginning of each search, the survey would appear, to ask people to select a response as to their reason for searching in the COMET database. The survey was designed so that it was just one question with a set of multiple-choice answers, which was fully contained within the pop-window. This meant that people could answer the question and close the window with a single click. People were asked only once to do this although they could have run a number of searches.
Description of studies
Following the removal of duplicates, 4980 citations were identified in the initial database search. A total of 4551 records were excluded during the title and abstract stage, and a further 400 were excluded following the assessment of full text papers (Fig 1). S1 Table provides a summary of the reasons for exclusion of the full text papers. Twenty-nine citations met the inclusion criteria. In addition to the database search, three additional citations were identified as being eligible for inclusion in the review following reference checking. In total, 32 reports relating to 29 new studies were included (S2 Table).
Year of publication.
The figure displaying the year of first publication of each COS that was included in the original review has been updated to include the 29 new studies identified in this updated review (Fig 2). The figure confirms that there has been a general increase in the number of COS over the years, and shows that there has been a consistently higher number of COS published annually in recent years than in most years before 2010. Of the 29 studies identified in this update, 28 studies were published between 2013 and 2014, and one study was published in 2010. This study was identified in the searches for the original review , but the lack of information on the COS in the title and abstract meant that it was excluded in the initial stages of that review. This study was identified in the updated review, through the checking of references in the included studies, thus highlighting the value of checking reference lists for locating studies in systematic reviews .
Scope of core outcome sets.
Fig 3 shows the number of COS developed according to disease category. The classification of 227 published COS studies are presented in S3 Table. The scope of published COS studies is summarised in Table 1 and includes both the 198 COS included in the original review and the 29 new COS that have been added by this updated review. This includes study aims, setting for intended use, population characteristics and intervention characteristics.
There has been an improvement in the description of the scope of a COS. In the original review, only 25% (49/198) of studies were found to clearly define the population characteristics, as compared to 55% (n = 16/29) of studies in the updated review. The majority, 58% (n = 115/198), of studies in the original review neglected to report intervention characteristics, compared to 17% (n = 5/29) in the update.
Methods used to select outcomes.
COS have been developed using a variety of methods . The inclusion of a literature/systematic review to inform the development has increased from 33% (n = 66/198) of studies in the original review to 72% (n = 21/29) of studies in the updated review. The use of the Delphi technique for assessing and developing consensus has risen from 15% (n = 29/198) in the original review to 31% (n = 9/29) in the updated review.
People involved in selecting outcomes.
The stakeholder groups regarded as key to developing a COS will vary between clinical areas. Clinical experts continue to be involved in almost all studies (see S4 Table), but there has been a trend towards greater involvement of patient and public representatives, increasing from 18% (31/174) in the original review, to 59% (13/22) in this updated review. Table 2 describes the degree of patient representation reported in COS studies included in this updated review.
During the four-week period of the survey, 396 different people searched the database. The survey achieved a 52% (206/396) response rate. Table 3 shows the frequency of the reasons selected. The most common were ‘I am thinking about developing a core outcome set’ and ‘I am planning a clinical trial’.
As in the original review, studies identified in this update covered various areas of health, with cancer and rheumatology being the most common. In addition to identifying the areas where COS are continuing to be developed, the review identified that COS had been developed for the first time in the areas of rehabilitation and eyes & vision. Regular review updates will allow us to identify the areas where there is an ongoing absence of COS, which may highlight future opportunities for COS developers.
It is apparent that many studies still do not report important characteristics pertaining to the scope of the COS. The development and implementation of a reporting guideline, may facilitate improved reporting of COS, a project currently underway .
A variety of methods continue to be used to develop COS. However, there has been an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. As in the original review, the updated review found clinical experts to be the most common group involved in developing COS. However, the proportion of studies that reported patient and public involvement in the process has increased. Since the systematic search was completed in January 2015, we are aware of a further six COS studies that have been published, and of these six studies, 100% have included patient and public representatives. In addition, patient and public representatives are also involved in 89% (64/72) of the planned and ongoing studies registered in the COMET database. It is important that clinical trials include outcomes that are considered to be important to patients and other users of health and social care. It appears that the issue of patient and public involvement is no longer about whether to involve, but rather how. In response to this need, COMET has set up the PoPPIE (People and Patient Participation, Involvement and Engagement) Working Group to develop resources in this area. As an example, plain language summaries are being developed in partnership with patients, http://www.comet-initiative.org/resources/PlainLanguageSummary, and a research agenda around methods for patient engagement has been set.
The studies identified in this updated review have been added to the COMET database; a freely accessible, publically available, searchable online resource that shows what work has been done in relation to outcomes to measure in effectiveness trials. This update has ensured that the database stores the most current research, and provides readily available information about existing COS, which will assist researchers, practitioners and other stakeholders, including the public, when designing new clinical trials, systematic reviews and clinical guidelines. Moreover, it will also help to avoid unnecessary duplication of efforts in COS development.
The results of the pop up survey demonstrate that people thinking about developing a COS are checking the COMET database to see whether a COS exists in their area of interest to avoid such duplication, hence emphasising the importance of keeping the database current. The pop-up survey also shows that the COMET database is being used for a variety of reasons by people besides COS developers, including clinical trialists, systematic reviewers, auditors, COS funders, and people who have been asked to participate in the development of COS. Interestingly, nearly one sixth of respondents were searching for reasons not anticipated, and further research is needed to understand those. Although 48% of visitors to the website did not respond to the survey, closing down the pop-up box immediately, this may reflect a typical response rate for such surveys.
There are a number of limitations of this review that should be noted. First, although two well-established, large bibliographic databases were searched to identify studies meeting the inclusion criteria, we may have missed some relevant studies within these databases, either because the reports of these studies are not indexed in these databases or because the information in the indexed record is insufficient to show that the report is eligible. It would be beneficial for bibliographic databases to introduce an indexing term to facilitate easier identification of COS. As shown by both our original review  and this update, the identification of COS is a complex task and we will keep the search strategy under review and refine it as necessary to minimise the possibility of missing eligible studies in the future.
A second limitation of the review is that the lack of an assessment tool means that there has been no formal quality assessment of the included studies. Determining the quality of a COS is not straightforward. A high quality COS would be one that leads to improved health outcomes but this would be difficult to measure. A tool is needed to assess how the COS developers minimised biases which would otherwise undermine the ability of the COS to have a positive impact on patient care and outcomes.
We have provided an updated review of studies that have addressed the development of COS for measurement and reporting in clinical trials. The updated review has demonstrated that COS are continuing to be developed across a range of health areas. Recent studies appear to have adopted (or, at least, reported) a more structured approach towards COS development, and public representation has increased. However, although general reporting quality has increased, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database is likely to be a useful resource for both COS developers and users of COS including clinical trialists and systematic reviewers.
S1 PRISMA Checklist. PRISMA checklist for content of a systematic review.
S1 Table. Reason for exclusion at stage 2 (assessment of full text reports).
S2 Table. Table of reports included in updated review (n = 32).
S3 Table. Classification of COS according to condition (n = 227 published studies).
S4 Table. Participant groups involved in selecting outcomes in new studies identified in the review update (n = 29).
The authors would like to thank Jennifer Weston and Binu Gurung for their assistance in screening studies for inclusion in the review.
Conceived and designed the experiments: EG PRW. Performed the experiments: SLG EG. Analyzed the data: SLG EG PRW MC JMB DGA. Wrote the paper: SLG EG PRW MC.
- 1. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374:86–89. doi: 10.1016/S0140-6736(09)60329-9. pmid:19525005
- 2. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383:101–104. doi: 10.1016/S0140-6736(13)62329-6. pmid:24411643
- 3. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132. doi: 10.1186/1745-6215-13-132. pmid:22867278
- 4. Sinha I, Jones L, Smyth RL, Williamson PR. A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Med. 2008;5:e96. doi: 10.1371/journal.pmed.0050096. pmid:18447577
- 5. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS ONE. 2014;9(6):e99111. doi: 10.1371/journal.pone.0099111. pmid:24932522
- 6. INVOLVE. Briefing notes for researchers: involving the public in NHS, public health and social care research. Eastleigh: INVOLVE; 2012.
- 7. Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET Initiative database: progress and activities from 2011 to 2013. Trials. 2014;15(1):279. doi: 10.1186/1745-6215-15-279
- 8. Gargon E, Williamson PR, Clarke M. Collating the knowledge base for core outcome set development: developing and appraising the search strategy for a systematic review. BMC Medical Research Methodology. 2015; 15:26. doi: 10.1186/s12874-015-0019-9. pmid:25888523
- 9. Horsley T, Dingwall O, Sampson M. Checking reference lists to find studies for systematic reviews. Cochrane Databaser Syst Rev. 2011;8:MR000026. doi: 10.1002/14651858.mr000026.pub2
- 10. Kirkham JJ, Gorst S, Altman DG, Blazeby J, Clarke M, Devane D, et al. COS-STAR: a reporting guideline for studies developing core outcome sets (protocol). Trials. 2015;16:373. doi: 10.1186/s13063-015-0913-9. pmid:26297658