MC, JMB, DGA, and PRW are members of the COMET Management Group and co-applicants on grants to support COMET and related work. EG is a member of the COMET Management Group and is the COMET Project Co-ordinator. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
Conceived and designed the experiments: EG PRW. Performed the experiments: SLG EG. Analyzed the data: SLG EG PRW MC JMB DGA. Wrote the paper: SLG EG PRW MC.
A COS represents an agreed minimum set of outcomes that should be measured and reported in all trials of a specific condition. The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development and application of COS, by including data on relevant studies within a publically available internet-based resource. In recent years, there has been an interest in increasing the development of COS. Therefore, this study aimed to provide an update of a previous review, and examine the quality of development of COS. A further aim was to understand the reasons why individuals are searching the COMET database.
A multi-faceted search strategy was followed, in order to identify studies that sought to determine which outcomes/domains to measure in clinical trials of a specific condition. Additionally, a pop up survey was added to the COMET website, to ascertain why people were searching the COMET database.
Thirty-two reports relating to 29 studies were eligible for inclusion in the review. There has been an improvement in the description of the scope of a COS and an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. Clinical experts continue to be the most common group involved in developing COS, however patient and public involvement has increased. The pop-up survey revealed the most common reasons for visiting the COMET website to be thinking about developing a COS and planning a clinical trial.
This update demonstrates that recent studies appear to have adopted a more structured approach towards COS development and public representation has increased. However, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database appears to be a useful resource for both COS developers and users of COS.
When designing clinical trials, it is important to measure appropriate outcomes, so that the results can be compared with other trials and will be as useful as possible to decision makers. At present, many studies which explore the effects of the same intervention on a specific health condition measure and report different outcomes, making it difficult to compare, contrast or combine their findings. This causes problems for people trying to use healthcare research. Inconsistency in outcome measurement and outcome reporting bias have led to avoidable waste in the production and reporting of research [
The COMET (Core Outcome Measures in Effectiveness Trials) initiative aims to collate and stimulate the development, application and promotion of COS, by including data on relevant individual studies within a publically available internet-based resource. The COMET database is a unique inventory that includes published accounts of COS development, as well as planned and ongoing work. Initially, the database included studies that had been identified by ad hoc means and a systematic review which addressed outcome selection in clinical trials in children [
Awareness of the need for COS and knowledge of the COMET Initiative has continued to grow, reflected in website and database usage figures [
To update the original systematic review [
The methods used in this updated review followed the same approach used in the original review [
As described in detail previously [
As previously [
In January 2015, we searched MEDLINE via Ovid and SCOPUS (including EMBASE) without language restrictions. The search identified studies that had been published from January 2013 onwards (MEDLINE) and August 2013 onwards (SCOPUS). The multifaceted search strategy developed in the original review using a combination of text words and index terms [
Records from each database were combined and duplicates were removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Two of three reviewers (SG, JW and BG) independently checked the title and abstract of each citation. One reviewer (SG) assessed the title and abstract of all citations. A second reviewer (JW) assessed the title and abstract of the first half of the citations and a third reviewer (BG) assessed the title and abstract of the second half of the citations. Citations were retained for further checking if agreement could not be reached. One reviewer (SG) assessed each full paper for inclusion in the review and another reviewer (BG) assessed half of the full papers. Reasons for exclusion at this stage were documented for articles judged to be ineligible.
During each stage of the review process, agreement between reviewers was assessed. Prior to independently assessing records, the three reviewers (SG, JW and BG) independently checked batches of abstracts and full papers for agreement.
Of the records that had been excluded on the basis of the title and abstract, full text papers were obtained for a 1% sample and a fourth reviewer (EG) assessed correct exclusion. If any studies were identified as being incorrectly excluded, further checking was performed within the other excluded records. Of the records that had been excluded after reading their full text papers, 5% were assessed for correct exclusion at that stage.
As described in detail previously [
As described previously [
During May-June 2015, a pop up survey consisting of one question was added to the COMET website search page, with the intention of finding out why people were searching in the COMET database. At the beginning of each search, the survey would appear, to ask people to select a response as to their reason for searching in the COMET database. The survey was designed so that it was just one question with a set of multiple-choice answers, which was fully contained within the pop-window. This meant that people could answer the question and close the window with a single click. People were asked only once to do this although they could have run a number of searches.
Following the removal of duplicates, 4980 citations were identified in the initial database search. A total of 4551 records were excluded during the title and abstract stage, and a further 400 were excluded following the assessment of full text papers (
The figure displaying the year of first publication of each COS that was included in the original review has been updated to include the 29 new studies identified in this updated review (
Original review n (%) | Updated review n (%) | Combined n (%) | |
---|---|---|---|
Specifically considered outcome selection and measurement | 97 (49) | 22 (76) | 119 (52%) |
Considered outcomes while addressing wider clinical trial design issues | 101 (51) | 7 (24) | 108 (48%) |
Clinical trials | 141 (71) | 19 (66) | 160 (71) |
Clinical research | 27 (14) | 4 (14) | 31 (14) |
Clinical research and practice | 11 (6) | 4 (14) | 15 (7) |
Clinical trials and clinical practice | 10 (5) | 0 (0) | 10 (4) |
Clinical trials and regulatory purposes | 3 (2) | 0 (0) | 3 (1) |
Trials and observational studies | 3 (2) | 0 (0) | 3 (1) |
Clinical trial extension studies | 1 (4) | 1 (<1) | |
Clinical trials, research and clinical record keeping | 1 (4) | 1 (<1) | |
Observational studies | 1 (<1) | 0 (0) | 1 (<1) |
Trials and case series | 1 (<1) | 0 (0) | 1 (<1) |
Clinical research, clinical practice and regulatory purpose | 1 (<1) | 0 (0) | 1 (<1) |
Adults | 10 (5) | 11 (38) | 21 (9) |
Children | 23 (12) | 2 (7) | 25 (11) |
Adults and children | 13 (7) | 1 (4) | 14 (6) |
Older adults | 3 (2) | 1 (4) | 4 (2) |
Adults and neonates | 0 (0) | 1 (4) | 1 (<1) |
Not specified | 149 (75) | 13 (45) | 162 (71) |
All intervention types | 7 (4) | 9 (31) | 16 (7) |
Drug treatments | 40 (20) | 4 (14) | 44 (19) |
Surgery | 13 (7) | 4 (14) | 17 (8) |
Vaccine | 2 (1) | 0 (0) | 2 (1) |
Rehabilitation | 1 (1) | 1 (4) | 2 (1) |
Exercise | 1 (1) | 1 (4) | 2 (1) |
Procedure |
5 (3) | 0 (0) | 5 (2) |
Device |
3 (2) | 0 (0) | 3 (1) |
Other |
11 (6) | 5 (17) | 16 (7) |
Not specified | 115 (58) | 5 (17) | 120 (53) |
*
**
***
Coronary angiogenesis.
Hip protectors.
Neuro-protective therapy (aka Neuroprotection).
Non-surgical treatment (no other detail given).
Operative and non-operative management.
Oral care products.
Ascorbic acid.
Fall injury prevention interventions.
Behavioural therapies or other kinds of nonpharmacologic therapies.
Psychological & behavioural: Psychosocial.
Maternity care.
There has been an improvement in the description of the scope of a COS. In the original review, only 25% (49/198) of studies were found to clearly define the population characteristics, as compared to 55% (n = 16/29) of studies in the updated review. The majority, 58% (n = 115/198), of studies in the original review neglected to report intervention characteristics, compared to 17% (n = 5/29) in the update.
COS have been developed using a variety of methods [
The stakeholder groups regarded as key to developing a COS will vary between clinical areas. Clinical experts continue to be involved in almost all studies (see
Methods used | Total number of participants | Number of public participants | % Public participants | |
---|---|---|---|---|
1 | Meeting and teleconferences (mixed) | 6 | 1 | 17% |
2 | Nominal Group Technique (mixed) | 25 | 3 | 12% |
3 | Meeting (mixed) | 12 | 1 | 8% |
4 | Focus groups (patient only) | 45 | 45 | |
Delphi (clinician only) | Round 1: 249 | Round 1: 249 | ||
Round 2: 247 | Round 2: 247 | |||
Round 3: 247 | Round 3: 247 | |||
Nominal Group Technique (mixed) | 28 | 5 | 18% | |
5 |
Delphi (patient only) | Round 1: 169 | Round 1: 169 | |
Round 2: 152 | Round 2: 152 | |||
Round 3: 147 | Round 3: 147 | |||
6 | Delphi (patient only) | Round 1: 71 | Round 1: 71 | |
Round 2: 67 | Round 2: 67 | |||
Round 3: 62 | Round 3: 62 | |||
Delphi (clinician only) | Round 1: 39 | Round 1: 39 | ||
Round 2: 35 | Round 2: 35 | |||
Round 3: 33 | Round 2: 33 | |||
Round 2: 35 | Round 2: 35 | |||
Round 3: 33 | Round 2: 33 |
*
During the four-week period of the survey, 396 different people searched the database. The survey achieved a 52% (206/396) response rate.
Reason | N |
---|---|
I am thinking about developing a core outcome set | 49 |
We are considering funding a core outcome set | 3 |
I am reviewing a funding application for the development of a core outcome set | 0 |
I have been asked to take part in a core outcome set study | 2 |
I am a person with a condition | 0 |
I am planning a systematic review of clinical trials | 19 |
I am planning a clinical trial | 33 |
I am reviewing a funding application for a trial | 4 |
We are considering funding a clinical trial | 2 |
I am involved in auditing a condition | 6 |
As part of a general educational activity | 29 |
General interest | 27 |
None of the above | 32 |
Did not respond | 190 |
396 |
As in the original review, studies identified in this update covered various areas of health, with cancer and rheumatology being the most common. In addition to identifying the areas where COS are continuing to be developed, the review identified that COS had been developed for the first time in the areas of rehabilitation and eyes & vision. Regular review updates will allow us to identify the areas where there is an ongoing absence of COS, which may highlight future opportunities for COS developers.
It is apparent that many studies still do not report important characteristics pertaining to the scope of the COS. The development and implementation of a reporting guideline, may facilitate improved reporting of COS, a project currently underway [
A variety of methods continue to be used to develop COS. However, there has been an increase in the proportion of studies using literature/systematic reviews and the Delphi technique. As in the original review, the updated review found clinical experts to be the most common group involved in developing COS. However, the proportion of studies that reported patient and public involvement in the process has increased. Since the systematic search was completed in January 2015, we are aware of a further six COS studies that have been published, and of these six studies, 100% have included patient and public representatives. In addition, patient and public representatives are also involved in 89% (64/72) of the planned and ongoing studies registered in the COMET database. It is important that clinical trials include outcomes that are considered to be important to patients and other users of health and social care. It appears that the issue of patient and public involvement is no longer about whether to involve, but rather how. In response to this need, COMET has set up the PoPPIE (People and Patient Participation, Involvement and Engagement) Working Group to develop resources in this area. As an example, plain language summaries are being developed in partnership with patients,
The studies identified in this updated review have been added to the COMET database; a freely accessible, publically available, searchable online resource that shows what work has been done in relation to outcomes to measure in effectiveness trials. This update has ensured that the database stores the most current research, and provides readily available information about existing COS, which will assist researchers, practitioners and other stakeholders, including the public, when designing new clinical trials, systematic reviews and clinical guidelines. Moreover, it will also help to avoid unnecessary duplication of efforts in COS development.
The results of the pop up survey demonstrate that people thinking about developing a COS are checking the COMET database to see whether a COS exists in their area of interest to avoid such duplication, hence emphasising the importance of keeping the database current. The pop-up survey also shows that the COMET database is being used for a variety of reasons by people besides COS developers, including clinical trialists, systematic reviewers, auditors, COS funders, and people who have been asked to participate in the development of COS. Interestingly, nearly one sixth of respondents were searching for reasons not anticipated, and further research is needed to understand those. Although 48% of visitors to the website did not respond to the survey, closing down the pop-up box immediately, this may reflect a typical response rate for such surveys.
There are a number of limitations of this review that should be noted. First, although two well-established, large bibliographic databases were searched to identify studies meeting the inclusion criteria, we may have missed some relevant studies within these databases, either because the reports of these studies are not indexed in these databases or because the information in the indexed record is insufficient to show that the report is eligible. It would be beneficial for bibliographic databases to introduce an indexing term to facilitate easier identification of COS. As shown by both our original review [
A second limitation of the review is that the lack of an assessment tool means that there has been no formal quality assessment of the included studies. Determining the quality of a COS is not straightforward. A high quality COS would be one that leads to improved health outcomes but this would be difficult to measure. A tool is needed to assess how the COS developers minimised biases which would otherwise undermine the ability of the COS to have a positive impact on patient care and outcomes.
We have provided an updated review of studies that have addressed the development of COS for measurement and reporting in clinical trials. The updated review has demonstrated that COS are continuing to be developed across a range of health areas. Recent studies appear to have adopted (or, at least, reported) a more structured approach towards COS development, and public representation has increased. However, although general reporting quality has increased, there remains a need for developers to adequately describe details about the scope of COS, and for greater public engagement. The COMET database is likely to be a useful resource for both COS developers and users of COS including clinical trialists and systematic reviewers.
(DOC)
(DOCX)
(DOCX)
(DOCX)
(DOCX)
The authors would like to thank Jennifer Weston and Binu Gurung for their assistance in screening studies for inclusion in the review.