Choosing important health outcomes for comparative effectiveness research: 4th annual update to a systematic review of core outcome sets for research

Background The Core Outcome Measures in Effectiveness Trials (COMET) database is a publically available, searchable repository of published and ongoing core outcome set (COS) studies. An annual systematic review update is carried out to maintain the currency of database content. Methods The methods used in the fourth update of the systematic review followed the same approach used in the original review and previous updates. Studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Searches were carried out in March 2018 to identify studies that had been published or indexed between January 2017 and the end of December 2017. Results Forty-eight new studies, describing the development of 56 COS, were included. There has been an increase in the number of studies clearly specifying the scope of the COS in terms of the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics. Public participation has continued to rise with over half (n = 27, 56%) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased, in particular participation from non-clinical research experts has risen from 32% (mean average in previous reviews) to 62% (n = 29). Input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) have all increased since the previous reviews. Conclusion This update included a pronounced increase in the number of new COS identified compared to the previous three updates. There was an improvement in the reporting of the scope, stakeholder participants and methods used. Furthermore, there has been an increase in participation from Australasia, Asia, South America and Africa. These advancements are reflective of the efforts made in recent years to raise awareness about the need for COS development and uptake, as well as developments in COS methodology.

Introduction Insufficient consideration of the outcomes measured in clinical trials, and other studies, is increasingly being addressed through the development and use of core outcome sets (COS). A COS is an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all trials for a specific clinical area [1]. The Core Outcome Measures in Effectiveness Trials (COMET) Initiative aims to collate and stimulate the development and application of COS, by maintaining a public repository of studies relevant to the development of COS (The COMET database, http://www.comet-initiative.org/studies/search). A survey of users of the database was undertaken to understand the reasons why people were searching the COMET database [2]. The most frequent users were people thinking about developing a COS to see whether a COS already exists in their area of interest to avoid any duplication, therefore emphasising the importance of keeping the database current.
A systematic review was conducted to initially populate the COMET database [3], and it has been subsequently updated annually to include all published COS up to, and including, 2016 [2,4,5]. The previous update showed that COS developers are adopting a more structured approach towards COS development, and public participation in COS development has also increased [5]. The review, however, highlighted a gap in the involvement of stakeholders from a wider range of geographical settings, in particular low and middle income countries (LMIC). This is important to increase the applicability of COS to global health and tackling the global burden of disease [6].
The database is therefore an integral resource to not only the development of COS, but also to the uptake of COS in research and in the avoidance of unnecessary duplication and waste of scarce resources. Eligible studies are added to the database as they are found; however, an additional annual update to the systematic review means that the database is kept up to date.
The aims of the current study were consistent with the previous update [5], specifically to: (i) update the systematic review in order to identify any further studies where a COS has been developed; (ii) to describe the methodological approaches taken in these studies, and (iii) to highlight areas for future COS development and improvement.

Systematic review update
The methods used in this updated review followed the same approach used in the original review [3] and previous updates [2,4,5]. The methods were described in detail previously [3,5]; a summary of methods is provided below.

Study selection
The inclusion and exclusion criteria were described in detail in the original systematic review [3]. Studies were eligible for inclusion if they had applied methodology for determining which outcome domains or outcomes should be measured in clinical trials or other forms of health research. As described previously, studies were eligible for inclusion if they reported the development of a COS, regardless of any restrictions by age, health condition or setting. Studies describing the development of a Patient Reported Outcome (PRO) COS (a core set of patientreported symptoms and health related quality of life domains) or Core Event Set (a core set of adverse events or complications) were eligible for inclusion in the review update. Studies describing the update of an existing COS were included as linked papers to the original COS.

Identification of relevant studies
In March 2018, MEDLINE via Ovid and SCOPUS were searched without language restrictions. The search identified studies that had been published or indexed since the previous systematic review update, between January 2017 and the end of December 2017. The multifaceted search strategy, developed for the original review using a combination of text words and index terms, was used for the current update [3] (S1 Table). Ovid included a new search option entitled 'MEDALL' that incorporates all previous search options and provides the most extensive coverage, including the 'in-process' citations as well as E-pub ahead of print citations. This search option was selected for this update of the systematic review in order to identify eligible studies at the earliest possible opportunity.
Hand searching was completed, including any studies that had been submitted directly to the COMET database, references cited in eligible studies, as well as those in ineligible studies that referred to or used a COS.

Selecting studies for inclusion in the review
As described previously [3,5], records from each database were combined and duplicates removed. Titles and abstracts were read to assess eligibility of studies for inclusion in the review (stage 1). Full texts of potentially relevant articles were obtained to assess for inclusion (stage 2). Two of five reviewers (EG, SG, NH, VS and KMS) independently checked the title and abstract of each citation. Citations were retained for further checking if agreement could not be reached. Two of the same five reviewers assessed each full paper for inclusion in the review. In cases of disagreement a third reviewer was consulted. The reasons for exclusion at this stage were documented for articles judged to be ineligible.

Checking for agreement between reviewers
One reviewer was new to this systematic review and had not been involved in previous updates (KMS). During each stage of the review process, agreement between KMS and the lead reviewer (EG) was assessed prior to independently assessing records. Agreement between all other reviewers had been checked in previous updates.
At full text stage. Of the records that had been excluded after reading the full text, 5% were assessed for correct exclusion. If any studies were identified as being incorrectly excluded at this stage, further checking was performed.

Data extraction
Described in full previously [3], data were extracted in relation to the study aim(s), health area, target population, interventions covered, methods of COS development and stakeholder groups involved. Data relating to the geographical locations of participants included in the development of COS were also extracted.

Data analysis and presentation of results
We report the review in accordance with PRISMA guidelines [7] (S1 File). We describe the studies narratively, and present the findings in text and tables. As in previous updates, we did not anticipate conducting any statistical analyses to combine the findings.

Description of studies
Following the removal of duplicates, 5140 records were identified in the database search. A total of 4626 records were excluded during the title and abstract stage, and a further 444 were excluded following the assessment of full text (Fig 1). Table 1 provides a summary of the reasons for exclusion of records at full text stage. Seventy records related to 45 new studies that met the inclusion criteria. In addition to the review search, three additional records were identified through database search alerts as being eligible for inclusion in the review. These three studies were not identified during the review search, as although they were published in 2017, they had not been indexed in the databases at the time we ran our search. A further 15 reports were identified by hand searching references of included studies. In total, 88 reports relating to 48 new studies describing the development of 56 COS were included for the first time in this update (S2 Table).

Included studies
Year of publication. The year of first publication of COS has been updated to include the 48 new studies included in this update (Fig 2). Of the 48 studies identified in this update, 36 were published in 2017, 11 were published in 2016 and one study was published in 2015.
Scope of core outcome sets. The scope of published COS studies is summarised in Table 2. This includes study aims, setting for intended use, population characteristics and intervention characteristics. Table 2 highlights that there has been an increase in the number of COS studies specifying the population (n = 43, 90%) and intervention (n = 48, 100%) characteristics of the newly developed COS. Fig 3 displays Table. Methods used to select outcomes. The methods used to develop the 48 new COS identified in the current review are presented in Table 3 alongside the methods used in the four previous systematic reviews. Table 3 highlights that the use of mixed methods to develop a COS has remained high (n = 35; 73%). The use of the Delphi technique has also remained high with 48% (n = 23) of studies using the technique in combination with other methods.
People involved in selecting outcomes. The list of stakeholders included in COS development (Table 4) has been updated to include the COS identified in this update. Of the 307 published COS studies, 273 (89%) provided information about the stakeholders who participated in the COS development. Clinical experts were included in 268 (98%) studies; this is in direct contrast to public representatives who were included in 89 (33%) studies. Public participation has continued to rise in recent years, with 56% (n = 27) of studies in the current review including input from members of the public. The rate of inclusion of all stakeholder groups has increased in comparison to the original review and previous review updates, with participation from non-clinical research experts, in particular, rising from 32% (n = 73) as the mean average in previous reviews to 62% (n = 29) in the current review.
As described previously [3,5], public representatives include patients, carers, health and social care service users and people from organisations who represent these groups. The degree of public participation within the development of the COS studies included in this updated review is described in Table 5. All of the 27 studies that reported including public participants provided some detail about their participation. Among Delphi studies that reported participation of both clinical experts and the public, levels of participation for public participants ranged from 6% [8] to 69% [9]. Table 6 lists the geographical locations, by continent, of the participants involved in developing the COS in this update and in the combined reviews, as reported by the included studies. In the current review, participant locations were provided in 41 of the 48 studies, however for the remaining seven studies, location information was only provided for the authors. The majority of COS were developed with the inclusion of participants located in Europe (n = 38;  93%) and North America (n = 28; 68%). However, input from participants located in Australasia (n = 17; 41%), Asia (n = 18; 44%), South America (n = 13; 32%) and Africa (n = 7; 17%) has increased since the previous reviews. The number of countries that have been involved in the development of a COS ranges from 1 to 76 (a median of 6). This review has identified an improvement in the reporting of COS scope (population and intervention characteristics), stakeholder participants and methods used. Core Outcome Set-STAndards for Reporting (COS-STAR) were published in 2016 [10]. This is the first update since the publication of the COS-STAR guidelines, and it is encouraging to see that ten of the 48 studies (21%) included in this update referenced these reporting guidelines. This provides a baseline with which to compare future updates, where we hope to see a continued increase in the number of studies reporting their work using these guidelines. Fourth annual update to a systematic review of core outcome sets for research

Discussion
The previous update highlighted an increase in the use of mixed methods, including the Delphi method, for COS development [5]. A recent survey of ongoing COS developers indicated that mixed methods, including Delphi, were commonly used in ongoing COS development, and there is an increased inclusion of public participants in ongoing studies [11]. As such, the percentage of new studies including public participation is expected to rise in future updates of this systematic review as ongoing studies are published. Participation of non-clinical research experts has increased, and as one of the criteria included in the recently published minimum standards for COS development [12], we would expect that this should continue to increase in new COS studies included in any further updates to this review. The Core Outcome Set-STAndards for Development (COS-STAD) recommendations will help users to assess whether a particular COS has been developed using a reasonable approach.

Survey only
In conclusion, we have completed the fourth update to a systematic review of COS, identifying COS published and indexed in 2017. COS methodology, and consensus development methodology more generally, have developed over recent years. This is reflected in the improvements seen throughout these updates, not only in the reporting of important COS details (such as scope), but in the methods chosen for development and the stakeholders who participate in the process. With the recent development and publication of guidelines and minimum standards for COS development [1,12], it is hoped that improvements in these areas will continue.
Supporting information S1 File. PRISMA Checklist. PRISMA checklist for content of a systematic review. (DOC)