A review of patient and carer participation and the use of qualitative research in the development of core outcome sets

Background To be meaningful, a core outcome set (COS) should be relevant to all stakeholders including patients and carers. This review aimed to explore the methods by which patients and carers have been included as participants in COS development exercises and, in particular, the use and reporting of qualitative methods. Methods In August 2015, a search of the Core Outcomes Measures in Effectiveness Trials (COMET) database was undertaken to identify papers involving patients and carers in COS development. Data were extracted to identify the data collection methods used in COS development, the number of health professionals, patients and carers participating in these, and the reported details of qualitative research undertaken. Results Fifty-nine papers reporting patient and carer participation were included in the review, ten of which reported using qualitative methods. Although patients and carers participated in outcome elicitation for inclusion in COS processes, health professionals tended to dominate the prioritisation exercises. Of the ten qualitative papers, only three were reported as a clear pre-designed part of a COS process. Qualitative data were collected using interviews, focus groups or a combination of these. None of the qualitative papers reported an underpinning methodological framework and details regarding data saturation, reflexivity and resource use associated with data collection were often poorly reported. Five papers reported difficulty in achieving a diverse sample of participants and two reported that a large and varied range of outcomes were often identified by participants making subsequent rating and ranking difficult. Conclusions Consideration of the best way to include patients and carers throughout the COS development process is needed. Additionally, further work is required to assess the potential role of qualitative methods in COS, to explore the knowledge produced by different qualitative data collection methods, and to evaluate the time and resources required to incorporate qualitative methods into COS development.


Methods
In August 2015, a search of the Core Outcomes Measures in Effectiveness Trials (COMET) database was undertaken to identify papers involving patients and carers in COS development. Data were extracted to identify the data collection methods used in COS development, the number of health professionals, patients and carers participating in these, and the reported details of qualitative research undertaken.

Results
Fifty-nine papers reporting patient and carer participation were included in the review, ten of which reported using qualitative methods. Although patients and carers participated in outcome elicitation for inclusion in COS processes, health professionals tended to dominate the prioritisation exercises. Of the ten qualitative papers, only three were reported as a clear pre-designed part of a COS process. Qualitative data were collected using interviews, focus groups or a combination of these. None of the qualitative papers reported an underpinning methodological framework and details regarding data saturation, reflexivity and resource use associated with data collection were often poorly reported. Five papers reported difficulty in achieving a diverse sample of participants and two reported that a large and varied range of outcomes were often identified by participants making subsequent rating and ranking difficult. PLOS

Introduction
Clinical trials in health care provide important evidence of the efficacy and safety of interventions and treatments [1], thereby informing future patient care, clinical guidelines and health policy [2,3]. The selection of outcomes to be measured and reported is an important part of the trial design process. Historically, the selection of outcomes has usually been based on the views of individual study teams informed by clinical and statistical considerations, and guided by regulatory considerations [4][5][6]. This is problematic since outcomes that matter to key stakeholders, including patients and carers, may be omitted. Furthermore, a wide variety of outcomes may be measured and reported across trials in the same health area "making it difficult or impossible to synthesise the results of different studies" [6] (p1). The difference in outcomes used across studies can also make it hard to detect reporting bias, where authors fail to report all findings because of the desire to report only positive results [7]. Greater uniformity in the reporting of outcomes and measures within a research area would help to facilitate the comparison and synthesis of research findings [7,8].
One potential solution to this problem is the use of core outcome sets (COS). A COS is an agreed standardised set of outcomes to be measured and reported as a minimum in all trials in a specific health related area [6]. If implemented, a COS may help to ensure that outcomes are relevant to a range of stakeholders and will provide consistent trial outcome data to inform evidence synthesis, clinical practice, shared-decision making and health policy [9,10]. Stakeholders can include patients, carers, patient representatives and patient advocates (reported as patients and carers from this point onwards), as well as healthcare professionals and decision makers including: funders, researchers, statisticians, health economists and pharmaceutical company representatives (reported as health professionals from this point onwards) [4,10]. Ultimately, it is the patients and those around them (carers) who benefit from improved healthcare and so it is important that their views and preferences are heard, particularly as there is evidence to suggest that patients' perspectives may differ from those of clinicians [11].
Despite the potential benefits of including a wide range of stakeholders in COS development, evidence to-date demonstrates that relatively few (18%) COS exercises include patients and carers and the reporting of the process does not always make it clear how they have participated [6]. Gargon et al. [6] recommended that further work is carried out to identify effective methods of eliciting patient and carer views in outcome set development. Data generated using qualitative methods can help to provide in-depth insight into patients' and carers' perspectives [12]. Therefore, qualitative methods could be well placed to identify outcomes that are important to patients and carers and to understand why that is. The aim of this review was therefore to: 1) review the methods by which patients and carers have been included as participants in COS development and 2) explore and describe the reported use of qualitative research with patients and carers. For the purpose of this review we were interested in participation, that is where patients and carers contribute data to COS development exercises as research participants, rather than involvement, where they contribute to the research process as an active research partner or advisor.

Data source
The Core Outcome Measures in Effectiveness Trials (COMET) Initiative "aims to bring together people interested in the development, reporting and promotion of COS, derived using rigorous consensus methods" [4]. The COMET Initiative database is an international repository of studies relevant to the development of COS, planned, ongoing and completed [13]. The COMET database was developed based on a systematic review using extensive searches [4] and the COMET Initiative conduct an annual search of the literature in order to keep the database up to date [13]. In addition, planned and ongoing COS exercises can be submitted to COMET by individuals or groups for inclusion in the database [13].

Search strategy
Given that the COMET database is a comprehensive source for COS development studies with the contents regularly updated, we limited our search to this one database.
COS Studies involving patient/carer involvement. The COMET database was searched on the 13 August 2015 using the following search categories: Carer organisations / Support group representatives, Charities, Conference participants, Consumers (caregivers), Consumers (patients), Families, Individuals with a known interest, Patient / Support group representatives, Service users, Guideline developers, with a study type of COS. Inclusion criteria were: papers developing COS with patients and carers as research participants. In addition, reference searches of the included papers were conducted.
Qualitative studies to inform COS development involving patients and carers. From these searches we identified all papers that described research using qualitative data collection methods (e.g. focus groups, interviews).

Data extraction and reporting
A data extraction pro-forma was developed, piloted and used to record study specific information: title, author, year, location of study (country), health area and data collection dates. For COS papers not using qualitative data collection methods we extracted the data collection methods used and the number of health professionals and patient and carer representatives participating in these. For papers reporting qualitative data collection methods we extracted; the stated qualitative methodological framework and rationale for this (please note: in qualitative research the methodological framework guiding research conduct, such as grounded theory, phenomenology or ethnography is distinguished from the methods used during conduct e.g. sampling, data collection, analytic approach) [14]; methods (sampling approach; data collection and analysis); sample characteristics; resource use (costs, resources and time involved); stated strengths and limitations and stated impact of the qualitative research). Some data items were informed by the CASP Qualitative Checklist [15].
The lead author (JJ) extracted data from all included papers and a second researcher (JMM, LLJ, MJC or TJHK) checked the extraction for accuracy on all papers reporting the use of qualitative methods. Any discrepancies were resolved via discussion within the research team. Data extraction were combined when a COS development exercise was reported across more than one paper. Data have been summarised descriptively. Recommendations have been made for the transparent reporting of qualitative research to inform COS development.

COS studies with patients and carers
Included papers. Of the 666 records on the COMET database, our initial search strategy returned 149 papers (Fig 1). Seventy-three of the returned studies were unpublished, and of the remaining 76, 24 were excluded because there were insufficient details to determine whether they were reporting a COS including patients or carers. Through reference searches of the included papers an additional seven papers [16][17][18][19][20][21][22] were identified as part of the included COS development exercises. Of these, four [16,19,21,22] were not archived on COMET at the point of the final search and three [17,18,20] papers were not returned in the initial search because they did not include patients or carers as participants; however, they described part of a COS pathway which included patient or carers reported in a separate publication.
Participants in the COS development. The 59 included papers represent a total of 34 COS development exercises covering 32 different health areas (Table 1). In 19 papers the number of participants was unclear.
The data collection methods used in COS development including patients and carers were; Delphi exercises (n = 14), consensus conferences/meetings (n = 20), surveys (n = 10), interviews (n = 6), focus groups (n = 6), nominal group techniques (n = 9) and other (n = 5). For the methods where participant numbers were reported fully, the mean percentage of patients and carers in each ranged from 20% to 86% (Table 2). Patients and carers formed the majority or all of the participants in surveys, interviews and focus groups. Health Professionals formed the majority of participants in nominal group techniques, Delphi exercises and consensus meetings. In all cases where qualitative data collection methods were utilised the primary aim was to identify outcomes of importance to the participants.

Qualitative studies to inform COS development involving patients and carers
Ten papers [16,19,21,22,24,30,32,34,35,37] reported using qualitative data collection methods to identify outcomes important to patients and carers (Table 3). However, of these only three were clearly reported as part of a COS development process [16,24,30]; five discussed outcomes with patients and carers but it was unclear whether the data were collected specifically with the intention to include them in a COS [21,32,34,35,37]; and two further studies were conducted for other primary research aims, such as exploring perceptions of access to care [19,22].
Methodological framework. Of the papers reporting a qualitative approach in the development of a COS [16, 19,    average of seven participants per group. A combination of interviews and focus groups were reported in two papers; Allard [16] who carried out ten individual interviews and 12 focus groups (97 participants) and Swigris [22] who conducted five individual interviews and three focus groups (15 participants). All studies were audio-recorded and transcribed verbatim, and all reported using topic or discussion guides. However, only four [16,19,30,34] provided details of the contents or derivation of these.  Data analysis. Only one of the papers [16] reported the use of a priori categories in the analysis, with most using exclusively inductive data-driven approaches. Seven papers [16, Sanderson [30] reported analysis based upon the principles of grounded theory. Four papers also referred to the use of the framework approach in the analytical description [16,30,34,35].
Triangulation. Data were analysed using multiple coders of and/or perspectives on the data in all of the included papers. Three [16,22,35] reported on the triangulation [74] of two different data collection methods (interviews and focus groups).
Reflexivity. Only two papers [19, 24] included any reflexive content. Potter [19] reported the use of a medically trained interviewer and reflected on the influence this may have had on the results. Arnold [24] discussed the use of an experienced facilitator with no prior knowledge of the condition under investigation, to avoid leading discussions.
Reported strengths and limitations. All studies acknowledged some strengths and limitations of their work. The main limitation noted in the included papers concerned recruitment. Five papers reported on the difficulty in recruiting an ethnically diverse and gender balanced sample [16,24,30,33,34]. Four discussed that they were unable to recruit participants to fulfil the desired sampling quota, for example, with not all categories of the disease/ investigation under investigation being included [16,24,30,37]. Sanderson [31] and Turk [37] reported that patients and carers identified a rather large and varied range of outcomes important to them, making subsequent rating and ranking very difficult. Potter [19] and Tierney [34] also highlighted the difficulty that participants may have when asked to recall their treatments and experience.
The participation of patients and carers in the core outcome set process was identified as a strength in 3 papers. Allard [16] stated that the differences between patients and parents were highlighted, and Sanderson [30] and Saketkoo [32] both reported that outcomes identified by patients and carers as important to them were not in current professional core outcome sets.
Resource use associated with qualitative methodology. Other than details on the length of the focus groups and interviews and reimbursements, very little information about the resources required to carry out qualitative data collection methods in COS development was reported in the included papers.
Planned and ongoing studies. As of 13 August 2015, 73 studies (S1 appendix) were registered on the COMET database as planned or ongoing COS development studies. Of these, all studies reported that they intended to include patients and carers as participants and 37 (52%) stated they wouldbe using qualitative methods as part of the COS pathway (S2 appendix). Of these 37, 21 planned to use both interviews and focus groups. Of the planned and ongoing studies five have published protocols [75][76][77][78][79]. Overall this supports the findings in Gorst's [80] recently updated systematic review which reports the increase in COS development studies and the increasing involvement of patients and carers.

Discussion
This study has described the different data collection methods used by COS developers in order to elicit patient and carer views on their preferred treatment outcomes, and has focused specifically on the reported use of methodology and methods within associated qualitative research. To our knowledge it is the first review to focus specifically on the use and reporting of qualitative research in COS development to date.
We have used the COMET database to identify COS studies. The database is based upon a systematic review of relevant studies and is regularly updated to ensure that new studies are added as they become available [4]. However, it is possible that there may be a lag time before studies are added to the database, or that COS relevant studies are not indexed or reported in ways that would mean they are captured within the database. We did perform reference searches of papers identified and found additional studies via this method. Therefore whilst the COMET database is an appropriate source of COS studies based on systematic review methodology, there is a possibility that there may be additional relevant studies not captured here. Furthermore, the focus of this review has been on participation in COS exercises. That is, we were concerned with the participation of patients and carers as research participants contributing data to the development of COSs. Of course patients and carers can also contribute to COS development via involvement in the research process as research partners and advisors, and in doing so influence the research and outputs. Our review has not focused on this involvement, which may not always be well reported and detailed in the outputs of COS studies. Further work to examine patient and carer involvement and its contribution to COS development would be useful. Recent work focused on patient and carer involvement demonstrates that this is a key issue that should be considered by researchers in the field [81].
To date, patients have participated in exercises that both identify relevant outcome domains for consideration in a COS, such as interviews, focus groups, and surveys, and also in the prioritisation and consensus methods that finalise a COS. However, the number and types of participants taking part in these, particularly patients and carers, were sometimes difficult to discern from the papers included in this review. Where participant numbers were reported, proportionately more patients participated in methods designed to identify relevant outcome domains for consideration, whilst Health Professionals were represented more than patients in prioritisation and consensus methods.
In this sample, qualitative and survey approaches have included patients more than Health Professionals to identify outcome domains. These methods may help incorporate patient perspectives that might otherwise go unheard, via the inclusion of patient preferred outcome domains in subsequent prioritisation and consensus methods. However, in prioritisation exercises (primarily Delphi methods in COS to date) where participant perspectives are aggregated and quantified then absolute numbers of participants from different stakeholder groups will influence outputs, particularly where views differ substantially between patients and carers and other stakeholder groups. Our observation that patients appear to be in the minority in these methods should be cause for reflection, although even if the number of participants from different stakeholder groups were balanced, there is some evidence to suggest that patients and carers rate many or all outcome domains as important in such exercises [31,37]. If this were the case then other stakeholder views may dominate as the outcome domains they do not value, on aggregate, will not be taken forward to the final COS. The inclusion of qualitative research to incorporate patient and carer perspectives as a precursor to group consensus approaches will not necessarily guarantee that patients' views are taken forward to the final COS.
In consensus meetings and conferences, one might expect Health Professionals to have more representation, as demonstrated here. The impact of patients and carers on outputs in these circumstances may well depend on the process and means by which their views are facilitated and accommodated, and on who takes part [81], as much as absolute numbers present versus other stakeholder groups.
Interestingly, one of the COS exercises we identified was expressly focused on understanding patient views and developing a patient core outcome set [30], building on the OMERACT work in rheumatoid arthritis. This approach does not rely on the integration of Health Professional and patient and carer views in a single COS exercise. Rather, it explicitly acknowledges that patient views are likely to be different to those of other stakeholders and therefore need specific consideration.
All of the qualitative research reviewed here was utilised as a means to ensure that patient and carer perspectives on outcome domains were accommodated in COS development processes. However, a key observation from this review is that some of this research (2 out of 10 papers) appear to have been designed and conducted for other primary research aims, not associated with COS development. Exceptions to this are Sanderson [30], Allard [16] and Arnold [24]. The remaining five papers [21, 32, 34, 35, 37] discussed outcomes with patients and carers but it is unclear whether the information was collected specifically with the intention to include it in a COS exercise. The availability of related research which is ready to feed into COS development, and the desire to include patient perspectives when specific COSfocused primary qualitative work is potentially time and resource intensive, could explain this. However, it does raise questions about the precise applicability of the underpinning qualitative research.
None of the papers stated a clear overarching qualitative methodological framework. Three [19, 24, 30] mentioned Grounded Theory, but only in descriptions of analytical approach. This may well be perfectly justifiable, for example if COS related qualitative research is being undertaken from an overtly generic qualitative research standpoint [82,83], though to date there does not appear to have been any reflection on this in the primary COS papers, or the COS methodological literature. There are longstanding debates within the qualitative methodological literature [82,83] about the use of methodological frameworks (e.g. grounded theory, phenomenology, ethnography), the coherency of underpinning research methods, and suitability for specific research aims. These have generally been in response to generic qualitative approaches that mix and match research methods, and which are more common in healthrelated research.
Reflection on methodology may help to further define the purpose and role of qualitative research in COS development. For example, if the primary aim is to understand and explain patients' perspectives as a means to influence COS development then an approach such as Grounded Theory may be appropriate; if one argues that the lived experience of illness and treatment is of chief concern as it describes the essence of disease experience which we are trying to improve via trial research then a phenomenological framework may be informative; or if we are simply wanting a pragmatic precursor that lists 'patient priorities' from descriptive accounts prior to COS prioritisation exercises, then forms of generic descriptive qualitative research may suffice. Whilst to our knowledge these issues have not been considered in COS research, suitable methodological frameworks have been discussed in analogous outcome related work in the patient reported outcome development literature concerning content validity [84]. It would seem that there are very clear parallels between this work and COS qualitative research.
Thinking about data collection methods specifically, this review demonstrates the use of interviews, focus groups, and a combination of these. Whilst there has been some recent attention to this in the methodological literature [85] empirical comparisons of the outputs and relative merits of different methods is needed in this developing area. Some of this work is currently underway (S3 appendix).
The qualitative papers reviewed note limitations around recruitment and diversity of samples. The reporting of sample characteristics is also variable. This is important when considering the generalisability (transferability in qualitative terms) of findings. Implicitly, COS development for trials assumes the generalisability of the final COS to varied populations and settings [81]. Conversely, qualitative approaches may not necessarily lay claim to widespread generalisability, acknowledging that patients' and carers' perspectives may vary, for example, in time, place and according to cultural factors. In addition to clear reporting of sample characteristics to aid judgements of transferability this issue needs acknowledgment. This has been addressed in some COS work, for example with work to culturally validate a patient core set amongst non-white patients in rheumatoid arthritis [86]. Outcomes and their importance in health research is often a difficult concept for patients and carers to understand [16,85]. The data collection methods used may have a direct impact on the depth of explanation of outcomes required [81]. For example, qualitative methods allow participants to talk about their experiences of illness without the need for an in-depth understanding of outcomes [87]. There is currently no guidance on how to discuss outcomes with patients and carers in qualitative research. The sharing of best practice and the publication of topic guides will aid future COS work [81,85].
There are some fairly simple reporting recommendations ( Table 4) that we would make for future qualitative COS work which include: clear reporting of aims in relation to the COS development; sampling and sample characteristics; data collection methods and derivation; use and reporting of the topic guide; data analysis; overt description of findings in the context of the COS; and reflection on strengths and limitations of approach. Beyond this we would suggest that there is a need for more fundamental consideration of the role of qualitative methods in COS and related methodological approaches, of the relative merits of different data collection approaches in terms of knowledge produced and time and resource requirements, as well as claims to generalisability.