Patient-Reported Outcome (PRO) Assessment in Clinical Trials: A Systematic Review of Guidance for Trial Protocol Writers

Background Evidence suggests there are inconsistencies in patient-reported outcome (PRO) assessment and reporting in clinical trials, which may limit the use of these data to inform patient care. For trials with a PRO endpoint, routine inclusion of key PRO information in the protocol may help improve trial conduct and the reporting and appraisal of PRO results; however, it is currently unclear exactly what PRO-specific information should be included. The aim of this review was to summarize the current PRO-specific guidance for clinical trial protocol developers. Methods and Findings We searched the MEDLINE, EMBASE, CINHAL and Cochrane Library databases (inception to February 2013) for PRO-specific guidance regarding trial protocol development. Further guidance documents were identified via Google, Google scholar, requests to members of the UK Clinical Research Collaboration registered clinical trials units and international experts. Two independent investigators undertook title/abstract screening, full text review and data extraction, with a third involved in the event of disagreement. 21,175 citations were screened and 54 met the inclusion criteria. Guidance documents were difficult to access: electronic database searches identified just 8 documents, with the remaining 46 sourced elsewhere (5 from citation tracking, 27 from hand searching, 7 from the grey literature review and 7 from experts). 162 unique PRO-specific protocol recommendations were extracted from included documents. A further 10 PRO recommendations were identified relating to supporting trial documentation. Only 5/162 (3%) recommendations appeared in ≥50% of guidance documents reviewed, indicating a lack of consistency. Conclusions PRO-specific protocol guidelines were difficult to access, lacked consistency and may be challenging to implement in practice. There is a need to develop easily accessible consensus-driven PRO protocol guidance. Guidance should be aimed at ensuring key PRO information is routinely included in appropriate trial protocols, in order to facilitate rigorous collection/reporting of PRO data, to effectively inform patient care.


Introduction
Patient-reported outcomes (PROs), including health-related quality of life (HRQL), symptoms such as pain or fatigue, and health utility, are increasingly assessed in clinical trials as a measure of effectiveness. [1][2][3] PRO trial data may be used to inform clinical care and decision-making, predict long-term outcomes and influence health-policy; but to do so, as with any trial outcome, they must be collected with rigor. Unfortunately, evidence shows that the quality of PRO data can be undermined in some trials by inconsistencies in data collection [4] and, in particular, by high rates of missing data [5]; this adversely affects the integrity and usefulness of such data in clinical practice.
To help ensure optimal PRO data collection, PRO-specific components should be considered during clinical trial design and clearly documented in the trial protocol. [6,7] The trial protocol is the cornerstone of a well-conducted trial, and should provide specific instruction on how to conduct all aspects of the study. [8] The protocol also allows external funding bodies, regulators, research ethics committees, journal editors, health care providers, systematic reviewers and policy makers to evaluate the design and methods. [6] Despite the importance of PROs, recent data suggests that some trial staff feel protocols provide little guidance regarding PRO-specific aspects of the trial, leading to ambiguity and the potential for significant inconsistency in the way PRO data are gathered, analysed, acted upon, and reported. [4,9,10].
The recent publication of the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guidance aims to promote the inclusion of important general methodological components in trial protocols [6]; however, it does not provide specific guidance related to PROs. It is currently unclear exactly what PRO-specific information should be included in trial protocols. The aim of this systematic review was to summarize current PRO-specific guidance for clinical trial protocol developers.

Ethics
This study received ethical approval from the University of Birmingham ethical review board (ERN_13-0047).
This review was conducted according to PRISMA guidelines (Checklist S1) [11] and a protocol is available. [12] Given the methodological focus of the review this was not registered with PROSPERO at http://www.crd.york.ac.uk/PROSPERO/.

Eligibility criteria
Papers were deemed eligible if they contained guidance (in the form of advice or formal recommendations) and/or a checklist on PRO related trial protocol content. As PROs is an 'umbrella term' papers including protocol guidance relating to specific types of PRO (such as HRQL) were also deemed eligible.

Information Sources and Search Strategy
The MEDLINE (Ovid), EMBASE, CINHAL and Cochrane Library databases were searched from inception to February 2013 (electronic search strategies are presented in full in Appendix S1). Other relevant articles were identified via two Internet search engines (Google and Google Scholar) using the key words 'Patient-Reported Outcomes' or 'Health-Related Quality of Life' in combination with 'Guidance', 'Guidelines' or 'Checklist'. Only the first 30 results (3 pages) of each search were reviewed as article relevance diminishes substantially with each page of results. [13] In addition, an international advisory group (MB, AG, JB, RMB and MK) were consulted via email to identify additional 'grey literature' directly relevant to the research question. Finally, PRO guidance/checklists and Standard Operating Procedures (SOPs) were requested from all members of the UK Clinical Research Collaboration registered clinical trials units (UK CRC-CTU) via email, with one follow up reminder. All citations were downloaded into Endnote software version X7 and duplicates deleted. Records were then screened by title/abstract before full-text articles/ documents were retrieved for eligibility evaluation. Remaining articles were subject to a citation search before a final hand-search of all reference lists.

Guidance selection
Papers and other guidance documents were deemed eligible if they provided guidance (advice or formal recommendations) and/ or a checklist describing key PRO-specific information that should be specified in clinical trial protocols. Non-English papers were screened by language specialists in the School of Health and Population Sciences, University of Birmingham. When more than one edition of a book was available, the latest edition was screened.

Data Extraction
Two reviewers (HDu and AG) independently screened the titles and abstracts of all citations. Full text versions of potentially eligible documents were independently reviewed (HDu and AG) with uncertainties resolved through discussion with a third investigator (MC/DK). Two investigators (MC and DK), independently and in duplicate, extracted both the publication details and all PRO-specific protocol recommendations from the final included documents. Data were extracted into Excel on prespecified forms to capture: publication source, year of publication, clinical and regulatory focus. Each recommendation was identified through independent review by MC and DK and iteratively added to the spreadsheet following checks for consistency.
Both explicit ('stated clearly and in detail, leaving no room for confusion or doubt' [14]) and implicit ('suggested though not directly expressed' [14]) recommendations for PRO related protocol content were extracted. Explicit recommendations specifically stated that an item of PRO information should be included in the trial protocol. Implicit recommendations described important PRO trial design issues, and were written in such a way as to suggest items should be included in the protocol -without specifically stating so.

Summary of Guidance
For ease of interpretation, PRO protocol recommendations were extracted and grouped as per SPIRIT guidance headings. [6] Duplicate recommendations within each of these sections were identified by MC and DK, and were merged where necessary following discussion with the international advisory group. The proportion of guidance documents associated with each recommendation was identified. To assess general trends in guidance over time, the proportion of guidance documents per recommendation was analysed retrospectively over 5 year time periods.

Selection of Guidance Documents
The literature search yielded 21,175 unique references. Following application of the inclusion criteria, 54 guidance documents [1, were included in the review (Figure 1). DK and MC independently included 53 of the 54 articles. One paper was initially excluded by MC but later included following discussion by the research team. [52] Of the 54 included documents, 8 were identified from searches of electronic databases, 5 from citation tracking, 27 from hand searching the reference lists of included articles, 7 from the grey literature review and 7 from expert recommendations.

Guidance Characteristics
Document characteristics are summarised in Table 1. The included materials dated from 1989 to 2013 and included 42 journal articles, 5 books and 7 organizational guideline documents, with the majority focused on HRQL/PRO assessment in cancer trials (n = 35, 64.8%) and written from a non-regulatory perspective (n = 44, 81.5%).

PRO Protocol Guidance
The included guidance documents contained 162 unique recommendations regarding PRO study design/conduct information that should be included in trial protocols. There was disagreement between data extractors on the inclusion of 10 items (6.2%), 6 of these items were included following discussion by the team, whilst 4 were felt to be duplicates of existing items. Of the 162 included items, 134 recommendations were explicit (e.g. 'All analyses should be clearly defined a priori in the research protocol' [41]), and 28 were implicit (e.g. '…investigators need to provide a rationale for the selection of a particular HRQL instrument' [46]). Protocol recommendations are summarised in Table 2 and presented in full in Appendix S2; in both places, they are grouped under SPIRIT guidance headings, with additional subheadings used to organise content. In addition, 10 PRO recommendations were related to other supporting trial documentation (Appendix S3).

Administrative information
There were n = 4 recommendations regarding trial administration, which centred around identifying the roles and responsibilities of PRO personnel and ranged from advocating involvement of the research nurse in PRO protocol development, to providing the contact details of the Quality of Life (QOL) sub-study coordinator where appropriate.

Introduction: Background, rationale, and objectives/ hypotheses
Eleven unique recommendations related to the inclusion of PROs in the introductory sections of the protocol. These focused on aspects surrounding: PRO specific background information (n = 2), for instance, the need to describe the PRO population of interest; specification of the PRO rationale (n = 5), for example, justifying the relevance of PRO assessment in the disease and population under investigation; or outlining the PRO hypothesis and objectives (n = 4).

Methods: Participants, interventions and outcomes
There were n = 25 unique recommendations within this section, focused on a number of areas, including: the PRO study setting (n = 1), the PRO-specific eligibility criteria (n = 3), the need to specify the PRO as an endpoint (n = 5), the PRO-specific sample size (n = 2) and blinding considerations (n = 2). Twelve different recommendations related to timing of the PRO assessment, ranging from: including PRO assessment timings in the main protocol assessment schedule and specifying time windows, to justifying timings according to the study research questions, length of recall of the questionnaire, the natural history of the disease under study, and any planned analysis.

Methods: assignment of interventions
There were no PRO-specific recommendations identified under this heading.

Methods: Data Collection, management and analysis
Ninety-four recommendations related to PRO-specific protocol guidance for data collection, management and analysis. These focused on data collection aspects including: identification/ description of the PRO instrument (n = 4), for instance, the need to outline the questionnaire domains and number of items; justifying the choice of instrument (n = 13), for example, the  importance of referencing the validity, reliability and responsiveness of the tool; detailing the data collection plan (n = 10), for example, stating who should administer the questionnaire; and describing the data collection/training guidelines (n = 16) (e.g. outlining the certification process for staff involved in PRO assessment) and plans to minimise missing data (n = 19) for example specifying who would check questionnaires for missing data. There were n = 7 recommendations concerning PRO specific quality assurance, ranging from the inclusion of guidance for data entry coding decisions regarding missing or ambiguous responses; to specifying procedures for a central PRO data monitoring system aimed at identifying and rectifying potential data collection problems. Finally, n = 13 recommendations focused on PRO analysis, specifically: the PRO-specific components of the statistical analysis plan (n = 13), for instance, the need to include an a priori estimation of expected change in PRO score; plans to address multiple hypothesis testing (n = 2), such as prespecification of sequence of testing; defining clinical significance (n = 6), for example, describing and justifying the minimal clinically important difference/change); and specifying methods to deal with missing PRO data (n = 4), for instance, defining proposed sensitivity analyses for imputation methods.

Methods: Monitoring
There were four recommendations regarding PRO specific trial monitoring, ranging from the need to define the role of the Data Monitoring Committee in relation to PROs, to the inclusion of a plan to manage PRO Alerts.

Ethics and Dissemination
There were n = 3 recommendations focused on PRO-specific consent information, for example, the need to include information for patients regarding who should be contacted for help with completing the PRO questionnaire. Two recommendations addressed PRO specific confidentiality issues, such as the need to specify whether QOL data will be used to influence patient management. Two recommendations focused on the need to include PRO-specific dissemination plans, through both peerreviewed scientific publication and direct participant contact.

Appendices
Fourteen recommendations focused on the inclusion of relevant PRO documents as protocol appendices, including: a copy of the PRO questionnaire(s), sample patient information and consent materials containing PRO information and a PRO-specific administration flow chart/checklist.

Other Trial Documentation
Ten recommendations focused on PRO information that should be included in protocol-related trial documents such as Standard Operating Procedures (SOPs), Case Report Forms (CRFs) or training manuals (Appendix S3).

Time trends and Common Recommendations
The availability of PRO-specific guidance over time is shown in Figure 2. The data suggest that there has been consistent publication of PRO protocol guidance, across all areas, over the last 25 years (Table 3). In addition, over 75% of recommendations extracted for this study have been available for at least 10 years.
Only 3% of recommendations appeared in more than half of the documents included in the study, highlighting a lack of consistency in the PRO guidance literature reviewed ( Table 2). These included (in order of frequency): the need to specify the timing of QOL assessment, the provision of PRO data collection guidelines and/or a training plan, specification (and justification) for the chosen PRO questionnaire, routine inclusion of a priori defined PRO analyses plans and specifying a named person within the trial with responsibility for overseeing QOL assessment.

Summary of Findings
Our review is the first to summarise the current PRO-specific guidance for clinical trial protocol developers. In total we identified 54 guidance documents [1,, which provided 162 recommendations regarding PRO-specific information that should be included in protocols containing a PRO endpoint.
Unfortunately, although PRO protocol guidance has been in existence for over 25 years, our findings suggest it remains difficult to implement in practice. First, with the exception of 8 publications [16,22,[39][40][41]59,62,63] sourced via electronic database searches, the guidance literature was particularly difficult to access. The remaining 46 documents, which provided more than half (56.7%) of all PRO protocol recommendations, were obtained via citation tracking, hand-searching reference lists of included articles, grey literature review and expert contact. It is unlikely that protocol developers would have the time or resources to carry out such a comprehensive search. As such, developers may be reliant on a small proportion of guidance documents available via easily accessible scientific databases. This is problematic, as these publications provide relatively little coverage of the current PRO protocol recommendations in circulation. As our findings show, recommendations are spread over a wide variety of sources, thus, over reliance on a small number of guidance documents may   If the sample size required for HRQL assessment is substantially less than for primary endpoint an unbiased strategy for selection of a subset of patients in whom HRQL will be assessed is possible provided that this strategy is clearly defined and justified in the protocol 4 (7.41)

PROs in relation to blinding
In a blinded study detail the use of PRO administration techniques to minimise the possibility of unblinding 1 (1.85) Research protocol must specify that interviewers be blind to intervention

Recommendation Number (%) of Guidance Documents
Specify that a named person at each centre (and/or centrally?) be nominated to take responsibility for admin, collection and checking of QoL forms -specify whether this is or is not the treating clinician

(50.00)
Provide instructions on how the patient should complete the form (e.g. without conferring with friends/relatives, all questions should be answered even if the patient feels them to be irrelevant)

(38.89)
Specify procedures for checking questionnaires/prevention of avoidable missing data, who will check form, and how will they deal with missing questionnaire(s) or items

(38.89)
Emphasise importance of good compliance/describe procedures to maximise compliance/minimise missing data 20 (37.04) Encourage staff to emphasise importance/rationale of QOL assessment to patients 13 (24.07) Provide reminders to staff to ensure baseline (and follow-up) questionnaires are completed 11 (20.37) Pre-specified procedures in protocol to avoid/minimise missing data 8 (14.81) Establish process for PRO assessment at withdrawal for patients that withdraw early from a study 7 (12.96) Establish process of how to follow patients who go''off-study''/"off treatment'' 7 (12.96) Provide interviewer training plan/format guidelines for PROs administered by interviewer (plus guidance on recording of interviews) 6 (11.11) Specify need to ensure backup data collection staff to cover leave/absence 6 (11.11) A plan should be included in protocol for systematically training and contacting local site personnel to ensure that they understand the content and importance of collecting PRO data. Ideally coordinated by a lead data manager who monitors patient adherence in real time and communicates with sites if patient non-adherent

(9.26)
Instruct site staff to routinely record (on clinical follow-up form) the reasons for any missing data 4 (7.41) Explain relevance and emphasise importance of QOL questions that might give rise to problems (e.g. sexual function questions) 4 (7.41) Detail/outline site-level incentives for good QOL submission rates/data quality and penalties for missing data (as appropriate) 4 (7.41) Encourage a sympathetic approach to patients who may be feeling particularly ill/show appreciation upon completion 4 (7.41)

Instruct site staff to give patients a full explanation about QoL assessment procedures 3 (5.56)
Instruct site staff to routinely record the source of PRO data in studies that allow proxies Specify procedures for a central PRO data monitoring system (aimed at identifying and rectifying potential data collection problems) 13 (24.07) Specify plan to monitor compliance 7 (12.96) Include data collection, data storage and data handling/transmission procedures 5 (9.26) Specify how an electronic PRO source will be maintained and how investigator will meet regulatory requirements and ensure data integrity and security

(3.70)
Specify procedure for monitoring adherence to timing windows' 2 (3.70) Include guidance for data entry on coding responses, missing responses or ambiguous responses 1 (1.85) Ensure plans for administration consistent with user manual 1 (1.85) All analyses should be clearly defined a priori in the protocol 29 (53.70) Specify ITT or per-protocol analysis. 6 (11.11) Pre-specify scoring 6 (11.11) State the anticipated response rate/effect size 4 (7.41) Specify conditions for positive outcome 3 (5.56) Ensure plans for scoring are consistent with those used in development 3 (5.56) Describe methods for scoring endpoints. Where possible, reference scoring manuals for summated scales from questionnaires (domain-specific and/or total),and methodological papers for composite endpoints (e.g. QTWiST)

(3.70)
Explain the assumptions of analyses 1 (1.85) Include an a priori estimation of expected change in PRO score

Pro-specific confidentiality procedures
Include guidance on discussing PRO confidentiality with patients (e.g. patients told how their questionnaires will be used) 9 (16.67) Specify whether QOL forms will be used to influence therapy or patient management (i.e. will clinician have knowledge) 4 (7.41) mean important PRO design considerations are overlooked. For example, even the two publications that provided most recommendations, Chassany [41] and Fairclough [58] (42 recommendations each, 24 shared), provided just 37.04% of the total in circulation. Second, developers wishing to use the guidance summarised in this review face the challenge of trying to incorporate a large number of recommendations into what is usually a rather limited amount of space within the protocol. For instance, we identified 94 unique recommendations concerning data collection, management and analysis, of which, 19 alone addressed minimising missing data. Tackling missing PRO data is clearly an important design consideration since it helps reduce bias and preserves statistical power [68], however, it may be unrealistic to expect protocol developers to incorporate all 19 recommendations within a study protocol.
It is important, therefore, for the scientific community to reach consensus on the essential PRO protocol content required to preserve trial integrity; and to provide guidance that is useful in practice. We advocate the development of consolidated, easily accessible and internationally endorsed consensus guidelines addressing this objective. Our review provides a useful starting point as it presents a comprehensive list of the PRO protocol guidance currently available, however, it remains unclear at this stage exactly which of the recommendations identified in this study should be incorporated into consolidated guidelines. A number of recommendations are supported by multiple sources and appear to be underpinned by a clear theoretical justification (for example, the need to provide a rationale for PRO measurement (recommended in 48.15% of guidance documents)), and may be promising candidates for inclusion. There were, however, a number of other recommendations that were less frequently cited, but still may have important implications for trial conduct, reporting and the quality of PRO results. For example, referencing the PRO instrument validity and reliability in the protocol (recommended in 25.93% of guidance documents) will help ensure that the psychometric properties of the PRO have been duly considered during the trial design and will help facilitate later reporting in accordance with the CONSORT-PRO extension. [69] In addition, only four publications provided guidance describing plans for the identification and management of PRO Alerts, that is: 'concerning levels of psychological distress or physical symptoms that may require an immediate response' [67]. However, evidence suggests that without clear, pre-specified, plans for the management of PRO Alerts, either in the trial protocol or supporting documentation, variation may occur in their management within/across trial sites risking co-intervention bias and suboptimal patient care. [67]. Consolidated PRO guidance for protocol developers is clearly necessary, however, our findings also show that guidelines should be developed using robust consensus methodology to ensure that the merits of all individual recommendations are carefully considered prior to selection/rejection. The definitive guidelines should aim to improve the quality of PRO trial design and reporting, resulting in more robust PRO trial data that will exert a greater influence on clinical practice and will provide an improved information base for future patients. Researchers should be supported in implementing the guidance through training and online resources. Furthermore, endorsement by funding bodies and Institutional Review Boards/Ethical Committees, who review the content of protocols, and journal editors, who are responsible for their publication, is important to ensure widespread adoption.

Strengths and Limitations
Our review has for the first time collated and summarised the existing PRO guidance available for protocol developers using systematic methods and multiple reviewers. A limitation of our approach is that the PRO item categorisation and indexing employed during our analysis is influenced by reviewer interpretation. Also, publications included in the study had to provide guidance on PRO-protocol content; however, such guidance was not always the main focus or aim of some of the included articles. Again, the interpretation of the reviewer may subtly alter the original meaning of the text drawn from such material. The use of independent dual data extraction by 2 investigators (with a third to mediate) sought to reduce these effects, however, they remain a legitimate concern. Furthermore no formal quality appraisal was undertaken given the diverse nature of the guidance documents. Relevant PRO guidance literature was difficult to source and appeared to be particularly poorly indexed. Whilst we employed a number of resources to comprehensively search the literature (including electronic databases, citation tracking and hand searching, internet search engines and expert contact) further PRO guidance probably exists that was not included in our study, such as disease specific documents citing EMA and FDA guidance included in our review. Consideration of alternative search strategies, such as 'citation pearl methodology' [70], may have further increased the comprehensiveness of our review. The International Society for Quality of Life Research Best Practice for PROs in Trials Taskforce (Co-chaired by authors, MC, MK and MB) is leading work in this area and would value the identification of further grey literature across a range of clinical settings on this  topic by interested readers to inform future guideline development.
A further limitation is that most of the international advisory group work in an oncology setting therefore are likely to have greater knowledge of grey literature in this field. However, oncologists have routinely used PROs for many years and as such we may anticipate more literature in this clinical area.

Conclusion
PRO-specific protocol guidance is difficult to access, lacks consistency and is unwieldy; with over 160 recommendations spread across 54 different publications. It is therefore extremely challenging to implement in practice. There is a need to develop easily accessible consolidated, and consensus-driven, PRO protocol guidelines. Guidance should aim to ensure key PRO information is routinely included in trial protocols with a PRO endpoint, in order to facilitate the rigorous collection and reporting of PRO data, thus maximising its capacity to effectively inform patient care.