Introduction

Every year, over 50 million people worldwide sustain traumatic brain injury (TBI). The global burden of TBI is enormous, dramatically impacting economic, societal, professional and personal welfare. Apart from the direct effects, which are often life-changing, there is growing evidence that TBI is a risk factor for development of major co-morbidities and neurodegenerative diseases [1]. As such, TBI has become a leading cause of death and disability globally across all ages, costing approximately $400 billion annually [2]. Despite decades of large-scale, government-funded research, effective treatments focused on neuroprotection and acceleration of recovery after TBI remain elusive. A recent systematic review of randomized controlled trials in acute moderate-to-severe TBI identified 191 completed trials, of which only 26 were judged as high-quality and with sufficient sample sizes. Of these, only three showed a statistically significant positive effect and none resulted in clearly discernible clinical benefit. The authors concluded that considerable investment of resources had resulted in very little clinically translatable evidence [3].

There is increasing concern that the use of insensitive, mis-matched, or psychometrically unproven clinical outcome assessment (COAs) measures in drug and device trials may be contributing to the failure to develop effective therapies for TBI [4–6]. Of the nearly 1000 COAs that have been employed in TBI research [2], including those that yield a quantitative score, most rely on subjective ratings made by either the examiner or respondent. This approach can increase error variance, introduce bias and compromise reproducibility. A second concern is that many COAs are not adequately responsive to important changes in function, including their ability to effectively measure response to treatment, which may predispose TBI clinical trials to false negative results [7]. Perhaps most importantly, very few COAs have been systematically vetted to assess their performance within well-defined contexts of use (COU), whether in TBI or other neurologic disorders [8]. The U.S. Food and Drug Administration (FDA) utilizes this concept in assessing whether a particular outcome measure is appropriate for use in a well-controlled clinical trial, as described in their guidance documents [8–13] on patient reported outcomes and drug development tools. At a minimum, the COU specifies, (1) the characteristics of the disease and the intended population to be tested and (2) the intended purpose of the COA (e.g., diagnostic categorization, outcome prediction). Each specific COU for a COA must meet a different set of performance criteria as a selected instrument may be valid and reliable when used for one purpose or population but not another [14].

The Glasgow Outcome Scale-Extended (GOSE) [15] is the most widely-used COA in TBI clinical studies [16] and in many TBI drug and device trials submitted to FDA. The scale was originally developed to assess global disability and recovery following TBI. Ratings are obtained through a structured interview and outcomes categorized into eight levels of function ranging from dead to good recovery. While the GOSE has demonstrated its utility in generally describing level of global functional recovery after TBI [16], its appropriateness for use in detecting therapeutic effectiveness has never be evaluated. This knowledge gap is amplified by the broad range of function subsumed by a single GOSE outcome category. For example, 20–30% of patients who fall into the highest level of the GOSE (ie, Level 8: Upper Good Recovery) have been shown to have persistent injury-related cognitive and psychological health problems when more specialized measures are administered [17]. On the other end of the severity spectrum, the GOSE’s Lower Severe Disability category (Level 3) lumps together patients who can live unsupervised at home for less than 8 hours a day with those whose only sign of consciousness is visual pursuit. These observations suggest that the GOSE may not be able to capture natural history or treatment-related changes at the upper and lower limits of the TBI continuum.

Several research funding agencies, including the National Institute of Neurological Disorders and Stroke (NINDS), Department of Defense (DOD), and National Institute on Disability, Independent Living and Rehabilitation Research (NIDILRR) prioritized the need to identify the most robust TBI COAs in the existing pool of outcome assessment tools. To advance this goal, the NINDS launched the Common Data Element (CDE) Project, which aims to improve the precision, reproducibility and cross-study comparability of data elements, including COAs. CDEs have now been established for 23 different neurologic disorders. The first iteration of outcome CDEs for TBI was published in 2010 [18], followed by updated recommendations in 2013 [19].

While the development of TBI CDEs has been an important step toward narrowing the field of potential COAs that may be selected for use in clinical trials and observational studies, a key shortcoming is that the CDEs included in the 2013 update were selected by expert consensus, not through evidentiary review. Many of the major TBI research funding agencies now mandate that grantees incorporate the recommended CDEs into their research design. Thus, investigators should have some assurance that the TBI CDEs will perform adequately, and at least as well as other COAs that have not been designated as CDEs.

In 2014, the U.S. Department of Defense established the TBI Endpoint Development (TED) initiative (#W81XWH-14-2-0176, PI: G. Manley) to identify and validate COAs and biomarkers that could serve as potential FDA Drug Development Tools (DDTs) to more accurately detect injury sequelae, identify patient subpopulations most likely to benefit from therapeutic interventions and detect changes attributable to efficacious therapies. In response to Priority #4 of the TED initiative, which called for the development of processes to assess the strength of clinical outcome measures, the authors were awarded a seed grant to design, build, and pilot-test an evidence-based assessment platform to enable the efficient, transparent, and systematic grading of TBI COAs for use in research and clinical practice. This effort culminated in the development of the Evidence-Based Clinical Outcome Assessment Platform (“EB-COP”), a semi-automated tool that guides an investigator through key questions designed to ascertain if the COA fits the intended COU, and presents the supporting evidence base for this assessment. The EB-COP extends the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) [20], a methodology for critically appraising and selecting appropriate outcome measurement instruments, by incorporating distinct sets of quality indicators (QIs) that are specific to different purposes of use. This feature helps the user determine the contexts for which a particular COA is appropriate. In this report, we describe the EB-COP framework, discuss how the platform grades COAs for each purpose of use and review the results of EB-COP pilot testing, which was conducted on the GOSE using the Qualtrics software platform.

Materials and methods

We convened a multidisciplinary team of experts to serve as the EB-COP Design Team. The team was composed of two TBI COA content experts who also co-led the Outcomes Core for the TED initiative (JG, MM), an evidence-based medicine methodologist (MJA), two database developers (SM, TSDG) from the American Academy of Neurology (AAN), a medical librarian (PE) from the Mayo Clinic, a regulatory affairs advisor (AR), and two TED post-doctoral fellows (AC, MB). A glossary of key terms and abbreviations is included in S1 File.

The conceptual framework adopted by the EB-COP Design Team was informed by the AAN’s Clinical Practice Guideline Process Manual [21], which follows recommendations proposed in two Institute of Medicine reports- “Finding What Works in Health Care: Standards for Systematic Reviews,” [22] and “Clinical Practice Guidelines We Can Trust,” [23] and guidance provided by the FDA entitled, “Roadmap to Patient-Focused Outcome Measurement in Clinical Trials [8].” In addition, the Design Team agreed that the EB-COP should adhere to three foundational principles:

• COAs should be assessed within distinct COUs
• Only studies meeting accepted standards for high-quality methodology [20, 22, 24, 25] should qualify for evidentiary review
• The review process should be systematic and transparent and lead to a clearly stated recommendation.

The criteria used to identify high-quality studies (i.e., those with low risk of bias) were informed by existing data quality standards, including the AAN’s Guideline Development Process Manual [21], COSMIN Checklist [26], Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) [27] and Quality in Prognostic Studies (QUIP) [28].

We selected six of the most common COA applications (i.e., purposes of use) for inclusion in the EB-COP: 1) diagnosis, 2) identification of TBI sequelae, 3) stratification of distinct TBI subgroups, 4) outcome prediction, 5) sensitivity to natural history changes, and 6) detection of treatment effects. After consulting the peer-reviewed literature [29–35] and additional on-line resources [8–10, 36, 37], 36 different QIs were identified as relevant to clinical outcome assessment. Next, we employed a two-part modified Delphi consensus process [38] to further narrow the QI pool. Prior to voting, panel members were furnished with a narrative description of each QI and summaries of previously-published recommendations concerning their use (e.g., FDA’s COA Qualification Guidelines and Clinical Trial Roadmap). The first part of the Delphi process was conducted to identify those QIs that were believed to be fundamental to all 6 purposes of use (n = 8). It was determined a-priori, that if any of the fundamental QIs were missing or inadequate, this would be considered a “fatal flaw” and the measure could not be recommended for any of the 6 purposes of use. For example, if an administration and scoring manual was not available for a particular COA, it would not be possible to ensure that the measure was being used in a standardized and reliable manner, resulting in low confidence in the measure across all applications. After the fundamental QIs were selected, a second modified Delphi poll was conducted to select QIs that were considered mandatory for each of the six purposes of use (n = 28). Depending on the number of missing purpose-specific QIs, the COA was either tentatively recommended or not currently recommended in view of the need for additional psychometric testing. A recommendation against use of the measure was levied when any of the mandatory QIs were found to be inadequate for the intended purpose. At the end of each round, participants’ votes and comments were synthesized and made available to the panel prior to voting in the next round. Consensus was defined as at least 80% agreement within three rounds of voting. A full transcript of the voting, including the instructions, voters’ responses and comments and resulting recommendations and actions, was maintained to ensure transparency in the process.

To facilitate the COA review process, the EB-COP was subsequently incorporated into a semi-automated on-line platform using Qualtrics software to facilitate data extraction and synthesis. The accompanying EB-COP Manual of Operating Procedures (MOP) (see S2 File) provides links to the on-line platform, instructs the user in how to navigate the EB-COP platform, defines every quality indicator included in the EB-COP platform, provides examples of key psychometric constructs and includes additional supporting materials to assist the user in answering the evidence questions. The software’s automation features aid the user in building the evidence question, guide the evidence review process using branch logic, display the mandatory quality indicators required for each purpose of use, produce an evidence summary table and grade the COA in relation to the COU.

Results

The EB-COP review process is comprised of six steps as shown in Fig 1. This procedure is intended to help ensure that the review process is systematic, transparent and considers only high-quality evidence.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Six-step EB-COP systematic review process.

https://doi.org/10.1371/journal.pone.0242811.g001

Step II: Assess fundamental quality indicators (QI)

Assessment of the COA begins by vetting the measure against a set of eight “fundamental” QIs that were selected by the Design Team through the modified Delphi process. These are: 1) the availability of documentation on how the COA was developed, 2) specification of the target population, 3) description of the intended COI, 4) information on content validity and 5) face validity, 6) clarity of the administration and scoring instructions, 7) extent of missing data in the population sampled, and 8) feasibility of application within the COU (e.g., cost, completion time, alternate forms). Table 1 shows the criteria used to assess each of the fundamental QIs. Source documents required for evaluation of the fundamental QIs include the COA’s administration and scoring manual and other materials that describe its development and use.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Criteria for assessment of fundamental quality indicators (QIs).

https://doi.org/10.1371/journal.pone.0242811.t001

The EB-COP’s on-line data entry platform defines each fundamental QI and prompts the user to respond to pre-programmed questions that determine whether the criteria for each QI has been met. The EB-COP MOP provides the user with additional guidance for responding to the questions. If the criteria for all eight fundamental QIs are met, the review process proceeds to the next step. If any one of the fundamental QIs fails, the user is advised that the COA is not appropriate for further review.

Step IV: Assess the relevance and methodologic quality of studies investigating the psychometric properties of the COA

Step IV is composed of four parts. In Step IVa, the abstracts retrieved in Step III are screened by the investigators to determine which articles should be acquired for full-text review. This is followed by a full-text review of the surviving studies to confirm the study’s relevance and sample size (Step IVb). The strength of the study design and generalizability of the results to the target population are assessed in Step IVc. Finally, in Step IVd, the mandatory QIs associated with the COA’s selected purpose(s) of use are examined based on the data extracted from the literature in Steps IVa-IVc. Step IV is designed to ensure that the final COA recommendations rely only on methodologically sound studies that can effectively evaluate the designated QIs. To control for bias, standardized data extraction forms are used to guide users through each of the steps described above. To help ensure reproducibility of the findings, two independent reviewers are required to screen the abstracts, extract data from the full-text articles, classify the strength of the study design and reconcile any discrepancies detected. After all of the harmonized data are entered into the on-line platform, the Qualtrics software automatically determines the final COA grade, based on the number of the number of mandatory quality criteria met.

Step IVa: Abstract review to assess study relevance and sample size.

Step IVa essentially serves as a high-level filter, eliminating articles that are unlikely to inform the evaluation of the COA due to weaknesses in the study design. The aims of Step IVa are to capture the abstracts of all published studies that may be relevant to the defined COU (as described in Step I), and ensure that extracted studies have a sufficient sample size to assess the performance of the COA for the selected purpose(s) of use. The relevance of the study is determined by reviewing the information provided in the abstract about the: 1) population studied (e.g., do the subjects have the specified condition of interest?), 2) age group (e.g., is the target population adult or pediatric?) and 3) measurement properties studied (e.g., are the indicators studied included in the EB-COP QI list?). The user also determines if the study enrolled the minimum number of subjects required to achieve the stated aims. If the abstract indicates that the study does not meet these criteria, it is excluded, but if it lacks information or is ambiguous, the study is retained for full-text review. Upon completion of this step, the EB-COP provides a summary of the user’s responses. Step IVa essentially serves as a high-level filter, eliminating articles that are unlikely to inform the evaluation of the COA due to low relevance to the COU or insufficient sample size.

Step IVb: Full-text review to confirm study relevance and sample size.

After both reviewers independently complete the abstract review and reconcile any disagreements about inclusion or exclusion, the full-text articles are retrieved by the user to confirm that the abstract describes an original research study and meets the requirements for relevance and sample size. Upon completion of this step, the EB-COP provides a summary of the user’s responses and indicates whether the requirements for proceeding to Step IVc have been met.

Step IVc: Confirmation of generalizability of results and overall methodological quality.

Step IVc is comprised of a series of questions designed to confirm that: 1) the design of the study under review provides adequate protection against risk of bias, and 2) the results can be generalized to the target population. These criteria follow guidelines proposed by the AAN [21], COSMIN [26], QUADAS-2 [27] and QUIP [28]. The criteria are examined through a series of queries about the recruitment method used (e.g., consecutive series or convenience sample), range of the target population sampled (i.e., narrow or broad spectrum), whether the COA was administered and scored in accord with standardized procedures and the percentage of missing data (e.g., </> 20%). If the study meets the requirements in this area, the user is furnished with a summary of the responses and the review advances to the next step.

Step IVd: Assessment of the adequacy of the QIs mandated for each purpose of use.

In step IVd, the user assesses the quality of the study investigating the QIs linked to the selected purpose of use. To ensure that only high-quality COA evidence is included in the data analysis and synthesis, only studies that meet all of the mandatory QI criteria are considered in the grading process and in the final recommendation. Fig 2 shows the QIs that have been assigned to each purpose of use. The full list of QIs included in Steps IVa through IVd is shown in the EB-COP MOP (S2 File).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. COA quality indicators organized by purpose of use and level of obligation.

QIs were selected for each of the six COA applications. Mandatory QIs are shaded green, non-mandatory QIs (i.e., should be assessed if investigated) are shaded purple and QIs that are not relevant to a specific purpose are shaded red.

https://doi.org/10.1371/journal.pone.0242811.g002

Step VI: Synthesize the evidence and develop a recommendation for the COA

In Step VI, the user reviews the Evidence Summary table, which includes all the data extracted from the high-quality studies reviewed in Step IV. Acceptability cut-offs for all the QIs (Table 2) are based on pre-specified, commonly accepted cut-off values [34, 35, 37, 41]. The Evidence Summary table indicates whether each mandatory purpose-specific QI is ‘adequate’ (i.e., meets threshold), ‘inadequate’ (i.e., fails to meet threshold), or ‘not determined’ (i.e., not investigated in a high-quality study). Based on these parameters, one of the following four recommendations is made:

• Grade I—Recommended without Reservation: For the specified purpose of use, all mandatory QIs were found to be “adequate.”
• Grade II—Recommended with Reservation(s): For the specified purpose of use, all mandatory QIs were found to be “adequate,” except for no more than 2 mandatory QIs that were found to be “undetermined.” No mandatory QIs were found to be inadequate. Additional research is indicated.
• Grade III—Not Currently Recommended: For the specified purpose of use, all mandatory QIs were found to be “adequate,” however more than 2 mandatory QIs were found to be “undetermined.” No mandatory QIs were found to be inadequate. Additional research is indicated.
• Grade IV—Not Recommended: For the specified purpose of use, one or more mandatory QIs were found to be “inadequate.”

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Quality indicators for grading and development of recommendations for COAs.

https://doi.org/10.1371/journal.pone.0242811.t002

The EB-COP does not stipulate which statistical tests should be used to calculate the cut-off values, but some general considerations are provided in the EB-COP MOP. The final COA grade and recommendation is based on the number of mandatory QIs that meet the pre-specified cut-off values as shown in Table 2. If any of the mandatory QIs is found to be ‘inadequate,’ the lowest grade (i.e., “Recommended Against”) is assigned.

EB-COP pilot-testing

We pilot-tested the EB-COP on the GOSE because it is among the most commonly used COAs in TBI research. The primary aim of the pilot-testing was to assess the utility of the EB-COP review process and the feasibility of the data entry platform. Pilot-testing was performed by two post-doctoral fellows who first participated in a primer on evidentiary methods conducted by the Design Team’s evidence-based medicine methodologist (MJA). Although we expected the results to be scientifically informative, our primary aim was to assess the EB-COP’s capacity to interrogate the GOSE within a well-defined COU. We chose detection of treatment effects as the purpose of use, given the GOSE’s widespread use in clinical trials. The results of each step are summarized below.

Step I: Determination of the GOSE evidence question.

In framing the evidence question concerning the GOSE, we selected the following parameters from the EB-COP’s drop-down menus: population = adults with moderate to severe TBI; injury duration = subacute; cause of injury = blunt trauma; Purpose = detection of treatment effects and concept of interest = global outcome. After these parameters (shown in bold in Fig 3) were entered, the EB-COP automatically generated the evidence question as shown below in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Screenshot of the evidence question generated by the EB-COP.

The evidence question describes the context of use for which the COA will be tested. Contextual factors include descriptive characteristics of the sample, the chronicity of the condition, the purpose for which the COA has been selected and the concept of interest the COA purports to measure.

https://doi.org/10.1371/journal.pone.0242811.g003

Step II. Assessment of fundamental QIs.

To assess the fundamental QIs, we identified three studies [15, 42, 43], a review article [16] and an on-line curated COA database [35] that included information about the development of the GOSE. Using these sources, we were able to assess each of the fundamental QIs. Based on the EB-COP criteria, all eight QIs were judged to be adequate, allowing the review to progress to the next step.

Step III: Results of the systematic literature review.

The Design Team’s reference librarian (PE) performed a literature search using the terms, “traumatic brain injury,” “Glasgow Outcome Scale” AND “Glasgow Outcome Scale—Extended.” The search was refined by employing the COSMIN filter [39] that was modified to capture TBI studies. We searched OVID MEDLINE, OVID, EMBASE, PsycINFO, EBSCO, CINAHL and SCOPUS and did not restrict the search to studies written in English. Dissertations, book chapters, conference proceedings and case studies were excluded. A total of 2,849 unique abstracts were identified for review, enabling transition to Step IV.

Step IV: Relevance and methodologic quality of identified studies investigating the GOSE.

Ninety abstracts were retained for full-text review after reconciliation by the two independent reviewers (Step IVa). Approximately 6% (170) of the abstracts required reconciliation. Abstracts that could not be definitively excluded were retained for full-text review. By far, the most common reason for exclusion was that the study clearly used the COA to test an intervention, rather than investigating its psychometric properties. During the full-text review (Step IVb), the reviewers excluded 79 studies for relevance because they used (versus tested) the GOSE as an outcome measure. In Step IVc, nine more studies were excluded, either because the study design had increased risk of bias or low probability of generalization. Finally, one additional study was excluded in Step IVd because the study did not include any of the QIs required for detection of treatment effects. Fig 4 shows the flow diagram for the abstract and full-text review process. The reviewers were able to extract and evaluate the data elements included in all four parts of Step IV in a systematic and transparent manner.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Flow diagram for the GOSE abstract and full-text review process.

The EB-COP uses a two-step review process which begins with the abstract and progresses to full-text review. The arrows and numbers indicate the reasons articles were excluded from the review, and the number of articles that were excluded for each reason, respectively.

https://doi.org/10.1371/journal.pone.0242811.g004

Steps V and VI: Evidence analysis and recommendation for use of the GOSE in detecting treatment effects.

The data extracted in Step IV were exported into an Excel spreadsheet by the Database Administrator and an Evidence Summary table was returned to the reviewers, indicating successful execution of Step V. Evidence analysis (Step VI) indicated that only one study met all of the EB-COP criteria required to complete the grading process. This study, published in 2003 by Pettigrew and colleagues [44], investigated two of the 16 QIs required for detection of treatment effects, test-retest and inter-rater reliability. Test-retest reliability could not be assessed due to insufficient sample size in the target population. Inter-rater reliability was assessed in a sample of 56 adults, including 25 with moderate or severe TBI. Subjects were interviewed by telephone by one rater and then face-to-face by a different rater up to one month later. The reported weighted kappa value, representing the level of agreement between the in-person and phone-based GOSE ratings, exceeded the EB-COP’s pre-specified threshold of 0.70 (K_w = 0.92, CI = 0.51–1.00) as shown in Table 2, however, because of the wide confidence interval, the results were considered inconclusive.

It is important to note that two highly cited papers, the original study of the GOSE structured interview by Wilson and colleagues [15], which assessed reliability, and a study by Levin and colleagues [45] that investigated the validity and sensitivity of the GOSE, were excluded by the EB-COP because their samples consisted primarily of subjects with mild TBI.

In the final analysis, while the EB-COP did not find any “inadequate” GOSE QIs, more than two QIs required for detection of treatment effects were characterized as “undetermined.” No purpose-specific GOSE data were available to assess internal consistency, test-retest reliability (i.e., cross-section, longitudinal), longitudinal inter-rater reliability, validity (i.e., construct, predictive, convergent, cross-cultural), responsiveness (i.e., minimum clinically important difference) or floor and ceiling effects. Consequently, a Grade III rating was assigned- “Not currently recommended” for detecting treatment effects on global function in adult patients with moderate to severe TBI assessed in the subacute setting.

Implications for clinical trial design. An investigator interested in using the GOSE score at 6 months post-injury as the primary endpoint in a clinical trial designed to test the benefits of a novel drug administered within 72 hours of sustaining severe traumatic brain injury would want to be confident that this measure is appropriate for detecting improvements in function that exceed those expected based on knowledge of the natural history of recovery. To achieve this, the investigator should be reasonably sure that the GOSE:

• Is comprised of items that are inter-correlated, independent of each other and behave lawfully (internal consistency, local independence, monotonicity, invariant item ordering);
• Captures a single underlying construct (unidimensionality);
• Can be reliably administered (interrater and test-retest reliability);
• Measures behaviors that are tied to functional ability, behaves like other measures of functional ability and predicts future functional status (construct, convergent and predictive validity);
• Is robust to cultural differences in the target population (cross-cultural validity)
• Detects “true” change (i.e., beyond measurement error and spontaneous recovery) that is clinically relevant across low and high performers (responsiveness, minimum clinically important difference, ceiling/floor effects).

Failure of a measure to demonstrate any one of these features can jeopardize accurate interpretation of the study findings and lead to either over-estimation or under-estimation of the effects of the intervention. The EB-COP guides the investigator through a systematic process that can be used to identify which COAs are best-suited to accurately detect treatment effects. The failure of a measure to meet the mandatory requirements does not mean that the measure should be scrapped, rather it reveals weaknesses that require further investigation before the measure can be confidently used for the desired purpose.

Conclusion.

The pilot test results demonstrated that the EB-COP platform is a feasible method of assessing COAs within a specific COU. Guided by a Qualtrics-driven software platform underpinned by branch logic, two independent reviewers systematically and transparently navigated a comprehensive evidence-based COA review of the GOSE. Pilot-testing also provided an opportunity to re-organize the order in which some of the questions are presented and to clarify language ambiguities. These changes have been incorporated into the current version of the EB-COP.

Discussion

The success of a clinical trial hinges, in part, on the performance of the selected COAs. Relying on existing literature on psychometric standards, well-established evidence-based methodology developed by the AAN and published FDA guidance, we developed a six-step evidence-based method (i.e., EB-COP) to evaluate COA performance within well-defined COUs, and operationalized this procedure on a semi-automated Qualtrics-driven software platform. To our knowledge, this combination of features is not available in any existing systematic review platform.

The EB-COP is distinct from the COSMIN guidelines in a number of important ways. First, it extends COSMIN by incorporating QIs that are specific to, and required for, different COA applications. This is critically important as a COA may perform well for one purpose or population, and poorly for another. As an illustration of this point, EB-COP pilot-testing suggested that the GOSE may not be suitable for detecting treatment effects in patients with moderate to severe TBI because most of the mandatory QIs for this purpose of use have not been investigated. This finding is not surprising since the GOSE was originally developed to assess global functional outcome after brain injury, not to detect treatment effects. Second, a modified Delphi consensus process involving a diverse stakeholder group was used to guide development of the EB-COP. The EB-COP’s six-tiered framework, the criteria for grading study methodology and the purpose-specific QIs were all subjected to an iterative consensus-building process. Third, the EB-COP’s grading system only considers evidence associated with moderate risk of bias or lower in accord with the AAN evidence classification scheme [21], ensuring that COA recommendations are driven only by high-quality studies. Fourth, the cloud-based Qualtrics software platform [46], used by more than 1,600 colleges and universities worldwide, will foster consistency across users and enable adaptation for use in a wide variety of clinical populations.

The EB-COP can be viewed as a decision-making tool that minimizes historical, subjective and idiosyncratic biases that influence COA selection in both clinical practice and research trials. We wish to emphasize that the EB-COP is not intended to be the only driver in determining which COAs should be selected for a specific TBI research application. Other factors, both practical and scientific, will also play a role. For example, investigators are likely to encounter situations in which a particular COA is not recommended for a specific purpose of use because of missing or inadequate QIs, but a suitable alternative is not available. Under these circumstances, some judgement will be required to determine whether the level of risk associated with the measure is acceptable, based on the number of missing or inadequate QIs, other COAs included in the assessment battery, the study aims and other considerations.

While our original focus was on TBI COAs, the EB-COP can be modified to address other populations and expanded to incorporate additional applications. From a regulatory standpoint, the EB-COP can sharpen the process used to determine the readiness of a COA for use as a clinical endpoint in drug development. This is consistent with the Agency’s efforts to encourage “community collaboration in the development of COAs for unmet measurement needs” [12] and supports its mission to develop efficient publicly-available medical device development tools for widespread use in device development programs [13]. For investigators, the EB-COP will help narrow the dense and confusing landscape of TBI COAs by identifying the most psychometrically-robust measures and the contexts in which they are most effective. This has direct relevance to the existing TBI CDEs for outcome measures. A recent critical review of CDE harmonization across three large multicenter TBI studies found substantial overlap between CDEs recommended for use during acute hospitalization and acute rehabilitation [47]. The EB-COP is well-equipped to help investigators optimize COA selection by assessing suitability for use within specific settings and phases of recovery. Finally, for clinicians, the EB-COP may improve diagnostic precision, symptom detection, prognostic accuracy and treatment monitoring through its inclusion of purpose-specific QIs.

The EB-COP has some limitations. Although selection of the QIs used in the EB-COP review process was guided by existing psychometric standards and approved by COA content experts from scientific, regulatory and clinical sectors using a formal consensus process, it is important to show that the grading scheme has criterion validity. This is typically accomplished by comparing a new measure to a “gold standard.” While no clear-cut gold standard exists, one approach might be to compare performance ratings generated for the same COA by the EB-COP and the COSMIN Risk of Bias Checklist [48]. A second issue is that while the EB-COP was designed for use across a variety of COA types (e.g. patient, clinician, observer-reported or performance-based), pilot-testing was limited to the GOSE, a clinician-reported COA. A separate unpublished study used the EB-COP to assess the appropriateness of the Rivermead Post-Concussion Questionnaire (RPQ) [49], a patient-reported COA, for three purposes of use- detecting post-concussive symptoms, stratifying PCS subtypes and monitoring resolution of PCS (see S4 File). Results demonstrated the feasibility of the EB-COP for assessment of the RPQ and were very similar to the findings for the GOSE. The review did not show that the RPQ was inadequate for the purposes specified, rather, it demonstrated that most of the QIs that are critical to these three applications have not been investigated. Additional field-testing is warranted to assess the generalizability of the EB-COP to other types of TBI COAs, including those intended for use in other health conditions. Third, the EB-COP is not designed to assess COAs that serve as screening measures. Screening is a distinct purpose of use that requires its own set of QIs, some of which differ from those currently included in the EB-COP. Future iterations of the EB-COP can be adapted to incorporate screening and other purposes of use. Fourth, budgetary constraints prevented us from conducting cognitive testing on the questions that guide the user through the EB-COP review. We intend to make the EB-COP publicly accessible so there should be ample opportunity to obtain user feedback for future updates. We envision the EB-COP as a “living” tool that can be modified and expanded as the field progresses. The EB-COP software platform can also be enhanced further by incorporating additional semi-automation technologies, including narrative search tools, to further streamline data extraction and synthesis. Lastly, the TBI literature, in general, suffers from a lack of well-designed psychometric studies on TBI outcome measures, despite the fact that they are integral to almost every type of TBI study. The utility of the EB-COP is dependent, in part, on the breadth and depth of the psychometric literature. The findings discussed above on the GOSE and RPQ illustrate this point. A better understanding of how COAs behave within different contexts is essential to improving precision in TBI research. The EB-COP can serve as a blueprint to aid in this endeavor.

Conclusion

We developed a semi-automated evidence-based platform (i.e., “EB-COP”) to facilitate the evaluation of TBI COAs for use in research and clinical practice. This tool is unique in that it includes quality indicators that assess the performance of TBI COAs within specific a-priori-defined COUs. This feature is critically important as it considers the psychometric strengths and limitations of the instrument in relation to the target population and intended purpose. These characteristics may help reduce type II error in research and improve diagnostic and prognostic sensitivity in clinical settings. Although designed for use in TBI, the underpinnings of the EB-COP framework are cross-cutting and may be applicable to other health conditions.

Supporting information

Acknowledgments

The authors wish to thank Christopher Malone, PsyD, Alyssa Taubert, OTD and Kristin Wilmoth, PhD, for their instrumental contributions to the development and review of the EB-COP Manual of Operating Procedures and for their meticulous approach to beta-testing the Qualtrics software platform.