Quality of the Development of Traumatic Brain Injury Clinical Practice Guidelines: A Systematic Review

Traumatic brain injury (TBI) is a leading cause of death worldwide and is increasing exponentially particularly in low and middle income countries (LMIC). To inform the development of a standard Clinical Practice Guideline (CPG) for the acute management of TBI that can be implemented specifically for limited resource settings, we conducted a systematic review to identify and assess the quality of all currently available CPGs on acute TBI using the AGREE II instrument. In accordance with PRISMA guidelines, from April 2013 to December 2015 we searched MEDLINE, EMBASE, Google Scholar and the Duke University Medical Center Library Guidelines for peer-reviewed published Clinical Practice Guidelines on the acute management of TBI (less than 24 hours), for any level of traumatic brain injury in both high and low income settings. A comprehensive reference and citation analysis was performed. CPGs found were assessed using the AGREE II instrument by five independent reviewers and scores were aggregated and reported in percentage of total possible score. An initial 2742 articles were evaluated with an additional 98 articles from the citation and reference analysis, yielding 273 full texts examined. A total of 24 final CPGs were included, of which 23 were from high income countries (HIC) and 1 from LMIC. Based on the AGREE II instrument, the best score on overall assessment was 100.0 for the CPG from the National Institute for Health and Clinical Excellence (NIHCE, 2007), followed by the New Zealand Guidelines Group (NZ, 2006) and the National Clinical Guideline (SIGN, 2009) both with a score of 96.7. The CPG from a LMIC had lower scores than CPGs from higher income settings. Our study identified and evaluated 24 CPGs with the highest scores in clarity and presentation, scope and purpose, and rigor of development. Most of these CPGs were developed in HICs, with limited applicability or utility for resource limited settings. Stakeholder involvement, Applicability, and Editorial independence remain weak and insufficiently described specifically with piloting, addressing potential costs and implementation barriers, and auditing for quality improvement.


Introduction
Traumatic brain injury (TBI) is one of the leading causes of death and disability in both developing and developed countries, with the highest incidences among young people less than 30 years of age [1,2]. While the current global burden is unknown, previous conservative estimates indicate an annual incidence of over 10 million people sustaining a TBI leading to hospitalization or death, with road traffic injuries causing a preponderance of cases [1,3]. TBI incidence is projected to continue to rise worldwide due to the continued increasing rates of road traffic injuries, particularly in low and middle-income countries (LMIC) where the rates are twice as high as in high-income countries [1,4]. Furthermore, the World Health Organization suggests that upwards of 90% of road traffic injury deaths occur in LMIC. These trends have been attributed to the rapid economic growth, urbanization, and motorization but limited infrastructure improvements in LMIC [1,4].
Unfortunately, as the burden of TBI continues to increase globally, appropriate prevention efforts have been limited, especially in LMIC, and healthcare quality remains poor, resulting in disproportionately higher mortality rates [5]. One healthcare quality improvement measure that has shown impact is the use of clinical practice guidelines (CPGs) [6][7][8][9]. In fact, in the last several years a significant number of CPGs for acute TBI care has been developed worldwide [5]. However, CPGs vary in quality and comprehensiveness, leading to difficulties with standardization of care, adaptation and implementation, particularly in resource limited settings [10]. There is limited literature comparing and evaluating the strengths and weaknesses of all available CPGs for the treatment of acute TBI.
The Brain Trauma Foundation first developed CPGs for the management of severe TBI in the United States in 1995, with subsequent updated editions published in 2000 and 2007 [11][12][13]. These guidelines have gradually gained acceptance and increased use internationally, with a number of countries adapting them to their individual needs. For example, until recently in Saudi Arabia, patients with severe TBI were managed per individual provider knowledge and experience. By implementing an ICU protocol derived from the Brain Trauma Foundation's guidelines, Saudi Arabian providers were able to significantly reduce hospital and ICU mortality due to TBI [6].
Despite this, a major criticism of the Brain Trauma Foundation CPGs is that they may not be appropriate for use in all locales due to differences in available resources. Subsequently, a number of newer CPGs have been developed in many different countries and by various practitioner groups based on data and capabilities specific to their respective practice environments [14][15][16]. Applying these CPGs in LMICs with limited resources is challenging. Additionally, these CPGs vary in quality and content, span across multiple disciplines, and are published in disparate literature bases, making effective utilization challenging. Previous studies evaluating quality of existing TBI CPGs, for instance, have focused on subsets of TBI severity such as mild TBI only [14,16], or review only a limited number of CPGs [17].
We aimed to assess and summarize the quality of all currently available international acute TBI CPGs by conducting a systematic review using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument [17]. We also compared the quality of CPGs created in high resource countries with those from low and middle income. We expect that these results will inform the development of a standard CPG for the acute management of TBI that can be implemented specifically for limited resource settings.

Protocol and Registration
This systematic review is reported in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) Statement [18], and is registered in the PROS-PERO database (International Prospective Register of Systematic Reviews) under the number CRD42013006008.

Eligibility Criteria
Articles mentioning clinical practice guidelines or recommendations for traumatic brain injury, which met the following inclusion criteria were considered: acute management of TBI (less than 24 hours), any level of traumatic brain injury, high and low income countries and publication in English. No CPG population age restrictions were added as we chose to evaluate both pediatric and adult CPGs. For multiple versions of CPGs, only the newest CPGs were included in the analysis and the older versions were excluded.

Information Sources
We employed an extensive search strategy from April 2013 to December 2015 to identify guidelines from the following electronic databases: MEDLINE, EMBASE and the Duke University Medical Center Guidelines located in the Duke University Medical Center Library and Archives [19]. The Duke University Medical Center Guidelines is an electronic repository curated by Duke University which gathers several national sources indexing CPGs, including the following: National Guideline Clearinghouse, Center for Disease Control Guidelines, Cochrane Library, AHRQ Evidence reports, American College of Emergency Medicine CPGs, American Academy of Child & Adolescent Psychiatry, American Psychiatric Association Guidelines, Canadian Medical Association, Health Services/Technology Assessment Text Collection, Infectious Disease Society of America, Practice guidelines, Joint National Committee, National Comprehensive Cancer Network Guidelines, National Institute for for Health and Clinical Excellence guidelines, NIH Consensus statements archive, US Preventative Services Task Force, Veterans Affairs Clinical Practice Guidelines. In addition, we manually searched the references and performed a citation analysis of the included studies using Google Scholar to include any potential CPG document that was not included in the initial steps.

Literature Search
The initial search comprised of the MeSH terms "Brain Injuries", "Guideline [Publication Type]", "Practice Guideline [Publication Type]", "Practice Guidelines as Topic", and their respectives entry terms. Appendix 1 presents the search strategy used in the PubMed database. We did not use limits for languages or date when searching the databases, but added to the final list only documents in English.

Study Selection
Titles and abstract of the retrieved articles were independently evaluated by two reviewers (A. P. and J.A.). Abstracts that did not provide enough information regarding the eligibility criteria were retrieved for full-text evaluation. Reviewers (A.P. and J.A.) independently evaluated fulltext articles and determined study eligibility. Disagreements were solved by consensus and if disagreement persisted, a third reviewer's opinion was sought (C.S). For CPGs with more than one version, we reported only the most recent versions given their large overlap and anticipated improvement in quality.

Quality of Clinical Practice Guidelines
Five appraisers independently assessed each eligible and selected guideline for quality in accordance with the AGREE II instrument [17,20]. AGREE II is an instrument designed to assess quality of clinical practice guidelines. It consists of 23 items divided into six domains: 1) scope and purpose, 2) stakeholder involvement, 3) rigor of development, 4) clarity of presentation, 5) applicability, and 6) editorial independence. Domain one, scope and purpose, evaluates specificity of the overall objective, clinical questions and the patient population described in the CPG. Domain two measures stakeholder involvement, inclusion of relevant professional groups, patients' views and preferences, and target users. Domain three evaluates the guideline's systematic methods, stating criteria for selecting evidence and explicit links between evidence and recommendations, strengths and limitations of the evidence, consideration of risks and benefits, external review prior to publication, and procedures for updating the guideline. The fourth domain, clarity of presentation, evaluates for clarity, lack of ambiguity, determines if different management options are presented, and assesses whether key recommendations are easily identifiable. Domain five assesses if the CPGs address recommendations for clinical application, barriers to application, cost or resource implications, and monitoring criteria. Finally, domain six evaluates for editorial independence from funding bodies and addressing of potential conflicts of interest.
Each item was scored on a seven-point scale (one = strongly disagree and seven = strongly agree). Scores for each domain were calculated by using the sum of all items within a domain and scaling the score as a percentage of the maximum possible score using the following formula: Obtained Score À Minimum Possible Score Maximum Possible Score À Minimum Possible Score The results from each guideline were summarized in a heatmap visualization with values for each domain. All 23 items of the AGREE-II instrument were assessed with results reported in percentage form for each of the six domains. A value of 100% indicated a domain in which all items were scored with seven points (strongly agree). A value of 0% corresponded to a domain in which all items were scored with one point (strongly disagree).
Each reviewer received a user's manual of the AGREE II instrument, containing its instructions. The six AGREE domains were reported independently for each included CPG. Additionally, an independent global assessment was conducted from the six domains and reported along with the appraiser's recommendation. Recommendations were measured in a threeoption scale ("Yes", "Yes with modifications", and "No"), with a qualitative comment about the CPG [17]. For each domain and for the global assessment, we report average values with respective standard deviations and range (minimum and maximum values).

Data Extraction
Two reviewers (A.P. and J.A.) independently conducted the data extraction and disagreements were resolved by a third reviewer (C.S.). Besides the AGREE assessment previously described, the following general characteristics of the studies were collected: year of publication, location where the guideline creation took place, the organization that created the guidelines, the main focus of the guideline, the patient population of the guideline, and the severity of TBI the guideline was addressing. Additionally, the countries of origin for each guideline were classified into high-, middle-, and low-income based on World Bank definitions [21]. The senior researcher (C.S.) moderated all discrepancies or doubts during the rating process.

Data Analysis
Data analysis was performed descriptively and with graphical representation. Overall assessments for each domain were calculated following the methods already described [17,20]. Consistency of evaluations of the AGREE II domain and for the overall assessment was calculated with an intra-class correlation coefficient (ICC). Graphical solutions were carried out with R software for statistical language [22].

Study Selection
The initial search strategy identified 2,742 titles and abstracts 80 of which were removed for duplicates. From these, 2487 were excluded after reviewing abstracts. A reference and citation analysis was performed on the remaining 175 articles yielding an additional 98 abstracts. Full text analysis was then performed on a total of 273 articles of which only 24  met inclusion criteria (Fig 1).

CPG Quality Assessment (AGREE)
All AGREE II assessments summaries (average per domain and standard deviation) are described per domain and with the overall assessment. Domain specific results are summarized in Fig 2. Overall assessment recommendations and comments are displayed in Table 2. Interrater reliability for each domain is expressed in Table 3. Domain Two-Stakeholder Involvement. The highest score of 86.9 was from the Early Management of Patients with a Head Injury; A National Clinical Guideline. (SIGN, 2009) The lowest score of 41.9 was from The Study Group on Head Injury of the Italian Society for Neurosurgery: Guidelines for minor head injured patients' management in adult age (SINch, 1996).
Domain Three-Rigor of Development. The lowest score was 28.3 from The Study Group on Head Injury of the Italian Society for Neurosurgery: Guidelines for minor head injured patients' management in adult age (SINch, 1996 (SINch, 1996) was the only CPG that 100% of reviewers reported they would not recommend for use. Comments on the strengths and weaknesses of each CPG are summarized in Table 2. Appraisers Consistency. Overall reliability was very good, with three of the quality domains (stakeholder involvement, rigor of development and editorial independence) and the overall assessment with values above 0.70, and two (scope and purpose and applicability) values around 0.65. The domain "clarity and presentation" had the lowest reliability (0.43) ( Table 3).

Discussion
This systematic review is the first to synthesize and collate all published clinical practice guidelines for all types of traumatic brain injury into a single large-scale quality review. Our study included 24 CPGs; across all TBI CPGs, the highest mean scores were achieved in clarity and presentation, scope and purpose and rigour of development, while the main weaknesses across CPGs were stakeholder involvement, applicability and editorial independence. The National Institute for Health and Clinical Excellence (NIHCE, 2007) guidelines, the New Zealand Guidelines Group (NZ, 2006), and the National Clinical Guideline (SIGN, 2009) were the three CPGs with best results. The majority of CPGs evaluated in this study were developed by high income countries (HICs), and are therefore minimally applicable in resource limited settings.

CPGs strengths and weaknesses
Overall, the strong scores in the clarity and presentation, scope and purpose, and rigor of development domains have been reported in other systematic reviews evaluating TBI CPGs [14, 17,  Review of Traumatic Brain Injury Practice Guideline Development  47,48]. This is likely attributed to the scientific rigor of developing a CPG, which typically involves a highly methodical approach [49]. In general, the guidelines that were more recently developed or updated, and those that had undergone numerous updates, most consistently demonstrated the highest quality by AGREE II scores. Our analysis indicates an overall improvement in the above domains in the most current CPGs, consistent with other studies [14][15][16]50]. In a 2011 systematic review of CPGs for managing mild TBI by Tavender et al, the NSW 2006 CPGs fared worse in all domains with the exception of domain scope and purpose when compared to our values for the 2011 version of the guidelines [14]. Similarly, another systematic review evaluated an older 2003 version of NIHCE determining an overall assessment score of 66.9 compared to a score of 100.0 on the 2007 version in our study [17]. It is noteworthy to mention that more recent CPGs have also the advantage of newer and more rigorous evidence-based medicine in addition to the availability of designing guidelines around the AGREE or AGREE II format. Nevertheless, a frequently criticized area in our results, within the rigor of development domain, was the lack of procedures for updating the guidelines for quality improvement (QI). Given the trend toward improved CPG quality with newer revisions, development of a systematic QI procedure may help to ensure quality of future CPGs without the resources required for a full new edition every few years.
Older reviews have demonstrated limited Stakeholder involvement in CPG development, a trend that persists in our current review of CPGs [14,15]. While there has been progressive improvement in CPG development, the domains of stakeholder involvement, applicability, and editorial independence remain weak, specifically when it comes to piloting interventions, addressing potential costs and barriers to implementation, and auditing for quality improvement. Recent literature suggests that successful implementation of CPGs reduces mortality and morbidity [6,[51][52][53][54], however applicability of guidelines to a given locale based on factors such as availability and cost of resources, provider skills, and population needs and values, are critically important for successful implementation of CPGs in a manner that will improve patient care. Consideration of stakeholder involvement and applicability are imperative considering these domains are intrinsically associated with CPG implementation and translation to other settings such as LMICs.
It has been suggested that adaptation of existing CPGs to local situations may be a more valid and cost-effective means of achieving high-quality CPGs worldwide [55]. However, recent attempts have revealed that this adaptation process remains complex and challenging, requiring careful planning and implementation to avoid additional costly resource utilization [56]. In particular, when the CPG recommendations require resources not present in a given locale, alternatives to the suggested "optimal" recommendations are required. Given our findings that most CPGs lacked information on addressing costs and barriers to implementation, such as in resource-poor settings, this demonstrates an important void in existing CPGs. Current attempts at adapting existing CPGs may require local practitioners and CPG developers to return to the primary literature to develop location-specific alternatives that are more practicable in their region or medical center. In addition, further research may be required to develop these potential cost-reducing alternatives and demonstrate efficacy.

CPGs and relevance to LMICs
As identified in this systematic review most CPGs have been developed in HICs, which makes them of questionable relevance to LMICs, especially for those populations with different costbenefit parameters for medical care. Many hospitals in LMIC, particularly in more rural areas, lack basic intensive or critical care capabilities, specialized staff, or even necessary diagnostic imaging, factors upon which the majority of the highest scoring guidelines rely upon for implementation [57]. LMICs have potentially greater challenges and barriers to implementation than HICs based on these factors, which need to be addressed to enable a CPG to be useful and beneficial. The few CPGs with high applicability scores were developed by and tailored for HICs [34,35], which makes them unlikely to be applicable for use in limited resourced settings and LMIC countries in their current forms.

Individual CPGs quality assessment and previous research
Comparing our results with other systematic reviews found many similarities. The EAST Evaluation and Management of the Mild Traumatic Brain Injury CPG (EAST, 2012) had similar evaluation of the best and worst domains. Our assessment of the Guidelines for the Acute Medical Management of Severe Traumatic Brain Injury in Infants, Children, and Adolescents (BTF, 2012) was similar to that of Grimmer et al., only differing in the applicability domain, which they assigned a fairly low score of 26.4, compared to our value of 66.7. However, the overall assessment was similar [24]. We found similar evaluations for domains, overall assessment and recommendations for the CPG from the National Institute for Health and Clinical Excellence (NIHCE, 2007) [14,17,47,48,58] and the National Clinical Guideline (SIGN, 2009) [14,58] in other studies.
We did find some differences in the domains of applicability and editorial independence in the systematic review mild TBI CPGs by Tavender et al. For the ACEP Clinical Policy, we assessed scores of 70 and 91.7 in the fields of applicability and editorial independence, respectively, compared to their findings of 17 and 50 [14]. The same study, in addition to a 2011 evaluation by Berrigan et al. found respective scores of 36 and 50 in the applicability domain for the New Zealand Guidelines Group, whereas our score was much higher at 82.5 [14,47]. Given the limited number of systematic reviews on this topic, we were not able to compare all CPGs included in this study.
Limitations. The main limitations of our study are the subjectiveness of the AGREE II tool and the potential bias of the reviewers performing the assessment. The AGREE II tool is a 23 question tool established to evaluate CPG quality. While it is a subjective tool, it is the current gold standard; the AGREE II guidelines suggest using at least two and preferably four appraisers with content specific knowledge [17]. We utilized five appraisers who all had content specific knowledge; four were emergency medicine physicians with experience in research and evidence-based methods and one was a masters level student with extensive research and content topic experience. Given that the AGREE II assessment requires evaluation of the CPGs based on the descriptions available in the published manuscripts, there is a small chance that inaccurate assessments would be due to poor descriptions in the manuscript. However, comparison of our AGREE II ratings to those of other researchers found similar scores [14].
Expanding the reviewers to include other relevant specialties such as neurosurgery, critical care, or neurology would have provided additional input on specialty-specific recommendations. Additionally, due to language limitations, we were only able to review CPGs in English. The language limitation did not provide a significant barrier to most CPGs, given that many of the articles written in different languages were also available in English, or failed to meet inclusion criteria. However, given that many LMICs are non-English speaking, they may have developed CPGs in other languages that we were unable to evaluate and subsequently missed. Finally, we did not search some clinical databases, like the TRIP and GIN repository. However, due to the several diverse repositories included through the Duke library search, it is unlikely that we missed published CPGs.
Conclusions. Our review identifies two specific areas for improvement in clinical practice guidelines addressing the acute management of TBI: (a) the domains of stakeholder involvement, applicability, and editorial independence remain weak and insufficiently described specifically when it comes to piloting interventions, addressing potential costs and implementation barriers, and auditing for quality improvement; (b) CPGs created specifically for use in low income settings are non-existent. Most of these CPGs were developed by high-income countries with only one CPG from an upper middle income country, which was found to have a poorer quality across all domains. This will limit the applicability and implementation capacity of CPGs for limited resourced settings.