Clinical Practice Guidelines and Consensus Statements in Oncology – An Assessment of Their Methodological Quality

Background Consensus statements and clinical practice guidelines are widely available for enhancing the care of cancer patients. Despite subtle differences in their definition and purpose, these terms are often used interchangeably. We systematically assessed the methodological quality of consensus statements and clinical practice guidelines published in three commonly read, geographically diverse, cancer-specific journals. Methods Consensus statements and clinical practice guidelines published between January 2005 and September 2013 in Current Oncology, European Journal of Cancer and Journal of Clinical Oncology were evaluated. Each publication was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) rigour of development and editorial independence domains. For assessment of transparency of document development, 7 additional items were taken from the Institute of Medicine’s standards for practice guidelines and the Journal of Clinical Oncology guidelines for authors of guidance documents. Methods Consensus statements and clinical practice guidelines published between January 2005 and September 2013 in Current Oncology, European Journal of Cancer and Journal of Clinical Oncology were evaluated. Each publication was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) rigour of development and editorial independence domains. For assessment of transparency of document development, 7 additional items were taken from the Institute of Medicine's standards for practice guidelines and the Journal of Clinical Oncology guidelines for authors of guidance documents. Findings Thirty-four consensus statements and 67 clinical practice guidelines were evaluated. The rigour of development score for consensus statements over the three journals was 32% lower than that of clinical practice guidelines. The editorial independence score was 15% lower for consensus statements than clinical practice guidelines. One journal scored consistently lower than the others over both domains. No journals adhered to all the items related to the transparency of document development. One journal’s consensus statements endorsed a product made by the sponsoring pharmaceutical company in 64% of cases. Conclusion Guidance documents are an essential part of oncology care and should be subjected to a rigorous and validated development process. Consensus statements had lower methodological quality than clinical practice guidelines using AGREE II. At a minimum, journals should ensure that that all consensus statements and clinical practice guidelines adhere to AGREE II criteria. Journals should consider explicitly requiring guidelines to declare pharmaceutical company sponsorship and to identify the sponsor’s product to enhance transparency.


Introduction
Consensus statements and clinical practice guidelines are widely used in oncology to improve the quality of patient care [1,2]. While both consensus statements and clinical practice guidelines are intended to provide guidance to clinicians, there are important differences between them. A clinical practice guideline (also called a medical guideline or clinical protocol) produces statements that are informed by a systematic review of the evidence and an assessment of the benefits and harms of alternative options [3]. A consensus statement is developed by an independent panel of experts, usually multidisciplinary, convened to review the research literature in an evidence-based manner for the purpose of advancing the understanding of an issue, procedure or method Table 1. Items from AGREE II (Domains 3 and 6) and additional items collected to assess Transparency of Document Development.

Criteria collected Source
Rigour of development Systematic methods were used to search for evidence.
The criteria for selecting the evidence are clearly described The strengths and limitations of the body of evidence are clearly described.
The methods for formulating the recommendations are clearly described. Domain 3 of AGREE II [5] The health benefits, side effects, and risks have been considered in formulating the recommendations.
There is an explicit link between the recommendations and the supporting evidence.
The guideline has been externally reviewed by experts prior to its publication.
A procedure for updating the guideline is provided.

Editorial independence
The views of the funding body have not influenced the content of the guideline. Domain 6 of AGREE II [5] Competing interests of guideline development group members have been recorded and addressed.
Additional items to assess transparency of document development Was a systematic review performed? (yes -systematic review performed and documented, no -systematic review not performed or not documented) IOM [3] JCO [10] How was the guideline group established? (invited, not disclosed, other), IOM [3] Was the group privately funded? (yes, no, not disclosed) JCO [10].

Consensus sponsor
For guidelines where a pharmaceutical product was evaluated was a specific product endorsed in the statement? (yes-name of product, no)  [4]. Both documents provide recommendations for optimizing patient care [3].
Although consensus statements address topics in which the evidence base is less extensive compared to clinical practice guidelines, their development should still be methodologically rigorous and transparent [4]. To assist with maintaining methodological rigor, organizations such as Appraisal of Guidelines for Research and Evaluation (AGREE) [5], Institute of Medicine (IOM) [3] and Guidelines International Network (GIN) [6] have developed criteria to ensure objective, scientifically valid, and consistent standards for the development and reporting of high quality guidance documents.
Given their widespread availability and importance for both clinical practice and funding decisions [7], we sought to evaluate the methodological quality of both consensus statements and clinical practice guidelines published in three commonly accessed oncology-specific journals through the domains of rigor of development and editorial independence Information around the transparency of document development was also collected to assess whether or not pharmaceutical company sponsored guidelines were more likely to endorse a product manufactured by the sponsoring company.

Methods
Three oncology specific journals were searched for consensus statements and clinical practice guidelines published from January 2005-September 2013. Current Oncology (CO), the European Journal of Cancer (EJC) and the Journal of Clinical Oncology (JCO) were chosen as they have editorial offices in different   [3]. As our primary focus related to evaluating the methodological quality, we opted to use Domain 3 of the AGREE II tool (Rigour of Development) and Domain 6 (Editorial Independence) to assess the documents. The rigour of development domain consists of 8 items, while the editorial independence domain consists of 2 items (items are shown in Table 1). AGREE II items are scored on a 7point Likert scale ranging from 1 (strongly disagree) to 7 (strongly   The development of evidence-based European guidelines on the management of depression in palliative cancer care [69] 2011 41 71 agree). Each domain score was calculated as per the AGREE II instructions included in the user's manual [5]. Domain score = [score obtained -minimum possible score]/[maximum possible score -minimum possible score 6 100], giving a percentage score between 0 and 100. As the Standards and Guidelines Evidence (SAGE) directory has used AGREE II to evaluate English language cancer guidelines released since 2003 [8], if a document had been included in the SAGE database, this appraisal was used and a primary assessment of our own was not performed. The SAGE assessment utilises two trained evaluators to assess each document, discrepancies of a certain magnitude are resolved by a third and if required, fourth evaluator [9]. As we also wanted to assess issues surrounding the transparency of document development, and specific to whether or not pharmaceutical company sponsorship of the guideline development process was associated with product endorsement, each document was assess using an additional 7 items. These additional items were derived from the IOM standards for trustworthy clinical practice guidelines [3] and the JCO criteria for publishing consensus statements and clinical practice guidelines [10] ( Table 1). These items included a statement on ''Was a systematic review conducted?'' Additional items related to transparency included, ''How was the group established?'', ''Was the group multidisciplinary?'', ''Was the group privately funded?'' and ''What was the name of the funding body?'' In order to assess any relationship between the sponsor of the group and recommendations, for pharmaceutical-related guidelines we also collected data on ''Was a specific product endorsed in statement?'', and if so, ''Who was the manufacturer of product?''.
Six reviewers appraised the documents, with each document appraised by two independent reviewers (see Acknowledgements). Discrepancy scores between reviewers for AGREE II were calculated using the concordance calculator for the SAGE database calculations [8]. We planned to resolve discrepancies in assessments as per SAGE, by third and if necessary fourth evaluators. For the additional items assessed, any discrepancies between the two reviewers were resolved by consensus.

Statistical analysis
For the two AGREE II domains of interests, we reported overall means with their 95% confidence intervals for each journal, stratified into separate categories of consensus statement and clinical practice guideline. We also stratified by year of document publication. We used the publication date of the IOM 'Clinical practice guidelines we can trust', March 2011 [3], as a time point in which to assess document quality over time. We compared overall differences between journals and between consensus statement or clinical practice guideline using analysis of variance (ANOVA). We also calculated the mean difference in scores between consensus statement and clinical practice guidelines with their corresponding 95% confidence intervals.
For the additional items collected addressing transparency of document development, we calculated the proportion of responses categorized as ''Yes'', ''No'', and ''Not Reported''. We assessed for differences in the journals' assessments using a chi-square test (or Fisher's Exact test when dealing with small cell counts in summary contingency tables) at a significance level of 5% while stratifying analyses into categories of consensus statement and clinical practice guideline. Finally, we compared overall items responses according to their consensus statement or clinical practice guideline category.
Agreement between reviewers was assessed by a concordance calculator, determining the number of standard deviations between reviewers, over each domain. A 'high' discrepancy score occurred when greater than 2 standard deviations were present between reviewers, 'medium' if .1.5 but ,2 standard deviations and 'low' if ,1.5 standard deviations.

Identified Literature
The search identified a total of 104 documents for review. Three were excluded as one was a physician survey, one was a review of guidelines, and one was a letter to the editor. Therefore, 34 consensus statements and 67 practice guidelines were retained for assessment. The numbers and types of documents for each journal were; CO-14 consensus statements, 24 clinical practice guidelines, EJC -9 consensus statements, 13 clinical practice guidelines and JCO-11 consensus statements, 30 clinical practice guidelines.

AGREE II Rigour of development scores
When assessed across all three journals ( Figure 1, Table 2), the mean scores for consensus statements were 32% (95% CI 27-38%) and for clinical practice guidelines 64% (95% CI 59-69%). The mean difference between guidelines was 32% (p,0.0001), indicating that clinical practice guidelines were scored significantly higher than consensus statements in terms of rigour of development. Analyses stratified by journal showed that rigour of development scores were significantly lower for consensus   Discrepancy levels between the reviewers were low with the exception of one consensus statement published in the Journal of Clinical Oncology [11] which had a high discrepancy score.

AGREE II Editorial independence scores
When assessed across all three journals ( Figure 2, Table 2), the mean score for consensus statements was 53% (95% CI 47-59%) and for clinical practice guidelines was 68% (95% CI 63-73%). The mean difference between consensus statement and clinical practice guideline scores was 15% (p = 0.0003), indicating that clinical practice guidelines were scored significantly higher than consensus statements with respect to editorial independence. Editorial independence scores were significantly lower for consensus statements than clinical practice guidelines in documents published in CO

Additional transparency of document development item scores
Consensus statements infrequently referenced or conducted a systematic review on the topic of the guideline (6/34 = 18%), a step which was much more common with clinical practice guidelines (56/67 = 83%) (p = 0.018) ( Table 3). The largest discrepancy was seen in JCO where 0/11 (0%) of consensus statements documented a systematic review compared to 28/30 (93%) of clinical practice guidelines. Neither consensus statements (50%) nor clinical practice guidelines (34%) consistently declared how their development group was established. Consensus statements were more likely than clinical practice guidelines to state that participants were ''invited'' (12/34 = 35% vs 14/67 = 21%; p = 0.01). Guideline groups were multidisciplinary in 21 out of 34 (62%) consensus statements and 50 out of 67 (75%) clinical practice guidelines groups. Group member roles were not declared in 35% (12/34) of the consensus statements nor in 25% (17/67) of clinical practice guidelines (p = 0.19).
With respect to whether or not a document endorsed a product made by the sponsoring company (Table 3), this occurred less frequently in clinical practice guidelines (2/67 = 3%) than in consensus statements (10/34 = 29%) (p,0.0001). In CO, consensus statements endorsed the product of the sponsoring company in 9/14 (64%) of cases. All of these documents declared financial support from the sponsoring company, but none explicitly declared the link between the sponsoring company and the product endorsed. Four percent of clinical practice guidelines published in CO endorsed the sponsor's product. This trend was seen to a lesser extent in EJC with 11% of consensus statements endorsing sponsors products and 8% of clinical practice guidelines. No document published by JCO documented a relationship between pharmaceutical company funding and product endorsement in the guideline.
Have consensus statements and clinical practice guidelines improved over time?
When assessed chronologically, there is no association with document quality over time, using the date of publication of the IOM 'Clinical practice guidelines we can trust', March 2011 as a reference point (Tables 4,5 and 6). There may be a trend of declining pharmaceutical sponsorship of documents in recent years.

Discussion
As the terms consensus statement and clinical practice guidelines are often used interchangeably and both are used to improve clinical care, their methodological rigour and transparency of development is essential. Here we report the results of a review of the methodological quality of consensus statements and clinical practice guidelines in a limited sample of the oncology literature. While others have published on quality assessment of clinical guidelines in oncology using either the AGREE or AGREE II tool [112][113][114][115][116][117], to our knowledge this is the first such comprehensive review of both consensus statement and practice guidelines in oncology.
As literature assessing the quality of consensus statements is limited [118], we used tools developed for clinical practice guidelines and collected additional information that would help assess the transparency of guideline development. AGREE II is a validated appraisal tool for assessing the methodological development quality and reporting of practice guidelines; it does not assess the actual content of clinical recommendations [5]. AGREE II assesses how well a guideline performs on each of the 6 domains (scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability and editorial independence). We felt the rigour of development (an assessment of the evidentiary base and methods used to formulate recommendations) and editorial independence (an assessment of bias and competing interests influencing recommendation formulation [5]) were the most appropriate for our evaluation.
For both the rigour of development and editorial independence domains, consensus statements scored consistently less well than did practice guidelines. In the only publication we found evaluating practice guidelines in comparison to consensus statements, although not specific to oncology [118], similar differences were seen, with consensus statements scoring significantly lower than clinical practice guidelines across 4 of the 6 AGREE II domains (stakeholder involvement, rigour of development, clarity, and presentation and applicability). We could show no improvement in document quality over time.
Performing a systematic review is an essential element of guideline development [119]. Both IOM [120] and JCO [10] state that ''clinical practice guideline developers should use systematic reviews'' and that ''guidelines/recommendations should be driven by a high level of evidence'' respectively. We felt it was necessary to specifically ask 'was a systematic review performed?' We asked this question even though AGREE II domain 3.1 assess if 'systematic methods were used to search for evidence' (scored on a continuum of whether a guideline reports what databases were searched, the search terms used, the search time periods and the inclusion of a full search strategy). In the current study systematic reviews were performed more frequently by clinical practice guidelines than consensus statements across all three journals. With respect to the processes by which a clinical practice guideline group was established and the role of individual members, this was inconsistently reported. There were however significant differences between these items in consensus statements and clinical practice guidelines.
Of particular interest was the role of the funding body for the development of the guidance document. While no information can be gleaned for whether this association is real or implied, several observations can be made. Overall, consensus statements and clinical practice guidelines published in the three journals studied either did not declare or were not explicit about the funding source for the document (funding source not declared in 65% consensus statements, 45% clinical practice guidelines). For documents with topics related to pharmaceutical products, when the document was sponsored by a pharmaceutical company, documents endorsed the sponsor's product in both consensus statements (29%) and to a lesser degree in clinical practice guidelines (3%). However, in the CO journal, 64% of consensus statements published endorsed the sponsors product, whereas only 4% of clinical practice guidelines endorsed the sponsors product. Further, this association was not reported in the conflict of interest statement. This absence of reporting contravenes standards published by medical societies [121,122] and could question the integrity and quality of published guidance documents [123,124].
We acknowledge a number of study limitations. Although we feel that consensus statements should be subjected to the same rigorous criteria for their development as practice guidelines, the AGREE II tool has not been validated for evaluation of consensus statements [5,118]. The additional items we included for assessment from the IOM guideline standards and JCO authorship guidance on consensus statements and clinical practice guidelines also have not been validated. Consensus statements and clinical practice guidelines analyzed here may not be representative of all oncology consensus statements and clinical practice guidelines released between January 2005 and September 2013, nor representative of all oncology journals. A brief PubMed search suggests over 900 oncology guidance documents were published in peer-reviewed journals over the same time period, translating to a sample of 11% of these documents. Finally, we chose only three journals from which to sample. Our rational for selecting them was that they commonly publish both consensus statements and clinical practice guidelines, are prominent journals in their locale of origin and are geographically diverse. We appreciate that these journals may not be representative of all oncology journals.

Conclusions
While consensus statements and clinical practice guidelines are developed with slightly different approaches and methods, both are used to inform clinical and policy decisions. As such both documents should be developed using equally rigorous and transparent methods and subjected to high quality standards.
Here we have shown that consensus statements score lower than clinical practice guidelines for scores of rigour of development and editorial independence. Consensus statements are also less likely to include a systematic review of the literature and were more likely to be sponsored by a pharmaceutical company and to endorse a specific pharmaceutical product. Unfortunately transparency of document development was generally poor in both types of documents and there was infrequent documentation of sources of funding, how guideline groups were established and who comprised their guideline development groups.
Given the important role of guidance we feel that both consensus statements and clinical practice guidelines should be subject to the same rigorous and high quality development criteria. We suggest that journals encourage authors of guidance documents to use the AGREE II and IOM criteria when developing their documents and require journal reviewers to use these same criteria when undertaking their peer-review of these documents. While there are quality differences between each of the journals sampled in our study, this was most pronounced around the issues of private funding and product endorsement. Readers of guidance documents published within these journals should be made aware of the presence of private funding and sponsorship should be made transparent through their reporting so that readers can acknowledge such conflicts and potential bias.