Several studies have reported that clinical practice guidelines (CPGs) in a variety of clinical areas are of modest or variable quality. The objective of this study was to evaluate the quality of an international cohort of CPGs that provide recommendations on pharmaceutical management of glycemic control in patients with type 2 diabetes mellitus (DM2).
Methods and Findings
We searched the National Guideline Clearinghouse (NGC) on February 15th and June 4th, 2012 for CPGs meeting inclusion criteria. Two independent assessors rated the quality of each CPG using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) instrument. Twenty-four guidelines were evaluated, and most had high scores for clarity and presentation. However, scope and purpose, stakeholder involvement, rigor of development, and applicability domains varied considerably. The majority of guidelines scored low on editorial independence, and only seven CPGs were based on an underlying systematic review of the evidence.
The overall quality of CPGs for glycemic control in DM2 is moderate, but there is substantial variability among quality domains within and across guidelines. Guideline users need to be aware of this variability and carefully appraise and select the guidelines that they apply to patient care.
Citation: Holmer HK, Ogden LA, Burda BU, Norris SL (2013) Quality of Clinical Practice Guidelines for Glycemic Control in Type 2 Diabetes Mellitus. PLoS ONE 8(4): e58625. doi:10.1371/journal.pone.0058625
Editor: Rocio I. Pereira, University of Colorado Denver, United States of America
Received: January 5, 2013; Accepted: February 6, 2013; Published: April 5, 2013
Copyright: © 2013 Holmer et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This manuscript was the result of work performed for the Agency for Healthcare Research and Quality under grant HS018500-01 (S. L. Norris). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: S.L. Norris has received funding from the US Agency for Healthcare Research and Quality (grant number 1R01HS018500) to study conflict of interest in clinical practice guideline development. H.K. Holmer, L.A. Ogden and B.U. Burda received salary support from this same grant to study conflict of interest in clinical practice guideline development. S.L. Norris has received funding from the US Centers for Disease Control and Prevention (CDC) and from the US Agency for Healthcare Research and Quality for development of guidelines related to diabetes. B.U. Burda currently receives salary support from the US Agency for Healthcare Research and Quality for work on the US Preventive Services Task Force guidelines. The authors have no other competing interests to declare.
High quality clinical practice guidelines (CPGs) provide recommendations based on a systematic review of the evidence, an assessment of balance of benefits and harms, and a transparent process for translating evidence to recommendations . CPGs have the potential to influence the care delivered by a large number of healthcare providers and thus the outcomes of patients . The quality of CPGs is therefore critically important. High-quality, or trustworthy guidelines promote the use of effective clinical services, decrease undesirable practice variation, reduce the use of services that are of minimal or questionable value, increase the use of effective but underused services, and target services to populations most likely to benefit .
The global burden of diabetes is enormous. Of the estimated 346 million people worldwide with diabetes, 90% have type 2 diabetes mellitus (DM2) . An estimated 3.4 million persons died in 2004 from causes related to elevated blood glucose and the World Health Organization predicts that diabetes-related deaths will double between 2008 and 2030 . Persons with diabetes have at least two times the risk of death than persons without diabetes , and morbidity from both macro-and microvascular disease is substantial. There are numerous pharmaceutical classes and specific agents used to treat hyperglycemia in DM2, with different mechanisms, pharmacokinetics, mean effects on blood glucose, and adverse effects.
A number of studies have reported that CPGs in a variety of clinical areas are of modest or variable quality , , , , . The objective of this study was to examine the quality of CPGs that include recommendations on pharmacotherapy for glycemic control in DM2.
We searched the National Guideline Clearinghouse (NGC) (www.guideline.gov) on February 15th and June 4th, 2012 for all guidelines that provided recommendations on pharmacotherapy for glycemic control in persons with DM2. We searched for CPGs on two separate dates because guidelines are continually being revised, updated, or archived in the NGC and we wanted to ensure that we identified all guidelines relevant to our topic and that we did not exclude a guideline because it was archived during the time of our review process.
The NGC is a publicly available online resource for evidence-based CPGs, funded by the United States government and produced by the Agency for Healthcare and Research Quality (AHRQ). For CPGs to be included in the NGC, guidelines must meet the following criteria: 1) the clinical practice guideline contains systematically developed statements that include recommendations, strategies, or information that assists physicians and/or other health care practitioners and patients to make decisions about appropriate health care for specific clinical circumstances; 2) the clinical practice guideline was produced under the auspices of medical specialty associations; relevant professional societies, public or private organizations, government agencies at the Federal, State, or local level; or health care organizations or plans; 3) corroborating documentation can be produced and verified that a systematic literature search and review of existing scientific evidence published in peer reviewed journals was performed during the guideline development; 4) the full text guideline is available upon request in the English language; 5) the guideline was developed, reviewed, or revised within the last 5 years .
In addition to meeting the NGC inclusion criteria, our study required that CPGs provided recommendations for glycemic control in any population with DM2, including adults, children, pregnant women, and persons with DM2 and any comorbid condition. If the full guideline was not available in the public domain, we purchased a copy.
Two coauthors with experience in quality assessment of CPGs independently scored each guideline using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) instrument  (Table 1). AGREE II consists of 23 items grouped into six domains: 1) scope and purpose; 2) stakeholder involvement; 3) rigor of development; 4) clarity of presentation; 5) applicability; and 6) editorial independence . The assessors then compared their individual scores for each item and came to consensus on discrepant scores (defined as scores varying by three points or more on the seven-point AGREE II scale). This approach accounted for frank error on the part of an assessor, when they had missed the relevant part of the guideline in their original assessment. If the two assessors were unable to reach consensus, a third person was consulted. If the two assessors' scores differed by two points they were averaged; if they differed by one point the lower score was kept. Standardized domain scores (expressed on a scale of 0–100) were calculated using the approach of AGREE II ([obtained score – minimum possible score] divided by [maximum possible score – minimum possible score]) . The overall AGREE II evaluation of recommend, recommend with modifications, or do not recommend each guideline was independently determined by each assessor and then consensus was achieved.
CPGs were considered to be based on a systematic review if there was either reference to a review or a review was contained within the guideline document, the review reported a search of one or more bibliographic databases, and a defined cohort of studies derived from the search was used to formulate recommendations.
Twenty-four guidelines met our inclusion criteria (Table 2; Figure S1). Ten of the guidelines were published between 2007 and 2009; the remainder were published in 2010 or later. The majority of the CPGs (n = 14; 58%) were developed by US-based organizations, followed by European (n = 5; 21%), Canadian (n = 3; 13%), and international (n = 2; 8%) organizations. The CPGs meeting inclusion criteria were developed primarily by non-profit organizations (25%), government agencies (21%), and medical specialty societies (21%).
The overall quality of the included CPGs varied considerably, both within and across AGREE II domains (Table 2). No guideline scored more than 50% in all six AGREE II domains. Across the CPGs, scores were highest for the domain of clarity and presentation (mean 81% of the maximum possible score). Most of the guidelines presented easily identifiable, specific key recommendations and different options for management of DM2. The domain of scope and purpose was also rated relatively high (mean 64% of the maximum possible score). The overall objectives of the guidelines and the specific populations to whom the guidelines were meant to apply were also well described in most CPGs.
Scores for the stakeholder involvement (mean 52% of the maximum possible score) and applicability (mean 43% of the maximum possible score) were variable across guidelines. Seven guidelines scored greater than 60% on stakeholder involvement , , , , , , , while only four CPGs scored that well on applicability , , , .
Scores for rigor of development were generally between 30–60%, with a few CPGs scoring very high and a few scoring very low (Figure 1). The Scottish Intercollegiate Guideline Network (SIGN)  and National Collaborating Centre for Women's and Children's Health (NCC-WCH)  guidelines scored the highest (both greater than 80%) and the New York State Department of Health (NY DoH)  and National Health Care for the Homeless Council (NHCHC)  guidelines scored the lowest (0% and 4%, respectively). Only seven CPGs , , , , , ,  reportedly based their recommendations on an underlying systematic review.
Scores are obtained from two of the domains of AGREE II (Appraisal of Guidelines for Research and Evaluation)  Guidelines: American Association of Clinical Endocrinologists (AACE), American College of Physicians (ACP), American Diabetes Association (ADA), American Medical Directors Association (AMDA), Canadian Agency for Drugs and Technologies in Health (CADTH), European Society of Cardiology (ESC), Institute for Clinical Systems Improvement (ICSI), International Diabetes Center (IDC), International Diabetes Federation (IDF), Joslin Diabetes Center (JDC), National Kidney Foundation (KDOQI), National Collaborating Centre for Acute and Chronic Conditions (NCC-ACC), National Collaborating Centre for Women's and Children's Health (NCC-WCH), National Health Care for the Homeless Council (NHCHC), National Institute for Health and Clinical Excellence (NICE), New York State Department of Health (NY DoH), Qatif Primary Health Care (QPHC), Scottish Intercollegiate Guidelines Network (SIGN), University of Michigan Health System (UMHS), Department of Veterans Affairs/Department of Defense (VA/DoD), Wisconsin Diabetes Prevention and Control Program (WDPCP).
Editorial independence was the domain with the lowest scores across guidelines (mean 26% of the maximum possible score, range 0–75%). CPGs infrequently described how the views of the funding body may or may not have influenced the content, and eight guidelines (33%) did not provide any information on conflicts of interest for the CPG developers. Of the 16 (66%) CPGs that did provide information on competing interests, only one guideline reported that they discussed and resolved their conflicts .
In the overall assessment, 13 guidelines (54%) were recommended, seven (29%) were recommended with modifications, and four (17%) were not recommended (Table 2). The four guidelines that were not recommended had little to no evidence base and lacked editorial independence. All other guidelines were recommended provided that they still needed improvement in one or more domains.
The overall quality of the 24 guidelines for glycemic control in DM2 was highly variable, and no guideline scored well in all domains of quality. There was also significant variability across domains within guidelines. Guidelines consistently scored well in the domain of clarity and presentation, suggesting that this component of guideline development may be easier to achieve or more highly valued by guideline development organizations. On the other hand, editorial independence was poorly addressed by almost all guidelines (the only exceptions were the CPGs developed by the American College of Physicians (ACP) , ). Perhaps guideline developers either do not appreciate the importance of conflict of interest disclosures and management, or choose not to address the issue in a transparent manner. There is considerable evidence that financial conflicts of interest are highly prevalent among CPGs in a variety of clinical areas , , , , and there is emerging evidence that conflict of interest may affect guideline recommendations .
Our assessment also suggests that guideline developers do not pay sufficient attention to the applicability of their recommendations to their target audiences and to implementation issues. Lack of attention to these issues has been noted in other studies examining the quality and usefulness of clinical practice guidelines , , .
Several studies have examined the quality of various cohorts of CPGs in diabetes, and findings vary. Bennett and colleagues  reported summary scores for the AGREE domain of rigor of development ranging between 17% and 100% across 11 CPGs from North America and the United Kingdom that examined oral agents for glycemic control. Eight of these guidelines had summary scores of less than 50%. Guidelines on the management of diabetes in pregnancy  also reported a great deal of variability in quality, with editorial independence the most problematic domain. Stone and colleagues  noted a great deal of variability across eight guidelines from Western Europe on the management of DM2, again with applicability and editorial independence scoring poorly. On the other hand, Mahmud and Mazza  scored all domains very high for five guidelines on preconception care in women with diabetes, with the exception of editorial independence. To our knowledge, no study has examined the broad spectrum of diabetes pharmacotherapy guidelines as in our study, which presents the largest cohort of published guidelines from around the globe.
Systematic reviews should form the basis for all high quality CPGs . In our cohort of 24 guidelines, however, only seven (produced by five organizations) included or referenced an underlying systematic review. This suggests a fundamental problem with the majority of these CPGs. Even when present, the systematic reviews underpinning CPGs varied in quality, as indicated by the domain of rigor of development in AGREE II.
There are several important issues with regards to using AGREE II to appraise the quality of CPGs. First, the AGREE II domain of rigor of development does not encompass all important aspects of the quality of a systematic review, as does a quality assessment instrument developed specifically for that purpose, such as AMSTAR . Second, and more importantly, AGREE II does not consider the relative importance of the six domains of quality: rigor of development is considered of equal importance to the other five domains. We think that this is problematic, and suggest that the domains of AGREE II should not be weighted equally. If the review underlying the guideline recommendations is either nonexistent or flawed (a low score on the domain of rigor of development), the guideline recommendations have a high risk of bias, and the other domains (no matter how well executed) are of little relevance in quality assessment.
The overall assessment in AGREE II of whether the CPG was recommended, recommended with modifications, or not recommended  is also problematic. There is no guidance in the AGREE II instrument as to how to make this assessment, and assessors may or may not weigh the various domains equally. For example, if most domains score high, but rigor of development scores low, an assessor might rate the CPG as “recommended”, and this could be misleading to potential users of the guideline. We suggest that AGREE II needs to be further revised to incorporate a hierarchy for appraisal, and to provide additional guidance on how to make the overall assessment.
This study has limitations, in addition to those imposed by AGREE II. Our cohort may not be representative of all diabetes guidelines, as we selected only those examining glycemic control for type 2 diabetes included in the NGC. Guidelines on other aspects of diabetes and those not in the NGC (which has minimum quality standards for inclusion) may differ in quality from those that we examined. In addition, the NGC does not contain all guidelines on diabetes: organizations choose to submit their guidelines to the NGC, and we did not search other sources for additional relevant guidelines.
We purposefully chose a low threshold for defining whether a systematic review was used to develop recommendations in the CPG. If we had imposed a more stringent definition such as one requiring a search of multiple bibliographic databases, assessment of quality of individual studies and of the body of evidence, and an explicit framework for developing recommendations from the body of evidence, the number of CPGs in our cohort that were considered to base recommendations on an underlying systematic review would have been far fewer.
In view of the potential impact of CPGs on health care delivery and patient outcomes, it is imperative that guidelines be of optimal quality. It is clear from this cohort of CPGs on glycemic control in DM2 that only a small minority of guidelines fulfill most criteria for a high quality guideline. The guideline user needs to beware, to critically appraise guidelines before use and to weigh the relative importance of the criteria for quality, starting with an assessment of whether a high quality systematic review underpins each recommendation.
PRISMA flow diagram. From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097.
Conceived and designed the experiments: HKH LAO BUB SLN. Performed the experiments: HKH LAO BUB SLN. Analyzed the data: HKH SLN. Wrote the paper: HKH LAO BUB SLN.
- 1. Institute of Medicine (2011) Clinical Practice Guidelines We Can Trust; Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, editors. Washington, D.C.: The National Academies Press.
- 2. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J (1999) Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 318: 527–530.
- 3. Institute of Medicine (2008) Knowing what works in health care: A roadmap for the nation. Washington, DC: The National Academies Press.
- 4. World Health Organization (2012) Diabetes Fact Sheet. Geneva, Switzerland: World Health Organization.
- 5. Burda BU, Norris SL, Holmer HK, Ogden LA, Smith BE (2011) Quality varies across clinical practice guidelines for mammography screening in women aged 40–49 years as assessed by AGREE and AMSTAR instruments. J Clinical Epidemiol 64: 968–976.
- 6. Burgers JS, Fervers B, Haugh M, Brouwers M, Browman G, et al. (2004) International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. J Clin Oncol 22: 2000–2007.
- 7. Harpole LH, Kelley MJ, Schreiber G, Toloza EM, Kolimaga J, et al. (2003) Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest 123: 7S–20S.
- 8. Ward JE, Grieco V (1996) Why we need guidelines for guidelines: a study of the quality of clinical practice guidelines in Australia. Med J Aust 165: 574–576.
- 9. de Haas ER, de Vijlder HC, van Reesema WS, van Everdingen JJ, Neumann HA, et al. (2007) Quality of clinical practice guidelines in dermatological oncology. Journal of the European Academy of Dermatology & Venereology 21: 1193–1198.
- 10. National Guideline Clearinghouse. Available: http://www.guideline.gov/about/index.aspx. Accessed 2012 Feb 15 and June 4.
- 11. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, et al. (2010) AGREE II: Advancing guideline development, reporting and evaluation in healthcare. CMAJ 182: E839–E842.
- 12. Canadian Agency for Drugs and Technologies in Health (2009) Optimal therapy recommendations for the prescribing and use of insulin analogues. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health.
- 13. Canadian Agency for Drugs and Technologies in Health (2010) Optimal therapy recommendations for the prescribing and use of second-line therapy for patients with diabetes inadequately controlled on metformin. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health.
- 14. National Collaborating Centre for Acute and Chronic Conditions (2009) Type 2 diabetes. The management of type 2 diabetes. London (UK): National Institute for Health and Clinical Excellence.
- 15. National Collaborating Centre for Women's and Children's Health (2008) Diabetes in pregnancy. Management of diabetes and its complications from pre-conception to the postnatal period. London (UK): National Institute for Health and Clinical Excellence.
- 16. Qatif Primary Health Care (2011) Cardiometabolic risk management guidelines in primary care. Qatif (Saudi Arabia): Qatif Primary Health Care.
- 17. Scottish Intercollegiate Guidelines Network (2010) Management of diabetes. A national clinical guideline. Edinburgh (Scotland): Scottish Intercollegiate Guidelines Network.
- 18. Department of Veteran Affairs, Department of Defense (2010) VA/DoD clinical practice guideline for the management of diabetes mellitus. Washington (DC): Department of Veteran Affairs, Department of Defense.
- 19. Institute for Clinical Systems Improvement (2012) Diagnosis and management of type 2 diabetes mellitus in adults. Bloomington (MN): Institute for Clinical Systems Improvement.
- 20. New York State Department of Health (2007) Prevention of secondary disease: diabetes. New York (NY): New York State Department of Health.
- 21. Brehove T, Joslyn M, Morrison S, Strehlow AJ, Wismer B (2007) Adapting Your Practice: Treatment and Recommendations for Homeless People with Diabetes Mellitus. Nashville: Health Care for the Homeless Clinicians' Network. 14 p.
- 22. Qaseem A, Humphrey LL, Chou R, Snow V, Shekelle P (2011) Use of intensive insulin therapy for the management of glycemic control in hospitalized patients: a clinical practice guideline from the American College of Physicians. Ann Intern Med 154: 260–267.
- 23. Qaseem A, Humphrey LL, Sweet DE, Starkey M, Shekelle P (2012) Oral pharmacologic treatment of type 2 diabetes mellitus: a clinical practice guideline from the American College of Physicians. Ann Intern Med 156: 218–231.
- 24. Norris SL, Holmer HK, Ogden LA, Selph SS, Fu R (2012) Conflict of Interest Disclosures for Clinical Practice Guidelines in the National Guideline Clearinghouse. PLoS ONE 7: e47343.
- 25. Norris SL, Holmer HK, Ogden LA, Burda BU (2011) Conflict of interest in clinical practice guideline development: a systematic review. PLoS ONE 6: e25153.
- 26. Neuman J, Korenstein D, Ross JS, Keyhani S (2011) Prevalence of financial conflicts of interest among panel members producing clinical practice guidelines in Canada and United States: cross sectional study. BMJ 343: d5621.
- 27. Mendelson TB, Meltzer M, Campbell EG, Caplan AL, Kirkpatrick JN (2011) Conflicts of interest in cardiovascular clinical practice guidelines. Arch Intern Med 171: 577–584.
- 28. Norris SL, Burda BU, Holmer HK, Ogden LA, Fu R, et al. (2012) Author's specialty and conflicts of interest contribute to conflicting guidelines for screening mammography. J Clin Epidemiol 65: 725–733.
- 29. Hogeveen SE, Han D, Trudeau-Tavara S, Buck J, Brezden-Masley CB, et al. (2012) Comparison of international breast cancer guidelines: are we globally consistent? Cancer Guideline AGREEment. Curr Oncol 19: e184–190.
- 30. Bennett WL, Odelola OA, Wilson LM, Bolen S, Selvaraj S, et al. (2012) Evaluation of guideline recommendations on oral medications for type 2 diabetes mellitus: a systematic review. Ann Intern Med 156: 27–36.
- 31. Greuter MJ, van Emmerik NM, Wouters MG, MW vT (2012) Quality of guidelines on the management of diabetes in pregnancy: a systematic review. BMC Pregnancy Childbirth 12: 58.
- 32. Stone MA, Wilkinson JC, Charpentier G, Clochard N, Grassi G, et al. (2010) Evaluation and comparison of guidelines for the management of people with type 2 diabetes from eight European countries. Diabetes Res Clin Pract 87: 252–260.
- 33. Mahmud M, Mazza D (2010) Preconception care of women with diabetes: a review of current guideline recommendations. BMC Womens Health 10: 5.
- 34. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, et al. (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7: 10.
- 35. Handelsman Y, Mechanick JI, Blonde L, Grunberger G, Bloomgarden ZT, et al. (2011) American Association of Clinical Endocrinologists medical guidelines for clinical practice for developing a diabetes mellitus comprehensive care plan. Endocrine Practice 17: 1–53.
- 36. American Diabetes Association (ADA) (2012) Standards of medical care in diabetes. Diabetes Care 35 (Suppl 1)S16–28.
- 37. American Medical Directors Association (2010) Diabetes management in the long-term care setting. Columbia (MD): American Medical Directors Association.
- 38. Canadian Agency for Drugs and Technologies in Health (2010) Third-line therapy for patients with type 2 diabetes inadequately controlled with metformin and a sulfonylurea. Ottawa (ON): Canadian Agency for Drugs and Technologies in Health.
- 39. Perk J, De Backer G, Gohlke H, Graham I, Reiner Z, et al. (2012) European Guidelines on cardiovascular disease prevention in clinical practice (version 2012). The Fifth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice. Eur Heart J 33: 1635–1701.
- 40. International Diabetes Center (2009) Type 2 diabetes. In: Prevention, detection and treatment of diabetes in adults 5th ed. Minneapolis (MN): International Diabetes Center.
- 41. International Diabetes Federation (2007) Guideline for management of postmeal glucose. Brussels, Belgium: International Diabetes Federation.
- 42. Joslin Diabetes Center (2007) Guideline for the care of the older adult with diabetes. Boston (MA): Joslin Diabetes Center.
- 43. National Kidney Foundation (2007) KDOQI clinical practice guidelines and clinical practice recommendations for diabetes and chronic kidney disease. NY (NY): National Kidney Foundation.
- 44. National Institute for Health and Clinical Excellence (2010) Liraglutide for the treatment of type 2 diabetes mellitus. London (UK): National Institute for Health and Clinical Excellence.
- 45. University of Michigan Health System (2009) Management of type 2 diabetes mellitus. Ann Arbor (MI): University of Michigan Health System.
- 46. Wisconsin Diabetes Prevention and Control Program (2011) Wisconsin diabetes mellitus essential care guidelines. Madison (WI): Wisconsin Diabetes Prevention and Control Program.