Guidelines traditionally focus on the diagnosis and treatment of single diseases. As almost half of the patients with a chronic disease have more than one disease, the applicability of guidelines may be limited. The aim of this study was to assess the extent that guidelines address comorbidity and to assess the supporting evidence of recommendations related to comorbidity.
We conducted a systematic analysis of evidence-based guidelines focusing on four highly prevalent chronic conditions with a high impact on quality of life: chronic obstructive pulmonary disease, depressive disorder, diabetes mellitus type 2, and osteoarthritis. Data were abstracted from each guideline on the extent that comorbidity was addressed (general comments, specific recommendations), the type of comorbidity discussed (concordant, discordant), and the supporting evidence of the comorbidity-related recommendations (level of evidence, translation of evidence). Of the 20 guidelines, 17 (85%) addressed the issue of comorbidity and 14 (70%) provided specific recommendations on comorbidity. In general, the guidelines included few recommendations on patients with comorbidity (mean 3 recommendations per guideline, range 0 to 26). Of the 59 comorbidity-related recommendations provided, 46 (78%) addressed concordant comorbidities, 8 (14%) discordant comorbidities, and for 5 (8%) the type of comorbidity was not specified. The strength of the supporting evidence was moderate for 25% (15/59) and low for 37% (22/59) of the recommendations. In addition, for 73% (43/59) of the recommendations the evidence was not adequately translated into the guidelines.
Our study showed that the applicability of current evidence-based guidelines to patients with comorbid conditions is limited. Most guidelines do not provide explicit guidance on treatment of patients with comorbidity, particularly for discordant combinations. Guidelines should be more explicit about the applicability of their recommendations to patients with comorbidity. Future clinical trials should also include patients with the most prevalent combinations of chronic conditions.
Citation: Lugtenberg M, Burgers JS, Clancy C, Westert GP, Schneider EC (2011) Current Guidelines Have Limited Applicability to Patients with Comorbid Conditions: A Systematic Analysis of Evidence-Based Guidelines. PLoS ONE 6(10): e25987. https://doi.org/10.1371/journal.pone.0025987
Editor: Peter McCulloch, University of Oxford, United Kingdom
Received: May 4, 2011; Accepted: September 15, 2011; Published: October 20, 2011
Copyright: © 2011 Lugtenberg et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Support was provided by The Commonwealth Fund (http://www.commonwealthfund.org/). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The views presented here are those of the authors and should not be attributed to The Commonwealth Fund or its directors, officers, or staff.
Competing interests: The authors have declared that no competing interests exist.
Traditionally, medical care is focused on the prevention, diagnosis and treatment of single diseases . Most research studies focus on the effectiveness of disease-specific interventions and patients with comorbidity or complex problems are often excluded from clinical trials , . In clinical practice, physicians are encouraged to adhere to evidence-based clinical practice guidelines (CPGs), as these are regarded as important tools for quality improvement . In line with both clinical practice and research traditions, most CPGs are disease-oriented documents focusing on the diagnosis and management of single diseases .
The emphasis of CPGs on single diseases may be problematic. Almost half of patients with chronic diseases have more than one disease , . Managing multiple conditions is more complex than managing single diseases and clinicians may find it challenging to provide optimal care for patients with multiple conditions –. Particularly when conditions are discordant, i.e. if they are not directly related in either their pathogenesis or management and do not share an underlying predisposing factor, patients are more likely to report conflicting instructions and problems with coordination of care –.
To the extent that CPGs focus on single diseases, they may offer insufficient guidance to physicians about care for patients with multiple conditions. Lack of applicability of CPGs due to comorbidity may pose an important barrier to guideline adherence among physicians , . Moreover, adhering to single disease CPGs in caring for patients with multiple conditions may adversely affect patient safety, if recommended treatments for one condition conflict with those for another condition .
Although prior studies suggest that physicians may find it challenging to provide care to patients with comorbidity, there are few systematic assessments of the comorbidity-related content of CPGs, and in particular the quality of the evidence that supports that content. The aim of this study was to explore the applicability of CPGs to patients with comorbidity by assessing the extent to which CPGs on high-prevalence chronic conditions address comorbidity and by assessing the quality of the evidence cited in support of recommendations related to comorbidity.
Two publicly-available international databases, the National Guideline Clearinghouse (NGC) and the Guidelines International Network Library (G-I-N),were used to select the guidelines.
Selection of chronic conditions.
In selecting the conditions, we focused on highly prevalent chronic diseases that have a high impact on quality of life. Both major depressive disorder ,  and diabetes mellitus type 2 ,  are highly prevalent and have been found to have a high impact on quality of life, particularly in combination , . We also included chronic obstructive pulmonary disease (COPD) and osteoarthritis, as pain and dyspnea may have a considerable impact on quality of life as well.
Selection of clinical practice guidelines
Guidelines were included if they:
- included a set of recommendations with an explicit link to their supporting evidence;
- were published in 2005 or later;
- addressed the treatment or management of the selected conditions;
- were published in English;
- were accessible in the public domain.
CPGs were excluded if they focused on a specific subgroup of patients (e.g. pregnant women, children, adolescents, homeless people).
One of the investigators (ML) abstracted data from the selected CPGs and the abstraction process was checked by a second investigator (JB). Any disagreement was resolved by discussion. General data were retrieved from the CPGs, and more detailed information was collected on the specific recommendations addressing comorbidity and their supporting evidence:
- General characteristics of the guideline: title; organization; country; target group; year of publication; number of pages and references; number of treatment recommendations.
- Characteristics of the guideline related to comorbidity: issue of comorbidity addressed (prevalence data, screening/diagnosing for comorbidity; considering comorbidity in treatment); discussion of patient-centered aspects (such as goals and burden of treatment, incorporating patient preferences), inclusion of specific comorbidity-related treatment recommendations (number and proportion). A recommendation was defined as a statement whose apparent intent is to provide guidance about the advisability of a clinical action . Contra-indications for medication or surgery were not considered as specific comorbidity-related recommendations, if no alternative treatments were provided.
- Type of recommendation: type of treatment addressed (general treatment, drug therapy, life-style advice, surgery, other); inclusion of patient-centered aspects.
- Number of comorbid conditions addressed;
- Type of comorbidity addressed: concordant or discordant. Concordant conditions were defined as representing the same overall pathophysiological risk profile and being more likely to be the focus of the same disease and self management plan . Discordant treatments are not directly related in either their pathogenesis or management. For each of the included conditions the authors developed a scheme of concordant and discordant comorbidities ( File S1). For diabetes, we did not consider cardiovascular risk factors such as hypertension and hyperlipidemia as concordant conditions but as part of the disease, because adequate management of diabetes is cardiovascular risk management including monitoring blood pressure and lipids.
- Link with underlying evidence described; (yes, no)
- Number of underlying studies;
- Level of evidence of underlying studies: high, moderate, low, not available. As grading systems differ per guideline, we considered the highest level of evidence as high, the lowest level as low, and intermediate levels as moderate.
- Translation of evidence: good, moderate or poor/unclear. Our judgment was based on the directness of the evidence and on whether the strengths and limitations of the evidence were discussed in the guideline. The translation was graded as: ‘good’ if the supporting evidence of the studies focused (at least partly) on the comorbidity part of the recommendation and the strengths and limitations of the supporting evidence were discussed in the guideline; as ‘moderate’ if either the supporting evidence of the studies focused (at least partly) on the comorbidity part of the recommendation or the strengths and limitations of the supporting evidence were discussed in the guideline; and as ‘poor or unclear’ if neither the supporting evidence of the studies focused on the comorbidity part of the recommendation nor were the strengths and limitations of the supporting evidence discussed in the guideline.
A total of 20 CPGs met our inclusion criteria, having been published in English and in the public domain since 2005 (Table 1). Six of the CPGs addressed COPD, four addressed major depressive disorder, seven addressed diabetes mellitus type 2 and three addressed osteoarthritis.
Eight CPGs were retrieved from the G-I-N database, six from the NGC database and six were available in both databases. The largest share of these 20 CPGs was produced in the United States (n = 7). Nine CPGs were produced by governmental agencies; five by professional societies and six by other types of organizations. The CPGs were predominantly developed in 2008 (7/20) and in 2007 (5/20).
Applicability of guidelines to patients with comorbidity
Of the 20 guidelines, 17 (85%) addressed the issue of comorbidity (Table 2). Eight guidelines (40%) provided comorbidity prevalence data, 16 guidelines (80%) recommended screening for comorbid conditions and 17 guidelines (85%) recommended considering comorbidity in treatment. Guidelines on depressive disorder and diabetes mellitus type 2 (100%) more often addressed the issue of comorbidity compared to the guidelines on COPD (83%) and osteoarthritis (33%).
Fourteen (70%) guidelines provided specific treatment recommendation for patients with comorbid conditions. The number of recommendations varied from 1 to 26 per guideline, with an average of 3 per guideline. The guidelines on COPD and osteoarthritis provided the fewest numbers of recommendations (0.7 per guideline), whereas the guidelines on diabetes mellitus type 2 included an average of 6.3 comorbidity-related recommendations.
The 20 guidelines provided a total of 59 comorbidity-related treatment recommendations (Table 3). Seventy-eight percent (46/59) of these recommendations addressed concordant comorbidities. Most of the diabetes mellitus type 2 guideline recommendations addressed concordant comorbidities such as coronary artery disease and heart failure. Relative to the other guidelines, the guidelines on depressive disorder included the largest proportion (33%) of recommendations on discordant comorbidities (such as cardiovascular disease). More than 90% of the recommendations were related to one comorbid condition; 10% focused on comorbidities in general and none of the recommendations specified the management of patients with more than one comorbid condition.
Fifty-four percent of the comorbidity-related recommendations concerned drug therapy (32/59); 25% related to other types of treatment such as psychotherapy or oxygen therapy (15/59). Few recommendations focused on surgery (10%; 6/59) and on life-style advice (3%; 2/59). Twelve percent of the recommendations (7/59) provided specific guidance on patient-centered aspects such as patient preferences, burden of disease and priority setting.
The link between guideline recommendation statements and the supporting evidence was described for 97% of the recommendations (57/59). The number of underlying studies varied between 1 and 12 per recommendation. The level of evidence of the studies was generally weak: 37% of the recommendations (22/59) had a ‘low’ level of evidence; for 25% of the recommendations (15/59) the level of evidence was described as ‘moderate’ (Table 4 and 5).
For 73% of the recommendations (43/59), the evidence underlying the studies was not adequately translated into the guideline with 48% (28/59) graded as ‘moderate’ and 25% (15/59) as ‘poor or unclear’ (Table 4 and 5). Translation of evidence was rated more frequently as ‘good’ for guidelines on diabetes mellitus type 2 (32% [14/44]) than those on depression (22% [2/9]); none of the guidelines on COPD and osteoarthritis received a ‘good’ rating for evidence translation (Table 4).
Patients with multiple comorbid conditions are very frequently encountered in clinical practice. However, our results suggest that evidence-based guidelines on four relatively prevalent chronic diseases may have limited applicability to patients with comorbid conditions. Most of these guidelines do not provide explicit guidance on treatment of patients with specific combinations of diseases. If comorbidity is addressed in the guidelines, it is often discussed in general; few specific treatment recommendations for patients with comorbid conditions are provided, particularly for discordant combinations. Moreover, the evidence supporting the available comorbidity-related recommendations was generally limited, had moderate to poor quality, and was often not adequately translated into the guidelines.
Among the guidelines in our study that included specific comorbidity-related recommendations, these recommendations were more likely to focus on concordant comorbidities with related treatment plans. We also found that none of the comorbidity-related recommendations specified the preferred action for patients with more than one concurrent condition. These results are consistent with previous American  and Australian  studies showing that guidelines pay little attention to patients with discordant comorbidities and to patients with multiple chronic conditions. This lack of attention contributes to limiting the applicability of single disease guidelines on patients with chronic diseases as almost one third of them have three or more conditions .
An important finding of our study is the limited evidence base that supports comorbidity-related recommendations. If specific recommendations for patients with comorbidity are provided, they are often based on limited evidence that is of moderate or poor quality. In addition, the supporting evidence rarely focuses directly on the groups of patients with comorbid conditions. Furthermore, the limitations of this evidence are not usually described in the guidelines. The failure to describe limitations of evidence in a guideline could give clinicians misplaced confidence in guideline recommendations.
Consistent with previous studies, our findings indicate that the evidence base for patients with multiple chronic conditions is limited , . The lack of evidence specific to comorbid conditions may explain the limited attention to comorbidity in the guidelines we studied. If future clinical trials included patients with comorbid conditions, at least for the most common combination of diseases and report the results, this would provide the evidence base that clinical guideline developers need , .
In light of the general absence of research evidence on patients with multiple conditions, guidelines should be more explicit about the applicability of their recommendations to patients with the most prevalent comorbid conditions and discuss the quality and directness of the evidence for these patients. This explicit approach should replace the implicit assumption that guideline recommendations are applicable to patients with comorbid conditions unless conflicting evidence is available , .
Our findings indicate that no systematic approach is used by guideline development groups for addressing comorbidity in guidelines. Compared to the guidelines on COPD, depressive disorder, and osteoarthritis, the guidelines on diabetes mellitus type 2 had better reporting of issues of comorbidity. Even for guidelines on the same condition, we found large variation between guidelines in the approach to addressing comorbidity. This applies to all levels of abstraction (guideline, recommendation, evidence). A previous study comparing diabetes guidelines from different countries, also found much variation in the supporting evidence, whereas the recommendations were similar . It would be helpful to develop guidance, as part of a handbook or manual for guideline developers ,  to facilitate and support this process and to create more uniformity. In addition, targeting educational activities to professional societies that do not yet incorporate comorbidity to a large extent in their guidelines might be useful.
The main strength of our study is that we systematically assessed the content of an international sample of evidence-based national and international guidelines in terms of addressing comorbidity. The guidelines included in our study are among the best in the clinical areas of interest and were produced by prominent governmental agencies or professional organizations. Furthermore, by simultaneously assessing the underlying evidence of the comorbidity-related recommendations, we were able to determine whether guidance was provided on treatment of patients with comorbid conditions and also to what extent this guidance was based on high-quality evidence.
Our study has several limitations. First, a limited number of chronic conditions were included in our study. Inclusion of a different set of chronic conditions could have yielded different results. However, we do not expect guidelines on other diseases to be more applicable to patients with multiple conditions than those for the included common conditions. Second, the number of selected guidelines varied between the conditions, with an overrepresentation of diabetes guidelines. This reflects the available number of high-quality guidelines on the selected diseases in the databases. Third, we did not assess all available comorbidity-related evidence for the included chronic conditions, but only the evidence that was described in the guidelines. A systematic search for evidence would be necessary to determine whether the guideline recommendations are based on the best available evidence. Future research on the selected conditions could be useful to draw firm conclusions on the availability of evidence for patients with multiple conditions, complementing the findings of our study.
Among a selected set of high-quality current evidence-based guidelines on prevalent chronic diseases, there is limited guidance on treatment of patients with comorbid conditions. Although the issue of comorbidity is recognized by guidelines, very few specific recommendations are provided and these are generally based on limited evidence of low or moderate quality. The supporting evidence often does not focus directly on groups of patients with comorbid conditions and it is rare that guidelines adequately describe the limitations of the evidence. Given the increasing prevalence of patients with multiple chronic diseases, guidelines should at least be explicit and transparent about the applicability of their recommendations to populations of patients with the most common combination of diseases. A guide for guideline developers could facilitate a systematic and uniform approach.
The authors would like to thank Klara Brunnhuber, MD, PhD (BMJ publishing group) and James Woodcock MD, MSc (London School of Hygiene and Tropical Medicine) for providing useful comments on our manuscript.
Conceived and designed the experiments: ML JB CC GW ES. Analyzed the data: ML JB. Wrote the paper: ML JB CC GW ES.
- 1. Tinetti ME, Fried T (2004) The end of the disease era. Am J Med 116: 179–185.ME TinettiT. Fried2004The end of the disease era.Am J Med116179185
- 2. Gross CP, Mallory R, Heiat A, Krumholz HM (2002) Reporting the recruitment process in clinical trials: who are these patients and how did they get there? Ann Intern Med 137: 10–16.CP GrossR. MalloryA. HeiatHM Krumholz2002Reporting the recruitment process in clinical trials: who are these patients and how did they get there?Ann Intern Med1371016
- 3. Guyatt GH, Sackett DL, Cook DJ (1994) Users' guides to the medical literature: II. How to use an article about therapy or prevention: B. What were the results and will they help me in caring for my patients? JAMA 271: 59–63.GH GuyattDL SackettDJ Cook1994Users' guides to the medical literature: II. How to use an article about therapy or prevention: B. What were the results and will they help me in caring for my patients?JAMA2715963
- 4. Grol R (2001) Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA 286: 2578–2585.R. Grol2001Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction.JAMA28625782585
- 5. Van Weel C, Schellevis FG (2006) Comorbidity and guidelines: conflicting interests [Comment]. Lancet 367: 550–551.C. Van WeelFG Schellevis2006Comorbidity and guidelines: conflicting interests [Comment].Lancet367550551
- 6. Hoffman C, Rice D, Sung HY (1996) Persons with chronic conditions: their prevalence and costs. JAMA 276: 1473–1479.C. HoffmanD. RiceHY Sung1996Persons with chronic conditions: their prevalence and costs.JAMA27614731479
- 7. Schoen C, Osborn R, Doty M, Bishop M, Peugh J, Murukutla N (2007) Toward higher-performance health systems: adults' health care experiences in seven countries. Health Aff 26: w717–734.C. SchoenR. OsbornM. DotyM. BishopJ. PeughN. Murukutla2007Toward higher-performance health systems: adults' health care experiences in seven countries.Health Aff26w717734
- 8. Wolff JL, Starfield B, Anderson G (2002) Prevalence, expenditures, and complications of multiple chronic conditions in the elderly. Arch Intern Med 162: 2269–2276.JL WolffB. StarfieldG. Anderson2002Prevalence, expenditures, and complications of multiple chronic conditions in the elderly.Arch Intern Med16222692276
- 9. Vogeli C, Shields AE, Lee TA, Gibson TB, Marder WD, et al. (2007) Multiple chronic conditions: prevalence, health consequences, and implications for quality, care management, and costs. J Gen Intern Med 22: S391–395.C. VogeliAE ShieldsTA LeeTB GibsonWD Marder2007Multiple chronic conditions: prevalence, health consequences, and implications for quality, care management, and costs.J Gen Intern Med22S391395
- 10. Fortin M, Soubhi H, Hudon C, Bayliss EA, Van den Akker M (2007) Multimorbidity's many challenges. BMJ 334: 1016–1017.M. FortinH. SoubhiC. HudonEA BaylissM. Van den Akker2007Multimorbidity's many challenges.BMJ33410161017
- 11. Redelmeier DA, Tan SH, Booth GL (1998) The treatment of unrelated disorders in patients with chronic medical diseases. N Engl J Med 338: 1516–1520.DA RedelmeierSH TanGL Booth1998The treatment of unrelated disorders in patients with chronic medical diseases.N Engl J Med33815161520
- 12. Piette JD, Kerr EA (2006) The impact of comorbid chronic conditions on diabetes care. Diabetes Care 29: 725–731.JD PietteEA Kerr2006The impact of comorbid chronic conditions on diabetes care.Diabetes Care29725731
- 13. Burgers JS, Voerman GE, Grol R, Faber MJ, Schneider EC (2010) Quality and coordination of care for patients with multiple conditions: results from an international survey of patient experience. Eval Health Prof 33: 343–364.JS BurgersGE VoermanR. GrolMJ FaberEC Schneider2010Quality and coordination of care for patients with multiple conditions: results from an international survey of patient experience.Eval Health Prof33343364
- 14. Francke A, Smit M, De Veer A, Mistiaen P (2008) Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak 8: 38.A. FranckeM. SmitA. De VeerP. Mistiaen2008Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review.BMC Med Inform Decis Mak838
- 15. Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS (2009) Why don't physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. Implement Sci 4: 54.M. LugtenbergJM Zegers-van SchaickGP WestertJS Burgers2009Why don't physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners.Implement Sci454
- 16. Boyd CM, Darer J, Boult C, Fried LP, Boult L, et al. (2005) Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA 294: 716–724.CM BoydJ. DarerC. BoultLP FriedL. Boult2005Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance.JAMA294716724
- 17. Moussavi S, Chatterji S, Verdes E, Tandon A, Patel V, et al. (2007) Depression, chronic diseases, and decrements in health: results from the World Health Surveys. Lancet 370: 851–858.S. MoussaviS. ChatterjiE. VerdesA. TandonV. Patel2007Depression, chronic diseases, and decrements in health: results from the World Health Surveys.Lancet370851858
- 18. Alonso J, Angermeyer MC, Bernert S, Bruffaerts TS, Brugha TS, et al. (2004) Disability and quality of life impact of mental disorders in Europe: results from the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psychiatr Scand 109: 38–46.J. AlonsoMC AngermeyerS. BernertTS BruffaertsTS Brugha2004Disability and quality of life impact of mental disorders in Europe: results from the European Study of the Epidemiology of Mental Disorders (ESEMeD) project.Acta Psychiatr Scand1093846
- 19. Rubin RR, Peyrot M (1999) Quality of life and diabetes. Diabetes Metab Res Rev 15: 205–218.RR RubinM. Peyrot1999Quality of life and diabetes.Diabetes Metab Res Rev15205218
- 20. De Grauw WJC, Van de Lisdonk EH, Behr RRA, Van Gerwen WHEM, Van den Hoogen HJM, et al. (1999) The impact of type 2 diabetes mellitus on daily functioning. Fam Pract 16: 133–139.WJC De GrauwEH Van de LisdonkRRA BehrWHEM Van GerwenHJM Van den Hoogen1999The impact of type 2 diabetes mellitus on daily functioning.Fam Pract16133139
- 21. Schram MT, Baan CA, Pouwer F (2009) Depression and quality of life in patients with diabetes: a systematic review from the European depression in diabetes (EDID) research consortium. Curr Diabetes Rev 5: 112–119.MT SchramCA BaanF. Pouwer2009Depression and quality of life in patients with diabetes: a systematic review from the European depression in diabetes (EDID) research consortium.Curr Diabetes Rev5112119
- 22. Hussain T, Michel G, Shiffman RN (2009) The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines. Int J Med Inform 78: 354–363.T. HussainG. MichelRN Shiffman2009The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines.Int J Med Inform78354363
- 23. Vitry AI, Zhang Y (2008) Quality of Australian clinical guidelines and relevance to the care of older people with multiple comorbid conditions. Med J Aust 189: 360–366.AI VitryY. Zhang2008Quality of Australian clinical guidelines and relevance to the care of older people with multiple comorbid conditions.Med J Aust189360366
- 24. Schoen C, Osborn R, How SKH, Doty MM, Puegh J (2008) In chronic condition: experiences of patients with complex health care needs, in eight countries. Health Aff 28: w1–16.C. SchoenR. OsbornSKH HowMM DotyJ. Puegh2008In chronic condition: experiences of patients with complex health care needs, in eight countries.Health Aff28w116
- 25. Gurwitz JH, Col NF, Avorn J (1992) The exclusion of the elderly and women from clinical trials in acute myocardial infarction. JAMA 268: 1417–1422.JH GurwitzNF ColJ. Avorn1992The exclusion of the elderly and women from clinical trials in acute myocardial infarction.JAMA26814171422
- 26. Altman DG, Bland JM (1995) Absence of evidence is not evidence of absence. BMJ 311: 485.DG AltmanJM Bland1995Absence of evidence is not evidence of absence.BMJ311485
- 27. Alderson P (2004) Absence of evidence is not evidence of absence. BMJ 328: 476–477.P. Alderson2004Absence of evidence is not evidence of absence.BMJ328476477
- 28. Burgers JS, Bailey JV, Klazinga NS, Van der Bij AK, Grol R, et al. (2002) Inside Guidelines. Diabetes Care 25: 1933–1939.JS BurgersJV BaileyNS KlazingaAK Van der BijR. Grol2002Inside Guidelines.Diabetes Care2519331939
- 29. Scottish Intercollegiate Guidelines Network (SIGN) (2008) SIGN 50: A guideline developer's handbook. Revised edition. Edinburgh: Scottish Intercollegiate Guidelines Network. Scottish Intercollegiate Guidelines Network (SIGN)2008SIGN 50: A guideline developer's handbook. Revised editionEdinburghScottish Intercollegiate Guidelines Network
- 30. National Institute for Health and Clinical Excellence (NICE) (2009) The guidelines manual 2009. London: National Institute for Health and Clinical Excellence. National Institute for Health and Clinical Excellence (NICE)2009The guidelines manual 2009LondonNational Institute for Health and Clinical Excellence