Figures
Abstract
Background
Annually, non-communicable diseases (NCDs) kill 38 million people worldwide, with low and middle-income countries accounting for three-quarters of these deaths. High-quality clinical practice guidelines (CPGs) are fundamental to improving NCD management. The present study evaluated the methodological rigor and transparency of Brazilian CPGs that recommend pharmacological treatment for the most prevalent NCDs.
Methods
We conducted a systematic search for CPGs of the following NCDs: asthma, atrial fibrillation, benign prostatic hyperplasia, chronic obstructive pulmonary disease, congestive heart failure, coronary artery disease and/or stable angina, dementia, depression, diabetes, gastroesophageal reflux disease, hypercholesterolemia, hypertension, osteoarthritis, and osteoporosis. CPGs comprising pharmacological treatment recommendations were included. No language or year restrictions were applied. CPGs were excluded if they were merely for local use and referred to NCDs not listed above. CPG quality was independently assessed by two reviewers using the Appraisal of Guidelines Research and Evaluation instrument, version II (AGREE II).
Main Findings
“Scope and purpose” and “clarity and presentation” domains received the highest scores. Sixteen of 26 CPGs were classified as low quality, and none were classified as high overall quality. No CPG was recommended without modification (77% were not recommended at all). After 2009, 2 domain scores (“rigor of development” and “clarity and presentation”) increased (61% and 73%, respectively). However, “rigor of development” was still rated < 30%.
Conclusion
Brazilian healthcare professionals should be concerned with CPG quality for the treatment of selected NCDs. Features that undermined AGREE II scores included the lack of a multidisciplinary team for the development group, no consideration of patients’ preferences, insufficient information regarding literature searches, lack of selection criteria, formulating recommendations, authors’ conflict of interest disclosures, and funding body influence.
Citation: Molino CdGRC, Romano-Lieber NS, Ribeiro E, de Melo DO (2016) Non-Communicable Disease Clinical Practice Guidelines in Brazil: A Systematic Assessment of Methodological Quality and Transparency. PLoS ONE 11(11): e0166367. https://doi.org/10.1371/journal.pone.0166367
Editor: Siew Hua Gan, Universiti Sains Malaysia, MALAYSIA
Received: July 8, 2016; Accepted: October 27, 2016; Published: November 15, 2016
Copyright: © 2016 Molino et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: CGRCM is supported by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), process number 164700/2015-3. The funding source did not influence the content of this study.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Non-communicable diseases (NCDs) are considered major public health problems, strongly affecting the poorest and most vulnerable population groups [1,2]. NCDs kill 38 million people worldwide each year, and 28 million of these deaths occur in low and middle-income countries. Due to NCD healthcare burdens, the World Health Organization (WHO) has emphasized a need for improving NCD treatment [1–3].
High-quality clinical practice guidelines (CPGs) are fundamental for improving healthcare management, as they are special tools that translate scientific research findings, provide explicit recommendations, and support evidence-based decision making [4,5]. CPG implementation is essential for improving NCD control and patient quality of life, as well as optimizing drug utilization and healthcare resources [6–8]. However, several studies have identified CPGs that suffer from low to moderate quality, which call into question the reliability of such measures among healthcare professionals and managers [9–18].
Several tools have been developed worldwide for evaluating CPG quality [19,20]. For instance, the Appraisal of Guidelines Research and Evaluation instrument [21] version II (AGREE II), published in 2009, has been extensively used, validated in several languages, and covers essential information for comprehensive CPG evaluation [19,20]. Several studies worldwide have been conducted for assessing CPG quality using the AGREE II [9–11,16–18,22–25]; however, very little is known regarding CPG quality among low income countries [10]. To date, only one study has evaluated CPG quality in Brazil [18], and none have critically assessed CPGs for NCD treatment quality within a Brazilian sample. Thus, the present study evaluated the methodological rigor and transparency of Brazilian CPGs that recommend pharmacological treatment for the most prevalent NCDs.
Methods
Search
The following chronic conditions were selected: asthma, atrial fibrillation, benign prostatic hyperplasia, chronic obstructive pulmonary disease, congestive heart failure, coronary artery disease and/or stable angina, dementia, depression, diabetes, gastroesophageal reflux disease, hypercholesterolemia, hypertension, osteoarthritis, and osteoporosis. These conditions were selected given their high observed prevalence within primary care settings [26–34]. We also included CPGs for rheumatic arthritis and Alzheimer´s disease given their high relevance for elderly adults.
We conducted a comprehensive literature search on October 30, 2015: MEDLINE (by PubMed) [35], LILACS (by Virtual Health Library website) [36], and Cochrane Library [37]. On September 9, 2015, we searched the National Guideline Clearinghouse [38] database. S1 Table contains the full search strategies used for all databases.
We also searched Google for the following (on September 9, 2015): “clinical guidelines” or “therapeutic guidelines” or “clinical protocol” or “clinical practice guidelines” and “Brazil” for each NCD. Finally, we searched for CPGs on the Brazilian Ministry of Health website (on October 31, 2015) available at: http://portalsaude.saude.gov.br/index.php/o-ministerio/principal/leia-mais-o-ministerio/840-sctie-raiz/daf-raiz/cgceaf-raiz/cgceaf/l3-cgceaf/11646-pcdt.
Guideline selection
A CPG was defined according to the Institute of Medicine: “clinical practice guidelines are statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” ([39], p4). Inclusion criteria were CPGs that comprised pharmacological treatment recommendations. No language or year restrictions were applied. Exclusion criteria included CPGs without pharmacological treatment recommendations, were only for local use, and/or referred to chronic conditions not already mentioned.
Two independent reviewers first assessed paper titles and abstracts for potential eligibility. Second, reviewers independently screened each full-text article for inclusion criteria. Finally, we checked medical society and Brazilian Ministry of Health websites for any supplemental material. Discrepancies at any stage were resolved through discussion between the two reviewers. When necessary, a third reviewer was included.
Data extraction and quality appraisal
One reviewer performed the data extraction. Next, a second reviewer checked the extraction. Included CPGs were assessed according to specific NCDs, publication year, publisher, guideline type (formulated or adapted), guideline references, funding, and quality appraisal.
CPG quality was assigned using the AGREE II instrument [21]. This instrument was chosen because it is a validated tool in Brazil [40] and is widely used [10,20,41]. The AGREE II comprises 23 items, which are grouped into 6 domains, as well as 2 overall assessment items that allow reviewers to rate overall CPG quality and recommend its use [42]. Items are rated on a 7-point scale: 1 (strongly disagree; there is no relevant information or the concept is poorly described) to 7 (strongly agree; quality of information is excellent and all criteria listed in the AGREE II User’s Manual are met) [21]. Reviewers judged the overall CPG quality taking into account all 23 items in accordance with the AGREE II User’s Manual [21].
Two independent reviewers (CGRCM and DOM) assessed CPG quality. The reviewers were trained on the AGREE II instrument as described in S1 Appendix. The reviewers had not previously authored any CPGs. Discrepancies in ratings were verified by calculating Kappa coefficients using VassarStats [43] for each CPG. Kappa coefficients were considered discrepant when ≤0.4, based on Landis and Koch’s criteria [44]. For discrepant CPGs, we identified domains that revealed high discrepancy using a concordance calculator developed by McMaster University [45]. The reviewers then discussed and independently reviewed high discrepant domains. Finally, domain scores were calculated according to the AGREE II User’s Manual.
Analyses
According to the AGREE II User’s Manual, domain scores must not be calculated as a single quality score. The scores of each domain are determined by the total item score percentage with reference to the range between maximum and minimum scores of that particular domain [42].
Despite having no validated method for an overall classification [42], we decided to adopt a metric for overall CPG quality, as shown in S1 Fig. We prioritized domain 3 for classifying the overall quality since this domain evaluates methodological rigor during CPG development. Other studies have used different classification metrics for determining overall quality [16,46,47]. Then, to differentiate CPGs’ quality regarding other domain scores, we divided it into high, moderate, and low categories with A to C grading. Thus, overall quality was divided into 9 types according to domain 3 and 2 other domains scores (S1 Fig).
Mann-Whitney test was used to test significant difference in domain scores between CPG published before and in/after 2009 (AGREE II publication year). P-values less than 0.05 were considered statistically significant.
Results
The search strategy retrieved 661 records, of which 58 were considered for full-text screening; 26 met our eligibility criteria and were assessed using the AGREE II (Fig 1). See S2 Table for excluded CPG details.
Most of the included CPGs were published in/after 2009 (85%). None conducted a systematic review of the literature, explicitly declared funding sources, or claimed to be an adapted CPG. Table 1 shows general characteristics and AGREE II scores of the included CPGs. Few CPGs were published by the Ministry of Health (6; 23%). Regarding the chronic conditions reviewed, only guidelines for dementia treatment were not found. Diabetes mellitus CPGs were the most common. Only two domains (“scope and purpose” and “clarity and presentation”) obtained scores greater than 60%. Most domains scored less than 30%.
Domain scores >60% are highlighted in bold and CPGs are sorted according to domain 3 score (from highest to lowest in each disease category).
Fig 2 shows that most CPGs were classified as low quality (16; 62%). None were classified as high quality.
In terms of whether reviewers would recommend implementing the CPG, we observed that none should be recommended without modifications, and 77% should not be recommended at all. Only CPGs from the Ministry of Health were recommended contingent on modifications.
Improvement in domain scores was observed when comparing CPGs published before and in/after 2009 (Fig 3). However, only 2 domain scores (“rigor of development” and “clarity and presentation”) demonstrated a significant improvement (p < 0.05).
(*) Statistical significant was observed for “rigor of development” (p = 0.03) and “clarity and presentation (p = 0.01).”
Discussion
We performed a systematic search of Brazilian CPGs for treating the most prevalent NCDs within primary care settings. Except for dementia, we identified 26 CPGs, with diabetes mellitus as the most common. Most CPGs had been created in the last 5 years; yet, none were of particularly high quality, and most were not recommended by our reviewers. Specifically, 62% of the selected CPGs were deemed “low quality”: low-A (8%), low-B (39%), and low-C (15%). In essence, most of the CPGs scored less than 30% on the “rigor of development” domain. This domain evaluated how evidence for the CPG was assembled and synthesized, how recommendations were formulated, and how the CPG would be updated. Given the deficiencies within this metric, we can conclude poor quality of our selected CPGs. Similar to our findings, Chinese CPGs have been evaluated for poor quality, particularly presenting low scores on the “rigor of development” domain [74–77].
Another important finding was that CPGs published in/after 2009 (AGREE II publication) were rated more positively within 2 domains: “rigor of development” and “clarity and presentation.” One prior study evaluating CPGs published between 1980 and 2007 observed improvement over time in most domains [10]. However, domain scores were still moderate to low [10]. In contrast to these findings, “scope and purpose,” “stakeholder involvement,” and “applicability” domains did not improve post-2009 in the present study. These differences might be explained, in part, by sample size issues, country of origin, and publication year. While we only evaluated 26 Brazilian CPGs, the aforementioned study assessed 42 CPGs across different continents. Additionally, we believe that international CPGs obtain greater scores because the AGREE instrument has been used worldwide since 2003 [20]. The first AGREE version was not validated in Brazil, and AGREE II was not available until 2009. It is interesting to note that only Ministry of Health guidelines were recommended by our reviewers. This result may be explained by the fact that the Ministry of Health guidelines presented better scores in the “rigor of development” domain, with a minimum of 41%. In addition, these guidelines reported AGREE use for developing CPGs, which may be associated with higher quality when compared to others.
Independent from publication year, “scope and purpose” and “clarity and presentation” were more highly rated; however, there is still room for improvement within these domains owing to the presence of scores lower than 30% and between 30 and 60%. As previously reported [10,17], these domains can be ameliorated by describing the target population and health questions, providing clear summaries, and standardizing the CPG presentation form.
“Stakeholder involvement” and “applicability” were the lowest scoring domains, accruing scores < 30%. Other studies [9,10,17,75] have also reported poor quality among these domains. We observed that the “stakeholder involvement” domain did not include consideration of patients’ perspectives, and developer groups were not multidisciplinary. With respect to the “applicability’ domain,” most CPGs did not evaluate resource implications and organizational barriers when applying recommendations, which has also been mentioned previously [9,10,17,75].
Several barriers affect physician adherence to CPG recommendations [78]. CPG developers should explore alternative tools for promoting adherence among healthcare providers. Tools, such as implementation instructions, flowcharts, digital books, and an outline of key recommendations, might be useful for promoting and disseminating CPGs [79]. This should improve “applicability” scores. Additionally, actions toward intensifying healthcare professionals’ participation in CPG development may also improve recommendation adherence [80,81], along with important domain scores. In Brazil, most CPG developers did not create multidisciplinary teams. For instance, specific medical societies only included specialty physicians, while the Brazilian Medical Association included some physicians from different expertise areas. Only the Ministry of Health included a multidisciplinary team and conducted public consultations. Furthermore, insufficient methods for procuring stakeholder views were reported. For instance, only the Ministry of Health’s guidelines conducted public consultations during the final step of guideline development. Few CPGs have conducted a literature search on stakeholder views. More effective CPG creation would likely include patients and patient representatives within the development group [82]. In accordance with our findings, a prior study analyzed 100 endocrine CPGs from North America and demonstrated that only 3 considered patient perspectives and included patients during CPG development [9].
It is interesting to note that all 26 CPGs reviewed presently were undermined by poor rigor of development, even when only analyzing guidelines published in/after 2009. Possible explanations for this result include: 1) no study performed a systematic review or adapted a high-quality CPG; 2) few detailed a specific search strategy and selection criteria; 3) few described the evidence considered and recommendations based on that evidence; and 4) a GRADE system was not used for most of the CPG recommendations. These problems directly affected the “rigor of development” score and overall CPG quality and reliability. Previous reports have noted that most CPGs do not perform a systematic literature search [10,83], as this can be very time-consuming. The AGREE II does not require a systematic review for a full score domain; however, to improve upon specific domain scores (namely “rigor of development,”), CPGs must explicitly describe their selection criteria and the type of evidence considered [42]. Low “rigor of development” scores might also be related to the various CPG versions that have been published (digital versions as eBooks, full vs. summary versions, etc.) [18]. For instance, the Ministry of Health provides partial methods sections and mentions specific CPG books. Medical societies often publish CPGs in scientific journals and digital books comprising flowcharts and recommendation summaries; however, most do not present a methods section. The Brazilian Medical Association provides digital books, including a general methods section for all CPGs published. Yet, no information regarding criteria selection and the formulation of recommendations is provided.
The “editorial independence” median score was 35.5% (interquartile range, 21–42%). Although most CPGs disclose authors’ competing interests, they did not address how identified conflicts would be managed. Moreover, none clearly disclosed funding sources. Thus, these results need to be interpreted with caution. AGREE II scores rely on CPG text; therefore, we believe that competing interests were properly assessed. A more careful description was lacking. Furthermore, low scores within this domain have been reported elsewhere [84]. Scores range from 16% among Chinese CPGs for hypertension to 30% for a separate analysis of 626 CPGs, and 41% for European CPGs specific to chronic diseases [10,17,75]. Moreover, a review of 250 CPGs observed the absence of author disclosures (40%), unavailable author disclosures within the public domain (42%), and at least 60% provided at least one author with a conflict [84].
In order to improve CPG quality, we suggest focusing on the following: 1) assembling a multidisciplinary development group; 2) take into account patients’ preferences; 3) describe literature search details, selection criteria, and the formulation of recommendations; and 4) explicitly declare any competing or financial interests among the authors. Finally, we believe high-quality CPGs will likely require a partnership between health institutions and universities with CPG development expertise. Previous work in Saudi Arabia has suggested that collaborative work between the Ministry of Health and McMaster University enabled the production of 10 CPGs, with 80 recommendations, within 4 months [85]. Thus, knowing that the Brazilian health system is a reference in Latin America, the Caribbean, and Portuguese-speaking African countries, the study of Brazilian CPGs is essential for healthcare professionals in many other countries. Health institutions should identify universities with CPG expertise and work collaboratively toward promoting high-quality CPG development and adaptation.
Limitations and strengths
To date, this is the first study to assess Brazilian CPGs’ quality for the most prevalent NCDs, as well as conduct a comprehensive search for identifying Brazilian guidelines. Prior Brazilian research evaluated only 8 Ministry of Health guidelines, and most were for managing rare diseases [18]. Nevertheless, the present study is limited by a subjective analysis of the AGREE II instrument. However, raters received exhaustive training on the instrument, which should ameliorate assessment concerns [21].
Conclusions
The present study evaluated Brazilian CPGs for treating the most prevalent NCDs. Our results revealed an urgent need to improve CPG development in Brazil. Healthcare professionals should be concerned with current CPG quality given that most were classified as low-B, suggesting poor “rigor of development” and less than stellar scores across additional domains. Overall, it is recommended that health institutions work in partnership with universities and adopt the AGREE II for improving CPGs’ quality.
Supporting Information
S1 Appendix. Reviewers training in AGREE II instrument.
https://doi.org/10.1371/journal.pone.0166367.s001
(DOCX)
S1 Fig. Overall quality classification for each CPG.
https://doi.org/10.1371/journal.pone.0166367.s002
(TIF)
Acknowledgments
We are grateful to Professor Airton Stein, Ana Marusic, and Bruno Caramelli for their review of this manuscript and helpful suggestions. We also thank Professor André Rolim Baby and Carlota de Oliveira Rangel Yagui for helpful advice on manuscript presentation.
Author Contributions
- Conceptualization: NSRL ER DOM.
- Formal analysis: CGRCM.
- Investigation: CGRCM DOM.
- Methodology: NSRL ER DOM.
- Project administration: DOM.
- Resources: DOM.
- Supervision: DOM NSRL ER.
- Visualization: CGRCM.
- Writing – original draft: CGRCM.
- Writing – review & editing: DOM NSRL ER.
References
- 1.
World Health Organization (WHO). Global Status Report On Noncommunicable Diseases 2014. Geneva, Switzerland: World Health Organization; 2014. Available: http://www.who.int/chp/en/
- 2.
Pan American Health Organization. Health in the Americas—Regional Outlook and Country Profiles. 2012th ed. Washington, DC: PAHO; 2012. Available: http://www.paho.org/healthintheamericas/
- 3.
World Health Organization (WHO). Noncommunicable diseases. In: WHO Mediacentre—Fact sheets. 2015. Available: http://www.who.int/mediacentre/factsheets/fs355/en/
- 4. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999;318: 527–30. pmid:10024268
- 5. Haines A, Jones R. Implementing findings of research. BMJ. 1994;308: 1488–1492. pmid:8019284
- 6. Wagner E, Austin B, Korff V. Organizing care for patients with chronic illness. Milbank Q. 1996;74: 511–44. pmid:8941260
- 7. Wagner E, Austin B, Von Korff M. Improving outcomes in chronic illness. Manag Care Q. 1996;4: 12–25.
- 8. Bodenheimer T, Wagner EH. Improving Primary Care for Patients With Chronic Illness. JAMA. 2002;288: 1775–1779. pmid:12365965
- 9. Bancos I, Cheng T, Prokop LJ, Montori VM, Murad MH. Endocrine clinical practice guidelines in North America. A systematic assessment of quality. J Clin Epidemiol. Elsevier Inc; 2012;65: 520–5. pmid:22280992
- 10. Alonso-Coello P, Irfan A, Solà I, Gich I, Delgado-Noguera M, Rigau D, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Heal Care. 2010;19: e58. pmid:21127089
- 11. Al-Ansary LA, Tricco AC, Adi Y, Bawazeer G, Perrier L, Al-Ghonaim M, et al. A Systematic Review of Recent Clinical Practice Guidelines on the Diagnosis, Assessment and Management of Hypertension. PLoS One. 2013;8. pmid:23349738
- 12. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA. 1999;281: 1900–5. pmid:10349893
- 13. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet (London, England). 2000;355: 103–6.
- 14. Kryworuchko J, Stacey D, Bai N, Graham ID. Twelve years of clinical practice guideline development, dissemination and evaluation in Canada (1994 to 2005). Implement Sci. 2009;4: 49. pmid:19656384
- 15. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: Two more decades of little, if any, progress. Arch Intern Med. 2012;172: 1628–33. pmid:23089902
- 16. Smith CAM, Toupin-April K, Jutai JW, Duffy CM, Rahman P, Cavallo S, et al. A systematic critical appraisal of clinical practice guidelines in juvenile idiopathic arthritis using the appraisal of guidelines for research and evaluation II (AGREE II) instrument. PLoS One. 2015;10: 1–22. pmid:26356098
- 17. Knai C, Brusamento S, Legido-Quigley H, Saliba V, Panteli D, Turk E, et al. Systematic review of the methodological quality of clinical guideline development for the management of chronic disease in Europe. Health Policy (New York). 2012;107: 157–67. pmid:22795610
- 18. Ronsoni RDM, Pereira CC de A, Stein AT, Osanai MH, Machado CJ. [Evaluation of eight Clinical Protocols and Therapeutic Guidelines under the Brazilian Ministry of Health using the AGREE II instrument: a pilot study]. Cad Saúde Pública. 2015;31: 1157–62. pmid:26200364
- 19. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramakers D. A systematic review of appraisal tools for clinical practice guidelines: Multiple similarities and one common deficit. Int J Qual Heal Care. 2005;17: 235–242. pmid:15743883
- 20. Siering U, Eikermann M, Hausner E, Hoffmann-E??er W, Neugebauer EA. Appraisal tools for clinical practice guidelines: A systematic review. PLoS One. 2013;8. pmid:24349397
- 21. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: Advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182: E839–42. pmid:20603348
- 22. Zhang Z, Guo J, Su G, Li J, Wu H, Xie X. Evaluation of the quality of guidelines for myasthenia gravis with the AGREE II instrument. PLoS One. 2014;9. pmid:25402504
- 23. Yan J, Min J, Zhou B. Diagnosis of pheochromocytoma: A clinical practice guideline appraisal using AGREE II instrument. J Eval Clin Pract. 2013;19: 626–632. pmid:22809219
- 24. Serón P, Lanas F, Ríos E, Bonfill X, Alonso-Coello P. Evaluation of the Quality of Clinical Guidelines for Cardiac Rehabilitation. J Cardiopulm Rehabil Prev. 2015;35: 1–12. pmid:25068394
- 25. Tudor KI, Kozina PN, Marusic A. Methodological Rigour and Transparency of Clinical Practice Guidelines Developed by Neurology Professional Societies in Croatia. PLoS One. 2013;8. pmid:23894555
- 26. Marengoni A, Angleman S, Melis R, Mangialasche F, Karp A, Garmen A, et al. Aging with multimorbidity: A systematic review of the literature. Ageing Res Rev. 2011;10: 430–39. pmid:21402176
- 27. Vitry AI, Zhang Y. Quality of Australian clinical guidelines and relevance to the care of older people with multiple comorbid conditions. Med J Aust. 2008;189: 360–5. pmid:18837677
- 28. Boyd CM, McNabrey MK, Brandt N, Correa-de-Araujo R, Daniel KM, Epplin J, et al. Guiding principles for the care of older adults with multimorbidity: an approach for clinicians: American Geriatrics Society Expert Panel on the Care of Older Adults with Multimorbidity. J Am Geriatr Soc. 2012;60: E1–E25. pmid:22994865
- 29. Menotti A, Mulder I, Nissinen A, Giampaoli S, Feskens EJM, Kromhout D. Prevalence of morbidity and multimorbidity in elderly male populations and their impact on 10-year all-cause mortalitym: The FINE study (Finland, Italy, Netherlands, elderly). J Clin Epidemiol. 2001;54: 680–86. pmid:11438408
- 30. Smith SM, Soubhi H, Fortin M, Hudon C, O’Dowd T. Managing patients with multimorbidity: systematic review of interventions in primary care and community settings. BMJ. 2012;345: e5205. pmid:22945950
- 31. Bower P, Macdonald W, Harkness E, Gask L, Kendrick T, Valderas JM, et al. Multimorbidity, service organization and clinical decision making in primary care: a qualitative study. Fam Pract. 2011;28: 579–87. pmid:21613378
- 32. Lefèvre T, D’Ivernois J-F, De Andrade V, Crozet C, Lombrail P, Gagnayre R. What do we mean by multimorbidity? An analysis of the literature on multimorbidity measures, associated factors, and impact on health services organization. Rev Epidemiol Sante Publique. 2014;62: 305–314. pmid:25444838
- 33. Lorgunpai SJ, Grammas M, Lee DSH, McAvay G, Charpentier P, Tinetti ME. Potential therapeutic competition in community-living older adults in the U.S.: Use of medications that may adversely affect a coexisting condition. PLoS One. 2014;9: e89447. pmid:24586786
- 34. Schmidt MI, Duncan BB, Mill JG, Lotufo PA, Chor D, Barreto SM, et al. Cohort Profile: Longitudinal Study of Adult Health (ELSA-Brasil). Int J Epidemiol. 2014; 1–8. pmid:24585730
- 35.
U.S. National Library of Medicine. MEDLINE/PubMed. Available: http://www.ncbi.nlm.nih.gov/pubmed
- 36.
BIREME—PAHO—WHO. Evidence Portal. In: Latin-American and Caribbean Center on Health Sciences Information. Available: http://evidences.bvsalud.org/php/index.php
- 37.
Cochrane Collaboration. Cochrane Library. Available: http://www.cochranelibrary.com/
- 38.
Agency for Healthcare Research and Quality (AHRQ). National Guideline Clearinghouse. Available: https://www.guideline.gov/
- 39.
Institute of Medicine (IOM). Clinical Practice Guidelines We Can Trust. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, editors. Washington, D.C.: The National Academies Press; 2011. Available: www.nap.edu
- 40. Khan GSC, Stein AT. [Cross-cultural adaptation of the instrument Appraisal of Guidelines For Research & Evaluation II (AGREE II) for assessment of clinical guidelines]. Cad Saúde Pública. 2014;30: 1111–4. pmid:24936826
- 41. Legido-Quigley H, Panteli D, Brusamento S, Knai C, Saliba V, Turk E, et al. Clinical guidelines in the European Union: Mapping the regulatory basis, development, quality control, implementation and evaluation across member states. Health Policy (New York). 2012;107: 146–156. pmid:22939646
- 42.
AGREE Next Steps Consortium (2009). The AGREE II Instrument [Electronic version]. Available: http://www.agreetrust.org
- 43.
Lowry R. VassarStats: Website for Statistical Computation. Available: http://vassarstats.net/kappa.html.
- 44. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;3: 159–74.
- 45.
McMaster University. McMaster AGREE II score calculator Excel spreadsheet. Available: http://fhswedge.csu.mcmaster.ca/cepftp/qasite/CriticalAppraisal.html
- 46. Poitras S, Avouac J, Rossignol M, Avouac B, Cedraschi C, Nordin M, et al. A critical appraisal of guidelines for the management of knee osteoarthritis using Appraisal of Guidelines Research and Evaluation criteria. Arthritis Res Ther. 2007;9: R126. pmid:18062805
- 47. Chui SY, Onishko C, Turner S, Coulthard K, McKinnon R. Incidence of Vancomycin-Induced Red Man Syndrome in a Women’s and Children’s Hospital. J Pharm Pract Res. 2007;37: 124–126.
- 48.
Brazil. Ministry of Health. [Clinical Protocol and Therapeutic Guidelines—Alzheimer’s disease]. 2013. Available: http://portalsaude.saude.gov.br/images/pdf/2014/abril/02/pcdt-doenca-de-alzheimer-livro-2013.pdf. Accessed 31 October 2015.
- 49.
Lopes L, Araújo L, Chaves M, Imamura M, Okamoto I, Ramos A, et al. [Alzheimer’s Disease: Prevention and Treatment]. In: Brazilian Medical Association. 2011 Available: http://www.projetodiretrizes.org.br/ans/diretrizes/doenca_de_alzheimer-prevencao_e_tratamento.pdf. Accessed 30 October 2015.
- 50.
Brazil. Ministry of Health. [Clinical Protocol and Therapeutic Guidelines—Asthma]. 2013 Available: http://portalsaude.saude.gov.br/images/pdf/2014/julho/22/PT-SAS-N—1317-alterado-pela-603-de-21-de-julho-de-2014.pdf. Accessed 31 October 2015.
- 51. Brazilian Society of Pneumology and Tisiology. [Brazilian Guidelines for the management of asthma—2012]. J Pneumol. 2012;38: S1–46. Available: http://www.sbpt.org.br/downloads/arquivos/COM_ASMA/SBPT_DIRETRIZES_MANEJO_ASMA_SBPT_2012.pdf
- 52. Zimerman LI, Fenelon G, Martinelli-Filho M, Grupi C, Atié J, Lorga-Filho A. [Brazilian atrial fibrillation guidelines]. Arq Bras Cardiol. 2009;92: 1–39.
- 53.
Cavalcanti A, Errico G, Araujo J, Ribeiro J, Scaletsky R. [Benign prostatic hyperplasia]. In: Brazilian Medical Association. 2006.Available: http://projetodiretrizes.org.br/5_volume/24-Hiperpla.pdf Accessed 30 October 2015.
- 54.
Brazil. Ministry of Health. [Clinical Protocol and Therapeutic Guidelines—Chronic obstructive pulmonary disease]. Available: http://portalsaude.saude.gov.br/images/pdf/2014/junho/13/Portaria-609-de-2013.pdf. Accessed 31 October 2015.
- 55.
Lundgren F, Maranhão B, Jardim J, Fernandes F, Mattos W, Teixeira P, et al. [Chronic obstructive pulmonary disease: exacerbation]. In: Brazilian Medical Association. 2012 Available: http://www.projetodiretrizes.org.br/diretrizes11/doenca_pulmonar_obstrutiva_cronica_exacerbacao.pdf. Accessed 30 October 2015.
- 56. Brazilian Society of Pneumology and Tisiology. [II Brazilian Consensus on Chronic obstructive pulmonary disease—COPD—2004]. J Bras Pneumol. 2004;30: S1–S42. ISSN 1806-3713
- 57. Bocchi EA, Marcondes-Braga FG, Bacal F, Ferraz AS, Albuquerque D, Rodrigues D, et al. [Updating of the Brazilian guideline for chronic heart failure—2012]. Arq Bras Cardiol. 2012;98: 1–33. http://dx.doi.org/10.1590/S0066-782X2012000700001 pmid:22392082
- 58. Cesar L, Ferreira J, Armaganijan D, Gowdak L, Mansur A, Bodanese L, et al. Guideline for Stable Coronary Artery Disease. Arq Bras Cardiol. 2014;103: 1–59. http://dx.doi.org/10.5935/abc.2014S004
- 59.
Fleck M, Chagas M, Guapo V, Brasil M, Giribela A, Toledo S, et al. [Unipolar depression: treatment]. In: Brazilian Medical Association. 2011. Available: http://www.projetodiretrizes.org.br/ans/diretrizes/depressao_unipolar-Tratamento.pdf. Accessed 30 October 2015.
- 60.
Brazilian Society of Diabetes. [Guidelines of the Brazilian Society of Diabetes 2014–2015]. São Paulo: AC Farmacêutica; 2015.
- 61. Soeiro A de M, Mansur A de P, Schaan BD, Caramelli B, Rochitte CE, Serrano CV Jr., et al. [I Guideline on specific aspects of diabetes (type 2) related to cardiology]. Arq Bras Cardiol. 2014;102: 1–41. pmid:27223869
- 62.
Macedo G, Moura F, Soriano EA, Ribas DF AN. [Type 2 Diabetes Mellitus: Insulinization]. In: Brazilian Medical Association. 2011. Available: http://www.projetodiretrizes.org.br/diretrizes10/diabetes_mellitus_tipo_2_insulinizacao.pdf. Accessed 30 October 2015.
- 63.
Sgarbi J, Villar H. [Diabetes Mellitus: pharmacological treatment]. In: Brazilian Medical Association. 2004. Available: http://www.projetodiretrizes.org.br/4_volume/13-Diabetes.pdf. Accessed 30 October 2015.
- 64.
Carvalhaes A, Ferrari A, Magalhães A, Nasy A, Paula e Silva C, Hashimoto C, et al. [Gastroesophageal reflux disease: pharmacological treatment]. In: Brazilian Medical Association. 2011. Available: http://www.projetodiretrizes.org.br/ans/diretrizes/doenca_do_refluxo_gastroesofagico-tratamento_farmacologico.pdf. Accessed 30 October 2015.
- 65.
Brazil. Ministry of Health. [Clinical Protocol and Therapeutic Guidelines—Dyslipidemia: prevention of cardiovascular and pancreatitis events]. 2013 Available: http://portalsaude.saude.gov.br/images/pdf/2014/abril/02/pcdt-dislipidemia-livro-2013.pdf. Accessed 31 October 2015.
- 66. Xavier HT, Izar MC, Faria Neto JR, Assad MH, Rocha VZ, Sposito AC, et al. [V Brazilian Guidelines on Dyslipidemias and Prevention of Atherosclerosis]. Arq Bras Cardiol. 2013;101: 1–20. pmid:24217493
- 67.
Martins D, Araújo L. [Diabetes Mellitus: treatment of hypertension]. In: Brazilian Medical Association. 2004. Available: http://www.amrigs.org.br/revista/47-02/Vacinas.pdf#page=2. Accessed 1 January 2015.
- 68. Brazilian Society of Cardiology, Brazilian Society of Hypertension, Brazilian Society of Nephrology. [VI Brazilian Guidelines on Hypertension]. Arq Bras Cardiol. 2010;95: 1–51.
- 69.
Coimbra I, Ferreira B, Coimbra A, Anderson M, Andrada N. [Use of nonsteroidal antiinflammatory drugs in chronic pain in patients with osteoarthritis (osteoarthrosis)]. In: Brazilian Medical Association. 2011. Available: http://www.projetodiretrizes.org.br/ans/diretrizes/uso_dos_anti-inflamatorios_nao_hormonais_na_dor_cronica_em_pacientes_com_osteoartrite_(osteoartrose).pdf
- 70.
Brazil. Ministry of Health. [Clinical Protocol and Therapeutic Guidelines—Osteoporosis]. 2014. Available: http://portalsaude.saude.gov.br/images/pdf/2014/dezembro/15/Osteoporose.pdf. Accessed 31 October 2015.
- 71.
Cunha EP; Steiner ML; Strufaldi R; Fernandes C; Laurindo IMM; Pereira RMR; Simões R. [Osteoporosis: treatment]. In: Brazilian Medical Association. 2011. Available: http://www.projetodiretrizes.org.br/ans/diretrizes/osteoporose-tratamento.pdf. Accessed 3o October 2015.
- 72.
Brazil. Ministry of Health. [Clinical Protocol and Therapeutic Guidelines—Rheumatoid arthritis]. 2015. Available: http://portalsaude.saude.gov.br/images/pdf/2016/marco/08/pcdt-Artrite-Reumat—ide—Portaria-SAS-996-PCDT-AR-30-09-2015.pdf Accessed 31 October 2015.
- 73. , Cruz BA, Brenol CV, Pereira IA, Rezende-Fronza LS, Bertolo MB, et al. 2012 Brazilian Society of Rheumatology Consensus for the treatment of rheumatoid arthritis. Rev Bras Reumatol. 2012;52: 152–74. pmid:22460407
- 74. Hu J, Chen R, Wu S, Tang J, Leng G, Kunnamo I, et al. The quality of clinical practice guidelines in China: a systematic assessment. J Eval Clin Pr. 2013;19: 961–7. pmid:22845397
- 75. Chen Y, Hu S, Wu L, Fang X, Xu W, Shen G. Clinical practice guidelines for hypertension in China: a systematic review of the methodological quality. BMJ Open. 2015;5: e008099. pmid:26179649
- 76. Jiang M, Liao L-Y, Liu X-Q, He W-Q, Guan W-J, Chen H, et al. Quality Assessment of Clinical Practice Guidelines for Respiratory Diseases in China: A Systematic Appraisal. Chest. The American College of Chest Physicians; 2015;148: 759–66. pmid:25950752
- 77. Chen Y-L, Yao L, Xiao X-J, Wang Q, Wang Z-H, Liang F-X, et al. Quality assessment of clinical guidelines in China: 1993–2010. Chin Med J (Engl). 2012;125: 3660–4.
- 78. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud P-AC, et al. Why Don’t Physicians Follow Clinical Practice Guidelines? A Framework for Improvement. JAMA. 1999;282: 1458–65. pmid:10535437
- 79. Murthy L, Shepperd S, Clarke MJ, Garner SE, Lavis JN, Perrier L, et al. Interventions to improve the use of systematic reviews in decision-making by health system managers, policy makers and clinicians. Cochrane Database Syst Rev. 2012;9: CD009401. pmid:22972142
- 80. Kunz R, Fretheim A, Cluzeau F, Wilt TJ, Qaseem A, Lelgemann M, et al. Guideline group composition and group processes: article 3 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9: 229–33. pmid:23256164
- 81. Fretheim A, Schünemann HJ, Oxman AD. Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Res Policy Syst. 2006;4: 15. pmid:17134482
- 82. Hudon C, Fortin M, Haggerty J, Loignon C, Lambert M, Poitras ME. Patient-centered care in chronic disease management: A thematic analysis of the literature in family medicine. Patient Educ Couns. 2012;88: 170–176. pmid:22360841
- 83. Piano V, Schalkwijk A, Burgers J, Verhagen S, Kress H, Hekster Y, et al. Guidelines for neuropathic pain management in patients with cancer: a European survey and comparison. Pain Pract. 2013;13: 349–57. pmid:23067004
- 84. Norris SL, Holmer HK, Ogden LA, Selph SS, Fu R. Conflict of interest disclosures for clinical practice guidelines in the national guideline clearinghouse. PLoS One. Public Library of Science; 2012;7: e47343. pmid:23144816
- 85.
Rayess Z Al, Wiercioch W. A new approach to CPG adaptation in Saudi Arabia: Adaptation of practice guidelines to a country-specific context using the GRADE/DECIDE evidence to decision framework. GIN Conference. Melbourne; 2014. Available: http://www.decide-collaboration.eu/sites/www.decide-collaboration.eu/files/uploads/140822_CPG_FridayRm1041100ZulfaAlRayess1.pdf