Clinical practice guidelines are systematically created documents that summarize knowledge and assist in delivering high-quality medicine by identifying evidence that supports best clinical care. They are produced not only by international professional groups but also by local professionals to address locally-relevant clinical practice. We evaluated the methodological rigour and transparency of guideline development in neurology formulated by professionals in a local medical community.
We analyzed clinical guidelines in neurology publicly available at the web-site of the Physicians’ Assembly in Croatia in 2012: 6 guidelines developed by Croatian authors and 1 adapted from the European Federation of Neurological Societies. The quality was assessed by 2 independent evaluators using the AGREE II instrument. We also conducted a search of the Cochrane Library to identify potential changes in recommendation from Cochrane systematic reviews included in guideline preparation.
The methodological quality of the guidelines greatly varied across different domains. „Scope and Purpose” and „Clarity of Presentation“ domains received high scores (100% [95% confidence interval (CI) 98.5–100] and 97% [77.9–100], respectively), the lowest scores were in “Stakeholder Involvement“ (19% [15.5–34.6]) and “Editorial Independence” (0% [0–19.2]). Conclusions of 3 guidelines based on Cochrane systematic reviews were confirmed in updated versions and one update provided new information on the effectiveness of another antidepressant. Two Cochrane reviews used in guidelines were withdrawn and split into new reviews and their findings are now considered to be out of date.
Neurological guidelines used in Croatia differ in structure and their methodological quality. We recommend to national societies and professional groups to develop a more systematic and rigorous approach to the development of the guidelines, timely inclusion of best evidences and an effort to involve target users and patients in the guideline development procedures.
Citation: Tudor KI, Kozina PN, Marušić A (2013) Methodological Rigour and Transparency of Clinical Practice Guidelines Developed by Neurology Professional Societies in Croatia. PLoS ONE 8(7): e69877. https://doi.org/10.1371/journal.pone.0069877
Editor: Adrian V. Hernandez, Universidad Peruana de Ciencias Aplicadas (UPC), Peru
Received: April 11, 2013; Accepted: June 13, 2013; Published: July 19, 2013
Copyright: © 2013 Tudor et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: These authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Clinical practice guidelines (CPGs) are systematically created documents that summarize the knowledge and assist in delivering high-quality medicine by identifying evidence that supports the best clinical care . They are produced not only by international organizations but also by professional associations in many countries, aiming to supply the local professionals with recommendations based on currently available evidence for best standard of care for the patients –.
In order to ensure that guidelines are valid and relevant for practice, it is important that they are prepared in a methodologically appropriate process. For this purpose, procedures to assess the methodological quality of CPGs have been developed, most notably the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument . AGREE instrument is a widely recognized tool for guideline evaluation of various fields of medicine –.
Neurology is a rapidly developing field where the new and relevant evidence may have important implications for patients’ health care and outcome prognosis. However, there are only a few guidelines for practice in neurology which been evaluated using a standardized instrument , . Our aim was to evaluate the methodological rigour and transparency of guideline development in neurology as formulated by professionals in a local medical community. We used the advantage of public access to clinical guidelines in neurology developed by the neurology experts from the Physicians’ Assembly in Croatia, which has a tradition in developing and maintaining professional standards in Croatia since 1874 . We also evaluated how many guidelines were based on Cochrane systematic reviews, considered to be the evidence of highest quality , , and whether there would be a change in recommendations based on recent evidence.
We analyzed 7 neurology guidelines that were available at the web-site of the Croatian Physicians’ Assembly in 2012 . Six were developed by Croatian authors and one was adapted (translated) from the European Federation of Neurological Societies (EFNS) guideline: 1) Consensus opinion on diagnosing brain death – Guidelines for use of confirmatory tests ; 2) Recommendations for neuropathic pain treatment ; 3) Recommendations for stroke management 2006 update ; 4) Evidence based guidelines for treatment of primary headaches ; 5) Guidelines for preoperative diagnostic evaluation of patients with pharmacoresistant epilepsy , 6) Epilepsy – therapeutic guidelines , 7) EFNS guidelines on pharmacological treatment of neuropathic pain .
We used The Appraisal of Guidelines for Research and Evaluation II (AGREE II) instrument  to assess the quality of guidelines. The instrument has 23 items, grouped in 6 quality domains: 1) Scope and Purpose (items 1–3), 2) Stakeholder Involvement (items 4–6), 3) Rigour of Development (items 7–14), 4) Clarity of Presentation (items 15–17), 5) Applicability (items 18–21) and 6) Editorial Independence (items 22–23). Each item was rated on a 7-point scale: from 1– strongly disagree to 7– strongly agree.
Two assessors independently performed the rating of guidelines. Each of two evaluators also independently judged the overall quality of the guideline from 1 (least quality) to 7 (highest quality). They independently submitted their scores to the third investigator, who then calculated the overall scores . The overall score for each domain was calculated by summing up all the scores of the individual items in the domain and the total was standardised as a percentage of the maximum possible score for that domain. The following formula was used:
The scaled domain score = (Obtained score − Minimum possible score)/(Maximum score − Minimum possible score)×100.
where “Obtained score” was the sum of the scores by individual assessors, Maximum score = 7 (strongly agree)×2×No. items in the domain, and minimum score = 1 (strongly disagree)×2×No. items in the domain .
The formula was also used to calculate the overall quality score.
We searched the Cochrane Database of Systematic Reviews to identify any potential change of recommendation from the version of the Cochrane systematic reviews included in guideline preparation. The search strategy included the terms from respective guidelines. The date of the Cochrane Library search was 13 May 2012.
Out of 7 guidelines, 2 were published within 2 years before the time of analysis, and 4 were published in 2005 or 2006 (Table 1). Only a single guideline stated that it was an update of a previous version, and 3 explicitly mentioned the method of development (Table 1).
The highest scores for all guidelines were observed in the domains “Scope and purpose” and “Clarity of Presentation”, and there was not much variability in the score among the guidelines (Table 2). The lowest scores (zero values) were given to “Editorial Independence” for all locally developed guidelines, except for EFNS guidelines  on pharmacological treatment of neuropathic pain (Table 2).
The domain “Stakeholder Involvement” also received low grades because the views and preferences of the target population (patients, public, etc.) were not often sought. The target users of the guideline were clearly defined only in the Consensus opinion on diagnosing brain death – Guidelines for use of confirmatory tests .
The scores in the domain “Applicability” were below 50% for all guidelines, because they did not have clear explanation of existing facilitators and barriers that could impact the application of guideline recommendations (Table 2). Again, EFNS guidelines on pharmacological treatment of neuropathic pain  fared better on this domain and received the composite score of 52% (Table 2).
The scores for the domain “Rigour of Development” varied across the guidelines (Table 1). Details of the strategy used to search for evidence were described in EFNS guidelines on pharmacological treatment of neuropathic pain  and in the Consensus opinion on diagnosing brain death – Guidelines for use of confirmatory tests . The criteria for selecting the evidence was described only in the EFNS guidelines . Methods for formulating recommendations were described in EFNS guidelines on pharmacological treatment of neuropathic pain  and in Recommendations for stroke management 2006 update . We found no evidence of external review prior to the publication for any of the evaluated guidelines. A clear statement about the procedure for updating the guideline was provided only in the EFNS guideline . The median score for “Overall Assessment” was 78.5% for Croatian guidelines, whereas the EFNS guideline achieved the full overall assessment score (Table 2).
The median score for of all guidelines was almost maximal for the “Scope and Purpose” and” Clarity and Presentation” domains and below 32% for other domains except Editorial independence which was 0 (Figure 1).
The inter-rater agreement (weighted kappa) between the two raters ranged from good (0.725, standard error 0.1449) to very good (0.949, standard error 0.051) across domains in individual guidelines .
In 3 out of 7 assessed guidelines, recommendations were partially based on evidence from Cochrane systematic reviews: EFNS guidelines on pharmacological treatment of neuropathic pain ; Recommendations for neuropathic pain treatment ; and Recommendations for stroke management 2006 update . The key conclusions made by prior studies were confirmed in updated versions of the reviews and one update provided new information on effectiveness of another antidepressant – venlafaxine . Additionally, 2 Cochrane reviews whose results were included in EFNS guideline guidelines on pharmacological treatment of neuropathic pain  and Recommendations for neuropathic pain treatment  were withdrawn and split into new reviews and their findings are now considered to be out of date , . Only a single guideline  had a procedure for updating the evidence for the recommendations.
Our evaluation of the methodological rigour and transparency of publicly available neurological guidelines used in Croatia demonstrated a general lack of structured procedure for guideline preparation, as well as significant deficiencies in the following domains: “Applicability”, “Editorial Independence” and “Stakeholder Involvement”. A single guideline provided information about the procedure for its updating , and most of the guideline were older than 4 years, which is close to the estimated “half life” of 5.5 years for systematic reviews . Even for more recent guidelines we identified a change in the evidence base , , i.e. withdrawal of Cochrane systematic reviews because the reviews were not updated.
The limitations of the study include the small number of evaluated guidelines so we did not attempt any statistical conclusion. Our aim was to assess only publicly available guidelines supported by professional groups and societies, as those were considered to be most influential and relevant for the clinical practice in Croatia. Although the AGREE instrument has been tested for validity in different clinical areas and by different professionals , its semiquantitive nature of the instrument does not allow conclusions on what is of high or low quality, so that the results of this study have to interpreted with caution and in the context of the local practices and health care system. It is also important to keep in mind that the AGREE instrument only evaluates how the individual items within separate evaluation domains were reported, which may create a bias if the individual items were included during the development of the guideline but were not reported. However, it is likely that guidelines that are better reported contain more appropriate and relevant recommendations. Finally, the instrument does not evaluate the validity of the actual recommendations in the guidelines. The raters were neurology residents without experience in guideline development and they were not blinded to the source and authors of the guidelines. This may have introduced bias, as it was shown that the instrument had low reliability when used by untrained professionals in developing, non-English language country setting . The two raters in our study read relevant literature about AGREE II Instrument and its applications and discussed individual items before actual scoring. They were consistent in their scores and their agreement was good or very good, as judged by kappa statistics.
The finding of low scores for Croatian guidelines in most of AGREE domains (“Rigour of Development”, “Stakeholder Involvement”, “Applicability” and “Editorial Independence) is similar to the finding for other national guidelines, as recently published for the general population of guidelines and specific otorhinolaryngology guidelines in China , . Like in China, Croatian guidelines have mostly been developed as expert opinions, which generally followed the methodology and presentation of a literary review, rather than methodologically rigorous analysis of evidence. This is probably the reason why guidelines derived by Croatian professional scored zero for “Editorial Independence” domain, while the guideline from a European organization, adopted by Croatian professionals, scored better on this domain (38%). EFNS guideline scored higher also on “Rigor of Development” and “Applicability” domains, indicating better methodological approach of professional groups creating guidelines at an international level, although the actual scores for the EFNS guideline leave a lot of space for improvement in methodological quality. A recent review of methodological quality of 28 national, as well as pan-European guidelines for the management of chronic disease in Europe demonstrated lower scores for “Editorial Independence” “Applicability”, “Stakeholder Involvement” and “Rigour of Development” . At both the European and national levels, there seems to be little coordination and methodological management of guideline development, leading to a non-transparent, decentralised and non-systematic approach to guidelines . Such situation does not help local/national health care communities, who could benefit from adopting international clinical practice guidelines and adapting them to local specificities instead of spending resources in producing their own guidelines.
Professional societies in neurology and other clinical disciplines, both at national and international levels, should take seriously the finding from our and other studies on the methodological quality of clinical practice guidelines. At a national level, a good approach may be the translation of AGREE instruments and other relevant documentation , which should be a part of professional training. Professional associations could provide leadership for such training and for systematic collection of data on the development and quality of guidelines. Even more effective approach to build the capacity of the medical profession for high quality clinical guidelines and their implementation in practice, would be education in research methodology and evidence-based medicine at the level of medical schools , .
Conceived and designed the experiments: AM. Performed the experiments: KIT PNK. Analyzed the data: AM KIT PNK. Contributed reagents/materials/analysis tools: AM KIT PNK. Wrote the paper: AM KIT PNK.
- 1. Wolf JS Jr, Hubbard H, Faraday MM, Forrest JB (2011) Clinical practice guidelines to inform evidence based clinical practice. World J Urol 29: 303–309.
- 2. Shinohara Y, Nagayama M, Origasa H (2009) Postpublication external review of the Japanese guidelines for the management of stroke 2004. Stroke 40: 1439–1443.
- 3. Chen YL, Yao L, Xiao XJ, Wang Q, Wang ZH, et al. (2012) Quality assessment of clinical guidelines in China: 1993–2010. Chin Med J (Engl) 125: 3660–3664.
- 4. Zhang ZW, Liu XW, Xu BC, Wang SY, Li L, et al. (203) Analysis of quality of clinical practice guidelines for otorhinolaryngology in China. PLoS One 8: e53566.
- 5. The AGREE Collaboration (2001) Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument, 2001. Available: http://www.agreecollaboration.org. Accessed 2012 Aug 1.
- 6. Polus S, Lerberg P, Vogel J, Watananirun K, Souza JP, et al. (2012) Appraisal of WHO guidelines in maternal health using the AGREE II assessment tool. PLoS One 7: e38891.
- 7. Legido-Quigley H, Panteli D, Brusamento S, Knai C, Saliba V, et al. (2012) Clinical guidelines in the European Union: mapping the regulatory basis, development, quality control, implementation and evaluation across member states. Health Policy 107: 146–156.
- 8. Knai C, Brusamento S, Legido-Quigley H, Saliba V, Panteli D, et al. (2012) Systematic review of the methodological quality of clinical guideline development for the management of chronic disease in Europe. Health Policy 107: 157–167.
- 9. Azermai M, Petrovic M, Elseviers MM, Bourgeois J, Van Bortel LM, et al. (2012) Systematic appraisal of dementia guidelines for the management of behavioural and psychological symptoms. Ageing Res Rev 11: 78–86.
- 10. Croatian Physicians’ Assembly. Professional Societies. Available: http://www.hlz.hr/portal/index.php?option=com_content&view=article&id=40&Itemid=79. Accessed 2012 Feb 1.
- 11. Delaney A, Bagshaw SM, Ferland A, Laupland K, Manns B, et al. (2007) The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med 35: 589–94.
- 12. Olsen O, Middleton P, Ezzo J, Gøtzsche PC, Hadhazy V, et al. (2001) Quality of Cochrane reviews: assessment of sample from 1998. BMJ 323: 829–32.
- 13. Demarin V, Lovrenčić- Huzjan A, Vargek-Solter V, Vuković V, Miškov S, et al. (2005) Consensus opinion on diagnosing brain death – Guidelines for use of confirmatory tests. Acta Clinica Croat 44: 5–79.
- 14. Demarin V, Basić-Kes V, Zavoreo I, Bosnar-Puretić M, Rotim K, et al. (2008) Recommendations for neuropathic pain treatment. Acta Clin Croat 47: 181–91.
- 15. Demarin V, Lovrenčić- Huzjan A, Trkanjec Z, et al. (2006) Recommendations for stroke management 2006 update. Acta Clinica Croat 45: 219–285.
- 16. Demarin V, Vuković V, Lovrenčić- Huzjan A, Lušić I, Jančuljak D, et al. (2005) Evidence based guidelines for treatment of primary headaches. Acta Clinica Croat 44: 139–183.
- 17. Petelin Ž, Hajnšek S, Wellmer J, Mrak G, Radoš M, et al. (2010) Guidelines for preoperative diagnostic evaluation of patients with pharmacoresistant epilepsy. Neurol Croat 59: 23–34.
- 18. Hajnšek S, Kovačević I, Petelin Ž (2010) Epilepsy – therapeutic guidelines. Neurol Croat 59: 35–62.
- 19. Attal N, Cruccu G, Haanpää M, Hansson P, Jensen TS, et al. (2006) EFNS guidelines on pharmacological treatment of neuropathic pain. Eur J Neurol 13: 1153–69.
- 20. The AGREE Next Steps Consortium. Appraisal of guidelines for research & evaluation II instrument. Available: http://www.agreecollaboration.org. Accessed 2012 Aug 1.
- 21. Altman DG (1991) Practical statistics for medical research. London: Chapman and Hall.
- 22. Saarto T, Wiffen P (2007) Antidepressants for neuropathic pain. Cochrane Database Syst Rev 4: CD005454.
- 23. Wiffen P, McQuay H, Edwards JE, Moore RA (2011) Withdrawn: Gabapentin for acute and chronic pain. Cochrane Database Syst Rev 3: CD005452.
- 24. Wiffen P, Collins S, McQuay H, Carroll D, Jadad A, et al. (2010) Withdrawn: Anticonvulsant drugs for acute and chronic pain. Cochrane Database Syst Rev 1: CD001133.
- 25. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, et al. (2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147: 224–233.
- 26. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, et al. (2010) Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 182: E472–478.
- 27. Rashidian A, Yousefi-Nooraie R (2012) Development of a Farsi translation of the AGREE instrument, and the effects of group discussion on improving the reliability of the scores. J Eval Clin Pract 18: 676–681.
- 28. Glasziou PP, Sawicki PT, Prasad K (2011) Montori VM; International Society for Evidence-Based Health Care (2011) Not a medical course, but a life course. Acad Med 86: e4.
- 29. Marušić A, Sambunjak D, Jerončić A, Malički M, Marušić M (2013) No health research without education for research - experience from an integrated course in undergraduate medical curriculum. Med Teach Mar 6. [Epub ahead of print].