http://orcid.org/0000-0002-5258-9684
Figures
Abstract
Each year, evidence-based clinical guidelines gain more space in the health professionals’ practice and in services organization. Due to the scarcity of scientific publications focused on diseases of poverty, the development of well-founded clinical guidelines becomes more and more important. In view of that, this paper aims to evaluate the quality of Brazilian guidelines for those diseases. The AGREE II method was used to evaluate 16 guidelines for poverty-related diseases (PRD) and 16 guidelines for global diseases whose treatment require high-cost technologies (HCD), with the ultimate aim of comparing the results. It was found that, in general, the guideline development quality standard is higher for the HCD guidelines than for the PRD guidelines, with emphasis on the "rigour of development" (48% and 7%) and "editorial independence" (43% and 1%) domains, respectively, which had the greatest discrepancies. The HCD guidelines showed results close to or above international averages, whereas the PRD guidelines showed lower results in the 6 domains evaluated. It can be concluded that clinical protocol development priorities need some redirecting in order to qualify the guidelines that define the healthcare organization and the care of vulnerable populations.
Citation: Santana RS, de Oliveira Lupatini E, Zanghelini F, de March Ronsoni R, Rech N, Leite SN (2018) The different clinical guideline standards in Brazil: High cost treatment diseases versus poverty-related diseases. PLoS ONE 13(10): e0204723. https://doi.org/10.1371/journal.pone.0204723
Editor: Karen Anne Grimmer, Flinders University, AUSTRALIA
Received: July 28, 2017; Accepted: September 13, 2018; Published: October 17, 2018
Copyright: © 2018 Santana et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files. All guidelines studied are available from the Brazilian Ministry of Health website.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The advancement of evidence-based health care has positively impacted patient care, improving diagnostic accuracy, guiding healthcare professionals on using the most effective therapy, and reducing patient exposure to ineffective or even harmful interventions [1].
In view of the great advance of scientific studies in the health area, one can state that access to the best available evidence is a right of the health professional and the patient, target of the health intervention. Well-developed clinical guidelines have the power to mediate different interests in health care policy, best practice, government funding, local contexts, and patient choice [2].
Yet, the great number of clinical guidelines continuously produced without an acceptable quality standard that supports their recommendations leads to discrediting of such a fundamental mechanism of health care qualification. Several guidelines developed for the care of the same disease and produced with different methods and objectives can generate unnecessary competition and a complex system of conflicting practices and interventions [3].
In Brazil, since 2000 there has been a movement for clinical guideline qualification using structured methods for the development and publication of the so-called “Clinical Protocols and Therapeutic Guidelines (PCDTs)”. Early versions of those documents include a small number of clinical conditions for which the cost of treatment, especially medicines, was considerably high.
After the enactment of Law No. 12,401 and Decree No. 7,508, both in 2011 (on therapeutic care and technology incorporation in the Brazilian health system), the PCDTs began to gain institutional legitimacy and greater coverage, becoming a reference document for disease or condition diagnosis, pharmacological and non-pharmacological treatment, patient follow-up and verification of therapeutic results [4]. Thus, in the context of the Brazilian Unified Health System (SUS, the national public health system), the PCDTs are referred to as Clinical Practice Guidelines, sometimes shortened as Clinical Guidelines.
The PDCTs are applied, also, on regulating and controlling the free access to high-cost treatment. High-cost medicines is an expression without a uniform definition but adopted by the Pan American Health Organization and partner countries to define therapies that are the major burden for the financing of the health system and / or high household expenditure. It is characterized generally by a greater interest of the pharmaceutical industry and consequently more number of launches / innovations and patents. These medicines are particularly applied for the so called “global diseases”, those that did not have the socioeconomic vulnerability bias as a fundamental determinant [5,6,7,8]. In Brazil, those diseases are covered by a program called Specialized Component of Pharmaceutical Assistance [4].
On the other hand, according to the conceptual propositions of the Médecins Sans Frontières (MSF, or Doctors Without Borders) and the World Health Organization (WHO), tropical, neglected or related to poverty diseases have opposite characteristics, with less interest of the pharmaceutical industry and consequently smaller number of medicines available, innovations and patents. They are often characterized by requesting older and less expensive drugs for the health system. In Brazil, these drugs are mostly produced by public laboratories and covered by the public program called Strategic Conditions Component of the Pharmaceutical Assistance [5, 6]. In this country, poverty-related diseases are particularly important health issues. They are the result of poor living conditions, insufficient access to adequate and healthy food, clean drinking water, medical care and education [5,6,7,8]. Some infectious diseases such HIV, considered related to poverty globally, in Brazil is not classified as neglected or poverty related. Particular healthcare facilities and guidelines are available and accessible.
Most of the so-called poverty-related and neglected diseases and conditions, however, are still addressed by the SUS in the form of different documents such as guides, manuals and handbooks formulated in diverse methodological ways rather than in a standardized way in line with the PCDTs’ structures.
In this sense, this paper aims to evaluate the methodological quality of the documents published by the Brazilian Ministry of Health addressing care guidelines for the population affected by poverty-related diseases, in an attempt to verify their compliance with evidence-based health standards and consequently the needs of health caregivers and patients.
Methods
Identification and selection of clinical guidelines
The study included only clinical guidelines published by the Brazilian Ministry of Health for the diseases or conditions covered by the Specialized Component of Pharmaceutical Assistance and the Strategic Conditions Component of the Pharmaceutical Assistance. Documents issued by states, municipalities or other health institutions were not included in the analysis. Likewise, posters, flyers or like documents that did not fit the structure of care guides or protocols were not considered, even if they contained therapeutic conduct guidelines.
Given the variety of documents for the same disease or condition, the quality assessment included the most recently published ones and with the most detailed information on care (clinical condition, diagnostic and therapeutic conduct and the target population of the proposed interventions). At that moment (2016), sixteen documents had been published by the Ministry of Health about diseases or conditions classified as poverty-related, presenting treatment guidelines and covered by the Strategic Conditions Component of Pharmaceutical Assistance. As a comparison criterion with the global disease guidelines, the study sample included all the PCDTs issued by the CONITEC (National Committee for Health Technology Incorporation in the SUS) and published as Ministry of Health Ordinances in 2015, the immediate year before the study. Sixteen PCSTs were published in that year. The main characteristic of these PCDTs is their focus on the so-called 'high-cost diseases', included in the Specialized Component of Pharmaceutical Assistance (CEAF). The 32 guidelines selected for the study are described in Table 1:
Methodological quality assessment: The AGREE II instrument
The quality analysis of the selected clinical guidelines was performed using the AGREE–Appraisal of Guidelines for Research and Evaluation, an assessment instrument developed from analyses of over 100 guidelines selected and evaluated independently by more than 200 evaluators from different parts of the world. It is used by the WHO and various technology assessment agencies worldwide as a reference tool for quality assessment of clinical practice guidelines. Its last edition (AGREE II) has 23 key items organized within six quality domains [9,10].
Following the AGREE II instructions, the guidelines selected for this study were appraised by four independent experts with previous experience in clinical practice guideline assessment [11]. As commonly observed, similar studies have been using two or three appraisers; however the instrument recommends ideally four appraisers [9,11–13].
Data extraction, management and analysis
Data were collected from the six quality domains described in the AGREE II Instrument: (i) scope and purpose; (ii) stakeholder involvement; (iii) rigour of development; (iv) clarity of presentation; (v) applicability, and (vi) editorial independence. The instrument provides a score sheet with ratings from 1 (strongly disagree) to 7 (strongly agree) for each of the 23 items. At the end, a quality score is calculated for each of the six AGREE II domains [11]. A percentage of suitability of each domain with values from 0% to 100% was also calculated using the score obtained by each evaluator and possible maximum score of the domain, following the guidelines of the AGREE II Instrument.
Although not suggested by the instrument, this study performed a concordance analysis among the appraisers, using the kappa test in order to avoid randomness or poor agreement. Kappa coefficients of moderate agreement (kappa> 0.4) were considered preferable for this type of study [12, 14].
For the concordance analysis, the appraisers decided jointly that scores 1 and 2 would be considered "low", scores between 3 and 5 would be "intermediate", and scores 6 and 7 "high". By evaluating the agreement between these categories, an initial kappa of 0.263 was obtained. Subsequently, a meeting was held with the appraisers to discuss the main dissenting opinions on the assessment criteria. The major divergences were remedied by re-reading and discussing together (all the appraisers) guidelines of the AGREE II Instrument [11]. After a new independent assessment, a final kappa of 0.598 was obtained, corresponding to a moderate agreement.
The non-parametric Mann-Whitney test was applied to compare the mean score difference of the guidelines in the six domains. The results were compared using program Statistical Package for the Social Sciences—SPSS (IBM, v. 20) with the level of significance set at 5%.
The instrument does not define a standard that indicates whether or not the guideline as a whole should be recommended. So, in order to make the assessment less subjective in that respect, it was agreed in this study that the domain (iii) “rigor of development” would be the main standard for the overall assessment of the guideline. Based on criteria suggested by other authors [9,10,12] the evaluators defined 50% as the minimum score for “rigor of development”, although some authors suggest a higher one. For a guideline to be considered as "recommended" it had to score higher than 50% in "rigor of development" and in two other domains; the guideline that scored between 30% and 50% in "rigor of development" and higher than 50% in two other domains was considered "recommended, with modifications"; and, lastly, the guideline that scored less than 30% in "rigor of development" was considered "not recommended".
Results and discussion
The various types of guideline documents evince the dissent among their denominations: "primary care handbooks", "guides", "protocols", "general guidelines", "manuals", "technical guidelines", among other terms adopted by the Brazilian Ministry of Health. Indeed, there are some conceptual differences between those terms, but there also seems to be no consensus in the Brazilian literature about the standard designation of this type of document, which we have chosen here to call generically as a “clinical guideline”. However, the lack of standardized publications by the federal health agency may hinder the access to this information, both for researchers and target health professionals [9,10,12].
The diversity of healthcare recommendations for the same disease seems harmful insofar as they can generate conflicts concerning the suggested clinical conduct given the different guideline production methods used and the different results presented in the publications in question.
The AGREE II instrument, however, is sensitive to different types of guidelines as it assesses comprehensive and necessary issues for any type of therapeutic care guideline. The assessment results are expressed as quality scores (0 to 100) for each domain and each of the 32 guidelines validated in this study (Table 2).
Table 3 displays the assessment results in a direct comparison between the two guideline groups evaluated (PRD x HCD), based on the ratings (1 to 7) obtained in each domain. There was a statistically significant difference between the mean scores contrasted.
Domain 1—scope and purpose
Fundamentally, this domain aims to analyze whether the guideline objectives, the diseases covered and the target population are well defined. The high-cost diseases guidelines showed higher scores (86% or 6.2) when compared to those of poverty-related diseases (74% or 5.4), although the latter still presented high scores.
It should be noted that the standard format of the HCDs accurately define the conditions covered by the guideline, using the codes of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) and clearly describing the "inclusion" and "exclusion" criteria. Molino et al. (2016) and Ronsoni et al. (2015) also found higher scores in this domain, although their studies were restricted to assessments of clinical protocols for non-communicable diseases [9,12].
Domain 2—stakeholder involvement
It evaluates the composition and degree of expertise of the group involved in drafting the guideline. In this domain, the ratings and quality scores dropped in both guideline groups; however the HCD mean scores (43% or 3.6) kept higher than those of PRD (26% or 2.6). The analysis revealed two weaknesses, namely the lack of commitment of all professional groups involved and the failure to promote participation of the target population. For instance, the sickle cell guideline (PRD 08), which had the lowest quality score in the domain (7%), was drawn up by only two medical specialists, with no reported participation of other health professionals or methodologists committed to evidence search and analysis.
An important difference between the assessed groups is that all the HCD guidelines were previously approved by the CONITEC and made available for public consultation and social collaboration in general. Only one guideline (PRD 14) of the PRD group was previously reviewed by the CONITEC and underwent a public consultation. None of the 32 guidelines evaluated in this study reported any kind of active patient participation in their development.
Effective patient participation in clinical guidelines development has been shown as a global difficulty due to issues concerning the closed structure of the Health Technology Assessment (HTA) area and patients' suspicion that the guideline developers might be influenced by pharmaceutical companies [15,16]. However, various studies have demonstrated that this strategy is crucial to improve the quality of the recommendations, since it allows identifying discordant points between professionals and patients, identifying priority needs from the perspective of users; points that are not well observed by guideline developers and managers which can help improve the strategies for guideline adherence and implementation [17,18].
Domain 3—rigor of development
This domain showed one of the greatest disparities among the groups evaluated, where the quality score of the HCD guidelines, i.e. the compliance level with the criteria of the instrument, was almost seven times greater (48%) than that of the PRD guidelines (7%), and their mean score (3.9) was almost three times higher (1.4), respectively.
It was found that some key questions for guideline quality assessment were totally neglected in the PRD guidelines analyzed, for instance, description of the search methods, evidence selection criteria, strengths and limitations of the evidence, formulation of recommendations, consideration of benefits and harms, in addition to external review methodology and procedure for updating the guideline.
Among the guidelines evaluated, the Guide to Infectious and Parasitic Diseases (PRD 02) had the lowest quality score (1%). It is a publication intended to be a quick guide to basic care. However, it does not have a section informing the reader about the methods used to formulate the recommendations and it does not even provide bibliographic references. Notwithstanding, the Guide has the highest circulation among the publications evaluated, with 90 thousand printed copies.
Molino et al. (2016) and Ronsoni et al. (2013) reviewed some HCD guidelines of the Brazilian Ministry of Health and found similar quality scores in this domain, with mean values of 41% and 36%, respectively [12, 19].
Decrees No. 7508/11 and No. 7646/11 establish that the PCDTs of the Brazilian Ministry of Health must be updated every two years, conferring to the CONITEC the constitution or modification of such guidelines. All of the HCD documents reviewed in this study met these standards. Of the 16 PRD guidelines evaluated, two (PRD 02 and PRD 09) were published before those decrees and, among the others, only one (PRD 14) was published in accordance with the aforementioned decrees [4, 20, 21].
Domain 4—clarity of presentation
This is the second domain with the highest scores, even with the trend of higher mean scores of the HCD guidelines (70% and 5.2) when compared to those of the PRD guidelines (61% and 4.6). In general, the guidelines in both groups were written in plain language, often addressing various aspects of the health issue.
The HCD guidelines are more concise than the PRD ones. The latter have a wide variety of formats and sizes of documents, contrasting with the first group, which had a standard format previously established by a specific ordinance that offered a roadmap for the PCDTs development, and consequently showed a more coherent presentation [22, 23].
Domain 5—applicability
The “applicability” domain scored low in both groups. This domain includes questions that describe facilitators and barriers to the guidelines’ application, potential resource implications and monitoring criteria [11].
Despite the small margin of difference, the PRD guidelines scores were higher (33% and 2.9%) than those of the HCD guidelines (25% and 2.5%). This demonstrates a certain concern of guideline development panels with the application of the guidelines in contexts of greater operational difficulty. The normative character and little approach to guideline implementation actions were previously described by other authors who evaluated some PCDTs of the Brazilian Ministry of Health, finding results similar to those of the present study, such as those by Ronsoni et al. (2015), Ronsoni (2013) (37.5% and 3.3) and Molino et al. (2016) (34%) [9, 12, 19].
Domain 6 –editorial independence
Similar to the "rigour of development" domain, this was one of the domains that showed the highest discrepancy between the scores of the HCD guidelines (43% and 3.6) and the PRD guidelines (1% and 1.1). This item evaluates the degree of reliability and equity in the guideline development. In this item, it is expected that there will be information and statements that demonstrate that the views or interests of the funding body did not influence the final recommendations.
The HCD guidelines did not reach higher scores solely because they did not specify more clearly the procedures used to avoid the funding bodies’ influence on their content and how potential competing interests were managed. However, there were supplementary documents and terms defining the procedures and responsibilities of each group of participants, and specifying that a declaration of competing interests was signed by all parties [12, 23].
On the other hand, the PRD guidelines did not have any similar procedure or standard, and there was no record attesting that a competing interest statement was signed or any actions were taken to avoid the funding bodies’ influence on the content and formulation of recommendations.
This points to a variability in the areas of the Ministry of Health concerning the procedures for declaring competing interests and how these conflicts are handled (or not). These are essential steps to ensure that the content of the recommendations is unbiased and the publications are reliable [2,3].
International comparison
With the intention of analyzing the results from an international standpoint, the quality scores per domain were compared with the findings of Alonso-Coello et al. (2010), who carried out a systematic review of studies from around the world that brought together 625 different guidelines. Fig 1 was prepared with the most current data from that study, including guidelines published since 2003 [13].
It can be seen that, in general, the HCD guidelines are in line with international standards, with similar and slightly higher scores in all the domains evaluated.
The PRD guidelines, on the other hand, showed lower scores in most domains. The main contrasts can be noticed in the "rigor of development" and "editorial independence" domains, reinforcing the findings that these documents have methodological criteria problems in what concerns evidence-based health and the procedures adopted to guarantee an unbiased process.
Overall guideline assessment
The comparison of this study’s results with international findings demonstrates the effectiveness of the method used. However, the participating appraisers did not consider it appropriate to individually assess whether a particular guideline would be recommended or not for use, as described in the AGREE II. Therefore, the appraisers decided to adopt criteria based on the analysis of previous domains.
Due to the greater emphasis on the "rigor of development" domain in similar studies [12,13,24,25], it was decided that this domain should be the central item for guideline assessment because of its relevance and greater number of analysis parameters.
According to the abovementioned criteria, the PCDTs for Rheumatoid Arthritis (HCD 17), Hepatitis C Virus Infection (HCD 18) and Breast Cancer (HCD 19) were considered as "recommended", and the other HCD guidelines (20 to 32) were considered as "recommended, with modifications". This result is in line with three other similar studies which pointed out that the PCDTs of the Ministry of Health, even with a methodological standard comparable to other international guidelines and superior to other Brazilian guidelines, need continuous improvement [9,12,19].
All the PRD guidelines (01 to 16) were considered as "not recommended" since, according to the AGREE II assessment, they did not meet the minimum criteria of the scientific method description, failing to provide whether or not the document was drawn up from an evidence-based health perspective.
The clinical guidelines with the highest score percentage in the AGREE II concern diseases that demand a heavy medicine expenditure of the Ministry of Health, namely, rheumatoid arthritis, hepatitis C virus infection and breast cancer, precisely those with a billion dollar budgetary impact for the Brazilian health system and for most health systems in developed countries [26, 27]. The lowest scores were of documents addressing the treatment of poverty-related diseases of low visibility such as intestinal parasitosis, human brucellosis, onchocerciasis, trachoma, among others. This indicates that the safety of patients suffering from such diseases and undergoing treatment cannot be ensured by the current treatment protocols.
This contrasts with the influence that budget pressure exerts on action priorities and investments in health qualification. This does not seem to be a coherent decision, since it is poverty-related diseases that have less potential for evidence generation due to less investment in clinical research and new technology development. Despite the decline in the number of cases of these diseases over the years, given the ongoing epidemiological transition, they still have a major impact on morbidity and mortality when we look at disease burden data, in addition to the contemporary risk of re-emergence of some previously controlled epidemics [6]. In addition, there is a trend for such issues to be less addressed in health training curricula, which consequently generates a greater need for consistent clinical guidelines [28].
Final remarks
Based on the findings of this study, it is possible to verify a double quality standard in clinical guideline development of the Brazilian Ministry of Health, where the guidelines for global diseases that demand high-cost technologies showed higher scores in the AGREE II assessment when compared to those for poverty-related diseases which, normally, demand old and low-cost technologies.
Discrepancies found in quality and methods of selection, use of scientific evidence, and editorial transparency criteria are of concern and should be reviewed for the poverty diseases guidelines. Future studies must include the clinical credibility and implementation of guideline recommendations, as recommended by the new AGREE-REX [29]. This study did not include some of the most ordinary diseases, as those most frequent in primary health care services (such as diabetes and hypertension) and can add a broader view of guidelines development in Brazilian health system.
It can be inferred that the budgetary power and resource pressure also influence the action priorities and the quality of care in the public sector, making diseases that are neglected by the pharmaceutical market also be neglected at certain levels by the health system.
The neglect of the poverty-related diseases observed in this study concerns not only financial, scientific and product development investments but also the standardization and quality of the clinical services offered to the affected population.
Supporting information
S1 File. Links to guidelines.
Links to clinical guidelines used in this study.
https://doi.org/10.1371/journal.pone.0204723.s001
(DOCX)
References
- 1. Rosenfeld RM, Shiffman RN. Clinical practice guideline development manual: A quality-driven approach for translating evidence into action. Otolaryngol—Head Neck Surg. 2009;140(6 SUPPL.):1–43.
- 2. Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: The current state of play. Int J Qual Heal Care. 2016;28(1):122–8.
- 3. Ribeiro RC. Diretrizes clínicas: como avaliar a qualidade? Rev Bras Clin Med. 2010;8(4):350–5.
- 4. Mega TP, Lopes AC de F, Santos VCC, Petramale CA. Protocolos clínicos e diretrizes terapêuticas no sus: histórico, desafios e perspectivas. Gestão e Saúde. 2015. p. Pág. 3275–3285.
- 5. Santana RS, Lupatini E de O, Leite SN. Registro e incorporação no SUS: Barreiras de acesso a medicamentos para doenças da pobreza? Cien Saude Colet. 2017; 22(5):1417–1428. pmid:28538914
- 6. Santana RS, Leite SN. Prioridades da pesquisa clínica com medicamentos no Brasil e as doenças da pobreza. Rev Panam Salud Publica. 2016;40(5):355–61.
- 7.
Médicos Sem Fronteiras. Desequilíbrio fatal: a crise em pesquisa e desenvolvimento de drogas para doenças negligenciadas. Geneva: Grupo de Trabalho de Drogas para Doenças Negligenciadas, Médicos Sem Fronteiras. 2001.
- 8.
World Health Organization. Global report for research on infectious diseases of poverty. 2012. 1–184 p.
- 9. Ronsoni RDM, Pereira CC de A, Stein AT, Osanai MH, Machado CJ. Avaliação de oito Protocolos Clínicos e Diretrizes Terapêuticas (PCDT) do Ministério da Saúde por meio do instrumento AGREE II: um estudo piloto. Cad Saúde Pública. 2015;31(6):1157–62.
- 10. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. Can Med Assoc J. 2010;182(18):839–42.
- 11.
AGREE Next Steps Consortium. The AGREE II Instrument [versão em português]. Disponível em: http://www.agreetrust.org/wp-content/uploads/2013/06/AGREE_II_Brazilian_Portuguese.pdf. 2009. 53p.
- 12. Molino C de GRC, Romano-Lieber NS, Ribeiro E, de Melo DO. Non-Communicable Disease Clinical Practice Guidelines in Brazil: A Systematic Assessment of Methodological Quality and Transparency. Gan SH, editor. PLoS One. 2016;11(11):e0166367. pmid:27846245
- 13. Alonso-Coello P, Irfan A, Sola I, Gich I, Delgado-Noguera M, Rigau D, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. BMJ Qual Saf. 2010;19(6):e58–e58.
- 14. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. pmid:843571
- 15. Van de Bovenkamp HM, Zuiderent-Jerak T. An empirical study of patient participation in guideline development: exploring the potential for articulating patient knowledge in evidence-based epistemic settings. Health Expect [Internet]. 2015;18(5):942–55. pmid:23634973
- 16. Van de Bovenkamp HM, Trappenburg MJ. Reconsidering Patient Participation in Guideline Development. Heal Care Anal [Internet]. Springer US; 2009;17(3):198–216.
- 17. Utens CMA, Dirksen CD, van der Weijden T, Joore MA. How to integrate research evidence on patient preferences in pharmaceutical coverage decisions and clinical practice guidelines: A qualitative study among Dutch stakeholders. Health Policy (New York). 2016;120(1):120–8.
- 18. Serrano-Aguilar P, Trujillo-Martin MM, del Pino-Sedeño T, Pérez de la Rosa A, de Pascual y Medina AM, Perestelo-Pérez L, et al. Patient participation in the development of a clinical guideline for inherited retinal dystrophies. Expert Opin Orphan Drugs [Internet]. Taylor & Francis; 2016;4(7):691–7.
- 19.
Ronsoni RDM. Avaliação dos protocolos clínicos e diretrizes terapêuticas do Ministério da Saúde segundo método AGREE II. Escola Nacional de Saúde Publica Sérgio Arouca; 2013.
- 20.
Brasil. Presidência da República. Decreto no 7.646 de 21 de dezembro de 2011 que Dispõe sobre a Comissão Nacional de Incorporação de Tecnologias no SUS e sobre o processo administrativo para incorporação, exclusão e alteração de tecnologias em saúde pelo SUS, e dá outras providências. Disponível em: http://www.planalto.gov.br/ccivil_03/_Ato2011-2014/2011/Decreto/D7646.htm. 2011.
- 21.
Brasil. Presidência da República. Decreto no 7.508 de 28 de Junho de 2011 o planejamento da saúde, a assistência à saúde e a articulação interfederativa, e dá outras providências. Disponível em: http://www.planalto.gov.br/ccivil_03/_ato2011-2014/2011/decreto/D7508.htm. 2011.
- 22.
Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Portaria no 375 de 10 de novembro de 2009 que aprova o roteiro a ser utilizado na elaboração de Protocolos Clínicos e Diretrizes Terapêuticas (PCDT). Disponível em: http://bvsms.saude.gov.br/bvs/saudelegis/sas/2009/prt0375_10_11_2009.html. 2009.
- 23.
Brasil. Ministério da Saúde. Secretaria de Ciência Tecnologia e Insumos Estratégicos. Portaria no 27, de 12 de junho de 2015 que aprova o Fluxo de trabalho para elaboração e atualização dos PCDT no âmbito da Conitec. Disponível: http://bvsms.saude.gov.br/bvs/saudelegis/sctie/2015/prt0027_12_06_2015.html. 2015.
- 24. Smith CAM, Toupin-April K, Jutai JW, Duffy CM, Rahman P, Cavallo S, et al. A Systematic Critical Appraisal of Clinical Practice Guidelines in Juvenile Idiopathic Arthritis Using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) Instrument. Nurmohamed M, editor. PLoS One. 2015;10(9):e0137180. pmid:26356098
- 25. Zhang Z, Guo J, Su G, Li J, Wu H, Xie X. Evaluation of the Quality of Guidelines for Myasthenia Gravis with the AGREE II Instrument. Zhang H-L, editor. PLoS One. 2014;9(11):e111796. pmid:25402504
- 26.
Brasil. Ministério da Saúde. Secretaria de Ciência e Tecnologia e Insumos Estratégicos. Componente Especializado da Assistência Farmacêutica: inovação para a garantia do acesso a medicamentos no SUS. Brasília: Ministério da Saúde; 2014. 163 p.
- 27. Andrieux-Meyer I, Cohn J, de Araújo ESA, Hamid SS. Disparity in market prices for hepatitis C virus direct-acting drugs. Lancet Glob Heal. 2015;3(11):e676–7.
- 28. Crump JA, Sugarman J, Working Group on Ethics Guidelines for Global Health Training (WEIGHT). Ethics and Best Practice Guidelines for Training Experiences in Global Health. Am J Trop Med Hyg. 2010;83(6):1178–82. pmid:21118918
- 29.
AGREE-REX: Recommendation EXcellence—AGREE Enterprise website [Internet]. Agreetrust.org. 2016 [cited 23 October 2016]. Available from: http://www.agreetrust.org/agree-research-projects/agree-rex-recommendation-excellence/"