http://orcid.org/0000-0001-6101-3946
Figures
Abstract
Background
The development of trustworthy guidelines requires substantial investment of resources and time. This highlights the need to prioritize topics for guideline development and update.
Objective
To systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines.
Methods
We searched Medline and CINAHL electronic databases from inception to July 2019, supplemented by hand-searching Google Scholar and the reference lists of relevant studies. We included studies describing prioritization exercises that have been conducted during the de novo development, update or adaptation of guidelines addressing clinical, public health or health systems topics. Two reviewers worked independently and in duplicate to complete study selection and data extraction. We consolidated findings in a semi-quantitative and narrative way.
Results
Out of 33,339 identified citations, twelve studies met the eligibility criteria. All included studies focused on prioritizing topics; none on questions or outcomes. While three exercises focused on updating guidelines, nine were on de novo development. All included studies addressed clinical topics. We adopted a framework that categorizes prioritization into 11 steps clustered in three phases (pre-prioritization, prioritization and post-prioritization). Four studies covered more than half of the 11 prioritization steps across the three phases. The most frequently reported steps for generating initial list of topics were stakeholders’ input (n = 8) and literature review (n = 7). The application of criteria to determine research priorities was used in eight studies. We used and updated a common framework of 22 prioritization criteria, clustered in 6 domains. The most frequently reported criteria related to the health burden of disease (n = 9) and potential impact of the intervention on health outcomes (n = 5). All the studies involved health care providers in the prioritization exercises. Only one study involved patients. There was a variation in the number and type of the prioritization exercises’ outputs.
Citation: El-Harakeh A, Lotfi T, Ahmad A, Morsi RZ, Fadlallah R, Bou-Karroum L, et al. (2020) The implementation of prioritization exercises in the development and update of health practice guidelines: A scoping review. PLoS ONE 15(3): e0229249. https://doi.org/10.1371/journal.pone.0229249
Editor: Vicki Jane Flenady, Mater Medical Research Institute, AUSTRALIA
Received: September 2, 2019; Accepted: February 2, 2020; Published: March 20, 2020
Copyright: © 2020 El-Harakeh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This study was supported by the Alliance for Health Policy and Systems Research, World Health Organization (WHO), Geneva. The funder was not involved in the design of the study, in the collection, analysis, and interpretation of data or in the writing of the manuscript.
Introduction
Health practice guidelines are “systematically developed statements” intended to optimize care at the clinical, public health and health systems levels [1]. The development of trustworthy guidelines requires substantial investment of resources [2, 3] and time, often taking an average of two to three years [4, 5]. In addition, and with the expansion of medical research and the emergence of new and innovative technologies, guidelines should be updated as necessary [6]. This highlights the need to prioritize topics for guideline development and update.
In fact, the importance of prioritization for guideline development has been recognized by many guideline developing groups [7]. It ensures that limited resources are aligned with priority needs for guideline development [8]. Furthermore, prioritization exercises engaging a wide range of stakeholders enhance the relevance and potential uptake of priority topics by end users [9–11]. This represents an important step toward improving the delivery of evidence-informed care.
The guideline development process includes, in addition to prioritizing topics, the prioritization of questions and outcomes [5, 12]. Also, prioritization should be considered when adapting guidelines to select priority questions from among those addressed in the original guidelines [13]. Similarly, guideline developers need to prioritize which guidelines, guideline sections, or recommendations should be updated [14].
With the growing interest among researchers in prioritizing topics for the de novo development, update and adaptation of guidelines, several exercises have been conducted to yield explicit and transparent prioritization [15–17]. Some investigators relied primarily on the use of criteria to select priority guideline topics [18], while others have followed multicomponent prioritization processes and have used established tools and approaches [19].
While some efforts have been invested in synthesizing the evidence on prioritization for guideline updating [20], none have described prioritization for the de novo development, update or adaptation of guidelines. As such, the objective of this study was to systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines.
Methods
We conducted a scoping review of published prioritization exercises implemented as part of the de novo development, update or adaptation of health practice guidelines. We followed standard methodology and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines for reporting scoping reviews [21] (see S1 File). This study is based on a detailed protocol available in S2 File. The project’s team included a multidisciplinary group of professionals in the clinical, public health, and health policy and systems fields, with expertise in guideline development and priority setting.
Eligibility criteria
- Paper type: We included descriptive reports and excluded commentaries, editorials, letters, correspondences, news, and abstracts.
- Scope: We included papers about the de novo development, update or adaptation of health practice guidelines addressing clinical, public health or health system topics. Also, we excluded papers reporting proposed approaches, without any applied exercise.
- Focus: We included papers that aimed to prioritize one of the following: guideline topics, questions/recommendations, or outcomes. We excluded papers reporting on individual prioritization criteria or items.
- Setting: We included eligible papers irrespective of the setting (low-, middle- or high-income countries; primary, secondary or tertiary healthcare facilities).
Information sources and literature search
We searched Medline and CINAHL electronic databases from their respective inception date to July 2019. We also manually searched Google Scholar in July 2019. We developed the search strategy with the assistance of a medical librarian. The search included both Medical Subject Headings (MeSH) terms and free-text words and combined various terms for health prioritization (see S3 File). We did not use any language or date restrictions. We screened the reference lists of included and other relevant papers.
Study selection
The study selection process consisted of two phases: title and abstract screening and full text screening. Teams of two reviewers (AEH, TL, AA, RZM, RF, LBK) independently and in duplicate screened the titles and abstracts of citations captured by the search for potential eligibility. The reviewers then obtained the full texts of citations judged as potentially eligible by at least one of the two reviewers. Then, they screened the full texts in duplicate and independently for eligibility using a standardized and pilot-tested screening form and following a calibration exercise. At this phase, the reviewers resolved disagreements by discussion or with the input of a third reviewer (EAA) when consensus could not be reached.
Data extraction
Prior to data extraction, we conducted two calibration exercises to enhance the validity of the process. Three reviewers (AEH, TL and AA) worked in duplicate and independently using a standardized and pilot-tested data extraction form (see S4 File). They resolved disagreements by discussion or with the help of a third reviewer (EAA). We collected the following data from each included paper:
- General characteristics: authors, lead entity, target audience, year of prioritization conduct, scope of prioritization, topic (specific domain), focus of prioritization, type of guideline development, and support/funding;
- Steps of prioritization:
- Pre-prioritization phase (development of guiding/ethical principles, generation of initial list of topics and collection of technical data to inform discussions);
- Prioritization phase (use of established prioritization methods, research gap analysis, use of criteria, prioritization/ranking);
- Post-prioritization phase (refinement of priorities into guideline topics, dissemination and implementation, revision mechanism, monitoring and evaluation).
Data synthesis
Due to the descriptive nature of data, we synthesized the findings in a semi-quantitative way. We used the extracted data to come up with common categorizations of relevant concepts (e.g., prioritization steps, generation of initial list of topics), using an iterative process of review and refinement. As part of this process, we analyzed the content of each study at least twice; when drafting the initial categories, and after producing an advanced draft. We reported the results narratively and in a tabular format.
The concepts addressed in our analysis included:
- Prioritization steps; we adopted 11 categories of prioritization steps, which we developed for a recent systematic review on prioritization for evidence synthesis [22];
- Generation of initial list of topics (descriptive analysis);
- Output of the priority setting exercises (descriptive analysis);
- Stakeholder involvement; we adopted the categories we developed for a recent systematic review on prioritization for evidence synthesis [22], which is based on the 7Ps framework [23];
- Prioritization criteria; we used a common framework of prioritization criteria that we developed for a recent systematic review on prioritization approaches in the development of health practice guidelines [24] (see S5 File).
Results
Study selection
Fig 1 shows the study flow diagram summarizing the study selection process. Out of the 33,339 identified citations, twelve papers met our inclusion criteria [14–19, 25–30]. We excluded 896 articles based on full text screening for the following reasons: not a paper type of interest (n = 49); not describing a reproducible prioritization exercise (n = 322); not about health practice guidelines (n = 525).
General characteristics of the included studies
Table 1 shows the general characteristics of the twelve included studies. One prioritization exercise was conducted in 1998 while the remaining ones were conducted between 2010 and 2017. Half of the prioritization exercises were implemented at a national level (n = 6) [14, 16, 19, 27, 29, 30], while the rest were implemented at regional (n = 3) [25, 26, 28], provincial (n = 2) [15, 18], or international levels (n = 1) [17]. All of the prioritization exercises focused on prioritizing guideline topics (as opposed to prioritizing questions or outcomes) and addressed clinical topics. While three exercises focused on updating guidelines [14, 15, 26], nine focused on de novo development [16–19, 25, 27–30], and none addressed adaptation of guidelines. Funding sources were mainly professional societies (n = 5) [16, 25, 26, 28, 29] or public funding (n = 4) [15, 18, 19, 30].
We present below our findings summarized as the steps of prioritization, with a focus on two of those steps for which detailed information was available (namely the generation of initial list of topics and prioritization criteria). Then, we review stakeholder involvement in the prioritization exercises. Finally, we review the processes and outputs of the prioritization exercises stratified by whether the prioritization exercise was related to de novo development or updating.
Steps of prioritization
Table 2 outlines the prioritization steps addressed in the twelve included studies across three phases: pre-prioritization, prioritization and post-prioritization phases. Although most of the studies (n = 11) addressed at least one step in the pre-prioritization phase [14–17, 19, 25–30], less than half (n = 5) addressed at least one step during the post-prioritization phase [14, 15, 17, 25, 27]. Four studies covered more than half of the 11 prioritization steps across the three phases [14, 15, 19, 25].
Prior to conducting the prioritization exercises, most of the included studies (n = 10) generated initial lists of topics [15–17, 19, 25–30], while only a few studies reported on the development of ethical principles to guide the conduct of the exercise (n = 3) [25, 26, 30], or on the collection of technical data to inform further discussions (n = 2) [14, 19].
Most studies used prioritization criteria (n = 8) [14, 15, 18, 19, 25, 26, 29, 30] and ranked priorities (n = 11) [14–17, 19, 25–30] during the prioritization phase. Out of eight studies, two refined priorities into guideline topics (excluding four studies where this step was not applicable as the exercise started with topics and not broad themes) [17, 25]. Less than half of the studies conducted or reported on a plan for dissemination and implementation (n = 3) [15, 17, 25] or monitoring and evaluation (n = 3) [14, 15, 27]. All of the studies involved stakeholders during various prioritization steps across the three phases, with the majority involving stakeholders in the generation of initial list of topics (n = 8) [16, 17, 25–30], use of criteria (n = 8) [14, 15, 18, 19, 25, 26, 29, 30] and prioritization/ranking of priorities (n = 10) [14–17, 19, 25, 26, 28–30]. Only one study engaged stakeholders in the post-prioritization phase [17].
Generation of initial list of topics.
Table 3 shows the steps involved in generating initial list of topics. One frequently used method for generating initial list of topics was seeking input from stakeholders (n = 8) [16, 17, 25–30]. Other methods included reviewing the literature (n = 7) [15, 17, 19, 25–27, 30], referring to the health information system (n = 1) [19] and to previous priority setting exercises (n = 1) [15].
Prioritization criteria.
Table 4 presents the prioritization criteria that 10 out of the 12 studies reported on. Eight studies used their proposed criteria as part of their prioritization exercises [14, 15, 18, 19, 25, 26, 29, 30], while two studies proposed criteria but did not use them in the exercise [16, 28].
The studies included a mean of seven prioritization criteria (range: 3–13), with a total of 70 criteria reported. We attempted to match the 70 criteria to a published framework of 20 guideline prioritization criteria classified into six domains (Table 4 and S5 File). During the matching process, we added two criteria that emerged from the included studies (i.e., availability of low certainty evidence and acceptability). Table 4 shows the classification of the identified prioritization criteria according to the new framework.
The most frequently reported criteria related to the health burden of disease (n = 9) [14–16, 18, 19, 25, 26, 28, 29] and potential impact of the intervention on health outcomes (n = 5) [16, 25, 26, 29, 30]. None of the studies included equity relevance of the disease or urgency as explicit criteria. Eleven (out of the total of 22 criteria listed in the framework) was the highest number of criteria reported by a study [16].
Stakeholder involvement
Table 5 shows the types of stakeholders involved in prioritizing guideline topics and the methods used to engage them. All included studies involved healthcare providers in the prioritization exercises, while four studies (33%) involved researchers [14, 19, 29, 30], and only one study (8%) involved patients (Table 5) [26]. In Loeffen et al., the authors reported not including patients, parents, and caretakers as they planned to understand the needs of the professionals beforehand [29]. Seven studies described stakeholder recruitment methods which ranged from the use of professional networks (e.g., members directory), to searching databases and emailing clinicians [16–18, 25, 26, 28, 30].
All prioritization exercises surveyed stakeholders (e.g., Delphi approach) as a method of engagement. Other methods included the nominal group technique (n = 1) [25] and consensus conference (n = 1) [14]. Stakeholders were engaged via an online platform (e.g., online surveys, email discussions) in all included studies, with two studies using both online and in-person meetings [14, 25]. The frequency of engagement varied from only once (n = 4), to twice (n = 6), or three times (n = 2).
Prioritization processes and outputs for de novo development (n = 9)
Table 6 describes the processes and outputs of the prioritization exercises. The nine studies that implemented prioritization processes for the de novo development of guidelines followed common steps of reviewing the literature (n = 5) [17, 19, 25, 27, 30] and/or engaging stakeholders (n = 9) [16–19, 25, 27–30], while considering the availability of existing guidelines on the suggested topics (n = 3) [16, 19, 28]. In fact, one study conducted the prioritization exercise regardless of existing guidelines which resulted in prioritizing 20 topics; all of which were covered by existing guidelines [16].
There was a variation in the types of outputs of the prioritization exercises. Most of the studies prioritized topics (n = 6) [16, 18, 25, 27–29], one prioritized clinical areas (n = 1) [17], one prioritized drug classes (n = 1) [30], and one prioritized chronic diseases (n = 1) [19]. None prioritized questions or outcomes. Most of the studies provided ranked lists of priorities (n = 8) [16, 17, 19, 25, 27–30], while one study had a topic suggested prior to the exercise and then confirmed as a result of prioritization [18]. The numbers of priorities derived from the initial lists varied between the studies (range 1–46).
Prioritization processes and outputs for updating (n = 3)
Studies that implemented prioritization exercises for updating (n = 3) either assessed candidate guideline documents for updating [15] or selected a specific guideline a priori and assessed potential topics or sections to be covered by updating [14, 26].
Agbassi et al. used a stepwise process in which two questionnaires were implemented to prioritize guidelines for updating and to assess the effect of new evidence on existing recommendations [15]. van der Veer et al. consulted clinicians and patients about priority topics to be covered by the update of 2007 vascular access guideline of the European Renal Best Practice [26]. Becker et al. classified guideline sections of a German clinical practice guideline based on evidence and clinical relevance (Table 6) [14]. Two studies used categories (e.g., urgent, high, medium, or low) to reflect the relative need for updating [14, 15]. One study reported a median of 167 days for the time taken to implement the prioritization process (range 18–358 days) [15].
In terms of outputs, two studies provided ranked lists of priorities [14, 26]. One study prioritized 8 out of 151 guideline documents for updating [15], another study prioritized 15 out of 35 guideline sections [14]. The third study generated a list of 42 topics from an initial list of 39 topics to be covered by the updated guideline [26].
Discussion
Summary of findings
We systematically reviewed the literature for prioritization exercises that have been conducted for the de novo development, update or adaptation of health practice guidelines. We identified twelve eligible studies that focused on prioritizing clinical topics and were predominantly conducted for the de novo development of guidelines; none addressed adaptation. The priority setting exercises consisted of several steps that we grouped in three phases: pre-prioritization, prioritization and post-prioritization. The two most commonly used steps were the generation of an initial list of topics (mostly by seeking input from stakeholders or by reviewing the literature) and ranking of priorities. The two least used steps were research gap analysis and having a revision mechanism. Most of the included studies used prioritization criteria as part of the exercises, with the most common criteria being the health burden and potential impact of the intervention on health outcomes. All studies involved stakeholders, particularly healthcare providers, in prioritizing guideline topics. Stakeholders were mainly involved in the generation of initial list of topics, use of criteria and ranking of priorities.
Interpretation of findings
We observed that the generated priority topics were generally broad and non-specific. This might have been due to the fact that the vast majority of the exercises did not describe a step of refinement of the priority topics. It is essential to refine the topics in a way that would enable an easy transition from topics into meaningful questions appropriate for guideline development [31].
In addition, one of the Institute of Medicine (IOM) standards in guideline development is establishing transparency [32]. The IOM emphasizes the need for detailed and publicly accessible guideline development processes including methods for priority setting [32, 33]. Such processes would increase the credibility and the potential uptake of the end results [34]. We found only three studies reporting on the development of ethical principles to guide the conduct of the exercises.
Prioritization should be supported by an effective dissemination plan to ensure that generated priorities inform prospective research and ultimately improve health [35]. The dissemination of priorities to researchers and funders helps in directing research agendas to guideline topics that are most important to stakeholders [36]. Although the number of prioritization exercises has been increasing over time, very few exercises reported on dissemination or implementation strategies. Consistent with findings of earlier reviews on prioritization for health research [35, 37], only three of the included exercises mentioned dissemination.
Furthermore, our reliance on a recently developed framework of prioritization criteria allowed us to categorize 66 out of the 70 criteria in the included studies. The most commonly used criterion was ‘health burden’. The majority of the remaining criteria were used by two or less exercises. For instance, although equity is one of the most frequently reported criteria in the priority setting literature [38], none of the exercises considered the equity relevance of the condition. An equity-oriented approach to priority setting is important for ensuring inclusiveness [38, 39]. While this could reflect a decision by the designers of the exercises to focus on few but relevant criteria, it could also point to the failure of these exercises to be comprehensive in their use of criteria. Indeed, Nast et al. highlighted the need for such exercises to address a wide range of explicit criteria that extend beyond disease-related factors [28].
Overall, the observed variation in the prioritization steps and criteria used in the included prioritization exercises could potentially be explained by the need to tailor the decision on how to conduct a prioritization exercise to the needs of relevant stakeholders and to the available resources, such as time and funding.
A recent systematic review highlighted the opportunity to engage diverse types of stakeholders in prioritizing guideline topics [24]. Incorporating views of various stakeholders in guideline development can potentially reduce a biased selection of topics by few groups and increase transparency [7, 31]. Moreover, considering the needs of different stakeholders may improve the uptake and usability of guidelines [40]. While all exercises included in this review involved health care providers, only four and one respectively engaged researchers and patients. None engaged the other eight types of stakeholders that we assessed. Patient involvement in priority setting for guideline development has been widely supported in the literature [41–44]. It helps direct guidelines toward questions that matter most to patients, expanding beyond the interests of researchers and clinicians [45–47]. However, potential barriers to patient involvement include limited resources (e.g., lack of funds and stakeholder time), slowed down and longer process, and difficulty in identifying appropriate representatives [42, 43, 48]. In addition, guidance on how to engage patients is limited [43]. Despite potential challenges, some of the available methods for engaging patients have been evaluated, and thus can be used to ensure appropriate patient involvement [49, 50]. Moreover, maintaining regular communication with patients or their representatives facilitates meaningful engagement [51].
The online approach to engaging stakeholders was adopted by all studies. Online platforms are considered practical and cost-efficient ways of engaging stakeholders [52]. Other methods that were not frequently used in the prioritization exercises include in-person meetings and workshops. Although not widely used (for practical and financial reasons), those methods might improve interactions and discussions between stakeholders and in turn generate different priorities. In addition, face-to-face meetings are one of the knowledge exchange methods with the greatest impact on policymaking [53]. Furthermore, most stakeholders were engaged through the Delphi survey method, which is a simple consensus tool for obtaining the views of a large group of relevant stakeholders [54] using structured and iterative group interactions [55]. The Delphi method is commonly used in both guideline development and in health research prioritization [56–58], explaining its use in prioritization for guideline development.
The included studies on prioritization for updating conducted the exercises at different time points of the updating process. One exercise was implemented to identify the clinical guidelines in greatest need of update after a surveillance process, while the other two exercises aimed to identify the topics or sections in greatest need of update for a selected guideline.
Strengths and limitations
We used a rigorous and transparent process including a comprehensive search strategy, duplicate and independent selection, and duplicate and independent data extraction [59]. In addition, and by drawing on an extensive body of literature since the 1990s and up to July 2019, this review synthesizes almost three decades of published research on prioritization for guideline development. On the other hand, we built on two recent systematic reviews of prioritization approaches to develop our data extraction and analysis framework (e.g., how to categorize the steps of prioritization, prioritization criteria).
There are limitations to our scoping review process. First, we did not appraise the quality of the included studies. However, this is consistent with the scoping review methodology [21] and no tool has been developed for the critical appraisal of priority setting exercises. Second, we did not search the grey literature, particularly websites of guideline developing organizations, due to time and resource constraints.
Comparison to other reviews
Our work adds to former reviews on the topic, e.g., the review by Garcia et al. which focused on the update of health decision-making tools, one of which was guidelines [20]. Consistent with our findings, Garcia et al. reported variability in the methods used to implement the prioritization exercises for updating. On the other hand, our study presents a more in-depth analysis of relevant characteristics such as the steps and criteria for prioritization exercises. Our list of criteria is consistent with, but a bit more comprehensive than the list by Garcia et al.
Implications for practice
Our findings can assist clinicians, researchers, funders, policymakers, and other stakeholders seeking to develop health practice guidelines in prioritizing topics to be addressed. Given that there are no standard prioritization best practices for guideline development [7], it might be challenging to provide specific guidance on which prioritization exercise to use. However, the decision on whether and how to conduct a prioritization exercise should be tailored to the needs of relevant stakeholders and to the available resources, including time and funding. Furthermore, the detailed lists of identified steps and criteria can serve as a menu of options for guideline developers to select from, as judged appropriate to the context, and through a transparent decision-making process.
Implications for future research
There is a need to develop methods and guidance for prioritization of not only topics, but also for questions and outcomes in guidelines projects. Exploring the same question of this study through the analysis of guideline handbooks would be helpful for that purpose. Further rigorous evaluation research can help with a better understanding of potential facilitators and barriers to prioritization. Moreover, and because all of the included conducted exercises were developed by researchers from high-income countries, future studies can focus on the effectiveness of the exercises in low- and middle-income countries. It is also essential to evaluate the impact of those exercises on resource allocation and on clinical outcomes.
Conclusions
This review identified 12 prioritization exercises that addressed different aspects of priority setting for guideline development and update. The detailed lists of prioritization steps and criteria can serve as a menu of options for guideline developers to select from, as judged appropriate to the context. This review also provided insight into the types of stakeholders involved in the prioritization of health practice guidelines. Engaging diverse stakeholders, particularly patients and their representatives, is essential to align guideline development with the needs and priorities of relevant stakeholders. However, the roles of stakeholders in the prioritization processes need to be further investigated.
References
- 1.
Field MJ, Lohr KN. Clinical practice guidelines: directions for a new program: National Academies Press; 1990.
- 2. Jaeschke R, Guyatt GH, Dellinger P, Schünemann H, Levy MM, Kunz R, et al. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ. 2008;337:a744. pmid:18669566
- 3. Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Developing guidelines. BMJ. 1999;318(7183):593–6. pmid:10037645
- 4. Morgan RL, Florez I, Falavigna M, Kowalski S, Akl EA, Thayer KA, et al. Development of rapid guidelines: 3. GIN-McMaster Guideline Development Checklist extension for rapid recommendations. Health Research Policy and Systems. 2018;16(1):63. pmid:30005679
- 5.
World Health Organization. WHO handbook for guideline development: World Health Organization; 2014.
- 6. Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: the current state of play. International journal for quality in health care: journal of the International Society for Quality in Health Care. 2016;28(1):122–8. Epub 2016/01/23. pmid:26796486; PubMed Central PMCID: PMC4767049.
- 7. Reveiz L, Tellez DR, Castillo JS, Mosquera PA, Torres M, Cuervo LG, et al. Prioritization strategies in clinical practice guidelines development: a pilot study. Health Res Policy Syst. 2010;8:7. Epub 2010/03/09. pmid:20205926; PubMed Central PMCID: PMC2846928.
- 8. Atkins D, Perez-Padilla R, MacNee W, Buist AS, Cruz AA. Priority setting in guideline development: article 2 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proceedings of the American Thoracic Society. 2012;9(5):225–8. pmid:23256163
- 9. Armstrong MJ, Rueda J-D, Gronseth GS, Mullins CD. Framework for enhancing clinical practice guidelines through continuous patient engagement. Health Expectations. 2017;20(1):3–10. pmid:27115476
- 10.
National Health and Medical Research Council. Guidelines for guidelines. Engaging stakeholders. 2018.
- 11. O’Haire C, McPheeters M, Nakamoto E, LaBrant L, Most C, Lee K, et al. Engaging stakeholders to identify and prioritize future research needs. 2011.
- 12. Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. Cmaj. 2014;186(3):E123–E42. pmid:24344144
- 13. Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE Evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. Journal of Clinical Epidemiology. 2017;81:101–10. pmid:27713072
- 14. Becker M, Jaschinski T, Eikermann M, Mathes T, Bühn S, Koppert W, et al. A systematic decision-making process on the need for updating clinical practice guidelines proved to be feasible in a pilot study. Journal of clinical epidemiology. 2018;96:101–9. pmid:29289763
- 15. Agbassi C, Messersmith H, McNair S, Brouwers M. Priority-based initiative for updating existing evidence-based clinical practice guidelines: the results of two iterations. Journal of clinical epidemiology. 2014;67(12):1335–42. pmid:25216900
- 16. Borgonjen RJ, van Everdingen JJ, van de Kerkhof PC, Spuls PI. Prioritizing dermatoses: rationally selecting guideline topics. Journal of the European Academy of Dermatology and Venereology: JEADV. 2015;29(8):1636–40. Epub 2014/09/02. pmid:25175822.
- 17. Kerr M, Scheepers M, Arvio M, Beavis J, Brandt C, Brown S, et al. Consensus guidelines into the management of epilepsy in adults with an intellectual disability. Journal of intellectual disability research: JIDR. 2009;53(8):687–94. Epub 2009/06/17. pmid:19527434.
- 18. Brouwers MC, Chambers A, Perry J. Can surveying practitioners about their practices help identify priority clinical practice guideline topics? BMC health services research. 2003;3(1):23. pmid:14687426
- 19. Jo H-S, Kim DI, Oh M-K. National Priority Setting of Clinical Practice Guidelines Development for Chronic Disease Management. J Korean Med Sci. 2015;30(12):1733–42. Epub 2015/11/30. pmid:26713047.
- 20. García LM, Pardo-Hernandez H, Superchi C, de Guzman EN, Ballesteros M, Roteta NI, et al. Methodological systematic review identifies major limitations in prioritization processes for updating. Journal of clinical epidemiology. 2017;86:11–24. pmid:28549931
- 21. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of internal medicine. 2018;169(7):467–73. pmid:30178033
- 22. Fadlallah R, El-Harakeh A, Bou-Karroum L, Lotfi T, El-Jardali F, Hishi L, et al. A common framework of steps and criteria for prioritizing topics for evidence syntheses: a systematic review. Journal of Clinical Epidemiology. 2019.
- 23. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27(8):985–91. Epub 2012/04/13. pmid:22528615.
- 24. El-Harakeh A, Morsi RZ, Fadlallah R, Bou-Karroum L, Lotfi T, Akl EA. Prioritization approaches in the development of health practice guidelines: a systematic review. BMC health services research. 2019;19(1):692. pmid:31615509
- 25. van der Veer SN, Van Biesen W, Bernaert P, Bolignano D, Brown EA, Covic A, et al. Priority topics for European multidisciplinary guidelines on the management of chronic kidney disease in older adults. International urology and nephrology. 2016;48(6):859–69. pmid:26984833
- 26. van der Veer SN, Haller MC, Pittens CA, Broerse J, Castledine C, Gallieni M, et al. Setting priorities for optimizing vascular access decision making–an international survey of patients and clinicians. PLoS one. 2015;10(7):e0128228. pmid:26151822
- 27. Van Der Sanden WJ, Mettes DG, Grol RP, Plasschaert AJ, Verdonschot EH. Development of clinical practice guidelines for dentists: methods for topic selection. Community dentistry and oral epidemiology. 2002;30(4):313–9. pmid:12147173
- 28. Nast A, Gaskins M, Eisert L, Werner R, Borradori L, Marinovic B, et al. Prioritizing topics in guideline development: results of a two‐phase online survey of dermatologist members of the EADV. Journal of the European Academy of Dermatology and Venereology. 2019;33(1):227–33. pmid:30317678
- 29. Loeffen E, Mulder R, Kremer L, Michiels E, Abbink F, Ball L, et al. Development of clinical practice guidelines for supportive care in childhood cancer—prioritization of topics using a Delphi approach. Supportive Care in Cancer. 2015;23(7):1987–95. pmid:25516211
- 30. Farrell B, Tsang C, Raman-Wilms L, Irving H, Conklin J, Pottie K. What are priorities for deprescribing for elderly patients? Capturing the voice of practitioners: a modified delphi process. PloS one. 2015;10(4):e0122246. pmid:25849568
- 31. Rosenfeld RM, Shiffman RN, Robertson P. Clinical practice guideline development manual: a quality-driven approach for translating evidence into action. Otolaryngology—Head and Neck Surgery. 2013;148(1_suppl):S1–S55.
- 32.
Institute of Medicine. Clinical practice guidelines we can trust.: In: Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, editors. The National Academies Press. Washington, DC; 2011.
- 33. Norris SL, Holmer HK, Burda BU, Ogden LA, Fu R. Conflict of interest policies for organizations producing a large number of clinical practice guidelines. PloS one. 2012;7(5):e37413. pmid:22629391
- 34. The L. Managing conflicts of interests in clinical guidelines. Lancet (London, England). 2019;394(10200):710.
- 35. McGregor S, Henderson KJ, Kaldor JM. How are health research priorities set in low and middle income countries? A systematic review of published reports. PloS one. 2014;9(10):e108787.
- 36. Majeed‐Ariss R, McPhee M, Bundy C, Griffiths C, Young H, Group PPSPS. Developing a protocol to identify and prioritize research questions for psoriasis: a James Lind Alliance Priority Setting Partnership. British Journal of Dermatology. 2018;178(6):1383–7. pmid:28906549
- 37. Tomlinson M, Chopra M, Hoosain N, Rudan I. A review of selected research priority setting processes at national level in low and middle income countries: towards fair and legitimate priority setting. Health Research Policy and Systems. 2011;9(1):19.
- 38. Kaplan W. Background Paper 3 Approaches to Priority Setting. Priority Medicines for Europe and the World; 2004.
- 39. Nasser M, Ueffing E, Welch V, Tugwell P. An equity lens can ensure an equity-oriented approach to agenda setting and priority setting of Cochrane Reviews. Journal of clinical epidemiology. 2013;66(5):511–21. pmid:23477991
- 40. Kastner M, Bhattacharyya O, Hayden L, Makarski J, Estey E, Durocher L, et al. Guideline uptake is influenced by six implementability domains for creating and communicating guidelines: a realist review. Journal of Clinical Epidemiology. 2015;68(5):498–509. pmid:25684154
- 41. Boivin A, Currie K, Fervers B, Gracia J, James M, Marshall C, et al. Patient and public involvement in clinical guidelines: international experiences and future perspectives. Quality and Safety in Health Care. 2010;19(5):e22-e. pmid:20427302
- 42. Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, et al. Patient and Stakeholder Engagement in the PCORI Pilot Projects: Description and Lessons Learned. J Gen Intern Med. 2016;31(1):13–21. pmid:26160480
- 43. Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane database of systematic reviews. 2006;(3).
- 44. Zhao G, Kennedy C, Mabaya G, Okrainec K, Kiran T. Patient engagement in the development of best practices for transitions from hospital to home: a scoping review. BMJ open. 2019;9(8):e029693. pmid:31383707
- 45. Woolf SH, Zimmerman E, Haley A, Krist AH. Authentic engagement of patients and communities can transform research, practice, and policy. Health Affairs. 2016;35(4):590–4. pmid:27044956
- 46. Armstrong MJ, Mullins CD, Gronseth GS, Gagliardi AR. Impact of patient involvement on clinical practice guideline development: a parallel group study. Implementation Science. 2018;13(1):55. pmid:29661195
- 47. Armstrong MJ, Bloom JA. Patient involvement in guidelines is poor five years after institute of medicine standards: review of guideline methodologies. Research Involvement and Engagement. 2017;3(1):19.
- 48. Rashid A, Thomas V, Shaw T, Leng G. Patient and public involvement in the development of healthcare guidance: an overview of current methods and future challenges. The Patient-Patient-Centered Outcomes Research. 2017;10(3):277–82. pmid:27830457
- 49. Madden M, Morley R. Exploring the challenge of health research priority setting in partnership: reflections on the methodology used by the James Lind Alliance Pressure Ulcer Priority Setting Partnership. Research involvement and engagement. 2016;2(1):12.
- 50. Manafo E, Petermann L, Vandall-Walker V, Mason-Lai P. Patient and public engagement in priority setting: A systematic rapid review of the literature. PLoS One. 2018;13(3):e0193579. Epub 2018/03/03. pmid:29499043; PubMed Central PMCID: PMC5834195.
- 51. Manafò E, Petermann L, Vandall-Walker V, Mason-Lai P. Patient and public engagement in priority setting: a systematic rapid review of the literature. PloS one. 2018;13(3):e0193579. pmid:29499043
- 52. Khodyakov D, Savitsky TD, Dalal S. Collaborative learning framework for online stakeholder engagement. Health Expectations. 2016;19(4):868–82. pmid:26295924
- 53. El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implementation science. 2012;7(1):2.
- 54. Yoshida S. Approaches, tools and methods used for setting priorities in health research in the 21st century. Journal of global health. 2016;6(1).
- 55. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. International journal of clinical pharmacy. 2016;38(3):655–62. pmid:26846316
- 56. Banno M, Tsujimoto Y, Kataoka Y. Reporting quality of the Delphi technique in reporting guidelines: a protocol for a systematic analysis of the EQUATOR Network Library. BMJ open. 2019;9(4):e024942. pmid:30948575
- 57. Bryant J, Sanson-Fisher R, Walsh J, Stewart J. Health research priority setting in selected high income countries: a narrative review of methods used and recommendations for future practice. Cost Effectiveness and Resource Allocation. 2014;12(1):23.
- 58. Viergever RF, Olifson S, Ghaffar A, Terry RF. A checklist for health research priority setting: nine common themes of good practice. Health research policy and systems. 2010;8(1):36.
- 59.
Higgins JP, Green S. Cochrane handbook for systematic reviews of interventions: John Wiley & Sons; 2011.