The implementation of prioritization exercises in the development and update of health practice guidelines: A scoping review

Background The development of trustworthy guidelines requires substantial investment of resources and time. This highlights the need to prioritize topics for guideline development and update. Objective To systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines. Methods We searched Medline and CINAHL electronic databases from inception to July 2019, supplemented by hand-searching Google Scholar and the reference lists of relevant studies. We included studies describing prioritization exercises that have been conducted during the de novo development, update or adaptation of guidelines addressing clinical, public health or health systems topics. Two reviewers worked independently and in duplicate to complete study selection and data extraction. We consolidated findings in a semi-quantitative and narrative way. Results Out of 33,339 identified citations, twelve studies met the eligibility criteria. All included studies focused on prioritizing topics; none on questions or outcomes. While three exercises focused on updating guidelines, nine were on de novo development. All included studies addressed clinical topics. We adopted a framework that categorizes prioritization into 11 steps clustered in three phases (pre-prioritization, prioritization and post-prioritization). Four studies covered more than half of the 11 prioritization steps across the three phases. The most frequently reported steps for generating initial list of topics were stakeholders’ input (n = 8) and literature review (n = 7). The application of criteria to determine research priorities was used in eight studies. We used and updated a common framework of 22 prioritization criteria, clustered in 6 domains. The most frequently reported criteria related to the health burden of disease (n = 9) and potential impact of the intervention on health outcomes (n = 5). All the studies involved health care providers in the prioritization exercises. Only one study involved patients. There was a variation in the number and type of the prioritization exercises’ outputs. Conclusions This review included 12 prioritization exercises that addressed different aspects of priority setting for guideline development and update that can guide the work of researchers, funders, and other stakeholders seeking to prioritize guideline topics.


Methods
We searched Medline and CINAHL electronic databases from inception to July 2019, supplemented by hand-searching Google Scholar and the reference lists of relevant studies. We included studies describing prioritization exercises that have been conducted during the de novo development, update or adaptation of guidelines addressing clinical, public health or health systems topics. Two reviewers worked independently and in duplicate to complete study selection and data extraction. We consolidated findings in a semi-quantitative and narrative way.

Results
Out of 33,339 identified citations, twelve studies met the eligibility criteria. All included studies focused on prioritizing topics; none on questions or outcomes. While three exercises focused on updating guidelines, nine were on de novo development. All included studies addressed clinical topics. We adopted a framework that categorizes prioritization into 11 steps clustered in three phases (pre-prioritization, prioritization and post-prioritization). Four studies covered more than half of the 11 prioritization steps across the three phases. The PLOS

Introduction
Health practice guidelines are "systematically developed statements" intended to optimize care at the clinical, public health and health systems levels [1]. The development of trustworthy guidelines requires substantial investment of resources [2,3] and time, often taking an average of two to three years [4,5]. In addition, and with the expansion of medical research and the emergence of new and innovative technologies, guidelines should be updated as necessary [6]. This highlights the need to prioritize topics for guideline development and update. In fact, the importance of prioritization for guideline development has been recognized by many guideline developing groups [7]. It ensures that limited resources are aligned with priority needs for guideline development [8]. Furthermore, prioritization exercises engaging a wide range of stakeholders enhance the relevance and potential uptake of priority topics by end users [9][10][11]. This represents an important step toward improving the delivery of evidenceinformed care.
The guideline development process includes, in addition to prioritizing topics, the prioritization of questions and outcomes [5,12]. Also, prioritization should be considered when adapting guidelines to select priority questions from among those addressed in the original guidelines [13]. Similarly, guideline developers need to prioritize which guidelines, guideline sections, or recommendations should be updated [14].
With the growing interest among researchers in prioritizing topics for the de novo development, update and adaptation of guidelines, several exercises have been conducted to yield explicit and transparent prioritization [15][16][17]. Some investigators relied primarily on the use of criteria to select priority guideline topics [18], while others have followed multicomponent prioritization processes and have used established tools and approaches [19].
While some efforts have been invested in synthesizing the evidence on prioritization for guideline updating [20], none have described prioritization for the de novo development, update or adaptation of guidelines. As such, the objective of this study was to systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines. standard methodology and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines for reporting scoping reviews [21] (see S1 File). This study is based on a detailed protocol available in S2 File. The project's team included a multidisciplinary group of professionals in the clinical, public health, and health policy and systems fields, with expertise in guideline development and priority setting.

Eligibility criteria
• Paper type: We included descriptive reports and excluded commentaries, editorials, letters, correspondences, news, and abstracts.
• Scope: We included papers about the de novo development, update or adaptation of health practice guidelines addressing clinical, public health or health system topics. Also, we excluded papers reporting proposed approaches, without any applied exercise.
• Focus: We included papers that aimed to prioritize one of the following: guideline topics, questions/recommendations, or outcomes. We excluded papers reporting on individual prioritization criteria or items.
• Setting: We included eligible papers irrespective of the setting (low-, middle-or high-income countries; primary, secondary or tertiary healthcare facilities).

Information sources and literature search
We searched Medline and CINAHL electronic databases from their respective inception date to July 2019. We also manually searched Google Scholar in July 2019. We developed the search strategy with the assistance of a medical librarian. The search included both Medical Subject Headings (MeSH) terms and free-text words and combined various terms for health prioritization (see S3 File). We did not use any language or date restrictions. We screened the reference lists of included and other relevant papers.

Study selection
The study selection process consisted of two phases: title and abstract screening and full text screening. Teams of two reviewers (AEH, TL, AA, RZM, RF, LBK) independently and in duplicate screened the titles and abstracts of citations captured by the search for potential eligibility. The reviewers then obtained the full texts of citations judged as potentially eligible by at least one of the two reviewers. Then, they screened the full texts in duplicate and independently for eligibility using a standardized and pilot-tested screening form and following a calibration exercise. At this phase, the reviewers resolved disagreements by discussion or with the input of a third reviewer (EAA) when consensus could not be reached.
• General characteristics: authors, lead entity, target audience, year of prioritization conduct, scope of prioritization, topic (specific domain), focus of prioritization, type of guideline development, and support/funding; • Steps of prioritization: • Pre-prioritization phase (development of guiding/ethical principles, generation of initial list of topics and collection of technical data to inform discussions); • Prioritization phase (use of established prioritization methods, research gap analysis, use of criteria, prioritization/ranking); • Post-prioritization phase (refinement of priorities into guideline topics, dissemination and implementation, revision mechanism, monitoring and evaluation).

Data synthesis
Due to the descriptive nature of data, we synthesized the findings in a semi-quantitative way. We used the extracted data to come up with common categorizations of relevant concepts (e.g., prioritization steps, generation of initial list of topics), using an iterative process of review and refinement. As part of this process, we analyzed the content of each study at least twice; when drafting the initial categories, and after producing an advanced draft. We reported the results narratively and in a tabular format. The concepts addressed in our analysis included: • Prioritization steps; we adopted 11 categories of prioritization steps, which we developed for a recent systematic review on prioritization for evidence synthesis [22]; • Generation of initial list of topics (descriptive analysis); • Output of the priority setting exercises (descriptive analysis); • Stakeholder involvement; we adopted the categories we developed for a recent systematic review on prioritization for evidence synthesis [22], which is based on the 7Ps framework [23]; • Prioritization criteria; we used a common framework of prioritization criteria that we developed for a recent systematic review on prioritization approaches in the development of health practice guidelines [24] (see S5 File).

Study selection
Fig 1 shows the study flow diagram summarizing the study selection process. Out of the 33,339 identified citations, twelve papers met our inclusion criteria [14][15][16][17][18][19][25][26][27][28][29][30]. We excluded 896 articles based on full text screening for the following reasons: not a paper type of interest (n = 49); not describing a reproducible prioritization exercise (n = 322); not about health practice guidelines (n = 525). Table 1 shows the general characteristics of the twelve included studies. One prioritization exercise was conducted in 1998 while the remaining ones were conducted between 2010 and 2017. Half of the prioritization exercises were implemented at a national level (n = 6) [14,16,19,27,29,30], while the rest were implemented at regional (n = 3) [25,26,28], provincial (n = 2) [15,18], or international levels (n = 1) [17]. All of the prioritization exercises focused on prioritizing guideline topics (as opposed to prioritizing questions or outcomes) and addressed clinical topics. While three exercises focused on updating guidelines [14,15,26], nine focused on de novo development [16-19, 25, 27-30], and none addressed adaptation of guidelines. Funding sources were mainly professional societies (n = 5) [16,25,26,28,29] or public funding (n = 4) [15,18,19,30]. We present below our findings summarized as the steps of prioritization, with a focus on two of those steps for which detailed information was available (namely the generation of initial list of topics and prioritization criteria). Then, we review stakeholder involvement in the prioritization exercises. Finally, we review the processes and outputs of the prioritization exercises stratified by whether the prioritization exercise was related to de novo development or updating.  Steps of prioritization Table 2 outlines the prioritization steps addressed in the twelve included studies across three phases: pre-prioritization, prioritization and post-prioritization phases. Although most of the studies (n = 11) addressed at least one step in the pre-prioritization phase [14-17, 19, 25-30], less than half (n = 5) addressed at least one step during the post-prioritization phase [14,15,17,25,27]. Four studies covered more than half of the 11 prioritization steps across the three phases [14,15,19,25]. Prior to conducting the prioritization exercises, most of the included studies (n = 10) generated initial lists of topics [15-17, 19, 25-30], while only a few studies reported on the development of ethical principles to guide the conduct of the exercise (n = 3) [25,26,30], or on the collection of technical data to inform further discussions (n = 2) [14,19].
Prioritization criteria. Table 4 presents the prioritization criteria that 10 out of the 12 studies reported on. Eight studies used their proposed criteria as part of their prioritization We did not include 'stakeholder engagement' as a separate step given that it was included across the three phases. The colored cells denote that stakeholders were engaged in the step. 2 Unclear whether participants were given an initial list and asked to suggest additional ones, or if it was based on suggestions only. 3 The exercise is based on former updating procedures of the Program in Evidence-based Care and procedures of other established guideline developing groups (e.g., Scottish Intercollegiate Guidelines Network and National Institute for Health and Care Excellence). 4 The pilot study followed a previously developed updating procedure (using information from a systematic review on the methods for updating clinical practice guidelines). https://doi.org/10.1371/journal.pone.0229249.t002 exercises [14,15,18,19,25,26,29,30], while two studies proposed criteria but did not use them in the exercise [16,28]. The studies included a mean of seven prioritization criteria (range: 3-13), with a total of 70 criteria reported. We attempted to match the 70 criteria to a published framework of 20 guideline prioritization criteria classified into six domains (Table 4 and S5 File). During the matching process, we added two criteria that emerged from the included studies (i.e., availability of low certainty evidence and acceptability). Table 4 shows the classification of the identified prioritization criteria according to the new framework. ✓ ✓ ✓ ✓ ✓ 1 The guideline topic was identified by the Neuro-oncology disease site group and was then confirmed through surveying practitioners. 2 The type of literature searched was not specified (i.e., reports outlining the prevalence and impact of polypharmacy). 3 Global Burden of Disease (GBD). All prioritization criteria have been worded in a way favoring prioritization (i.e., a favorable assessment of the criterion indicates higher priority). 2 Criteria were proposed but were not used in the prioritization exercise. https://doi.org/10.1371/journal.pone.0229249.t004 The most frequently reported criteria related to the health burden of disease (n = 9) [14-16, 18, 19, 25, 26, 28, 29] and potential impact of the intervention on health outcomes (n = 5) [16,25,26,29,30]. None of the studies included equity relevance of the disease or urgency as explicit criteria. Eleven (out of the total of 22 criteria listed in the framework) was the highest number of criteria reported by a study [16]. Table 5 shows the types of stakeholders involved in prioritizing guideline topics and the methods used to engage them. All included studies involved healthcare providers in the prioritization exercises, while four studies (33%) involved researchers [14,19,29,30], and only one study (8%) involved patients (Table 5) [26]. In Loeffen et al., the authors reported not including patients, parents, and caretakers as they planned to understand the needs of the professionals beforehand [29]. Seven studies described stakeholder recruitment methods which ranged from the use of professional networks (e.g., members directory), to searching databases and emailing clinicians [16-18, 25, 26, 28, 30].

Stakeholder involvement
All prioritization exercises surveyed stakeholders (e.g., Delphi approach) as a method of engagement. Other methods included the nominal group technique (n = 1) [25] and consensus conference (n = 1) [14]. Stakeholders were engaged via an online platform (e.g., online surveys, email discussions) in all included studies, with two studies using both online and in-person meetings [14,25]. The frequency of engagement varied from only once (n = 4), to twice (n = 6), or three times (n = 2). Table 6 describes the processes and outputs of the prioritization exercises. The nine studies that implemented prioritization processes for the de novo development of guidelines followed common steps of reviewing the literature (n = 5) [17,19,25,27,30] and/or engaging stakeholders (n = 9) [16-19, 25, 27-30], while considering the availability of existing guidelines on the suggested topics (n = 3) [16,19,28]. In fact, one study conducted the prioritization exercise regardless of existing guidelines which resulted in prioritizing 20 topics; all of which were covered by existing guidelines [16].

Prioritization processes and outputs for updating (n = 3)
Studies that implemented prioritization exercises for updating (n = 3) either assessed candidate guideline documents for updating [15] or selected a specific guideline a priori and assessed potential topics or sections to be covered by updating [14,26].
Agbassi et al. used a stepwise process in which two questionnaires were implemented to prioritize guidelines for updating and to assess the effect of new evidence on existing recommendations [15]. van (Table 6) [14]. Two studies used categories (e.g., urgent, high,   Following the initiation of a review, review outcomes (endorsement, archive, and update) should be approved by a larger expert panel comprising a multidisciplinary team of clinicians and other stakeholders. https://doi.org/10.1371/journal.pone.0229249.t005 Table 6. Prioritization processes and outputs of the conducted prioritization exercises.

Output (final list of priorities)
De novo development of guidelines Brouwers, 2003 [18] One suggested topic ! One topic was identified by the Neuro-oncology disease site group (DSG) ! Members of the DSG conducted a survey of Ontario clinicians to confirm topic selection and determine their support for the topic ! Survey results showed variation in current practice and that practitioners feel that a guideline is necessary ! One topic was confirmed for guideline development One confirmed topic Borgonjen, 2015 [16] 157 topics and 10 criteria ! 157 dermatological topics were selected and ranked by 118 dermatologists as priority topics regardless of existing guidelines via a survey ! 10 criteria were included for ranking by dermatologists who were asked to add any missing criterion ! 20 topics were prioritized based on pooled scores; all of which were covered by existing guidelines; with an overlap between topics 12-20 and a further set of 15 topics and a significant difference in raking between top 5 and other 30 topics. ! 8 criteria were prioritized, and 3 additional criteria were suggested • excluding diseases for which guidelines are being developed by KAMS ! Data on burden of disease and a list of available guidelines were provided to the experts for consideration in prioritization ! Analytic hierarchy process and subjective assessment were used for ranking ! Top 20 diseases were derived from each assessment Two ranked lists of 20 chronic diseases Kerr, 2009 [17] 30 clinical areas ! An initial list of 30 clinical areas was derived based on the below steps: • experts were sent a CD-ROM with detailed evidence from Cochrane review (abstracts from review and all interventional trials identified) • each expert was invited to submit 10 areas where "it was felt new clinical guidance statements were necessary" ! Participants were then asked to rank the top 10 areas in another mailing ! A large difference was observed in ranking score after area ranked 11 so this was the cut-off point

Output (final list of priorities)
Nast, 2019 [28] 265 topics under disease categories ! 265 topics were suggested by participants of round 1 of the survey ! Suggestions were combined into 35 broader topics to be ranked by participants ! Suggestions well covered by existing guidelines or not specific to decide whether they are covered by existing guidelines were excluded ! Respondents were asked to rank their top 10 topics in round 2 of the survey ! Top 10 topics were generated according to total weighted points Ranked list of 10 topics van der Sanden, 2002 [27] 1027 topics ! 1027 topics were obtained from 3 methods (survey, peer group, and literature) as follow: • Survey: dentists confirming important of guideline development proposed a maximum of 5 topics with justification • Analysis of peer group (8-10 dental practitioners) discussions over 1989-1998 • All national dental journals and periodicals (n = 8) were screened over 1992-1997 ! 4 th method: linear regression lines of topics obtained from literature ! Two researchers collected data and reclassified topics into 9 groups; topics reported less than 6 times within a method were excluded ! An overall rank was obtained by adding rank numbers assigned to the 4 methods for each topic; lowest value indicated highest priority ! The reliability of a method was tested by the ''item-rest sum correlation"; a correlation of rank positions of one method with the sum of rank positions obtained by the other 3 methods ! 34 topics were prioritized as a result of ranking and reclassification Ranked list of 34 topics belonging to 9 topic groups van der Veer, 2016 [25] 48 topics in 6 categories ! 48 topics in 6 categories were generated from a scoping literature review (813 titles) and views of an international expert panel ! List was refined by the panel into 47 topics in 7 categories via an open online consultation of professionals (asked to select 50% of topics within each category as priority topics and add suggestions) ! Expert consensus meeting with 2-round voting yielded a ranked list of the 46 topics based on overall median and range. One topic was not included in the rating process due to an administrative error

Updating of guidelines
Agbassi, 2014 [15] 151 PEBC guideline documents ! 151 PEBC guideline documents were assessed in consultation with a clinical expert and facilitated by a methodologist: 37 archived, 33 deferred, 6 special cases and 75 need review ! Documents within review category underwent prioritization into low-(n = 20), medium-(n = 10), high-(n = 18) and urgent-priority (n = 27) ! In order of priority, clinical expert and methodologist reviewed guidelines (via conducting a streamlined systematic review) to determine effect of new evidence on existing recommendation and further action ! Then documents were classified as either endorsed (n = 15), updated (n = 8) or archived (n = 7), while others either required a new version (n = 1), had a review initiated (n = 7) or the process was incomplete (n = 35) ! Review outcomes (endorsement, archive, and update) should be approved by a larger expert panel comprising a multidisciplinary team of clinicians and other stakeholders Categorization of guideline documents into endorse (n = 15), update (n = 8) or archive (n = 7) Becker, 2018 [14] 35 guideline sections ! Limited search yielded 902 abstracts of potentially relevant evidence on the 35 guideline sections ! Further new evidence was identified via an online survey of CGP group members who also rated the sections based on evidence and clinical relevance ! Sections were subdivided in groups with high (15), medium (9), or low (11) need for update based on median scores ! A consensus conference was held to finalize the list of sections with "high" need for update; 7 sections were allocated from low and middle to high need for update and 2 new sections were suggested based on median voting. However, these additions were not presented in the paper.
Ranked list of 15 guideline sections (Continued ) medium, or low) to reflect the relative need for updating [14,15]. One study reported a median of 167 days for the time taken to implement the prioritization process (range 18-358 days) [15].
In terms of outputs, two studies provided ranked lists of priorities [14,26]. One study prioritized 8 out of 151 guideline documents for updating [15], another study prioritized 15 out of 35 guideline sections [14]. The third study generated a list of 42 topics from an initial list of 39 topics to be covered by the updated guideline [26].

Summary of findings
We systematically reviewed the literature for prioritization exercises that have been conducted for the de novo development, update or adaptation of health practice guidelines. We identified twelve eligible studies that focused on prioritizing clinical topics and were predominantly conducted for the de novo development of guidelines; none addressed adaptation. The priority setting exercises consisted of several steps that we grouped in three phases: pre-prioritization, prioritization and post-prioritization. The two most commonly used steps were the generation of an initial list of topics (mostly by seeking input from stakeholders or by reviewing the literature) and ranking of priorities. The two least used steps were research gap analysis and having a revision mechanism. Most of the included studies used prioritization criteria as part of the exercises, with the most common criteria being the health burden and potential impact of the intervention on health outcomes. All studies involved stakeholders, particularly healthcare providers, in prioritizing guideline topics. Stakeholders were mainly involved in the generation of initial list of topics, use of criteria and ranking of priorities.

Interpretation of findings
We observed that the generated priority topics were generally broad and non-specific. This might have been due to the fact that the vast majority of the exercises did not describe a step of refinement of the priority topics. It is essential to refine the topics in a way that would enable an easy transition from topics into meaningful questions appropriate for guideline development [31].
In addition, one of the Institute of Medicine (IOM) standards in guideline development is establishing transparency [32]. The IOM emphasizes the need for detailed and publicly accessible guideline development processes including methods for priority setting [32,33]. Such processes would increase the credibility and the potential uptake of the end results [34]. We found only three studies reporting on the development of ethical principles to guide the conduct of the exercises. Prioritization should be supported by an effective dissemination plan to ensure that generated priorities inform prospective research and ultimately improve health [35]. The dissemination of priorities to researchers and funders helps in directing research agendas to guideline topics that are most important to stakeholders [36]. Although the number of prioritization exercises has been increasing over time, very few exercises reported on dissemination or implementation strategies. Consistent with findings of earlier reviews on prioritization for health research [35,37], only three of the included exercises mentioned dissemination. Furthermore, our reliance on a recently developed framework of prioritization criteria allowed us to categorize 66 out of the 70 criteria in the included studies. The most commonly used criterion was 'health burden'. The majority of the remaining criteria were used by two or less exercises. For instance, although equity is one of the most frequently reported criteria in the priority setting literature [38], none of the exercises considered the equity relevance of the condition. An equity-oriented approach to priority setting is important for ensuring inclusiveness [38,39]. While this could reflect a decision by the designers of the exercises to focus on few but relevant criteria, it could also point to the failure of these exercises to be comprehensive in their use of criteria. Indeed, Nast et al. highlighted the need for such exercises to address a wide range of explicit criteria that extend beyond disease-related factors [28].
Overall, the observed variation in the prioritization steps and criteria used in the included prioritization exercises could potentially be explained by the need to tailor the decision on how to conduct a prioritization exercise to the needs of relevant stakeholders and to the available resources, such as time and funding.
A recent systematic review highlighted the opportunity to engage diverse types of stakeholders in prioritizing guideline topics [24]. Incorporating views of various stakeholders in guideline development can potentially reduce a biased selection of topics by few groups and increase transparency [7,31]. Moreover, considering the needs of different stakeholders may improve the uptake and usability of guidelines [40]. While all exercises included in this review involved health care providers, only four and one respectively engaged researchers and patients. None engaged the other eight types of stakeholders that we assessed. Patient involvement in priority setting for guideline development has been widely supported in the literature [41][42][43][44]. It helps direct guidelines toward questions that matter most to patients, expanding beyond the interests of researchers and clinicians [45][46][47]. However, potential barriers to patient involvement include limited resources (e.g., lack of funds and stakeholder time), slowed down and longer process, and difficulty in identifying appropriate representatives [42,43,48]. In addition, guidance on how to engage patients is limited [43]. Despite potential challenges, some of the available methods for engaging patients have been evaluated, and thus can be used to ensure appropriate patient involvement [49,50]. Moreover, maintaining regular communication with patients or their representatives facilitates meaningful engagement [51].
The online approach to engaging stakeholders was adopted by all studies. Online platforms are considered practical and cost-efficient ways of engaging stakeholders [52]. Other methods that were not frequently used in the prioritization exercises include in-person meetings and workshops. Although not widely used (for practical and financial reasons), those methods might improve interactions and discussions between stakeholders and in turn generate different priorities. In addition, face-to-face meetings are one of the knowledge exchange methods with the greatest impact on policymaking [53]. Furthermore, most stakeholders were engaged through the Delphi survey method, which is a simple consensus tool for obtaining the views of a large group of relevant stakeholders [54] using structured and iterative group interactions [55]. The Delphi method is commonly used in both guideline development and in health research prioritization [56][57][58], explaining its use in prioritization for guideline development.
The included studies on prioritization for updating conducted the exercises at different time points of the updating process. One exercise was implemented to identify the clinical guidelines in greatest need of update after a surveillance process, while the other two exercises aimed to identify the topics or sections in greatest need of update for a selected guideline.

Strengths and limitations
We used a rigorous and transparent process including a comprehensive search strategy, duplicate and independent selection, and duplicate and independent data extraction [59]. In addition, and by drawing on an extensive body of literature since the 1990s and up to July 2019, this review synthesizes almost three decades of published research on prioritization for guideline development. On the other hand, we built on two recent systematic reviews of prioritization approaches to develop our data extraction and analysis framework (e.g., how to categorize the steps of prioritization, prioritization criteria).
There are limitations to our scoping review process. First, we did not appraise the quality of the included studies. However, this is consistent with the scoping review methodology [21] and no tool has been developed for the critical appraisal of priority setting exercises. Second, we did not search the grey literature, particularly websites of guideline developing organizations, due to time and resource constraints.

Comparison to other reviews
Our work adds to former reviews on the topic, e.g., the review by Garcia et al. which focused on the update of health decision-making tools, one of which was guidelines [20]. Consistent with our findings, Garcia et al. reported variability in the methods used to implement the prioritization exercises for updating. On the other hand, our study presents a more in-depth analysis of relevant characteristics such as the steps and criteria for prioritization exercises. Our list of criteria is consistent with, but a bit more comprehensive than the list by Garcia et al.

Implications for practice
Our findings can assist clinicians, researchers, funders, policymakers, and other stakeholders seeking to develop health practice guidelines in prioritizing topics to be addressed. Given that there are no standard prioritization best practices for guideline development [7], it might be challenging to provide specific guidance on which prioritization exercise to use. However, the decision on whether and how to conduct a prioritization exercise should be tailored to the needs of relevant stakeholders and to the available resources, including time and funding. Furthermore, the detailed lists of identified steps and criteria can serve as a menu of options for guideline developers to select from, as judged appropriate to the context, and through a transparent decision-making process.

Implications for future research
There is a need to develop methods and guidance for prioritization of not only topics, but also for questions and outcomes in guidelines projects. Exploring the same question of this study through the analysis of guideline handbooks would be helpful for that purpose. Further rigorous evaluation research can help with a better understanding of potential facilitators and barriers to prioritization. Moreover, and because all of the included conducted exercises were developed by researchers from high-income countries, future studies can focus on the effectiveness of the exercises in low-and middle-income countries. It is also essential to evaluate the impact of those exercises on resource allocation and on clinical outcomes.