Reporting Items for Updated Clinical Guidelines: Checklist for the Reporting of Updated Guidelines (CheckUp)

Background Scientific knowledge is in constant development. Consequently, regular review to assure the trustworthiness of clinical guidelines is required. However, there is still a lack of preferred reporting items of the updating process in updated clinical guidelines. The present article describes the development process of the Checklist for the Reporting of Updated Guidelines (CheckUp). Methods and Findings We developed an initial list of items based on an overview of research evidence on clinical guideline updating, the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument, and the advice of the CheckUp panel (n = 33 professionals). A multistep process was used to refine this list, including an assessment of ten existing updated clinical guidelines, interviews with key informants (response rate: 54.2%; 13/24), a three-round Delphi consensus survey with the CheckUp panel (33 participants), and an external review with clinical guideline methodologists (response rate: 90%; 53/59) and users (response rate: 55.6%; 10/18). CheckUp includes 16 items that address (1) the presentation of an updated guideline, (2) editorial independence, and (3) the methodology of the updating process. In this article, we present the methodology to develop CheckUp and include as a supplementary file an explanation and elaboration document. Conclusions CheckUp can be used to evaluate the completeness of reporting in updated guidelines and as a tool to inform guideline developers about reporting requirements. Editors may request its completion from guideline authors when submitting updated guidelines for publication. Adherence to CheckUp will likely enhance the comprehensiveness and transparency of clinical guideline updating for the benefit of patients and the public, health care professionals, and other relevant stakeholders.


Background
Trustworthy clinical guidelines aim to assist decision making by providing recommendations that are informed by the best available evidence and include an assessment of the benefits and harms of alternative care options [1,2]. Because of the continuous emergence of new research evidence (i.e., changes in available interventions, effects, or cost) [3], appropriate updating to maintain the trustworthiness of clinical guidelines is challenging since it requires regular surveillance and reviewing of the new evidence [4,5].
Trustworthiness standards for guidelines have been published by both the Institute of Medicine (IOM) and the Guidelines International Network (G-I-N) [1,2]. Additionally, instruments are available for assessing the quality of clinical guidelines, such as the Appraisal of Guidelines for Research and Evaluation (AGREE) II Instrument [19], while others, such as the GIN-McMaster Guideline Development Checklist [20], support developing and implementing trustworthy clinical guidelines. However, guideline updating requires some different methodological considerations and unique communication procedures. Currently, none of the existing tools address these issues.
To address this gap, in a partnership of the Iberoamerican Cochrane Centre (www. cochrane.org), the AGREE Collaboration (www.agreetrust.org), and the G-I-N Updating Guidelines Working Group (www.g-i-n.net/working-groups/updating-guidelines), we have developed the Checklist for the Reporting of Updated Guidelines (CheckUp). This article about CheckUp is targeted at guideline developers and users of guidelines. In the article, we present the methodology of the development process and the final checklist. In a supplementary file, we present explanations and examples for each item (S1 Appendix).

Methodology
For reporting the development process of CheckUp, we followed Enhancing the QUAlity and Transparency Of health Research(EQUATOR) and Moher's criteria [21,22]. The development of CheckUp consisted of four phases: (1) panel selection, (2) generation of the initial checklist, (3) optimisation of the checklist, and (4) approval of the final checklist (Fig 1).

Panel Selection
To advise on the development of the CheckUp, a panel comprising individuals with relevant experience in clinical guideline development and updating and/or in systematic reviews/guidelines research methodology was convened. Invited panel participants were identified based on a review of the main authors in the field, as well as the AGREE Trust (www.agreetrust.org) and the G-I-N (www.g-i-n.net) members. The purpose of the panel was to provide expert advice during the development process and to participate in the Delphi survey. A core group (RWMV, PAC, MB, and LMG) was established to design the protocol and provide more timesensitive and operational advice.

Generation of the Initial Checklist
The core group first developed an initial list of items-including explanation and examplesthrough brainstorming and discussion, taking into account (1) research evidence in the field [15,16,18], (2) the AGREE II Instrument [19], and (3) the panel experience. We used three core updating publications as a starting point from which the initial version of the checklist was generated [15,16,18]. These studies include an overview of the available guidance from guideline methodological handbooks [18], a systematic review of the published methodological research [16], and an international survey about the experiences of the main guideline developers [15].

Optimisation of the Checklist
We optimised the initial checklist through a multistep process that included an assessment of existing updated clinical guidelines, semistructured interviews, a Delphi consensus survey, and an external review with clinical guideline methodologists and users (Fig 1).
Assessment of updated clinical guidelines. We piloted the initial checklist among a convenience sample of updated clinical guidelines to assess the used terminology, to identify missing items, and as a first step to explore its face validity. We included updated clinical guidelines that were (1) developed by G-I-N members, (2) published in English or Spanish, and (3) published between 2011 and 2013. We searched the G-I-N library (www.g-i-n.net) and the National Guideline Clearinghouse (www.guidelines.gov). Two reviewers (RWMV and SPS) applied the checklist, solving disagreements by consensus. The core group discussed the results and refined the initial list of items.
Semistructured interviews. To refine the checklist and to identify missing items, we conducted semistructured interviews with clinical guideline experts. We chose a convenience sample of participants, outside the CheckUp panel, with (1) experience in updating clinical guidelines, defined as having participated in the updating process of at least one clinical guideline over the past year, and (2) fluency in English. We identified the participants by contacting professionals associated with G-I-N or researchers in the field. When someone did not respond or could not participate, a new person was recruited. We continued to recruit participants and collect data until data saturation was achieved.
In each interview, participants were asked about their experiences and challenges in updating clinical guidelines. Subsequently, the participant was prompted by the interviewer (RWMV) to reflect on the strengths, weaknesses, missing concepts, and redundancies in the checklist. The interviews were audiotaped, and key themes were identified. The core group discussed the results and refined the list of items.
Delphi consensus survey. We reviewed the refined list of items through a Delphi consensus survey [23], with all members of the CheckUp panel. The Delphi participants assessed the inclusion, comprehensiveness, clarity, and coverage of each item and tried to identify potentially additional items for the checklist.
Using a seven-point Likert scale (one meaning strongly disagree and seven meaning strongly agree) [24], we asked participants to rate whether the item should be included in the checklist. For each item, participants were asked whether their perceptions of (1) the completeness, (2) the usability, and (3) the quality of a clinical guideline would be influenced if the item was reported. We included a free text box for suggestions to modify the items, the explanation, or the examples. We used online software to design the survey and to collect the responses (www.surveymonkey.com).
We calculated the median score for inclusion, completeness, usability, and quality for each item and classified them into (1) items with a median score of 0 to 3 points, which were excluded; (2) items with a median score of 4 to 5 points or with substantial comments that needed important revision, which were retained, modified, and further tested; and (3) items with a median score of 6 to 7 points and without substantial comments, which were included and not evaluated further in the following rounds.
One reviewer (RWMV) analysed the quantitative and qualitative results and suggested potential solutions. The core group discussed the results and potential solutions and refined the list of items accordingly. We continued with additional rounds until consensus for inclusion or exclusion was reached and no more relevant comments were provided.
External review with clinical guideline methodologists. To evaluate the usability of the checklist, we conducted a survey with clinical guideline methodologists who had experience in updating clinical guidelines, as measured by having participated in the updating process of at least one clinical guideline over the past year. We also invited all of the G-I-N institutional member contacts to participate in the external review. If the contact person was not able to participate, we asked them to provide contact details of another expert working at the same institution.
Using a seven-point scale (one meaning strongly disagree and seven meaning strongly agree), we asked participants to rate the usability of each item and its influence on the confidence in an updated clinical guideline if the item was reported. A free text option was included for suggestions to modify the items, the explanation, or the examples. We used online software to design the survey and to collect the responses (www.surveymonkey.com).
We calculated the median score for usability and confidence for each item. One reviewer (RWMV) analysed the quantitative and qualitative results and suggested potential solutions. The core group discussed the results and potential solutions and refined the list of items accordingly.
External review with clinical guideline users. We conducted semistructured interviews with clinical guideline users to evaluate the usability of the checklist. We engaged individuals who were (1) health care professionals who used clinical guidelines in clinical practice and (2) located in Canada, Spain, or the Netherlands. We identified the participants with the help of the panel members. When someone did not respond or could not participate, a new person was recruited. We continued to recruit participants and collect data until the information was repeated and no new information emerged (data saturation).
For each interview, participants were asked whether reporting of the item in an updated clinical guideline would increase their confidence in the guideline. The participant and interviewer (RWMV) reviewed the checklist, and the participants were prompted to consider missing concepts, redundancy, and the usability of the checklist. The interviews were audiotaped, and key themes were identified. The core group discussed the results and refined the list of items.
Approval of the final checklist. The checklist was presented and discussed in a workshop at the 2015 G-I-N Conference in Amsterdam. In this workshop, we asked the participants whether the checklist was deemed adequate for assessment of the updated clinical guidelines [25]. We also asked the participants to give an overall impression of the checklist. The core group discussed the results and agreed on the final list of items.

Generation of the Initial Checklist
The initial checklist included 13 items within the following domains: updating rationale, scope and purpose of the updated clinical guideline, participants in the updating panel, conflicts of interest, updating methodology, differentiating original and new information, and reasons for the changes in the recommendations.

Optimisation of the Checklist
Assessment of updated clinical guidelines. Initially, we assessed a convenience sample of ten updated clinical guidelines from the G-I-N library with the initial checklist [26][27][28][29][30][31][32][33][34][35]. The items more frequently reported in the included clinical guidelines were related to the literature search strategy (60%), the composition of the panel (50%), and the external review (40%). The other checklist items (e.g., assessment for the need of updating, evidence selection, rationale for updating, and rationale for changes) were reported in less than 20% of the included clinical guidelines. No additional concepts or items emerged (Table 1).
Semistructured interviews. We conducted semistructured interviews with clinical guideline developers (5 from Europe, 5 from North America, and 3 from South America). In total, we interviewed 13 participants, at which point saturation was reached. As a result, we modified five items and added four new ones. The modifications were related to (a) differences in the objectives, purpose, or aim between the original and updated version; (b) the identification of new evidence; (c) the rationale for changing recommendations; and (d) the funding. The new items were related to (a) the scope of the update (partial or complete), (b) the target audience, (c) the changes in the recommendations, and (d) the plans and methodology reported to update the clinical guideline (Table 1).
Delphi consensus survey. All the members of the CheckUp panel (n = 33) were invited to participate in the Delphi consensus survey. Twenty-seven (82%) members participated in the first Delphi round, thirty-one members (93.9%) in the second Delphi round, and all (100%) members in the third and final round.
In the first round, the participants provided substantial feedback on various items, their explanation, and the accompanying examples. This feedback triggered modification in the order of the items, phrasing of the items, explanations, and examples (Table 1). All items met the inclusion criteria, and no major comments were reported. The median score for whether the participants believed that an updated clinical guideline would be more complete, usable, and of higher quality whenever the item was reported was six for all questions. Table 1. CheckUp: Stages of the optimisation process (objective, sample, and results by optimisation processes).

Stage
Objectives Sample (n) Main Results

Assessment of updated clinical guidelines
• Assess whether the terminology was consistent between the checklist and updated clinical guidelines • Identify items that were lacking in the checklist.
• No items were modified.
• No new items were added.

Semistructured interviews
• Explore challenges and issues regarding the clinical guideline updating process. • Identify items that were lacking in the checklist.
• Five items had major modifications.
• Four new items were added.

Delphi consensus survey
• Assess the inclusion, comprehensiveness, clarity, and coverage of each item. • Identify items that were lacking in the checklist.
• Clinical guideline updating process experts (33) (CheckUp panel). In the second round (n = 17 items), the amount of feedback was substantially smaller than in the first round (Table 1). We merged some items, and the checklist was reduced to 16 items. Again, all items met the inclusion criteria, and no major comments were reported. The median score for whether the participants believed the clinical guideline would be more complete, usable, and of higher quality was 7.0, 6.0, and 6.0, respectively.
In the third and last round, a general consensus was reached for all items, explanations, and examples. The median score for item inclusion, completeness, usability, and quality was !6 in all items (except for two items with a median score of 5.5 in the usability and quality question) ( Table 2).
External review. External review with clinical guideline methodologists. We conducted a survey with 53 clinical guideline methodologists (53/59, response rate 90%). The median scores of usability and confidence for each item were !6 ( Table 3). Participants provided comments that improved the writing style of the items, explanations, and examples (Table 1).
External review with clinical guideline users. We had conducted semistructured interviews with 10 clinical guideline users (3 from Spain, 2 from the Netherlands, and 5 from Canada) when saturation was reached. All participants acknowledged that all items were useful to evaluate the reporting of the updating process in updated clinical guidelines. Neither new items nor modifications were proposed (Table 1). Whether this information was present or not would influence your perceptions of the completeness of the reporting of an updated clinical guideline; c Whether this information was present or not would influence your perceptions of the usability of an updated clinical guideline;

Final Checklist
The checklist includes 16 items that can be broadly categorised into three themes: (1) presentation (e.g., clinical guideline sections and recommendations), (2) editorial independence (e.g., the working group and funding), and (3) the methodology used (e.g., search strategy and evidence synthesis) (Table 4). Those attending the presentation of the checklist workshop at the G-I-N 2015 conference reviewed and agreed with the final version of the checklist.

Discussion
We developed CheckUp through a comprehensive development process, including the use of systematic reviews, assessment of updated clinical guidelines, and engagement of the international guideline community through semistructured interviews, a Delphi consensus survey, and an external review.

Main Findings
Across the different processes, an alignment and consensus of opinion emerged between what was documented in the literature and the expectations of clinical guideline developers, users, and researchers in regards to what information ought to be reported in an updated clinical guideline. CheckUp includes 16 items regarding the presentation of the updated clinical guideline, editorial independence, and the methodology used in the clinical guideline updating process. CheckUp was primarily developed to evaluate the completeness of reporting in updated guidelines. Additionally, the tool can inform guideline developers about strategies for updating clinical guidelines and their reporting requirements. An explanation and elaboration article for the CheckUp is published as a supporting information article (S1 Appendix). CheckUp can be used in several ways. The checklist can provide guidance to developers who update clinical guidelines, by providing methodological principles that should be incorporated into the updating process, as well as strategies for reporting this information. The checklist can be applied by users or appraisers of clinical guidelines to assess whether updated clinical guidelines align with the CheckUp items. We suggest that a minimum of two reviewers assess the reporting of the guideline updating process independently, with the help of a third reviewer if there is a need of reaching consensus.

Our Results in the Context of Previous Research
Updating is a crucial part of maintaining the trustworthiness of clinical guidelines [1,2]. Since clinical guidelines have a limited lifespan, updating clinical guidelines is crucial to maintain the validity of the recommendations [3,9,36,37]. Although the importance of regular updating has been recognised and clinical guidelines may have an "expiration date," little research has been conducted in the field so far [13,[15][16][17][18]. Published standards for trustworthy guidelines require the description of updating plans [1,2]; however, these standards do not provide specific guidance about the detailed reporting of the updating process of guidelines.

Strengths and Limitations
Our CheckUp proposal has several strengths. For the development process, we systematically reviewed the evidence and followed EQUATOR and Moher's criteria [21,22]. Also, by applying a formal consensus method (Delphi survey) and collecting experts' opinions (semistructured interviews and external reviews), we reached a fair understanding of clinical guideline methodologists' and users' perceptions about the updating of clinical guidelines. Finally, there was fairly strong overall consensus during the development of CheckUp.
Our study has some limitations. We used consensus methods and convenience samples of clinical guideline stakeholders. However, across the different processes, an alignment and consensus of opinion emerged on what clinical guideline developers, users, and researchers expect to see reported in updated clinical guidelines. Another potential limitation is that CheckUp includes some items that may partially overlap with some items that are present in other instruments [19,20]; however, we think this is a minor limitation as CheckUp differs for the most part and has a very specific and differentiated goal. Finally, we did not collect potential conflicts of interest in our panel.

Implications for Practice and Research
CheckUp can be used for multiple purposes. Firstly, guideline developers can use it both for the reporting of their guidelines and to plan their updating processes. Guideline users can assess the reporting of updated guidelines. Editors may request its completion from guideline authors. CheckUp provides an overall picture of how complete the updating process is reported in updated clinical guidelines. Being a reporting checklist, CheckUp does not evaluate the quality of the updating processes, as there are no gold standards for this process. Currently, the G-I-N Updating Guidelines Working Group (http://www.g-i-n.net/working-groups/ updating-guidelines) is undertaking an analysis of current guideline updating methods worldwide. From this work, strategies or advice might come on how we might assess guideline updating quality.
There are currently no gold standards for guideline updating methodology. Nonetheless, updating is key to ensuring trustworthy, implementable, and clinically relevant recommendations. Current guideline evaluation tools or guideline method resources (e.g., AGREE II, Grading of Recommendations Assessment, Development, and Evaluation (GRADE), IOM Standards, and the like) are not simply transferable to the conceptual requirements of an updated guideline. CheckUp addresses the gap: it has been supported by our study participants and is a resource that complements (rather than competes with) the other high-quality tools available in the guideline enterprise.
Further rigorous research in updating clinical guidelines is warranted, and we invite users to comment on the items and the usability of CheckUp. It would be important to assess the impact of CheckUp in the updating clinical guideline field over the next few years [16]. When dynamic or living guidelines become a reality, [38] some adaptation of CheckUp could potentially be necessary. Finally, the G-I-N Updating Guidelines Working Group will continue to play a key role in this work and in moving forward the updating agenda in the clinical guideline enterprise.