Comparing the usability of the World Health Organization’s conventional tuberculosis guidelines to the eTB recommendations map: A two-arm superiority randomised controlled trial

Best practices for the dissemination of global health guidelines has not undergone rigorous research. We used a new approach to digitizing World Health Organization (WHO) global tuberculosis guideline recommendations (eTB RecMap) and compared its usability to the conventional method of accessing TB recommendations using the WHO website. We conducted a two-arm superiority randomised controlled trial using a survey among global stakeholders who were past or planned future users of TB guidelines, recommendations, or policy advice. We assigned participants randomly (1:1) to complete an activity using the WHO eTB RecMap or the conventional website. The primary outcome was the accessibility of information and secondary outcomes understanding, satisfaction, and preference for one of the two formats. Between February 26 and August 29, 2021, we received 478 responses from stakeholders, of whom 244 (122 per group) were eligible and provided analysable results. Participants rated the eTB RecMap as more accessible, on average, when compared to the conventional website (on a seven-point scale, the mean difference {MD} was 0.9; 95% confidence interval {CI}: 0.6, 1.2; p < 0.001) and were more likely to correctly answer understanding questions. This is the first trial comparing digitized dissemination formats of health guideline recommendations. Stakeholders rated the WHO eTB RecMap as more accessible than the conventional WHO website for the tested recommendations. They also understood presented information better. The findings support better usability of TB information through the eTB RecMap and contribute to the effort to end the TB epidemic. Trial registration: This trial was registered with ClinicalTrials.gov (NCT04745897) on February 9, 2021.


Background
Tuberculosis (TB) is the leading cause of death from a single infectious agent worldwide, with an estimated 10 million new cases in 2019. 1 In a concerted effort to end the TB epidemic, the WHO Global TB Programme (WHO-GTB) has issued guidelines with recommendations on TB prevention, diagnosis, treatment, and care. 2,3 Since 2009, these guidelines have been developed using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) method 3 , which is a transparent, evidence-based framework for the assessment of the certainty in a body of evidence and recommendation development. 4 Specifically, GRADE assists guideline developers in question prioritization, certainty assessments, balancing benefits and risks, and considering, among other criteria, cost, equity, acceptability, and feasibility in context. 4,5 WHO TB recommendations are located in many discrete publications on the WHO website, including standard, consolidated, interim, and emergency guidelines. Given the large number and variety of publications, ways to enhance their connectivity and accessibility should be explored like for guidelines in other fields. A mixed-methods study by Hajizadeh, et al. 6 suggested that stakeholders, including members of the WHO-GTB guideline development group (GDG), desire direct access to WHO TB recommendations and supplementary information, such as evidence to decision (EtD) tables and evidence profiles. 6 In an effort to improve accessibility and use of WHO TB recommendations, we developed a WHO eTB recommendation map (RecMap) in a collaboration between the WHO Collaborating Center for Infectious Diseases, Research Methods and Recommendations, the Michael G DeGroote Cochrane Canada Centre and the WHO-GTB. 6,7 This RecMap identifies, lists, and maps WHO TB recommendations using recommendation mapping methodology. This methodology visually organizes recommendations, thus allowing the identification of clusters and gaps. 6 Furthermore, it is anticipated to facilitate the adoption, adaptation, or de novo development of recommendations in a variety of countries and settings through an integrated linkage and access to original content on GRADEpro. 8 In order to compare the accessibility of the new WHO eTB RecMap with the conventional method of accessing TB recommendations using the WHO website, we will conduct a randomised controlled trial.

Methods
This study will be reported in accordance with the most recent guidance from the Consolidated Standards of Reporting Trials (CONSORT). 9 We registered the trial with ClinicalTrials.gov (trial registration: NCT04745897).

Study Setting
This study was a two-arm randomised controlled superiority trial to compare the accessibility of the WHO eTB RecMap (WHO eTB) to the conventional method of accessing TB recommendations through the WHO publications website (WHO TB). It will be administered using a SurveyMonkey® questionnaire, accessible through a link shared via email. Participants will respond to demographic questions and will subsequently be randomised using 1:1 allocation to access a recommendation using either WHO eTB or WHO TB (platforms). Randomization is stratified by participant background (e.g. patient, healthcare provider, policymaker) in order to ensure balance between groups. Participants will complete Likert-scale and multiple-choice questions about the platform which they were allocated. After completing the key portion of the trial, they will receive information about the alternative platform to respond to a question on their preference. Stakeholders who consider themselves to be users or potential users of TB guidelines, recommendations, and policy advice are eligible. For the purposes of this trial, we define a user as someone who responds "yes" to the question "have you ever accessed TB guidelines, recommendations or policy advice in the past?". A potential user is someone who responds "yes" to the question "do you plan on accessing TB guidelines, recommendations or policy advice in the future?". Eligible participants could be part of any group with a stake in TB, including the public, healthcare providers, policymakers, and researchers, and there are no restrictions on country of origin, level of education, or prior TB work experience. Individuals who were involved in WHO eTB development are not eligible, thus will not be invited to participate.

Recruitment
We will use a targeted snowball recruitment strategy by emailing survey links to WHO TB Guideline Development Group (GDG) members, program managers, and other stakeholders involved in the process of using and applying TB guideline recommendations. We will request that these members disseminate the survey within their networks, which may have included healthcare providers, policymakers, researchers, and people living with TB. Furthermore, we plan to share the survey link with TB, infectious disease, and guideline groups on the social media platforms Twitter and LinkedIn.

Intervention and Comparison
In this randomised controlled superiority trial, the intervention is the new WHO eTB RecMap (WHO eTB), and the comparison is the conventional method of accessing TB recommendations using the WHO website (WHO TB). We will ask survey participants to complete an activity in searching for the same recommendation using the one platform which they had been randomly allocated. Instructions and questions for both arms are worded as similarly as possible. See Table  1 for an overview of the differences between WHO eTB and WHO TB.

Outcomes
This trial uses several of the outcomes that have been validated in the evaluation of GRADE Summary of Findings (SoF) tables and are defined as follows. [10][11][12][13] Primary Outcome Accessibility of Information The primary outcome of interest is the accessibility of information available on the WHO eTB compared to WHO TB. We define accessibility as the ability to access and use the presented information. This outcome considers the four following domains: (1) how easy it was to find the information (2) how easy it was to understand the information (3) whether the presentation facilitated decision-making (4) whether the website was easy to navigate. We will use the composite of these four outcomes as primary outcome (see statistical analysis), and the individual domains will also be analyzed as secondary outcomes.

Secondary Outcomes Understanding
We define understanding as the correct comprehension of findings. This outcome will be measured using three multiple-choice questions with five choices and one correct answer.

Satisfaction
We define satisfaction as a stakeholder's impression of platform presentation. This outcome considers the presentation of three domains: (1) home page (2) recommendation list (3) individual recommendation.

Preference
We define preference as a greater liking of one platform over the other. All participants will be provided with a short demonstration of both platforms. They will subsequently be asked 'between the WHO Tuberculosis Guidelines (current website), and the WHO eTB Guidelines (alternative website), which do you prefer?'.

Outcomes Measurement
We will use the original Likert-scale to obtain responses for the outcomes of accessibility and satisfaction. We will measure preference using a Likert-type scale to express the degree of preference with seven answer options (1 = strongly prefer WHO TB, 2 = prefer WHO TB, 3 = somewhat prefer WHO TB, 4 = same preference for WHO TB and eTB, 5 = somewhat prefer WHO eTB, 6 = prefer WHO eTB, 7 = strongly prefer WHO eTB).

Sample Size Calculation
Sample size was calculated using the primary outcome of accessibility in WINPEPI® (PEPI-for-Windows) version 11.65. For this two-sided ( = 0.05) superiority analysis, these computations were made based on a t-test with the null hypothesis that there is no difference between the WHO eTB and WHO TB in the accessibility of information.

H0: WHO eTB = WHO TB H1: WHO eTB ≠ WHO TB
With sample sizes of 122 per arm (244 total) we would achieve 80% power to detect a difference on the Likert-scale of 0.5 (effect size) with a standard deviation of 1.0 between intervention and control groups. We applied the effect size and standard deviation from previous studies on GRADE SoF tables. [10][11][12][13] We assumed that 15% of those starting would not complete the survey, but we did not factor stratifying participants by stakeholder group into the calculation, as the aim of stratification was to balance participants rather than be sufficiently powered to detect subgroup effects.

Randomization
Participant response to a question on their role as a participant in this study will be used for stratification into one of four categories described in Figure 1. Participants within each of these categories will be randomly assigned in a 1:1 ratio to the WHO eTB or WHO TB arms. Participants will also be randomly assigned to access one of two recommendations in a 1:1 ratio. The same two recommendations will be presented for both arms and selected because they contain mostly plain language and were more accessible to non-clinical participants.

Allocation Concealment
The allocation sequence is concealed using SurveyMonkey® software based on a commercial, but unknown algorithm without a pre-identified sequence.

Blinding
Participants will not be aware of their random allocation to WHO eTB or WHO TB until disclosure (see Figure 1). Thus, participants will be blinded for all outcomes except the secondary outcome of preference. Neutral language is used in both trial arms to prevent promotion of the intervention or comparison. Nevertheless, some participants may have accessed WHO TB recommendations in the past and a small number may have been aware of the development of WHO eTB.

Consultation and Pilot Testing
We will perform a pilot test for survey length, question relevance, and question clarity. Participants will be researchers and information technology developers affiliated with McMaster University. Revisions will be made to the questionnaire using pilot test feedback. Pilot test participants will not be included in the final analysis.

Statistical Analysis
We will conduct the analyses using IBM SPSS® (Statistical Package for Social Sciences) version 23.

Descriptive Analysis
We will summarize participant baseline characteristics and outcomes using means and standard deviations (SD) for continuous variables, and proportions for categorical variables.

Inferential Analysis
We will perform a primary analysis including all randomised participants, except for those who completed the survey in less than five minutes. We determined this cut-off a priori because user testing determined that it would be impossible to complete the work in that time. We will conduct a second per-protocol analysis which excludes participants who were flagged by SurveyMonkey® software (see below). For the outcomes of accessibility and satisfaction, we will use t-tests and mean differences 95% confidence intervals (95% CIs) to compare means and standard deviations (SDs) between the intervention and control groups. For the outcome of understanding, we will use 2 tests and risk difference (RD) with 95% CIs to compare the proportion of correct responses between groups. Finally, for preference, we will present preference as mean (SD) overall and for both trial arms. Skewness, Shapiro-Wilk tests, and histograms will be used to evaluate whether the distribution was shifted toward the same preference in both groups. Levene's test of equal variances will be used for all t-tests and degrees of freedom were adjusted for p < 0.05. We will report all p-values to three decimal places, with values less than 0.001 reported as < 0.001.
Dropouts, missing data, and poor-quality responses We will use available case analysis for data from participants who responded to some survey questions after being assigned to the intervention. In order to prevent dropouts and missing data, we plan to implement the following strategies: (1) disseminating short emails to target stakeholder groups with a direct link to the survey, (2) informing participants that the survey will require 15 minutes, (3) participants who complete the survey have the option to enter a draw for a gift card, (4) all outcome questions are marked as mandatory, and (5) participants are randomised only after the collection of baseline characteristics.
We define participants who spend less than five minutes on the survey as inappropriate in our analyses, with the rationale that practical comprehension and completion of the survey could not be performed in five minutes. The average survey completion time is estimated to be 15 minutes. This is anticipated to prevent analysis of erroneous responses from participants who sought to simply gain access to the content of the survey or enter the gift card draw. For the perprotocol analysis, we will use a pre-defined but unknown SurveyMonkey® algorithm which flags poor-quality responses for straight-lining. Straight-lining is defined by SurveyMonkey® as responses to questions with the same answer option or pattern. Participants flagged for straightlining, as well as those who spend less than five minutes on the survey, will be removed from the per-protocol analysis.

Interim Analysis
We will perform pre-planned interim analysis based on the thesis defence date of the first author (MM). The interim analysis will not be used to stop the study or draw final conclusions.

Discussion
Research in Context To our knowledge, this is the first RCT comparing stakeholder feedback on the presentation of two guideline platforms. Nevertheless, several studies have explored stakeholder perceptions of guideline development and presentation, as well as the factors that influence their uptake. One qualitative study by Fearns, et al. 14 explored public perceptions of clinical practice guidelines and found that participants desired information to help them make decisions, but current numerical formats may not always be accessible to a public audience. Additionally, a content analysis by Santesso, et al. 15 found that patient versions of guidelines may not always address stakeholder needs, as they rarely include important EtD information, such as beliefs, values and preferences, accessibility, costs and feasibility. Furthermore, a realist review by Kastiner, et al. 16 which sought to identify the factors associated with guideline uptake, found that effective communication of content, including simple, clear and persuasive language, improved the implementability of guidelines by stakeholders.
This trial will expand on the methods of previous RCTs used to evaluate the presentation of guideline information, specifically, comparing new GRADE SoF tables to conventional formats. These studies evaluated participant understanding, accessibility, satisfaction and preference. 10,12,13 Carrasco-Labra, et al. 10 and Vandvik, et al. 13 identified that stakeholders preferred the presentation of risk differences over absolute risk estimates, as well as the inclusion of narrative statements to supplement numerical data. Furthermore, Akl, et al. 17 found that participants demonstrated a better understanding of strength of recommendations and quality of evidence when this information was presented as symbols, rather than numbers. Similar to our trial, these studies objectively evaluate perceptions of new to conventional formats to identify areas of improvement.

Strengths and Limitations
This study is anticipated to have several strengths. First, we use a randomised design conducted and reported in accordance with the CONSORT statement on randomised trials, which reduces the risk of confounding, selection and reporting bias. 9 Second, we use several previously validated outcomes from similar trials. 10-13 Third, we gather feedback from diverse group of stakeholders, thus improving the generalizability of findings using a carefully developed approach to presenting recommendations. 6 This study also has some limitations. First, the ability to blind participants is limited, as some may have been aware of eTB development. We consider this probability to be small, as the main publication and awareness campaigns will begin after the majority of participants have been recruited. Second, data will be collected using an online survey and, thus, there is limited control over the environment in which the survey is performed. Third, participants may claim to be part of more than one stakeholder group (e.g. a healthcare provider involved in research), but they are required to select just one for stratification. Fourth, we did not power our trial to conclusively evaluate results by participant strata.
Implications for Policy and Practice Tuberculosis guideline recommendations developed by the WHO assist stakeholders in making evidence-informed decisions on TB prevention, diagnosis, treatment, and care. 2,3 According to the Guidelines International Network (G-I-N) and the National Institute for Health and Care Excellence (NICE), stakeholder engagement is important to ensuring that guideline products are feasible and acceptable to end-users. 18,19 This study will engage stakeholders in evaluating the presentation of recommendations through the outcomes of accessibility, understanding, and satisfaction, which are surrogates for the correct implementation of evidence in practice.

Implications for Research
This study is anticipated to demonstrate how RCTs may be used to compare stakeholder feedback on guideline platforms. Future studies should seek to explain the findings of this trial through qualitative and user testing techniques, such as the study by Rosenbaum, et al. for SoF tables. 11 Furthermore, additional trials should focus on specific stakeholder groups, such as patients and the public, to determine optimal ways to present recommendations.