Skip to main content
Advertisement
  • Loading metrics

A cluster randomized trial assessing the effect of a digital health algorithm on quality of care in Tanzania (DYNAMIC study)

  • Rainer Tan ,

    Contributed equally to this work with: Rainer Tan, Godfrey Kavishe

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Software, Supervision, Writing – original draft, Writing – review & editing

    rainer.tan@unisante.ch

    Affiliations Centre for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland, Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania, Swiss Tropical and Public Health Institute, Allschwil, Switzerland, University of Basel, Basel, Switzerland

  • Godfrey Kavishe †,

    Contributed equally to this work with: Rainer Tan, Godfrey Kavishe

    † Deceased.

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Software, Supervision, Writing – original draft, Writing – review & editing

    Affiliation National Institute of Medical Research–Mbeya Medical Research Centre, Mbeya, United Republic of Tanzania

  • Alexandra V. Kulinkina,

    Roles Conceptualization, Project administration, Supervision, Writing – review & editing

    Affiliations Swiss Tropical and Public Health Institute, Allschwil, Switzerland, University of Basel, Basel, Switzerland

  • Sabine Renggli,

    Roles Data curation, Project administration, Software, Supervision, Writing – review & editing

    Affiliation Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania

  • Lameck B. Luwanda,

    Roles Conceptualization, Project administration, Supervision, Writing – review & editing

    Affiliation Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania

  • Chacha Mangu,

    Roles Project administration, Supervision, Writing – review & editing

    Affiliation National Institute of Medical Research–Mbeya Medical Research Centre, Mbeya, United Republic of Tanzania

  • Geofrey Ashery,

    Roles Project administration, Supervision, Writing – review & editing

    Affiliation Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania

  • Margaret Jorram,

    Roles Supervision, Writing – review & editing

    Affiliation Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania

  • Ibrahim Evans Mtebene,

    Roles Software, Writing – review & editing

    Affiliation Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania

  • Peter Agrea,

    Roles Software, Writing – review & editing

    Affiliation National Institute of Medical Research–Mbeya Medical Research Centre, Mbeya, United Republic of Tanzania

  • Humphrey Mhagama,

    Roles Supervision, Writing – review & editing

    Affiliation National Institute of Medical Research–Mbeya Medical Research Centre, Mbeya, United Republic of Tanzania

  • Kristina Keitel,

    Roles Conceptualization, Writing – review & editing

    Affiliations Swiss Tropical and Public Health Institute, Allschwil, Switzerland, University of Basel, Basel, Switzerland, Pediatric Emergency Department, Department of Pediatrics, University Hospital Bern, Bern, Switzerland

  • Marie-Annick Le Pogam,

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    Affiliation Centre for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland

  • Nyanda Ntinginya ,

    Roles Funding acquisition, Project administration, Supervision, Writing – review & editing

    ‡ NN, HM and VD also contributed equally to this work.

    Affiliation National Institute of Medical Research–Mbeya Medical Research Centre, Mbeya, United Republic of Tanzania

  • Honorati Masanja ,

    Roles Conceptualization, Funding acquisition, Project administration, Supervision, Writing – review & editing

    ‡ NN, HM and VD also contributed equally to this work.

    Affiliation Ifakara Health Institute, Dar es Salaam, United Republic of Tanzania

  •  [ ... ],
  • Valérie D’Acremont

    Roles Conceptualization, Funding acquisition, Supervision, Writing – review & editing

    ‡ NN, HM and VD also contributed equally to this work.

    Affiliations Centre for Primary Care and Public Health (Unisanté), University of Lausanne, Lausanne, Switzerland, Swiss Tropical and Public Health Institute, Allschwil, Switzerland, University of Basel, Basel, Switzerland

  • [ view all ]
  • [ view less ]

Abstract

Digital clinical decision support tools have contributed to improved quality of care at primary care level health facilities. However, data from real-world randomized trials are lacking. We conducted a cluster randomized, open-label trial in Tanzania evaluating the use of a digital clinical decision support algorithm (CDSA), enhanced by point-of-care tests, training and mentorship, compared with usual care, among sick children 2 to 59 months old presenting to primary care facilities for an acute illness in Tanzania (ClinicalTrials.gov NCT05144763). The primary outcome was the mean proportion of 14 major Integrated Management of Childhood Illness (IMCI) symptoms and signs assessed by clinicians. Secondary outcomes included antibiotic prescription, counseling provided, and the appropriateness of antimalarial and antibiotic prescriptions. A total of 450 consultations were observed in 9 intervention and 9 control health facilities. The mean proportion of major symptoms and signs assessed in intervention health facilities was 46.4% (range 7.7% to 91.7%) compared to 26.3% (range 0% to 66.7%) in control health facilities, an adjusted difference of 15.1% (95% confidence interval [CI] 4.8% to 25.4%). Only weight, height, and pallor were assessed statistically more often when using the digital CDSA compared to controls. Observed antibiotic prescription was 37.3% in intervention facilities, and 76.4% in control facilities (adjusted risk ratio 0.5; 95% CI 0.4 to 0.7; p<0.001). Appropriate antibiotic prescription was 81.9% in intervention facilities and 51.4% in control facilities (adjusted risk ratio 1.5; 95% CI 1.2 to 1.8; p = 0.003). The implementation of a digital CDSA improved the mean proportion of IMCI symptoms and signs assessed in consultations with sick children, however most symptoms and signs were assessed infrequently. Nonetheless, antibiotics were prescribed less often, and more appropriately. Innovative approaches to overcome barriers related to clinicians’ motivation and work environment are needed.

Author summary

Digital health tools have been created to help healthcare workers provide better care, but their real-world impact is still uncertain. This study evaluated the use of ePOCT+, a digital health clinical decision support algorithm for health providers, combined with point-of-care diagnostic tools (CRP tests, pulse oximetry), training, and mentorship. The study compared 9 primary care health facilities using ePOCT+ with 9 facilities providing care as usual. The study found that using the digital health intervention improved the assessment of important symptoms and signs in children aged 2 to 59 months with acute illnesses. However, many symptoms and signs were still not frequently assessed. In addition, antibiotic prescriptions were halved in facilities using the tool, and the appropriateness of prescriptions improved significantly. Despite these benefits, challenges related to healthcare workers’ motivation and work environments must be explored to fully realize the potential of such tools. These findings suggest that digital health tools can improve the quality of care and address issues like overprescription of antibiotics. However, broader strategies are needed to support healthcare workers in delivering comprehensive, high-quality care in similar settings globally.

Introduction

Millions of preventable deaths are attributed to suboptimal healthcare quality [1]. Factors such as staff shortages, inadequate budget allocation, poor clinical knowledge, and limited access to quality medical education, mentorship and supervision collectively contribute to this issue [24]. In response to this challenge, and to reduce childhood mortality, the World Health Organization developed the Integrated Management of Childhood Illness (IMCI) Chartbook [5]. Since its inception, over 100 countries have implemented the guidelines, and IMCI may reduce mortality and improve quality of care [6,7]. However, poor adherence to IMCI is common, limiting its benefits [810].

Digital Clinical Decision Support Algorithms (CDSAs) were devised to enhance adherence to clinical guidelines. These tools, typically operating on electronic tablets or mobile phones, guide healthcare providers through the consultation process, by prompting the evaluation of symptoms, signs, and recommended diagnostic tests, to finally propose the appropriate diagnosis and treatment [11,12]. While several studies have found that using these digital CDSAs improve adherence to IMCI, a noteworthy research gap is that many of these investigations were conducted in controlled study settings, and most lacked randomization [1320].

ePOCT+, a digital CDSA, was developed based on insights from two previous generations of CDSAs [21,22], specifically addressing challenges by our CDSAs and others, such as limited scope and information technology difficulties [23]. The present study aimed to assess whether this CDSA associated with point-of-care tests, training, and mentorship, would improve the quality of care for sick children compared to usual care, by comparing adherence to IMCI in a pragmatic cluster randomized trial.

Methods

Study design

The present study is an open-label, parallel-group, cross-sectional cluster randomized trial within the DYNAMIC Tanzania project. An external clinical researcher observed a sample of consultations from health facilities from both study arms documenting adherence of healthcare providers to quality-of-care indicators. The study was a planned ancillary study within a larger cluster randomized trial conducted between 1 December 2021 and 31 October 2022 using a sample of the clusters [24]. A cluster design was chosen since the intervention was targeted at the health facility and healthcare provider. The trial design and rationale are outlined in the protocol available in the parent trial registration on ClinicalTrials.gov number NCT05144763 and in the supplementary materials (S1 File). The detailed statistical analysis plan for this ancillary study is also available in the supplementary materials (S2 File).

The study design and implementation were collaboratively executed between both Tanzanian (Ifakara Health Institute, National Institute for Medical Research—Mbeya Medical Research Centre) and Swiss (Centre for Primary Care and Public Health [Unisanté]–University of Lausanne, and Swiss Tropical and Public Health Institute) partners. The design was guided by input from patients, and health providers during the implementation of similar trials in Tanzania [14,22,25]. Over 100 community engagement meetings involving over 7,000 participants were conducted before and during the study. These meetings included discussions with Community and Regional Health Management Teams in Tanzania.

Participants

The health facility was the unit of randomization since the intervention targeted both the healthcare provider and health facility. Primary care health facilities (dispensaries or health centers) were eligible for inclusion if they performed on average 20 or more consultations with children 2 months to 5 years per week, were government or government-designated health facilities, and were located less than 150 km from the research institutions. Specific to this study, consultations were only included if healthcare providers had been trained to use ePOCT+, and the ePOCT+ tool was functioning on the day of observation (no IT issues related to power outages, or crashes reported).

In contrast to the larger trial that included children aged 1 day to 14 years, this ancillary study included only children aged 2 to 59 months old presenting for an acute medical or surgical condition at participating health facilities. Children presenting solely for scheduled consultations for a chronic disease (e.g. HIV, tuberculosis, malnutrition), for routine preventive care (e.g. growth monitoring, vaccination), or a follow-up consultation were excluded.

The study was conducted in 5 councils within the Mbeya and Morogoro regions of Tanzania, with two councils being semi-urban and three rural. Malaria prevalence in febrile children was low in three councils, and moderately high in two. HIV prevalence among children less than 5 years old in Tanzania is 0.4% [26]. Healthcare for children under 5 years of age is free for all acute illnesses at government or government-designated primary health facilities, including the cost of medications [27]. Nurses and clinical officers routinely provide outpatient care in dispensaries, while in health centers medical doctors sometimes provide care as well. Clinical Officers, the predominant healthcare providers at primary health facilities, are non-physician health professionals with 2–3 years of clinical training following secondary school [28].

Interventions

The intervention involved equipping health facilities with ePOCT+, an electronic clinical decision support algorithm on an Android based tablet (Fig 1), along with associated point-of-care tests (C-Reactive Protein, Hemoglobin, pulse oximetry), training, and mentorship. ePOCT+ prompts the healthcare provider to answer questions about demographics, symptoms, signs, and tests [23]. Based on the answers, ePOCT+ proposes one or more diagnoses, treatments, and management plans including referral recommendation. Healthcare providers had the possibility to deviate from ePOCT+ recommendations, and had the option to not use the tool. In order to move forward within the different sections of the digital tool, it was mandatory to respond to all IMCI symptoms and signs, except for height and mid-upper arm circumference (MUAC) which was optional. The tool allowed some signs to be estimated (temperature, respiratory rate) or based on recent measurements (weight). Detailed description on the development process and features of ePOCT+ and the medAL-reader application can be found in separate publications [23,29].

thumbnail
Fig 1. Screen-shots of different stages of ePOCT+ running on the medAL-reader application.

Stages are shown in order of appearance, however not all stages are shown in the figure.

https://doi.org/10.1371/journal.pdig.0000694.g001

The implementation team provided mentorship to intervention health facilities. This mentoring consisted of regular visits to health facilities every 2–3 months, and frequent communication via phone calls or group messages (3–4 times per month) to address issues, offer guidance, and gather feedback on the new tools. Quality-of-care dashboards were shared through group messages, enabling healthcare providers to compare their antibiotic prescription rates, uptake, and other quality-of-care indicators with other facilities (benchmarking). Control health facilities continued with usual care, did not have access to clinical data dashboards, and only received visits from the implementation team to help resolve issues related to the electronic case report forms (eCRFs).

The infrastructure provided to all health facilities (control and intervention) included a tablet for each outpatient consultation room, a router, a local server (Raspberry Pi), internet connectivity, and backup power (battery or solar system if needed). If unavailable weighing scales, mid-upper arm circumference (MUAC) bands, and thermometers were provided to all health facilities. Healthcare providers from both intervention and control facilities underwent equivalent clinical refresher training on IMCI and concepts of antibiotic stewardship. Additionally, specific training was provided on the use of the ePOCT+ CDSA in intervention facilities and the use of the eCRF in control facilities.

Outcomes

The primary outcome was the mean proportion of 14 pre-identified major IMCI symptoms and signs assessed by the healthcare provider, as observed by an external clinical research assistant. The included symptoms were fever, cough or difficult breathing, convulsions in this illness, diarrhea, ear pain or discharge, child unable to drink or breastfeed, and child vomits everything. The included signs were measurement of temperature, respiratory rate, pallor, weight, mid-upper arm circumference (MUAC), height, and skin turgor. In specific circumstances, some patients were not included in the denominator for specific signs as they were not clinically indicated as defined by IMCI: they include MUAC in children less than 6 months old, respiratory rate in the absence of cough or difficult breathing, and skin turgor in the absence of diarrhea. If cough or difficult breathing was not assessed, then we took the most conservative approach assuming that respiratory rate should have been measured, and the same for diarrhea and skin turgor. Of note “lethargic and unconscious” was considered as assessed if the clinician asked the caregiver if it was present during the illness, and not based on observation of the child as being “lethargic or unconscious”.

Secondary outcomes include the proportion of consultations during which each major IMCI symptom and sign were assessed, the proportion of which other symptoms and signs were assessed (S2 File), the proportion of consultations where different IMCI counseling was conducted, and proportion of consultations for which antibiotics were prescribed. The rationale for the distinction between “major” IMCI and “other” symptoms and signs are described in detail in the statistical analysis plan (S2 File). Prescription of antibiotics was assessed by the research assistant by observing the actual prescription prescribed. The appropriateness of antibiotic prescription in relation to the retained diagnosis was also assessed. An antibiotic prescription was considered appropriate if one of the retained diagnoses required an antibiotic as per IMCI or the WHO hospital pocket book, and the absence of a prescription if no diagnosis required an antibiotic [30,31]. An appropriate antimalarial prescription was a prescription of any antimalarial if there was a positive malaria test. Assessment of appropriateness was conducted blinded to the study arm.

All outcomes pertained to the cluster level (health facility), and were assessed by an external clinical research assistant who observed the consultations in the consultation room without interfering with the consultation. The external clinical research assistants were clinical officers with experience in primary care consultations for children. Data was collected using a structured and pre-tested observation form programmed on ODK, and collected on an Android-based tablet. The observation form was based on the 2012 DHS Service Provision Assessment Survey form [32]. Of note, a more recent modification to this survey was developed after the initial planning of this study [32]. Modifications were made to the survey form, to shorten the duration of the evaluation and align it more closely with the aim of the evaluation, to incorporate additional signs and symptoms in line with IMCI 2014 guidelines such as duration of symptoms and the symptomatic assessment of lethargic or unconscious, and additions used by similar evaluations conducted previously [17].

Initially, antibiotic prescription was considered a co-primary outcome alongside the current primary outcome (proportion of 14 major IMCI symptoms and signs) but was later reclassified as a secondary outcome. We made this change to focus the analysis on quality of care, given that antibiotic prescription was already the primary outcome of the large longitudinal cluster randomized trial [24].

Sample size

The original sample size calculation was based on the previous co-primary outcome of antibiotic prescription. To detect a 25% absolute decrease in mean antibiotic prescription from a baseline of 50%, using an intraclass correlation coefficient (ICC) of 0.10 and an alpha of 0.05, a sample size of 25 patients in 9 clusters (health facilities) per arm was required to have 80% power. The ICC was based on studies evaluating prescription variations among different health care facilities/practices, ranging from 0.07 to 0.10 [3336].

Expecting a high variability in baseline values of symptoms and signs assessed by a clinician and between clinicians [8,13,17,37], the above sample size would have 67–93% power to detect a 30% absolute increase in the assessment of major IMCI symptoms and signs, considering a baseline value of 40–60%, an ICC of 0.15–0.25, and an alpha of 0.05.

Randomization

Within the parent trial, health facilities were randomized 1:1 by an independent statistician, stratified by monthly attendance, type of health facility (dispensary or health center), region, and council [24]. For the present ancillary study, another independent statistician sampled 18/40 facilities to be included. This included 8/8 health centers (4 intervention, and 4 control), and 10/32 dispensaries. Among the 32 dispensaries, 10 were randomly sampled, stratified by study arm and region (following the same 3:2 ratio in favor of the Morogoro region as done in the parent trial). Due to the nature of the intervention, it was not feasible to blind the healthcare providers, patients, study implementers, or external clinical research assistants (observers) to the intervention.

A convenience sampling was employed, whereby the external clinical researcher observed all eligible consultations while present at the health facility during normal standard working hours (Monday to Friday, 8:00 to 15:00).

Statistical methods

All analyses were performed using an intention-to-treat approach, i.e. all children with a recorded outcome were included in the analysis regardless if the intervention, ePOCT+, was used or not. The eCRF was designed to prevent missing data, as such all data was complete. All analyses were performed using a clustered-level analysis approach instead of an individual-level analysis due to the small number of clusters included [38,39]. Such an approach has been shown to be robust in cluster randomized trials with less than 30 clusters, with cluster sizes of >10, when assessing outcomes with a prevalence of > 10%, and performs best when cluster sizes are similar, and in case of high ICC [38,39]. The analysis was performed using a two-stage approach as outlined by Hayes et al [39,40], to adjust for both the cluster-level and individual-level covariates. In the first stage, we used a logistic regression model for binary outcomes and a linear regression model for continuous outcomes adjusting for covariates and ignoring clustering and trial arm. Cluster-level residuals were then calculated for each cluster. In the second stage, the residuals were compared to estimate risk ratios for the binary outcomes and mean risk difference for the continuous outcome (including the primary outcome) between study arms. Pre-specified cluster-level covariates were the type of health facility, council, healthcare worker cadre and healthcare worker years of experience. Pre-specified individual-level covariates of patients were age and sex. Covariates were pre-specified based on their potential influence on the study outcomes. All analyses were performed using Stata v16, v17 and v18 [41].

Ethics

Written informed consent was obtained from all parents or guardians of participants when attending the participating health facility during the enrollment period. We also requested, written informed consent from all healthcare providers for which their consultations were observed during this ancillary study. Ethical approvals were granted from the Ifakara Health Institute (IHI/IRB/No: 11–2020), the Mbeya Medical Research Ethics Committee (SZEC-2439/R.A/V.1/65), the National Institute for Medical Research Ethics Committee (NIMR/HQ/R.8a/Vol. IX/3486 and NIMR/HQ/R.8a/Vol. IX/3583) in Tanzania and from the cantonal ethics review board of Vaud (CER-VD 2020–02800) in Switzerland.

Results

Baseline characteristics of health facilities, healthcare providers, and patients

Between 23 March 2022 to 3 June 2022, 225 consultations were observed in 9 intervention facilities, and 225 consultations in 9 control facilities (Fig 2). The type of health facility, urban/rural localization, and region were well distributed between both study arms (Table 1). A total of 17 healthcare providers saw patients in the control arm, and 22 in the intervention arm during the study. Distribution of sex, age, working experience and cadre of healthcare providers were similar in both study arms. The number of healthcare providers with less than 3 years of experience was slightly higher in the control arm. Among the included patients, there were slightly more female patients and median age was slightly higher in the intervention arm compared to the control. Within the intervention arm, ePOCT+ was used throughout the whole consultation in 213/225 (95%) of consultations, partially used in 5/225 (2%), used after the consultation in 6/225 (3%), and not used at all in 1/225 (0.4%) of consultations.

thumbnail
Table 1. Characteristics of health facilities, healthcare providers, and patients.

https://doi.org/10.1371/journal.pdig.0000694.t001

Assessment of symptoms and signs, and counseling

The primary outcome of mean proportion of major IMCI symptoms and signs assessed was higher by an adjusted difference of 15.1% (95% confidence interval [CI] 4.8% to 25.4%), p-value 0.007) in intervention health facilities (mean of 46.4%, range 7.7% to 91.7%) compared to control health facilities (mean of 26.3%, range 0% to 66.7%) (Table 2 and Fig 3). Weight, mid-upper arm circumference (MUAC) and pallor were the only individual assessments among the pre-identified major IMCI symptoms and signs that showed a statistically significant difference in the adjusted risk ratio (Table 2 and Fig 4). Among other symptoms and signs assessed, there was a statistically significant difference in the proportion of patients for which the duration of cough, duration of diarrhea, mother’s HIV status were assessed, and the proportion of children who were undressed for the physical examination (Table 3). There was no statistical difference in the proportion of counseling topics covered by healthcare providers between study arms (Table 4). For most outcomes the intraclass correlation coefficient (ICC) was relatively high suggesting high variability in adherence to IMCI between health facilities. For the primary outcome of major IMCI symptoms and signs assessed, the ICC was higher in intervention health facilities (0.608) compared to control health facilities (0.354). This difference can also be seen when visualizing the mean proportion of major IMCI symptoms and signs assessed from each health facility in a scatterplot (Fig 3).

thumbnail
Fig 3. Scatter-plot of the mean proportion of the major IMCI symptoms and signs assessed.

https://doi.org/10.1371/journal.pdig.0000694.g003

thumbnail
Fig 4. Proportion of individual major IMCI symptoms and signs assessed.

https://doi.org/10.1371/journal.pdig.0000694.g004

Antibiotic and antimalarial prescription

Antibiotic prescription as observed by the external clinical researcher was lower in intervention health facilities compared to control health facilities with an adjusted risk ratio of 0.5, 95%CI 0.4 to 0.7 p-value <0.001 (Table 5 and Fig 5). The documented antibiotic prescription by healthcare providers (in the ePOCT+ tool for intervention facilities, and eCRF in control facilities) during the same 5-week period collected by the external clinical researchers in the same health facilities of the present analysis, was slightly lower than that observed by the external clinical researchers. The adjusted risk ratio remained nonetheless similar, 0.4, 95% CI 0.2 to 0.7, p-value 0.005.

thumbnail
Fig 5. Reported and observed antibiotic prescription and appropriateness.

Panel A presents the antibiotic prescription as observed by the external clinical researchers in 225 consultations in the control arm and 225 consultations in the intervention arm. It also presents the reported antibiotic prescription as documented by the healthcare providers in the ePOCT+ digital health tool among 239 new consultations in patients aged 2–59 months in intervention health facilities, and the documented antibiotic prescription within the eCRF in the control health facilities during the same days and same health facilities when the external clinical researchers were present. Panel B presents the appropriateness of antibiotic prescription according to the diagnoses documented or stated by the healthcare providers in the 225 control arm consultations and 225 intervention arm consultations.

https://doi.org/10.1371/journal.pdig.0000694.g005

thumbnail
Table 5. Antibiotic prescription and appropriate antibiotic and antimalarial prescription.

https://doi.org/10.1371/journal.pdig.0000694.t005

81.9% of antibiotic prescriptions were appropriate in the intervention arm, compared to 51.4% in the control arm, adjusted relative risk of 1.5, 95% CI 1.2–1.5, p-value 0.003. All patients with malaria appropriately received an antimalarial treatment, and no patient without malaria received an antimalarial in both study arms.

Discussion

This cluster randomized controlled trial in Tanzania found that the use of ePOCT+, a digital clinical decision support algorithm, for the management of sick children aged 2–59 months old, moderately increased (by 15%) the mean proportion of major IMCI symptoms and signs assessed by healthcare providers in primary care level health facilities. The overall proportion of IMCI symptoms and signs assessed at each individual consultation however remained low.

The overall increase in the mean proportion of major IMCI symptom and signs assessed by healthcare providers aligns with previous studies in Tanzania [13], Afghanistan [37], Nigeria [17], and Burkina Faso [16], but in contrast to findings in South Africa [42], which did not find improvements in the assessment of IMCI symptoms and signs. While these findings suggest that digital clinical decision support algorithms can positively enhance quality of care for sick children, there is much room for improvement. Indeed, the proportion of most individual IMCI symptoms and signs assessed in both control and intervention arms was low, much lower than in previous studies, and many symptoms and signs were not assessed more frequently in intervention health facilities. Differences compared to other studies, may be explained by the more pragmatic nature of the study compared to other studies conducted in more controlled settings (shorter pilot studies) where the Hawthorne effect may have a greater influence on healthcare provider behavior [13]. In addition, our study found much poorer adherence to IMCI in the control arm compared to most other similar studies; which could partly be explained by differences in the setting and healthcare providers, notably in the frequency and type of IMCI training provided to healthcare providers [16,17]. The low proportion of children assessed for danger signs in order to identify children at highest risk of mortality is most concerning (33% for convulsions, 48% for unable to drink or breastfeed, and 36% for vomiting everything in the intervention arm). Respiratory rate was also infrequently assessed (29% in intervention arm), despite being an essential sign to distinguish children with cough or difficulty breathing requiring antibiotics or not [30].

The statistically significant increase in the assessment of weight (aRR 4.9 [95% CI 1.9 to 12.9]), and MUAC (aRR 5.5 [95% CI 1.7 to 17.6]) is noteworthy, as they are critical anthropometric measures to identify children with malnutrition, a condition that contributes significantly to childhood morbidity and mortality. Systematically measuring weight and MUAC to identify and manage severe malnutrition can indeed improve clinical outcomes as well as the long-term health status of children [43]. The low proportion of children assessed for height/length reflects the difficulty and constraints of this measurement in particular [44], and the impact of not requiring the measurement to be mandatory within the digital tool.

While improved adherence to the assessment of IMCI symptoms and signs would likely be beneficial, translation to improved clinical outcomes should not automatically be assumed. Healthcare providers often integrate a number of clinical cues that may allow them to distinguish which child would truly require danger signs to be assessed, or respiratory rate to be measured. For example, a 2 year old child presenting to a primary care health facility smiling and playing in the consultation room, with complaints of cough and runny nose for the past 2 days without difficulty breathing or fever, is unlikely to have danger signs and very often not have fast breathing or chest indrawing. This raises the important question on how to best assess quality of care, and whether it can be done without assessment of clinical outcome.

The large variation in mean proportion of major IMCI symptoms and signs between intervention health facilities provide clues to higher potential benefits of the intervention. The three best health facilities have a mean score of 60% or above, and the three worst below 30%. While clinical decision support algorithms may help improve knowledge and information on what symptoms and signs to assess, it does not completely address the other barriers and bad habits linked to poor adherence to IMCI [9,10]. As with many complex health interventions, implementation of new interventions, and or guidelines may often not succeed with training alone, instead concomitant and meaningful mentorship must accompany it [45]. Indeed training, mentorship, and dashboards integrating benchmarking were part of the current intervention package, however these supportive interventions were not targeted towards assuring adherence to the assessment of many IMCI symptoms and signs but rather targeted on antibiotic stewardship, and overall uptake on the use of the tools. Adaptations to these supportive tools to target specific IMCI quality of care measures may help [4648]. Further qualitative investigations are underway to better understand healthcare provider perspectives on barriers in adhering to ePOCT+ and the IMCI chartbook.

The study also revealed a two-fold reduction in antibiotic prescriptions (adjusted relative risk 0.5, 95% CI 0.4 to 0.7), and 50% improvement in the appropriate use of antibiotics in health facilities using ePOCT+ (adjusted relative risk 1.5 (95% CI 1.2 to 1.8). These are critical findings given the global concern for bacterial antimicrobial resistance [49], and validating the results found in the parent trial [24]. Similar reductions were found in the parent trial using the intention-to-treat results over the full 11 month trial period (adjusted relative risk 0.6 (95% CI 0.5 to 0.6) [24], and in the same 5-week period as the present trial (adjusted relative risk 0.4, 95% CI 0.2 to 0.7). However, the documented antibiotic prescription as observed by external clinical researchers was slightly higher than that documented by the healthcare providers in ePOCT+ (intervention health facilities) and the eCRF (control health facilities), suggesting that some healthcare providers may under report antibiotic prescription in ePOCT+ and the eCRF.

There were several limitations to this study. First, the Hawthorne effect, the presence of an external clinical researcher observing the consultation may have influenced the healthcare provider’s practice in both study arms. Indeed the uptake of the ePOCT+ tool in this study was higher compared to the parent trial (95% versus 76%), likely due to the presence of the researchers. Despite this, adherence to the IMCI chartbook was relatively low, and substantially lower compared to other studies, suggesting that the Hawthorne effect may not have had such a big impact, and the desirability bias minimal. Second, sample size; while the study was sufficiently powered for the primary outcome, interpretation of the secondary outcomes would have benefited from a higher sample size. Indeed many secondary outcomes did not show statistical significance despite relatively high effect size, likely due to the higher than expected heterogeneity between health facilities as indicated by the high ICC. Third, the intervention package included not only the digital tool, but also mentorship and benchmarking quality of care dashboards. It is thus not possible to understand what part, and to what extent the intervention package impacted quality of care. Finally as discussed in previous paragraphs, adherence to IMCI is an imperfect proxy for the measurement of quality of care.

In conclusion, a digital clinical decision support algorithm package can help improve quality of care, however adherence to IMCI remained low for many symptoms and signs in a close to real world assessment. Further efforts including innovative approaches to improve quality of care are highly needed. The implementation of multiple interventions, such as the development and improvement of supportive mentorship of clinicians, better healthcare provider incentives, task-shifting, ongoing training and performance accountability may help address the many barriers to quality of care.

Acknowledgments

We would like to first thank all the participating healthcare providers, patients and caregivers. We acknowledge the contributions of the research assistants at the Ifakara Health Institute and Mbeya Medical Research Centre–National Institute of Medical Research, who assisted in the data collection, the Information Technology staff at Unisanté (Sylvain Schaufelberger, Greg Martin), and staff at Wavemind (Emmanuel Barchichat, Alain Fresco, and Quentin Girard) for their work on the medAL-suite during the study, and Community Health Management Team members in the 5 participating councils in Tanzania for their collaboration in implementing the study. We acknowledge researchers of the Tools of Integrated Management of Childhood Illness and DYNAMIC Rwanda project for their contributions to ePOCT+ and the many common research activities (Dr Fenella Beynon, Dr Lena Matata, Dr Helene Langet, Dr Ludovico Cobuccio, Dr Victor Rwandacu, Dr Robert Moshiro). We would also like to acknowledge Janet Urquhart-Ducharme for her assistance in proofreading a version of this manuscript. We would like to acknowledge Dr Irene Masanja for her work on the initial development of the study, who regrettably passed away before the start of the study. Dr Godfrey Kavishe passed away before the submission of the final version of this manuscript. Rainer Tan accepts responsibility for the integrity and validity of the data collected and analyzed.

References

  1. 1. Kruk ME, Gage AD, Joseph NT, Danaei G, García-Saisó S, Salomon JA. Mortality due to low-quality health systems in the universal health coverage era: a systematic analysis of amenable deaths in 137 countries. The Lancet. 2018;392(10160):2203–12. pmid:30195398
  2. 2. Di Giorgio L, Evans DK, Lindelow M, Nguyen SN, Svensson J, Wane W, et al. Analysis of clinical knowledge, absenteeism and availability of resources for maternal and child health: a cross-sectional quality of care study in 10 African countries. BMJ Glob Health. 2020;5(12). Epub 2020/12/24. pmid:33355259; PubMed Central PMCID: PMC7751199.
  3. 3. Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, et al. High-quality health systems in the Sustainable Development Goals era: time for a revolution. The Lancet Global Health. 2018;6(11):e1196–e252. pmid:30196093
  4. 4. World Health Organization. World Health Report, Working Together for Health. Geneva: World Health Organization; 2006.
  5. 5. World Health Organization. Integrated Management of Childhood Ilness Chartbook. Geneva: 2014.
  6. 6. Boschi-Pinto C, Labadie G, Dilip TR, Oliphant N.L. Dalglish S, Aboubaker S, et al. Global implementation survey of Integrated Management of Childhood Illness (IMCI): 20 years on. BMJ open. 2018;8(7):e019079. pmid:30061428
  7. 7. Gera T, Shah D, Garner P, Richardson M, Sachdev HS. Integrated management of childhood illness (IMCI) strategy for children under five. Cochrane Database Syst Rev. 2016;(6):Cd010123. Epub 2016/07/06. pmid:27378094; PubMed Central PMCID: PMC4943011.
  8. 8. Krüger C, Heinzel-Gutenbrunner M, Ali M. Adherence to the integrated management of childhood illness guidelines in Namibia, Kenya, Tanzania and Uganda: evidence from the national service provision assessment surveys. BMC Health Serv Res. 2017;17(1):822. Epub 2017/12/15. pmid:29237494; PubMed Central PMCID: PMC5729502.
  9. 9. Kiplagat A, Musto R, Mwizamholya D, Morona D. Factors influencing the implementation of integrated management of childhood illness (IMCI) by healthcare workers at public health centers & dispensaries in Mwanza, Tanzania. BMC public health. 2014;14:277. Epub 2014/03/29. pmid:24666561; PubMed Central PMCID: PMC3987128.
  10. 10. Lange S, Mwisongo A, Mæstad O. Why don’t clinicians adhere more consistently to guidelines for the Integrated Management of Childhood Illness (IMCI)? Soc Sci Med. 2014;104:56–63. Epub 2014/03/04. pmid:24581062.
  11. 11. Beynon F, Guérin F, Lampariello R, Schmitz T, Tan R, Ratanaprayul N, et al. Digitalizing Clinical Guidelines: Experiences in the Development of Clinical Decision Support Algorithms for Management of Childhood Illness in Resource-Constrained Settings. Global Health: Science and Practice. 2023. pmid:37640492
  12. 12. Keitel KD’Acremont V. Electronic clinical decision algorithms for the integrated primary care management of febrile children in low-resource settings: review of existing tools. Clinical microbiology and infection: the official publication of the European Society of Clinical Microbiology and Infectious Diseases. 2018;24(8):845–55. Epub 2018/04/24. pmid:29684634.
  13. 13. Mitchell M, Hedt-Gauthier BL, Msellemu D, Nkaka M, Lesh N. Using electronic technology to improve clinical care—results from a before-after cluster trial to evaluate assessment and classification of sick children according to Integrated Management of Childhood Illness (IMCI) protocol in Tanzania. BMC medical informatics and decision making. 2013;13:95. Epub 2013/08/29. pmid:23981292; PubMed Central PMCID: PMC3766002.
  14. 14. Rambaud-Althaus C, Shao A, Samaka J, Swai N, Perri S, Kahama-Maro J, et al. Performance of Health Workers Using an Electronic Algorithm for the Management of Childhood Illness in Tanzania: A Pilot Implementation Study. The American journal of tropical medicine and hygiene. 2017;96(1):249–57. Epub 2017/01/13. pmid:28077751; PubMed Central PMCID: PMC5239703.
  15. 15. Horwood C, Haskins L, Mapumulo S, Connolly C, Luthuli S, Jensen C, et al. Electronic Integrated Management of Childhood Illness (eIMCI): a randomized controlled trial to evaluate an electronic clinical decision-making support system for management of sick children in primary health care facilities in South Africa. BMC Health Services Research. 2024;24(1):177. pmid:38331824
  16. 16. Sarrassat S, Lewis JJ, Some AS, Somda S, Cousens S, Blanchet K. An Integrated eDiagnosis Approach (IeDA) versus standard IMCI for assessing and managing childhood illness in Burkina Faso: a stepped-wedge cluster randomised trial. BMC health services research. 2021;21(1):354–. pmid:33863326.
  17. 17. Bernasconi A, Crabbé F, Adedeji AM, Bello A, Schmitz T, Landi M, et al. Results from one-year use of an electronic Clinical Decision Support System in a post-conflict context: An implementation research. PLoS One. 2019;14(12):e0225634. Epub 2019/12/04. pmid:31790448; PubMed Central PMCID: PMC6886837.
  18. 18. Bernasconi A, Crabbé F, Rossi R, Qani I, Vanobberghen A, Raab M, et al. The ALMANACH Project: Preliminary results and potentiality from Afghanistan. International journal of medical informatics. 2018;114:130–5. pmid:29330009
  19. 19. Beynon F, Salzmann T, Faye PM, Thiongane A, Ndiaye O, Luwanda LB, et al. Paediatric digital clinical decision support for global health. Revue Medicale Suisse. 2023;19(836):1398–403.
  20. 20. Agarwal S, Glenton C, Tamrat T, Henschke N, Maayan N, Fønhus MS, et al. Decision-support tools via mobile devices to improve quality of care in primary healthcare settings. Cochrane Database Syst Rev. 2021;7(7):Cd012944. Epub 2021/07/28. pmid:34314020; PubMed Central PMCID: PMC8406991.
  21. 21. Rambaud-Althaus C, Shao AF, Kahama-Maro J, Genton B, d’Acremont V. Managing the Sick Child in the Era of Declining Malaria Transmission: Development of ALMANACH, an Electronic Algorithm for Appropriate Use of Antimicrobials. PLoS One. 2015;10(7):e0127674. Epub 2015/07/15. pmid:26161753; PubMed Central PMCID: PMC4498609.
  22. 22. Keitel K, Kagoro F, Samaka J, Masimba J, Said Z, Temba H, et al. A novel electronic algorithm using host biomarker point-of-care tests for the management of febrile illnesses in Tanzanian children (e-POCT): A randomized, controlled non-inferiority trial. PLoS medicine. 2017;14(10):e1002411. Epub 2017/10/24. pmid:29059253; PubMed Central PMCID: PMC5653205.
  23. 23. Tan R, Cobuccio L, Beynon F, Levine GA, Vaezipour N, Luwanda LB, et al. ePOCT+ and the medAL-suite: Development of an electronic clinical decision support algorithm and digital platform for pediatric outpatients in low- and middle-income countries. PLOS Digital Health. 2023;2(1):e0000170. pmid:36812607
  24. 24. Tan R, Kavishe G, Luwanda LB, Kulinkina AV, Renggli S, Mangu C, et al. A digital health algorithm to guide antibiotic prescription in pediatric outpatient care: a cluster randomized controlled trial. Nature Medicine. 2024;30(1):76–84. pmid:38110580
  25. 25. Shao AF, Rambaud-Althaus C, Samaka J, Faustine AF, Perri-Moore S, Swai N, et al. New Algorithm for Managing Childhood Illness Using Mobile Technology (ALMANACH): A Controlled Non-Inferiority Study on Clinical Outcome and Antibiotic Use in Tanzania. PLoS One. 2015;10(7):e0132316. Epub 2015/07/15. pmid:26161535; PubMed Central PMCID: PMC4498627.
  26. 26. Tanzania Commission for AIDS (TACAIDS), (ZAC) ZAC. Tanzania HIV Impact Survey (THIS) 2016–2017: Final Report. Dar es Salaam, Tanzania 2018.
  27. 27. Amu H, Dickson KS, Kumi-Kyereme A, Darteh EKM. Understanding variations in health insurance coverage in Ghana, Kenya, Nigeria, and Tanzania: Evidence from demographic and health surveys. PLoS One. 2018;13(8):e0201833. pmid:30080875
  28. 28. Mullan F, Frehywot S. Non-physician clinicians in 47 sub-Saharan African countries. The Lancet. 2007;370(9605):2158–63. pmid:17574662
  29. 29. Cobuccio LG, Faivre V, Tan R, Vonlanthen A, Beynon F, Barchichat E, et al. medAL-suite: a software solution for creating and deploying complex clinical decision support algorithms. https://zenodo.org/records/13889290
  30. 30. World Health Organization. Integrated Management of Childhood Ilness Chart Booklet. Geneva: 2014 2014. Report No.
  31. 31. World Health Organization. Pocket Book of Hospital Care for Children: Guidelines for the Management of Common Childhood Illnesses. 2nd ed. Geneva: World Health Organization; 2013.
  32. 32. Demographic and Health Suveys (DHS). Service Provision Assessments (SPA) 2024 [cited 2024 05.02.2024]. Available from: https://dhsprogram.com/Methodology/Survey-Types/SPA.cfm.
  33. 33. Gilroy K, Winch PJ, Diawara A, Swedberg E, Thiéro F, Kané M, et al. Impact of IMCI training and language used by provider on quality of counseling provided to parents of sick children in Bougouni District, Mali. Patient Education and Counseling. 2004;54(1):35–44. pmid:15210258
  34. 34. Rowe AK, Onikpo F, Lama M, Osterholt DM, Rowe SY, Deming MS. A multifaceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin. Am J Public Health. 2009;99(5):837–46. Epub 2009/03/19. pmid:19299681.
  35. 35. Lemiengre MB, Verbakel JY, Colman R, De Burghgraeve T, Buntinx F, Aertgeerts B, et al. Reducing inappropriate antibiotic prescribing for children in primary care: a cluster randomised controlled trial of two interventions. The British journal of general practice: the journal of the Royal College of General Practitioners. 2018;68(668):e204–e10. Epub 2018/02/12. pmid:29440016.
  36. 36. Flottorp S, Oxman AD, Håvelsrud K, Treweek S, Herrin J. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ. 2002;325(7360):367. pmid:12183309
  37. 37. Bernasconi A, Crabbé F, Raab M, Rossi R. Can the use of digital algorithms improve quality care? An example from Afghanistan. PLoS One. 2018;13(11):e0207233–e. pmid:30475833.
  38. 38. Thompson JA, Leyrat C, Fielding KL, Hayes RJ. Cluster randomised trials with a binary outcome and a small number of clusters: comparison of individual and cluster level analysis method. BMC Medical Research Methodology. 2022;22(1):1–15.
  39. 39. Hayes RJ, Moulton LH. Cluster randomised trials: CRC press; 2017.
  40. 40. Thompson JA, Leurent B, Nash S, Moulton LH, Hayes RJ. Cluster randomized controlled trial analysis at the cluster level: The clan command. The Stata Journal. 2023;23(3):754–73. pmid:37850046
  41. 41. StataCorp. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC.; 2019.
  42. 42. Horwood C, Haskins L, Mapumulo S, Connolly C, Luthuli S, Jensen C, et al. Electronic Integrated Management of Childhood Illness (eIMCI): a randomized controlled trial to evaluate an electronic clinical decision-making support system for management of sick children in primary health care facilities in South Africa. 2023.
  43. 43. Tan R, Kagoro F, Levine GA, Masimba J, Samaka J, Sangu W, et al. Clinical Outcome of Febrile Tanzanian Children with Severe Malnutrition Using Anthropometry in Comparison to Clinical Signs. American Journal of Tropical Medicine and Hygiene. 2020;102(2):427–35. WOS:000512881500035. pmid:31802732
  44. 44. Myatt M, Khara T, Collins S. A Review of Methods to Detect Cases of Severely Malnourished Children in the Community for Their Admission into Community-Based Therapeutic Care Programs. Food and Nutrition Bulletin. 2006;27(3_suppl3):S7–S23. pmid:17076211
  45. 45. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ: Canadian Medical Association journal = journal de l’Association medicale canadienne. 1995;153(10):1423–31. Epub 1995/11/15. pmid:7585368; PubMed Central PMCID: PMC1487455.
  46. 46. Odendaal WA, Anstey Watkins J, Leon N, Goudge J, Griffiths F, Tomlinson M, et al. Health workers’ perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2020;3(3):Cd011942. Epub 2020/03/28. pmid:32216074; PubMed Central PMCID: PMC7098082.
  47. 47. Kruse C, Betancourt J, Ortiz S, Valdes Luna SM, Bamrah IK, Segovia N. Barriers to the Use of Mobile Health in Improving Health Outcomes in Developing Countries: Systematic Review. J Med Internet Res. 2019;21(10):e13263. Epub 2019/10/09. pmid:31593543; PubMed Central PMCID: PMC6811771.
  48. 48. Renggli S, Mayumana I, Mboya D, Charles C, Maeda J, Mshana C, et al. Towards improved health service quality in Tanzania: An approach to increase efficiency and effectiveness of routine supportive supervision. PLoS One. 2018;13(9):e0202735. pmid:30192783
  49. 49. Murray CJL, Ikuta KS, Sharara F, Swetschinski L, Robles Aguilar G, Gray A, et al. Global burden of bacterial antimicrobial resistance in 2019: a systematic analysis. The Lancet. 2022. pmid:35065702