https://orcid.org/0000-0001-8427-2035
Figures
Abstract
Sepsis occurs predominantly in low-middle-income countries. Sub-optimal triage contributes to poor early case recognition and outcomes from sepsis. Improved recognition and quality of care can lead to improved outcomes. We evaluated the impact of Smart Triage using improved time to intravenous antimicrobial administration in a multisite interventional study. Smart Triage, a digital platform with a risk score and clinical dashboard, was implemented (with control sites) in Kenya (February 2021-December 2022) and Uganda (April 2020-April 2022). Children presenting to the outpatient departments with an acute illness were enrolled. A controlled interrupted time series was used to assess the effect on time from arrival at the facility to intravenous antimicrobial administration. Secondary analyses included antimicrobial use, admission rates and mortality (NCT04304235). During the baseline period, the time to antimicrobials decreased significantly in Kenya (132 and 58 minutes) at control and intervention sites. In Uganda, the time to antimicrobials marginally decreased (3 minutes) at the intervention site. Then, during the implementation period in Kenya, the time to antimicrobials at the intervention site decreased by 98 min (57%, 95% CI 81-114) but increased by 49 min (21%, 95% CI: 23-76) at the control site. In Uganda, the time to antimicrobials initially decreased but was not sustained and there was no significant difference between intervention and control sites. At both intervention sites, there was a significant reduction in antimicrobial utilization of 47% (Kenya) and 33% (Uganda) compared to baseline. There was a reduction in admission rates of 47% (Kenya) and 33% (Uganda) compared to baseline. Mortality reduced by 25% (Kenya) and 75% (Uganda) compared to the baseline period. We showed significant improvements in time to intravenous antibiotics in Kenya but not Uganda, likely due to COVID-19, a short study period and resource constraints. The reduced antimicrobial use and admission and mortality rates are remarkable and welcome benefits. The admission and mortality rates should be interpreted cautiously as these were secondary outcomes. This study underlines the difficulty of implementing technologies and sustaining quality improvement in health systems.
Author summary
Implementing the Smart Triage platform and quality improvement program for children in Kenya and Uganda resulted in inconsistent improvements in time to intravenous antimicrobial administration. The time to IVA decreased significantly in Kenya during baseline and reduced further during the intervention while increasing at the control site. In Uganda the time to treatment initially decreased but was not sustained. The treatment times were significantly influenced by the improvements during baseline data collection and multiple external health system factors such as drug shortages, the COVID -19 pandemic, staff shortages and strikes. The dramatic reduction in treatment, admission, and mortality rates should be further investigated.
Citation: Ansermino JM, Pillay Y, Tagoola A, Zhang C, Dunsmuir D, Kamau S, et al. (2025) Implementation of Smart Triage combined with a quality improvement program for children presenting to facilities in Kenya and Uganda: An interrupted time series analysis. PLOS Digit Health 4(3): e0000466. https://doi.org/10.1371/journal.pdig.0000466
Editor: Ziad El-Khatib, Karolinska Institutet, SWEDEN
Received: February 9, 2024; Accepted: January 8, 2025; Published: March 10, 2025
Copyright: © 2025 Ansermino et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Study materials (protocol, data collection tools, data dictionary, software, analysis code, and metadata) are publicly available through the Pediatric Sepsis Data CoLaboratory’s (Sepsis CoLab) Dataverse on https://borealisdata.ca/dataverse/ST_implementation. Due to the sensitive nature of clinical data and the potential risk for re-identification of research participants, the de-identified dataset is available through moderated access on the Sepsis CoLab Dataverse on https://borealisdata.ca/dataverse/ST_implementation and through the KWTRP Research Data Repository Dataverse https://kemri-wellcome.org/dataverse/. Access to these data will be granted on a case-by-case basis following approval from the authors and the Data Governance Committees.
Funding: The funding for this project was provided by a Wellcome Trust (https://wellcome.org/) Innovator Award (215695/Z/19/Z) to SA and MA as co-principal investigators in January 2020. The funders played no role in the study design, data collection, analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Background
Among critically ill children, sepsis is the leading cause of preventable deaths globally. Countries in Sub-Saharan Africa report disproportionately high case fatality rates [1,2]. Many of these deaths are from malaria, pneumonia and diarrheal diseases resulting in sepsis, which may respond to time-sensitive treatment [3]. Early treatments, usually within the first hour of presentation, improve survival rates [4,5]. However, delayed recognition and treatment within health facilities remain significant barriers to reducing sepsis-related deaths and complications [6].
Critically ill children can be rapidly identified using triage, which prioritizes patients and the provision of medical care according to illness severity [7]. The Emergency Triage Assessment and Treatment (ETAT) system [8,9], the Pediatric South African Triage Scale (PSATS) [10] and Integrated Management of Childhood Illnesses (IMCI) guidelines [11] were developed based on expert consensus to facilitate triage processes for low-resourced environments. Despite adoption and scaling in Lower-and-Middle-Income Countries (LMICs), sustained implementation into clinical practice at scale has been a persistent challenge, resulting in suboptimal impact [9,12,13]. Therefore, triage remains under-used in many LMIC settings and patients are still frequently seen on a first-come-first-served basis [14].
Recent efforts to improve triage and treatment decisions include clinical decision support tools implemented in digital platforms using guidelines based on expert opinion [15]. Digital tools can also provide automated guidance based on real-world data using an individualized precision public health approach [15] and data-driven feedback that can be used for quality improvement (QI) by health workers [16] and to address local implementation challenges [17]. Nevertheless, multifaceted system-wide interventions are still needed to effectively improve care practices in complex, low-resourced settings [3].
We have developed, validated, implemented, and evaluated a digital triage platform called Smart Triage [18–22]. The Smart Triage platform comprises data-driven risk assessment (triage), patient and treatment tracking, a real-time dashboard, and automated reports to improve emergency case recognition and time-to-treatment. The Smart Triage algorithm predicts admission, as a data-driven surrogate for illness severity, and is combined with emergency and priority signs as guardrails to generate a triage tool suitable for all acutely ill patients.
QI studies evaluating digital triage tools in low-income countries, under real-world conditions, are scarce [13]. We report on the findings of a study investigating the Smart Triage platform implementation at pediatric outpatient departments (OPD) of four tertiary hospitals (two hospitals in Uganda and two hospitals in Kenya). We hypothesized that the Smart Triage platform would reduce the time to treatment. The secondary aims were to evaluate the effect of Smart Triage and QI on patient outcomes, including mortality, admission and readmission.
Methods
We use an interrupted time series analysis to assess changes in the time from arrival at the facility to administration of an appropriate sepsis bundle of care following implementation of the Smart Triage platform.
Study sites
This controlled interrupted time series study was conducted in the OPDs of four public tertiary facilities in Kenya and Uganda. Facilities included one control and one intervention site per country. In Kenya, the facilities were Mbagathi County Hospital (intervention site) and Kiambu County Referral Hospital (control site). In Uganda the facilities were Jinja Regional Referral Hospital (intervention site) and Gulu Regional Referral Hospital (control site).
Contextual and temporal factors
COVID-19-related lockdowns restricting gatherings and travel between districts were enforced from 15 April-5 May 2020 and 7 June-19 July 2021 in Uganda. Outside of these dates, a strict curfew and other infection prevention measures were observed from April 2020 to December 2021. The study was initiated in Kenya after in-country COVID-19-related restrictions had ended. In addition, there were healthcare worker shortages due to strikes (nurses in Kenya and junior doctors in Uganda). Planned staff rotations also significantly impacted compliance with triage and QI. The control site in Uganda was added a year later than the intervention site in Uganda due to delays in study initiation at the control site in Kenya, which was initially planned as the sole control site.
Participant recruitment
All pediatric arrivals presenting with an acute illness to the OPD were eligible. The age limits were in keeping with each hospital’s practice for pediatric admissions (Mbagathi 0-15 years, Kiambu 0-15 years, Jinja 0-19 years and Gulu 0-12 years). Informed consent was obtained from parents or legal guardians. Assent was obtained from children over 8 years (Uganda) and 13 years (Kenya). We excluded trauma, burns, elective cases (surgery, change of dressing), immunization visits or clinical review or follow-up appointments. Patients with trauma or burns were classified as emergency during implementation.
Standard procedures
The implementation design and procedural information have previously been published [20]. In brief, a preintervention phase (baseline), interphase (model and technology development) and intervention phase (implementation) were used. The implementation is described in this report. We used a systematic method of sampling. Following consent, predefined candidate predictor variables and outcomes were collected on patients waiting to be seen in the OPD by trained study nurses using Android tablets on a custom-built mobile application [18,20]. Dedicated timekeepers, with at least a high school level education, logged the time of each IVA treatment manually in the treatment room. The implementation and control sites continued data collection for the entire study duration. Oxygen saturation (SpO2) and heart rate were measured using the Masimo iSpO2 (Masimo Corporation, California, USA). Respiratory rate was measured using the RRate Application directly on the tablet [23,24]. The Welch Allen SureTemp 692 thermometer was used to measure temperature (Welch Allyn, New York, USA).
Smart triage platform
Using the baseline data at intervention sites, we developed and recalibrated a nine-variable risk prediction model [19,25] and deployed this within a mobile application for triage. The triage algorithm included independent triggers based on the ETAT triage guidelines [26]. The independent triggers ensure that no danger or priority sign or single significantly abnormal vital sign is missed. The Smart Triage platform comprises data-driven risk assessment (triage), patient and treatment tracking using a Bluetooth Low Energy (BLE) tracking system called Smart Spot, a real-time dashboard, and automated reports. The Smart Spot system uses a BLE tag (Fig 1) attached to a color-coded lanyard around the caregiver’s neck. A BLE reader detects the signal strength from each tag. Signal strengths that exceed the reader’s threshold are considered to be in that room (thresholds differ based on room size). Readers were strategically located in the hospital to provide the patient’s location as they move through the facility. Treatment tags in the treatment or emergency room were used to track the time and type of treatment given. All the information collected at triage and via the Smart Spot system was accessible through the clinical dashboard. The dashboard was available to clinicians, laboratory, pharmacy, nutrition team and administrators on laptops, tablets or computer screens. A large screen with a public-facing dashboard was displayed in the waiting areas. The public-facing dashboard displayed coded patient queues, health information posters and videos targeting caregivers. The information was intended to improve understanding of the triage process, Smart Spot system and general health information.
The feasibility and usability of the platform were assessed through questionnaires, direct observation and usability scenarios using a think-aloud methodology [20]. Qualitative interviews were used to optimize the platform [21]. A cost-effectiveness study was completed to determine utilization costs and implications for scale-up [22].
Implementation
Hospital triage staff routinely utilized the digital triage platform during the intervention phase. Following triage with Smart Triage algorithm, families were given colour-coded lanyards, which corresponded to the predicted level of risk, with an attached Smart Spot BLE tag. Red lanyards corresponded to emergency cases, yellow to priority and green to non-urgent cases.
The Smart Triage platform was implemented as part of the overall QI program that focused on delivering a sepsis bundle of care. At implementation sites, healthcare workers were trained on the importance of early sepsis recognition and treatment, in using the platform and in performing QI using a train-the-trainer model. Weekly feedback reports were provided using data collected by the Smart Triage platform. The reports included information on the number of children triaged, triage categories, duration of triage, treatment times and metrics of specific interest to the facility. The metrics and report layout were developed collaboratively with facility leadership and implementors to improve data-driven QI processes at the facility. Ongoing training, support supervision, job aides and manuals were also available to health workers.
QI initiatives were aimed at optimal triage and time to antimicrobial administration and were led and customized by each implementation facility. Triage QI initiatives included reducing the time from arrival to triage, improving the triage completion rate, increasing completeness of vital sign measurement, and creating a daily updated list of drug and supply stock-outs to improve communication between departments and caregivers. The causes of delayed time to treatment were identified by using a fishbone diagram. The workflow was optimized at both sites with changes in patient queuing at admission, assessment, laboratory, pharmacy and the emergency room. Specific benches were allocated for emergency cases. An emergency pharmacy cabinet with after-hours access was installed in Uganda to reduce the impact of delays in supplying drugs from the pharmacy. Stock-outs and staff shortages were the most significant barriers identified at both sites (S1 Table).
Follow-up
In-hospital and post-discharge mortality and readmission data were collected during a phone call at seven-days post discharge (or post-visit for those who were not admitted).
Outcome measures
The primary outcome was the time to IVA from the time of OPD arrival, among those treated with IVA. The initially planned primary outcome was the time to receive an appropriate sepsis bundle of care (antibiotics, antimalarials, oxygen or fluids if clinically indicated). However, it became evident that IVA was most robustly collected by the timekeepers and most commonly occurred after the other elements of the bundle.
Secondary outcomes included treatment rates, the admission rate, readmission rate (both among those admitted and among those not initially admitted), length of stay among admissions, and overall mortality at seven days. In addition, the time to IVA in each triage category and the proportion of triaged patients was calculated.
Data management
All data was captured digitally, either on a tablet or through the study dashboard. At the end of each day, data was uploaded directly to REDCap, on a Kenya Medical Research Institute (KEMRI) server in Kenya and on the central study server at the BC Children’s Hospital Research Institute for Uganda.
Sample size
Based on the feasibility study [22], the increase in the proportion of children receiving a bundle of care within one hour in this study was 21.4%, and the proportion of children receiving IVA was 8.2%. Based on these observations, we estimated that using an alpha of .05, at a power of 80%, we would require 10 600 triages (1290 treatments) in the pre-intervention and post-intervention phases combined, to confirm this effect observed in the previous study.
Analysis
We used Mann-Whitney and Kruskall-Wallis (for continuous variables) and Fischer’s exact test (for categorical variables) to compare patient characteristics for patients who received IVA and outcomes between control and intervention sites at baseline and following implementation. A p-value of <0.001 was considered statistically significant.
We used a quasi-experimental interrupted time series to assess the immediate effects and effects over time of the intervention but excluding the interphase period. The post-implementation period was set to occur the day following the baseline period. Quantile regression was used to estimate the change in median time to IVA (in minutes) per increase in days since enrollment. The independent variable included each individual IVA administration while a weekly median IVA was derived to visually summarize data. The regression error terms were fitted using an Ordinary Least Square method. Separate regression analysis was done for each of the control and intervention sites. We estimated secular trends for the pre- and post- implementation period and compared the post-implementation period with the counterfactual trend to determine immediate and longer-term effect of the implementation using slope and level changes, respectively. An interaction term with sites was fitted to estimate the difference in median time to IVA between control and intervention sites.
A univariate logistic regression was fitted to obtain the odds ratio of admission, readmission, mortality, and receiving IVA between sites at baseline and following implementation as well as within sites at the intervention sites for the full cohort and separately by triage categories. Confidence intervals were calculated using maximum likelihood.
Ethical considerations
Ethics approval was obtained from Makerere University School of Public Health (MUSPH) Institutional Review Board (IRB00011353). The Ugandan National Council for Science and Technology (UNCST) (HS528ES) in Uganda provided approval for study activities to be conducted at Jinja Regional Referral Hospital and Gulu Regional Referral Hospital. The KEMRI Scientific and Ethics Review Unit (SERU) (KEMRI/SERU/CGMR-C/183/3958) provided approval for the study activities to be conducted at Mbagathi County Hospital and Kiambu County Hospital in Kenya. The study was also approved by the University of British Columbia Research Ethics Board in Canada (H19-02398-A006). The trial was registered on Clinical Trials.gov. Identifier: NCT04304235, Registered 11 March 2020.
Results
There were 6784 children enrolled at baseline from three sites and 5410 at two intervention sites and 6032 at two control sites during the implementation phase (Fig 2). There were a few negative times as patients were identified as emergency cases and treated as soon as they arrived before triage. In such cases, negative times were set to zero, and times greater than 8 hours were censored at 8 hours. Triage was completed on 5331 (98.5%) children enrolled at the intervention sites during the implementation period, which formed our analysis cohort. The median (IQR) time for fluids and/or oxygen to be initiated was 70 (32-137) min compared to IVA at 210 (130-304) min at all sites during both phases (S2 and S3 Tables). In addition, only 85 (13.6%) children received fluids or oxygen after IVA. Time to IVA among children who went on to receive IVA was thus used as the study endpoint for all analyses based on these observations.
Control site was added later than intervention site in Kenya due to site initiation delays in intervention site and control site in Uganda resulting from the COVID-19 pandemic.
In Kenya, the baseline time to IVA decreased by 133 min (44%, 95% CI: 93 to 171) at the control site and by 57 min (37%, 95% CI: 38 to 77) at the intervention site (Fig 3). Paradoxically, there was a 74-min (76%; 95% CI: 54-94) increase in time to IVA following the 2-week interphase period at the Kenya intervention site. During the implementation phase, the time to IVA decreased by 98 min (57%, 81-114) at the Kenya intervention site, while during the same period it increased by 50 min (21%, 95% CI: 23-76) at the control site (Table 1).
The vertical dotted black line indicates Smart Triage implementation. Secular trends are indicated by solid lines and counterfactual trends are indicated by dotted colored lines. Weekly median time to IVA is summarized by “+”.
In Uganda, the baseline time to IVA decreased by 3 min (2%, 95% CI: -12 to 5) at the intervention site (Fig 4). During the intervention, the time to IVA initially decreased by 19 min (8%, 95% CI: 19-21) compared to the end of baseline, but this was not sustained. The median time to IVA during the baseline period was 16 min (6%, 95% CI: 6 to 40) more than the implementation phase. Time to IVA at the Uganda control site decreased by 13 min (6%, 95% CI: 6 to 20) during the implementation phase, but there was no statistically significant difference between intervention and control sites (Table 2). We have no baseline data from the Uganda control site, so we could not evaluate the trend over time for the entire study period.
The vertical dotted black line indicates Smart Triage implementation. Secular trends are indicated by solid lines and counterfactual trends are indicated by dotted colored lines. Weekly median time to IVA is summarized by “+”. Due to delays related to COVID-19, the Ugandan control site was initiated during the implementation phase at the intervention site.
In Kenya, there was a reduction in admission rates between the baseline and intervention of 47% (OR: 0.51, 95% CI: 0.42-0.62, p<0.0001) (Table 3). Admission rates in the control site were unchanged (4.7%). IVA use was also reduced at the intervention site compared to the baseline by 47% (OR: 0.51 CI: 0.41-0.63) (Table 3). IVA use in the Kenya control site was also unchanged (2.9%). Mortality within 7 days was also reduced by 25% (OR: 0.29, 95% CI: 0.14-0.59) compared to the baseline period.
In Uganda, the reduction in admission rate between baseline and intervention was 34% (OR: 0.59, 95% CI 0.50-0.71) (Table 2). IVA use was reduced by 33% (OR: 0.62 CI: 0.52-0.75) (Table 2). Mortality within 7 days was reduced by 75% (OR: 0.27, 95% CI: 0.08-0.84) at the intervention sites compared to the baseline period.
In addition to the overall reduction in admission rates, there was a reduction in priority admission rates of 64% (OR: 0.35, 95% CI: 0.22-0.55) in Kenya (S4 Table) and 32% (OR: 0.62, 95% CI: 0.47-0.83) in Uganda (S5 Table). There was a 19% (OR: 0.79 95% CI: 0.54-1.17) reduction in the admission rates of priority cases at the Kenya control site compared to the baseline. Emergency cases who presented to the Kenya intervention site showed a 161 min (62%, 95% CI: 113 to 209) reduction in time to IVA when compared to the Kenya control site (S4 Table). In Uganda, the reduction was 42 min (17%; 95% CI: 13 to 71) compared to the baseline period (S5 Table).
Discussion
We report the results of a controlled interrupted time series analysis implementing the Smart Triage platform to improve the time to treatment. We found an inconsistent reduction in the time to IVA. The time to IVA at the start of baseline was much lower in Kenya than in Uganda, with a much larger reduction in time to IVA during the baseline compared to the intervention and control sites in Uganda. In Kenya, we observed an immediate increase in time to IVA at both our intervention and control sites between the end of the baseline and the start of the implementation phase. This effect could be explained by the timing of the Smart Triage implementation, which started during the December holiday period when staff shortages was common and there was staff changeover. In contrast, the sustained improvement observed at the intervention site versus the control site supports implementation success at this site. In Uganda at the start of the intervention, the control site had a lower time to IVA than the intervention site and continued to improve over time while the intervention site worsened. The comparison between sites is compromised by delayed initiation of the control site due to the COVID-19 pandemic, the lack of simultaneous baseline data and the differences between two facilities in different countries with different confounders [27].
The consistent reduction of time to IVA administration at all sites during the baseline period would indicate a system change associated with baseline data collection. A similar contamination has been observed in cluster randomized trials of digital health interventions that mask the impact of the intervention [28,29]. The downward slope in time to IVA during the baseline period would also suggest that this effect had not reached steady state.
What was surprising in our study was the decrease in treatment and admission rates in both Uganda and Kenya. This finding may be due to adherence to triage criteria in almost all patients and less reliance on personal preferences among clinicians. However, without more robust long-term follow-up (>7 days) of all children after discharge we cannot be certain that the lower admission rates reflect better triage practices. However, we did not see an increase in antimicrobial utilization or readmission rates. A similar reduction in antimicrobial utilization has been shown in a randomized trial using an electronic algorithm-guided management platform [30]. These findings warrant additional investigation into the impact on longer-term mortality following triage as post-discharge mortality is common and may also be more prevalent in patients who were not admitted [31].
The finding of a decrease in mortality by 25% (OR: 0.29, 95% CI:0.14-0.59) in Kenya and by 75% (OR: 0.27, 95% CI: 0.08-0.84) in Uganda compared to the baseline period is similar to an effect that has been reported in the US in a large 19 hospital American Academy of Paediatrics Paediatric Severe Sepsis Collaborative that were focused on sepsis QI in emergency department care [32]. In these collaboratives, there were shared criteria for sepsis, standardized screening tools and electronic health record (EHR) embedded tools (such as triage screens and sepsis order sets). Improvements were noted in time to first clinical assessment, fluid bolus and antibiotics, with an associated improvement in 30-day all-cause mortality from 2.3% to 1.4%. Interestingly, even sites newer to improvement work demonstrated process metric improvement as well as mortality reductions, corroborating the benefits of an ‘all teach - all learn’ QI collaborative methodology [32,33].
Our findings of inconsistent improvement are not surprising because the uptake and effectiveness of clinical decision support tools have had mixed success due to complexities in implementation contexts [15,34]. Beyond the setting, technology-supported innovations face additional complexity in implementation that stems from each aspect of the innovation. These complexities include addressing the clinical condition (especially when poorly characterized like sepsis), interacting technology components, and obtaining organizational, user and patient buy-in [35]. These factors are amplified in poorly resourced settings which are plagued by overcrowding and understaffing in emergency departments, inadequate resources for timely laboratory testing and imaging, inadequate stock of antibiotics, and inadequate overall resources for paediatric sepsis management [16,36,37]. Thus, to be effective, digital interventions addressing triage and QI must consider individual and systems-related factors and processes that influence facility performance [38–40].
Sepsis care in children is complex. Consideration must be given to biology of both pathogens (malaria, dengue, bacteria and viruses) and the patient including age and genetics; along with seasonality of infections, resource availability (personnel, drugs, diagnostic tools) and access, timing of presentation and progress of deterioration and co-morbidities such as malnutrition and HIV [41]. Care occurs in facilities which are complex, dynamic environments continuously influenced by process inputs, organizational structure and resources, culture and individual motivations [42]. In this milieu of constant shifts and changes it is not surprising that QI is difficult to initiate and sustain. An assumption of the interrupted time series methodology is that the baseline trend is not changing during the intervention [43]. By selecting a process indicator and prioritizing randomization and prospective enrollment on arrival at the facility; we hoped to account for confounding related to variable case mix and severity, however this may not be the case.
The time to IVA was chosen because it is the most important factor in improving outcomes from sepsis and easy to track and compare between hospitals [44]. However, IVA is vulnerable to changes in resource availability [45,46]. For example, using the data-driven QI process, we were able to identify that delivery of timely treatment was influenced not just by identification of emergency cases, but also staff availability, drug and reagent shortages, and lab equipment for diagnostics and caregivers’ ability to pay for the drug and other sundries. These issues were exacerbated by the COVID-19 pandemic and persisted throughout the study period. The best chance for success, therefore, relies on a holistic approach which not only addresses the clinical elements of the treatment guideline but also include supply chain management, maintenance policies and coevolution of health systems infrastructure with pediatric triage and sepsis guidelines [27,47,48].
Strengths and limitations
The major limitation of this study is that the methodology chosen was unable to account for the significant confounders during data collection such as the COVID-19 pandemic and significant disruptions in supplies and staffing. We were also limited by the lack of synchronous data during the baseline in Uganda. We underestimated the impact of data collection during the baseline on time to IVA and would have preferred to have a longer period of observation to allow for traction in QI initiatives and account for seasonality. Using timekeepers to track treatment times manually at the point of care may have influenced clinical staff behaviour. However, this was consistent between baseline, control and intervention sites.
The strengths of the study are completeness of data, and the adjustment of the intervention using small iterative changes (responsive feedback), co-developed with the QI teams on site. We actively addressed issues related to usability, training and staff rotations through continued training, resources, and job aides. Sustainability and continued staff-led problem solving is supported through an ongoing QI program and partnerships with implementors.
Conclusion
Smart Triage was successfully implemented for routine use in two resource-constrained settings. The improvement in time to IVA was inconsistent between sites and impacted by numerous health system factors. However, the reduction in admission and treatment times and the reduction in mortality are benefits worthy of further investigations. We have also highlighted the significant challenges in undertaking clinical evaluation of digital health tools in complex real-world clinical settings. The results can be leveraged to understand delivery gaps, strengthen implementation strategy and methodology and inform future adaptions of the Smart Triage platform.
Supporting information
S1 Table. Delayed treatments and reasons for delays.
https://doi.org/10.1371/journal.pdig.0000466.s001
(PDF)
S2 Table. Characteristics of patients who received IV antimicrobials in the study.
https://doi.org/10.1371/journal.pdig.0000466.s002
(PDF)
S3 Table. Time to the components of sepsis bundle.
https://doi.org/10.1371/journal.pdig.0000466.s003
(PDF)
S4 Table. Kenya: Time to intravenous antimicrobial (IVA) based on of triage categories, sites and periods.
https://doi.org/10.1371/journal.pdig.0000466.s004
(PDF)
S5 Table. Uganda: Time to intravenous antimicrobial (IVA) based on triage categories, sites and periods.
https://doi.org/10.1371/journal.pdig.0000466.s005
(PDF)
Acknowledgments
We are grateful for the quality improvement efforts of facility staff and leadership at Jinja Regional Referral Hospital and Mbagathi County Hospital. Further, we would like to thank members of the Smart Triage research team based at Walimu, Uganda; Kenya Medical Research Institute, Kenya; and the Institute for Global Health, Canada. This includes but is not limited to, Savio Mwaka, Clare Komugisha, Bamwesigye Emmanuel, Annet Mary Nabweteme, Busense Bosco, Kisaame Meshack Moshin, Kantono Miria, Nakasagga Barbra, Monero Angel, Isaac Omara, Komakech Francis, Patricia Aloya, Joan Lamagi, Priscilla Antimango, Reagan Obalim, Angela Wanjiru, Anastasia Gathigia, Felix Kimani, Mercy Mutuku, Emmah Kinyanjui, Esther Muthoni, Faith Wairimu, Sidney Kipkorir, Verah Karasi, Victor Achiro, Kevin Bosek, Brian Ochieng, John Mboya, Deborah Lester, Jessica Rigg, Katija Pallot, Parnian Hosseini, Alishah Mawji and Edmond C K Li.
References
- 1. Liu L, Oza S, Hogan D, Chu Y, Perin J, Zhu J, et al. Global, regional, and national causes of under-5 mortality in 2000-15: an updated systematic analysis with implications for the Sustainable Development Goals. Lancet. 2016;388(10063):3027–35. pmid:27839855
- 2. Kissoon N, Reinhart K, Daniels R, Machado M, Schachter R, Finfer S. Sepsis in children: global implications of the World Health Assembly resolution on sepsis. Pediatric Critical Care Medicine. 2017;18(12):e625–7.
- 3. Barasa EW, Ayieko P, Cleary S, English M. A multifaceted intervention to improve the quality of care of children in district hospitals in Kenya: a cost-effectiveness analysis. PLoS Med. 2012;9(6):e1001238. pmid:22719233
- 4. Mittal Y, Sankar J, Dhochak N, Gupta S, Lodha R, Kabra SK. Decreasing the Time to Administration of First Dose of Antibiotics in Children With Severe Sepsis. J Healthc Qual. 2019;41(1):32–8. pmid:30247207
- 5. Koenig C, Schneider C, Morgan JE, Ammann RA, Sung L, Phillips B. Association of time to antibiotics and clinical outcomes in patients with fever and neutropenia during chemotherapy for cancer: a systematic review. Support Care Cancer. 2020;28(3):1369–83. pmid:31264188
- 6. Weiss SL, Peters MJ, Alhazzani W, Agus MSD, Flori HR, Inwald DP, et al. Surviving Sepsis Campaign International Guidelines for the Management of Septic Shock and Sepsis-Associated Organ Dysfunction in Children. Pediatr Crit Care Med. 2020;21(2):e52–106. pmid:32032273
- 7. Moorthy GS, Pung JS, Subramanian N, Theiling BJ, Sterrett EC. Causal Association of Physician-in-Triage with Improved Pediatric Sepsis Care: A Single-Center, Emergency Department Experience. Pediatr Qual Saf. 2023;8(3):e651. pmid:37250616
- 8. Buys H, Muloiwa R, Westwood C, Richardson D, Cheema B, Westwood A. An adapted triage tool (ETAT) at Red Cross War Memorial Children’s Hospital Medical Emergency Unit, Cape Town: an evaluation. S Afr Med J. 2012;103(3):161–5. pmid:23472679
- 9.
World Health O. Emergency triage assessment and treatment (ETAT). Geneva: World Health Organization; 2005.
- 10. Rominski S, Bell SA, Oduro G, Ampong P, Oteng R, Donkor P. The implementation of the South African Triage Score (SATS) in an urban teaching hospital, Ghana. Afr J Emerg Med. 2014;4(2):71–5. pmid:28344927
- 11. Bryce J, Victora C, Habicht J-P, Black R, Scherpbier R. Programmatic pathways to child survival: results of a multi-country evaluation of Integrated Management of Childhood Illness. Health Policy and Planning. 2005;20(suppl_1):i5–17.
- 12. Hategeka C, Mwai L, Tuyisenge L. Implementing the Emergency Triage, Assessment and Treatment plus admission care (ETAT+) clinical practice guidelines to improve quality of hospital care in Rwandan district hospitals: healthcare workers’ perspectives on relevance and challenges. BMC Health Serv Res. 2017;17(1):1.
- 13. Hansoti B, Jenson A, Keefe D, De Ramirez SS, Anest T, Twomey M, et al. Reliability and validity of pediatric triage tools evaluated in Low resource settings: a systematic review. BMC Pediatr. 2017;17(1):37. pmid:28122537
- 14.
Schultz M, Dünser M, Dondorp A, Adhikari N, Iyer S, Kwizera A, et al. Current challenges in the management of sepsis in ICUs in resource-poor settings and suggestions for the future. Sepsis management in resource-limited settings. 2019:1–24.
- 15. Beynon F, Guérin F, Lampariello R, Schmitz T, Tan R, Ratanaprayul N. Digitalizing clinical guidelines: experiences in the development of clinical decision support algorithms for management of childhood illness in resource-constrained settings. Global Health: Science and Practice. 2023.
- 16. Melendez E, Bachur R. Quality improvement in pediatric sepsis. Curr Opin Pediatr. 2015;27(3):298–302. pmid:25944306
- 17. Clark M, Spry E, Daoh K, Baion D, Skordis-Worrall J. Reductions in inpatient mortality following interventions to improve emergency hospital care in Freetown, Sierra Leone. PLoS One. 2012;7(9):e41458. pmid:23028427
- 18. Mawji A, Li E, Chandna A, Kortz T, Akech S, Wiens MO, et al. Common data elements for predictors of pediatric sepsis: A framework to standardize data collection. PLoS One. 2021;16(6):e0253051. pmid:34111209
- 19. Mawji A, Li E, Dunsmuir D, Komugisha C, Novakowski SK, Wiens MO, et al. Smart triage: Development of a rapid pediatric triage algorithm for use in low-and-middle income countries. Front Pediatr. 2022;10:976870. pmid:36483471
- 20. Mawji A, Li E, Komugisha C, Akech S, Dunsmuir D, Wiens MO, et al. Smart triage: triage and management of sepsis in children using the point-of-care Pediatric Rapid Sepsis Trigger (PRST) tool. BMC Health Serv Res. 2020;20(1):493. pmid:32493319
- 21. Novakowski SK, Kabajaasi O, Kinshella M-LW, Pillay Y, Johnson T, Dunsmuir D, et al. Health worker perspectives of Smart Triage, a digital triaging platform for quality improvement at a referral hospital in Uganda: a qualitative analysis. BMC Pediatr. 2022;22(1):593. pmid:36229790
- 22. Li ECK, Tagoola A, Komugisha C, Nabweteme AM, Pillay Y, Ansermino JM, et al. Cost-effectiveness analysis of Smart Triage, a data-driven pediatric sepsis triage platform in Eastern Uganda. BMC Health Serv Res. 2023;23(1):932. pmid:37653477
- 23. Karlen W, Gan H, Chiu M, Dunsmuir D, Zhou G, Dumont GA, et al. Improving the accuracy and efficiency of respiratory rate measurements in children using mobile devices. PLoS One. 2014;9(6):e99266. pmid:24919062
- 24. Gan H, Karlen W, Dunsmuir D, Zhou G, Chiu M, Dumont GA, et al. The Performance of a Mobile Phone Respiratory Rate Counter Compared to the WHO ARI Timer. J Healthc Eng. 2015;6(4):691–703. pmid:27010948
- 25. Kigo J, Kamau S, Mawji A, Mwaniki P, Dunsmuir D, Pillay Y, et al. External validation of a paediatric Smart triage model for use in resource limited facilities. PLOS Digit Health. 2024;3(6):e0000293. pmid:38905166
- 26. Kamau S, Kigo J, Mwaniki P, Dunsmuir D, Pillay Y, Zhang C, et al. Comparison between the Smart Triage model and the Emergency Triage Assessment and Treatment guidelines in triaging children presenting to the emergency departments of two public hospitals in Kenya. PLOS Digit Health. 2024;3(8):e0000408. pmid:39088404
- 27. English M. Improving emergency and admission care in low-resource, high mortality hospital settings-not as easy as A, B and C. Health Policy Plan. 2022;37(6):808–10. pmid:34673985
- 28. Payne B, Kinshella MLW, Bawani S, Sheikh S, Hoodbhoy Z, Charanthimath U, et al. 216. Evaluation of the PIERS on the MOVE mobile health tool for pre-eclampsia triage: The users’ perspective. Pregnancy Hypertension. 2018;13:S101.
- 29. Payne B, Dunsmuir D, Qureshi R, Sevene E, Munguambe K, Goudar S, et al. 218. Implementation of the PIERS on the Move mobile health app in the Community Level Interventions for Pre-eclampsia trials. Pregnancy Hypertension. 2018;13:S101–2.
- 30. Tan R, Kavishe G, Luwanda LB, Kulinkina AV, Renggli S, Mangu C, et al. A digital health algorithm to guide antibiotic prescription in pediatric outpatient care: a cluster randomized controlled trial. Nat Med. 2024;30(1):76–84. pmid:38110580
- 31. Chaudhry M, Knappett M, Nguyen V, Trawin J, Mugisha NK, Kabakyenga J, et al. Pediatric post-discharge mortality in resource-poor countries: A protocol for an updated systematic review and meta-analysis. PLoS One. 2023;18(2):e0281732. pmid:36827241
- 32. Depinet H, Macias CG, Balamuth F, Lane RD, Luria J, Melendez E, et al. Pediatric Septic Shock Collaborative Improves Emergency Department Sepsis Care in Children. Pediatrics. 2022;149(3):e2020007369. pmid:35229124
- 33. Britto MT, Fuller SC, Kaplan HC, Kotagal U, Lannon C, Margolis PA, et al. Using a network organisational architecture to support the development of Learning Healthcare Systems. BMJ Qual Saf. 2018;27(11):937–46. pmid:29438072
- 34. Keitel K, D’Acremont V. Electronic clinical decision algorithms for the integrated primary care management of febrile children in low-resource settings: review of existing tools. Clin Microbiol Infect. 2018;24(8):845–55.
- 35. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, et al. Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework. BMC Med. 2018;16(1):66. pmid:29754584
- 36. Cruz AT, Lane RD, Balamuth F, Aronson PL, Ashby DW, Neuman MI, et al. Updates on pediatric sepsis. J Am Coll Emerg Physicians Open. 2020;1(5):981–93. pmid:33145549
- 37. Souza DC, Jaramillo-Bustamante JC, Céspedes-Lesczinsky M, Quintero EMC, Jimenez HJ, Jabornisky R, et al. Challenges and health-care priorities for reducing the burden of paediatric sepsis in Latin America: a call to action. Lancet Child Adolesc Health. 2022;6(2):129–36. pmid:34902315
- 38. Odetola FO, Freed G, Shevrin C, Madden B, McCormick J, Dombkowski K. In-Hospital Quality-of-Care Measures for Pediatric Sepsis Syndrome. Pediatrics. 2017;140(2):e20170350. pmid:28739652
- 39. Schlapbach LJ, Kissoon N. Defining Pediatric Sepsis. JAMA Pediatr. 2018;172(4):312–4. pmid:29459982
- 40. Dalwai M, Tayler-Smith K, Twomey M, Wallis L. Developing a reference standard for assessing paediatric triage scales in resource poor settings. African J Emergency Medicine. 2015;5(4):181–4.
- 41. Carrol ED, Ranjit S, Menon K, Bennett TD, Sanchez-Pinto LN, Zimmerman JJ, et al. Operationalizing Appropriate Sepsis Definitions in Children Worldwide: Considerations for the Pediatric Sepsis Definition Taskforce. Pediatr Crit Care Med. 2023;24(6):e263–71. pmid:37097029
- 42. English M, Irimu G, Wamae A, Were F, Wasunna A, Fegan G, et al. Health systems research in a low-income country: easier said than done. Arch Dis Child. 2008;93(6):540–4. pmid:18495913
- 43. Hategeka C, Ruton H, Karamouzian M, Lynd LD, Law MR. Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review. BMJ Glob Health. 2020;5(10):e003567. pmid:33055094
- 44. Evans IVR, Phillips GS, Alpern ER, Angus DC, Friedrich ME, Kissoon N, et al. Association Between the New York Sepsis Care Mandate and In-Hospital Mortality for Pediatric Sepsis. JAMA. 2018;320(4):358–67. pmid:30043064
- 45. English M, Ntoburi S, Wagai J, Mbindyo P, Opiyo N, Ayieko P, et al. An intervention to improve paediatric and newborn care in Kenyan district hospitals: understanding the context. Implement Sci. 2009;4:42. pmid:19627588
- 46. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. pmid:25791983
- 47. King C, Dube A, Zadutsa B, Banda L, Langton J, Desmond N, et al. Paediatric Emergency Triage, Assessment and Treatment (ETAT) - preparedness for implementation at primary care facilities in Malawi. Glob Health Action. 2021;14(1):1989807. pmid:34779363
- 48. Bryce J, Victora CG, Habicht J-P, Vaughan JP, Black RE. The multi-country evaluation of the integrated management of childhood illness strategy: lessons for the evaluation of public health interventions. Am J Public Health. 2004;94(3):406–15. pmid:14998804