Study design and setting

The study was conducted in the outpatient department (OPD) of GRRH over a two-year period, from June 23, 2022, to June 30, 2024. GRRH is a public hospital funded by the Uganda Ministry of Health and its OPD serves as a primary healthcare provider for over 23,000 pediatric and adult patients annually, offering free services to residents of five districts in Northern Uganda: Amuru, Gulu, Kitgum, Lamwo, and Pader.

The follow-up intervention augments the Smart Triage platform, a pediatric digital triage system that uses a data-driven risk prediction model and treatment tracking system for children under 18 years old [2–6]. Developed over the past decade, Smart Triage aims to collaboratively develop and implement clinically validated tools to allocate resources more efficiently in low-resource contexts and improve outcomes for children with an acute illness upon arrival to outpatient (emergency) departments. Initially, all caregivers were enrolled during triage for telephone follow-up with a study nurse seven days after discharge. The follow-up phone calls were supplemented with the automated two-way text messaging system in June 2022.

The intervention

QI to increase the message response rate.

The baseline response rate was established during the historical period (June 2022–March 2023), prior to the implementation of any QI changes. Beginning in April 2023, multiple iterative strategies were implemented on a monthly basis to improve response rates and refine the messaging process (Fig 1 and S1 Table). Response rates were monitored over time to assess the impact of each intervention, with adjustments made based on feedback and evolving needs to optimize follow-up engagement. For the purpose of analysis, the study period was divided into three phases: a historical period (June 2022–March 2023), a QI intervention period (April–November 2023), and a post-QI period (December 2023–June 2024). The QI period corresponded to the time during which iterative improvements to the follow-up system were actively implemented. The post-QI period was defined to assess whether improvements in response rates were sustained after the conclusion of active QI activities.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Response rate change by month.

https://doi.org/10.1371/journal.pone.0322969.g001

Data collection

Responses to the automated messages were automatically recorded in REDCap in real-time, ensuring accurate and timely data entry. If a caregiver responded with “1” (improved) the case was marked as complete and the caregiver was sent an acknowledgment. Responses of “2” (not improved) and non-responses were tracked in a REDCap report, which was monitored daily. The study nurses attempted to call all caregivers in this report. During the phone call, caregivers were asked a standardized set of questions about their child’s health and the study nurses manually entered the responses into REDCap immediately following the interaction.

Outcome measures

Ethics

Ethics approval for this study was obtained from the institutional review boards at Makerere University School of Public Health in Uganda (SPH-2021–41) and the Uganda National Council for Science and Technology (HS1745ES). As this is a QI project, patient consent was waived. Ethics approval for this study was not required by the University of British Columbia’s (UBC) Research Ethics Board. UBC adheres to the Canadian government’s Tri-Council Policy 2 (TCPS2) Statement which states that quality assurance and quality improvement (QA/QI) studies, program evaluation activities, and performance reviews, or testing within normal educational requirements, when used exclusively for assessment, management or improvement purposes, do not constitute research under the TCPS2 and do not fall under the scope of REB review. See section 4.4.1 of the UBC Clinical Research Ethics General Guidance Notes. No data was collected in Canada. This QI project was reported according to the Standards for Quality Improvement Reporting Excellence (SQUIRE) 2.0 guidelines [13]. Additional information regarding the ethical, cultural, and scientific considerations specific to inclusivity in global research is included in the Supporting Information (S3 File).

Study population

During the study period, 6,826 children triaged through the Smart Triage platform were enrolled in the automated follow-up system. The median age was 1.4 years (IQR: 0.6 to 3.5 years), with 84% (5731) of children under 5 years of age. Overall, 54% (3673) of participants were male. The most common presenting complaints at triage were cough and/or respiratory illness (2732, 40%), diarrhea and/or vomiting (1505, 22%), and fever (1034, 15%). In terms of triage acuity, 52% (3531) of children were classified as non-urgent, 36% (2450) as priority, and 12% (845) as emergency cases at initial triage (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of the study population, including age, sex, presenting complaint, and triage category at initial presentation. In cases where multiple presenting complaints were listed, the most severe complaint was selected for this table (e.g., convulsions were selected if cough, diarrhea, convulsions were all listed).

https://doi.org/10.1371/journal.pone.0322969.t001

Delivery rate

Among enrolled participants, 84% (5715/6826) of caregivers chose SMS as their preferred channel and 16% (1111/6826) chose WhatsApp (Fig 2). The overall delivery rate was 95%. Of the 357 (5%) undelivered messages, most (98%) were WhatsApp messages, while 2% were SMS messages.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Messages sent and received.

https://doi.org/10.1371/journal.pone.0322969.g002

Response rate

Between June 2022 and March 2023, before the implementation of any QI changes, the response rate was 22% (507/2292). Response rates improved from 20% in April 2023 to 40% in June 2024, following the introduction of iterative QI changes. During the post-QI period (December 2023 – June 2024), response rates remained stable between 33% and 41%, indicating sustained improvements beyond the active intervention phase. Compared to the historical period (June 2022 – March 2023), the overall response rate during the post-QI period was significantly higher (observed difference: 15.4%; 95% CI: 12.5% to 18.2%, p < 0.001, S2 Table). The Cochran-Armitage trend test also demonstrated a statistically significant positive trend in response rates during the QI intervention period (April 2023 – November 2023) (Z = 5.32, p < 0.001; S2 Table). Between June 2022 and June 2024, 1856 responses were received. Out of those, 1244 (67%) caregivers responded that their child had improved, and 612 (33%) responded that their child had not improved.

The QI changes implemented were informed by the feedback from caregivers (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Reasons for not responding.

https://doi.org/10.1371/journal.pone.0322969.t002

The data gathered through follow-up calls with non-responders identified key barriers such as message delivery times, confusion about the sender, and lack of understanding of how to respond. For example, follow-up messages were initially sent at 9:00 AM. After receiving feedback that caregivers were often busy in the mornings, the message time was shifted to 4:00 PM to better align with their availability. Another strategy was to add a reminder message if no response was received three days after the initial message, to prompt caregivers who may have forgotten or missed the first message. In addition, to ensure caregivers understood how to respond, a demo message was sent to the caregiver’s phone at the time of their in-person triage. A health worker demonstrated and explained the follow-up message to caregivers. Additionally, caregiver education was enhanced using posters displayed in the OPD and information cards distributed during triage. The posters and cards provided additional information regarding the automated follow-up system, including the definitions of the response options and understanding the response process.

Health outcomes

A total of 5582 caregivers were scheduled for follow-up phone calls by the study nurse. Those scheduled included 4551 caregivers (82%) who did not respond to the automated message, 612 (11%) who replied that their child had not improved, 41 (0.7%) who sent an invalid response, 21 (0.4%) who unsubscribed, and 357 (6%) for whom the message delivery failed (Fig 2). As part of the protocol, caregivers who unsubscribed from the messages were still followed up with by phone call. A total of 5619 caregivers were actually reached by phone, including 177 caregivers (3%) who received a follow-up phone call even though they responded that their child had improved (calls were not required for this group). This occurred due to either delayed caregiver responses, which arrived after the follow-up call had already been made, or errors by the study nurse. In total, 140 caregivers (3%) were lost to follow-up.

Among the 4551 non-responders, 206 caregivers (5%) reported seeking care, 33 (0.7%) stated their child had been readmitted, and 3 children (0.07%) had died by the time of follow-up (Table 3). Among the 612 caregivers who reported that their child had not improved, 58 (9%) indicated they had sought some care that did not lead to an improvement, 12 (2%) reported that their child had been readmitted, and none had died by the time of the phone call. The most common reasons for readmission were malaria (5 cases, 41%) and upper respiratory tract infections (URTI; 2 cases, 17%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Health outcomes collected during follow-up phone calls.

https://doi.org/10.1371/journal.pone.0322969.t003

Comparison with other studies

SMS-based interventions are effective in healthcare settings in low-and middle-income countries (LMICs) and are reported to improve childhood immunization coverage and timeliness [11], increase attendance at maternal and neonatal healthcare visits [14], and support treatment adherence in the management of hypertension and HIV [15, 7]. However, many studies using text messages have primarily focused on sending reminders or behavior change interventions aimed at improving specific outcomes, such as appointment attendance and medication adherence, and mostly in non-urgent conditions where there is time for deliberation [16].

Strategies to improve response rates for text-based interventions include personalizing messages, such as incorporating the patient’s name or tailoring content to specific health conditions [17, 18]. Sending messages at times aligned with recipients’ routines or preferences also improves response rates [17, 18]. In our study, strategies such as personalizing messages to include the child’s name and adjusting the timing of message delivery were implemented, leading to an overall increase in response rates.

The potential for text-based interventions to improve post-discharge outcomes is highlighted by a study conducted in rural Kenya [19]. The study evaluated the use of SMS for following up with caregivers of children after discharge. The findings revealed improvements in caregiver knowledge, increased efficacy in childcare, strengthened relationships with healthcare staff, and reduced hospital readmission rates [19].

Strengths and limitations

The strength of our study is the iterative QI approach which tailored the system for the environment in which it was routinely used. We drew on successful strategies from previous studies, including the WelTel Kenya1 study [7], which demonstrated the effectiveness of using simple messages such as “How are you?” to encourage responses and enable self-reporting of problems. Building on this approach, our system utilized two-way text messaging to confirm whether caregivers received and engaged with the messages and to capture valuable feedback on the health of children once they returned to their community. This feedback may allow health workers to respond quickly when caregivers report a lack of improvement in their child’s condition, potentially preventing adverse outcomes.

There is a critical need for community post-discharge outcome data and follow-up care for high-risk children, as well as a more efficient approach to reduce the burden on health workers and healthcare resources. Before the introduction of the automated follow-up system, health workers attempted to call every caregiver of a discharged child—a highly resource-intensive process, with each call taking at least five minutes. A more targeted approach, in which follow-up calls focus only on caregivers who do not respond to messages or report that their child has not improved, can streamline efforts and make follow-up more feasible. While individual phone calls remain time-intensive, prioritizing high-risk cases using data-driven strategies can reduce overall resource demands compared to traditional methods. Additionally, the system ensures timely data collection, which is essential for driving QI initiatives and refining clinical risk prediction models. The growing response rates, along with the simplicity and accessibility of text messaging, suggests SMS/WhatsApp messaging could be a viable alternative to traditional follow-up methods, such as return visits, particularly in settings with limited infrastructure and long travel distances. Beyond facilitating follow-up, the system captures valuable community-level outcomes, such as readmissions and post-discharge mortality, providing crucial data for clinical and public health planning. Furthermore, the collected data may help identify key trends, including seasonal variations in disease prevalence and shifts in care-seeking behaviors, which can inform targeted public health interventions.

Limitations are related to the technology and barriers faced by caregivers. Undelivered messages due to rurality and infrastructure (e.g., caregivers lacking active accounts or compatible devices, inconsistent mobile network coverage, and occasionally disrupted message delivery) were significant, given the total number of messages sent. Caregiver-related barriers included difficulty understanding the messages, competing priorities, unawareness of the sender or purpose of the message, and concerns about cost (although they were assured that this is not an issue). Incorrect or outdated phone numbers and broken screens and keypads were also problems. These findings highlight the importance of addressing behavioural and technological obstacles to improve response rates.

The costs associated with this system are relatively high in the Ugandan context, with fixed monthly and annual payments to Africa’s Talking and the Ugandan Communications Commission amounting to USD $3,776.00 per year. These costs are fees paid to the mobile network operators and for the use of a dedicated short code. Although the cost per individual message was low— USD $0.016 for outgoing WhatsApp and USD $0.0018 for incoming and outgoing SMS—these expenses can be costly at scale and accumulate significantly with high patient volumes. Twilio charges a small per-message fee for each outgoing WhatsApp message, which decreases as the volume of messages increases. However, the system could be cost-effective if it is shared across multiple facilities, distributing the fixed costs more efficiently. Mobile applications like WhatsApp offer a more cost-effective solution for communication compared to SMS. However, in our study, most caregivers preferred SMS as their communication channel. This preference may be attributed to the study’s setting in a relatively poor region of Uganda, where access to smartphones is likely limited for many caregivers. In the Acholi region which encompasses Gulu district, 68% of households owned a mobile phone compared to 93% in higher income areas like Kampala [20]. Despite these costs, the system has the potential to enable efficient care and reduce hospitalizations by improving follow-up, but its cost-effectiveness and scalability remain to be formally assessed. Leveraging predictive models [3, 4] to identify and target follow-ups for the most vulnerable children could maximize the impact of an automated follow-up system while minimizing unnecessary expenditures and could be a valuable investment in resource-constrained settings.

Additionally, several limitations are associated with the study design and analysis. The observational design without a comparator or control group limits the ability to draw causal inferences about the effectiveness of the interventions. While we observed changes over time, we cannot rule out the influence of external factors. The study was a quality improvement initiative and was not designed for formal hypothesis testing. Therefore, we primarily used descriptive statistics to track trends over time and guide adaptations to the intervention. While additional post-hoc statistical analyses were conducted to examine changes in response rates, these were not pre-specified and should be interpreted cautiously. In particular, the designation of a post-QI period was applied retrospectively to assess whether improvements were sustained. Smaller sample sizes and variable trends across other periods and months would have limited the ability to detect statistically significant differences. We found that this comparison between the historical period and the post-QI period was well-powered and showed a significant difference, but the findings should be interpreted as indicative of implementation trends rather than definitive proof of intervention effectiveness. These limitations reflect the real-world, pragmatic nature of the project. Future studies using more rigorous designs and statistical methods would be valuable to further evaluate this approach.

While mobile messaging can support post-discharge follow-up, it also raises ethical concerns. Not all caregivers had reliable access to phones, mobile networks, or the digital literacy needed to engage with the system. This may have unintentionally excluded some families and deepened existing health inequities. Although we prioritized caregiver privacy, digital follow-up still requires the collection of personal information. Even when anonymized, this raises important issues about consent, data ownership, and autonomy, particularly in low-resource settings where power imbalances may exist between families and the healthcare system. Future implementation should focus on improving digital access, clearly communicating how data will be used, and involving caregivers in the design and oversight of such systems.