Data sources.

ePOCT+ stores all the information (date of consultation, anthropometric measures, vitals, presence/absence of specific symptoms and signs prompted by the algorithm, diagnoses, medicines, managements, etc.) entered by the HW in the tablet during consultations. We retrospectively analyzed data from 20,085 outpatient consultations conducted between November 2021 and October 2022 with children aged 1 day to 14 years with an acute condition, in the 16 HCs where the intervention was deployed. The data were accessed for research purposes on 22 November 2022. All data used in the study were anonymized, and the authors did not have access to information that could identify individual participants during or after data collection. It is important to clarify that while our analysis was conducted retrospectively, the data itself was collected prospectively during routine ePOCT+ consultations. This distinction is crucial, as it means that typical biases associated with retrospective data collection do not apply to our study. Data cleaning, management, and analyses were conducted using R software (version 4.2.1).

Signs of interest.

Most of the signs in ePOCT+ are binary (presence/absence), which can be challenging to ground-truth (and are potentially biased toward negative responses as described above). On the other hand, some of the clinical signs, particularly measurements that result in numerical values, can be observed in a more objective and offer informative distributions for detecting outliers and other meaningful patterns. Thus, we focused on quantifiable and objective measures: temperature, mid-upper arm circumference (MUAC), weight, height, MUAC for age z-score, weight for age z-score, weight for height z-score, heart rate, respiratory rate and blood oxygen saturation (SpO₂). Z-scores were considered because MUAC, weight, and height distributions are age-specific and require normalization to be meaningful. The way each sign is measured and recorded by the HWs (or automatically calculated in the application such as z-scores) is described in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of signs of interest, how they were assessed by healthcare workers, entered in the tablet, and subsequently analyzed.

https://doi.org/10.1371/journal.pone.0318284.t001

For all the signs of interest, we identified the most frequent value (the value and its frequency) and the percent of measures that was declared as not feasible by the HW and marked as a missing value in the dataset. For the anthropometric signs (MUAC, weight, height) we also considered the percent of extreme values (<-2 or >2) in the z-scores. For temperature, blood oxygen saturation, heart rate and respiratory rate, we considered the median and the inter-quartile range (IQR).

Identification of potential skill gaps.

We compared the statistical parameters for each sign of interest in individual HCs against the same parameter across all HCs (Table 2), which we considered as our reference due to a lack of a true gold standard. We conducted the analysis at HC level, allowing us to identify the HCs that exhibited most significant deviations, guiding further investigation in subsequent analyses. Based on the features of the statistical parameters, signs (per HC) were labeled with one of the four distinct pattern types (i.e., skipper, mono-repeater, multi-repeater, and wrongly evaluated), along with one of the three corresponding levels of expression (i.e., mild, moderate, or severe). The level of expression was determined by comparing the statistical parameters (mean, interquartile range [IQR], frequency of not feasible/missing values, and value distribution) of clinical measurements across different HCs with the combined average from all HCs. The degree of deviation from this combined baseline was then categorized as mild, moderate, or severe, depending on how much the values diverged, as illustrated in Table 2. Examples of how data were visualized, and patterns were identified are provided in S1–S5 Figs and S1 Table, Supporting Information.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Types of pattern and definitions for their level of deviation from the mean values and distributions for all health centers (considered as reference).

https://doi.org/10.1371/journal.pone.0318284.t002

Temperature.

In the retrospective analysis, 3 HCs exhibited either mono- or multi-repeater type of patterns (Table 3), with the most frequent values being 36°C, 36.5°C, and 37°C (S4 Fig). In the observational study, the temperature measurement had the lowest skip rate of all signs (Fig 1). In the few instances where the temperature measurement was skipped, HWs primarily cited forgetfulness as the reason. When the temperature was measured, most of the time it was deemed of insufficient quality (Fig 1), with the predominant reasons being inadequate location (e.g., on the chest instead of forehead) or distance from the skin (for infrared thermometers) (Table 4).

MUAC.

In the retrospective analysis, 7 HCs exhibited either skipper, wrongly evaluated, mono- or multi-repeater type of patterns (Table 3). Often more than one type of pattern was present per HC. The most frequent values were 14, 15, 16 and 17 cm (S4 Fig). The observational study found that MUAC measurement, like temperature, was rarely skipped (Fig 1), with the main reasons for skipping being considering the measurement unnecessary for the child’s condition, considering the child too old, or asking the caregiver for the child’s health card to retrieve a MUAC measurement performed in the latest previous consultation (Table 4). MUAC had the highest rate of incorrect values (Fig 1), with the vast majority being done at a wrong place (other than at the mid-upper arm distance) (Table 4).

Weight and height.

In the retrospective analysis, no skipper behavior patterns were identified for weight, whereas for height 6 HCs exhibited skipper type of patterns. However, 6 HCs exhibited wrongly evaluated type of pattern, primarily based on weight-for-age and weight-for-height z-scores analyses (Table 3). In the observational study, weight and height were both rarely skipped (Fig 1). However, when omitted, it was because HWs considered them irrelevant to the child’s complaint, or relied on parent-reported weight, or visually estimated height without actual measurement. When weight or height was measured incorrectly, it was mainly because of failure to undress the child; often weight was taken while the child wore heavy clothes and/or shoes and height while the child wore shoes or a hat (Table 4). For weight measurements, additional issues included partial weight application on the balance and problems with balance calibration. For height measurements, issues included incorrect positioning on the measurement device, such as heels, shoulders, or head not touching the flat surface, and child not looking straight ahead during measurement (Table 4).

Heart rate and blood oxygen saturation measured using an oximeter.

In the retrospective analysis, inadequate patterns of heart rate measurement were identified in 8 HCs, predominantly as skipper (5 HCs), but also as mono- (3 HCs) and multi- (2 HC) repeaters (Table 3). The most frequent values were 68, 96 and 99 beats/min (S4 Fig). For blood oxygen saturation, inadequate type of patterns were identified in 7 HCs, manifesting as skipper (4 HCs), mono- (3 HCs), multi- (3 HCs) repeaters, and wrongly evaluated (1 HC). The most frequently frequent values were 96, 98, and 99 (S4 Fig). The observational study found that heart rate and blood oxygen saturation were indeed frequently skipped (Fig 1) due to time constraints, perceived absence of signs of respiratory distress, and dysfunctional equipment (Table 4). Additionally, incorrectly performed measurements were primarily attributed to the use of inadequate pulse oximeter probes and improper probe positioning (Table 4).

Respiratory rate.

In the retrospective analysis, respiratory rate was the most problematic sign, with patterns identified in 9 HCs, predominantly as mono-repeaters (4 HCs), but also as skippers and wrongly evaluated (3 HCs each) and multi-repeaters (2 HCs) (Table 3). In the observational study, respiratory rate measurement was the most frequently skipped sign (Fig 1). Justifications included the perceived absence of respiratory distress, time constraints, the possibility to tick “not feasible” in the app, forgetfulness, or reported lack of timing equipment (despite the fact that the RRate app was installed on all tablets) (Table 4). When respiratory rate was still measured, quality was adequate (Fig 1). Notably, the RRate app was used as frequently as the timer (39 times each), but the former resulted in better quality assessments, with 29 out of 39 measurements (74%) being sufficient compared to 18 out of 39 (46%) for the timer. Mistakes included underestimation or overestimation, mismanagement of measurement timing (e.g., starting the timer a lot before starting to calculate RR, not resetting the timer before starting to count, or not making the measure over a full minute), and failure to uncover the abdomen (Table 4).

Perceived relevance of measurement.

Our results underscore a significant issue with the perceived relevance of certain measurements. Despite the ePOCT+ prompting signs such as heart rate and blood oxygen saturation based on specific criteria related to the child’s condition, these measurements were frequently skipped. Notably, these signs, along with respiratory rate, were among the most frequently omitted. This may be attributed to the lack of integration of pulse oximetry into the Rwanda IMCI guidelines [30], making them only familiar to the HWs in the context of the DYNAMIC study and potentially leading to their misperception as unnecessary. Additionally, despite the availability of a ‘not feasible’ option, the retrospective analysis suggested that some HWs entered a value without actually taking the measurement, possibly to avoid having to explain their decision not to measure during the monitoring visits. This behavior was also observed, though less often, during our observational study, potentially due to the Hawthorne bias. Accurate measurement of respiratory rate is particularly crucial as it can help identify critical signs of respiratory distress and differentiate between mild and severe respiratory infections, which can significantly impact clinical outcomes.

To address this, targeted training should emphasize the importance of each measurement and provide practical strategies for integrating them effectively into consultations. Another approach could involve case-based training sessions where HWs analyze real scenarios demonstrating the impact of missing or inaccurate measurements on a patient’s diagnosis or treatment plan, which could reinforce their clinical relevance. Additionally, to improve the perceived relevance of these measurements, the contextual alerts within ePOCT+ can be revised to briefly explain why specific measurements are necessary based on the child’s symptoms and provide supporting evidence. Furthermore, if a measurement is skipped, HWs could be required to enter a justification, ensuring they reconsider the importance of the measurement and its relevance to the child’s condition. However, this approach should be considered with caution, as making the opting out option more difficult for HW could further encourage entry of false values without taking the measurement.

Training gaps and supervision.

Regular assessment of basic measurements such as temperature, weight, height and MUAC is part of a primary clinical assessment, regardless of the child’s complaint. In the post-pandemic period in Rwanda, there has been an emphasis on creating triage-like stations in the patient registration area where these measurements are commonly assessed –these stations were present in most (15 of 16) studied HCs but most frequently staffed by non-clinicians. As a result, the frequency of having vital signs and basic measurement were assessed was high; however the quality of their measurements was compromised due to methodological errors. While task-shifting remains a valuable strategy, it is crucial to ensure that staff who take on these tasks receive high-quality training and ongoing supervision by member of the nursing staff. Training should focus on a specific set of signs measured at the triage station, emphasizing the quality of these essential measurements over the quantity of tasks performed. It may be beneficial to designate a nurse HW as the leader of the triage team, responsible for conducting regular refresher training and ensuring the accuracy of measurements taken during triage. While it is unclear whether such a role currently exists in all HCs, implementing this approach could enhance both the training and supervision of triage staff, potentially improving the overall quality of primary clinical assessments. Additionally, digital refresher courses and interactive modules allowing HWs to review proper measurement techniques at their own pace, would be beneficial.

To further enhance measurement accuracy, the tablet-based application used in the triage stations includes warnings for numerical values that fall outside acceptable ranges. However, triage station staff who took the measurements wrote them on paper and nurses subsequently entered the values into the tablet in the consultation rooms. As such, any possible alerts incorporated in to ePOCT+ were not available to the staff taking the measurements. Secondly, variations in inadequate measurements by HWs were sometimes insufficient to trigger some alerts. This was highlighted by the retrospective analysis, which revealed extreme values in Z-scores (MUAC-for-age, weight-for-age, and weight-for-height) that did not activate the alert thresholds. Therefore, the algorithm’s alerts may need adjustment to account for Z-scores, not just raw MUAC, height, or weight values.

Equipment related challenges.

New or less familiar equipment has also presented significant challenges. The introduction of infrared thermometers in HCs during the COVID-19 pandemic was an important initiative aimed at saving time for HWs. However, due to the crisis situation, there may have been insufficient training and guidelines provided for the correct use of these new thermometers, especially to non-clinicians staffing the triage stations. Similarly, the addition of pulse oximeters as part of the DYNAMIC study introduced SpO₂ and heart rate measurements, which were not traditionally part of primary care assessments. Despite the training provided on pulse oximeter use, issues with its application persisted, potentially due to the high turnover rate of HWs in HCs. Although discouraged by the study team, new staff may have worked with equipment for which they have not received sufficient training.

To address these challenges, a dedicated person at the overseeing district hospital responsible for monitoring and supervision of nurses in HCs could be provided with access to a dashboard with similar comparisons that were explored in the retrospective analysis. When issues with measurements are identified using the data patterns, nurses in that particular HC could be pointed to an instructional video or in-person hands-on refresher training on the use of the tools.

Time constraints.

Time constraints also emerged as a significant factor affecting the quality of measurements. Respiratory rate, in particular, was the most frequently skipped measurement, with time pressure often cited as a primary justification, even though the RRate app was introduced and available to facilitate its accurate and efficient measurement (i.e., to measure respiratory rate with a timer requires a full minute, whereas the RRate app can determine it accurately in approximately 15 seconds).

While task-shifting has proven effective in redistributing responsibilities, the key challenge remains to ensure that time constraints do not compromise the quality of clinical assessments. One approach could involve increasing the number of triage measurements collected at the reception area based on the patient’s complaint, allowing HWs more time during consultations to focus on more complex assessments. Of course, this would require ensuring that the triage staff receive proper training to perform these measurements accurately and consistently, enabling them to make adequate and reliable assessments at the triage stage. Additionally, streamlining procedures to better integrate time-saving tools, such as ensuring the use of the RRate app, can further alleviate time pressures. Enhancing training on these efficient measurement techniques is also crucial, ensuring HWs are both confident and proficient in using tools that save time without sacrificing accuracy. To reinforce the importance of complete and accurate measurements under time constraints, providing recognition or incentives for HWs who consistently perform high-quality assessments can help sustain motivation and improve overall performance.