Field testing two existing, standardized respiratory severity scores (LIBSS and ReSViNET) in infants presenting with acute respiratory illness to tertiary hospitals in Rwanda – a validation and inter-rater reliability study

Introduction There is a substantial burden of respiratory disease in infants in the sub-Saharan Africa region. Many health care providers (HCPs) that initially receive infants with respiratory distress may not be adequately skilled to differentiate between mild, moderate and severe respiratory symptoms, which may contribute to poor management and outcome. Therefore, respiratory severity scores have the potential to contributing to address this gap. Objectives to field-test the use of two existing standardized bronchiolitis severity scores (LIBSS and ReSViNET) in a population of Rwandan infants (1–12 months) presenting with respiratory illnesses to urban, tertiary, pediatric hospitals and to assess the severity of respiratory distress in these infants and the treatments used. Methods A cross-sectional, validation study, was conducted in four tertiary hospitals in Rwanda. Infants presenting with difficulty in breathing were included. The LIBSS and ReSViNET scores were independently employed by nurses and residents to assess the severity of disease in each infant. Results 100 infants were recruited with a mean age of seven months. Infants presented with pneumonia (n = 51), bronchiolitis (n = 36) and other infectious respiratory illnesses (n = 13). Thirty-three infants had severe disease and survival was 94% using nurse applied LIBSS. Regarding inter-rater reliability, the intra-class correlation coefficient (ICC) for LIBSS and ReSViNET between nurses and residents was 0.985 (95% CI: 0.98–0.99) and 0.980 (0.97–0.99). The convergent validity (Pearson’s correlation) between LIBSS and ReSViNET for nurses and residents was R = 0.836 (p<0.001) and R = 0.815 (p<0.001). The area under the Receiver Operator Curve (aROC) for admission to PICU or HDU was 0.956 (CI: 0.92–0.99, p<0.001) and 0.880 (CI: 0.80–0.96, p<0.001) for nurse completed LIBSS and ReSViNET respectively. Conclusion LIBSS and ReSViNET were designed for infants with bronchiolitis in resource-rich settings. Both LIBSS and ReSViNET demonstrated good reliability and validity results, in this cohort of patients presenting to tertiary level hospitals. This early data demonstrate that these two scores have the potential to be used in conjunction with clinical reasoning to identify infants at increased risk of clinical deterioration and allow timely admission, treatment escalation and therefore support resource allocation in Rwanda.


Introduction
There is a substantial burden of respiratory disease in infants in the sub-Saharan Africa region. Many health care providers (HCPs) that initially receive infants with respiratory distress may not be adequately skilled to differentiate between mild, moderate and severe respiratory symptoms, which may contribute to poor management and outcome. Therefore, respiratory severity scores have the potential to contributing to address this gap.

Objectives
to field-test the use of two existing standardized bronchiolitis severity scores (LIBSS and ReSViNET) in a population of Rwandan infants (1-12 months) presenting with respiratory illnesses to urban, tertiary, pediatric hospitals and to assess the severity of respiratory distress in these infants and the treatments used.

Methods
A cross-sectional, validation study, was conducted in four tertiary hospitals in Rwanda. Infants presenting with difficulty in breathing were included. The LIBSS and ReSViNET scores were independently employed by nurses and residents to assess the severity of disease in each infant.

Introduction
Acute respiratory distress is a frequent cause of pediatric emergency department attendance. Common etiologies include bronchiolitis and pneumonia [1,2]. The burden of respiratory disease remains in Low-Income Countries (LICs), where mechanical ventilation facilities are limited or unavailable [3]. Deaths from pneumonia and bronchiolitis have been linked to low health coverage, lack of exclusive breastfeeding, malnutrition, incomplete immunization and lack of access to an appropriate health care service [4,5]. Many health care providers (HCPs) that initially receive infants with respiratory distress may not be adequately skilled to differentiate between mild, moderate and severe respiratory symptoms, which may contribute to poor management and outcome [6]. Untreated or unsupported pediatric respiratory distress can lead to respiratory failure, identified as the main cause of cardiac arrest in children [7]. Continuous positive airway pressure (CPAP) can help to prevent progression to respiratory failure [3]. However, identifying those children who need admission and those that need higher levels of care is dependent upon HCPs with the necessary skills and tools. Risk prediction models offer the potential to support such clinical decision making, and in respiratory disease can help to identify children that require admission and respiratory support [8]. One such model is a clinical measurement instrument consisting of clinical signs and symptoms that are grouped together to measure an intended construct. The validation of such an instrument should ideally include: assessment of reliability, responsiveness, and usability [9][10][11][12][13][14][15]. Instruments could be used by HCPs in their decision making, such as when to admit children, or when to escalate care of infants with respiratory distress. In order to be fit for purpose, such an instrument should be validated, reliable, quick to perform, and straightforward to interpret in the clinical context [15]. It should not involve complex measurements, descriptions or equipment.
The Liverpool Bronchiolitis Severity Score (LIBSS) and the Respiratory Syncytial Virus Network Scale (ReSViNET), are both bronchiolitis specific scores. LIBSS and ReSViNET have both previously been assessed for apparent validity (to develop the model) and internal validity in infants in developed countries with bronchiolitis symptoms [25,28].

Objectives
This study sought to field-test the use of two scoring instruments (LIBSS and ReSViNET), assessing the severity of respiratory distress in a population of Rwandan infants (1-12 months) presenting to urban, tertiary, pediatric hospitals. Specifically, we assessed the usability, interrater reliability and internal consistency of the two instruments. The secondary aim was the description of the severity of respiratory distress in these infants and the treatments used.

Reasoning for choice of LIBSS and ReSViNET scores
Both the LIBSS and ReSViNET instruments share five parameters and employ clinical parameters assessed by HCPs [15]. The parameters of both scores are applicable to all respiratory diseases that can cause respiratory distress in infants in settings such as Rwanda, and can be undertaken with no equipment (ReSViNET) or with just a saturation monitor (LIBSS). Specifically we sought to continue the work of the LIBSS and ReSViNET teams to see if these scores could be used in our setting, without requiring an assessment of HIV, malaria or chronic nutritional status, which are required for RISC scores.

Study design
This was a cross-sectional, multi-centre, validation study, which took place from September 2018 to February 2019. Reporting of this study has been verified in accordance with the TRI-POD checklist for reporting prediction models [39,40].

Study sites (data source)
The principal investigator (BH) undertook the study to complete his MMed pediatric residency at the University of Rwanda. The study was therefore conducted at four public university, or university affiliated hospitals in Rwanda, namely; Kigali University Teaching hospital (CHUK), Rwanda Military Hospital (RMH), Butare University Teaching hospital (CHUB) and Ruhengeri Referral Hospital (RRH). Recruitment was planned but unsuccessful at King Faisal Hospital (KFH), a collaborative public-private, university, tertiary hospital. CHUK, KFH and RMH are all located in Kigali city, the capital of Rwanda. CHUB and RRH are in provincial towns. All hospitals are tertiary referral centres. These sites are all located in urban settings but receive patients from both rural and urban settings. Patients using the national health insurance system ("mutuelle de sante") cannot self-refer to tertiary hospitals and are therefore referred from Health Centers and District Hospitals. A small number of paying patients will self-refer.

Study population
Participants were recruited prospectively from children in both outpatient departments and the pediatric emergency rooms (ER) at the study sites.
Inclusion criteria. infants 1-12 months of age presenting with respiratory distress due to any respiratory illness and whose parents could give consent [41,42].
Exclusion criteria. infants with known chronic lung disease (CLD), or infants who presented with a non-respiratory cause of respiratory distress (e.g. cardiac disease).
Sampling. recruitment was opportunistic at the study sites, with infants presenting to the study sites and meeting the case definition of respiratory distress being approached for recruitment.
Patient recruitment. The resident pediatrician and/or resident on duty identified eligible infants. Parents were provided with both verbal and written information about the study. If they agreed to participate in the study, written informed consent was gained and demographic details collected.

Clinical care of infants
All data-collectors were HCPs and were involved in the clinical care of the participants, and so were not blind to the infants' condition, treatment interventions and patient outcomes. If an infant required stabilization this was done before recruitment and severity scoring. These interventions were administration of oxygen, antibiotics and fluids. 3. LIBSS: A validated score for use in children with bronchiolitis between the ages of 0-12 months. LIBSS has two scoring systems 0-3 months score and 3-12 months to take into account for age-dependent vital signs [28]. The LIBSS includes ten parameters, namely; General condition; Apnea; Increased work of breathing; Sa02; Respiratory Rate; Appearance; Heart Rate; Feeding; Urine output; Capillary refill time.

Data collection tool
4. ReSViNET: a validated clinical severity scale that was developed to assess the severity of illness in infants with bronchiolitis [25,43]. ReSViNET includes seven parameters, namely; Feeding intolerance; Medical intervention; Respiratory difficulty; Respiratory rate; Apnea; General condition; Fever 5. Study-specific follow-up questionnaire for final outcomes: admission to pediatric ward; admission to high dependency unit (HDU) or pediatric intensive care unit (PICU); or outpatient treatment. This section of the questionnaire was specifically designed for the purposes of this study.

Instrument translation
Most HCPs in Rwanda speak Kinyarwanda (local language), English and/or French. However, English may not be the first language. Therefore, to overcome language-related barriers, LIBSS and ReSViNET were translated into Kinyarwanda by the principal investigator (PI) and then back-translated for accuracy by a native-Kinyarwanda speaking pediatric resident. Translation discrepancies were reviewed with a third native-Kinyarwanda speaking pediatric resident for consensus. Within the scoring instruments, the Kinyarwanda translation was presented alongside the original English. Instrument availability. The Translated LIBSS and ReSViNET, used in this study, are available online (https://doi.org/10.7910/DVN/N4O05G). The original ReSVinet was published in PLoS and is therefore available under Creative Commons licence CC BY [25] and the original LIBSS is available under a CC BY attribution [28].

Data-collection
One pediatric resident and one nurse independently assessed each infant using LIBSS and ReSViNET. The resident and nurse were instructed to not share their LIBSS/ReSViNET findings with each other. Infants were recruited and scored within an hour of presentation to hospital.

Training data-collectors/assessors
Prior to undertaking assessments, HCPs were given three hours of training on two consecutive days on how to use the two scores and questionnaires.

Sample size
Data was collected on paper forms, entered into Microsoft Excel and analyzed using SPSS version 24. The 'rule of thumb' to determine the sample size for a clinical field-test is 10-15 times the number of parameters in a test [44]. This gave a minimum sample size of 100 for LIBSS (10 parameters) and 70 for ReSViNET (7 parameters).
Validity. Criterion and convergent validity were used to assess whether the instrument measures what it was intended to measure. Criterion validity (predictive) testing was determined using the area under aROC based on severity scores of nurse-assessed LIBSS and ReSViNET. aROCs of 0.50-0.70, 0.70-0.90 and >0.90 were pre-defined as indicating low, moderate and high validity respectively [44]. Construct validity was assessed by examining the correlation between LIBSS and other measures of the impact of the disease: hospital admission, length of stay, HDU/PICU admission and mortality. Length of stay was categorised into binary criteria of long/short dependent on the median length of stay as a cut-off.
The two scores use 12 different parameters. Five parameters are shared by the two tools, namely: Apnea; Feeding intolerance; General condition; Increased work of breathing; and respiratory rate. ReSViNET uses two additional parameters (Fever and medical intervention), and LIBSS employs five unique parameters (Appearance, Capillary refil time, heart rate, oxygen requirement and urine output). Therefore, convergent validity of LIBSS and ReSViNET scores was assessed using Pearson's r correlation coefficient. Coefficients of 0.4-0.59, 0.6-0.79 and 0.8-1.0 indicating moderate, strong and very strong relationships respectively [45].
Reliability testing. Inter-rater reliability (agreement) was evaluated between paired pediatric resident and nurse responses using the Intraclass Correlation Co-efficiency (ICC) with ICC <0.75 poor to moderate, >0.75 is good, >0.9 is excellent [44,46]. Internal consistency of the parameters within each score was measured using Cronbach alpha, whereby Cronbach's: <0.70 poor, >0.70 good (if <7 items), interpretation is dependent on number of parameters [44].

Ethical considerations and declarations
Financial disclosure. The author(s) sought nor received any specific funding for this work.
Potential conflict of interest. The authors have declared that no competing interests exist.
Confidentiality. Unique Individual Patient Identifier codes were used when completing the questionnaires. Patient-identifiable information was kept in a separate database to protect confidentiality.
Informed consent. Informed, written consent, was gained from parents or caregivers. Incentives for subjects. No incentives, financial or otherwise, were offered to participants.
Risk to subjects (including safeguards to mitigate these risks). No significant physical, social, emotional, legal or financial risks were identified.
Ethical approval. The project proposal was reviewed by the Institutional Review Board (IRB) at the University of Rwanda. Ethical approval was given (Ref: NO237/CMHSIRB/2018). The proposal was then reviewed and approved by the ethics committees of each enrolling site.
Availability of data and materials: The study data-set is available online at https://doi.org/ 10.7910/DVN/N4O05G/AKSNOZ.

Recruitment
A total of n = 107 eligible infants were recruited (Fig 1): CHUK (n = 47); RRH (n = 46), CHUB (n = 9) and RMH (n = 5). Data from seven infants could not be used due to data collection being incomplete or inadequate, and therefore, these cases were removed. Data most commonly omitted were the heart rate and respiratory rate.

Data quality
No data points were missing from the final data set.
Criterion validity (predictive). Both LIBSS and ReSViNET performed well for predicting hospital admission, HDU/PICU admission and mortality (Table 4). However, they performed only moderately well for predicting prolonged length of stay (Fig 3).
Internal reliability (consistency). Internal consistency for both scores was good with marginally higher internal consistency for data from the ReSViNET score (Cronbach = 0.850 and 0.848 for nurse and resident scoring respectively) ( Table 4). Table 4 demonstrates the reliability of each score if each item is deleted from the score, with only the removal of apnea resulting a modest increased reliability in both scores.

Discussion
This study sought to field-test the use of two scoring instruments (LIBSS and ReSViNET), assessing the severity of respiratory distress in a population of 100 Rwandan infants (1-12 months) consulting urban, tertiary, pediatric hospitals. The predictive validity, reliability between raters, and the internal consistency of the two instruments was measured and both instruments performed well.

Severity of disease
The majority of patients had moderate or severe disease. This likely represents the location of the field testing and the health structure in Rwanda. Children cannot self-present to tertiary sites and are therefore referred from Health Centers and District Hospitals, where they will likely have provided the necessary care of infants with mild disease, without referring to the tertiary hospitals. The pre-established cut-points of ReSViNET identified fewer infants as having severe disease.

Treatments used
Though assessing severity is important, this study also highlights the importance of the implementation of evidence into clinical practice to ensure that evidence-based treatments are employed. Many patients received unnecessary treatments (Table 5). Antibiotics, adrenaline and/or bronchodilators or not warranted in bronchiolitis but were used frequently in children with this condition and discontinuing the use of these three interventions as a key priority [47][48][49]. Not only is the efficacy of these medications not backed up by the evidence in the literature, they are also costly for the health care system and families. Further work needs to be undertaken into how to reduce the use of these ineffective treatments. This is not a Rwandan-specific issue, and this problem has also been described in developed [50].

Validity of the scores
Many healthcare facilities in resource-limited settings will not have medical doctors. Therefore, the nurse-assessed LIBSS and ReSViNET were used for the validity analyses. There is no "gold-standard" for assessing respiratory disease. When LIBSS and ReSViNET were measured against each other, there was a strong correlation between the two scores (Pearson's >0.8).
The scores use 12 parameters, and as they share five of parameters (apnoea, feeding intolerance, general condition and respiratory rate), the strong correlation is perhaps not surprising, but it is reassuring that they are scoring severity similarly. Both scores performed highly for predicting hospital admission, HDU/PICU admission and mortality. HDU/ICU are not infrequently needed to optimize respiratory and medical support, especially in sub-Saharan Africa [2], however, such care may require the transfer of the patient, is expensive and labour-intensive. Identifying the right patients for this level of care is, therefore, important. The case-definition used in our inclusion criteria overlapped with the scoring systems that were measured. Therefore, when interpreting the validity of the instruments it is important to consider that the sample taken were done so using this case-definition and therefore it is feasible that the tools will perform very differently in unselected children and in primary care levels.

Reliability
Interrater reliability. The inter-rater reliability was good, with lower confidence limits of 0.98 and 0.97 for LIBSS and ReSViNET, whereby scoring >0.9 is "excellent" [44,46]. In the UK, field-testing of LIBSS, the lowest confidence interval for ICC was 0.75 [28], and the ReSVi-NET gained lowest ICCs of 0.76 between professionals [25]. It is therefore interesting that the Rwandan professionals scored considerably higher than both these original settings.
Internal consistency. The consistency was better within the ReSViNET, and this is despite it having fewer parameters (seven versus ten). It is well known that some items may particularly affect reliability within the data and removing this item could see an improvement in internal consistency, at the cost of content validity. Our data revealed good internal reliability and only the removal of the measurement of apnoea would have marginally improved the internal reliability. As Apnoea is an important aspect of respiratory illness in infants (content validity) we would not advocate removing it.

Use of LIBSS and ReSViNET in the Rwandan setting
The most common co-morbidities were: malnutrition (12%), prematurity (5%) and HIV (2%) ( Table 2). According to the Rwandan DHS 2015 data, 38% of under-5 children are stunted where 18% of infants between 6 to 8 months are stunted [51]. RISC requires and assessment of nutritional status as these infants are particularly prone to worse outcomes. Therefore, using a respiratory distress score including HIV and/or nutritional assessment is likely to be beneficial. However, doing so requires additional skills (nutritional assessment) which can add to the complexity of the scoring.

Comparison of the population with original tools
The major consideration here is that the original scoring tools were designed for infants from resource-rich settings and for infants with bronchiolitis alone, with other respiratory illnesses being assessed. The cut-points for disease severity were therefore determined in a different setting with cohorts of patients very different from ours. These factors may account for the lower criterion validity for HDU/PICU admission with ReSViNET (aROC 0.880) compared to LIBSS (0.956).

Use of valid and reliable clinical scores in research
As well as using scores in clinical practice severity scores are also useful for clinical trials, in order to consistently and reliably assess severity of diseases and the effects of treatments. Both the scores described here are simple to use, without complex investigations or assessments, and have the potential to be used in research activities in this setting and age group.

Bias
Our population of outpatient-managed cases was only small (n = 7), probably reflecting the type of hospitals where assessment was undertaken. Usually tertiary and/or referral hospitals in Rwanda receive critically ill patients transferred from peripheral hospitals. Due to the scarcity of hospital beds, priority is given to the most critically ill patients. But self-referred patients with private financial capacity can consult as outpatients. Therefore our population was biased to either tertiary, referred patients or private self-referring patients.

Limitations
Limitations of our study include the lack of a gold standard for the evaluation of children with acute respiratory distress and the lack of a mechanism to have an ongoing follow-up. The other major limitation was our inability to hire full-time data collector nurses at each study site. This would have potentially increased recruitment and reduced incomplete datasets that led to seven cases being excluded. We attempted to recruit at a University public-private hospital, King Faisal Hospital (KFH), a public-private, but only recruited three cases, which was felt insufficient for inclusion. The reason for poor recruitment was high perceived work load and reported low admissions of patients with respiratory distress. The sample size was suitable for field-testing the severity scores and to assess if they are feasible for use. There is however a risk of over-fitting in relation to the prediction measures (aROCs). The tests we used were created for assessing bronchiolitis rather than all causes of respiratory illness. They do not assess for HIV, malaria or chronic nutritional status, and these may have important prognostic implications that require additional resources and HCP skills, therefore, may impede utilization both in clinical and research applications.

Next steps
In this study, it was not feasible to undertake repeated assessments, and therefore, responsiveness was not assessed. Testing responsiveness could have allowed us to monitor progress in hospital and see if the scores were helpful to identify deterioration or improvement and guide care. Applying evidence into practice is challenging. These two scores have not been applied in Rwanda and doing so would require additional work, such as assessing stakeholder attitudes, along with addressing training needs of those using scores in primary care facilities. When LIBSS and ReSViNET were measured against each other, there was a strong correlation between the two scores (Pearson's >0.8). However, ReSViNET described fewer infants (n = 16, 16%) as having severe disease compared to LIBSS (n = 33, 33%) ( Table 2). Therefore a larger study to identify appropriate cut-off points, along with responsiveness, would be an important piece of work to undertake.

Conclusion
The findings of this study are important to the Rwandan health system where facilities have limited resources, and the decision to admit and escalate care must be carefully considered. This early data demonstrate that these two scores have the potential to be used in conjunction with clinical reasoning to identify infants at increased risk of clinical deterioration and allow timely admission, treatment escalation and therefore support resource allocation in Rwanda.