Background and Aims

Acute Respiratory Infections (ARI), and more specifically bronchiolitis, constitute a major cause of morbidity and mortality in infants throughout the world [1,2,3]. Even though the clinical characteristics, pathophysiology and course of ARI in infancy are well known, there is no consensus with regards to how best to measure the severity of disease and the impact of therapeutic or preventive interventions in clinical trials [4, 5].

Many clinical scores have been published and even validated for the evaluation of infants with ARI and/or bronchiolitis, although none of them has been sufficiently validated to allow meaningful use in children [5]. These scales incorporate a variety of end points, and they all have important limitations. These include the following: they are designed for medical personnel, they rely on in-person assessment of the patient by a physician (neglecting clinical records and parental opinion), many of them are focused only in the respiratory status of the patient, and in most cases they require parameters that are not universally available (such as oxygen saturation or invasive procedures) [6,7].

No score has been validated as a research tool in the setting of pediatric respiratory infections; many of them are focused only in the respiratory status of the patient, and most, if not all, require in-person evaluation, which may be particularly difficult to ensure in outpatient clinical trials [5, 7]. We also currently lack a scale that could be used by parents and physicians alike. Clinicians in emergency departments and primary settings, as well as professionals involved in outpatient clinical trials, could all benefit greatly from such a scoring system. Doctors could have a reliable tool, allowing parents to monitor the evolution of children at home, establishing objective thresholds for patient referral or to seek of medical assistance. In addition, in the setting of clinical trials, having a validated score that parents can use would allow objective follow-up after hospital discharge or in case of mild diseases.

The goal of this project was to develop and evaluate a global clinical severity scale (ReSVinet scale) that would be amenable to be used in outpatient and inpatient studies without the need of in-person assessment. Internal reliability, construct validity, and ease of applicability of the scale were assessed. Additionally, the validity and reproducibility of an adapted version of this score in non-medical language for parental use was evaluated.

Methods

All necessary ethical permissions were obtained from the regional ethics committee in Galicia (Spain) prior to the beginning of this study, which was approved by the aforementioned Institution on 23 March 2010 (registration number 2010/015). Written informed consent was obtained from all participants’ legal guardians before each patient was included in the study.

a- Score design

The main investigators, with substantial research experience in pediatric respiratory infections, systematically reviewed studies that reported on the use of a respiratory score in children with ARI. The following symptoms and clinical findings were initially considered as indicative of disease severity: general condition, respiratory difficulty, oxygen saturation by pulse oxymeter, cyanosis, apnea, food tolerance, tachypnea, and tachycardia. Finally some of these symptoms were either included as part of a single component of the score, or excluded for being considered redundant or for limiting the applicability of the score in non-hospital settings and/or by non-medical personnel.

The proposed score was submitted for evaluation to 90 pediatricians of outpatient, hospital care and pediatric critical setting from the Galician Pediatric Research Network (REGALIP) and from the FIVE research group [8]. They agreed on the ease of recording and value of the parameters included. The final ReSVinet scale is detailed in Table 1, consisting of seven different items with different value ranges assigned, which add up to a total score that spans from 0 to 20.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. ReSVinet scale.

This table presents the original scale, and was the one used by the three investigators.

https://doi.org/10.1371/journal.pone.0157665.t001

We aimed to validate this scale using information obtained from patients’ medical records. Our secondary aim was to provide the parents with a simple tool for evaluating the severity of their children’s disease. The scale was thus developed to cover an important limitation of the scoring systems currently available: none of them was designed for use by non-medical personnel. All the components as explained in the scale should be easily evaluated by a parent/caregiver with no previous training. This resulted in an adapted version of the ReSVinet scale written in plain Spanish language (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Adapted version of the ReSVinet scale for parental use.

https://doi.org/10.1371/journal.pone.0157665.t002

Results

The three investigators completed all of the scales, whereas 60 of the 61 parents who agreed to do so completed all the items of the score (one of the patients was discharged without having his parents filled in their version of the score). Demographics and clinical aspects of “professional” and “parental” cohorts are summarized in Tables 3 and 4, respectively. Median age (interquartilic range) was 6.5 (2.0–9.5) months for the “professional” cohort while it was 2.3 (1.8–6.3) months for the “parental” cohort. Score results for each subject can be seen in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Comparison of results obtained (A) by the three observers, and (B) by the parents and one of the observers. In graph A: O1- Results obtained by observer 1. O2- Results obtained by observer 2. O3- Results obtained by observer 3. In graph B: O1- Results obtained by observer 1. P—Results obtained by the parents.

https://doi.org/10.1371/journal.pone.0157665.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results of the ReSVinet scale in the professional cohort (Group 1).

This table categorizes patients according to clinical, epidemiological and outcome variables in the cohort evaluated by three pediatricians.

https://doi.org/10.1371/journal.pone.0157665.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Results of the ReSVinet scale in the parental cohort (group 2).

This table reflects the results of the score obtained by a physician (observer 1) and parents and the relation of the values obtained with clinical, epidemiological and outcome variables.

https://doi.org/10.1371/journal.pone.0157665.t004

Cronbach’s alpha for the results obtained by each of the observers and for parental subgroup was greater than 0.70 in all cases (r-value ranged from 0.72 to 0.79) for the professional cohort (G1), 0.73 for the parental cohort (G2).

Weighed kappa values for each paired couple of investigators was 0.77 for O1-O2, 0.71 for O1-O3 and 0.74 for O2-O3 in the first cohort. Light’s kappa for more than two observers was 0.74. The weighed kappa for the group evaluated by parents and O1 was 0.73. There was a strong correlation between the results obtained by investigators (r-value ranged from 0.76 to 0.83) and also between the results obtained by the parents and the investigators (r = 0.73).

O1 allocated 2.4% of the patients evaluated to the extreme endpoints of the scale (Total score 0 or 20). Rates for O2 and O3 were 0.59% and 0.0%, respectively. Only 1.7% of children reviewed by their parents were given an extreme value.

Changes in the score depending on several clinical conditions were sought. Only those patients who were admitted to a Paediatric Intensive Care Unit (p<0.001 for both cohorts) or those receiving Heliox (a fixed mixture of 30% oxygen and 70% helium) (P-value < 0.001 for G1 and P-value = 0.002 for G2) were found to have a statistically significant higher score in both cohorts. A previous episode of wheezing could be related to a more severe disease in the sample assessed by professionals (P-value = 0.012) but not in the parental cohort (P-value = 0.672). RSV was identified as the agent causing the disease in 77.1% (G1) and 81% (G2) of the cases, but could not be definitively linked to a more aggressive course of infection (i.e, achieving higher scores in the scale). No differences were found in the score when comparing younger patients to older ones, regardless of the person carrying out the assessment, but this observation lacked statistical significance (P-value = 0.790 for G1 and P-value = 0.880 for G2). Interventions such as nebulized epinephrine and inhaled or oral corticosteroids could not be related with significant changes in the final score. Children on antibiotics did receive a higher score value in the paediatrician-assessed cohort G1 (mean difference range: 2.2–2.4), and this difference could be attributable to endovenous prescriptions (P-value < 0.001), but not to oral ones (P-value = 0.980). Similarly, scores of children with suspected bacterial super-infection presented also a greater score than those discharged without that suspicion (mean difference range: 1.7–1.9, P-value = 0.001).

Severity was defined by admission to PICU. Those infants who required intensive care presented a significantly higher score than those admitted to wards (15.7 ± 2.6 vs. 10.2 ± 2.5 when evaluated by O1, 15.4 ± 2.7 vs. 9.2 ± 2.4 for O2, 14 ± 2.6 vs. 9.4 ± 2.4 for O3 in the professional cohort, P-value < 0.001 for all of these values). These results could be replicated in the parental subgroup (15.9 ± 2.5 for vs. 10.6 ± 2.7 for those children assessed by their parents, P-value < 0.001). Cut-points analysis shows that a “severe episode” can be defined when a child scores ≥ 14 points in our scale. If severity of the episode was defined instead by obtaining a value in the “severe” range of the WDS (face value), we also found that values ≥ 14 in the ReSVinet scale were associated with values of ≥ 7 in the WDS; this association was found to bear statistical significance when comparing these patients to those who presented a “moderate” severity (defined by obtaining a 4, 5 or 6 in the WDS): in the first cohort results were 15.1 ± 3.1 vs. 11.1 ± 2.7 for O1, 14.7 ± 3.0 vs. 10.2 ± 2.7 for O2, 13.7 ± 2.9 vs. 10.2 ± 2.5 for O3; and parental values (G2) were 15.9 ± 2.5 vs. 11.7 ± 2.0 (P-value < 0.001 for all of these observations). Mean values and standard deviation from O1 for patients in parental cohort (G2) were 16.8 ± 3.1 vs. 11.4 ± 2.6. In both cohorts, higher results meant a higher WDS (P-value < 0.001 for G1 and G2).

Length of stay had a significant positive correlation with the total value of the scale (r: 0.48–0.60 for G1, P-value < 0.001, and r: 0.33–0.35 for G2, P-value = 0.027).

Parents found the scale easy to understand, and they could complete it in less than two minutes. For investigators, all the information required in the scale was accessible through electronic clinical records. Investigators needed about five minutes to review the clinical record of a patient.

Discussion

The ReSVinet scale fulfills most of the requirements of an acceptable scoring system (Table 5) [5, 12,13,14] with the added value of avoiding the need for in-person evaluation and allowing an adapted version that is useful for non-professional assessment of the patient status.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Check-list of the characteristics of the ReSVinet scale according to the desired properties of a clinical scale validated for use in infants with acute respiratory infections^*.

The “considerations” column explains whether the ReSVinet scale meets the requirement. N.A. = not assessed.

https://doi.org/10.1371/journal.pone.0157665.t005

Our scale showed a good internal consistence and an acceptable inter-observer reliability even when tested by non-medical personnel. Most of the literature concerning scale validation deals with information obtained by only two investigators [5, 15], so having an acceptable weighted kappa index for more than two investigators is a relevant asset [5]. Intra-observer reliability or test-retest procedures can be assumed to be good if the inter-observer reliability is acceptable [12]. Hypotheses chosen as outcomes for testing face validity were correct for the WDS and discriminating the severity of the illness, and length of hospital stay also correlated positively with higher values of the scale in both groups.

So far, none of the many scores used for assessing the degree of dyspnea in patients with acute respiratory infections has been sufficiently validated to allow clinically meaningful use in children [5]. Our scale performed positively in 9 of the 15 items required. To the best of our knowledge, there are only a few scales whose validation criteria have been tested so thoroughly, and even fewer meeting more quality criteria than the ReSVinet scale [5]. The exclusion of different parameters by an expert panel is debatable, as there is ample evidence that some of them (such as peripheral oxygen saturation) are associated with longer admission and invasive measures [7]. The main points of disagreement concerned the inclusion of oxygen saturation and heart frequency, as both of them are frequently used in other scoring systems [16]. In the end, both were intentionally left out of the scale. Oxygen saturation is not always available in an outpatient setting. When used, recent studies show that its value in the decision-making process when facing a child with bronchiolitis may have been overestimated, leading to a higher rate of admissions [17]; physicians tend to abuse this “objective” measurement even when many do not understand its basis, placing great value on a difference of 2%, which is the accuracy most manufacturers report for their machines [18]. Parents also rely greatly on peripheral oxygen saturation when their child is admitted [19], and they could be prone to underestimate the true severity of the disease if this particular parameter is good enough for them. This variable was definitively discarded as its inclusion in the ReSVinet scale would hamper its usability by non-medical personnel–given the need of a device and training in probe placement and data interpretation–as well as its applicability in an outpatient setting. The need of oxygen supplementation (the intervention normally undertaken in patients with low oxygen saturation) is in any case covered in Q2. The “heart rate” parameter is featured in the most widely used scoring system, WDS [9], but its cut-off (120 beats per minute) is poorly discriminative in children under two years as it can be secondary to fever, irritability, pain, excitement, bronchodilators, or even a physiological heart rate in youngest ones. Another scores have defined higher cut-off values for this parameter. In the score designed by Marlais et al., the heart rate was the best of the five items chosen as predictors for admission[16], but since this was a retrospective study, this parameter was not compared with the heart rate of children being discharged directly from the emergency department; furthermore a higher heart rate may in any case be related to a more aggressive course of bronchodilators prior to admission. Wheezing is another item frequently included in respiratory assessment scales [4,6,9], but it can be difficult to explain, medical material (stethoscope) could be required for its assessment, and case definition of bronchiolitis in several countries does not require wheezing for the diagnosis [20, 21]. We chose instead to address the “respiratory noise” in Q3, as this concept is more understandable by non-medical personnel. The ReSVinet scale covers feeding difficulties (Q1), which have been related in several studies with greater risk of admission and invasive medical interventions [22,23,24] but are neglected by many of the scores already in use in clinical trials [4,5,7]. What is more, some of the most widely used scales deal only with the respiratory impact of the disease, like the Respiratory Disease Assessment Instrument (RDAI) or the Respiratory Assessment Change Score (RACS), both of which are widely used but have shown poor to moderate construct validity [4]; they even neglect findings such as hypoventilation and episodes of apnea. This particular concern is also found in scales more directed to asthmatic patients, like the Pulmonary Score [25]. Last, general condition is another parameter of our clinical scoring system which is usually not included in most of the existing scales, despite the fact that most clinicians seek and value the parental opinion on how severely affected they think their offspring are.

Our results suggest that this score has a reliable performance when calculating the respiratory difficulty from clinical records, making in-person evaluation not necessary. This feature could be of extraordinary value for clinical trials, where frequent in-person assessment and close follow-up is usually required when a participant presents with an adverse event such as bronchiolitis. Moreover, this could also be an asset in everyday clinical practice, allowing the primary care physician to obtain an impression of the condition of the infant at the moment of discharge from the emergency department, or their previous condition in a center that has just referred the patient for interment in another institution. In cases where medical records are not available or are not sufficient to allow the correct use of this score by the physician, parents could evaluate their children with results that can be compared to those of a physician. To the best of our knowledge, there is no other scale which shares these two advantages with the ReSVinet scale.

Limitations of our study include the lack of responsiveness in our currently gathered data, that the score was tested only on hospitalized patients younger than two years of age, and the inexistence of a gold standard for the evaluation of children with acute lower respiratory infection. Most of our patients suffered from acute bronchiolitis which, although the most frequent ARI during infancy, may not represent the whole spectrum of respiratory infections in the first 2 years. A memory bias was a possible confounding factor for the parents who filled in the score (as they completed it at the moment of discharge with the sole help of what they remembered), whereas quality and accuracy of clinical records could influence the results of the observers. Another reservation about the results presented in this study is that the professionals reviewing the clinical records were experienced pediatricians. Further testing may be required to ensure investigators with a different profile conduct the evaluation of clinical records with comparable precision. It could also be argued that clinicians may also bias the results as medical interventions influence the final outcome of the scoring system. However, the opposite could be argued too, as many decisions of pediatricians could be affected by the final score of any scale they use. Despite this potential risk of doctors’ decisions impacting the outcome of the scale, it was not possible to demonstrate that use of drugs, such as nebulized epinephrine or oral medication (corticosteroids or antibiotics), was associated with a significant increase in the final score in any of the two cohorts. In fact, severity range in the ReSVinet Scale correlated well with the definition of severity in the WDS, where medical interventions are not considered. It must be noted that there was a significant overlap between values that would correspond to “mild” and “moderate” episodes in the WDS, a finding that could be explained because the scale has only been applied to hospitalized children.

The final score value for patients in our setting did not show any difference dependent on the pathogen infecting them. In studies concerning other scoring systems, patients suffering from RSV had a higher total score [26]. Our scale failed to demonstrate any statistically significant impact of prematurity in the score, even though those children are at risk of worse episodes of bronchiolitis and many of them should receive prophylaxis against RSV [27]. Children of younger age in our study had a similar mean score to that obtained in older patients, although this finding lacked statistical significance. This can be explained by an insufficient number of subjects in both cohorts or the fact that we excluded children with chronic pulmonary conditions (which is a frequent problem in prematurely born babies); however, this outcome has already been described in medical literature [26], probably explained by the widely accepted measure of preemptively interning prematurely born children and younger infants. Aspects such as distance to a health provider, parental stress, or need for endovenous rehydration influence the decision of discharging a patient from the Emergency Department, and they are usually not gathered in respiratory scales nor clinical records. At the same time, a previous episode of wheezing was linked to a more severe disease in the cohort assessed by professionals but not in the parental cohort, which may be attributed to the significant lower mean age of the latter group. Neither the impact of maternal smoking during pregnancy nor breastfeeding were addressed in this study, although there is ample evidence of their influence in the severity of the disease [2].

The next steps for completing the validation of this score should include the application of this scale in outpatient settings, expansion of the age span of the scale to those children older than two years, and the inclusion of other pathologies which primarily manifest with respiratory difficulty in older children (asthma, atypical causes of pneumonia…) or in patients with underlying conditions. More importantly, the responsiveness of the ReSVinet scale should also be assessed. If the ReSVinet scale’s performance regarding responsiveness is acceptable, this scale will prove to be an important tool for the evaluation of infants suffering from respiratory infections.

Conclusions

Our results indicate that the ReSVinet severity score has adequate criterion validity, adequate construct validity, adequate inter-rater reliability, and appropriate usability for infants under two years of age hospitalized for acute respiratory infections, even when used by non medical-personnel or when evaluating patients using clinical records and therefore avoiding the need for in-person evaluation of the subject by a physician. Although this scale still does not meet all the criteria for a perfectly validated scale, it complies with far more of them than the vast majority of the alternatives currently available and used in the clinical practice.

Acknowledgments

Our most sincere thanks to the members of the ReSVinet group (Louis Bont¹, Anne Greenough², Harish Nair³, Paolo Manzoni⁴, Nikolaos Papadopoulos⁵, Eugenio Baraldi⁶, Brigitte Fauroux⁷, Terho Heikkinen⁸, Asunción Mejías⁹, Fernando Polack¹⁰, Mike Sharland¹¹, Octavio Ramilo¹², Renato Stein¹³) and to colleagues working in the FIVE research group (S. Alfayate Miguélez, C. Alvarez Alvarez, F. Álvez González, J.V. Arcos Machancoses, J. Arístegui Fernández, F. Baquero Artigao, E. Bernaola Iturbe, S. Bueno Pardo, MJ. Cabero Pérez, G. Cabrera Roca, C. Calvo, I.M. Ceballos Rodríguez, S. Cerdán Oncala, L.M. Ciria Calavia, J.A. Couceiro Gianzo, J.M. de Cea, F. de Juan Martín, M. Delgado Cardoso, G.M. Escudero Bueno, M. Fernández de Sevilla, L. Fernández Silveira, M. García Barreiro, M.J. García de Miguel, M.L. García-García, F. Giménez Sánchez, A. Gimeno Díaz de Atauri, M. González González, M.I. González Sánchez, M.I. González Tomé, B. González García, S. Guillén Martín, X. Hernández Fernández, J.F. Hurtado Díaz, M. Lillo Lillo, M. López Sousa, S. Martínez Megías, A.I. Menasalvas Ruiz, D. Moreno-Pérez, C. Otero Reigada, M.J. Peláez Cantero, R. Pérez Gorricho, M.E. Pérez Gutiérrez, R. Piñeiro Pérez, R. Rodríguez Fernández, P. Ruiz del Arbol Sánchez, M. Ruiz Jiménez, J.M. Rumbao Aguirre, M. Sánchez Forte, M.A. Tejero Hernández, M. Tobeña Rué, M. Triviño Rodríguez, A. Urda Cardona, N. Viguria Sánchez). We would also like to acknowledge all the members of the REGALIP research network (Miguel Ángel Fernández Pérez, José Manuel Fernández Bustillo, Mercedes del Río Garma, Josefina Pena Nieto, María Elena Álvarez Garnelo, Ángel López-Silvarrey Varela, Consuelo Alonso Alonso, Carmen Murias Taboada, Inés del Río Pastoriza, Julio Regueiro Martínez, Natalia García Sánchez, María Cristina López del Olmo, Lucía Castro Paz, Juan Manuel Sánchez Lastres, María Teresa Valls Durán, Manuel Sampedro Campos, Miguel Angel San José González, Patricia Vilas Rodríguez). Last and not least, we would like to express our admiration to those parents who cooperated with us in the development of this score while they took care of the most precious thing they have.