Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Development and Internal Validation of a Predictive Model Including Pulse Oximetry for Hospitalization of Under-Five Children in Bangladesh

  • Shahreen Raihana,

    Affiliation Centre for Child and Adolescent Health, International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka, Bangladesh

  • Dustin Dunsmuir,

    Affiliations Centre for International Child Health, British Columbia Children’s Hospital, Vancouver, British Columbia, Canada, The Department of Anesthesiology, Pharmacology & Therapeutics, The University of British Columbia, Vancouver, British Columbia, Canada

  • Tanvir Huda,

    Current address: School of Public Health, Sydney Medical School, University of Sydney, Sydney, Australia

    Affiliation Centre for Child and Adolescent Health, International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka, Bangladesh

  • Guohai Zhou,

    Affiliation The Department of Statistics, The University of British Columbia, Vancouver, British Columbia, Canada

  • Qazi Sadeq-ur Rahman,

    Affiliation Centre for Child and Adolescent Health, International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka, Bangladesh

  • Ainara Garde,

    Affiliation The Department of Electrical & Computer Engineering, The University of British Columbia, Vancouver, British Columbia, Canada

  • Md Moinuddin,

    Affiliation Centre for Child and Adolescent Health, International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka, Bangladesh

  • Walter Karlen,

    Affiliation The Department of Electrical & Computer Engineering, The University of British Columbia, Vancouver, British Columbia, Canada

  • Guy A. Dumont,

    Affiliations Centre for International Child Health, British Columbia Children’s Hospital, Vancouver, British Columbia, Canada, The Department of Electrical & Computer Engineering, The University of British Columbia, Vancouver, British Columbia, Canada

  • Niranjan Kissoon,

    Affiliations Centre for International Child Health, British Columbia Children’s Hospital, Vancouver, British Columbia, Canada, The Department of Pediatrics, The University of British Columbia, Vancouver, British Columbia, Canada

  • Shams El Arifeen,

    Affiliation Centre for Child and Adolescent Health, International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka, Bangladesh

  • Charles Larson,

    Affiliations Centre for International Child Health, British Columbia Children’s Hospital, Vancouver, British Columbia, Canada, The Department of Pediatrics, The University of British Columbia, Vancouver, British Columbia, Canada

  • J. Mark Ansermino

    anserminos@yahoo.ca

    Affiliations Centre for International Child Health, British Columbia Children’s Hospital, Vancouver, British Columbia, Canada, The Department of Anesthesiology, Pharmacology & Therapeutics, The University of British Columbia, Vancouver, British Columbia, Canada

Development and Internal Validation of a Predictive Model Including Pulse Oximetry for Hospitalization of Under-Five Children in Bangladesh

  • Shahreen Raihana, 
  • Dustin Dunsmuir, 
  • Tanvir Huda, 
  • Guohai Zhou, 
  • Qazi Sadeq-ur Rahman, 
  • Ainara Garde, 
  • Md Moinuddin, 
  • Walter Karlen, 
  • Guy A. Dumont, 
  • Niranjan Kissoon
PLOS
x

Correction

15 Jan 2016: Raihana S, Dunsmuir D, Huda T, Zhou G, Rahman QSu, et al. (2016) Correction: Development and Internal Validation of a Predictive Model Including Pulse Oximetry for Hospitalization of Under-Five Children in Bangladesh. PLOS ONE 11(1): e0147560. https://doi.org/10.1371/journal.pone.0147560 View correction

Abstract

Background

The reduction in the deaths of millions of children who die from infectious diseases requires early initiation of treatment and improved access to care available in health facilities. A major challenge is the lack of objective evidence to guide front line health workers in the community to recognize critical illness in children earlier in their course.

Methods

We undertook a prospective observational study of children less than 5 years of age presenting at the outpatient or emergency department of a rural tertiary care hospital between October 2012 and April 2013. Study physicians collected clinical signs and symptoms from the facility records, and with a mobile application performed recordings of oxygen saturation, heart rate and respiratory rate. Facility physicians decided the need for hospital admission without knowledge of the oxygen saturation. Multiple logistic predictive models were tested.

Findings

Twenty-five percent of the 3374 assessed children, with a median (interquartile range) age of 1.02 (0.42–2.24), were admitted to hospital. We were unable to contact 20% of subjects after their visit. A logistic regression model using continuous oxygen saturation, respiratory rate, temperature and age combined with dichotomous signs of chest indrawing, lethargy, irritability and symptoms of cough, diarrhea and fast or difficult breathing predicted admission to hospital with an area under the receiver operating characteristic curve of 0.89 (95% confidence interval -CI: 0.87 to 0.90). At a risk threshold of 25% for admission, the sensitivity was 77% (95% CI: 74% to 80%), specificity was 87% (95% CI: 86% to 88%), positive predictive value was 70% (95% CI: 67% to 73%) and negative predictive value was 91% (95% CI: 90% to 92%).

Conclusion

A model using oxygen saturation, respiratory rate and temperature in combination with readily obtained clinical signs and symptoms predicted the need for hospitalization of critically ill children. External validation of this model in a community setting will be required before adoption into clinical practice.

Introduction

In 2013, an estimated 6.3 million children died worldwide before their 5th birthday [1]. Approximately 50% of these deaths are due to infectious causes that may be preventable if detected early and treated with simple cost effective therapies such as appropriate antibiotics and oral or intravenous fluids. The lack of clinical expertise for early and accurate diagnosis of serious infection in children less than 5 years of age, especially in community settings, is a significant roadblock to reducing mortality [2]. Integrated community case management (iCCM) is an attempt to address this issue. It is a strategy to train, support, and supply community health workers to diagnose and treat sick children of families with difficult access to case management at health facilities. However, even in the developed world, deaths from bacterial infection may be associated with a delay in seeking care, under evaluation of severity, and delayed antibiotic administration [3].

The iCCM can reduce mortality in diseases such as pneumonia by up to 70% [4]. However, case finding in the community is essential as it has been estimated that over 80% of deaths in children with pneumonia occur outside of facilities [5]. There is increasing evidence that large scale implementation of iCCM can reduce childhood mortality in diverse clinical contexts [6]. iCCM diagnosis is based on signs and symptoms that are designed to have a high degree of sensitivity in order to avoid missing any seriously sick child. However, the high sensitivity of the iCCM algorithms may result in high rates of referral of non-severe cases, with an increased burden on already under resourced and weak health care systems. An objective, point of care tool that would accurately identify children at higher or lower risk would help refine decisions about case management, such as community case management or the need for referral for higher level supportive care. At the community level, the challenge is identifying children who are critically ill as soon as possible to allow time for them to reach the nearest health facility.

Of particular interest in this study is the added predictive value of oxygen saturation (SpO2). Monitoring SpO2 with pulse oximetry in resource-poor settings, when coupled with a reliable oxygen supply has been shown to reduce mortality from pneumonia by as much as 35% [7,8,9]. In addition, a single estimate of SpO2 has been shown to be a significant predictor of radiological pneumonia in children [10]. However, the clinical utility of SpO2 as an isolated measurement is hampered by the lack of a clear threshold limit and the lack of availability of pulse oximetry for assessment of the illness severity in children in the community [11]. SpO2 in combination with other clinical signs and symptoms has been used to predict the need for admission to hospital in children with bronchiolitis [12], asthma [13] and pneumonia [14] in high resource settings and as a prognostic predictor of mortality in low resource settings [9,15,16].

The purpose of this study was to develop and internally validate a simple model that would be predictive of the need for admission to a health facility using clinical signs and symptoms in combination with oxygen saturation and heart rate measured by pulse oximetry. We chose to study children presenting at this facility, rather than in the community, to ensure that we would have reliable assessments performed by the experienced facility physicians. The practicality of collecting a large enough sample size and the need for good outcome data dictated this decision. Moreover, this tertiary hospital was the only health resource for the catchment population.

Methods

Ethics Statement

Institutional review board approval was obtained from the Research Review Committee (RRC) and Ethical Review Committee (ERC) of the Institutional Review Board of International Centre for Diarrhoeal Disease Research, Bangladesh and the University of British Columbia/Children’s and Women’s Health Centre Research Ethics Board. We explained the study and obtained written informed consent from the caretaker accompanying the child.

Study Design and Population

We conducted a prospective observational study at the Kumudini Women’s Medical College Hospital’s (KWMCH). KWMCH is a not-for-profit private tertiary level hospital located in the Mirzapur sub-district of Tangail district in Bangladesh, with more than 12,000 annual outpatient visits for children less than 5 years of age. KWMCH provides primary level care to a catchment population of approximately 500,000 people.

All children less than 5 years of age presenting at the outpatient department or emergency department from October 2012 to April 2013 (winter is from November to February) were eligible for inclusion in this study. We assessed the first presenting case of a day followed by the next eligible case in the queue after completion of the preceding assessment. Children were seen based on time of arrival and no triage process occurred before cases were selected. We excluded children presenting with chronic diseases or those who had documented low SpO2 levels due to other conditions, such as cardiac disease.

The attending hospital physician undertook a complete history and physical examination and made a treatment recommendation. Our study physicians then extracted data from the hospital medical record and documented all demographic data, clinical signs, and symptoms such as weight, respiratory rate, and temperature in a structured case report form. In addition, the study physicians performed a one minute recording of the photoplethysmographic (PPG) waveform and extracted SpO2 and heart rate (HR) using an Xpod® external original equipment manufacturer (OEM) pulse oximeter (Nonin Medical Inc., Plymouth, USA; model 3012LP) connected to a mobile device the 4th generation iPod touch (Apple Inc., Cuppertino, USA) and a custom data collection app (‘PhoneOxR2’ version 2.2.3 [17]). The interface of the PhoneOxR2 application displayed the PPG waveform with a color coded (8 shades of green to red) background based on the calculated signal quality index (SQI). The SQI was calculated using the amplitude of the PPG and variability of the SpO2 and HR and pulse oximeter module generated alerts. The research physicians were trained to optimize the SQI for data quality purposes. The recorded SpO2 was kept hidden from the facility and study physicians, as the SpO2 measurement was not routinely available in this clinical setting. The SpO2 reading was not used for making any clinical decisions and was only available during analysis of the data. The application also included the measurement of respiratory rate by tapping the screen of the mobile device with each breath [18]. This application was built using Lambda Native [19], a cross-platform open-source development environment written in Scheme. In PhoneOxR2, physiological trends from PPG were recorded to a comma-separated-value file at 1 Hz and the PPG waveform was recorded at 75 Hz. No decisions on hospitalization were made based on the research data collection.

The clinical course of all hospitalized and non-hospitalized children was followed and documented. Children who were sent home from the outpatient facility or emergency department were followed-up with a telephone call 96 to 120 hours after their initial visit at the facility.

Data Quality Management

All case record forms were entered using double data entry and checked for completeness and accuracy. Validation rules such as range check, uniqueness check, and skip rules were used with the data entry interface to verify the consistency of data. The PhoneOxR2 software provided automatic encryption and synchronization through a wireless connection and Application Programming Interface (API) to a web-based data collection system (REDCap) [20].

Statistical Analysis

Statistical analyses were performed using STATA/SE 12.1 (StataCorp LP, College Station, USA) and R (3.1.0) [21]. The primary outcome for the analysis was the need for hospitalization for ongoing care. Admission was considered necessary if children were admitted and stayed for more than 24 hours in the hospital, were sent home initially but admitted at a subsequent visit within 1 week, were advised for admission but the advice was not followed, or were transferred or referred to another facility providing a higher level of care for admission. We considered all children who were sent home or admitted and discharged within 24 hours as not having illness of sufficient severity to require hospitalization for ongoing care.

Candidate Variable Selection

The candidate predictor variables were selected based on a literature review, a priori consensus of clinical importance, prevalence in the study population and ease of measurement in resource-poor settings. The relationship between predictor variables and the outcome was assessed using univariate logistic regression using odds ratio [OR] (with confidence intervals). All continuous variables were assessed for co-linearity. Nonlinear predictors were transformed using logarithmic, exponential or square transformations, based on the pattern of nonlinearity. A weight-for-age z-score was calculated. A physiological transformation based on the shape of the relationship between SpO2 and virtual shunt [22] was used to linearize the SpO2 measurements [70*log10(104-SpO2)-57]. This virtual shunt was used as an index of disease severity. To evaluate the added value of pulse oximetry, we calculated the net reclassification improvement (NRI) and integrated discrimination improvement (IDI) of the model with SpO2 [23].

To reduce co-linearity, high multi-co-linearity, as measured by correlation coefficient (-0.9 < corr < +0.9), and variable inflation factor (≥ 10.0) between variables was determined and only the more clinically relevant variable of a pair of highly correlated variables was retained. The majority of the categorical variables were found to be highly correlated. When a high degree of correlation existed between two categorical variables they were re-coded as a combined indicator variable when possible, but were otherwise retained in the model, if considered important in the outcome prediction.

Missing Values

Predictor variables with a low number of missing values (< 10 cases missing) had these cases excluded. The SQI of the recorded PPG waveform was low in a number of children and was considered clinically related to both the age and disease severity of the child. A new variable based on the SQI and an estimate of virtual shunt from the SpO2 was included as a candidate predictor [21]. Missing data in weight for age z-scores were calculated based on imputed values from age and sex variables. The missing values for weight, respiratory rate and temperature were imputed using a multiple imputation approach [24] to avoid potential bias introduced by ignoring missing data uncertainty. Each missing value was imputed using five plausible values, leading to five complete datasets. As the rate of missing information was low, five imputations were considered adequate [25]. Standard logistic regression analysis was conducted for each of the five complete datasets. The results were later pooled to yield estimates, confidence intervals and p-values that incorporated missing data uncertainty. Association between the primary outcome (i.e., cases requiring admission), and each of the clinical signs and symptoms was examined in a series of univariate analyses.

Model Building

A series of multivariate logistic regression models were incorporated to identify the individual variables that predicted the primary outcome. A stepwise selection approach was applied, starting with an empty model and adding or removing one predictor at each step. Models were compared using the BIC (Bayes Information Criterion), AIC (Akaike Information Criterion) and LASSO (Least Absolute Shrinkage and Selection Operator) methods. Further inclusion into the list of variables was made based on clinical knowledge. The Hosmer-Lemeshow goodness-of-fit test was conducted to evaluate the calibration of the final model.

The sample size required for model development was determined on the basis of the minimum standard of 10 events per effective variable considered in the model according to the formula N = (n*10)/I where N is the sample size, n is the number of candidate predictor variables, and I is the estimated event rate in the population [26]. An estimated event rate of 20% was used for a model with 39 effective candidate predictor variables. A minimum of 1950 children were estimated to be required. It was estimated that a 6 month data period would provide a significantly large sample size.

Internal Model Validation

The bootstrap procedure was employed for internal validation [27]. The predictor selection was applied to each bootstrap sample to obtain a final model, and the optimism was estimated by comparing the final model performance to the original data for each bootstrap sample. The bootstrap corrected area under the curve was computed by subtracting the optimism from the original area under the curve.

Model Performance

To assess the calibration capability of the model to identify children who required admission to a facility, we used crossover testing on the same sample set to test if the prediction rule successfully predicted individual case requirements. We calculated the classification performance (sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios) for each possible risk threshold. A risk stratification table was used to assess the distribution of the outcome in each risk category stratified by the model prediction [28]. We selected a range of possible thresholds to identify the trade-off between identifying sick children and unnecessary admission of cases that could be used in different clinical contexts. A weighted classification score, (the number of correct identifications of true positives) + (the number of correct identifications of true negatives), was calculated using various ratios of false negative cases to false positive cases (1:3; 1:5 and 1:10) and plotted to identify important inflection points. The final threshold was chosen to maximize the weighted classification score.

Results

We enrolled 3374 children less than 5 years of age in the study. The median (interquartile range) age was 1.02 (0.42–2.24) years and 62% were male. Twenty-five percent were admitted and 102 of these children (11.9%) were discharged within 24 hours. Thirteen (< 1%) children who were sent home following the initial visit were subsequently admitted to KWMCH or elsewhere. One child died at home and 687 (20%) were lost to follow-up (Fig 1). The children lost to follow- up were on average older, had lower weight for age z-scores, a higher prevalence of fever and a lower prevalence of diarrhea. No other systematic differences were found between the children lost to follow-up and those that had been followed-up (see S1 File for more details).

The most common diagnoses in children requiring admission were acute lower respiratory tract infections (40.6%), and acute upper respiratory or eye or ear infection (20.1%). In children discharged home, the most common diagnosis was acute upper respiratory or eye or ear infection (50.4%), diarrhea (16.1%), fever of unknown cause (14%) and other gastrointestinal complains (8.7%). The prevalence of acute lower respiratory infection in children not requiring admission was very low (1.2%) (Table 1).

thumbnail
Table 1. Primary diagnosis reported by facility physician for children included in the study*.

https://doi.org/10.1371/journal.pone.0143213.t001

Univariate Analysis

Children who required admission were younger, had lower median weight for age z-scores and SpO2 values, and had higher likelihood of breathing difficulty, fever, cough, chest in-drawing and lethargy compared to children who did not require admission. Children who were sent home were more likely to have symptoms such as diarrhea and vomiting although these were more likely to be present for more than 24 hours in those requiring admission. The univariate analysis with lethargy, low SpO2 or fast breathing revealed an area under the receiver operating characteristic curve (AUC) of over 0.70 (Table 2).

thumbnail
Table 2. Predictor variable distribution and odds ratio for study population#.

https://doi.org/10.1371/journal.pone.0143213.t002

Multivariate Analysis and the Final Model

BIC consistently identified the most parsimonious model. AIC and the LASSO methods did suggest a few additional predictors, however, their inclusion only marginally improved prediction. Therefore, the set of predictors identified by BIC were used. The predictor selection was repeated for each of the five complete datasets following imputation for missing variables (12.77% for weight, 1.04% for respiratory rate and 0.42% for temperature) and the same set of predictors were consistently identified in four of the five complete datasets.

The final model was reduced to 10 predictor variables (Table 3). This final model had an AUC of 0.89 (95% CI: 0.87 to 0.90) (Fig 2). The Hosmer-Lemeshow goodness-of-fit test indicated evidence of lack of fit (p = 0.01), however, a closer look at the standardized residuals within each decile reveals that all discrepancies come from the 20% sickest cases, with the highest 20% of predicted admission probabilities. The exclusion of subjects with convulsions or who were unconscious significantly improved the fit (Fig 3).

thumbnail
Table 3. Adjusted odds ratios of predictor variables in the final prediction model.

https://doi.org/10.1371/journal.pone.0143213.t003

thumbnail
Fig 2. Receiver operating characteristic curve of the final model in the study cohort.

AUC ROC = area under the curve of the receiver operating characteristic. Sens = sensitivity. Spec = specificity. PPV = positive predictive value. NPV = negative predictive value.

https://doi.org/10.1371/journal.pone.0143213.g002

thumbnail
Fig 3. Calibration plot of the final 10-predictor model applied to the 3263 cases excluding subjects who were unconscious or who had experienced convulsions (Hosmer-Lemeshow goodness-of-fit p-value = 0.53).

The 45 degree straight line corresponds to the line of perfect calibration on which model predicted risks coincide with the observed risks.

https://doi.org/10.1371/journal.pone.0143213.g003

The addition of SpO2 to the final predictors exhibited 7% net improvement in the classification of the non-admitted and a 4% net improvement in the classification of the admitted subjects, at a risk threshold of 25%. The IDI was 0.005 (95% CI: 0.002 to 0.007, p-value <0.001).

Risk Prediction

At a risk threshold of 25% the model can correctly classify 77% of children requiring admission (sensitivity) and 87% of children not requiring admission (specificity), with positive predictive value (PPV) of 70% and negative predictive value (NPV) 91% (Fig 2). Lowering the risk threshold to 15% increases the sensitivity to 85% and decreased the specificity to 75%. The PPV decreases to 57% and NPV increases to 93% (Table 4 and Fig 4).

thumbnail
Table 4. Classification performance measures of final prediction model at different risk thresholds.

https://doi.org/10.1371/journal.pone.0143213.t004

thumbnail
Fig 4. Weighted classification score for the full range of thresholds using different trade-offs between false negative and false positive cases.

https://doi.org/10.1371/journal.pone.0143213.g004

Internal Validation

Each of the 10 final selected predictors was also selected in at least 80% of the bootstrap samples indicating a stable predictive capability. The bootstrap corrected AUC was 0.87 (95% CI: 0.86 to 0.89). The model stability was further ascertained to be consistent for each of the 5 imputed datasets. The details of internal validation can be found in S2 File.

Discussion

Summary of Findings

We have developed and internally validated a clinical prediction model to predict the need for hospitalization in children under 5 years of age presenting at an outpatient department. Breathing difficulty, chest in-drawing, irritability, lethargy, an increased temperature, oxygen saturation and respiratory rate were the strongest predictors of the need for hospitalization in line with previous studies and current international guidelines. The presence of cough was a positive predictor in the univariate prediction but cough and diarrhea were protective in predicting admission. A model with 10 predictors has provided an AUC of 0.89 (95% CI: 0.87 to 0.90) and at a risk threshold of 25% has PPV value 70% (95% CI: 67% to 73%) and NPV of 91% (95% CI: 90% to 92%).

Importance of Study

This prediction model has concentrated on variables that could be readily measured in the community. Predictors that have been used by more experienced clinicians such as ‘gut feeling’ and chest auscultation were not considered [29]. In addition, the use of a pulse oximeter provides an objective measure of SpO2 and HR but would be dependent on the availability of the device in the community. The barrier of availability can be overcome by using a mobile app and low cost sensor attached to the same mobile device [30].

There is increasing evidence to suggest that mHealth (the use of mobile devices with software applications to provide health services and manage patient information) can be used to deliver increased and enhanced health care services to individuals and communities, while helping to strengthen health systems and retain community health workers [31]. The computing power and display capability of even the entry level smartphones in low resource settings provide the opportunity to repurpose these as medical diagnostic devices for community measurement of vital signs [32]. Mobile devices also provide a platform to implement clinical guidelines such as iCCM and prediction models in low resource settings, with low training and follow-up support overhead and at an affordable cost.

The inclusion of vital signs including SpO2 as continuous variables is a significant advance in the development of this prediction model. It has previously been demonstrated that using categorical variables in prediction models for HR and respiratory rate results in significant information loss and lower predictive ability [33]. The inclusion of multiple continuous vital sign measurements in the model allows us to capture the known clinical interactions between variables such as respiratory rate and age or temperature [34]. While a low SpO2 has been clearly demonstrated to predict both the need for hospitalization and mortality [35], many of these studies were performed at high altitude. The lower partial pressure of inspired oxygen at altitude moves the normal range of the SpO2 curve to the steeper and more linear slope of the S shaped dissociation curve [36, 37]. We have chosen to perform a transformation based on the physiological concept of virtual shunt [23]. This transformation will linearize the flat end of the oxygen curve and allows us to use a continuous variable for SpO2. The transformation overcomes the limitations of using a single threshold value for SpO2 such as that adopted by the World Health Organization (WHO) [38]. The barometric pressure available with many mobile devices will also allow us to correct this SpO2 variable for the change in inspired oxygen partial pressures when the prediction is used at different altitudes [8].

Interpreting Risk

A prediction model is only clinically useful if it is able to support clinical decision making. At a threshold of 25% the model showed a 77% sensitivity, 87% specificity, 70% positive predictive value and 91% negative predictive value for need of hospital admission. The sensitivity can be further increased (88%) at a lower risk threshold but at a cost of a decrease in specificity (69%). In a similar clinical setting, the use of Integrated Management of Childhood Illness guidelines resulted in a sensitivity of 86% and specificity of 64% for predicting hospital admission [39]. There is always a trade-off between acceptable risk and undue burden to the family, patient and health system. A marginal score could be used to direct further reassessment of the child to prevent depriving them of potentially lifesaving interventions. The appropriate risk threshold and actions taken would depend on the specific local context such as transportation, local treatment options and family choice. For developing countries with limited resources and where the health system is already over burdened due to several supply side constraints [35], it is probably more appropriate to use a guideline with a higher specificity [40]. The lack of optimal sensitivity would mean that some children requiring admission would be sent home. Parents would need to be encouraged to return for further assessment even if they fell below the risk threshold and they were still concerned about the clinical condition of their child.

Model Calibration

The lack of calibration of the model was not unexpected. During the predictor variable selection process we did identify certain danger signs, such as unconsciousness or convulsions, which would indicate the need for admission to a facility without the need to calculate a prediction score. The Hosmer-Lemeshow test is known to be sensitive to sample size when there is small deviation from the perfect fit. We believe that as the fit is poor only in the highest 20% of predicted admission probabilities, the model with 10 predictors provides an acceptable calibration for the 3374 eligible cases. If we remove the 105 (3%) cases with unconsciousness and convulsions, the derived final model still contains the same 10 predictors with similar coefficients, but with significantly improved calibration (p = 0.53).

Limitations

There are a number of potential limitations of this study. The data was collected at a tertiary level facility rather than in the community. We would anticipate that the case mix would not be significantly different than that encountered in the community. This assumption will need to be validated in future studies. There was a risk that the facility physicians would be overly conservative in their decision making knowing that their clinical outcomes were being followed. To reduce this risk of bias in the model we excluded cases that were discharged within 24 hours, we did not provide the pulse oximetry data to the facility physicians and we used dedicated study physicians to collect the data. To avoid missing any cases that were inadvisably sent home we attempted to contact all families sent home, we specifically looked for children who were admitted at a subsequent visit and we documented the families who chose not to follow the advice for admission to the facility. However, these cases only constituted a small fraction of the total number of cases studied. There was significant variability in the diagnosis listed by the facility physician in the children that were not admitted to the facility, while the diagnoses of the children admitted to the facility were much easier to verify based on further investigations and the clinical course of the disease. There may have been children who improved very rapidly and warranted admission and early discharge based on a response to treatment. The decision to classify children based on these criteria was made a priori. There were a very low number (107) of children discharged within 24 hours.

Conclusion

This simple model was able to predict which children were admitted to a health facility. However, the model requires prospective external validation before introduction into clinical practice. The use of SpO2 as a continuous variable with appropriate transformation based on well described physiological principles reduces the information loss produced by using a single threshold for classification of risk.

Supporting Information

S1 File. Characteristics of children lost to follow-up.

https://doi.org/10.1371/journal.pone.0143213.s001

(DOC)

Acknowledgments

We would like to thank Rollin Brant from the Department of Statistics for expert statistical advice and support, Joanne Lim for her editorial assistance and the study coordinators from The University of British Columbia and British Columbia Children’s Hospital who contributed to the study design, processing of the PPG waveforms, data analysis and manuscript writing.

Author Contributions

Conceived and designed the experiments: SR DD TH QSR AG MM WK GAD NK SEA CL JMA. Performed the experiments: SR TH QR MM SEA. Analyzed the data: SR DD TH GZ QSR AG MM WK GAD JMA. Contributed reagents/materials/analysis tools: SR DD TH GZ QSR AG MM WK JMA. Wrote the paper: SR DD TH GZ QSR AG MM WK GAD NK SEA CL JMA.

References

  1. 1. Liu L, Oza S, Hogan D, Perin J, Rudan I, Lawn J, et al. Global, regional, and national causes of child mortality in 2000–13, with projections to inform post-2015 priorities: an updated systematic analysis. Lancet. 2015; 385 (9966): 430–40. pmid:25280870
  2. 2. UNICEF. Committing to Child Survival: A Promise Renewed. 2012. UNICEF Progress Report 2012. Available: http://www.un.org/ru/publications/pdfs/renewed_%20progress%20report%202012.pdf
  3. 3. Launay E, Gras-Le Guen C, Martinot A, Assathiany R, Martin E, Blanchais T, et al. Why children with severe bacterial infection die: A population–based study of determinants and consequences of suboptimal care with a special emphasis on methodological issues. PLoS One. 2014; 9 (9): e107286. pmid:25247401
  4. 4. Theodoratou E, Al-Jilaihawi S, Woodward F, Ferguson J, Jhass A, Ballliet M, et al. The effect of case management on childhood pneumonia mortality in developing countries. Int J Epidemiol. 2010; 39(Suppl. 1):i155–71. pmid:20348118
  5. 5. Nair H, Simões EA, Rudan I, Gessner BD, Azziz-Baumgartner E, Zhang JSF, et al. Global and regional burden of hospital admissions for severe acute lower respiratory infections in young children in 2010: a systematic analysis. Lancet. 2013; 6736(12): 4–6.
  6. 6. Marsh DR, Hamer DH, Pagnoni F, Peterson S. Introduction to a special supplement: Evidence for the implementation, effects, and impact of the integrated community case management strategy to treat childhood infection. Am J Trop Med Hyg. 2012; 87(Suppl 5): 2–5.
  7. 7. Duke T, Wandi F, Jonathan M, Matai S, Kaupa M, Saavu M, et al. Improved oxygen systems for childhood pneumonia: a multihospital effectiveness study in Papua New Guinea. Lancet. 2008; 372(9646): 1328–1333. pmid:18708248
  8. 8. Duke T, Subhi R, Peel D, Frey B. Pulse oximetry: technology to reduce child mortality in developing countries. Ann Trop Paediatr. 2009; 29(3): 165–75. pmid:19689857
  9. 9. Subhi R, Adamson M, Campbell H, Weber M, Smith K, Duke T, et al. The prevalence of hypoxaemia among ill children in developing countries: a systematic review. Lancet Infect Dis. 2009; 9(4): 219–27. pmid:19324294
  10. 10. Modi P, Munyanezy RB, Goldberg E, Choy G, Shailam R, Sagar P, et al. Oxygen saturation can predict pediatric pneumonia in a resource-limited setting. J Emerg Med. 2013; 45(5): 752–60. pmid:23937809
  11. 11. Ginsburg AS, Van Cleve WC, Thompson MI, English M. Oxygen and pulse oximetry in childhood pneumonia: a survey of healthcare providers in resource-limited settings. J Trop Pediatr. 2012; 58(5):389–393. pmid:22170511
  12. 12. Walsh P, Rothenberg SJ, O’Doherty S, Hoey H, Healy R. A validated clinical model to predict the need for admission and length of stay in children with acute bronchiolitis. Eur J Emerg Med. 2004; 11: 265–272. pmid:15359199
  13. 13. Keahey L, Bulloch B, Becker AB, Pollack CV, Clark S, Camargo C, et al. Initial oxygen saturation as a predictor of admission in children presenting to the emergency department with acute asthma. Ann Emerg Med. 2002; 40(September): 300–307.
  14. 14. Van den Bruel A, Haj-Hassan T, Thompson M, Buntinx F, Mant D, et al. Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet. 2010; 375(9717): 834–845. pmid:20132979
  15. 15. Berkley JA, Ross A, Mwangi I, Osier FH, Mohammed M, Shebbe M, et al. Prognostic indicators of early and late death in children admitted to district hospital in Kenya: cohort study. BMJ. 2003; 326(7385): 361. pmid:12586667
  16. 16. Laman M, Ripa P, Vince J, Tefuarani N. Can clinical signs predict hypoxaemia in Papua New Guinean children with moderate and severe pneumonia? Ann Trop Paediatr. 2005; 25(1): 23–27. pmid:15814045
  17. 17. Dunsmuir D, Petersen C, Karlen W, Lim J, Dumont GA, Ansermino JM. The Phone Oximeter for mobile spot-check. Anesth Analg. 2012; 115(2 Suppl): S21.
  18. 18. Karlen W, Gan H, Chiu M, Dunsmuir D, Zhou G, Dumont GA, et al. Improving the accuracy and efficiency of respiratory rate measurements in children using mobile devices. PLoS One. 2014; 9(6): e99266. pmid:24919062
  19. 19. Petersen CL, Görges M, Dunsmuir D, Ansermino JM, Dumont GA. Experience report: functional programming of mHealth applications. In Proceedings of the 18th ACM SIGPLAN international conference on Functional programming. 2013; 48 (9): 357–362.
  20. 20. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2): 377–381. pmid:18929686
  21. 21. R Development Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: the R Foundation for Statistical Computing. 2012. ISBN: 3-900051-07-0.
  22. 22. Benatar SR, Hewlett AM, Nunn JF. The use of iso-shunt for control of oxygen therapy. Br J Anaesth. 1973; 45(7): 711–718. pmid:4581075
  23. 23. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27 (2): 157–172; discussion 207–212. pmid:17569110
  24. 24. Little RJA and Rubin DB. Statistical analysis with missing data. 2nd edition. Hoboken: Wiley-Interscience; 2002.
  25. 25. Schafer JL. Multiple imputation: a primer. Stat Methods Med Res. 1999; 8(1): 3–15. pmid:10347857
  26. 26. Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996; 15: 361–387. pmid:8668867
  27. 27. Steyerberg EW, Harrell FE Jr, Borsboom GJ, Vegouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001; 54(8): 774–81. pmid:11470385
  28. 28. Janes H, Pepe MS, Gu W. Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med. 2008; 149: 751–760. pmid:19017593
  29. 29. Thompson M, van den Bruel A, Verbakel J, Lakhanpaul M, Haj-Hassan T, Stevens R, et al. Systematic review and validation of prediction rules for identifying children with serious infections in emergency departments and urgent-access primary care. Health Technol Assess. 2012; 16(15): 1–99. pmid:22452986
  30. 30. Petersen CL, Chen TP, Ansermino JM, Dumont GA. Design and evaluation of a low-cost smartphone pulse oximeter. Sensors. 2013; 13(12):16882–93. pmid:24322563
  31. 31. Källander K, Tibenderana JK, Akpogheneta OJ, Strachan DL, Hill Z, ten Asbroek AH, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low- and middle-income countries: A review. J Med Internet Res. 2013; 15(1): e17. pmid:23353680
  32. 32. Ansermino JM. Universal access to essential vital signs monitoring. Anesth Analg. 2013; 117(4): 883–890. pmid:24023026
  33. 33. Spruijt B, Vergouwe Y, Nijman RG, Thompson M, Oostenbrink R. Vital signs should be maintained as continuous variables when predicting bacterial infections in febrile children. J Clin Epidemiol. 2013; 66(4): 453–7. pmid:23306061
  34. 34. Nijman RG, Thompson M, van Veen M, Perera R, Moll HA, Oostenbrink R. Derivation and validation of age and temperature specific reference values and centile charts to predict lower respiratory tract infection in children with fever: prospective observational study. BMJ. 2012; 345(July): e4224.
  35. 35. Onyango FE, Steinhoff MC, Wafula EM, Wariua S, Musia J, Kitonyi J. Hypoxaemia in young Kenyan children with acute lower respiratory infection. BMJ. 1993; 306(6878): 612–5. pmid:8369033
  36. 36. West JB. Respiratory Physiology: The Essentials. 9th edition. Philadelphia: Lippincott Williams and Wilkins, 2011. ISBN-13: 978–1609136406
  37. 37. Severinghaus JW. Simple, accurate equations for human blood O2 dissociation computations. J Appl Physiol Respir Environ Exerc Physiol. 1979; 46(3):599–602. pmid:35496
  38. 38. World Health Organization. Revised WHO classification and treatment of pneumonia in children at health facilities: evidence summaries. 2014. Available: http://www.who.int/maternal_child_adolescent/documents/child-pneumonia-treatment/en/
  39. 39. Kalter HD, Schillinger JA, Hossain M, Burnham G, Saha S, de Wit V, et al. Identifying sick children requiring referral to hospital in Bangladesh. Bull World Health Organ. 1997; 75 Suppl 1:65–75. pmid:9529719
  40. 40. Bang AT, Reddy HM, Deshmukh MD, Baitule SB, Bang R. Neonatal and infant mortality in the ten years (1993 to 2003) of the Gadchiroli field trial: effect of home-based neonatal care. J Perinatol. 2005; 25 Suppl 1: S92–S107. pmid:15791283