To assess whether a patient’s in-hospital length of stay (LOS) and mortality can be explained by early objective and/or physicians’ subjective-risk assessments.
Data Sources/Study Setting
Analysis of a detailed dataset of 1,021 patients admitted to a large U.S. hospital between January and September 2014.
We empirically test the explanatory power of objective and subjective early-risk assessments using various linear and logistic regression models.
The objective measures of early warning can only weakly explain LOS and mortality. When controlled for various vital signs and demographics, objective signs lose their explanatory power. LOS and death are more associated with physicians’ early subjective risk assessments than the objective measures.
Citation: Azadeh-Fard N, Ghaffarzadegan N, Camelio JA (2016) Can a Patient’s In-Hospital Length of Stay and Mortality Be Explained by Early-Risk Assessments? PLoS ONE 11(9): e0162976. https://doi.org/10.1371/journal.pone.0162976
Editor: Delmiro Fernandez-Reyes, University College London, UNITED KINGDOM
Received: March 7, 2016; Accepted: August 31, 2016; Published: September 15, 2016
Copyright: © 2016 Azadeh-Fard et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are provided in the Supporting Information files.
Funding: The authors received no specific funding for conducting this study.
Competing interests: The authors have declared that no competing interests exist.
Risk evaluation and control are important components of healthcare operations. Ideally, providers would like to be able to predict health risks early in hospital admissions to take subsequent controlling actions. Different methods and techniques have been developed for this purpose, one of which is the early warning system. An early warning system is a measurement tool to assess patient health risks objectively and to quickly determine the degree of illness .
Several major early warning systems exist and each type offers a slight variation in risk- assessment parameters. The Modified Early Warning Score (MEWS) , the focus in our study, is a commonly used triage tool to quickly determine the severity of a hospitalized patient’s illness [2–5]. The MEWS system utilizes five major vital signs—systolic blood pressure, pulse rate, respiratory rate, temperature, and the AVPU score (A: alert, V: responding to voice, P: responding to painful stimuli, U: unresponsive)—to assign an aggregate number to each patient, which comprises the patient’s health risk. Obtaining a MEWS involves assigning a number between 0 and 3 to each of the 5 vital signs. The sum of the 5 numbers yields the patient’s total MEWS score—between 0 and 15. A total score of four or higher prompts a nurse to call the patient’s physician or the hospital’s rapid response team (for more information, see reference ). Other early warning systems follow a similar logic. The Standardized Early Warning System (SEWS)  adds oxygen saturation level (SpO2%) to the five MEWS vital signs to detect a patient’s deterioration. The Decision-Tree Early Warning Score (DTEWS)  is a decision-tree analysis based on a database of vital signs. The National Early Warning Score (NEWS) [8–11] utilizes seven variables to identify deteriorating patients: respiratory rate, oxygen saturation, any supplemental oxygen, temperature, systolic blood pressure, heart rate, and level of consciousness.
The main use of early warning scores is to provide early warnings to health providers to spur quick, preventive reactions. Some experts have argued that these measures in fact have much more to offer, such as helping to predict patients’ length of stay (LOS) in hospitals or health outcomes—e.g., the chance of in-hospital mortality [4, 6, 12, 13]. Given the importance of LOS and mortality in assessing healthcare quality, resource allocations, and costs , the hope is that early warning scores can make it possible to explain and improve hospital utilization and outcomes.
While a warning system can be an efficient and rapid way of reducing or preventing life-threatening events, there is mixed evidence regarding the effectiveness of early warning scores. On the one hand, MEWS can predict an increased risk of death or admission to an intensive care unit (ICU) or high dependency unit (HDU) . MEWS can also be used to identify patients who need hospital admission as well as those at higher risk of in-hospital death . The proportion of patients admitted and who subsequently died in the hospital was significantly higher for higher MEWS values . SEWS have also been correlated with a patient’s LOS . On the other hand, some studies have suggested that further work is needed to derive and validate early warning scores, and that scores that utilize inappropriate parameters and cut-off points should be replaced with those with higher diagnostic accuracy [15–18].
A potential limitation to these and many other similar studies is a lack of detailed administrative-level data on patient and physician characteristics. Some of these studies focus on assessing simple correlations between two variables without controlling for variation across patients or across physicians. Thus, whether early warning scores influence physicians’ diagnoses and risk assessments remains an open question. In other words, we do not know if physicians are actually using these measures and if such measures are preferred over early subjective assessments.
In this study, we assess whether a patient’s admission MEWS and vital signs are associated with LOS and in-hospital mortality. We also compare and contrast MEWS as an objective measure with physicians’ subjective-risk assessments early after admission.
We collected detailed data for 1,021 randomly selected patients admitted between January and September 2014 to an over-500-bed medical center (Lewis Gale Medical Center) in the state of Virginia. Our study was approved by the Lewis Gale Medical Center Education Department as patient records were anonymized and de-identified prior to analysis and the dataset is considered exempt by Virginia Tech’s Institutional Review Board. Patient records were accessed through the electronic patient management systems (MEDITECH and Crimson), which contain full patient demographic data along with individual records and medical documents including various vital signs, MEWS, clinicians’ early assessments of severity level and mortality risk, and patient outcomes such as hospital LOS and disposition conditions. Physicians’ identifiers were included in the data. Newborns and patients admitted to the ICU were excluded from our study. The data is available in S1 Data file.
Table 1 shows the descriptive statistics for some of the main variables in our analysis that are stratified by dead versus survivors. Severity level and mortality risk are categorical variables between 1 and 4, where 1 represents the lowest severity level or mortality risk. In addition, we controlled for gender, weight, height, BMI, physician, and additional physiological measurements including oxygen saturation level and diastolic blood pressure (DBP). The variable physician is a nominal variable assigned to each attending physician who makes the subjective measures. AVPU is a categorical and nominal variable with four values (A, P, V, and U).
Table A1 in S1 Appendix reports the variables and correlations between any pairs of the variables. We found no serious correlation between the variables other than between two subjective assessments of physicians—that is, physicians’ assessment of mortality risk and severity level, which was expected.
We investigated associations between different subjective and objective measures and our outcome variables. Our analysis included a range of stepwise regressions. In this study, and in the interest of parsimony, we focused on two main models: an ordinary least square, fixed-effect model to explain LOS, and a logistic regression model to explain mortality; both models were based on a wide range of independent and control variables. In addition, we analyzed effects of vital signs and early-risk indicators on physicians’ early subjective-risk assessments.
Table 2 summarizes the results of our stepwise regression analysis for LOS in which different independent variables were controlled in each step. In addition to the presented variables, patient demographics were controlled. Although the first column in Table 2 shows that the MEWS’s coefficient is significant, the low value of R2 (0.02) depicts that MEWS is only slightly better than a random guess for explaining LOS, This means that the whole model only predicted 2% of LOS variation.
In models M2–M6, however, MEWS is no longer significant. This is not unexpected, and one may argue that it does not speak against the usefulness of MEWS. We point out two major observations: (1) the model with only MEWS as a control variable has a very low predictive power (R2) even though MEWS is significant. This resonates with the common statistical argument that “significance” is different than “meaningfulness”: MEWS is significant in M1, but model M1 is only slightly better than a random guess; (2) as shown in the results of M2–M6, not all variables within MEWS (i.e., vital signs) are significant. In other words, better warning scores can be constructed based on weighted scores of some but not all of the vital signs. In models M5 and M6, vital signs lose their significance. In this step-wise analysis, R2 increases from 0.02 in M1 to 0.33 in model 6.
Finally, M7 shows the results of a model with the subjective measures and MEWS (but not the components of MEWS) in which MEWS is not a significant variable.
In sum, as shown in the table, in more complete models, none of the MEWS or vital signs at the time of admission is associated with LOS. However, physicians’ subjective assessments represented by their assessment of mortality risk and severity level are significantly associated with LOS. We later analyze how physicians make subjective assessments.
We now look at the mortality outcome variable. Table 3 shows the results of our mortality analysis with logistic regressions and reports the odds ratios. In model M1, MEWS is positively associated with mortality. In models M2–M5, we find that in controlling for more variables, MEWS loses its statistical significance and that the direction of association is negative in M3–M5. Physicians’ subjective assessments seem to have a higher explanatory power of mortality. Specifically, physicians’ subjective assessment of mortality risk is significantly and positively associated with the actual mortality. This finding leads to the conclusion that patients with higher mortality risk values at the time of admission are more likely to die. In another model, we added “physician” as a control variable in our analysis, but it was not significantly associated with mortality.
The results show that LOS is significantly and negatively associated with mortality; that is, the probability of mortality decreases the longer a patient stays in the hospital. It should be noted that, this can also be due to a reverse causality effect meaning that people who survive tend to stay longer in the hospital. As previously stated, no serious correlation exists between any pairs of variables in our analysis other than the correlation between the two subjective measures (Table A1).
In short, although both analyses show that using MEWS as the only independent variable can somewhat explain LOS and death, these models are weak and only slightly better than random guesses. If more variables are added, it better explains LOS and death. Both of the models point to the importance of physicians’ subjective assessments. It is important to look at the effects of different variables, including vital signs, demographic characteristics, and MEWS on physicians’ subjective assessments of severity level and mortality risks. The results are shown in Tables 4 and 5, which demonstrate that adding vital signs make a stronger model. In other words, it seems that physicians look at some of the vital signs (especially pulse rate, SBP, and AVPU) rather than the aggregate measure of MEWS in assigning risk measures.
Briefly, our analysis shows that LOS is more associated with physicians’ early subjective assessments. In this analysis, MEWS only weakly explains LOS and death in models in which we exclude any vital sign and demographics. Physicians’ risk assessments are more influenced by some of the risk indicators than the MEWS system and physicians’ early assessment of risk determines the duration of a hospital stay.
Our results contradict findings from some previous studies that stressed the predictive power of MEWS [4, 12]. We believe this is because we controlled for a much greater number of variables in our analysis, and that we have a better control of in-hospital processes such as physicians’ subjective assessments, which trigger the intensity of healthcare services.
Our study hypothesizes that humans’ (physicians’) early judgments are more associated with both LOS and outcomes than several of the physiological measures. The negative association between LOS and mortality as well as the positive association between physicians’ assessment of mortality risk and LOS indicate that, like all settings in which human decision making plays an important role, early-risk measures can be more useful in guiding subsequent actions—i.e., how physicians react to initial vital signs and how such reactions may prevent catastrophic events such as mortality. Our hypothesis resonates with the results of a previous study that concludes that physicians’ and nurses’ assessments of risk of mortality is accurate . While understanding the human component and that subjective assessments are important, improving the accuracy of those measures is still a necessity.
Our study focused on one specific hospital. The goal here was to ensure internal validity by focusing on a specific setting, gathering as many datapoints as possible, and being aware of all potential processes that may influence the data. Our knowledge of the hospital, the number of variables, and datapoints helped to validate our study of this specific hospital. While we do not see a major issue that may limit generalization of our study, we would like to invite other researchers to be cautious about generalizing our insights. We suggest similar studies in other hospital settings, with different governance structures, different patient demographics, and different types of physician training. In addition, we suggest more studies on effects of risk factors on patients’ perceptions and decisions in visiting emergency departments [20, 21].
Developing effective warning scores is critical for risk analysis and effective healthcare management. Our study shows that current early warnings are not effective in explaining LOS and mortality. Major feedback loop(s) exist that affect how physicians react to vital signs as well as their own subjective measures that can later compensate for large early-risk values. This comes down to the question of the purpose of early warning scores. If the scores are for predicting risk, the goal should be developing measures that provide a quick estimation of risk. However, for outcome prediction, we need more sophisticated models that embed how physicians react to different measures of risk, processes, resources, and technologies in hospitals. To conclude, we invite more studies to develop scores that can raise physician attention and improve early- assessment accuracy as well as measures that can better predict outcomes.
S1 Appendix. Correlations between variables.
Table A1 in this appendix shows the matrix for correlation coefficients between control variables.
- Conceptualization: NA NG JC.
- Data curation: NA.
- Formal analysis: NA NG JC.
- Funding acquisition: NG JC.
- Investigation: NA NG.
- Methodology: NA NG.
- Project administration: NA NG JC.
- Resources: NA NG JC.
- Software: NA NG.
- Supervision: NG JC.
- Validation: NA NG JC.
- Visualization: NA NG.
- Writing – original draft: NA NG.
- Writing – review & editing: NA NG JC.
- 1. Institute for Healthcare Improvement. Early Warning System: Scorecards That Save Lives 2014 [cited 2014]. Available from: http://www.ihi.org/resources/Pages/ImprovementStories/EarlyWarningSystemsScorecardsThatSaveLives.aspx.
- 2. Subbe C, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. Qjm. 2001;94(10):521–6. pmid:11588210
- 3. Gardner-Thorpe J, Love N, Wrightson J, Walsh S, Keeling N. The value of Modified Early Warning Score (MEWS) in surgical in-patients: a prospective observational study. Annals of the Royal College of Surgeons of England. 2006;88(6):571. pmid:17059720
- 4. Cei M, Bartolomei C, Mumoli N. In-hospital mortality and morbidity of elderly medical patients can be predicted at admission by the Modified Early Warning Score: a prospective study. International journal of clinical practice. 2009;63(4):591–5. pmid:19220521
- 5. Subbe C, Davies R, Williams E, Rutherford P, Gemmell L. Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia. 2003;58(8):797–802. pmid:12859475
- 6. Paterson R, MacLeod D, Thetford D, Beattie A, Graham C, Lam S, et al. Prediction of in-hospital mortality and length of stay using an early warning scoring system: clinical audit. Clinical Medicine. 2006;6(3):281–4. pmid:16826863
- 7. Badriyah T, Briggs JS, Meredith P, Jarvis SW, Schmidt PE, Featherstone PI, et al. Decision-tree early warning score (DTEWS) validates the design of the National Early Warning Score (NEWS). Resuscitation. 2014;85(3):418–23. pmid:24361673
- 8. Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS—towards a national early warning score for detecting adult inpatient deterioration. Resuscitation. 2010;81(8):932–7. pmid:20637974
- 9. Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84(4):465–70. pmid:23295778
- 10. McGinley A, Pearse RM. A national early warning score for acutely ill patients. Bmj. 2012;345.
- 11. Hill K. National Early Warning Score. Nursing in Critical Care. 2012;17(6):318-.
- 12. Burch V, Tarr G, Morroni C. Modified early warning score predicts the need for hospital admission and inhospital mortality. Emergency Medicine Journal. 2008;25(10):674–8. pmid:18843068
- 13. Kim WY, Shin YJ, Lee JM, Huh JW, Koh Y, Lim C-M, et al. Modified Early Warning Score Changes Prior to Cardiac Arrest in General Wards. PloS one. 2015;10(6):e0130523. pmid:26098429
- 14. Brownell M, Roos N. Variation in length of stay as a measure of efficiency in Manitoba hospitals. CMAJ: Canadian Medical Association Journal. 1995;152(5):675. pmid:7882230
- 15. Cuthbertson BH, Smith G. A warning on early-warning scores! British journal of anaesthesia. 2007;98(6):704–6. pmid:17519260
- 16. Barlow G, Nathwani D, Davey P. The CURB65 pneumonia severity score outperforms generic sepsis and early warning scores in predicting mortality in community-acquired pneumonia. Thorax. 2007;62(3):253–9. pmid:16928720
- 17. Price C, Sitch A, Banda P, Kellett J, Nyirenda M, Rylance J. Early warning scores generated in developed healthcare settings are not sufficient at predicting early mortality in Blantyre, Malawi: a prospective cohort study. PloS one. 2013;8(3):e59830. pmid:23555796
- 18. Kyriacos U, Jelsma J, James M, Jordan S. Monitoring vital signs: development of a modified early warning scoring (MEWS) system for general wards in a developing country. PloS one. 2014;9(1):e87073. pmid:24475226
- 19. Brabrand M, Hallas J, Knudsen T. Nurses and physicians in a medical admission unit can accurately predict mortality of acutely admitted patients: a prospective cohort study. PloS one. 2014;9(7):e101739. pmid:25019354
- 20. Hosseinichimeh N, Martin EG, Weinberg S. Do Changes in Health Insurance Coverage Explain Interstate Variation in Emergency Department Utilization? World Medical & Health Policy. 2016;8(1):58–73.
- 21. Hosseinichimeh N, Weinberg S. The effect of the Massachusetts healthcare reform on emergency department use. World Medical & Health Policy. 2014;6(1):9–21.