http://orcid.org/0000-0002-5582-5873
Figures
Abstract
Introduction
Early warning scores (EWS) are being increasingly embedded in hospitals over the world due to their promise to reduce adverse events and improve the outcomes of clinical patients.
The aim of this study was to evaluate the clinical use of an automated modified EWS (MEWS) for patients after surgery.
Methods
This study conducted retrospective before-and-after comparative analysis of non-automated and automated MEWS for patients admitted to the surgical high-dependency unit in a tertiary hospital. Operational outcomes included number of recorded assessments of the individual MEWS elements, number of complete MEWS assessments, as well as adherence rate to related protocols. Clinical outcomes included hospital length of stay, in-hospital and 28-day mortality, and ICU readmission rate.
Results
Recordings in the electronic medical record from the control period contained 7929 assessments of MEWS elements and were performed in 320 patients. Recordings from the intervention period contained 8781 assessments of MEWS elements in 273 patients, of which 3418 were performed with the automated EWS system. During the control period, 199 (2.5%) complete MEWS were recorded versus 3991 (45.5%) during intervention period. With the automated MEWS systems, the percentage of missing assessments and the time until the next assessment for patients with a MEWS of ≥2 decreased significantly. The protocol adherence improved from 1.1% during the control period to 25.4% when the automated MEWS system was involved. There were no significant differences in clinical outcomes.
Conclusion
Implementation of an automated EWS system on a surgical high dependency unit improves the number of complete MEWS assessments, registered vital signs, and adherence to the EWS hospital protocol. However, this positive effect did not translate into a significant decrease in mortality, hospital length of stay, or ICU readmissions. Future research and development on automated EWS systems should focus on data management and technology interoperability.
Citation: Mestrom E, De Bie A, Steeg Mvd, Driessen M, Atallah L, Bezemer R, et al. (2019) Implementation of an automated early warning scoring system in a surgical ward: Practical use and effects on patient outcomes. PLoS ONE 14(5): e0213402. https://doi.org/10.1371/journal.pone.0213402
Editor: Raffaele Serra, University Magna Graecia of Catanzaro, ITALY
Received: November 7, 2018; Accepted: February 20, 2019; Published: May 8, 2019
Copyright: © 2019 Mestrom et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files have been deposited to the Dryad Data Repository and may be accessed via DOI: 10.5061/dryad.000mn47.
Funding: The sponsor of this study was the Catharina Hospital. The study is a part of PhD research focusing on perioperative monitoring within a collaboration project of the Catharina Hospital, Technical University Eindhoven and Philips Research (IMPULS 2). Authors Rick Bezemer (RB) and Louis Atallah (LA) are employees of Philips Research and were only involved in the preparation of the study design and the manuscript. Dr. R.A. Bouwman and H.H.M. Korsten have acted as clinical consultants for Philips Research in Eindhoven, the Netherlands. R.A. Bouwman is also a member of the associate editorial board of the British Journal of Anaesthesia and received travel support from CSLBehring to visit Network for the Advancement of Patient Blood Management, Haemostasis and Thrombosis (NATA) 2015. URL to the sponsors’ website is https://www.catharinaziekenhuis.nl/.
Competing interests: R. Bezemer: employed by Philips Research, Perinatal Care and Parenting Solutions. L. Atallah: employed by Philips Research, Clinical Decision Support Research Area. R. A. Bouwman: Since January 2016, R. Bouwman act as clinical consultant for Philips Research in Eindhoven, The Netherlands. H. Korsten: Since January 2016, H. Korsten act as clinical consultant for Philips Research in Eindhoven, The Netherlands. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Automated early warning score (EWS) systems are increasingly embedded in clinical practice to improve registration and awareness of vital signs and enhance rapid response teams (RRT) notifications[1]. The impact on clinical outcomes of these systems remains uncertain in various populations, for example in the high-risk surgical population, which is the focus of this study.
Complications that are frequently encountered on the general ward can lead to major adverse events such as unplanned intensive care unit (ICU) admissions, cardiorespiratory arrest, and mortality[2, 3]. The EWS has been developed as an objective bedside tool to help clinicians identify patients at risk of adverse events[4, 5]. Since its introduction in the late nineties, studies have shown varied results on the predictive value of EWS as well as its implications on clinical outcomes[4, 6–9]. Consequently, newer versions of EWS have been developed to improve clinical relevance, such as the modified early warning score (MEWS)[10]. Digital automated systems have been introduced to assist clinicians in completing the EWS assessment. Several studies emphasized that these systems provide a faster completion of EWS with increased accuracy. This is important since complete and accurate registration on a regular base is essential for the effectiveness of EWS, especially when upcoming assisting medical technologies, such as clinical decision support tools, rely on these data[1, 11–16].
In addition, many studies have shown that automated systems reduce mortality and length of hospital stay. These studies also showed an improvement in the survival of patients treated by RRTs after EWS-triggered notification[1, 11, 17, 18]. On the other hand, the study of Dawes et al. showed no significant improvement in mortality among 3184 patients admitted to an acute medical unit after an electronic alerting physiological scoring system was introduced[19].
The automation of MEWS systems may be more effective for specific subgroups of patients such as high-risk surgical patients on the ward since postoperative complications develop more often and documentation on vital signs is known to be lacking there[20, 21].
The aim of this retrospective before-and-after cohort study is to investigate whether an automated MEWS system on a surgical high dependency unit (HDU) had a positive effect on clinical practice, in terms of improved documentation of vital signs and complete EWS assessments, and EWS protocol adherence. Secondary aim of the study was to evaluate the impact on clinical outcomes, such as mortality, length of stay, and ICU readmissions.
Methods
Study design
This is a retrospective before-and-after comparative analysis of clinical practice and outcomes for patients admitted to the eight-bed surgical HDU in the Catharina Hospital, a tertiary teaching hospital in Eindhoven, the Netherlands. This unit functions as a step-down unit between the ICU and the regular surgical ward in the postoperative phase after major elective and acute surgeries. These surgeries include major gastro-intestinal, oncologic, and vascular surgeries, such as pylorus preserving pancreaticoduodenectomy, open vascular aortic surgery and hyperthermic intraperitoneal chemotherapy.
Inclusion criteria were patients admitted for step-down care after surgery, who required an ICU admission for postoperative hemodynamic surveillance. Exclusion criteria were patients under 18 years old, ICU admission for other reasons than hemodynamic surveillance, such as electrolyte surveillance after thyroidectomy, and patients who had a second admission on the HDU after a hospital discharge during the study period or who were admitted for other reasons than step-down care.
The study period consisted of two 15 month phases: the control phase (January 2012 until March 2013) and the intervention phase (June 2013 until August 2015). Data from a three-month period between the phases was omitted to account for any influences of the training period directly after implementation of the automated MEWS system, the Philips IntelliVue Guardian Solution (Guardian). The study was approved by the Medical research Ethics Committees United (MEC-U; study ID:non-WMO 2015–87). The study was classified as non-WMO by the MEC-U based on the retrospective design. Therefore obtaining an informed consent was not deemed necessary as it conforms to the Dutch Agreement on Medical Treatment Act. All data were analyzed anonymously.
Control period.
In correspondence with common daily practice, the MEWS was used during the control period. This MEWS was introduced in 2011 in the hospital during a national study by Ludikhuize et al (Table 1) [17]. MEWS parameters are heart rate, oxygen saturation (SpO2), respiratory rate, non-invasive blood pressure, temperature, AVPU scale, and 24-hour urine production. According to the MEWS hospital protocol, these parameters were assessed at bedside within certain timeframes based on the previous MEWS (Table 1). All measured parameters were manually recorded in the electronic medical record (EMR). Nurses could manually calculate the MEWS using cards containing the MEWS algorithm. They were instructed to alert the physician on call in case of a MEWS ≥3 for assessment and potential treatment (Table 2). Nurses could also escalate to call the RRT of the ICU directly if the physician on call were not available or if initiated treatments after a certain MEWS did not lead to any improvement.
Intervention period.
The electronic EWS system Philips IntelliVue Guardian Solution (Guardian) was implemented on the HDU over a three-month period between the control and intervention phase. This system facilitated the acquisition of vital signs and the completion of MEWS to provide automated clinical decision support and awareness to the nursing staff. The device consisted of two spot-check monitors which were taken to the bedside of the patients to measure respiratory rate, non-invasive blood pressure, heart rate and SpO2. Urine output, level of consciousness (AVPU scale), temperature, and the nurse’s level of concern were manually entered in the Guardian software. The device calculated the MEWS values and showed them on the screen of the device as well as on a monitor at the central nurse station. In addition to the MEWS value, a short advice was displayed on the screen for further monitoring, such as the recommended time until the next assessment, or recommended actions, such as alerting a physician or the RRT. In addition, the monitor at the central nurse station also displayed these features. Every spot-check observation was stored in a database, which was not connected to the electronic medical record (CS-EZIS test, Chipsoft BV, Amsterdam, The Netherlands). Nurses were required to copy the measured vital parameters and MEWS into the EMR. All nurses received appropriate training on data collection before implementation of the Guardian system. All parameters needed to be recorded within a 15-minute timeframe to be considered in a single MEWS measurement. During the intervention period, the conventional methods used during the control period were still available to collect MEWS parameters and record them in the EMR, and although nurses were trained to use the automated system, they were free to work according to their preference.
Data collection and outcomes
The following data were retrieved from the EMR or the Guardian database during the periods: patient characteristics, vital signs and all other elements from the MEWS, and outcomes such as ICU re-admission, mortality, and length of stay. APACHE II, APACHE IV and SAPS II scores were collected during the first postoperative admission at the ICU or in the event of an ICU readmission. The results of this study were divided into two categories: operational outcomes (primary outcomes) and clinical outcomes (secondary outcomes).
Primary outcomes.
Operational outcomes under study include the practical clinical use of the automated EWS system, such as the number of documented MEWS elements and calculated MEWS values recorded in the EMR. In addition, the percentage of complete MEWS, daily patterns of the assessments, and the time interval between assessments were calculated. Time interval between assessments was also used to see if the next assessment was performed conform MEWS hospital protocol (Table 1). Measurements with a time interval of less than 15 minutes were considered a single assessment.
Secondary outcomes.
The clinical outcomes analyzed include the impact of the use of the automated MEWS system on the length of stay in the hospital, in-hospital and 28-day mortality, and ICU readmission rate. An ICU readmission was defined as a transfer to a higher level of care, such as ICU, medium care, or cardiac care unit.
Sub-analysis was performed for readmitted patients at the ICU; APACHE II, APACHE IV, and SAPS II scores on admission to the ICU were compared between the control and intervention group. Other outcomes for this subgroup analysis were the length of stay at the ICU and ICU mortality. In addition, a similar subgroup-analysis was performed for different age groups (≤49 years, 50–69 years and ≥70 years) or if certain patient characteristics differed significantly between the control and intervention period.
Statistical analysis
All statistical analyses were performed using IBM SPSS Statistics version 23 (IBM Corp, Armonk, New York, released 2015) [22]. Normality of the data sets was tested using a Kolmogorov-Smirnov test. Continuous data are presented as means and standard deviations or medians and interquartile data based on the distribution of the data. Categorical data are presented as proportions or percentages. The Mann-Whitney test was used to test for differences in continuous variables with non-normal distributions and the chi-square test Fisher’s exact test was used to test for differences in categorical groups. A value of p<0.05 was considered statistically significant.
Results
A total of 594 patients were included for analysis, 320 patients for the control group and 274 patients for the intervention group. Both groups were comparable in terms of age, gender, and APACHE II and -IV scores at the time of first ICU admittance postoperatively (Table 3). The control group consisted of significantly less patients undergoing oncologic abdominal surgery (69.7 versus 80.3%; p = 0.01) and more patients undergoing aortic surgery (20.0 versus 12.4%; p 0.01).
Operational outcomes
During the control period, a total of 7929 records of one or more elements of the MEWS were retrieved from the EMR. During the intervention period the total number of recorded assessments was 8781. Of these 8781 assessments, 3418 (39%) were recorded with the automated MEWS system. The other 5363 (61.1%) assessments were recorded with conventional monitoring systems. Results are shown in Fig 1.
The number of complete recorded MEWS and protocol adherence according to these MEWS for the control period and the intervention period, subdivided in conventional and automated MEWS assessments during the intervention period.
The adherence to MEWS hospital protocol improved when the automated MEWS system was involved, from 1.1% (88 of 7929 assessments) in the control group to 25.4% (2237 of 8781 assessments) in the intervention group. Within the intervention group, the adherence to MEWS hospital protocol improved from 10.8% (599 of 5363 assessments) when a conventional method was used to 47.9% (1638 of 3418 assessments) when the automated MEWS system was used (Fig 1).
The implementation of the automated MEWS system resulted in significantly more completed MEWS containing all the MEWS elements from 199 (2.5%) complete assessments in the control group to 3991 (45.5%) complete assessments in the intervention group (p<0.001). The number and percentage of missing elements in the MEWS for each documented record is shown in Fig 2. The most pronounced difference was observed for respiratory rate (96% versus 3%) and level of consciousness (100% versus 3%). The daily pattern is shown in Fig 3 and was comparable in both groups with three peaks during 24 hours corresponding to the daily nursing rounds. The median time until the next assessment was significantly shorter when a MEWS of 2 and higher was measured with the automated MEWS system compared to the conventional method between both groups and within the intervention group (Table 4.)
Pattern for control and intervention group.
Clinical outcomes
There were no significant differences in outcomes on mortality or length of stay (Table 5). In addition to that, the number of readmitted patients at the ICU and their severity of illness at readmission based on the SAPS II, and APACHE II and–IV did not significantly differ between the control and the intervention group.
Subgroup-analyses of three age groups (≤49 years, 50–69 years and ≥70 years) or per type of surgery did not result in significant differences between groups.
Discussion
The results of this retrospective study in a surgical high-dependency unit show that the use of an automated MEWS system improves the recording separate MEWS elements and complete MEWS assessments, as well as the resulting adherence to the MEWS hospital protocol. The use of this system improved the MEWS hospital protocol adherence for MEWS assessments using conventional methods during the intervention period. After implementing the automated MEWS system, 39% of the assessments were performed during the intervention period. Although there was a trend towards improved clinical outcomes in this period, this study did not show significant differences in mortality, length of stay, ICU readmission rate, or severity of illness at ICU readmission.
Similar to other studies, our study demonstrated improved accuracy and completeness in the recording of vital signs and complete MEWS after implementation of an automated MEWS system[14, 23]. These positive results on registration outcomes are important since valuable, reliable data become available for research, as quality of care indicators, and as input for clinical decision support system to improve clinical attendance and patient survival [1, 11, 13–16, 23, 24].
In contrast to similar studies, however, this study did not find a significant improvement of clinical outcomes[1, 11, 13, 14]. The most important difference between this work and others who have found such improvement was the practical and logistical use of the automated MEWS system. The implementation and data management of the automated MEWS in this study was different from previous studies[1, 13–16, 24]. For example, in this study, the automated MEWS system functioned as a CDSS (clinical decision support system) on the bedside and results needed to be copied manually into the hospital EMR system. The lack of direct integration with the EMR and also use of the conventional method during the intervention period were in contrast to previous studies where data collection changed from paper records during the control period to digital recording with automated EWS for the intervention period. In addition, some previous studies applied automated EWS assessments to provide clear overviews with trends on big screens or connections to beepers of ward physicians and RRT [1, 13–16, 24]. These differences in integration of automated EWS systems in daily clinical practice might explain the differences in clinical outcomes.
Second, several previous studies vary in methods used to assess the EWS, like single parameter scoring, EWS, and MEWS[1, 11, 13, 14]. Consequently, this heterogeneity prevents adequate comparison of clinical outcomes.
Third, the present study focused on non-elderly, high-risk surgical patients, while previously studied populations were mainly performed on general surgical and medical wards, and found positive effects in the elderly[1, 11, 13, 14].
Fourth, the difference in outcomes might be due to the variation in study design. Observation periods between studies differed and could have led to educational or Hawthorne effects in studies with relatively short intervention periods [11, 14]. Additionally, prospective research may be more prone for higher acuity or over-triage of less severe ill patients for ICU admission, possibly leading to diversion of real practice during the intervention periods. For example, the largest prospective multi-center study reporting positive clinical outcomes of Bellomo et al. readmitted more patients to the ICU while the need for ICU interventions like vasopressors, arterial lines and mechanical ventilation in their intervention period was significantly lower compared to the control period[11]. Even though this is explained as a result of earlier recognition of the deteriorating patient, one should be aware that this might also be explained by over-triage. This retrospective study was not able to find significant differences in severity of illness, ICU interventions or clinical outcomes in the overall study population and ICU readmissions. Therefore, retrospective long term designed studies may provide important insights on the effect of automated EWS systems while preventing diversion of real practice [25].
The retrospective nature of this study in inherent for the limitations of having missing data. Assessments might have been performed without being recorded in the EMR or automated MEWS system. Another limitation is in the size of the groups analyzed. If the study were to be powered based on mortality alone, there would have been a need for increasing patient cohort size from approximately 500 to more than 10,000 patients. In addition to that, the administrative burden for the nursing staff can result in missing data due to the automated MEWS system lacking interoperability with the EMR and hospital’s computer server. This is likely to be an important reason why only a trends towards improved clinical outcomes was found in this retrospective real-life cohort study.
Automated EWS systems that provide more complete and accurate recording of data have a great potential for future clinical decision support systems and early deterioration detection. Especially if additional data is included such as laboratory, pharmaceutical, and historical data, in combination with the upcoming use of machine learning and artificial intelligence for department or patient-specific MEWS algorithms[26–30]. Therefore, interoperability between automated MEWS systems and the EMR or other medical devices seems essential to prevent solely data input without data management and valuable output to save time for medical staff while achieving more consistent improved clinical outcomes.
Conclusion
Implementation of an automated MEWS on a surgical high dependency unit improves the number of complete MEWS, registered vital signs, and adherence to the local MEWS hospital protocol. However, this positive effect did not translate into a significant decrease in mortality, hospital length of stay, or ICU readmissions. Future research and development on automated EWS systems should focus on data management and technology interoperability to provide actionable insights to the right person at the right time.
Acknowledgments
Clémence Petit, MSc was involved in the study design and conducted the clinical part of the study.
References
- 1. Subbe CP, Duller B, Bellomo R. Effect of an automated notification system for deteriorating ward patients on clinical outcomes. Crit Care. 2017;21(1):52. Epub 2017/03/16. pmid:28288655; PubMed Central PMCID: PMCPMC5348741.
- 2. Pearse RM, Moreno RP, Bauer P, Pelosi P, Metnitz P, Spies C, et al. Mortality after surgery in Europe: a 7 day cohort study. Lancet (London, England). 2012;380(9847):1059–65. Epub 2012/09/25. pmid:22998715; PubMed Central PMCID: PMCPmc3493988.
- 3. Abbott TEF, Ahmad T, Phull MK, Fowler AJ, Hewson R, Biccard BM, et al. The surgical safety checklist and patient outcomes after surgery: a prospective observational cohort study, systematic review and meta-analysis. British journal of anaesthesia. 2018;120(1):146–55. pmid:29397122.
- 4. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PW. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation. 2014;85(5):587–94. Epub 2014/01/29. pmid:24467882.
- 5. Franklin C, Mathew J. Developing strategies to prevent inhospital cardiac arrest: analyzing responses of physicians and nurses in the hours before the event. Critical care medicine. 1994;22(2):244–7. Epub 1994/02/01. pmid:8306682.
- 6. De Meester K, Das T, Hellemans K, Verbrugghe W, Jorens PG, Verpooten GA, et al. Impact of a standardized nurse observation protocol including MEWS after Intensive Care Unit discharge. Resuscitation. 2013;84(2):184–8. Epub 2012/07/17. pmid:22796310.
- 7. Moon A, Cosgrove JF, Lea D, Fairs A, Cressey DM. An eight year audit before and after the introduction of modified early warning score (MEWS) charts, of patients admitted to a tertiary referral intensive care unit after CPR. Resuscitation. 2011;82(2):150–4. Epub 2010/11/09. pmid:21056524.
- 8. Paterson R, MacLeod DC, Thetford D, Beattie A, Graham C, Lam S, et al. Prediction of in-hospital mortality and length of stay using an early warning scoring system: clinical audit. Clinical medicine (London, England). 2006;6(3):281–4. Epub 2006/07/11. pmid:16826863.
- 9. Subbe CP, Williams E, Fligelstone L, Gemmell L. Does earlier detection of critically ill patients on surgical wards lead to better outcomes? Annals of the Royal College of Surgeons of England. 2005;87(4):226–32. Epub 2005/08/02. pmid:16053678; PubMed Central PMCID: PMCPmc1963939.
- 10. Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM: monthly journal of the Association of Physicians. 2001;94(10):521–6. pmid:11588210.
- 11. Bellomo R, Ackerman M, Bailey M, Beale R, Clancy G, Danesh V, et al. A controlled trial of electronic automated advisory vital signs monitoring in general hospital wards. Critical care medicine. 2012;40(8):2349–61. Epub 2012/07/20. pmid:22809908.
- 12. Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive care medicine. 2007;33(4):667–79. Epub 2007/02/24. pmid:17318499.
- 13. Schmidt PE, Meredith P, Prytherch DR, Watson D, Watson V, Killen RM, et al. Impact of introducing an electronic physiological surveillance system on hospital mortality. BMJ quality & safety. 2015;24(1):10–20. Epub 2014/09/25. pmid:25249636.
- 14. Jones S, Mullally M, Ingleby S, Buist M, Bailey M, Eddleston JM. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol. Critical care and resuscitation: journal of the Australasian Academy of Critical Care Medicine. 2011;13(2):83–8. Epub 2011/06/02. pmid:21627575.
- 15. Evans RS, Kuttler KG, Simpson KJ, Howe S, Crossno PF, Johnson KV, et al. Automated detection of physiologic deterioration in hospitalized patients. Journal of the American Medical Informatics Association: JAMIA. 2015;22(2):350–60. pmid:25164256; PubMed Central PMCID: PMC5566187.
- 16. Huh JW, Lim CM, Koh Y, Lee J, Jung YK, Seo HS, et al. Activation of a medical emergency team using an electronic medical recording-based screening system*. Critical care medicine. 2014;42(4):801–8. pmid:24335439.
- 17. Ludikhuize J, Smorenburg SM, de Rooij SE, de Jonge E. Identification of deteriorating patients on general wards; measurement of vital parameters and potential effectiveness of the Modified Early Warning Score. Journal of critical care. 2012;27(4):424.e7–13. Epub 2012/02/22. pmid:22341727.
- 18. Ludikhuize J, Borgert M, Binnekade J, Subbe C, Dongelmans D, Goossens A. Standardized measurement of the Modified Early Warning Score results in enhanced implementation of a Rapid Response System: a quasi-experimental study. Resuscitation. 2014;85(5):676–82. Epub 2014/02/25. pmid:24561029.
- 19. Dawes TR, Cheek E, Bewick V, Dennis M, Duckitt RW, Walker J, et al. Introduction of an electronic physiological early warning system: effects on mortality and length of stay. British journal of anaesthesia. 2014;113(4):603–9. Epub 2014/06/01. pmid:24878563.
- 20. McGain F, Cretikos MA, Jones D, Van Dyk S, Buist MD, Opdam H, et al. Documentation of clinical review and vital signs after major surgery. The Medical journal of Australia. 2008;189(7):380–3. Epub 2008/10/08. pmid:18837681.
- 21. Hands C, Reid E, Meredith P, Smith GB, Prytherch DR, Schmidt PE, et al. Patterns in the recording of vital signs and early warning scores: compliance with a clinical escalation protocol. BMJ quality & safety. 2013;22(9):719–26. Epub 2013/04/23. pmid:23603474.
- 22.
IBM Corp. Released 2015. IBM SPSS Statistics for Windows, Version 23.0. Armonk, NY: IBM Corp.
- 23. Schmidt PE, Meredith P, Prytherch DR, Watson D, Watson V, Killen RM, et al. Impact of introducing an electronic physiological surveillance system on hospital mortality. BMJ quality & safety. 2015;24(2):176–7. Epub 2015/01/22. pmid:25605956.
- 24. Kollef MH, Chen Y, Heard K, LaRossa GN, Lu C, Martin NR, et al. A randomized trial of real-time automated clinical deterioration alerts sent to a rapid response team. Journal of hospital medicine. 2014;9(7):424–9. pmid:24706596; PubMed Central PMCID: PMC4354800.
- 25. Cohen AT, Goto S, Schreiber K, Torp-Pedersen C. Why do we need observational studies of everyday patients in the real-life setting? European Heart Journal Supplements. 2015;17(suppl_D):D2–D8.
- 26. Florath I, Albert A, Hassanein W, Arnrich B, Rosendahl U, Ennker IC, et al. Current determinants of 30-day and 3-month mortality in over 2000 aortic valve replacements: Impact of routine laboratory parameters. European journal of cardio-thoracic surgery: official journal of the European Association for Cardio-thoracic Surgery. 2006;30(5):716–21. Epub 2006/09/13. pmid:16965919.
- 27. Loekito E, Bailey J, Bellomo R, Hart GK, Hegarty C, Davey P, et al. Common laboratory tests predict imminent death in ward patients. Resuscitation. 2013;84(3):280–5. Epub 2012/08/07. pmid:22863543.
- 28. Khurana HS, Groves RH Jr., Simons MP, Martin M, Stoffer B, Kou S, et al. Real-Time Automated Sampling of Electronic Medical Records Predicts Hospital Mortality. The American journal of medicine. 2016;129(7):688–98.e2. Epub 2016/03/29. pmid:27019043; PubMed Central PMCID: PMCPmc4916370.
- 29. Hackmann G, Chen M, Chipara O, Lu C, Chen Y, Kollef M, et al. Toward a two-tier clinical warning system for hospitalized patients. AMIA Annual Symposium proceedings AMIA Symposium. 2011;2011:511–9. Epub 2011/12/24. pmid:22195105; PubMed Central PMCID: PMCPmc3243239.
- 30. Petit C, Bezemer R, Atallah L. A review of recent advances in data analytics for post-operative patient deterioration detection. Journal of clinical monitoring and computing. 2017. Epub 2017/08/23. pmid:28828569.