http://orcid.org/0000-0001-9072-2469
Figures
Abstract
Background
Early risk stratification for guiding treatment priority in the emergency department (ED) is becoming increasingly important. Existing prediction models typically use demographics, vital signs and laboratory parameters. Laboratory-based models require blood testing, which may cause substantial delay. However, these delays can be prevented by the use of point-of-care testing (POCT), where results are readily available. We aimed to externally validate a laboratory-based model for mortality and subsequently assessed whether a POCT model yields comparable performance.
Methods
All adult patients visiting the ED of a university hospital between January 1st, 2012 and December 31st, 2016 were retrospectively reviewed for inclusion. Primary outcome was defined as 30-day mortality after ED presentation. We externally validated one existing prediction model including age, glucose, urea, sodium, haemoglobin, platelet count and white blood cell count. We assessed the predictive performance by discrimination, expressed as Area under the Curve (AUC). We compared the existing model to an equivalent model using predictors that are available with POCT (i.e. glucose, urea, sodium and haemoglobin). Additionally, we internally validated these models with bootstrapping.
Results
We included 34,437 patients of whom 1,942 (5.6%) died within 30 days. The AUC of the laboratory-based model was 0.794. We refitted this model to our ED population and found an AUC of 0.812, which decreased only slightly to 0.790 with only POCT parameters.
Conclusions
Our POCT-model performs similar to existing laboratory-based models in identifying patients at high risk for mortality, with results available within minutes. Although the model needs further validation and evaluation, it shows the potential of POCT for early risk stratification in the ED.
Citation: Brink A, Schuttevaer R, Alsma J, Zietse R, Schuit SCE, Lingsma HF (2020) Predicting 30-day mortality using point-of-care testing; an external validation and derivation study. PLoS ONE 15(9): e0239318. https://doi.org/10.1371/journal.pone.0239318
Editor: Itamar Ashkenazi, Technion - Israel Institute of Technology, ISRAEL
Received: June 4, 2020; Accepted: September 4, 2020; Published: September 24, 2020
Copyright: © 2020 Brink et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data will be available from the RePub, Erasmus University Repository of the Erasmus University Medical Center, Rotterdam. The data are from now on available on:(http://hdl.handle.net/1765/129069).
Funding: We would like to state that the authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: AUC, area under the curve; bpm, beats per minute; CI, confidence interval; dL, decilitre; ED, emergency department; Erasmus MC, Erasmus University Medical Center; ESI, Emergency Severity Index; EWS, Early Warning Scores; g, gram; IQR, interquartile range; L, litre; MEWS, Modified Early Warning Score; mmol, millimole; MTS, Manchester Triage Systems; NEWS, National Early Warning Score; POCT, point-of-care testing; ROC, Receiver Operating Characteristic; SD, standard deviation
Introduction
Identifying patients in the Emergency Department (ED) at risk of dying remains challenging. The existing prediction models are typically based on demographics and vital signs.
Triage systems are initially used to identify the most severely ill patients. However, current triage systems, such as the Emergency Severity Index (ESI) [1] and the Manchester Triage System (MTS) [2], were mainly introduced for trauma patients. The performance of triage systems in all ED patients is poor [3–5]. Early warning scores (EWS) are also used in the ED, either as replacement or as addition to triage [6, 7]. Examples of EWS are the Modified Early Warning Score (MEWS) and the National Early Warning Score (NEWS). However, EWS are developed to detect inpatient clinical deterioration and the predictive value of EWS for mortality in the ED varies [8, 9] and both validation and calibration are inadequate or lacking [10].
Using laboratory parameters is another approach for early risk stratification. An advantage of laboratory parameters in prediction models is their objective measurement. Asadollahi et al. provided a laboratory-based prediction model derived from 1,650 acute medical and surgical patients, which performed well with an AUC of 0.848 [11]. The model was internally-externally validated using data from the same hospital in a different period of time. The model uses age, glucose, urea, sodium, haemoglobin, platelets and white blood cell count as predictors. These six laboratory parameters were selected from a large array of potential parameters and are known to correlate with adverse outcome [12–15]. Laboratory models require blood testing and can therefore cause a substantial delay if blood samples are analysed by a central laboratory. However, these delays might be prevented by point-of-care testing (POCT), which yields results within minutes [16, 17].
The aim of this study is to determine whether a laboratory model can be implemented using only POCT laboratory testing. Since external validation is a critical step to implementation in clinical practice, and to potentially improve the feasibility of the model, the first aim of this study is to externally validate the laboratory-based model by Asadollahi et al. [11] in a large unselected population of ED patients. The second aim is to assess whether a model based only on POCT available laboratory parameters yields comparable performance.
Methods
Study design and setting
We performed a retrospective cohort study in the ED of the Erasmus University Medical Center Rotterdam (Erasmus MC), Rotterdam, the Netherlands, which is a large tertiary referral centre, situated in an urban area. The ED has approximately 32,000 visits annually. Data from all patients were automatically extracted from the electronic health records on a regular basis and collected in a database.
Selection of participants
We included all ED visits from January 1st 2012 up to December 31st 2016. Adult patients, aged 18 years and over, in which laboratory diagnostics were performed were selected for this study. Per patient only the first visit to the ED was included.
Measurements and outcomes
We extracted demographic data (i.e. sex, age) and presenting vital parameters (i.e. body temperature, heart rate, respiratory rate, oxygen saturation, blood pressure and consciousness level using AVPU-scale). Furthermore, we extracted acuity scale according to MTS category, disposition (i.e. in hospital admission), and arrival (i.e. by ambulance). In line with the study of Asadollahi et al. we extracted haemoglobin, serum sodium, plasma glucose, white blood cell count, serum urea and platelet count. These laboratory values were afterwards categorized (Table 1).
Haemoglobin levels were converted from mmol/L to g/dL [11]. Subsequently, we selected the laboratory parameters that are measurable with POCT, i.e. glucose, urea, sodium and haemoglobin. At Erasmus MC, the ABL800 FLEX (Radiometer America Inc., Westlake, OH) blood gas analyser for POCT is used, which yields results within two minutes. Outcome was defined as 30-day mortality after the index ED visit. Mortality data were retrieved from the patient records, which are linked to municipal mortality records. The Medical Ethics Committee of the Erasmus MC reviewed the study and concluded that our study did not fall under the scope of the Medical Research Involving Human Subjects Act and therefore no informed consent needed to be obtained.
Statistical analysis
Patient characteristics were presented as mean (SD), median (interquartile range (IQR)) or absolute numbers (percentage), when appropriate. Missing data were handled using multiple imputations (n = 5) with a chained equations procedure, which means that the expected value of the missing data point is estimated based on the available data.
We examined all patient characteristics of patients that were alive versus the patients who died within 30 days from the index ED visit. Data were compared using Pearson chi-squared tests or unpaired t-tests, based on distribution of data.
Model performance of the laboratory-based model was described as discrimination and calibration. Discrimination was assessed using the area under the Receiver Operating Characteristic-curve (AUC). We assessed calibration with a calibration plot in which the slope indicates the relation between the observed and the predicted outcome (i.e. ideally close to 1) and the intercept indicates whether the predictions are systematically deviant (i.e. ideally close to 0). We calculated likelihood ratios for all cut-off points from the total score and determined the ideal cut-off point using Youden’s index (i.e. the cut-off point combining the optimal sensitivity and specificity). Interval likelihood ratios were established for several different intervals. Furthermore, we refitted the model on our data and subsequently reduced the model by only including parameters which could be tested using POCT. These models were internally validated using five hundred times bootstrap resampling.
Analyses were conducted with IBM SPSS Statistics for Windows version 26 (IBM Corp., Armonk, New York, USA) and R statistics version 3.6.1. A significance level of p<0.05 was considered as statistically significant.
Results
Patient characteristics
116,398 adult ED visits were recorded between January 2012 and December 2016. Laboratory testing was performed in 54,753 of these visits. Selecting only first ED visits, yielded 34,437 patients eligible for analysis (Fig 1). The majority of the population was male (54.8%). Median age (IQR) was 54 years (37–67). Admission rate was 55.7% and in total 1,942 (5.6%) patients died within 30 days after the ED visit. Patients who died presented to the ED with more abnormal vital signs (i.e. higher heart rate (93 vs. 87 per minute), lower systolic blood pressure (135 vs. 140 mmHg), abnormal consciousness level (37.3 vs. 7.5%), p<0.001) and were significantly older (68 vs. 52 years, p<0.001) (Table 2).
Model performance
Haemoglobin was most frequently tested (in 98.8% patients), whereas blood urea nitrogen was least often tested (72.0%). All predictive effects we found corresponded to the original model (e.g. low platelet count, haemoglobin level, sodium levels were associated with 30-day mortality) (Tables 1 and 2). Of all predictors in the model, age ≥ 65 years was the strongest predictor for 30-day mortality in univariate analysis (OR [95% CI] = 4.4 [4.0; 4.8]) (Table 3).
External validation showed an AUC of 0.796 [0.788–0.806] (S1 Fig). The calibration curve had a slope of 0.77 and an intercept of 0.34 (Fig 2).
We refitted the laboratory-based model on our own dataset with subsequent reduction to a model with only POCT parameters (Table 3). The refitted laboratory-based model yielded an internally validated AUC of 0.813, which slightly decreased to 0.790 when only including age and POCT parameters.
Likelihood ratios of score intervals of 0 to 5, 6 to 13 and 14 to 20 were 0.31, 1.77 and 5.04 respectively. Positive and negative likelihood ratios for dichotomous cut-off points are found in S1 Table. The highest Youden’s index was found using a cut-off point score of 8.
Discussion
In this study we first externally validated the model by Asadollahi et al. that uses laboratory parameters and the patients’ age to predict mortality. Next we assessed that the Asadollahi et al. model based only on POCT available laboratory parameters yielded comparable performance. To our knowledge, we are the first to validate this laboratory-based model and to perform calibration. Our external validation resulted in a reasonable AUC of 0.796. One benefit of our study is that we used a large database for the external validation, which limits uncertainty in the performance of the model and thus increases the clinical relevance [18].
Despite the fact that most models are introduced without calibration, it is a critical step preceding implementation of a model. Calibration describes the agreement between the calculated, based on the prediction model, and the observed number of occurrences. The calibration of this model was suboptimal, which indicates the model slightly overestimates the mortality risk.
A major disadvantage of the study by Asadollahi et al. is the case-controlled study design. The authors included deceased and non-deceased patients in a 1:2 ratio, which yields a mortality rate of 33%. This results in an overestimation of the prevalence of 30-day mortality. Since we conducted a cohort study, our mortality rate reflects the 30-day mortality prevalence more accurately.
One of the merits of the model by Asadollahi et al. is that it uses only a few parameters. Additionally, the parameters within this model are virtually always assessed in patients with indication for laboratory diagnostics admitted to the ED. Therefore this model is generally applicable and easy to use by clinicians. Its simple interpretation accommodates usage implementation in electronic patient files. The power of laboratory values in prediction research is that they provide an objective measurement, especially compared to manually collected vital parameters. Vital parameters that are manually collected are prone to interrater variability. Also, vital parameters are subject to influences that are not always taken into account (pain, stress, normality for an individual patient). A downside of laboratory values is they take time to become available and are therefore difficult to implement in a decision model in the ED.
Ideally, prediction models in the ED should consist of readily available parameters and as little parameters as possible, making the model convenient for clinical practice. As most laboratory test results take more than an hour, the second aim of our study was to assess whether a model based on POCT parameters yielded similar predictive performance in predicting 30-day mortality. We found that our model with age and POCT parameters had similar performance. We provided the regression coefficients and an intercept allowing to replicate this study, but also to facilitate implementation of this model in clinical practice. POCT is promising since it only takes minutes to analyse blood samples. This may lead to a reduction in time to diagnosis and initiation of treatment. Furthermore, POCT-systems are already used in EDs. Although presently not every ED has a POCT-analyser, which may limit the applicability of this study, this study may encourage to invest in POCT-analysers.
A general limitation of literature concerning prediction models in the ED, is that hardly any study provides sufficient information to execute external validation. There are several models based on laboratory values published which performed well in general with an AUC above 0.80 [19–21].
A limitation of our study is its retrospective study design, which makes our study prone to bias. Nevertheless, laboratory data were automatically retrieved from the laboratory testing machines thus the quality of the data is high and not subject to human mistakes. Furthermore, we had missing data, mainly vital signs, which we replaced using multiple imputation. This is a valid way to manage even large samples of missing data [22], although a database with all data available is obviously superior. Therefore, we should strive to collect data as complete as possible. Last, this validation study took place in a tertiary care centre which corresponds to the derivation study [11]. Therefore, our results might be less generalizable to other centres with patients with different complexity and pathology. We therefore recommend external validation of our model in another centre, before implementation. In addition, we encourage considering POCT in prediction model development, researching both its discrimination and calibration.
In conclusion, the performance of the model by Asadollahi et al. was adequate in identifying patients at high risk for mortality in the ED. However, our POCT-model performs similar with results available within minutes. Although our model needs further validation and evaluation, it shows the potential of POCT in early risk stratification in the ED.
Supporting information
S1 Table. Sensitivity, specificity, LR+, LR-, Youden’s index for different cut-off points.
https://doi.org/10.1371/journal.pone.0239318.s002
(DOCX)
References
- 1. Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N. Reliability and validity of a new five-level triage instrument. Acad Emerg Med. 2000;7(3):236–42. pmid:10730830
- 2.
Mackway-Jones K, Marsden J, Windle J. Emergency Triage. 3rd ed. West Sussex: John Wiley & Sons, Ltd.; 2014. 192 p.
- 3. Geier F, Popp S, Greve Y, Achterberg A, Glockner E, Ziegler R, et al. Severity illness scoring systems for early identification and prediction of in-hospital mortality in patients with suspected sepsis presenting to the emergency department. Wien Klin Wochenschr. 2013;125(17–18):508–15. pmid:23934184
- 4. Steiner D, Renetseder F, Kutz A, Haubitz S, Faessler L, Anderson JB, et al. Performance of the Manchester Triage System in Adult Medical Emergency Patients: A Prospective Cohort Study. J Emerg Med. 2016;50(4):678–89. pmid:26458788
- 5. van der Wulp I, Schrijvers AJ, van Stel HF. Predicting admission and mortality with the Emergency Severity Index and the Manchester Triage System: a retrospective observational study. Emerg Med J. 2009;26(7):506–9. pmid:19546272
- 6. Griffiths JR, Kidney EM. Current use of early warning scores in UK emergency departments. Emergency Medicine Journal. 2012;29(1):65–6. pmid:21965177
- 7. Tian L, Fang Z, Xiao H, Li L, Li Y. Value of triage early warning score for trauma patients in an emergency department. Zhong Nan Da Xue Xue Bao Yi Xue Ban. 2015;40(5):549–57. pmid:26032074
- 8. Alam N, Vegting IL, Houben E, van Berkel B, Vaughan L, Kramer MH, et al. Exploring the performance of the National Early Warning Score (NEWS) in a European emergency department. Resuscitation. 2015;90:111–5. pmid:25748878
- 9. Mitsunaga T, Hasegawa I, Uzura M, Okuno K, Otani K, Ohtaki Y, et al. Comparison of the National Early Warning Score (NEWS) and the Modified Early Warning Score (MEWS) for predicting admission and in-hospital mortality in elderly patients in the pre-hospital setting and in the emergency department. PeerJ. 2019;7:e6947. pmid:31143553
- 10. Brink A, Alsma J, Fortuin AW, Bramer WM, Zietse R, Lingsma HF, et al. Prediction models for mortality in adult patients visiting the Emergency Department: a systematic review. Acute Med. 2019;18(3):171–83. pmid:31536055
- 11. Asadollahi K, Hastings IM, Gill GV, Beeching NJ. Prediction of hospital mortality from admission laboratory data and patient age: a simple model. Emerg Med Australas. 2011;23(3):354–63. pmid:21668723
- 12. Asadollahi K, Hastings IM, Beeching NJ, Gill GV. Laboratory risk factors for hospital mortality in acutely admitted patients. Qjm-Int J Med. 2007;100(8):501–7.
- 13. Jung WJ, Lee HJ, Park S, Lee SN, Kang HR, Jeon JS, et al. Severity of community acquired hypernatremia is an independent predictor of mortality. Intern Emerg Med. 2017;12(7):935–40. pmid:28474207
- 14. Waikar SS, Mount DB, Curhan GC. Mortality after Hospitalization with Mild, Moderate, and Severe Hyponatremia. Am J Med. 2009;122(9):857–65. pmid:19699382
- 15. Whelan B, Bennett K, Oriordan D, Silke B. Serum sodium as a risk factor for in-hospital mortality in acute unselected general medical patients. Qjm-Int J Med. 2009;102(3):175–82.
- 16. Singer AJ, Taylor M, LeBlanc D, Meyers K, Perez K, Thode HC Jr., et al. Early Point-of-Care Testing at Triage Reduces Care Time in Stable Adult Emergency Department Patients. J Emerg Med. 2018;55(2):172–8. pmid:29887410
- 17. Kankaanpaa M, Holma-Eriksson M, Kapanen S, Heitto M, Bergstrom S, Muukkonen L, et al. Comparison of the use of comprehensive point-of-care test panel to conventional laboratory process in emergency department. Bmc Emerg Med. 2018;18.
- 18. Riley RD, Snell KI, Ensor J, Burke DL, Harrell FE Jr., Moons KG, et al. Minimum sample size for developing a multivariable prediction model: PART II—binary and time-to-event outcomes. Stat Med. 2019;38(7):1276–96. pmid:30357870
- 19. Kristensen M, Iversen AKS, Gerds TA, Ostervig R, Linnet JD, Barfod C, et al. Routine blood tests are associated with short term mortality and can improve emergency department triage: a cohort study of > 12,000 patients. Scand J Trauma Resus. 2017;25.
- 20. Silke B, Kellett J, Rooney T, Bennett K, O'Riordan D. An improved medical admissions risk system using multivariable fractional polynomial logistic regression modelling. Qjm-Int J Med. 2010;103(1):23–32.
- 21. Cournane S, Conway R, Byrne D, O'Riordan D, Silke B. Predicting Outcomes in Emergency Medical Admissions Using a Laboratory Only Nomogram. Comput Math Method M. 2017.
- 22. Pedersen AB, Mikkelsen EM, Cronin-Fenton D, Kristensen NR, Pham TM, Pedersen L, et al. Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol. 2017;9:157–65. pmid:28352203