https://orcid.org/0000-0002-8278-9513
Figures
Abstract
Objectives of the study
Demographic changes alongside medical advances have resulted in older adults accounting for an increasing proportion of emergency hospital admissions. Current measures of illness severity, limited to physiological parameters, have shortcomings in this cohort, partly due to patient complexity. This study aimed to derive and validate a risk score for acutely unwell older adults which may enhance risk stratification and support clinical decision-making.
Methods
Data was collected from emergency admissions in patients ≥65 years from two UK general hospitals (April 2017- April 2018). Variables underwent regression analysis for in-hospital mortality and independent predictors were used to create a risk score. Performance was assessed on external validation. Secondary outcomes included seven-day mortality and extended hospital stay.
Results
Derivation (n = 8,974) and validation (n = 8,391) cohorts were analysed. The model included the National Early Warning Score 2 (NEWS2), clinical frailty scale (CFS), acute kidney injury, age, sex, and Malnutrition Universal Screening Tool. For mortality, area under the curve for the model was 0.79 (95% CI 0.78–0.80), superior to NEWS2 0.65 (0.62–0.67) and CFS 0.76 (0.74–0.77) (P<0.0001). Risk groups predicted prolonged hospital stay: the highest risk group had an odds ratio of 9.7 (5.8–16.1) to stay >30 days.
Conclusions
Our simple validated model (Older Persons’ Emergency Risk Assessment [OPERA] score) predicts in-hospital mortality and prolonged length of stay and could be easily integrated into electronic hospital systems, enabling automatic digital generation of risk stratification within hours of admission. Future studies may validate the OPERA score in external populations and consider an impact analysis.
Citation: Arjan K, Forni LG, Venn RM, Hunt D, Hodgson LE (2021) Clinical decision-making in older adults following emergency admission to hospital. Derivation and validation of a risk stratification score: OPERA. PLoS ONE 16(3): e0248477. https://doi.org/10.1371/journal.pone.0248477
Editor: Andrea L. Conroy, Indiana University School of Medicine, UNITED STATES
Received: June 8, 2020; Accepted: February 26, 2021; Published: March 18, 2021
Copyright: © 2021 Arjan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In the United kingdom (UK) the over 65 years demographic have demonstrated the largest increase in emergency hospital admissions of any age group, a trend projected to continue further in the coming decade [1]. Care pathways for this cohort must not only optimise clinical management but also involve patients and their kin in decision-making with regard to escalation and limitations of care. This should be aided by objective measures, including where appropriate, accurate prognostication [2]. A generalisable prediction score developed for acutely unwell older patients may help develop and inform such pathways.
In routine UK acute clinical practice, methods to monitor and determine escalations of care are limited to the National Early Warning Score (NEWS) and its iteration NEWS2 [3], which combines physiological parameters into a risk score, and clinical judgement. However, NEWS has demonstrated only moderate predictive power for mortality in older cohorts [4, 5].
Frailty has been investigated as a method to risk assess this demographic [6]. In UK hospital settings, frailty is commonly quantified using the Clinical Frailty Scale [7]. This tool assesses domains such as co-morbidity, function, and cognition to create a score ranging from very fit to terminal illness. Risk prediction models incorporating frailty are yet to be implemented in the acute clinical setting for older patients.
Clinical frailty, acute physiology (NEWS2), and other variables such as nutrition assessments, blood parameters, previously coded past medical history, and drugs prescriptions, are being increasingly employed in electronic systems which may themselves facilitate enhanced clinical risk stratification [8, 9]. This study aimed to derive and validate a prediction score for in-hospital mortality in the older adult patient presenting to hospital, using readily accessible electronic information, that would allow subsequent digital automation and presentation to the clinical team.
Materials and methods
Source of data and participants
Data was collected prospectively for all patients (≥65 years) admitted through two emergency departments (EDs) of a non-specialist hospital organisation on the South coast of England that has 870 beds and a combined annual ED attendance over 135,000 (April 2017 –April 2018). The two hospital sites, with minimal overlap of staffing were treated as independent populations to derive (site 1) and validate (site 2) the model. Participants were excluded if: no NEWS2 or clinical frailty scale (CFS) was recorded, or the length of stay in hospital was <1 day, to limit non-urgent attendances.
Outcome
The primary outcome was in-hospital mortality. Secondary outcomes were: 48-hour mortality, 7-day mortality, hospital stay >30 days, and re-admission <30 days of discharge.
Predictors
Variables collected included: age, sex, co-morbidity (ICD-10 [10]) coding of congestive cardiac failure, diabetes, liver disease, chronic kidney disease (eGFR <60mls/min), community-acquired acute kidney injury (CA-AKI) [11], NEWS2, clinical frailty scale (CFS), Malnutrition Universal Screening Tool (MUST) [12] and palliative care or a do not attempt resuscitation (DNAR) status recorded. Definitions and explanations of the variables measured are presented in Appendix 1 in S1 File. These variables are routinely collected for older patients at both sites and are known to be used in other scoring systems.
Missing data
The missingness of NEWS2 and CFS, in patients who did not have it documented, was assessed for correlation with in-hospital mortality. Any remaining missing data, after the exclusion of patients without a documented NEWS2 and CFS, was handled with multiple imputation, using the most appropriate method [13].
Statistical analysis
Binary logistic regression was carried out using a backwards stepwise elimination method on accessible and clinically significant variables. The identified independent predictors of in-hospital mortality were used to create a risk prediction score. Model performance was assessed for calibration with calibration plot analysis and Hosmer-Lemeshow test and area under receiver operating curve (AUC) for discrimination, and R2 for determination (Cox-Snell). Re-calibration and re-adjustment of the model were undertaken to optimise the risk score in the external validation group [14]. A scoring system was created by assigning points to the independent predictors of in-hospital mortality. Points were allocated by dividing their regression coefficients (β), by the smallest β of all variables. These numbers were then rounded to the nearest integer [15]. A step-by-step example of points allocation is provided in Appendix 2 in S1 File.
Information loss between the models, when using regression coefficients or the points score, was assessed by differences in AUC [16]. Sensitivity analysis on the full model examined performance differences when excluding patients under palliative care or with a DNAR order in-place.
Risk groups
Low, medium, high and severe risk groups were created using cut-offs that optimised the sensitivities, specificities, and mortality risk for use in clinical practice.
Development and validation
Both hospitals are within the same NHS England Trust. There were no systematic differences in the method of data collection or admission criteria between derivation and validation datasets.
Data was analysed using IBM SPSS Statistics version 25® and Stata version 15.1®.
Reporting guidelines
This study remained concordant with the TRIPOD checklist for predictive modelling [13].
Results
Participants characteristics
14,274 non-elective non-day case admissions aged ≥65 years, were admitted in site 1 and 14,951 in site 2, during the study period. 5,300 and 6,560 were excluded from site 1 and 2 respectively, for missing either NEWS2 or CFS, leaving 8,974 and 8,391 patients to be included for analysis in the derivation and validation groups respectively (consort chart presented in Appendix 3 in S1 File). Clinical, demographic, and missing data for the derivation and validation sites are presented in Table 1. Univariate analysis for in-hospital mortality is presented in Appendix 4 in S1 File.
Missing data
358 (4.0%) cases had missing MUST scores in the derivation cohort and 105 (1.3%) in the validation. The Markov Chain Monte Carlo method of multiple imputation was employed (7,020 values were imputed for 468 missing items over 15 iterations) [17]. Independent predictors identified were not different between the original data and after multiple imputation.
Model specification
Regression analysis identified six independent predictors of mortality: male sex, age, CA-AKI, MUST, NEWS2, and CFS (Table 2), with no evidence of collinearity (Appendix 5 in S1 File).
A points-based score was created using these independent predictors and for ease of use was capped at 50 with the model abbreviated to the Older Persons’ Emergency Risk Assessment (OPERA) score. For the MUST variable, one point is added for every increase in risk group (low 0, medium 1, and high risk 2 points, respectively).
Model performance
There was no statistically significant difference in AUC for mortality prediction when using the regression coefficients or OPERA points score (p = 1), indicating negligible information loss. Likewise, R2 and Hosmer-Lemeshow tests demonstrated negligible difference. For in-hospital mortality, the derivation model demonstrated an R2 of 0.087 and AUC of 0.77 (95% confidence interval 0.75–0.78) (Fig 1). The score in the validation group had an R2 of 0.082 and an AUC of 0.79 (0.78–0.80). The model’s calibration in the validation cohort as measured by the Hosmer-Lemeshow test was acceptable (p = 0.302). Excluding the variable requiring a laboratory result (creatinine to define CA-AKI), AUC was lowered to 0.76 (0.74–0.77) in the derivation cohort and 0.77 (0.76–0.79) in the validation cohort.
(A) derivation and (B) validation groups. OPERA score (Blue line), CFS (Red line) and NEWS2 (Green line). CFS—clinical frailty scale, NEWS2—national early warning score.
AUC for NEWS2 was 0.66 (0.64–0.68) and 0.65 (0.62–0.67) in derivation and validation sets, respectively. AUC for CFS were 0.68 (0.66–0.70) and 0.76 (0.74–0.77) in the derivation and validation set, respectively. Pairwise comparisons for in-hospital mortality AUC were significantly different between OPERA and for NEWS2 and CFS (p<0.0001) [16].
OPERA was used to classify individuals into low, medium, high, and severe risk groups (sensitivity, specificity, positive and negative predictive values are presented for each risk group in Appendix 6 in S1 File). For the validation set negative predictive value (NPV) was greater than 95% for each risk group.
In derivation, AUCs to predict secondary outcomes were: 48-hour mortality 0.86 (0.83–0.90), 7-day mortality 0.83 (0.81–0.85), hospital stay >30 days 0.66 (0.64–0.67), and 30-day re-admission 0.50 (0.49–0.52). In the validation cohort AUCs for the same outcomes were: 0.84 (0.78–0.90), 0.83 (0.79–0.86), 0.68 (0.65–0.70), and 0.52 (0.50–0.54), respectively (receiver operating curves for secondary outcomes are presented in Appendix 7 in S1 File). OPERA risk groups were investigated as indicators of extended hospital stay (>30 days) with odds ratios (ORs). When compared to the low risk group, the medium, high, and severe groups demonstrated increased ORs of hospital stay >30 days: 3.7 (95% CI 2.9–4.8), 7.9 (4.8–13.1), and 9.7 (5.8–16.1), respectively (ORs for hospital stay >7 days and >48 hours in derivation and validation sets are presented in Appendix 6 in S1 File).
Sensitivity analysis
OPERA demonstrated higher discrimination for in-hospital mortality when excluding patients with limitations in care in place or with palliative care input: AUC 0.79 (0.76–0.82) in derivation and 0.80 (0.76–0.83) in validation. In derivation AUCs for mortality in those >75 years and >85 were 0.75 (0.73–0.77) and 0.74 (0.71–0.76), respectively; on the validation set, AUCs were 0.78 (0.76–0.80) and 0.75 (0.73–0.78), respectively.
A nomogram based on the derivation risk equation is presented in Fig 2, with a worked example of an 85-year old man, an admission NEWS2 of 9, CFS of 7, high-risk on the MUST assessment and community-acquired AKI, giving an OPERA score of 35 with an estimated in-patient mortality of 75.9%. A further worked example using the regression equation and a graphical representation of mortality risk versus points scored are in the S1 File (Appendixes 8 and 9 in S1 File, respectively).
Example shown is of an acutely admitted 85-year-old male with a CFS of 7, creatinine 163 μmol/L (baseline 80), high risk MUST, and a NEWS2 of 9. Age 85 = 4 points (one point added for every 5 years above 65), CFS 7 = 14 points (individual CFS multiplied by two), AKI = 4 points, high risk MUST = 2 points (low = 0, medium = 1 and high = 2), male sex = 2 points, NEWS2 of 9 = 9 points (one point for every NEWS2 score). Total OPERA score 35 points. AKI—acute kidney injury, MUST—Malnutrition Universal Screening Tool.
An age constant “65(0.04)” within the equation allows for OPERA points only to be given for age past 65, as opposed to adding points from starting from age zero.
Model updating
Calibration plot analysis on the validation cohort demonstrated a small, statistically significant over-estimation of mortality risk (intercept difference -0.310) (Fig 3). The model was successfully re-calibrated through intercept updating (Appendix 10 in S1 File).
(A) before recalibration, (B) after recalibration. Reference line, 95% confidence interval, lowess smoothing curve, and the distribution of mortality and survival against predicted probabilities.
Discussion
This study derived and externally validated the Older Persons’ Emergency Risk Assessment (OPERA) score, using routinely collected demographic and clinical information. OPERA could offer clinicians an objective method to help risk stratify for early risk of death and prolonged hospital stay and inform discussions with patients and families about potential escalation of management or limitations of care. The model’s high NPV could be particularly useful in identifying those truly low risk, unlikely to require care escalation. Two large and geographically separated populations presenting non-specifically to ED were used to derive and validate OPERA. This supports potential generalisability and reduces the likelihood of over-optimism, though further external validation is desirable. Model over-fitting was limited by assessing a succinct number of clinical variables.
The model demonstrated good discrimination in both cohorts. Calibration plot analysis revealed a small over-prediction of mortality risk in the validation set however this was addressed with simple re-calibration techniques. The initial disparity between predicted and observed mortality may be a result of the significant differences in demographic and medical presentations between the hospitals. Future studies may investigate these differences between both hospitals. Controlling mortality predictions for these previously unaccounted variables may help elucidate the cause of the disparity. Additionally, the regression method used, created a linear coefficient of risk for each variable; some variables may have non-linear associations with mortality. This can be integrated into scoring systems, however it increases complexity and makes it harder to use in a busy clinical setting.
Excluding those without a documented frailty score limited sample size. Within both hospitals, all patients over the age of 65 are expected to have a CFS assessment, however, this may not have been judged necessary by hospital staff and hence not done if the patient was relatively well. Further analysis supports this, as a documented CFS was independently associated with increased age, NEWS2, MUST, palliative care input, a do not attempt resuscitation (DNAR) form, and all co-morbidities. This may explain why having a documented frailty score was associated with an increased odds ratio for in-hospital mortality (Appendix 11 in S1 File). A key potential benefit of the OPERA model however, is to aid mortality prediction, and those with a complete CFS assessment are more likely to form part of this at-risk group for whom such risk stratification is important. Importantly, OPERA revealed better discrimination when excluding patients with a DNAR form in-situ or receiving palliative care. This group have already had decisions made on appropriate care and as such, it is the remaining cohort for whom such a score may be of most clinical use. Individuals without documented NEWS2 were also excluded. It is suspected the majority of patients missing NEWS2 were medically fit patients, either admitted to ED in order to access other services such as dialysis in the renal unit or, admitted overnight to await a home package of care. As such, these would not be the demographic that would require risk stratification. Other causes for missing NEWS2 scores may include self-discharge prior to triage, declining physical observations, death in-between entering ED and triage, or data not being entered in the electronic notes system.
OPERA performed better for short-term mortality, likely due to the function of NEWS2 to quantify acute illness severity. The ability of the model to predict longer-term outcomes requires further investigation. OPERA provided odds ratios for extended hospital stay depending on the patient’s risk category, which could be useful for clinicians and hospital management as it provides an objective risk estimate, that could support discharge planning and resource allocation. Importantly however, the confidence intervals produced for each risk group overlapped. This may be due to the lower incidence of patients experiencing a length of stay over 30 days thus increasing uncertainty over these results.
OPERA did not discriminate for 30-day re-admission, aligning with previous research demonstrating the limitation of physiological data in predicting this outcome [18]. Alternatively, more historical variables such as the number of previous admissions and adverse events post-admission may do better [19, 20]. The most similar variable in this regard is frailty, however it too struggles to predict longer term out-of-hospital outcomes [21]. Future studies could derive prediction models focusing on broader holistic and patient centred outcomes, which could also include quality of life following discharge and likelihood of discharge to long-term institutional care.
Although still performing well, discrimination was slightly lower when the model was applied only to patients >85; at such extremes of age accurate prognostication will remain challenging. Few other tools have been described in this field. The Risk Index for Geriatric Acute Medical Admission (RIGAMA) has similar aims of this study, with similar discrimination in validation [22]. RIGAMA aimed to quantify frailty and mortality risk based on the index of accumulated deficits, where the number of deficits contribute to individual risk, regardless of what those deficits are [23]. OPERA uses conventional modelling methods where each variable contributes its own amount of risk and unlike RIGAMA, does not require a working diagnosis and the results for bloods tests indicated only for certain diagnoses. Soong et al., reported models using frailty with AUCs ranging 0.62–0.66 [24], and Gilbert et al., reported a frailty risk score from ICD-10 diagnostic codes with a C-statistic of 0.60 [25].
Clinical use and future implications
The ability of OPERA to predict in-hospital mortality makes it potentially applicable for use in older patients presenting acutely to hospital, for instance at the point of a post-take ward round. The score uses routinely collected data and is simple to use. Both MUST and the CFS can be quickly obtained by staff [26, 27], with the latter also demonstrating acceptable inter-rater agreement [7]. Additionally, NEWS2 is widely adopted in multiple UK healthcare settings [3]. Importantly, this data can be easily accessed from electronic patient record systems and calculated within hours of admission.
OPERA may be beneficial for junior staff and nurses, who make-up the majority of out-of-hours staffing, providing objective short-term prognostication and prompting potential need for care escalation or the opinion of a senior decision-maker. Furthermore, objective risk scores can inform difficult discussions with both patients and families by providing quantifiable population estimates in addition to an individual clinical assessment, rather than replacing clinical acumen. OPERA could also contribute to the early identification of palliative care needs which in turn may avoid inappropriate intensive treatment and help deliver appropriately holistic care [28]. As all variables were imported from the hospital’s electronic system, future services could automate elements of care recommendations, or provide real-time prompts to healthcare providers; computer based prediction tools can save time and limit mistakes compared to those limited to pen and paper [29]. Lastly, classification of patients into low, medium, high, and severe risk could be used to help determine care pathways. The utility of OPERA in acute older patient care could be further investigated with decision curve analysis [30] and impact studies [31].
Conclusion
This study derived and validated OPERA, a risk score that could be digitally integrated into currently available healthcare electronic hospital systems at the point of hospital admission. Future research could validate OPERA in external populations and consider impact analysis.
Acknowledgments
The authors wish to thank Mr Tim Short and Mr Mark Dennis for their invaluable help in the process of data retrieval.
References
- 1.
Wittenberg R, Sharpin L, McCormick B, Hurst J. Understanding emergency hospital admission of older people. 2014. https://www.chseo.org.uk/downloads/report6-emergencyadmissions.pdf
- 2. Baillie L, Gallini A, Corser R, Elworthy G, Scotcher A, Barrand A. Care transitions for frail, older people from acute hospital wards within an integrated healthcare system in England: a qualitative case study. International Journal of Integrated Care. 2014;14. pmid:24868193
- 3.
Royal College of Physicians. National early warning score (NEWS) 2. Standardising the assessment of acute-illness severity in the NHS. 2017. https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news-2
- 4. Churpek MM, Yuen TC, Winslow C, Hall J, Edelson DP. Differences in vital signs between elderly and non-elderly patients prior to ward cardiac arrest. Critical care medicine. 2015;43(4):816–822. pmid:25559439
- 5. de Groot B, Stolwijk F, Warmerdam M, Lucke JA, Singh GK, Abbas M, et al. The most commonly used disease severity scores are inappropriate for risk stratification of older emergency department sepsis patients: an observational multi-centre study. Scandinavian journal of trauma, resuscitation and emergency medicine. 2017;25(1):91. pmid:28893325
- 6. Church S., Rogers E., Rockwood K. et al. A scoping review of the Clinical Frailty Scale. BMC Geriatr 20, 393 (2020). pmid:33028215
- 7. Rockwood K, Song X, MacKnight C, Bergman H, Hogan DB, McDowell I, et al. A global clinical measure of fitness and frailty in elderly people. CMAJ. 2005;173(5):489–495. pmid:16129869
- 8. Faisal M, Scally A, Richardson D, Beatson K, Howes R, Speed K, et al. Development and external validation of an automated computer-aided risk score for predicting sepsis in emergency medical admissions using the patient’s first electronically recorded vital signs and blood test results. Critical care medicine. 2018;46(4):612–6188. pmid:29369828
- 9. Faisal M, Howes R, Steyerberg E, Richardson D, Mohammed MA. Using routine blood test results to predict the risk of death for emergency medical admissions to hospital: an external model validation study. QJM: An International Journal of Medicine. 2017;110(1):27–31. pmid:27486263
- 10.
Organization WH. International statistical classification of diseases and related health problems: World Health Organization; 2004.
- 11. Kellum JA, Lameire N, Aspelin P, Barsoum RS, Burdmann EA, Goldstein SL, et al. Kidney disease: improving global outcomes (KDIGO) acute kidney injury work group. KDIGO clinical practice guideline for acute kidney injury. Kidney international supplements. 2012;2(1):1–138.
- 12.
Elia M. The ‘MUST’ report. Nutritional screening for adults: a multidisciplinary responsibility Development and use of the ‘Malnutrition Universal Screening Tool’(‘MUST’) for adults A report by the Malnutrition Advisory Group of the British Association for Parenteral and Enteral Nutrition; 2003. https://www.bapen.org.uk/pdfs/must/must-report.pdf
- 13. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Annals of internal medicine. 2015;162(1):W1–W73. pmid:25560730
- 14. Janssen K, Moons K, Kalkman C, Grobbee D, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. Journal of clinical epidemiology. 2008;61(1):76–86. pmid:18083464
- 15. Mehta HB, Mehta V, Girman CJ, Adhikari D, Johnson ML. Regression coefficient–based scoring system should be used to assign weights to the risk index. J Clin Epidemiol. 2016;79: 22–28. pmid:27181564
- 16. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics. 1988;44(3):837–845. pmid:3203132
- 17. Karangwa I, Kotze D. Using the Markov chain Monte Carlo method to make inferences on items of data contaminated by missing values. American Journal of Theoretical and Applied Statistics. 2013;2(3):48–53.
- 18. LaMantia MA, Platts-Mills TF, Biese K, Khandelwal C, Forbach C, Cairns CB, et al. Predicting hospital admission and returns to the emergency department for elderly patients. Academic emergency medicine. 2010;17(3):252–259. pmid:20370757
- 19. García-Pérez L, Linertová R, Lorenzo-Riera A, Vázquez-Díaz J, Duque-González B, Sarría-Santamera A. Risk factors for hospital readmissions in elderly patients: a systematic review. QJM: An International Journal of Medicine. 2011;104(8):639–651. pmid:21558329
- 20. Franchi C, Nobili A, Mari D, Tettamanti M, Djade CD, Pasina L, et al. Risk factors for hospital readmission of elderly patients. European journal of internal medicine. 2013;24(1):45–51. pmid:23142413
- 21. Wou F, Gladman JR, Bradshaw L, Franklin M, Edmans J, Conroy SP. The predictive properties of frailty-rating scales in the acute medical unit. Age and ageing. 2013;42(6):776–781. pmid:23666406
- 22. Romero-Ortuno R, O’Dwyer C, Byrne D, O’Riordan D, Silke B. A Risk Index for Geriatric Acute Medical Admissions (RIGAMA). Acute Med. 2014;13(1):6–11. pmid:24616899
- 23. Rockwood K, Mitnitski A. Frailty in relation to the accumulation of deficits. The Journals of Gerontology Series A: Biological Sciences and Medical Sciences. 2007;62(7):722–727. pmid:17634318
- 24. Soong J, Poots A, Scott S, Donald K, Bell D. Developing and validating a risk prediction model for acute care based on frailty syndromes. BMJ open. 2015;5(10):e008457. pmid:26490098
- 25. Gilbert T, Neuburger J, Kraindler J, Keeble E, Smith P, Ariti C, et al. Development and validation of a Hospital Frailty Risk Score focusing on older people in acute care settings using electronic hospital records: an observational study. The Lancet. 2018;391(10132):1775–1782. pmid:29706364
- 26. Stratton RJ, Hackston A, Longmore D, Dixon R, Price S, Stroud M, et al. Malnutrition in hospital outpatients and inpatients: prevalence, concurrent validity and ease of use of the ‘malnutrition universal screening tool’(‘MUST’) for adults. British Journal of Nutrition. 2004;92(5):799–808.
- 27. Gregorevic KJ, Hubbard RE, Katz B, Lim WK. The clinical frailty scale predicts functional decline and mortality when used by junior medical staff: a prospective cohort study. BMC geriatrics. 2016;16(1):117. pmid:27250650
- 28. Khandelwal N, Kross EK, Engelberg RA, Coe NB, Long AC, Curtis JR. Estimating the effect of palliative care interventions and advance care planning on ICU utilization: a systematic review. Critical care medicine. 2015;43(5):1102. pmid:25574794
- 29. Prytherch DR, Smith GB, Schmidt P, Featherstone PI, Stewart K, Knight D, et al. Calculating early warning scores—a classroom comparison of pen and paper and hand-held computer methods. Resuscitation. 2006;70(2):173–178. pmid:16806641
- 30. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565–574. pmid:17099194
- 31. Kappen TH, van Klei WA, van Wolfswinkel L, Kalkman CJ, Vergouwe Y, Moons KG. Evaluating the impact of prediction models: lessons learned, challenges, and recommendations. Diagnostic and prognostic research. 2018;2(1):11.