https://orcid.org/0000-0003-0062-676X
Figures
Abstract
Objectives
The Nutrition Risk in the Critically Ill (NUTRIC) score has been advocated as a screening tool for nutrition risk assessment in critically ill patients. It was developed and validated to predict 28-day mortality using Acute Physiology and Chronic Health Evaluation II (APACHE II) score as one of its components. However, nowadays the Simplified Acute Physiology Score 3 (SAPS 3) demonstrates better performance. We aimed to test the performance of NUTRIC score in predicting 28-day mortality after replacement of APACHE II by SAPS 3, and the interaction between nutrition adequacy and mortality.
Methods
Adult patients who received nutrition therapy and remained >3 days in intensive care unit were retrospectively evaluated. In order to replace APACHE II component, we used ranges of SAPS 3 with similar predicted mortality. Discrimination between these tools in predicting 28-day mortality was assessed using the ROC curve, calibration was evaluated with calibration belt, and correlation with intraclass correlation. The relationship between nutritional adequacy and mortality was assessed in a subgroup with available data.
Results
542 patients were analyzed (median age of 78 years old, 73.4% admitted for non-surgical reasons and 28-day mortality was 18.1%). Mortality prediction discrimination did not differ between tools (p>0.05), but showed a good agreement (intraclass correlation 0.86) with good calibration. In the subgroup analysis for nutritional adequacy (n = 99), no association with mortality was observed.
Citation: Souza IAO, Ribeiro PC, Jonckheer J, De Waele E, Taniguchi LU (2022) Performance of NUTRIC score to predict 28-day mortality in critically ill patients after replacing APACHE II with SAPS 3. PLoS ONE 17(7): e0270455. https://doi.org/10.1371/journal.pone.0270455
Editor: Chiara Lazzeri, Azienda Ospedaliero Universitaria Careggi, ITALY
Received: March 20, 2022; Accepted: June 10, 2022; Published: July 1, 2022
Copyright: © 2022 Souza et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
List of abbreviations: NUTRIC score, NUTrition Risk in the Critically ill score; APACHE, Acute Physiology and Chronic Health Evaluation; SAPS 3, Simplified Acute Physiology Score 3; ROC, receiver operating characteristics curve; ICU, intensive care unit; SAPS-NUTRIC, NUTRIC after replacement of APACHE II by SAPS 3; BMI, body mass index; CI, confidence interval; NRI, net reclassification index
Introduction
Since malnutrition is prevalent in the acute hospital setting and associated with poor clinical outcomes [1], the latest American guidelines suggest nutritional risk assessment in all patients admitted to the intensive care unit (ICU) [2]. Such evaluation would allow for the appropriate adjustment of nutritional therapies, and identification of those most likely to benefit from early nutritional support. The Nutrition Risk in the Critically Ill (NUTRIC) score was originally proposed in 2011 as a tool to assess the relationship between nutritional risk and 28-day mortality in critically ill patients [3]. It has been validated by other authors [4], including a study applying a Portuguese version of the tool [5].
In the conceptual model of NUTRIC score, acuity would be one of mediators of worse clinical outcomes [3]. Acute Physiology and Chronic Health Evaluation II (APACHE II) score was originally included as such a severity-of-illness variable. However, nowadays APACHE II is outdated, and no longer accurate to represent acuity [6], and can only be calculated after 24 hours of ICU stay. Therefore, better performing severity-of-illness scores might be more adequate to update NUTRIC score. Moralez et al. recently demonstrated that Simplified Acute Physiology Score 3 (SAPS 3), which is performed with information within the first hour of ICU stay, is the most accurate prognostic model in a multicenter Brazilian cohort [7]. Consequently, it is one of the best options.
Our study was designed to test the performance of NUTRIC after replacement of APACHE II by SAPS 3 (SAPS-NUTRIC) and the relationship between nutritional adequacy and 28-day mortality stratified by NUTRIC and SAPS-NUTRIC.
Methods
Study design and patient setting
This retrospective cohort study included adult patients (≥ 18 years old) admitted to the medical-surgical ICU of Hospital Sírio-Libanês (located in São Paulo, Brazil) between February 2016 and February 2019, who received enteral and / or parenteral nutrition therapy and remained > 3 days in the ICU. Exclusion criteria were pregnant women and patients with 48-hours death expectancy, palliative care or transferred from another hospital.
Cloud-based administrative (Epimed®) [8] and multidisciplinary nutrition therapy team databases were use as source of patient’s data, comprising demographic, clinical and body mass index (BMI); invasive organ supports (mechanical ventilation, use of vasoactive drugs, dialysis); organ failure (SOFA score) [9], nutritional therapy received during hospitalization (enteral, parenteral, amount of calories and protein); and clinical outcomes (discharges, deaths, transfers). The study was approved by the local institutional ethics committee (CAAE: 65382117.8.0000.5461), which waived informed consent due to the observational and retrospective design of the study.
Scores calculation and study definitions
APACHE II score and SAPS 3 were calculated as originally described [10–12]. Of note, APACHE II uses the worst value of its parameters in the first 24 hours of ICU stay, while SAPS 3 parameters are from the first hour of ICU stay. NUTRIC score was calculated as the modified version validated by Rahman et al. without interleukin-6 [4]. High nutritional risk was considered when the NUTRIC score had values ≥ 5 as previously defined in guidelines [2]. Frailty was defined as a Clinical Frailty Scale ≥ 5 [13].
Nutrition adequacy was calculated as the proportion of the caloric prescription received (either enterally or parenterally) during the first week of ICU admission and was considered adequate when patients received ≥ 80% of calories and protein prescribed. Goal of caloric intake was calculated according to the American guidelines [2]: 25 Kcal/kg body weight/day for patients with BMI < 30 or 14 Kcal/kg body weight/day for those patients with BMI ≥ 30. Goal of protein intake was 1.3 g/kg body weight /day. This analysis was restricted to patients whose data on the prescribed and infused calorie were available.
SAPS-NUTRIC derivation
We derived SAPS-NUTRIC score from the previous one described by Rahman et al. [4]. Common variables were age, Sequential Organ Failure Assessment (SOFA), number of comorbidities and length of hospital stay before ICU admission. In order to convert APACHE II score into SAPS 3 as a substitute variable in SAPS-NUTRIC we first estimated the predicted range of probability of death for each stratum of APACHE II in NUTRIC score (i.e. less than 15; from 15 to 19; from 20 to 27; higher or equal to 28) without adjustments of admission reason. Then, SAPS 3 values with equivalent predicted probability of death were chosen (less than 53; from 53 to 57; from 58 to 74; higher or equal to 75). This procedure was chosen in order to maintain similar severity of illness prediction from the original NUTRIC derivation, but with a more accurate and updated score.
Statistical analysis
Continuous variables were tested for normality using the Kolmogorov-Smirnov test. Continuous parametric variables were compared by unpaired t-test for analysis between groups and by paired t-test for paired analysis. Categorical variables were evaluated using the χ² test or Fisher’s exact test, when appropriate. Continuous nonparametric variables were evaluated using the Wilcoxon or Mann-Whitney tests.
Primary outcome was 28-day mortality (same primary outcome in the original study and validation studies) [3, 4]. To evaluate the predictive performance of NUTRIC and SAPS-NUTRIC score, we first compared discrimination between these models with the area under the Receiver Operating Characteristics (AUROC) curve by the Delong method [14]. Then, calibration was assessed by the calibration belt method as described by the GiViTI group [15]. This method applies a generalized polynomial logistic function between the outcome and the logit transformation of the estimated predicted probability, with the respective 95% confidence intervals (CI) boundaries. A statistically significant deviation from the bisector (the line of perfect calibration) occurs when the 95% CI boundaries of the calibration belt do not include the bisector [15]. Finally, Brier score as an overall performance measure was calculated using the standard formula [16].
Correlation between NUTRIC and SAPS-NUTRIC score was compared with intraclass correlation (2-way mixed effects model). Patients’ reclassification was analyzed with net reclassification index (NRI). Since mortality rate are not expected to be equal to survival rate (i.e. 50% to 50%) in our cohort, absolute NRI is expected to be more accurate than additive NRI [17]. Furthermore, the association between nutritional adequacy and 28-day mortality was assessed by stratifying the patients according to their nutritional risk group and compared with chi-squared test. Statistical analyzes were performed by using the software SPSS® Statistics 20 (IBM Corp., Armonk, New York, USA), STATA (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC) and R (http://www.r-project.org). Significant differences were stated at 5% level.
Results
During the study period, we studied 542 patients; 282 (52%) were categorized as low and 260 (48%) as high nutritional risk according to the NUTRIC score (Table 1). Those patients categorized as high nutritional risk were older, more frequently admitted for non-surgical reasons, with higher prevalence of comorbidities and frailty compared to low nutritional risk patients. They also had higher severity-of-illness scores at ICU admission, required invasive organ support more frequently and had higher 28-day mortality compared to low risk patients (25.4% vs 11.4% respectively, p<0.001). Some differences could be observed between low and high nutritional risk groups regarding types of nutritional support (Table 2).
Comparison between NUTRIC and SAPS-NUTRIC performance
Mean values of NUTRIC and SAPS-NUTRIC scores were 4.5 ± 1.9 and 4.1 ± 1.8 respectively. In our cohort, 326 (60.1%) were categorized as low and 216 (39.9%) as high nutritional risk according to the SAPS-NUTRIC score.
Our analysis showed that both NUTRIC and SAPS-NUTRIC demonstrated similar performance in discriminating 28-day mortality (p = 0.504), with adequate calibration evaluated by the calibration belt method. Brier scores were similar (Table 3 and S1 Appendix). Intraclass correlation between these two tools was 0.86 (CI 95% 0.83–0.88). Reclassification analysis demonstrated an additive NRI of -0.06 with an absolute NRI of 5.16% (S2 Appendix).
Nutritional adequacy and 28-day mortality
For the 99 patients analyzed for nutritional adequacy, 35 patients (35.4%) were categorized as low nutritional risk and 64 patients (64.6%) as high nutritional risk, according to the NUTRIC score, and 52 patients (52.5%) were categorized as low nutritional risk and 47 patients (47.5%) as high nutritional risk, according to the SAPS-NUTRIC score. No differences in 28-day mortality could be observed between those groups whose energy or protein goals ≥ 80% of prescribed were achieved compared with those whose goals were not reached (Fig 1).
28-day mortality rate stratified by nutritional risk according to the adequacy of energy (A) and protein (B) received. No differences in 28-day mortality were observed in critically ill patients who achieved or not adequate energy (A) and protein supply (B), regardless their nutritional risk (p > 0.05 for each comparison).
Discussion
The main findings of the present study were: (1) the APACHE II replacement by SAPS 3 in NUTRIC score (SAPS-NUTRIC) had comparable performance in our single-center retrospective cohort to predict 28-day mortality, and (2) in a subgroup analysis, we did not observe an interaction between nutritional adequacy, NUTRIC or SAPS-NUTRIC categories and mortality.
Prevalence of high nutritional risk categorized by the NUTRIC score is variable in the literature (ranging from 32% to 57%) [5, 18–20], probably due to different case-mix in published cohorts. Therefore, discrimination to predict mortality described by ROC curves ranges between 0.64–0.79 [3–5, 18, 21]. It is reassuring that the ROC curves in our study are within these reported ranges and no difference was observed between NUTRIC and SAPS-NUTRIC score, suggesting similar discrimination. Of note, our results are similar to those reported in some of the largest multicenter validation studies (ROC curve of 0.65 in Rahman et al’ publication with 1199 patients [4] and 0.66 in Mendes et al’ publication with 1143 patients [5]) and a similar publication from Brazil (0.62 in Toledo et al’ publication) [21].
Calibration is another aspect of predictive performance that is relevant for new models, specially because it deteriorates over time [22–24]. Since Hosmer and Lemeshow goodness of fit test usually lacks statistical power to reject poor calibration [22], new methods to assess calibration have been described such as the calibration belt [15]. Both NUTRIC and SAPS-NUTRIC models had excellent calibration without any underestimation or overestimation in our cohort. This is relevant because calibration is considered the “most important property of a model” [17] and good calibration is required to properly evaluate reclassification. Our results with an absolute NRI of 5.16% suggest an improvement in reclassifying patients with SAPS-NUTRIC instead of NUTRIC model.
However, why should one bother to use the new SAPS-NUTRIC instead of the old NUTRIC score since their predictive performance showed similar results of discrimination, calibration and Brier scores? In fact, SAPS 3 can be calculated in the first hour of ICU admission with fewer variables while APACHE II can only be calculated after 24 hours, which might allow one to screen nutritional risk at admission and formulate a nutritional plan. Also, many units have adopted newer and updated prognostic scores instead of APACHE II (for examples, in Brazil the National Board of Intensive Care Medicine uses SAPS 3 in the National Registry of Intensive Care Units) [25]. Consequently, to calculate NUTRIC score this would demand calculation of two prognostic scores. Finally, performance between NUTRIC and SAPS-NUTRIC scores were similar with high intraclass correlation (suggesting good content agreement), but absolute NRI suggested some proper reclassification in a proportion of patients with SAPS-NUTRIC.
Our subgroup analysis did not demonstrate an association between 28-day mortality and energy-protein adequacy, even in those categorized as high nutritional risk by both tools. This finding is similar to those demonstrated by Arabi et al. who studied in a post-hoc analysis of the PermiT Trial the interaction between nutritional risk groups (stratified by NUTRIC score) and permissive underfeeding or standard feeding [26]. Similar outcomes considering risk groups were observed in their study, raising the question whether NUTRIC score, which has not been prospectively validated, is appropriate to identify who might benefit for early aggressive feeding and who might be harmed [27].
Globally, our results suggest that NUTRIC score may represent the severity of illness and predicted mortality, and are in accordance to a recent systematic review [28]. However, the utility of this tool, which does not incorporate anthropometric or nutritional-related variables, to screen nutritional risk in critically ill patients still need validation in prospective studies. Most of previous literature are from observational studies, which may suffer from confounding or indication bias. Even in the original publication of NUTRIC score the authors explained that the lack of nutritional variables was mainly due to difficulty to obtain these information in their database (>70% of missingness), a fact that might influence the inclusion of these variables in their model [3]. Thus, the utility of a prognostic score such as NUTRIC as a clinical decision tool should be further evaluated to avoid misapplication [29].
Our study has some limitations. First, our study is a single-center cohort, which may be influenced and biased by local practice and case-mix. However, our performance prediction analysis is in concordance with some previous studies and this is reassuring. Second, our nutritional adequacy analysis was performed in a smaller subgroup, what is prone to error type II and might explain the lack of association between energy-protein supply and outcomes. However, a post-hoc analysis of a large randomized controlled trial suggested the same results [26]. Third, our population had a median age of 78 years old. If our results are applicable to younger patients remains to be demonstrated. Nevertheless, Toledo et al. demonstrated similar findings in a younger population (mean age of 63) [21]. Finally, functional and health-related quality of life measures would be one of the most relevant outcomes in nutrition studies, but usually are secondary outcomes in large randomized controlled trials [30]. We do not have this information in our database, but we acknowledge the importance of this outcome for future studies in critical care nutrition, since like our study the other published validation cohorts of NUTRIC Score applied mortality as the primary outcome.
Conclusions
In conclusion, in our cohort the performance of NUTRIC score with SAPS 3 (SAPS-NUTRIC) is similar to the original tool, with some improvement in reclassification. Therefore, it might be an option when APACHE II is not available. However, the utility of both tools to guide clinical decisions are yet to be demonstrated in further studies.
Supporting information
S1 Appendix. Calibration plot assessed by the calibration belt method.
https://doi.org/10.1371/journal.pone.0270455.s001
(DOCX)
S1 Dataset. Dataset with anonymized information for primary analysis replication.
https://doi.org/10.1371/journal.pone.0270455.s003
(XLSX)
References
- 1. Barker LA, Gout BS, Crowe TC. Hospital malnutrition: prevalence, identification and impact on patients and the healthcare system. Int J Environ Res Public Health. 2011;8: 514–527. pmid:21556200
- 2. McClave SA, Taylor BE, Martindale RG, Warren MM, Johnson DR, Braunschweig C, et al. Guidelines for the provision and assessment of nutrition support therapy in the adult critically ill patient: Society of Critical Care Medicine (SCCM) and American Society for Parenteral and Enteral Nutrition (A.S.P.E.N.). JPEN J Parenter Enteral Nutr. 2016;40: 159–211. pmid:26773077
- 3. Heyland DK, Dhaliwal R, Jiang X, Day AG. Identifying critically ill patients who benefit the most from nutrition therapy: the development and initial validation of a novel risk assessment tool. Crit Care. 2011;15: R268. pmid:22085763
- 4. Rahman A, Hasan RM, Agarwala R, Martin C, Day AG, Heyland DK. Identifying critically-ill patients who will benefit most from nutritional therapy: Further validation of the "modified NUTRIC" nutritional risk assessment tool. Clin Nutr. 2016;35: 158–162. pmid:25698099
- 5. Mendes R, Policarpo S, Fortuna P, Alves M, Virella D, Heyland DK, et al. Nutritional risk assessment and cultural validation of the modified NUTRIC score in critically ill patients-A multicenter prospective cohort study. J Crit Care. 2017;37: 249. pmid:27938907
- 6. Soares M, Dongelmans DA. Why should we not use APACHE II for performance measurement and benchmarking? Rev Bras Ter Intensiva. 2017;29: 268–270. pmid:28876406
- 7. Moralez GM, Rabello L, Lisboa TC, Lima M, Hatum RM, De Marco FVC, et al. External validation of SAPS 3 and MPM0-III scores in 48,816 patients from 72 Brazilian ICUs. Ann Intensive Care. 2017;7: 53. pmid:28523584
- 8. Zampieri FG, Soares M, Borges LP, Salluh JIF, Ranzani OT. The Epimed Monitor ICU Database(R): a cloud-based national registry for adult intensive care unit patients in Brazil. Rev Bras Ter Intensiva. 2017;29: 418–426. pmid:29211187
- 9. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22: 707–710. pmid:8844239
- 10. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13:818–29. pmid:3928249
- 11. Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med. 2005;31: 1336–1344. pmid:16132893
- 12. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005;31: 1345–1355. pmid:16132892
- 13. Bagshaw SM, Stelfox HT, McDermid RC, Rolfson DB, Tsuyuki RT, Baig N, et al. Association between frailty and short- and long-term outcomes among critically ill patients: a multicentre prospective cohort study. CMAJ. 2014;186: E95–E102. pmid:24277703
- 14. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44: 837–845. pmid:3203132
- 15. Finazzi S, Poole D, Luciani D, Cogo PE, Bertolini G. Calibration belt for quality-of-care assessment based on dichotomous outcomes. PLoS One. 2011;6: e16110. pmid:21373178
- 16. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21: 128–138. pmid:20010215
- 17. Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ, et al. Discrimination and calibration of clinical prediction models: Users’ guides to the medical literature. JAMA. 2017;318: 1377–1384. pmid:29049590
- 18. Mukhopadhyay A, Henry J, Ong V, Leong CS, Teh AL, van Dam RM, et al. Association of modified NUTRIC score with 28-day mortality in critically ill patients. Clin Nutr. 2017;36: 1143–1148. pmid:27582120
- 19. Compher C, Chittams J, Sammarco T, Nicolo M, Heyland DK. Greater protein and energy intake may be associated with improved mortality in higher risk critically ill patients: A multicenter, multinational observational study. Crit Care Med. 2017;45: 156–163. pmid:28098623
- 20. Compher C, Chittams J, Sammarco T, Higashibeppu N, Higashiguchi T, Heyland DK. Greater nutrient intake is associated with lower mortality in Western and Eastern critically ill patients with low BMI: A multicenter, multinational observational study. JPEN J Parenter Enteral Nutr. 2019;43: 63–69. pmid:29959851
- 21. Toledo DO, Junior JM, Toloi JM, de Assis T, Serra LM, do Carmo PG, et al. NUTRIC-S proposal: Using SAPS 3 for mortality prediction in nutritional risk ICU patients. Clin Nutr. Exp. 2020;31: 19–27.
- 22. Labarere J, Renaud B, Fine MJ. How to derive and validate clinical prediction models for use in intensive care medicine. Intensive Care Med. 2014;40: 513–527. pmid:24570265
- 23. Soares M, Salluh JI. Validation of the SAPS 3 admission prognostic model in patients with cancer in need of intensive care. Intensive Care Med. 2006;32: 1839–1844. pmid:16977481
- 24. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34: 1297–1310. pmid:16540951
- 25.
Relatório de Unidades de Terapia Intensiva [http://www.amib.org.br/censoamib/relatorio-de-unidades-de-terapia/]; 2010
- 26. Arabi YM, Aldawood AS, Al-Dorzi HM, Tamim HM, Haddad SH, Jones G, et al. Permissive underfeeding or standard enteral feeding in high- and low-nutritional-risk critically ill adults. Post hoc analysis of the PermiT Trial. Am J Respir Crit Care Med. 2017;195: 652–662. pmid:27589411
- 27. Marik PE, Rice TW. Feeding the critically ill: more questions than answers! Am J Respir Crit Care Med. 2017;195: 555–556. pmid:28248150
- 28. Reis AMD, Fructhenicht AVG, Moreira LF. NUTRIC score use around the world: a systematic review. Rev Bras Ter Intensiva. 2019;31: 379–385. pmid:31618358
- 29. Moskowitz A, Andersen LW, Cocchi M, Donnino MW. The misapplication of severity-of-illness scores toward clinical decision making. Am J Respir Crit Care Med. 2016;194: 256–258. pmid:27299379
- 30. Bear DE, Wandrag L, Merriweather JL, Connolly B, Hart N, Grocott MPW. Enhanced Recovery After Critical Illness Programme Group i: The role of nutritional support in the physical and functional recovery of critically ill patients: a narrative review. Crit Care. 2017;21: 226.