https://orcid.org/0000-0002-4709-0600
Figures
Abstract
Objective
To establish and validate an individualized nomogram to predict mortality risk within 30 days in patients with sepsis from the emergency department.
Methods
Data of 1205 sepsis patients who were admitted to the emergency department in a tertiary hospital between Jun 2013 and Sep 2021 were collected and divided into a training group and a validation group at a ratio of 7:3. The independent risk factors related to 30-day mortality were identified by univariate and multivariate analysis in the training group and used to construct the nomogram. The model was evaluated by receiver operating characteristic (ROC) curve, calibration chart and decision curve analysis. The model was validated in patients of the validation group and its performance was confirmed by comparing to other models based on SOFA score and machine learning methods.
Results
The independent risk factors of 30-day mortality of sepsis patients included pro-brain natriuretic peptide, lactic acid, oxygenation index (PaO2/FiO2), mean arterial pressure, and hematocrit. The AUCs of the nomogram in the training and verification groups were 0.820 (95% CI: 0.780–0.860) and 0.849 (95% CI: 0.783–0.915), respectively, and the respective P-values of the calibration chart were 0.996 and 0.955. The DCA curves of both groups were above the two extreme curves, indicating high clinical efficacy. The AUC values were 0.847 for the model established by the random forest method and 0.835 for the model established by the stacking method. The AUCs of SOFA model in the model and validation groups were 0.761 and 0.753, respectively.
Citation: Wang B, Chen J, Pan X, Xu B, Ouyang J (2024) A nomogram for predicting mortality risk within 30 days in sepsis patients admitted in the emergency department: A retrospective analysis. PLoS ONE 19(1): e0296456. https://doi.org/10.1371/journal.pone.0296456
Editor: Jincheng Wang, Hokkaido University: Hokkaido Daigaku, JAPAN
Received: June 29, 2023; Accepted: December 4, 2023; Published: January 25, 2024
Copyright: © 2024 Wang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data contain potentially sensitive information, and data are owned by Affiliated Dongyang Hospital of Wenzhou Medical University. The Department of Scientific Research has imposed them. If there is any interest, please contact the Maofeng Wang (Email: wzmcwmf@163.com).
Funding: This work was funded by The Science and Technology Bureau of JinHua with grant number 2023-4-213.The funders had no role in study design,data collection analysis,decision to publish,or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Sepsis is a life-threatening condition resulting from an excessive immune response to infection, and is characterized by systemic inflammation that eventually culminates to multi-organ dysfunction [1, 2]. An estimated 11 million sepsis-related deaths occur worldwide every year, which accounts for 19.7% of all deaths. The global age-standardized mortality rate for sepsis is 148 deaths per 100,000 person-years [3]. Even in developed countries, the incidence rate of sepsis is 437 cases per 100,000 person-years and the hospital mortality rate due to sepsis is 17% [4]. The risk of death increases considerably once the sepsis progresses to septic shock [5]. Furthermore, the high fatality rate of sepsis leads to increased healthcare costs. Therefore, it is crucial to identify the sepsis patients with a high risk of death in order to improve prognosis through early interventions [6].
At present, the outcome of patients with sepsis is predicted using scoring systems based on various parameters. The “Acute Physiology and Chronic Health Evaluation II” or APACHEII score is widely used to assess the prognosis of critically ill sepsis patients [7]. However, studies show that the APACHE II score underestimates the risk of death in sepsis patients [8]. Similarly, the “Simplified Acute Physiology Score II” (SAPS II) has not been reliable in external validation studies [9, 10]. On the other hand, the sequential organ failure assessment or SOFA score is based on respiratory, cardiovascular, hepatic, coagulation, renal and neurological indices systems, and describes the development of multiple organ dysfunction, but is not effective for evaluating prognosis [11]. In addition, there is delay in prediction for the scoring systems mentioned above.
Several risk prediction models have been developed to evaluate the prognosis of sepsis patients, but there are still some deficiencies, the collinearity between continuous variables and logitp and the multi-collinearity between enrolled variables and the cross-interactions between enrolled variables have not evaluated before multivariate logistic regression analysis [12–14]. For machine learning models, it is hard to evaluate what variables were included and how much the enrolled variables contributed to the outcomes, leading to limited significance in practice [15, 16]. Although a few prediction models have been established to predict the death risk of sepsis patients, they are focused on the severe sepsis patients with shock or decreased blood pressure and they are accompanied by higher mortality [17–20]. In addition, the sepsis patients who are admitted in the emergency department usually suffer from emergent episode, consequently resulting uncertainty in prognosis. Thus, there have not been prediction model to evaluate the short-term prognosis of sepsis patients effectively who are admitted at emergency department. To this end, our aim was to establish a model to evaluate the mortality risk among the sepsis patients who are admitted at emergency department, displayed as a nomogram graph, which could provide a more understandable outcome measure and have been broadly adopted [21, 22].
Materials and methods
Study patient inclusion and exclusion
Sepsis patients who were admitted to the emergency department in Dongyang People’s Hospital from June 1, 2013, to September 1, 2021 were enrolled in this study (a retrospective analysis). The data collection was performed by extracting the information from clinical record database, which was constructed (by Hangzhou Le9 Healthcare Technology Co., Ltd) based on the patients’ clinical record after removing the personalized identifying information. The inclusion criteria were as follows: 1) definite diagnosis of an infection, and 2) an increase in SOFA score by 2 points or more [23]. Patients younger than 18 years of age, or with positive diagnosis of leukemia, lymphoma, chronic respiratory failure or end-stage tumor, uncertain prognosis indicators, missing data, incomplete treatment (given up treatment) or unknown infection were excluded.
This study was approved by the Ethics Committee of Dongyang People’s Hospital (NO. 2021-YX-093). The written consents of the patients or their families were obtained. Personal information was completely deleted from the data. This study was conducted in accordance with the principles of the Helsinki Declaration and its amendments.
Variable collection
The basic demographic information (gender and age), Glasgow coma score (GCS), and the mean arterial pressure (MAP) were obtained from nursing record. The hematological indices at the time of admission such as oxygenation index (PaO2/FiO2), C reactive protein (CRP), creatinine (Cr), emergency procalcitonin (PCT), activated partial thromboplastin time (APTT), prothrombin time (PT), albumin, pro-brain natriuretic peptide (pro-BNP), lactic acid (LAC), hematocrit (HCT), platelet (PLT), white blood cell (WBC) count, total bilirubin (T-bil), sodium (Na), were measured and recorded in the clinical record within 24h of admission. The currently used international common units were used in this study.
Data processing and selection of variables
Due to no linearity with logitp or cross-interactions between each other or multicollinearity among the enrolled variables, some variables were transformed into categorical variables.The patients were randomly divided into the training and validation groups at the ratio of 7:3 using the createDataPartition function in R statistical package and baseline characteristics between these two groups were analyzed. Univariate analysis was performed using the twogrps function of the CBCgrps package to identify the significant risk factors of mortality in training groups [24]. Multiple-collinearity among the variables and the significant variables were identified using variance inflation factor (VIFs) values. Variables had no cross-interactions each other (P > 0.05), were linear with logitp and without multiple collinearities (VIFs<10) were further included in the multivariate regression analyses and stepwise regression analysis. The independent risk factors were then used to establish the predictive logistic regression model and a nomogram graph was drawn to visually evaluate how much they contributed to the mortality risk [25].
Evaluating the performance of established predict model
The discriminatory power of the model was analyzed by plotting receiver operating characteristic (ROC) curve, and the area under the curve (AUC) was calculated. AUC > 0.75 indicates good discrimination ability [26]. The cutoff value was determined based on the maximal Youden indexes in the ROC analysis, along with the sensitivity, specificity, the prediction accuracy, negative predictive value (NPV) and positive predictive value (PPV). A calibration chart was used to evaluate the goodness of fit, with a P value greater than 0.05 suggesting an adequate goodness of fit [27]. The decision curve analysis (DCA) was plotted to assess the clinical validity of the model. The distance of the model curve from the “All” and “None’ reference curves is indicative of clinical significance [28, 29]. Then, the established model was validated on patients in the validation group.
Comparing to other prediction methods
Subsequently, random forests and stacking methods were employed to establish a model in the training group, the discrimination power of the models was compared to nomogram prediction model in the validation group by Delong test. Moreover, models based on the SOFA score were conducted in training group, the AUCs of the models were compared to nomogram prediction model in both training group and validation group.
Statistical analysis
Variables with normal distribution were expressed as mean ± standard deviation, and data that were not normally distributed were expressed as median with quartiles. The categorical variables were displayed as counts with percentages. The significances between two groups were analyzed by twogrps function in CBCgrps packages. Other packages included caret, CBCgrps, tidyverse, car, rms, foreign, DataExplorer, boot, gbm, caretEnsemble, C50, xgboost, randomForest, mlr, reportROC.
Results
Baseline characteristics of the training population and validation population
A total of 2257 patients with sepsis from the emergency department were initially recruited (S1 Fig), of which 1052 patients were excluded due to age below 18 years (n = 75), unknown infections (n = 356), incomplete treatment (n = 32), lack of prognostic information (n = 125), comorbidities such as leukemia and chronic respiratory failure (n = 155), and missing data (n = 309). Finally, 1205 patients were included that were randomly divided into the training group with 844 cases and 139 deaths, and the verification group with 361 cases and 52 deaths. Accordingly, the mortality was calculated to be 16.5%. All baseline characteristics of the training population and validation population are shown in Table 1. There were no significant differences in the baseline characteristics between the training population and validation population (all p>0.05).
Identification of variables associated with mortality risk in sepsis patients
In the training group, univariate analysis showed that 7 indices, including MAP, creatinine, oxygenation index, albumin, hematocrit, Pro.bnp, lactic acid were significantly correlated to prognosis (p<0.001) (Table 2).
None of the significant variables showed obvious cross-interactions and multiple collinearity (S1 and S2 Tables), and they were all categorical variables. Multivariate logistics analysis further indicated that pro-bnp, lactic acid, PaO2/FiO2, map, and hematocrit were significantly associated with death (p<0.05; Table 3). Finally, stepwise regression analysis identified all of the above variables as independent risk factors related to the death of sepsis patients within 30 days (p<0.05; Table 3).
Establishment and validation of the predictive nomogram for 30-day mortality
The independent risk factors of 30-day mortality were used to establish prediction model and was displayed as a nomogram (Fig 1). The score of each variable was calculated after matching to upper score line, and the individual scores were summed up to obtain the total score. Using the lower score lines and parallel prediction line, the total score was matched to the death risk.
Map, mean arterial pressure; pro.bnp, pro-brain natriuretic peptide.
ROC curves, calibration belts and DCA curves of established model in the training group and the validation group
The discriminatory power of the nomogram was assessed by ROC analysis (Fig 2A). The AUC value of the nomogram in the training group was 0.820 (95% CI: 0.780–0.860), which is indicative of good discrimination ability. The cutoff value was 0.259, with sensitivity 89.7% (95% CI:87.2%-91.9%) and specificity 59.9% (95% CI:50.4%-66.9%). The prediction accuracy was 0.846 (95% CI:84.6%-84.6%), with a PPV of 0.529 (95% CI:45.0%-60.8%) and a NPV of 0.917 (95% CI:89.7%-93.8%).
A, ROC curve; B, calibration chart; C, DCA curve.
The calibration chart was used to assess the goodness of fit, and the p-value, R2 and slope in the training group were 0.996, 0.376 and 1 respectively, suggesting good fit (Fig 2B). Finally, the DCA curves of the nomogram in the training group (Fig 2C) were above the two extreme curves, indicating good clinical efficacy.
The prediction model was further validated in the validation group. As shown in Fig 3A, the AUC of the model in the validation group was 0.849 (95% CI: 0.783–0.914), comparable to that in the training group. The prediction accuracy was 0.842 (95% CI:84.1%-84.3%), with a PPV of 0.468 (95% CI:35.8%-57.8%) and a NPV of 0.947 (95% CI:92.1%-97.3%).
A, ROC curve; B, calibration chart; C, DCA curve.
The calibration chart (p = 0.955, R2–0.407, slope—1; Fig 3B) and DCA curves (Fig 3C) of the nomogram respectively showed good fitting and clinical efficacy in the verification group.
Comparing to other prediction methods
The AUC value of the random forests modeling was 0.847 in the validation group (Fig 4A), which was comparable to the established logistic prediction model (P = 0.430). In the stacking method, three methods (SVM, support vector machine; XGBoost, extreme gradient boosting; C 5.0) were chosen. The calibration of the final integrated model is poor in the validation group (Fig 4B). The AUC value of the final integrated model was 0.835 in the validation group, which was comparable to the established logistic prediction model (P = 0.280) (Fig 4C).
A, ROC curve of the model by Random Forest method; B, calibration chart of the model by Stacking method; C, ROC curve of the model by Stacking method.
Based on the SOFA scoring system, the AUC of the prediction model in the training group was 0.761 (Fig 5A), which was significantly lower than that of established nomogram prediction model (P = 0.008, Delong test). The AUC of the prediction model in the validation group was 0.753 (Fig 5B), which was significantly lower than that of established nomogram prediction model (P = 0.022).
A, ROC comparison in training group; B, ROC comparison in validation group.
Discussion
Sepsis is the result of a disproportionate immune response to infection, which is unable to eliminate invading pathogens [30, 31] and manifests as multiple organ dysfunction entailing coagulation disorder, cardiac dysfunction, renal insufficiency, nutritional disorder etc. We established a prediction model for 30-day mortality in sepsis patients on the basis of pro-bnp, lactic acid, hct,and map at admission. The discriminatory power, calibration and clinical efficacy of the nomogram were significant, which indicates its potential for identifying sepsis patients at high risk of death in order to initiate intervention measures at the earliest.
Lactic acid is an indicator of oxygen metabolism and is used to gauge the prognosis in sepsis and other critical diseases [32–34]. Pro-BNP is an indicator of cardiac function and is relevant in sepsis since the latter can trigger septic cardiomyopathy. One study showed that the death rate is higher in sepsis patients with septic cardiomyopathy [35]. Moreover, inappropriate rehydration during treatment can also increase pro-BNP levels. Regardless of the cause, an aberrant increase in pro-BNP is an indicator of poor prognosis in sepsis patients [36].
The low MAP and high heart rate in sepsis patients are indicative of low perfusion shock and severe stage of the disease. Studies show that the mortality rate of septic shock is 33.5%-61% [37, 38], which significantly increases the risk of death in patients with sepsis. Many studies have explored the relationship between sepsis and acute respiratory distress syndrome (ARDS) [18], and have reported that ARDS increases the mortality rate of sepsis patients [39]. Nevertheless, little is known regarding the relationship between sepsis and oxygenation index. We found that the early decline in the oxygenation index is an independent risk factor for sepsis. In addition, ARDS, cardiac dysfunction and excessive fluid resuscitation can decrease the oxygenation index, and should be actively prevented and treated in the sepsis patients. Finally, we observed that the HCT value is predictive of the short-term prognosis of sepsis patients. Sub-normal HCT was associated with worse prognosis, suggesting that treating anemia can improve the prognosis of sepsis patients.
Various scoring systems are used to predict the prognosis of sepsis patients, such as the APACHEII and SOFA scoring systems, although they are fraught with lower predictive accuracy and reliability, along with insufficient discrimination power [8, 34, 40]. In addition, four scoring systems were evaluated by Arabi et al (including APACHEII and SOFA) in ICU patients with sepsis, all of which showed poor calibration [10]. Moreover, the clinical relevance of these scoring systems has not been evaluated previously [41]. In this study, the nomogram prediction model was superior than SOFA system regarding on the discrimination power (AUC), which could be partly explained by the difference among the enrolled variables of these two groups. Furthermore, the predictive factors included in this study are objective and simple and conveniently obtained after admission, indicating potential clinical efficacy. Predicting the death risks after admission could help clinical physicians to make effective therapeutic regimens with good prognosis.
This nomogram prediction model was comparable to those based on random forests model and stacking model regarding on the discrimination power. The stacking method integrates three models of SVM, C5.0 and Xgboost. The nomogram graph, based on logistic regression model, could visually show how much the enrolled variables contributed to the predicted risk. For random forest method and stacking method, how the enrolled variables contributed to the predicted event remained unclear, and some showed little improvement to the performance of prediction model, resulting into less practice. Only when the increased complexity leads to significant improvement in the efficiency of a model, it is reasonable to add complex algorithms to establish a model [42]. Our nomogram model owned comparable AUCs to models based on the stacking method, with less variables enrolled and more feasibility.
Nevertheless, there are some limitations in our study that ought to be considered. First, as a retrospective study, the selection bias could not be avoided as patients with missing enrolled variables were excluded. Second, it was a single-center study with limited number of cases, and a real external validation dataset was lacking. Third, this established model could not be compared to APACHE II scoring system due to unavailable data from clinical record. Moreover, other potential risk factors such as comorbidities, were not considered in this study. These limitations might reduce the prediction ability of established model in another patient population.
Conclusion
A novel nomogram based on multiple clinical indices was constructed that can predict the short-term prognosis of sepsis patients accurately and reliably, which will be helpful for clinical decision-making.
Supporting information
S1 Checklist. STROBE 2007 (v4) checklist of items to be included in reports of observational studies in epidemiology*.
https://doi.org/10.1371/journal.pone.0296456.s001
(PDF)
S1 Fig. The flowchart for patient inclusion and exclusion.
https://doi.org/10.1371/journal.pone.0296456.s002
(TIF)
S1 Table. The interaction of variables in training group.
https://doi.org/10.1371/journal.pone.0296456.s003
(PDF)
Acknowledgments
Thank you for the technical support provided by Le9 Health platform and the efforts and contributions of each author in this study.
References
- 1. Rhodes A, Evans LE, Alhazzani W, Levy MM, Antonelli M, Ferrer R, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017;43(3):304–77. Epub 2017/01/20. pmid:28101605.
- 2. Egi M, Ogura H, Yatabe T, Atagi K, Inoue S, Iba T, et al. The Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock 2020 (J-SSCG 2020). J Intensive Care. 2021;9(1):53. Epub 2021/08/27. pmid:34433491; PubMed Central PMCID: PMC8384927.
- 3. Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet. 2020;395(10219):200–11. Epub 2020/01/20. pmid:31954465; PubMed Central PMCID: PMC6970225.
- 4. Fleischmann C, Scherag A, Adhikari NK, Hartog CS, Tsaganos T, Schlattmann P, et al. Assessment of Global Incidence and Mortality of Hospital-treated Sepsis. Current Estimates and Limitations. Am J Respir Crit Care Med. 2016;193(3):259–72. Epub 2015/09/29. pmid:26414292.
- 5. Marx G, SepNet Critical Care Trials G. Correction to: Incidence of severe sepsis and septic shock in German intensive care units: the prospective, multicentre INSEP study. Intensive Care Med. 2018;44(1):153–6. Epub 2017/12/03. pmid:29196793.
- 6. Rahmel T. [SSC International Guideline 2016—Management of Sepsis and Septic Shock]. Anasthesiol Intensivmed Notfallmed Schmerzther. 2018;53(2):142–8. Epub 2018/02/10. pmid:29426052.
- 7. Yu S, Leung S, Heo M, Soto GJ, Shah RT, Gunda S, et al. Comparison of risk prediction scoring systems for ward patients: a retrospective nested case-control study. Crit Care. 2014;18(3):R132. Epub 2014/06/28. pmid:24970344; PubMed Central PMCID: PMC4227284.
- 8. Huang CT, Ruan SY, Tsai YJ, Ku SC, Yu CJ. Clinical Trajectories and Causes of Death in Septic Patients with a Low APACHE II Score. J Clin Med. 2019;8(7). Epub 2019/07/25. pmid:31330785; PubMed Central PMCID: PMC6678558.
- 9. Nassar AP Jr., Mocelin AO, Nunes AL, Giannini FP, Brauer L, Andrade FM, et al. Caution when using prognostic models: a prospective comparison of 3 recent prognostic models. J Crit Care. 2012;27(4):423 e1-7. Epub 2011/10/29. pmid:22033059.
- 10. Arabi Y, Al Shirawi N, Memish Z, Venkatesh S, Al-Shimemeri A. Assessment of six mortality prediction models in patients admitted with severe sepsis and septic shock to the intensive care unit: a prospective cohort study. Crit Care. 2003;7(5):R116–22. Epub 2003/09/17. pmid:12974979; PubMed Central PMCID: PMC270727.
- 11. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707–10. Epub 1996/07/01. pmid:8844239.
- 12. Ren Y, Zhang L, Xu F, Han D, Zheng S, Zhang F, et al. Risk factor analysis and nomogram for predicting in-hospital mortality in ICU patients with sepsis and lung infection. BMC Pulm Med. 2022;22(1):17. Epub 2022/01/08. pmid:34991569; PubMed Central PMCID: PMC8739695.
- 13. Zeng Q, He L, Zhang N, Lin Q, Zhong L, Song J. Prediction of 90-Day Mortality among Sepsis Patients Based on a Nomogram Integrating Diverse Clinical Indices. Biomed Res Int. 2021;2021:1023513. Epub 2021/11/02. pmid:34722755; PubMed Central PMCID: PMC8550845.
- 14. Liu H, Zhang L, Xu F, Li S, Wang Z, Han D, et al. Establishment of a prognostic model for patients with sepsis based on SOFA: a retrospective cohort study. J Int Med Res. 2021;49(9):3000605211044892. Epub 2021/09/30. pmid:34586931; PubMed Central PMCID: PMC8485318.
- 15. Kong G, Lin K, Hu Y. Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU. BMC Med Inform Decis Mak. 2020;20(1):251. Epub 2020/10/04. pmid:33008381; PubMed Central PMCID: PMC7531110.
- 16. Yao RQ, Jin X, Wang GW, Yu Y, Wu GS, Zhu YB, et al. A Machine Learning-Based Prediction of Hospital Mortality in Patients With Postoperative Sepsis. Front Med (Lausanne). 2020;7:445. Epub 2020/09/10. pmid:32903618; PubMed Central PMCID: PMC7438711.
- 17. Seo MH, Choa M, You JS, Lee HS, Hong JH, Park YS, et al. Hypoalbuminemia, Low Base Excess Values, and Tachypnea Predict 28-Day Mortality in Severe Sepsis and Septic Shock Patients in the Emergency Department. Yonsei Med J. 2016;57(6):1361–9. Epub 2016/09/07. pmid:27593863; PubMed Central PMCID: PMC5011267.
- 18. Wang B, Chen J. Establishment and validation of a predictive model for mortality within 30 days in patients with sepsis-induced blood pressure drop: A retrospective analysis. PLoS One. 2021;16(5):e0252009. Epub 2021/05/21. pmid:34015023; PubMed Central PMCID: PMC8136670.
- 19. Carrara M, Baselli G, Ferrario M. Mortality Prediction Model of Septic Shock Patients Based on Routinely Recorded Data. Comput Math Methods Med. 2015;2015:761435. Epub 2015/11/12. pmid:26557154; PubMed Central PMCID: PMC4628694.
- 20. Vallabhajosyula S, Jentzer JC, Kotecha AA, Murphree DH Jr., Barreto EF, Khanna AK, et al. Development and performance of a novel vasopressor-driven mortality prediction model in septic shock. Ann Intensive Care. 2018;8(1):112. Epub 2018/11/24. pmid:30467807; PubMed Central PMCID: PMC6250607.
- 21. Osawa T, Abe T, Takada N, Ito YM, Murai S, Shinohara N. Validation of the nomogram for predicting 90-day mortality after radical cystectomy in a Japanese cohort. Int J Urol. 2018;25(7):699–700. Epub 2018/04/24. pmid:29682815.
- 22. Zhang X, Wang J, Wu B, Li T, Jin L, Wu Y, et al. A Nomogram-based Model to Predict Neoplastic Risk for Patients with Gallbladder Polyps. J Clin Transl Hepatol. 2022;10(2):263–72. Epub 2022/05/10. pmid:35528981; PubMed Central PMCID: PMC9039700.
- 23. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):801–10. Epub 2016/02/24. pmid:26903338; PubMed Central PMCID: PMC4968574.
- 24. Zhang Z, Gayle AA, Wang J, Zhang H, Cardinal-Fernandez P. Comparing baseline characteristics between groups: an introduction to the CBCgrps package. Ann Transl Med. 2017;5(24):484. Epub 2018/01/05. pmid:29299446; PubMed Central PMCID: PMC5750271.
- 25. Harrell FE Jr., Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. JAMA. 1982;247(18):2543–6. Epub 1982/05/14. pmid:7069920.
- 26. Silva TB, Oliveira CZ, Faria EF, Mauad EC, Syrjanen KJ, Carvalho AL. Development and validation of a nomogram to estimate the risk of prostate cancer in Brazil. Anticancer Res. 2015;35(5):2881–6. Epub 2015/05/13. pmid:25964571.
- 27. Niu XK, He WF, Zhang Y, Das SK, Li J, Xiong Y, et al. Developing a new PI-RADS v2-based nomogram for forecasting high-grade prostate cancer. Clin Radiol. 2017;72(6):458–64. Epub 2017/01/11. pmid:28069159.
- 28. Nattino G, Finazzi S, Bertolini G. A new test and graphical tool to assess the goodness of fit of logistic regression models. Stat Med. 2016;35(5):709–20. Epub 2015/10/07. pmid:26439593.
- 29. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–74. Epub 2006/11/14. pmid:17099194; PubMed Central PMCID: PMC2577036.
- 30. Lelubre C, Vincent JL. Mechanisms and treatment of organ failure in sepsis. Nat Rev Nephrol. 2018;14(7):417–27. Epub 2018/04/25. pmid:29691495.
- 31. Rubio I, Osuchowski MF, Shankar-Hari M, Skirecki T, Winkler MS, Lachmann G, et al. Current gaps in sepsis immunology: new opportunities for translational research. Lancet Infect Dis. 2019;19(12):e422–e36. Epub 2019/10/22. pmid:31630991.
- 32. Liu W, Peng L, Hua S. Clinical significance of dynamic monitoring of blood lactic acid, oxygenation index and C-reactive protein levels in patients with severe pneumonia. Exp Ther Med. 2015;10(5):1824–8. Epub 2015/12/08. pmid:26640556; PubMed Central PMCID: PMC4665687.
- 33. Houwink AP, Rijkenberg S, Bosman RJ, van der Voort PH. The association between lactate, mean arterial pressure, central venous oxygen saturation and peripheral temperature and mortality in severe sepsis: a retrospective cohort analysis. Crit Care. 2016;20:56. Epub 2016/03/13. pmid:26968689; PubMed Central PMCID: PMC4788911.
- 34. Liu Z, Meng Z, Li Y, Zhao J, Wu S, Gou S, et al. Prognostic accuracy of the serum lactate level, the SOFA score and the qSOFA score for mortality among adults with Sepsis. Scand J Trauma Resusc Emerg Med. 2019;27(1):51. Epub 2019/05/02. pmid:31039813; PubMed Central PMCID: PMC6492372.
- 35. Takasu O, Gaut JP, Watanabe E, To K, Fagley RE, Sato B, et al. Mechanisms of cardiac and renal dysfunction in patients dying of sepsis. Am J Respir Crit Care Med. 2013;187(5):509–17. Epub 2013/01/26. pmid:23348975; PubMed Central PMCID: PMC3733408.
- 36. Cheng H, Fan WZ, Wang SC, Liu ZH, Zang HL, Wang LZ, et al. N-terminal pro-brain natriuretic peptide and cardiac troponin I for the prognostic utility in elderly patients with severe sepsis or septic shock in intensive care unit: A retrospective study. J Crit Care. 2015;30(3):654 e9–14. Epub 2015/01/13. pmid:25575850.
- 37. Zhou J, Qian C, Zhao M, Yu X, Kang Y, Ma X, et al. Epidemiology and outcome of severe sepsis and septic shock in intensive care units in mainland China. PLoS One. 2014;9(9):e107181. Epub 2014/09/17. pmid:25226033; PubMed Central PMCID: PMC4167333.
- 38. Huang CT, Tsai YJ, Tsai PR, Yu CJ, Ko WJ. Epidemiology and Outcome of Severe Sepsis and Septic Shock in Surgical Intensive Care Units in Northern Taiwan. Medicine (Baltimore). 2015;94(47):e2136. Epub 2015/12/04. pmid:26632737; PubMed Central PMCID: PMC5059006.
- 39. Force ADT, Ranieri VM, Rubenfeld GD, Thompson BT, Ferguson ND, Caldwell E, et al. Acute respiratory distress syndrome: the Berlin Definition. JAMA. 2012;307(23):2526–33. Epub 2012/07/17. pmid:22797452.
- 40. Wang S, Li T, Li Y, Zhang J, Dai X. [Predictive value of four different scoring systems for septic patient’s outcome: a retrospective analysis with 311 patients]. Zhonghua Wei Zhong Bing Ji Jiu Yi Xue. 2017;29(2):133–8. Epub 2017/06/20. pmid:28625260.
- 41. Rousson V, Zumbrunn T. Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies. BMC Med Inform Decis Mak. 2011;11:45. Epub 2011/06/24. pmid:21696604; PubMed Central PMCID: PMC3148204.
- 42. Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17(1):195. Epub 2019/10/31. pmid:31665002; PubMed Central PMCID: PMC6821018.