Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

SOFA score performs worse than age for predicting mortality in patients with COVID-19

  • Raphael A. G. Sherak ,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing

    Raphael.Sherak@Yale.edu

    Affiliation Yale Department of Emergency Medicine, Yale School of Medicine, New Haven, CT, United States of America

  • Hoomaan Sajjadi,

    Roles Data curation, Formal analysis, Investigation, Writing – review & editing

    Affiliation Department of Computer Science and Engineering, Center for Remote Health Technologies and Systems, Texas A&M Univ, College Station, TX, United States of America

  • Naveed Khimani,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Writing – review & editing

    Affiliation Department of Computer Science and Engineering, Center for Remote Health Technologies and Systems, Texas A&M Univ, College Station, TX, United States of America

  • Benjamin Tolchin,

    Roles Conceptualization, Resources, Supervision, Writing – review & editing

    Affiliations Department of Neurology, Yale School of Medicine, New Haven, CT, United States of America, Yale New Haven Health Center for Clinical Ethics, New Haven, CT, United States of America

  • Karen Jubanyik,

    Roles Conceptualization, Methodology, Resources, Supervision, Writing – review & editing

    Affiliation Yale Department of Emergency Medicine, Yale School of Medicine, New Haven, CT, United States of America

  • R. Andrew Taylor,

    Roles Methodology, Project administration, Resources, Supervision, Writing – review & editing

    Affiliation Yale Department of Emergency Medicine, Yale School of Medicine, New Haven, CT, United States of America

  • Wade Schulz,

    Roles Methodology, Resources, Supervision, Writing – review & editing

    Affiliations Department of Laboratory Medicine, Yale School of Medicine, New Haven, CT, United States of America, Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, CT, United States of America

  • Bobak J. Mortazavi,

    Roles Conceptualization, Formal analysis, Methodology, Resources, Supervision, Writing – review & editing

    Affiliations Department of Computer Science and Engineering, Center for Remote Health Technologies and Systems, Texas A&M Univ, College Station, TX, United States of America, Center for Outcomes Research and Evaluation, Yale University, New Haven, CT, United States of America

  • Adrian D. Haimovich

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

    Affiliations Yale Department of Emergency Medicine, Yale School of Medicine, New Haven, CT, United States of America, Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, MA, United States of America

Abstract

The use of the Sequential Organ Failure Assessment (SOFA) score, originally developed to describe disease morbidity, is commonly used to predict in-hospital mortality. During the COVID-19 pandemic, many protocols for crisis standards of care used the SOFA score to select patients to be deprioritized due to a low likelihood of survival. A prior study found that age outperformed the SOFA score for mortality prediction in patients with COVID-19, but was limited to a small cohort of intensive care unit (ICU) patients and did not address whether their findings were unique to patients with COVID-19. Moreover, it is not known how well these measures perform across races. In this retrospective study, we compare the performance of age and SOFA score in predicting in-hospital mortality across two cohorts: a cohort of 2,648 consecutive adult patients diagnosed with COVID-19 who were admitted to a large academic health system in the northeastern United States over a 4-month period in 2020 and a cohort of 75,601 patients admitted to one of 335 ICUs in the eICU database between 2014 and 2015. We used age and the maximum SOFA score as predictor variables in separate univariate logistic regression models for in-hospital mortality and calculated area under the receiver operator characteristic curves (AU-ROCs) and area under precision-recall curves (AU-PRCs) for each predictor in both cohorts. Among the COVID-19 cohort, age (AU-ROC 0.795, 95% CI 0.762, 0.828) had a significantly better discrimination than SOFA score (AU-ROC 0.679, 95% CI 0.638, 0.721) for mortality prediction. Conversely, age (AU-ROC 0.628 95% CI 0.608, 0.628) underperformed compared to SOFA score (AU-ROC 0.735, 95% CI 0.726, 0.745) in non-COVID-19 ICU patients in the eICU database. There was no difference between Black and White COVID-19 patients in performance of either age or SOFA Score. Our findings bring into question the utility of SOFA score-based resource allocation in COVID-19 crisis standards of care.

Introduction

The Sequential Organ Failure Assessment (SOFA) score was developed in 1994 by a European Society of Intensive Care Medicine working group to objectively quantify the degree of organ dysfunction and failure in intensive care unit (ICU) patients with sepsis [1]. The SOFA score assigns a value of 0 to 4 to the dysfunction of 6 organ systems (respiratory, coagulation, hepatic, cardiovascular, neurologic, and renal), with higher numbers indicating more dysfunction. The score is calculated using the worst clinical values observed in the previous 24 hours. It may be used to describe each organ system or as a summative measure.

Although the initial intention of the SOFA score was to describe morbidity in ICU patients with sepsis, subsequent studies have used the SOFA score or SOFA score-based models to predict mortality [24]. While the SOFA score on admission performed well for mortality prediction in sepsis in some studies [5,6], other studies have suggested only intermediate discriminatory accuracy [7,8]. The use of SOFA score has also expanded beyond patients with sepsis [9,10] and is used outside of the ICU [6,11].

With COVID-19 positive patients occupying up to 90% of ICU beds during the global SARS-CoV-2 pandemic [12], the SOFA score gained new applications. One study reported that 20 of the 26 COVID-19 ventilator triage policies surveyed used the SOFA score [13]. Many of these guidelines for crisis standards of care involve withholding resources from patients with an expected low likelihood of survival, with the rationale of saving the most lives possible with the limited resources available. One way this is done is by assuming patients with a SOFA score at or above a predetermined threshold are unlikely to survive the hospitalization. However, there is a dearth of data about the use of the SOFA score to predict mortality in COVID-19, potentially leading to improper allocation of resources [14,15].

One prominent study of 675 COVID-19 positive patients from a single healthcare system found that SOFA score had inferior discriminant prognostic accuracy for in-hospital mortality compared to patient age [16]. However, the study was limited by its small sample size with missing data for approximately 25% of its cohort and restriction to patients requiring mechanical ventilation. Moreover, the study population had higher and less variable SOFA scores than the average ICU population [6,7].

Here, we expand on prior work of SOFA score mortality prediction in COVID-19 by assessing SOFA score performance for an undifferentiated population of admitted patients. The primary aim of this retrospective study was to assess how well SOFA score predicts in-hospital mortality in a consecutive cohort of patients with COVID-19 admitted to a quaternary medical center in the Northeast United States (U.S.). We compare the discriminative performance of SOFA to age alone. To examine whether any findings are specific to patients with COVID-19, we contrast these findings to a large cohort of general ICU patients from the publicly accessible eICU database. As a secondary objective, given concerns that prediction models can perpetuate systemic inequities [1722], we compared the prognostic value of SOFA across race in both cohorts.

Methods

This was a retrospective study comprising two separate patient cohorts.

Data acquisition and preprocessing

COVID-19 cohort.

Patients in the COVID-19 cohort consisted of all patients with an age ≥ 18 admitted to any of the 5 hospitals in the Yale-New Haven Health System (YNHH) from March 29th, 2020 to August 1, 2020 with a diagnosis of COVID-19, defined as either a positive PCR test for COVID-19 or designated as a COVID-19 patient by an attending physician. Data was obtained retrospectively from the YNHH electronic medical record (EMR, Epic Systems Corporation, Verona, WI). SOFA score was automatically calculated for all admitted patients and recorded every 4 hours by the YNHH EMR. Patients were excluded if they did not have a SOFA score recorded in the first 24 hours of admission or if they had a prior admission with a COVID-19 diagnosis. All records were de-identified prior to analysis. The patient cohort is described extensively in a prior, publicly available manuscript [23].

eICU cohort.

The eICU Collaborative Research Database is a publicly available database containing data from over 139,000 patients hospitalized between 2014 and 2015 at one of the 335 American ICUs participating Philips Healthcare eICU program [24].

Patients with an ICU stay of at least 24 hours were included. Also, for the patients having more than one hospitalization, only the most recent hospitalization was used. The SOFA scores on eICU were calculated and extracted using Python’s Pandas library, based on its standard definition on the values of creatinine, bilirubin, platelets, fraction of inspired oxygen, partial pressure of inspired oxygen, Glasgow Coma Scale, mean arterial pressure, and mechanical ventilation status. As part of the eICU patient de-identification process, all patients with age >89 are grouped together and were assigned an age of 90 for the analysis. Then, the corresponding tables and columns were pre-processed to calculate the final scores. The overall steps taken in this procedure are described in the supplement.

In both cohorts, gender was dichotomized and race was classified as a single category based on predefined fields in the EMR. Depending on a patient’s clinical status, gender and race were either self-selected by patients or assigned by hospital registration staff.

Ethics statement.

This study was reviewed and approved by the Yale University Institutional Review Board (Study Number 2000027747). The study was deemed exempt from the requirement for consent because all data were analyzed anonymously.

Logistic regression model development

Age in years at time of admission and the maximum SOFA score recorded in the first 24 hours after admission were each used as predictor variables in separate univariate logistic regression models for the binary outcome of in-hospital mortality. Each model was fit to a sample of 60% of the respective cohorts.

Model assessment

The remaining 40% of each cohort were used as a validation cohort to calculate area under the receiver operator characteristic curves (AU-ROCs). AU-ROC intervals were estimated using DeLong’s method [25]. This method was then repeated to calculate area under precision-recall curves (AU-PRCs). Calibration curves were produced using R’s predtools package [26] (S1 Fig). In addition, we conducted a decision curve analysis [27] using the dcurves R package [28]. All analyses were performed in Python (Version 3.7.7) and R (Version 1.4.1717).

We performed several secondary analyses. We stratified each cohort by race including only patients identifying as Black/African American or White/Caucasian as we had insufficient sample sizes for further groups. Because the eICU cohort is restricted to patients admitted to the ICU, we performed an additional analysis of the COVID-19 cohort restricted to ICU patients. We also conducted an exploratory analysis of the COVID-19 cohort of survival rates at and above a given SOFA score. We then stratified by race to explore whether there was a difference in survival rates at various SOFA score thresholds. The binomial exact test was used to calculate confidence intervals.

Results

Patient characteristics

A total of 2,648 patients were included in the COVID-19 cohort with a median age of 65 (IQR 50–79), mean SOFA score of 2.1 (SD 2.7), median SOFA score of 1 (IQR 0–3) and an in-hospital mortality rate of 17.1%. The eICU cohort consisted of 75,601 patients with a median age of 65 (IQR 52–77), mean SOFA score of 7.7 (SD 2.7), median SOFA score of 8 (IQR 6–9) and an in-hospital mortality rate of 10.8%. The COVID-19 and eICU cohorts respectively were 52.4% and 46.1% female and 25.4% and 10.7% Black/African American (Fig 1 and Table 1).

thumbnail
Fig 1. Distribution of SOFA scores.

Histogram of SOFA scores in the COVID-19 (orange) and eICU (blue) cohorts.

https://doi.org/10.1371/journal.pone.0301013.g001

Primary outcome

Among the COVID-19 cohort, age (AU-ROC 0.795, 95% CI 0.762, 0.828) had a significantly better discrimination than SOFA score (AU-ROC 0.679, 95% CI 0.638, 0.721) for mortality prediction (Fig 2). When restricted to patients admitted to the ICU, age (AU-ROC 0.698, 95% CI 0.631, 0.765) still generally performed better than SOFA score (AU-ROC 0.562, 95% CI 0.486, 0.639), for mortality prediction in COVID-19 patients (S2 Fig). Conversely, age (AU-ROC 0.628 95% CI 0.608, 0.628) significantly underperformed compared to SOFA score (AU-ROC 0.735, 95% CI 0.726, 0.745) in non-COVID-19 eICU patients (Fig 2 and Table 2).These findings were consistent with the decision curve analysis where the age-based model had a higher net benefit across most threshold probabilities in the COVID-19 cohort, while the SOFA score-based model had a higher net benefit than the age-based model across all threshold probabilities greater than 5% in the eICU cohort (Fig 3).

thumbnail
Fig 2. AU-ROCs and AU-PRCs of age and SOFA score for mortality prediction.

Age-based model is in orange and SOFA score-based model is in blue.

https://doi.org/10.1371/journal.pone.0301013.g002

thumbnail
Fig 3. Decision curve analysis plots of age-based and SOFA score-based models in the COVID-19 and eICU cohorts.

Decision curve analysis plots for age-based (blue) and SOFA score-based (purple) models in both cohorts. The “null” options of “treat all” and “treat none” are represented by red and green lines respectively.

https://doi.org/10.1371/journal.pone.0301013.g003

Secondary analyses

When stratified by race, age performed similarly between Black and White patients in both cohorts (Fig 4 and Table 3). There also was no significant difference in the performance of the SOFA score between Black and White patients with COVID-19. In the eICU cohort, SOFA score was better than age at discriminating between survivors and non-survivors both in Black and White patients. Additionally, the SOFA score performed better in Black patients compared to White patients in the eICU cohort.

thumbnail
Fig 4. Predictor performance stratified by cohort and race.

Plot of AU-ROCs with 95% CIs for age-based (red) and SOFA score-based (blue) models in each cohort stratified by race.

https://doi.org/10.1371/journal.pone.0301013.g004

thumbnail
Table 3. AU-ROC of age and SOFA score for mortality prediction stratified by cohort and race.

https://doi.org/10.1371/journal.pone.0301013.t003

In the COVID-19 cohort, there was no significant difference in survival rates between Black and White patients at and above SOFA scores of 3 (S3 Fig). The survival rate of COVID-19 patients only dropped below 50% at SOFA scores greater than or equal to 12 (S4 Fig).

Discussion

Age significantly outperformed SOFA score for predicting mortality in hospitalized COVID-19 patients, including those in the ICU. This phenomenon may be unique to COVID-19 as SOFA score was significantly better at predicting mortality in the eICU cohort. The finding that a simple metric such as age outperformed SOFA score for mortality prediction in COVID-19 patients suggests that caution should be taken when applying established prediction models to a completely novel disease process. This is especially prudent when using mortality prediction models to guide treatment decisions and resource allocation. Many guidelines for crisis standards of care suggested a SOFA score threshold of ≥ 6 to identify patients with low likelihood of survival [29,30]. However, there was only a 33% mortality rate using that threshold in our COVID-19 cohort.

Although the SOFA score was originally created for ICU patients [1], the inclusion of non-ICU patients in the COVID-19 cohort does not explain our findings. When restricted to COVID-19 cohort to ICU patients, a similar trend was noted. Age performed better than SOFA score for mortality prediction, with minimally overlapping confidence intervals. However, these results were limited by the relatively small sample size (642) of COVID-19 ICU patients. While applying SOFA score to non-ICU patients is outside of the original intent of SOFA score, so is using it for mortality prediction. In the original study describing the SOFA score, Vincent et al. state, “it is important to realize that the SOFA score is designed not to predict outcome but to describe a sequence of complications”[1].

Prior studies have theorized that SOFA score underperforms for mortality prediction in COVID-19 patients because the illness affects fewer organ systems resulting in lower variability in scores [16]. Contradictory to that theory, patients in the COVID-19 cohort had the same standard deviation in scores (2.7) as the eICU cohort. Since age >65 is one of the characteristics with the strongest association with increased mortality in COVID-19 positive patients [31,32], it may partially explain why age outperforms SOFA score only in COVID-19 patients.

There was no difference between White and Black COVID-19 patients in the performance of either age or SOFA score for mortality prediction. Additionally, there was no significant difference in mortality rate between White and Black COVID-19 patients at SOFA scores greater than 2. Notably, a prior study of patients from the same COVID-19 cohort, Black patients had 1.5 times the odds of a SOFA score ≥ 6 than white patients, even when adjusting for age, sex, insurance status, BMI, liver and renal diseases [23]. This suggests that Black patients with COVID-19 may be more likely than White patients to be assigned higher SOFA scores but will have similar mortality rates at those higher SOFA scores.

This study builds on a recent analysis from a single health system of 675 COVID-19 positive patients requiring mechanical ventilation in which SOFA score had inferior prognostic accuracy for in-hospital mortality compared to simply using patient age [16]. However, the study was limited by its cohort having higher and less variable SOFA scores than the average ICU population as well as missing data for approximately 25% of the cohort. Our study had over 2,600 COVID-19 positive patients with an average SOFA score similar to prior analyses on non-COVID-19 patients [6,7] and a mortality rate comparable to other cohorts of COVID-19 patients that were admitted to U.S. hospitals during a similar time period [31]. Moreover, the SOFA scores used in the COVID-19 cohort were automatically calculated by the electronic health record, a pragmatic approach that would be used in triage scenarios.

Our study has several limitations. The cohort of COVID-19 patients was restricted to a single health system in the northeast U.S. which may not have a comparable population to COVID-19 patients at other academic health systems or those in the eICU database. Due to sample size, we also restricted our analysis to two racial groups and did not consider patients that identified as multi-racial. We did not consider sex or its potential interaction with race or age in our study. Furthermore, an unknown proportion of patients in the COVID-19 cohort were too critically ill to answer demographic questions and had their race and sex recorded by a hospital clerk based on assumption. Additionally, this study only describes the performance of max SOFA score within 24 hours of hospital admission. Many COVID-19 positive patients present to the hospital with respiratory complaints and develop multisystem organ dysfunction later in their disease course [32]. However, information on SOFA subscores in the cohort of COVID-19 patients were not available so we were unable to test this hypothesis. SOFA score may have greater utility for predicting mortality with serial measurements or later in disease course [2].

Although COVID-19’s status as an official U.S. public health emergency ended May 11, 2023 [33], hospital system capacity continues to be strained by both the direct and in-direct effects of COVID-19 [34,35]. Nationwide, there remains key gaps in planning for crisis standards of care, and SOFA score is often proposed as a guide for resource allocation [36]. This study suggests caution should be used when considering SOFA score as triage tool, as it has limited prognostic performance in COVID-19 and may suffer similar limitations in future pandemics, especially when novel disease processes are involved.

Supporting information

S1 Fig. Calibration curves for SOFA score and age based models in the COVID-19 and eICU cohorts.

Calibration curves for age-based (red) and SOFA score-based (blue) models in both cohorts. The black line with a slope of 1 represents a perfectly calibrated model.

https://doi.org/10.1371/journal.pone.0301013.s001

(TIF)

S2 Fig. AU-ROCs of age and SOFA score for mortality prediction in COVID-19 cohort patients admitted to the ICU.

AU-ROC of age-based (orange) and SOFA score-based (blue) models calculated on subset of COVID-19 cohort that required ICU admission.

https://doi.org/10.1371/journal.pone.0301013.s002

(TIF)

S3 Fig. Mortality in COVID-19 patients above sofa score threshold by race with 95% CIs.

Mortality percentage at and above a given SOFA score in Black (red line) and White (blue line) patients in the COVID-19 cohort with 95% confidence intervals in shaded region.

https://doi.org/10.1371/journal.pone.0301013.s003

(TIF)

S4 Fig. Mortality in COVID-19 patients above SOFA score threshold.

Mortality percentage at and above a given SOFA score for all patients in the COVID-19 cohort.

https://doi.org/10.1371/journal.pone.0301013.s004

(TIF)

References

  1. 1. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonça A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22: 707–710. pmid:8844239
  2. 2. Ferreira FL, Bota DP, Bross A, Mélot C, Vincent JL. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA. 2001;286: 1754–1758. pmid:11594901
  3. 3. Vincent J-L, de Mendonca A, Cantraine F, Moreno R, Takala J, Suter PM, et al. Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: Results of a multicenter, prospective study. Crit Care Med. 1998;26: 1793. Available: https://journals.lww.com/ccmjournal/Fulltext/1998/11000/Use_of_the_SOFA_score_to_assess_the_incidence_of.16.aspx.
  4. 4. Cárdenas-Turanzas M, Ensor J, Wakefield C, Zhang K, Wallace SK, Price KJ, et al. Cross-validation of a Sequential Organ Failure Assessment score-based model to predict mortality in patients with cancer admitted to the intensive care unit. J Crit Care. 2012;27: 673–680. pmid:22762932
  5. 5. Jones AE, Trzeciak S, Kline JA. The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Crit Care Med. 2009;37: 1649–1654. pmid:19325482
  6. 6. Kovach CP, Fletcher GS, Rudd KE, Grant RM, Carlbom DJ. Comparative prognostic accuracy of sepsis scores for hospital mortality in adults with suspected infection in non-ICU and ICU at an academic public hospital. PLoS One. 2019;14: e0222563. pmid:31525224
  7. 7. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R, et al. Prognostic Accuracy of the SOFA Score, SIRS Criteria, and qSOFA Score for In-Hospital Mortality Among Adults With Suspected Infection Admitted to the Intensive Care Unit. JAMA. 2017;317: 290–300. pmid:28114553
  8. 8. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315: 762–774. pmid:26903335
  9. 9. Minne L, Abu-Hanna A, de Jonge E. Evaluation of SOFA-based models for predicting mortality in the ICU: A systematic review. Crit Care. 2008;12: R161. pmid:19091120
  10. 10. Kashyap R, Sherani KM, Dutt T, Gnanapandithan K, Sagar M, Vallabhajosyula S, et al. Current Utility of Sequential Organ Failure Assessment Score: A Literature Review and Future Directions. Open Respir Med J. 2021;15: 1–6. pmid:34249175
  11. 11. Yu S, Leung S, Heo M, Soto GJ, Shah RT, Gunda S, et al. Comparison of risk prediction scoring systems for ward patients: a retrospective nested case-control study. Crit Care. 2014;18: R132. pmid:24970344
  12. 12. Douin DJ, Ward MJ, Lindsell CJ, Howell MP, Hough CL, Exline MC, et al. ICU Bed Utilization During the Coronavirus Disease 2019 Pandemic in a Multistate Analysis-March to June 2020. Crit Care Explor. 2021;3: e0361. pmid:33786437
  13. 13. Antommaria AHM, Gibb TS, McGuire AL, Wolpe PR, Wynia MK, Applewhite MK, et al. Ventilator Triage Policies During the COVID-19 Pandemic at U.S. Hospitals Associated With Members of the Association of Bioethics Program Directors. Ann Intern Med. 2020;173: 188–194. pmid:32330224
  14. 14. Shahpori R, Stelfox HT, Doig CJ, Boiteau PJE, Zygun DA. Sequential Organ Failure Assessment in H1N1 pandemic planning. Crit Care Med. 2011;39: 827–832. pmid:21263327
  15. 15. Khan Z, Hulme J, Sherwood N. An assessment of the validity of SOFA score based triage in H1N1 critically ill patients during an influenza pandemic. Anaesthesia. 2009;64: 1283–1288. pmid:19860754
  16. 16. Raschke RA, Agarwal S, Rangan P, Heise CW, Curry SC. Discriminant Accuracy of the SOFA Score for Determining the Probable Mortality of Patients With COVID-19 Pneumonia Requiring Mechanical Ventilation. JAMA. 2021. pmid:33595630
  17. 17. Tolchin B, Hull SC, Kraschel K. Triage and justice in an unjust pandemic: ethical allocation of scarce medical resources in the setting of racial and socioeconomic disparities. J Med Ethics. 2020. pmid:33067315
  18. 18. Egede LE, Walker RJ. Structural Racism, Social Risk Factors, and Covid-19—A Dangerous Convergence for Black Americans. N Engl J Med. 2020;383: e77. pmid:32706952
  19. 19. Miller WD, Han X, Peek ME, Charan Ashana D, Parker WF. Accuracy of the Sequential Organ Failure Assessment Score for In-Hospital Mortality by Race and Relevance to Crisis Standards of Care. JAMA Netw Open. 2021;4: e2113891. pmid:34143190
  20. 20. Ashana DC, Anesi GL, Liu VX, Escobar GJ, Chesley C, Eneanya ND, et al. Equitably Allocating Resources During Crises: Racial Differences in Mortality Prediction Models. Am J Respir Crit Care Med. 2021. pmid:33751910
  21. 21. Gershengorn HB, Holt GE, Rezk A, Delgado S, Shah N, Arora A, et al. Assessment of Disparities Associated With a Crisis Standards of Care Resource Allocation Algorithm for Patients in 2 US Hospitals During the COVID-19 Pandemic. JAMA Netw Open. 2021;4: e214149. pmid:33739434
  22. 22. Sarkar R, Martin C, Mattie H, Gichoya JW, Stone DJ, Celi LA. Performance of intensive care unit severity scoring systems across different ethnicities in the USA: a retrospective observational study. Lancet Digit Health. 2021;3: e241–e249. pmid:33766288
  23. 23. Tolchin B, Oladele C, Galusha D, Kashyap N, Showstark M, Bonito J, et al. Racial disparities in the SOFA score among patients hospitalized with COVID-19. PLoS One. 2021;16: e0257608. pmid:34535009
  24. 24. Pollard TJ, Johnson AEW, Raffa JD, Celi LA, Mark RG, Badawi O. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci Data. 2018;5: 180178. pmid:30204154
  25. 25. Sun X, Xu W. Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves. IEEE Signal Process Lett. 2014;21: 1389–1393.
  26. 26. Sadatsafavi M, Safari A, Lee TY. predtools: Prediction Model Tools. 2023. Available: https://CRAN.R-project.org/package=predtools.
  27. 27. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26: 565–574. pmid:17099194
  28. 28. Sjoberg DD. dcurves: Decision Curve Analysis for Model Evaluation. 2022. Available: https://github.com/ddsjoberg/dcurves, https://www.danieldsjoberg.com/dcurves/.
  29. 29. White DB, Lo B. A Framework for Rationing Ventilators and Critical Care Beds During the COVID-19 Pandemic. JAMA. 2020;323: 1773–1774. pmid:32219367
  30. 30. Tolchin B, Latham SR, Bruce L, Ferrante LE, Kraschel K, Jubanyik K, et al. Developing a Triage Protocol for the COVID-19 Pandemic: Allocating Scarce Medical Resources in a Public Health Emergency. J Clin Ethics. 2020;31: 303–317. Available: https://www.ncbi.nlm.nih.gov/pubmed/32991327. pmid:32991327
  31. 31. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O’Donnell L, Chernyak Y, et al. Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study. BMJ. 2020;369: m1966. pmid:32444366
  32. 32. Wiersinga WJ, Rhodes A, Cheng AC, Peacock SJ, Prescott HC. Pathophysiology, Transmission, Diagnosis, and Treatment of Coronavirus Disease 2019 (COVID-19): A Review. JAMA. 2020;324: 782–793. pmid:32648899
  33. 33. RENEWAL OF DETERMINATION THAT A PUBLIC HEALTH EMERGENCY EXISTS. [cited 7 Jan 2024]. Available: https://aspr.hhs.gov/legal/PHE/Pages/COVID19-9Feb2023.aspx.
  34. 34. Covid and flu rising ahead of holidays, increasing ER visits. The Washington Post. 12 Dec 2023. Available: https://www.washingtonpost.com/health/2023/12/12/covid-rising-flu-rsv-variant/. Accessed 7 Jan 2024.
  35. 35. CDC. COVID Data Tracker. [cited 7 Jan 2024]. Available: https://covid.cdc.gov/covid-data-tracker/#datatracker-home.
  36. 36. Hick JL, Hanfling D, Wynia M. Hospital Planning for Contingency and Crisis Conditions: Crisis Standards of Care Lessons from COVID-19. Jt Comm J Qual Patient Saf. 2022;48: 354–361. pmid:35595654