Prediction model and risk scores of ICU admission and mortality in COVID-19

This study aimed to develop risk scores based on clinical characteristics at presentation to predict intensive care unit (ICU) admission and mortality in COVID-19 patients. 641 hospitalized patients with laboratory-confirmed COVID-19 were selected from 4997 persons under investigation. We performed a retrospective review of medical records of demographics, comorbidities and laboratory tests at the initial presentation. Primary outcomes were ICU admission and death. Logistic regression was used to identify independent clinical variables predicting the two outcomes. The model was validated by splitting the data into 70% for training and 30% for testing. Performance accuracy was evaluated using area under the curve (AUC) of the receiver operating characteristic analysis (ROC). Five significant variables predicting ICU admission were lactate dehydrogenase, procalcitonin, pulse oxygen saturation, smoking history, and lymphocyte count. Seven significant variables predicting mortality were heart failure, procalcitonin, lactate dehydrogenase, chronic obstructive pulmonary disease, pulse oxygen saturation, heart rate, and age. The mortality group uniquely contained cardiopulmonary variables. The risk score model yielded good accuracy with an AUC of 0.74 ([95% CI, 0.63–0.85], p = 0.001) for predicting ICU admission and 0.83 ([95% CI, 0.73–0.92], p<0.001) for predicting mortality for the testing dataset. This study identified key independent clinical variables that predicted ICU admission and mortality associated with COVID-19. This risk score system may prove useful for frontline physicians in clinical decision-making under time-sensitive and resource-constrained environment.


Introduction
The coronavirus disease 2019 (COVID- 19) is an infectious disease that can cause severe respiratory illness [1,2]. First reported in Wuhan, China in December 2019 [3], COVID-19 was declared a pandemic on March 11, 2020 [4]. Currently, more than 3 million people have been infected and more than 300,000 have died of COVID-19 [ to be even higher due to testing shortages [6]. Numbers of infection and mortality are predicted to continue to rise in the near future and there are likely to be second waves and future recurrence [7]. There is an urgent need to help frontline clinicians to effectively triage patients in the COVID-19 pandemic. Many patients deteriorate rapidly after a period of relatively mild symptoms, emphasizing the need for early risk stratification [1,2]. Current literature has already identified several clinical features associated with the severity of COVID-19 infection, but a simple scoring system specific to COVID-19 is lacking [1,2,8]. Unfortunately, established early risk scores, such as Sequential Organ Failure Assessment (SOFA) and Modified Early Warning Score (MEWS), have mixed accuracy in predicting COVID-19 severity [9][10][11]. Here we report a predictive model and a risk score system to predict intensive care unit (ICU) admission and in-hospital mortality in laboratory-confirmed COVID-19 patients. This model was developed and internally validated using data from the COVID-19 persons under investigation (PUI) registry of 4997 patients from a major academic hospital in New York.

Methods
Stony Brook University Hospital, located about 40 miles east of New York City on Long Island, is the only academic hospital in Suffolk County with a population of approximately 1.5 million. The incidence of COVID-19 infections in Suffolk County has been among the highest in the country [12]. As the pandemic evolved, our hospital reconfigured to increase ICU beds and inpatient capacity. The Army Corps of Engineers had also built five mobile field hospitals on our campus, totaling 1000 additional beds.

Study population and data collection
This was a retrospective study from Stony Brook University Hospital (March 9, 2020 to April 20, 2020). The study was approved by the Human Subjects Committee with an exemption for informed consent and HIPAA waiver. The COVID-19 PUI registry consisted of 4997 patients. Only hospitalized patients who were diagnosed by positive tests of real-time polymerase chain reaction (RT-PCR) for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) were included in this study. Of those, we excluded patients who were still hospitalized because their outcomes were unknown at the time, thus risking grouping them incorrectly. Patients who were younger than 18 years of age, and those with incomplete past medical history were also excluded. One patient who expired very quickly after admission and did not receive any laboratory testing was also excluded. Clinical data at hospital admission, including demographic information, chronic comorbidities, vital signs, symptoms, laboratory tests, and outcomes were collected from the Electronic Medical Record and REDCAP database of our COVID-19 PUI registry [13].

Outcomes
The primary outcomes were admission to the ICU and death. Among the COVID-19 patients admitted to the hospital, those who were admitted to the ICU met any one of the following criteria: (i) patients developing respiratory failure requiring mechanical ventilation; (ii) patients who have another organ failure requiring ICU monitoring. Our criteria of ICU admission meet the definition of critical illness as described in previous literature of COVID-19, thus we believe it is an appropriate primary outcome for this study [1,14]. Our mortality group included patients who died during their hospital stay. Both outcomes were compared to a group of patients requiring only general admission, which included patients who were hospitalized and discharged without receiving ICU care at any point of their hospitalization.

Statistical analysis and modeling
We followed the TRIPOD guideline for developing a multivariable regression model [15]. Categorical variables were described as frequencies and percentages and were compared using χ 2 tests or Fisher exact tests. Continuous variables were presented as medians and interquartile ranges (IQR) and compared using Mann-Whitney U tests. Total data were split into 70% training set and 30% testing set. Data collected from March 9 to April 14, 2020 were used to build the prediction model (n = 454, 70%). Data collected from April 14 to 20, 2020 were used for independent internal validation (n = 187, 30%). We recognized that this choice (instead of random sampling) would likely yield the worst possible performance to challenge our model. Logistic regression models were fit for ICU admission and mortality against general admission as dependent variables, and all available demographic and laboratory variables were included as independent variables. Laboratory variables were transformed to dichotomous variables based on optimal cut off values using the Youden index determined by a nonparametric Kernel regression method for maximizing the summation of sensitivity and specificity [16]. Backward selection was performed and only significant predictors (p<0.05) were kept in the final models. Predictors in the regression model were checked for collinearity and independence. A risk score was developed by assigning each significant variable remaining in the final model to a value of one based on the cut off value. We also developed a weighted risk score model, in which each variable was assigned a weight based on their odds ratio. Prediction performance was evaluated by the area under the curve (AUC) of the receiver operating characteristic (ROC) curve. All analyses were performed in SPSS v26 and SAS v9.4.

Characteristics of the ICU admission group
As summarized in Table 1, the median age of patients who were admitted to the ICU was 60 years (IQR, 50.0-70.0) and that of those who were not admitted to the ICU (general admission) was 58 years (IQR, 46-71) (p = 0.13). Male gender was significantly associated with higher risk of ICU admission (69.7% vs. 55.8%, p = 0.001). None of the comorbidities were significantly associated with the ICU group (p>0.05). A higher proportion of ICU patients presented with shortness of breath (78.5% vs. 64.8%, p = 0.001), while diarrhea was more common in patients without critical illness (25.6% vs. 17.9%, p = 0.04). Other symptoms on the initial presentation including nausea or vomiting, fever, cough, fatigue, sputum, myalgia, sore throat, rhinorrhea, loss of smell, loss of taste, headache, and chest discomfort were not significantly different between those in the ICU and general hospitalization (p>0.05).
The majority of patients in both the ICU (91.7%) and general admission (81.1%) groups had positive chest x-ray (CXR) findings, but more patients in the ICU group had positive CXR findings (p = 0.001). Similarly, both the ICU group (89.3%) and general admission (78.8%) groups also had bilateral involvement on the CXR findings, but more patients in the ICU group had bilateral involvement (p = 0.003).
For vital signs, elevated heart rate (100 vs. 98 rate/min, p = 0.04), elevated respiratory rate (23 vs. 20 rate/min, p<0.001), and decreased pulse oxygen saturation (SpO 2 ) (93% vs. 95%, p<0.001) were significantly associated with ICU admission. However, the magnitude of each of these differences was small and unlikely to be of clinical significance. There was no significant difference in systolic blood pressure and temperature of patients in the ICU compared to those in general admission group (p>0.05).

Characteristics of mortality group
The demographics, comorbidities, symptoms, imaging findings, vital signs, and laboratory findings of the mortality group compared with the survival group are shown in Table 2. Of the total 82 patients who died, the median age was 77 years (IQR, 66-85), and 53 (64.6%) were male.

Risk scores for ICU admission and mortality
We then developed a risk score to predict ICU admission and mortality in COVID-19 positive patients from the top predictors we identified (Fig 2). The risk score for predicting ICU admission ranged from 0 to 5, from lowest to highest risk. The score was calculated by assigning 1 point each for the five top predictors of ICU admission in Table 3. The risk score for predicting mortality ranged from 0 to 7, from lowest to highest risk. The score was calculated by assigning 1 point for each of the seven top predictors of mortality ( Table 3). The percentage of patients in ICU care increased with increasing ICU-admission risk score, while the percentage of patients in non-ICU care decreased with increased risk score. Similarly, the percentage of   (n = 187). The sensitivity and specificity of the risk scores were 10.5% and 99.2% for predicting ICU admission, and 7.1% and 100% to predict mortality. The high specificity and low sensitivity were due to the imbalance of sample sizes where there were more patients in general admission group compared to ICU admission or death group. We also developed a weighted risk score model, where each variable was assigned a weight based on their odds ratio. However, the AUC was comparable to the non-weighted model. Consequently, we determined that the non-weighted model was superior because of its simplicity and good accuracy.

Discussion
The present study used data obtained early in the course of COVID-19 infection to develop simple risk scores that elucidate two issues: 1. Providing objective evidence to aid in the decision of ICU admission, and 2. Quantifying the risk of mortality from this infection, based on parameters during their initial (emergency department) presentation. To our knowledge, this is the first risk score developed using a large US sample. The strengths of our study include rigorous methods for developing multivariable regression models and a separate dataset for validation of our risk scores. The simplified risk score system yielded an AUC of 0.74 for predicting ICU admission, and 0.82 for predicting mortality for the testing dataset. These tools can help direct COVID-19 patient flow and appropriately allocate resources. The common top predictors of ICU admission and mortality were elevated LDH and procalcitonin, and reduced SpO 2 . Elevated LDH indicates cell death and injury and is associated with a poor host immune response, resulting in a higher susceptibility to severe viral infections [17][18][19]. Procalcitonin is a widely-used indicator for critical illness due to infectious etiology, albeit with controversy [20]. It has been reported that viral infections induce the release of inflammatory cytokines such as interferon-γ which decreases procalcitonin levels [21]. Substantial increases in procalcitonin during a viral infection may therefore indicate a deficient immune response to clear the infection [21]. Severe respiratory failure and death caused by COVID-19 may result from damaged alveoli and edema formation, negatively affecting the lung's ability to oxygenate blood, as reflected in reduced oxygen saturation [22,23]. As arterial blood-gas measurement is invasive and not regularly performed at the first point of care, SpO 2 serves as a more available indicator for oxygenation for triage purposes.
Some variables were uniquely associated with either ICU admission or mortality. Reduced lymphocyte count was amongst the top predictors of ICU admission but not of mortality. Lymphopenia is a common characteristic of infection caused by the body's cytokine-induced reaction [24]. The reduced CD4+ T-cell and CD8+ T-cell levels promote viral survival and predict worse outcome [25]. In terms of past medical history and comorbidities, smoking history was a unique top predictor of ICU admission but was not associated with increased mortality in our cohort. However, medical histories of heart disease, COPD, and elevated heart rate at presentation were unique top predictors of mortality but not ICU admission, suggesting that these cardiopulmonary risk factors may contribute more to mortality in COVID-19 patients. Our findings were consistent with a recent study that found SARS-CoV-2 to cause myocardial injury through mechanisms of inflammation, resulting in fatal cardiac dysfunction and arrythmias [26].
Interestingly, unlike previous literature which generally cited age as an important predictor of mortality, our model did not find age to be amongst the top predictors for mortality in our cohort [27,28]. These results suggested that comorbidities associated with aging, rather than advanced age itself, contribute to a worse prognosis. CXR findings in our cohort were not predictive of ICU admission or mortality. However, our imaging analysis included only the presence of any abnormalities and the laterality of lung involvement. Detailed imaging findings, such as ground glass opacity and consolidation, and radiologist scoring of disease severity, is under investigation. It is not surprising that symptoms were not amongst the top predictors. Although some were significantly different between groups, it was likely attributed to the subjective reporting and non-specific nature of symptoms such as fever, cough, and shortness of breath. Ethnicity and race were not amongst our top predictors although further studies are warranted [29,30].
It is also possible to construct a model to classify hospitalized and non-hospitalized COVID-19 patients. However, data from non-hospitalized patients were limited and consisted mainly of vitals and demographics, usually without laboratory and imaging tests. A recent paper from our hospital supported this finding. In that study, Singer et al. found that confirmed COVID-19 patients had worse outcomes than COVID-19 negative patients [13]. Our study instead focused only on COVID-19 positive patients and covered different time periods, explaining some differences in the predictor variables identified from the prior study at our institution.
A few studies have attempted to develop a predictive or risk score model to facilitate clinical decision making associated with COVID-19 patients. General risk scores, such as SOFA and MEWS, lack sensitivity and specificity to predict mortality when applied to COVID-19 infection [9,10]. Lu et al. created a three-tiered risk score based on only two variables, age and CRP thresholds, to determine mortality [28]. Ji et al. identified comorbidities, age, lymphocyte count and LDH to be predictors of mortality [27]. Xie et al. reported age, lymphocyte count, LDH and SpO 2 to be independent predictors of mortality but a risk score was not developed [31]. None of these previous studies statistically ranked the predictors nor validated the risk scores using independent datasets. While Zhou et al. developed a nomogram for predicting severity of COVID-19 from comorbidities and symptoms, they did not include any laboratory values, which is an important component in the clinical realm [32]. All these prediction models were also only based on Chinese patient cohorts, who have different health profiles to US cohorts. As a result, our model may have more implications in US hospitals, which are experiencing a later surge than the rest of the world. Variables like smoking history, COPD and heart failure for example, were not part of their models, but play a significant role in our risk score.
While our model has some overlapping variables with previous risk scores, the variability between our studies can be attributed to different patient cohorts with different lifestyles, different outcomes measures being investigated (mortality, ICU admission, and disease severity), hospital environments, and statistical methods employed, among other factors. Thus, a good predictive model needs to be flexible to incorporate new and local data. Unlike most of the previous predictive models, our model was validated internally using independent datasets. Our risk-score model also incorporated 5 to 7 significant predictors, more than those in previously studies but still a manageable amount to paint an accurate clinical picture without being too broad.
This study had several limitations. First, while our risk score identified five predictors of ICU admission, it does not mean these should be the only criteria for determining ICU admission. These predictors should instead be used in conjunction to support a clinician's decision to admit a patient to the ICU. We also recognized that our hospital's criteria for ICU admission can differ from that of other hospitals. In addition to how our study measured mortality, we realized that there are other measures, such as a "time-to-event" approach or fixed follow-up period. This was also a retrospective study based on a single institution. We will collaborate with other institutions to test our model to improve generalizability. This study included clinical data only at presentation. Incorporating longitudinal clinical data are currently under investigation. While this prediction model was based on logistic regression, machine learning and other methods are being explored. Finally, it is important to note that the COVID-19 pandemic circumstance is unusual and evolving. Flow of patients (i.e., to general admission or ICU) and mortality may depend on individual hospital's patient load, practice, and available resources, which also differ amongst countries. Thus, this prediction model may need to be retrained with regional data, which can be readily accomplished.

Conclusion
This study identified the highly ranked clinical features that accurately predict ICU admission and mortality associated with COVID-19 infection. Based on these findings, we developed a practical risk-score model to stratify patients into general versus ICU admission, and to predict mortality. Our model can be readily enhanced or retrained using additional data and data from other institutions. This approach has the potential to provide frontline physicians with a simple and objective tool to stratify patients based on risks so that they can triage COVID-19 patients more effectively in time-sensitive, stressful and potentially resource-constrained environments.