https://orcid.org/0000-0002-6639-727X
Figures
Abstract
Objective
To develop and validate a scoring system to predict mortality among hospitalized patients with COVID-19.
Methods
Retrospective cohort study. We analyzed 5,062 analyzed hospitalized patients with COVID-19 treated at two hospitals; one each in Quito and Guayaquil, from February to July 2020. We assessed predictors of mortality using survival analyses and Cox models. We randomly divided the database into two sets: (i) the derivation cohort (n = 2497) to identify predictors of mortality, and (ii) the validation cohort (n = 2565) to test the discriminative ability of a scoring system. After multivariate analyses, we used the final model’s β-coefficients to build the score. Statistical analyses involved the development of a Cox proportional hazards regression model, assessment of goodness of fit, discrimination, and calibration.
Results
There was a higher mortality risk for these factors: male sex [(hazard ratio (HR) = 1.32, 95% confidence interval (95% CI): 1.03–1.69], per each increase in a quartile of ages (HR = 1.44, 95% CI: 1.24–1.67) considering the younger group (17–44 years old) as the reference, presence of hypoxemia (HR = 1.40, 95% CI: 1.01–1.95), hypoglycemia and hospital hyperglycemia (HR = 1.99, 95% CI: 1.01–3.91, and HR = 1.27, 95% CI: 0.99–1.62, respectively) when compared with normoglycemia, an AST–ALT ratio >1 (HR = 1.55, 95% CI: 1.25–1.92), C-reactive protein level (CRP) of >10 mg/dL (HR = 1.49, 95% CI: 1.07–2.08), arterial pH <7.35 (HR = 1.39, 95% CI: 1.08–1.80) when compared with normal pH (7.35–7.45), and a white blood cell count >10 × 103 per μL (HR = 1.76, 95% CI: 1.35–2.29). We found a strong discriminative ability in the proposed score in the validation cohort [AUC of 0.876 (95% CI: 0.822–0.930)], moreover, a cutoff score ≥39 points demonstrates superior performance with a sensitivity of 93.10%, a specificity of 70.28%, and a correct classification rate of 72.66%. The LR+ (3.1328) and LR- (0.0981) values further support its efficacy in identifying high-risk patients.
Conclusion
Male sex, increasing age, hypoxemia, hypoglycemia or hospital hyperglycemia, AST–ALT ratio >1, elevated CRP, altered arterial pH, and leucocytosis were factors significantly associated with higher mortality in hospitalized patients with COVID-19. A statistically significant Cox regression model with strong discriminatory power and good calibration was developed to predict mortality in hospitalized patients with COVID-19, highlighting its potential clinical utility.
Citation: Dueñas-Espín I, Echeverría-Mora M, Montenegro-Fárez C, Baldeón M, Chantong Villacres L, Espejo Cárdenas H, et al. (2023) Development and validation of a scoring system to predict mortality in patients hospitalized with COVID-19: A retrospective cohort study in two large hospitals in Ecuador. PLoS ONE 18(7): e0288106. https://doi.org/10.1371/journal.pone.0288106
Editor: Alonso Soto, Universidad Ricardo Palma Facultad de Medicina Humana, PERU
Received: June 11, 2022; Accepted: June 19, 2023; Published: July 17, 2023
Copyright: © 2023 Dueñas-Espín et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: IDE, MEM and CMF were supported by Pontificia Universidad Católica del Ecuador (Pontifical Catholic University of Ecuador, https://www.puce.edu.ec/) grant number QINV0318-IINV533020000. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
World Health Organization (WHO) has reported a global total of 767,984,989 confirmed cases of COVID-19 caused by Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), with 6,943,390 deaths. Additionally, as of 13 June 2023, a total of 13,397,292,784 vaccine doses have been administered worldwide [1]. In 2021, hospitalizations from COVID-19 have increased worldwide. COVID-19 hospitalizations reached a record high of more than 142,000 on January 14, 2021, and last topped 100,000 on September 11 [2], with a greater rate of hospitalization and death among patients with comorbidities [2], such as obesity [3], and other specific clinical conditions [4]. Moreover, diabetes has been more prevalent among fatal cases than non-fatal cases [5].
Although the mortality rate of patients with COVID-19 in the general population has been reduced by the introduction of vaccination, causing important reductions in infections, hospital admissions, and deaths [6], there is still a large burden of death from the infection [7].
Globally, the vaccination process has been carried out unequally, irregularly, and very slowly. In 2022, 90 countries have not yet reached the goal set by the World Health Organization for 2021 of vaccinating 40% of their population, and 36 countries have inoculated less than 10% of their population [8, 9]. Currently, and despite that 69.7% of the world population has received at least one dose of a COVID-19 vaccine, 13.31 billion doses have been administered globally, and 954,258 are now administered each day, only 27.7% of people in low-income countries have received at least one dose [10]. The slow process of vaccinating could have contributed to virus mutations, resulting in new variants. Specifically, despite Delta variant caused more severe, the extremely contagious Omicron variant has resulted in a new collapse of public and private healthcare facilities increasing the risk for morbidity and mortality of the population.
Importantly, despite the evidence that vaccination may make breakthrough cases less severe and people who are fully vaccinated may be less likely to spread the disease to others, the rate of hospitalization in the first months of 2022 was even higher than that reported during the year 2020 or 2021 [10].
Therefore, it is necessary to estimate factors that can predict the risk of death among hospitalized patients [11]. In addition, it is important to consider that the rise of new variants is causing impacts in terms of public health that have not been seen since 2020 [12]. These variants have increased the number of hospitalized patients throughout the region, and specifically in Ecuadorian territory.
In Ecuador, given the sanitary collapse that occurred between March and May 2020, there is evidence of underreporting COVID-19 deaths in infographics from the Ministry of Health [13]. Our country had one of the highest mortality rates from COVID-19 in Latin America in 2020 and 2021 [14, 15]; currently, 34,533 people have died due to COVID-19 in the country [16]. More than 46,656 excessive deaths have been registered compared to the years prior to the pandemic, giving an excess mortality rate of 266 per 100,000 population [16].
At present, despite the Ecuadorian prevalence of compliance with the complete vaccination protocol is being close to 79.06% (one of the highest in the region) [10, 13, 17], there are difficulties reaching populations where vaccination coverage is complex (i.e., rural areas population), a notable increase in cases has been reported during January 2022 [7]. Among the different possible explanations, the following aspects are the ones that could mainly explain it: (i) vaccines are not 100% effective, (ii) unvaccinated people could be fueling transmission, (iii) spreading the disease by vaccinated cases, (iv) people that are moving around more to restart economic activities or during vacation periods and festivities, (v) the relaxation of protection measures like distancing measures or the increase in capacity in establishments; and (vi) the current global dominance of the SARS-CoV-2 Omicron (BA.1 or B.1.1.529) and several subvariants that are appearing more frequently, with a higher transmissibility, even in vaccinated individuals [18].
With this background, decisions to contain the pandemic should be taken based on proper tools for estimating the risk bands to allocate resources at both clinical and public health levels [19]. This study aimed to identify the prognostic factors associated with in-hospital mortality of COVID-19 from the information of adult patients admitted to two Ecuadorian Social Security Institute Social, the Hospital General Norte of Guayaquil and the Hospital General del Sur of Quito, between February and July 2020, and to validate and to assess its discriminative ability in the form of a score.
Materials and methods
Design
Retrospective cohort study. This is a survival analysis of a retrospectively reconstructed cohort of patients hospitalized for COVID-19 in two large hospitals in Ecuador and developing and validation of a score for predicting in-hospital mortality. The supplementary material containing the TRIPOD [20] statement checklist has been included in the S1 Table.
Population and database
This is an individual-based study whose population is patients over 18 years of age diagnosed with COVID-19 and treated at two large hospitals of the social security system: one each in Guayaquil and Quito, Ecuador, from February to July 2020.
Inclusion Criteria: (i) age ≥18 years, (ii) confirmed diagnosis of COVID-19 through Reverse Transcriptase-Polymerase Chain Reaction (RT-PCR) testing or findings of computed tomography criteria consistent with COVID-19 [21], (iii) hospitalized in one of the aforementioned health facilities, (iv) hospitalized from February to July 2020. Exclusion criteria: (i) pregnant or in the immediate, early, or late puerperium, (ii) primary or congenital immunodeficiency, and (iii) history of neoplasms.
Main outcome
In-hospital death. For the survival analyses, we employed the time from hospitalization to in-hospital death measured in days (survival time). We calculated incidence rates of mortality (mortality rates).
Explanatory variables
We analyzed several possible explanatory variables as predictors of in-hospital death. We collected information about sex, age, anthropometric measurements, vital signs, and clinical and common-use laboratory biomarkers. We also used the variable hypoxemia. Although there is no specific reference available for the altitude of Quito, efforts have been made to establish a consistent and justified criterion. The operational definition of hypoxemia was based on previous studies, such as the work of Luks and Swenson (2011) [22] and Hupperets et al. (2004) [23], who explored the effects of altitude on respiratory physiology and the hypoxic ventilatory response. In the absence of widely established literature defining Quito’s altitude-specific hypoxemia, a consensus was reached based on clinical guidelines and expert opinion [24]. According to this consensus, hypoxemia was defined as an oxygen saturation level below 92% in Quito. Additionally, for sea level (Guayaquil), a threshold below 95% was chosen to define hypoxemia.
Statistical analyses
The analyses were based on the whole anonymized database of hospitalized patients at both hospitals (Guayaquil and Quito) from February to July 2020. For sample size calculation, based on Cox PH regression with an alpha risk of 0.05, beta risk of 0.2, and a minimum hazard ratio of 1.3, assuming a mortality rate of 14% among patients in the low-risk group [25], it yielded an estimated sample size of 1365 with 191 expected events.
Given that the whole unimputed database had 28.6% of the values missing, we assessed the presence and patterns of missing values and, assuming the missing-at-random hypothesis (i.e., missingness conditional on measured patients’ characteristics), we used multiple imputations (10 imputed datasets) with the method of chained equations [26]. We compared the characteristics of chained equations patients in the complete case to the imputed datasets (S2 Table). Given that it was not missing values of the in-hospital mortality, except for the analyses of the mortality rates in both hospitals, all analyses were performed using the imputed dataset.
To randomly divide the database into a derivation cohort and a validation cohort, we constructed a sequence of integer values from 1 to 2, as each patient’s identifier code was sorted, which divided the entire imputed database in two; therefore, we studied 2,497 patients in the derivation cohort, and 2,565 patients in the validation cohort. Importantly, the number of subjects available was more than twice the required sample, according to sample size calculations.
Building the model in the derivation cohort
We developed a univariate description of the variables in the first part of the study, followed by a bivariate analysis of in-hospital mortality from the derivation database. Then, we performed multivariate Cox proportional hazards models.
We started by building crude models between one explanatory variable and survival time, which led to the development of a saturated Cox proportional-hazard model including all the variables that achieved a p-value equal to or below 0.25 in the bivariate models. Our saturated model also included variables such as weight, height, body mass index, cardiac rate, diastolic blood pressure, body temperature, ferritin, procalcitonin, platelet count, partial thromboplastin time (PTT), and D-dimer that did not meet the cutting point but were clinically relevant according to the clinicians’ and the researchers’ expertise. Subsequently, we built a parsimonious model using a one-to-one variable reduction from the saturated model. It was performed using the researchers’ expertise instead of only the p-value as a parameter.
The saturated and parsimonious models were compared, and the final model was chosen according to the p-value of the likelihood ratio test. When the final model was selected, we assessed our model by testing proportional hazards and the overall goodness of fit. The assessment of the Cox proportional hazards model was based on a test of proportional hazards and goodness of fit. We evaluated the goodness of fit through the Cox-Snell residuals graph to visualize a 45-degree angle in the slope to confirm the adequacy of the model and assessed for multicollinearity by using the variance inflation factor.
The model was tested for both goodness of fit and the proportional hazards assumption. Goodness of fit was assessed using the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) through the “estat ic” command. The proportional hazards assumption was tested with the “estat phtest” command, which involved running chi-square tests for each covariate included in the model.
We stratified the analyses by running the final model in men and women, looking for any possible interaction between sex and other explanatory variables. We also performed a sensitivity analysis by running the final model excluding: (i) patients in both extremes of age quartiles, (ii) patients with ≥180 mmHg of systolic blood pressure or ≥120 mmHg of diastolic blood pressure; and (iii) patients with ≥200 mg/dL of serum glucose.
Building the score
We used the final model’s β-coefficients that were multiplied by a number (20) and rounded to the more approximate integer to build a score. We included the coefficients of the statistically significant variables in the score. Then, we tested the discriminative power of the score, which we called the COVID-19 in-hospital mortality score, in the derivation (internal validation) and the validation (external validation) cohorts, by calculating the area under the curve (AUC) in Receiver Operating Characteristic (ROC) curve analyses [27]. To assess if there was a good separation in mortality, we categorized the score as tertiles and assessed if there was differential mortality across risk bands by the log-rank test. For reference, an AUC between 0.7 and 0.8 is considered acceptable, while an AUC between 0.8 and 0.9 is considered to have excellent discriminative power [28].
The categorization scheme for determining risk levels was established using the provided sensitivity, specificity, LR+, and LR- values. The derivation dataset was initially used to identify the cutoff points, and subsequently, the categorization scheme was applied to the validation dataset to evaluate its performance; and then, calculated the survival functions by Kaplan–Meier survival estimates and Log-Rank test.
Validating the model in the validation cohort
We tested the cutoff value of the score in the validation cohort by calculating the AUC in ROC curve analyses. We used the same reference for assessing the discriminative power. Again, to assess if there was a good separation in mortality, we categorized the score into tertiles and assessed if there was differential mortality across risk bands. We compared the discriminatory power of the score between hospitals and between sex categories, calculating the AUC in the two hospitals and both sex categories (S3 Table). These analyses were also performed in the validation cohort only.
Calibration of the model in the validation cohort
The calibration analysis of the model in the derivation dataset was performed using the Stata command "stcoxcal" [29]. These commands allowed us to assess the calibration of the Cox proportional hazards model by testing the intercepts and slopes against specified values. The "test" command tested the hypothesis that the intercepts are equal to 0 and the slopes are equal to 1, while the "trend" command tested the hypothesis that the slopes are equal to 1 with the intercepts estimated. These analyses were conducted to evaluate the goodness of fit and assess the agreement between the observed and predicted event probabilities, providing valuable insights into the model’s calibration performance. Furthermore, we employed the "trend" command in Stata to assess the calibration of the Cox proportional hazards model in the validation dataset. Unlike the "test" command, the "trend" command allows for examining the overall trend of calibration over time.
In the calibration assessment of the Cox model applied to Quito (S4 Table), non-significant coefficients were found across all the tests, as evidenced by p-values greater than 0.05. Additionally, chi-square tests further reinforced these findings, with the null hypotheses not being rejected in all cases. These results suggest that the model is well-calibrated for this high-altitude city, as the model predictions do not significantly differ from the observed data. Therefore, the Cox model demonstrates an acceptable predictive accuracy for the city of Quito. Unfortunately, the application of the “stcoxcal” command for the Cox model to Guayaquil was hampered due to the limited number of observations in the original (non-imputed) dataset. This unfortunate circumstance restricts the evaluation of the calibration and predictive model for this city, which underscores the need to collect additional data to provide a more complete understanding of the model’s applicability to Guayaquil.
We considered that there were two-tailed statistically significant differences when the p-value was <0.05, and all analyses were performed with Stata 14.2 (Statistical Software Stata: Release 16.1 College Station, TX: StataCorp LP).
Ethics approval
This study was conducted with the authorization of the Ethics Committee of Research on Humans Beings of the Ministry of Public Health, number MSP-CGDES-2020-0183-O, in September 2020. The need for consent was waived by the ethics committee. The information was extracted from patients’ clinical records by personnel not related to the research team, who oversaw the systems and technology department of both hospitals. Data were anonymized before being accessed by the researchers.
Results
A total of 5,062 patients were analyzed. Cumulative mortality during the study was 22.6% (1,139 of 5,062 patients died in six months), which represents an in-hospital mortality rate of 236 deaths per 10,000 person-days (considering a total person-days of follow up of 48,263). Specifically, in the Hospital General Norte of Guayaquil, we found a mortality rate of 3.4 deaths per 10,000 person-days; and, in the Hospital General del Sur of Quito, the mortality rate was 1.4 per 10,000 person-days.
Table 1 shows the characteristics of the 2,497 patients who were included in the derivation cohort. Most patients were male (63%), with a mean (standard deviation, SD) of 56.5 (16.6) years of age and a mean (SD) body mass index (BMI) of 27.0 (6.0) Kg/m2. Importantly, 88.8% of patients presented a systemic inflammatory response syndrome, 33.7% had hyperglycemia, 16% had abnormal creatinine levels, and 13.6% had high blood pressure. The characteristics of the 2,565 patients in the validation cohort are shown in S5 Table.
We found several significant differences in the characteristics of patients who died in the hospital and those who did not (Table 2). Among those who died, there was a higher proportion of males (73% vs. 60%, p-value <0.001), the mean age was higher (66.5 vs. 53.6 years old, p-value <0.001), they had a higher respiratory rate (24.1 vs. 21.9 breaths per minute, p-value <0.001), and had a lower oxygen saturation (89.7% vs. 91.2%, p-value <0.001). Compared with survivors, deceased patients presented with significantly higher values of serum glucose (159.2 vs. 124.5 mg/dL, p-value <0.001), blood urea nitrogen (BUN) (27.0 vs. 16.1 mg/dL, p-value <0.001), lactic acid dehydrogenase (LDH) (340.2 vs. 252.1 U/L, p-value <0.001), pCO2 (33.2 vs. 30.1 mmHg, p-value <0.001), and white blood cell count (12.33 vs. 8.38 × 103 cells/μL, p-value <0.001); furthermore, they had lower arterial pH (7.36 vs. 7.39, p-value <0.001). In addition, deceased patients presented with a higher percentage of (i) systemic inflammatory response syndrome (98.1% vs. 96.1, p-value <0.001), (ii) hyperglycemia (55% vs. 28%, p-value <0.001), (iii) elevated serum creatinine (30% vs. 12%, p-value <0.001), and (iv) anemia (38% vs. 15%, p-value <0.001).
After multivariate analyses, we found several statistically significant and independent predictors of mortality (Table 3); male sex [(hazard ratio (HR) = 1.32, 95% confidence interval (CI): 1.03–1.69] compared to female, per each increase in a quartile of ages (HR = 1.44, 95% CI: 1.24–1.67) considering the younger group (17–44 years old) as the reference, presence of hypoxemia (HR = 1.40, 95% CI: 1.01–1.95), hypoglycemia and hospital hyperglycemia (HR = 1.99, 95% CI: 1.01–3.91, and HR = 1.27, 95% CI: 0.99–1.62, respectively) when compared with normoglycemia, an AST–ALT ratio >1 (HR = 1.55, 95% CI: 1.25–1.92), a C-reactive protein (CRP) level >10 mg/dL (HR = 1.49, 95% CI: 1.07–2.08), arterial pH <7.35 (HR = 1.39, 95% CI: 1.08–1.80) when compared with normal pH (7.35–7.45), and a white blood cell count >10 × 103 per μL (HR = 1.76, 95% CI: 1.35–2.29). The final model is shown in detail in the S6 Table, and the comparison of accuracy and other metrics between Guayaquil and Quito can be seen in the S7 Table. Goodness of fit was assessed using the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC), which yielded AIC = 286.4317 and BIC = 330.3455. Chi-square tests for each variable, together with their corresponding p-values, suggest that the risks are proportional over time. The overall test p-value for the model was 0.9814, further supporting the proportional hazards assumption. These results demonstrate that the Cox regression model used in the analysis provides a good fit to the data and satisfies the proportional hazards assumption. Furthermore, when we stratified the analyses across sex categories the estimates did not modify (S8 Table). Similarly, excluding younger and older patients or those with very high hypertension or hyperglycemia did not modify the estimates (S9 Table).
After building the score, it was named “COVID-19 in-hospital mortality score” (Table 4). We found an excellent discriminative ability when tested in the validation cohort (Fig 1), with an AUC of the ROC curve of 0.876 (95% CI: 0.822–0.930), sensitivity and specificity of 93.1% and 70.3%, respectively, and a cutoff value of ≥39 points (range, 0–86 points). The discriminative ability when tested the score in the validation cohort by hospitals and sex categories showed less discriminative capacity when the stratified analyzes were performed for the categories of the hospital and sex (S3 Table), when the analysis was performed in Guayaquil hospital only (AUC of the ROC curve of 0.742, 95% CI: 0.712 to 0.773) and when it was performed in females only (AUC of the ROC curve of 0.829, 95% CI: 0.671 to 0.986).
The results of the calibration analysis in the validation dataset indicated that the model demonstrated good calibration overall (Table 5). The chi-square tests for intercepts and slopes did not provide evidence of miscalibration (Test 1: p = 0.9321; Test 2: p = 0.7206). Furthermore, the joint test of intercepts and slopes also showed no significant evidence of miscalibration (Test 3: p = 0.9739). These findings suggest that the model’s predicted event probabilities align well with the observed event probabilities in the validation dataset.
The intercept test, slope test, joint test, and interaction test were performed to evaluate the model’s calibration.
Furthermore, a cutoff score of ≥39 is considered a better cutoff over other scores because of its balance between sensitivity and specificity, along with other performance indicators. With a cut-off score of 39, a sensitivity of 93.10%, a specificity of 70.28% and a correct classification of 72.66% are achieved. In addition, the LR+ (3.1328) and LR- (0.0981) values support the usefulness of the cutoff score of 39 in identifying patients at high risk of mortality. These indicators suggest that a cut-off score of 39 provides a reasonable balance between detecting patients at risk and identifying those who are not, offering good discriminatory power.
When comparing between both hospitals, (S10 Table) there were differences in sensitivity, specificity, correct classification, LR+ (positive likelihood ratio) and LR- (negative likelihood ratio) between the populations of Guayaquil and Quito at each cut-off point of the COVD-19 in-hospital score tested (only show selected cut points for simplicity). It is observed that, in Guayaquil, the selected score points had higher sensitivity values, but lower specificity compared to those in Quito. In addition, the score in Guayaquil exhibited lower overall precision and more modest likelihood ratios than the score in Quito.
Also, and Based on the sensitivity and specificity, as well as the LR+ and LR- values, we can establish risk levels using the following cut-off points: low risk (score ≤ 30) with high specificity (56.18%) and LR- of 0.0000, indicating a high probability of negative results; moderate risk (score between 31 and 65) with a range of sensitivity (100% to 18.92%) and specificity (60.96% to 99.20%), suggesting a mix of positive and negative probabilities; and high risk (score > 65) with a sensitivity of 10.81%, a specificity of 99.20%, and an LR+ of 13.5675, indicating significantly higher odds of positive results compared with lower scores (S11 Table). These cut-off points help classify people into different levels of risk based on the balance between sensitivity and specificity. The Kaplan–Meier curves were statistically significant (Log-Rank test, p-value <0.001) when comparing the survival between intermediate-risk, high-risk, and very high-risk groups in both cohorts (Fig 2).
When comparing between both hospitals, (S12 Table) there are differences in sensitivity, specificity, correct classification, LR+ (positive likelihood ratio) and LR- (negative likelihood ratio) between the populations of Guayaquil and Quito at each cut-off point of the COVD-19 in-hospital score tested (only show selected cut points for simplicity). It is observed that, in Guayaquil, the selected score points had higher sensitivity values, but lower specificity compared to those in Quito. In addition, the score in Guayaquil exhibited lower overall precision and more modest likelihood ratios than the score in Quito.
Discussion
Comparing patients who died with those who survived presented a significantly higher frequency of male sex, older age, hypoxemia, hypoglycemia, or hospital hyperglycemia (a hospital glucose level >140 mg/dL) [30], AST–ALT ratio >1, elevated CRP, and altered arterial pH. These factors were used to construct a prognostic clinical scoring system, the “COVID-19 in-hospital mortality score”, with the excellent ability to discriminate the risk of mortality in patients hospitalized for COVID-19.
In the presented study, it was observed that the "COVID-19 in-hospital mortality score" demonstrated greater utility in predicting mortality in unvaccinated patients and those residing in higher altitude areas. It is important to note that this finding may be related to specific biological adaptations and environmental factors in these populations [53, 54]. However, further research in diverse populations and contexts is required to confirm and better understand these findings.
Higher altitude populations have been shown to exhibit a lower rate of COVID-19 related mortality in some studies [33]. This could be due to biological adaptations to hypoxia, which may play a role in the body’s response to the virus. Additionally, specific environmental factors of high-altitude areas could influence the spread and severity of the disease [34]. On the other hand, in unvaccinated populations, the lack of immunity acquired from vaccination might lead to more severe disease outcomes, which could explain the higher utility of the score in these populations.
Nevertheless, it was found that the score exhibited lower discriminative ability when analyzed in patients from Guayaquil’s hospital and in women [S2 Table]. This suggests that the score might be less effective in certain population groups and contexts, emphasizing the importance of validating and adapting the score to different populations and settings.
In conclusion, although the "COVID-19 in-hospital mortality score" appears to be particularly useful in unvaccinated patients and those residing in higher altitude areas, further research is necessary to validate and adapt this score to different populations and contexts. This would enhance its applicability and utility in clinical practice and healthcare resource planning. Several studies have proposed mortality scoring systems in patients hospitalized for COVID-19 [35]. The most common predictors of in-hospital death used in these scores are (i) male sex [36], (ii) older age [37], (iii) presence of comorbidities (such as high blood pressure and diabetes) [38, 39], (iv) alteration of vital signs (such as hypoxemia) [40], and (v) obesity [41]. Moreover, important laboratory parameters predicting a worse prognosis included: increased concentrations of BUN [42], creatinine [43], serum glucose [44], AST [45], increased AST–ALT ratio [46], LDH [47], creatinine phosphokinase (CPK) [48], CRP [49], procalcitonin [50], ferritin [51, 52], altered blood count parameters, such as leucocytosis, leukopenia [53], lymphocytosis [54], lymphopenia [53], altered neutrophil and lymphocyte index [55], thrombocytopenia [56], and thrombocytosis [55], as well as, alteration of the prothrombin time [57], and increase in the concentration of D-dimer [52]. Present results are consistent with these investigations, supporting the feasibility for the clinical use of the “COVID-19 in-hospital mortality score”. As a systematic review showed, scoring systems for predicting severe COVID-19 showed an AUC from 0.52 to 0.98 [58], with most articles from China, Europe, and North America, where COVID- 19 hit the hardest. In the same review, despite prediction models reporting validation, most studies have a significant risk of bias, likely due to limited transparency on the predictor and outcome assessment and a limited number of samples. In that sense, and despite our retrospective approach, we overcame the small sample size of other studies, the outcome was clearly defined, and overfitting was not a limitation of the current study. Nevertheless, when we circumscribed the calculation of the AUC to female hospitalized patients and to patients of the Guayaquil hospital the AUC was reduced.
These findings could be explained by several factors: (i) the score could be less effective in predicting in-hospital death at the sea level, maybe the lower COVID-19 related deaths in altitude were the results from a biological adaptation to hypoxemia, due to environmental factors [34], or related to the different contexts between both cities/altitudes [59], (ii) the score could be less effective in predicting in-hospital death for females maybe because of the lack of information regarding comorbidities which are, generally, more frequent among females [60], and (iii) the lack of a similar representativeness of Guayaquil and female population by comparison with their counterparts (patients from Quito and male), when building the score. In summary, the score seems to work best for hospitalized men in Quito. Moreover, we strongly recommend the external validation of the “COVID-19 in-hospital mortality score” in independent populations to assess its generalizability and applicability across different settings. Future validation studies involving diverse patient cohorts will provide valuable insights into the performance and robustness of the scoring system, ensuring its reliability and usefulness in clinical practice.
Regarding the clinical course and risk factors for mortality of adult patients with COVID-19 in the studied population, we found that mortality rates between both hospitals were different (338 vs. 144 per 10,000 person-days). As other studies have found, we believe that context differences in both cities and the health system’s response adaptability factors could contribute to such differences in the mortality rate. We speculate that two main factors could explain the differences in mortality: (i) Guayaquil had a more severe impact from the pandemic at the very beginning, collapsing the health system and creating the impossibility of organizing the system in the face of overcrowded health services; and, (ii) as several studies have confirmed, context factors such as altitude [34] could improve survival in Quito (2850 m) by comparison with Guayaquil (4 m). In that sense, we recognize that we did not have enough information about the differences in the health system organization and context information. In that regard, we recommend further investigation to confirm such context determinants on COVID-19 mortality.
Moreover, the calibration analysis of the Cox proportional hazards model in the validation dataset demonstrated its ability to accurately estimate event probabilities. This indicates that the model can be considered reliable for predicting outcomes in similar population; Moreover, the tests of calibration, including the interaction between the slope and times, did not show significant differences, supporting the model’s adequacy in terms of calibration over time.
The scoring system developed in the present study has potential implications in clinical settings and in the field of public health. Specifically, we believe that this score could allow clinicians to make timely decisions to stratify the risk of patients hospitalized with COVID-19 [61]. In the field of public health, the use of the “COVID-19 in-hospital mortality score” could contribute to a rational allocation of resources for health care [62], since risk bands could be identified early with the timely allocation of resources and clinical management [63].
We believe that our study has some strengths. It was carried out in a real scenario, with an important sample of patients from the two main Ecuadorian cities, which also have very different geographic, social, economic, and climatic characteristics, allowing estimates with proper external validity. As a retrospective cohort study, it is possible to estimate mortality risk factors [64] with potential use for scoring systems.
We recognize some limitations in our study; however, these are not significant sources of bias. Probably the most important limitation is the lack of information on a majority of the variables for the patients included in the sample. Although the percentage of missing values was high, the use of multiple imputation techniques, using chained equations, have resulted in realistic estimates with standard errors very close to reality and that have already been used for studies like ours [65].
Another possible limitation to the present study is the lack of information concerning other possible health predictors, such as the need for mechanical ventilation, use of intensive care therapy, and use of inotropic drugs. Unfortunately, the database administration process prevented us from carrying out these analyses because of the urgency in responding to other pandemic health necessities. Currently, it is impossible to link the anonymized information with that from the medical records. We recognize the possible source of residual confusion in the lack of information regarding ethnicity, deprivation, specific symptoms and clinical signs, sociodemographic factors, and specific comorbidities that have been shown to be possible predictors of poor health outcomes in patients with COVID-19. Nevertheless, according to a systematic review [58] that found nine studies reporting prediction models that were rated as low risk of bias and low concerns for applicability, several scoring systems that allow proper prognosis in patients with COVID-19 do not always include these variables. Therefore, we believe that this is a source of residual confounding that is unlikely to bias our estimates substantially. A final limitation is the lack of validation of the score in a different population since the internal and external validation was carried out in the same group of patients. In that sense, we encourage a process of validation of our scoring system in other contexts.
Conclusion
Male sex, increasing age, hypoxemia, hypoglycemia or hospital hyperglycemia, AST–ALT ratio >1, elevated CRP, altered arterial pH, and leucocytosis were factors significantly associated with higher mortality in patients hospitalized with COVID-19. These factors were included to construct a prognostic clinical score system with the ability to determine the risk of mortality in patients hospitalized for COVID-19. Based on the statistical significance of the Cox regression model, its discriminatory ability (AUC-ROC), and calibration, the results suggest that the developed model is a valuable tool for predicting mortality in hospitalized COVID-19 patients. The model demonstrates robust statistical performance, with significant associations between the predictor variables and mortality outcomes. Furthermore, the model exhibits good discrimination, as evidenced by a high AUC-ROC value. The calibration analysis confirms the model’s ability to accurately estimate mortality probabilities. These findings collectively indicate the potential clinical utility of the model in identifying individuals at higher risk of mortality and informing appropriate interventions.
Supporting information
S1 Table. Prediction model development and validation checklist (adapted from (1))—for illustrative purposes only.
https://doi.org/10.1371/journal.pone.0288106.s002
(DOCX)
S2 Table. Comparison of characteristics between imputed and non-imputed data sets (derivation cohort).
https://doi.org/10.1371/journal.pone.0288106.s003
(DOCX)
S3 Table. Comparison of the discriminatory power of the score between hospitals and between sex categories, calculating the area under the ROC curve in the validation cohort.
https://doi.org/10.1371/journal.pone.0288106.s004
(DOCX)
S4 Table. Summary of the Cox model calibration results for Quito.
https://doi.org/10.1371/journal.pone.0288106.s005
(DOCX)
S5 Table. Characteristics at the time of admission of the patients treated at the two hospitals in Quito and Guayaquil (validation cohort).
https://doi.org/10.1371/journal.pone.0288106.s006
(DOCX)
S6 Table. Cox regression modelling with multiple imputations was applied.
https://doi.org/10.1371/journal.pone.0288106.s007
(DOCX)
S7 Table. Comparison of accuracy and other metrics between Guayaquil an Quito.
https://doi.org/10.1371/journal.pone.0288106.s008
(DOCX)
S8 Table. Crude and adjusted associations between baseline variables at admission and in-hospital death, according to our parsimonious model on Table 3 of the main text (Cox proportional hazards model) stratified by sex (validation cohort).
https://doi.org/10.1371/journal.pone.0288106.s009
(DOCX)
S9 Table. Crude and adjusted associations between baseline variables at admission and in-hospital death, according to our parsimonious model on Table 3 of the main text (Cox proportional hazards model) excluding: (i) patients in both extreme of age quartiles, (ii) patients with ≥180 mmHg of systolic blood pressure or ≥90 mmHg of diastolic blood pressure; and (iii) patients with ≥200 mg/dL of serum glucose (validation cohort).
https://doi.org/10.1371/journal.pone.0288106.s010
(DOCX)
S10 Table. Comparative detailed report of sensitivity and specificity between Guayaquil and Quito.
https://doi.org/10.1371/journal.pone.0288106.s011
(DOCX)
S11 Table. Sensitivity, specificity, and classification accuracy of the proposed risk classification cutpoints.
https://doi.org/10.1371/journal.pone.0288106.s012
(DOCX)
S12 Table. Comparative detailed report of sensitivity and specificity between Guayaquil and Quito.
https://doi.org/10.1371/journal.pone.0288106.s013
(DOCX)
Acknowledgments
Authors thank the contributions from the personnel in charge of anonymizing data from Hospital Los Ceibos, Guayaquil, and Hospital General del Sur de Quito—IESS, Quito.
References
- 1.
WHO. WHO Coronavirus (COVID-19) Dashboard With Vaccination. World Health Organization. 2021.
- 2. Antos A, Kwong ML, Balmorez T, Villanueva A, Murakami S. Unusually high risks of COVID-19 mortality with age-related comorbidities: An adjusted meta-analysis method to improve the risk assessment of mortality using the comorbid mortality data. Infect Dis Rep. 2021;13(3):700–11. pmid:34449622
- 3. Huang Y, Lu Y, Huang YM, Wang M, Ling W, Sui Y, et al. Obesity in patients with COVID-19: a systematic review and meta-analysis. Metabolism [Internet]. 2020;113:154378. Available from: pmid:33002478
- 4. Gimeno-Miguel A, Bliek-Bueno K, Poblador-Plou B, Carmona-Pírez J, Poncel-Falcó A, González-Rubio F, et al. Chronic diseases associated with increased likelihood of hospitalization and mortality in 68,913 COVID-19 confirmed cases in Spain: A population-based cohort study. PLoS One. 2021;16(11 November):1–14.
- 5. Gold MS, Sehayek D, Gabrielli S, Zhang X, McCusker C, Ben-Shoshan M. COVID-19 and comorbidities: a systematic review and meta-analysis. Postgrad Med [Internet]. 2020;132(8):749–55. Available from: pmid:32573311
- 6. Bernal JL, Andrews N, Gower C, Robertson C, Stowe J, Tessier E, et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: Test negative case-control study. BMJ. 2021;373.
- 7.
World Health Organization. WHO Coronavirus Disease (COVID-19) Dashboard With Vaccination Data | WHO Coronavirus (COVID-19) Dashboard With Vaccination Data [Internet]. World Health Organization. 2021 [cited 2022 May 17]. p. 1–5. Available from: https://covid19.who.int/
- 8.
WHO. Vaccine equity [Internet]. The World Health Organization. 2022 [cited 2022 Mar 29]. Available from: https://www.who.int/campaigns/vaccine-equity
- 9. Ghebreyesus TA. Five steps to solving the vaccine inequity crisis. PLOS Glob Public Heal. 2021 Oct 13;1(10):e0000032. pmid:36962118
- 10. Hasell J, Ortiz-Ospina E, Ritchie H, Roser M. “Coronavirus Pandemic (COVID-19)”. Our World in Data. In: https://ourworldindata.org/coronavirus [Internet]. 2022 [cited 2022 Mar 27]. Available from: https://ourworldindata.org/coronavirus
- 11. Perez-Guzman PN, Daunt A, Mukherjee S, Crook P, Forlano R, Kont MD, et al. Clinical Characteristics and Predictors of Outcomes of Hospitalized Patients With Coronavirus Disease 2019 in a Multiethnic London National Health Service Trust: A Retrospective Cohort Study. Clin Infect Dis. 2021;73(11):e4047–57. pmid:32766823
- 12. Le Rutte EA, Shattock AJ, Chitnis N Phd, Kelly SL, Penny MA Phd. Assessing impact of Omicron on SARS-CoV-2 dynamics and public health burden. medRxiv [Internet]. 2021 Dec 14 [cited 2022 Mar 27];2021.12.12. Available from: https://www.medrxiv.org/content/
- 13.
Boletines epidemiológicos coronavirus por semanas–Ministerio de Salud Pública [Internet]. [cited 2022 Mar 27]. Available from: https://www.salud.gob.ec/boletines-epidemiologicos-coronavirus-por-semanas/
- 14. Ortiz-Prado E, Simbaña-Rivera K, Barreno LG, Diaz AM, Barreto A, Moyano C, et al. Epidemiological, socio-demographic and clinical features of the early phase of the COVID-19 epidemic in Ecuador. PLoS Negl Trop Dis. 2021;15(1):1–18. pmid:33395425
- 15. León Cabrera JM, Kurmanaev A. Ecuador’s Death Toll During Outbreak Is Among the Worst in the World. New York Times [Internet]. 2020 [cited 2022 May 17]; Available from: https://www.nytimes.com/2020/04/23/world/americas/ecuador-deaths-coronavirus.html
- 16. Observatorio Social del Ecuador. Personas fallecidas | coronavirus Ecuador [Internet]. Personas fallecidas por coronavirus en Ecuador. 2021 [cited 2022 May 17]. Available from: https://www.covid19ecuador.org/fallecidos
- 17. Statista. COVID-19: dosis y porcentaje de vacunados por país de América Latina y el Caribe | Statista [Internet]. Porcentaje de vacunados y dosis. 2021 [cited 2022 May 17]. Available from: https://es.statista.com/estadisticas/1258801/porcentaje-y-numero-vacunados-contra-covid-19-en-latinoamerica-por-pais/
- 18.
Khandia R, Singhal S, Alqahtani T, Kamal MA, El-Shall NA, Nainu F, et al. Emergence of SARS-CoV-2 Omicron (B.1.1.529) variant, salient features, high global health concerns and strategies to counter it amid ongoing COVID-19 pandemic. Vol. 209, Environmental Research. Academic Press Inc.; 2022.
- 19. Jeon J, Baruah G, Sarabadani S, Palanica A. Identification of risk factors and symptoms of COVID-19: Analysis of biomedical literature and social media data. J Med Internet Res. 2020;22(10):1–10. pmid:32936770
- 20. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD Statement. Eur Urol [Internet]. 2015;67(6):1142–51. Available from: pmid:25572824
- 21. Simpson S, Kay FU, Abbara S, Bhalla S, Chung JH, Chung M, et al. Radiological society of North America expert consensus document on reporting chest CT findings related to COVID-19: Endorsed by the society of thoracic radiology, the American college of radiology, and RSNA. Radiol Cardiothorac Imaging. 2020;2(2).
- 22. Luks AM, Swenson ER. Pulse oximetry at high altitude. High Alt Med Biol. 2011;12(2):109–19. pmid:21718156
- 23. Hupperets MDW, Hopkins SR, Pronk MG, Tiemessen IJH, Garcia N, Wagner PD, et al. Increased hypoxic ventilatory response during 8 weeks at 3800 m altitude. Respir Physiol Neurobiol. 2004;142(2–3):145–52. pmid:15450476
- 24. Llano M, Villamagua B, Garelli DZ, Freund P, Francisco V, Fernando AL, et al. Biomedical journal. Biomed J. 2016;1(1):1–9.
- 25. Matta S, Chopra KK, Arora VK. Morbidity and mortality trends of Covid 19 in top 10 countries. Indian J Tuberc [Internet]. 2020;67(4):S167–72. Available from: pmid:33308665
- 26. Donders ART, GJMG van der Heijden, Stijnen T, Moons KGM. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol [Internet]. 2006 Oct [cited 2014 Jul 12];59(10):1087–91. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16980149 pmid:16980149
- 27. Hajian-Tilaki K. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Casp J Intern Med. 2013;4(2):627–35. pmid:24009950
- 28. Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol [Internet]. 2010;5(9):1315–6. Available from: pmid:20736804
- 29. Royston P. Tools for checking calibration of a Cox model in external validation: Prediction of population-averaged survival curves based on risk groups. Stata J. 2015;15(1):275–91.
- 30. Care D, Suppl SS. 16. Diabetes Care in the Hospital: Standards of Medical Care in Diabetes—2022. Diabetes Care. 2022;45(January):S244–53. pmid:34964884
- 31. Hall P, Cash J. What is the real function of the liver “function” tests?: Discovery Service for Endeavour College of Natural Health Library. Ulster Med J [Internet]. 2012;81(1):30–6. Available from: www.ebscohost.com
- 32. Ruíz‐Argüelles GJ, Sánchez‐Medal L, Loría A, Piedras J, Córdova MS. Red cell indices in normal adults residing at altitudes from sea level to 2670 meters. Am J Hematol. 1980;8(3):265–71.
- 33. Arias-Reyes C, Zubieta-DeUrioste N, Poma-Machicao L, Aliaga-Raduan F, Carvajal-Rodriguez F, Dutschmann M, et al. Does the pathogenesis of SARS-CoV-2 virus decrease at high-altitude? Respir Physiol Neurobiol. 2020;277(April). pmid:32333993
- 34. Simbaña-Rivera K, Jaramillo PRM, Silva JVV, Gómez-Barreno L, Campoverde ABV, Novillo Cevallos JF, et al. High-altitude is associated with better short-term survival in critically ill COVID-19 patients admitted to the ICU. PLoS One. 2022;17(3):e0262423. pmid:35358185
- 35. Vicka V, Januskeviciute E, Miskinyte S, Ringaitiene D, Serpytis M, Klimasauskas A, et al. Comparison of mortality risk evaluation tools efficacy in critically ill COVID-19 patients. BMC Infect Dis [Internet]. 2021;21(1):1–7. Available from: https://doi.org/10.1186/s12879-021-06866-2
- 36. Haitao T, Vermunt J V., Abeykoon J, Ghamrawi R, Gunaratne M, Jayachandran M, et al. COVID-19 and Sex Differences: Mechanisms and Biomarkers. Mayo Clin Proc [Internet]. 2020;95(10):2189–203. Available from: pmid:33012349
- 37. Starke KR, Reissig D, Petereit-Haack G, Schmauder S, Nienhaus A, Seidler A. The isolated effect of age on the risk of COVID-19 severe outcomes: A systematic review with meta-analysis. BMJ Glob Heal. 2021;6(12):1–12. pmid:34916273
- 38. Al‐Sabah S, Al‐Haddad M, Al‐Youha S, Jamal M, Almazeedi S. COVID ‐19: Impact of obesity and diabetes on disease severity. Clin Obes. 2020;10(6). pmid:33079448
- 39. Fang L, Karakiulakis G, Roth M. Are patients with hypertension and diabetes mellitus at increased risk for COVID-19 infection? Lancet Respir Med [Internet]. 2020;8(4):e21. Available from: pmid:32171062
- 40. Marateb HR, von Cube M, Sami R, Haghjooy Javanmard S, Mansourian M, Amra B, et al. Absolute mortality risk assessment of COVID-19 patients: the Khorshid COVID Cohort (KCC) study. BMC Med Res Methodol [Internet]. 2021;21(1):1–9. Available from: https://doi.org/10.1186/s12874-021-01340-8
- 41. Kompaniyets L, Goodman AB, Belay B, Freedman DS, Sucosky MS, Lange SJ, et al. Body Mass Index and Risk for COVID-19–Related Hospitalization, Intensive Care Unit Admission, Invasive Mechanical Ventilation, and Death—United States, March–December 2020. MMWR Surveill Summ. 2021;70(10):355–61.
- 42. Küçükceran K, Ayrancı MK, Girişgin AS, Koçak S, Dündar ZD. The role of the BUN/albumin ratio in predicting mortality in COVID-19 patients in the emergency department. Am J Emerg Med. 2021;48:33–7. pmid:33838471
- 43. Ghahramani S, Tabrizi R, Lankarani KB, Kashani SMA, Rezaei S, Zeidi N, et al. Laboratory features of severe vs. non-severe COVID-19 patients in Asian populations: A systematic review and meta-analysis. Eur J Med Res [Internet]. 2020;25(1):1–10. Available from: https://doi.org/10.1186/s40001-020-00432-3
- 44. Kumar B, Mittal M, Gopalakrishnan M, Garg MK, Misra S. Effect of plasma glucose at admission on covid-19 mortality: Experience from a tertiary hospital. Endocr Connect. 2021;10(6):589–98. pmid:33971617
- 45. Moon AM, Barritt AS. Elevated Liver Enzymes in Patients with COVID-19: Look, but Not Too Hard. Dig Dis Sci [Internet]. 2021;66(6):1767–9. Available from: pmid:32875529
- 46. Qin C, Wei Y, Lyu X, Zhao B, Feng Y, Li T, et al. High aspartate aminotransferase to alanine aminotransferase ratio on admission as risk factor for poor prognosis in COVID-19 patients. Sci Rep [Internet]. 2020;10(1):1–10. Available from: https://doi.org/10.1038/s41598-020-73575-2
- 47. Peiró ÓM, Carrasquer A, Sánchez-Gimenez R, Lal-Trehan N, del-Moral-Ronda V, Bonet G, et al. Biomarkers and short-term prognosis in COVID-19. Biomarkers [Internet]. 2021;26(2):119–26. Available from: pmid:33426934
- 48. Akbar MR, Pranata R, Wibowo A, Lim MA, Sihite TA, Martha JW. The prognostic value of elevated creatine kinase to predict poor outcome in patients with COVID-19—A systematic review and meta-analysis: Creatinine Kinase in COVID-19. Diabetes Metab Syndr Clin Res Rev. 2021;15(2):529–34.
- 49. Smilowitz Nathaniel R., Kunichoff Dennis, Garshick Michael, Shah Binita, Pillinger Michael, Hochman Judith S., et al. C-reactive protein and clinical outcomes in patients with COVID-19. Eur Heart J. 2021;42(23):2280–3.
- 50. Hu R, Han C, Pei S, Yin M, Chen X. Procalcitonin levels in COVID-19 patients. Int J Antimicrob Agents. 2020;56(2):8–10.
- 51. Vargas M, Rojo C. Ferritin levels and COVID-19. Ther Adv Vaccines. 2020;9(5):2019–20.
- 52. Hussein Ali M., Taha Zhala B., Ahmed Gailan Malek Kamgar Akram Rasul, Dur Qasim Hazim Reman Jalal Ahmed, et al. D-Dimer and Serum Ferritin as an Independent Risk Factor for Severity in COVID-19 Patients. Mater Today Proc J. 2021.
- 53. Huang G, Kovalic AJ, Graber CJ. Prognostic value of leukocytosis and lymphopenia for coronavirus disease severity. Emerg Infect Dis. 2020;26(8):1839–41. pmid:32384045
- 54. Jamal M, Bangash HI, Habiba M, Lei Y, Xie T, Sun J, et al. Immune dysregulation and system pathology in COVID-19. Virulence [Internet]. 2021;12(1):918–36. Available from: pmid:33757410
- 55. Rodriguez-Morales AJ, Cardona-Ospina JA, Gutiérrez-Ocampo E, Villamizar-Peña R, Holguin-Rivera Y, Escalera-Antezana JP, et al. Clinical, laboratory and imaging features of COVID-19: A systematic review and meta-analysis. Travel Med Infect Dis. 2020;34(January). pmid:32179124
- 56. Araya S, Mamo MA, Tsegay YG, Atlaw A, Aytenew A, Hordofa A, et al. Blood coagulation parameter abnormalities in hospitalized patients with confirmed COVID-19 in Ethiopia. PLoS One [Internet]. 2021;16(6 June):1–9. Available from: pmid:34153056
- 57. Guan W, Ni Z, Hu Y, Liang W, Ou C, He J, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. N Engl J Med. 2020;382(18):1708–20. pmid:32109013
- 58. Miller JL, Tada M, Goto M, Chen H, Dang E, Mohr NM, et al. Prediction models for severe manifestations and mortality due to COVID ‐19: A systematic review. Acad Emerg Med. 2022;29(2):206–16. pmid:35064988
- 59. Abbasi BA, Chanana N, Palmo T, Pasha Q. Disparities in COVID-19 incidence and fatality rates at high-altitude. PeerJ. 2023;11(e14473):1–16. pmid:36788813
- 60. Gu H, Wang C, Zhang X, Jiang Y, Li H, Meng X. Ten-Year Trends in Sex Differences in Cardiovascular Risk Factors, In-Hospital Management, and Outcomes of Ischemic Stroke in China: Analyses of a Nationwide Serial Cross-Sectional Survey from 2005 to 2015. Int J Stroke. 2023;Feb 8:1747(Epub ahead of print):2023.
- 61. Chu K, Alharahsheh B, Garg N, Guha P. Evaluating risk stratification scoring systems to predict mortality in patients with COVID-19. BMJ Heal Care Informatics. 2021;28(1):1–6. pmid:34521623
- 62. Huang J, Xu Y, Wang B, Xiang Y, Wu N, Zhang W, et al. Risk stratification scores for hospitalization duration and disease progression in moderate and severe patients with COVID-19. BMC Pulm Med. 2021;21(1):1–15.
- 63. Cherian R. Clinical risk stratification in COVID-19: the need for a revised approach? Pulm Circ. 2021;11(1):0–3. pmid:33614018
- 64.
Munnangi S, Boktor SW. Epidemiology Of Study Design [Internet]. StatPearls. StatPearls Publishing; 2019 [cited 2022 Mar 29]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK470342/
- 65. Kuroda S, Matsumoto S, Sano T, Kitai T, Yonetsu T, Kohsaka S, et al. External validation of the 4C Mortality Score for patients with COVID-19 and pre-existing cardiovascular diseases/risk factors. BMJ Open. 2021;11(9):1–9. pmid:34497086