Validation and repurposing of the MSL-COVID-19 score for prediction of severe COVID-19 using simple clinical predictors in a triage setting: The Nutri-CoV score

Background During the COVID-19 pandemic, risk stratification has been used to decide patient eligibility for inpatient, critical and domiciliary care. Here, we sought to validate the MSL-COVID-19 score, originally developed to predict COVID-19 mortality in Mexicans. Also, an adaptation of the formula is proposed for the prediction of COVID-19 severity in a triage setting (Nutri-CoV). Methods We included patients evaluated from March 16th to August 17th, 2020 at the Instituto Nacional de Ciencias Médicas y Nutrición, defining severe COVID-19 as a composite of death, ICU admission or requirement for intubation (n = 3,007). We validated MSL-COVID-19 for prediction of mortality and severe disease. Using Elastic Net Cox regression, we trained (n = 1,831) and validated (n = 1,176) a model for prediction of severe COVID-19 using MSL-COVID-19 along with clinical assessments obtained at a triage setting. Results The variables included in MSL-COVID-19 are: pneumonia, early onset type 2 diabetes, age > 65 years, chronic kidney disease, any form of immunosuppression, COPD, obesity, diabetes, and age <40 years. MSL-COVID-19 had good performance to predict COVID-19 mortality (c-statistic = 0.722, 95%CI 0.690–0.753) and severity (c-statistic = 0.777, 95%CI 0.753–0.801). The Nutri-CoV score includes the MSL-COVID-19 plus respiratory rate, and pulse oximetry. This tool had better performance in both training (c-statistic = 0.797, 95%CI 0.765–0.826) and validation cohorts (c-statistic = 0.772, 95%CI 0.0.745–0.800) compared to other severity scores. Conclusions MSL-COVID-19 predicts inpatient COVID-19 lethality. The Nutri-CoV score is an adaptation of MSL-COVID-19 to be used in a triage environment. Both scores have been deployed as web-based tools for clinical use in a triage setting.


Introduction
The pandemic caused by the SARS-CoV2 virus, which is causative of COVID-19, has led to increased morbidity and mortality, posing challenges to healthcare systems worldwide [1]. Since its arrival in Mexico in late February 2020 to date, it has caused over 800,000 cases and more than 90,000 deaths attributable to COVID-19 [2]. Although most of these cases will remain mild or moderate, a group of patients could develop a severe to critical form of COVID-19 which will require quick medical assessment to prevent adverse clinical outcomes [3]. Most of these cases with severe to critical disease have underlying chronic cardiometabolic comorbidities (e.g., obesity, type 2 diabetes, arterial hypertension), chronic kidney disease, pulmonary obstructive disease and immunosuppression of any cause [4]. The high prevalence of obesity, type 2 diabetes and other chronic diseases, in addition with socioeconomic disparities, unequal access to prompt medical attention, and a lack of sufficient health infrastructure pose intense challenges in the public health care system in Mexico [5,6]. As the disease continues to spread locally, the implementation of clinical tools to assess those patients with a higher risk for complications and lethality are needed [7,8]. Recently, some predictive scores have been proposed and validated for COVID-19 in a hospital care setting, especially to predict critical illness hospitalization, intense care unit (ICU) admission, and mechanical ventilation support. However, these scores include specialized image acquisition or laboratories, which might not be available at most primary care scenarios in Mexico and elsewhere [9][10][11]. Recently, our group developed a novel mechanistic score for lethality attributable to COVID-19 (MSL-COVID- 19) using age, self-reported comorbidities, and clinically suspected pneumonia, which performed adequately in a real-world scenario [12]. This tool considers the major contribution of chronic diseases to develop severe forms of COVID-19. Hence, the objective of the present study is to validate the MLS-COVID-19 to predict relevant hospitalization outcomes. Furthermore, we sought to develop an improved score based in the MLS-COVID-19 to predict disease severity including simple clinical measurements obtained in a triage setting.

Source of data and study population
This is a secondary analysis of the registry data of COVID-19 patients attended at the Instituto Nacional de Ciencias Médicas y Nutrición Salvador Zubirán (INMCNSZ). Briefly, we included patients aged >18 years with complete clinical data from March 16 th to August 17 th , 2020 who were evaluated at triage at INCMNSZ, a COVID-19 reference center in Mexico City, and had confirmed SARS-CoV-2 infection by RT-PCR test in respiratory samples. All patients were followed up until September 15 th , 2020. All clinical procedures and measurements were approved by the INCMNSZ Research and Ethics Committee, written informed consent was waived due to the observational nature of the study. Amongst all evaluated patients within the study period, we considered consecutive patients with complete data to estimate the MSL-COVID-19 score (n = 3,007). This report adheres to the TRIPOD guidelines for development and validation of predictive models.

Outcome and timepoint setting
Clinical recovery was defined as hospital discharge based on the absence of clinical symptoms requiring inpatient management. ICU requirement was based on clinical judgment of the attending physician. Severe COVID-19 was determined as a composite event of either death, ICU admission requirement or mechanical ventilation; we estimated performance of metrics for this outcome at 7, 10, 15, 20 and 30 days, with a main focus being at 15 days given previous research indicating the disease course of mild COVID-19 cases. Follow-up time was estimated from date of symptom onset, considered as time zero of follow-up, up to last follow-up (censoring) or the composite event of severe COVID-19, which included death, requirement for invasive ventilation or ICU admission.

Predictor variables
Information was collected prospectively at the time of triage and emergency department evaluation; candidate predictor variables included predictors which were routinely measured in a triage setting without laboratory measurements. We only considered predictors available in >80% or participants. Predictors included demographic variables including age and sex, medical history of comorbidities including type 2 diabetes, obesity, chronic obstructive pulmonary disease (COPD), asthma, hypertension, immunosuppression, HIV, cardiovascular disease (CVD), chronic kidney disease (CKD), chronic liver disease (CLD), smoking habits, and current symptoms, as described elsewhere [13]. Physical examination included weight (measured in kilograms) and height (measured in meters) to estimate the body-mass index (BMI), vital signs including pulse oximetry (SpO 2 ), respiratory rate (RR), heart rate (HR) and arterial blood pressure (BP). The Charlson Comorbidity Index (CCI), the National Early Warning Score (NEWS, NEWS2), and the quick Sequential Organ Failure Assessment (qSOFA) were also estimated to predict risk of severe COVID-19 [11,14,15]. Using this information, we also calculated the Mechanistic Score for COVID-19 Lethality (MSL-COVID-19), developed for prediction of COVID-19 case lethality in Mexicans using nation-wide case data (5).

Missing data
We included complete-case analysis for validation of MSL-COVID-19 under the assumption of data missing completely at random. Next, we analyzed patterns of missing data on clinical predictors to assess patterns of missing data and performed multiple imputation for variables assumed to be completely missing at random or missing at random using multivariable imputation with chained equations within the mice R package, generating five multiply imputed datasets and pooling the results of modeling using Rubin's rules. Matrices of missing data are fully presented in S1 File.

Sample size estimation
Sample size was estimated for a time-to-event model using the guidelines proposed by Riley et al implemented with the pmsamplesize R package [16]. We considered the prior estimated cstatistic for the MSL-COVID- 19

Statistical analysis
Outcome assessment. We compared patients with and without Severe COVID-19 using chi-squared tests for categorical variables and Student's t-test or Mann-Whitney's U for continuous variables depending on variable distribution. We evaluated each variable for prediction of mortality, ICU admission, requirement of mechanical invasive ventilation and/or severe disease using Kaplan-Meier curves and Cox Proportional Hazard Regression models; model assumptions were verified using Schöenfeld residuals. A multivariable model was fitted to assess for predictor independence with model selection being carried out using minimizations in the Bayesian Information Criterion (BIC).

MSL-COVID-19 model validation and predictors of severe disease.
To validate the MSL-COVID-19 score we used Cox Proportional Hazard regression models on the continuous score and assessed model performance using Harrel's c-statistic, Sommer's D xy score and the calibration slope in the overall sample for prediction of mortality, ICU and/or invasive ventilation requirement and severe/critical COVID-19. To estimate the optimism in these metrics, we carried out bootstrapping (B = 1,000) using the rms R package, using bias corrected accelerated (BCa) bootstrapping.
Derivation of the Nutri-CoV score. To increase precision of the MSL-COVID-19 score, we developed a two-step model whereby the score would be used as a first step and a second step would consider the previous risk and update it with clinical information. We included patients with complete clinical and physical examination data (n = 3,007) and split it into a training and validation datasets for patients admitted up to June 4 th 2020 (training) and from that date up to August 17 th (validation). To address overfitting and improve generalizability of our findings, we used Elastic Net Cox Regression, a regularization algorithm which handles multicollinearity and perform variable selection to increase generalizability using a λ penalization parameter and an α mixture parameter. We fitted an Elastic Net Cox Proportional Risk Regression Model using the both the MAMI and gmlnet R packages to incorporate coefficient estimation from multiply imputed data including al predictor variables depicted in Table 1; the ideal λ penalization parameter was estimated using k-fold cross-validation (k = 10) in the training dataset (S1 File). We selected the optimum alpha mixture parameter using simultaneous cross-validation for consecutive alpha mixture values ranging from 0 to 1 using 0.1 increments across each multiply imputed dataset and obtaining an average from all alpha values; to implement this, we used the cva.glmnet function of the glmnetUtils R package. We explored models including non-linear terms using restricted cubic splines, deciding the number of knots based on BIC minimization, and categorized models. We compared the performance of models with non-linear terms and categorized variables using cross-validation; since we observed only marginal decreases in performance with categorized variables, we proceeded with models using categorized predictors for simplification. Model coefficients from the Elastic Net Model were normalized to its ratio with the lowest absolute β coefficient to develop a point system and categories were developed to maximize separation assessed using Kaplan-Meier curves in the training dataset.
Comparison of Nutri-CoV with other severity measures. We assessed the performance of the Nutri-CoV score in both the training and validation cohorts using a simple imputed dataset with mice and rms R packages, with correction for overoptimism being estimated using bias corrected accelerated (BCa) bootstrapping. We also estimated additional performance metrics including time-dependent sensitivity, specificity, positive and negative predictive values were obtained using the timeROC R packages using Inverse Probability of Censoring

PLOS ONE
Weighting (IPCW) estimation of Cumulative/Dynamic time-dependent ROC curves for 7, 10, 15, 20 and 30 days after symptom onset with weighting performed using Cox proportional risk regression. Finally, we compared performance of MSL-COVID-19, ABC-GOALS, qSOFA, NEWS, NEWS2, and the CCI to predict disease severity using decision curve analyses estimated with the rmda R package. A p-value <0.05 was considered as statistical significance threshold. All analyses were performed using R software version 3.6.2. To facilitate the use of the score, we implemented both the MSL-COVID-19 and the Nutri-CoV score in a web-based tool using the ShinyApps R package hosted at: https://uiem.shinyapps.io/nutri_cov/, and also available in Spanish at https://uiem.shinyapps.io/nutri_cov_es/.

Sensitivity analyses
We performed recalculation of validation metrics using the rms package under the following scenarios: 1) Comparing participants with and without comorbidities, 2) excluding participants who were not yet discharged and were censored at the end of follow-up, 3) excluding participants who had severe COVID-19 upon admission, 4) comparing cases above and below 60 years of age, and 5) considering complete case analysis vs. multiply imputed analysis.

Predictors of COVID-19 disease severity
In univariate analyses using Cox proportional risk regression, we observed a higher risk of severe COVID-19 with increasing age (HR 1. In the fully adjusted models, the only independent predictors for severe COVID-19 were increasing age, and higher RR, whilst higher SpO 2 was a protective factor ( Table 2). Inclusion of use of supplementary oxygen in the model did not attenuate the observed associations.  Table 2).

Derivation of the Nutri-CoV score
Next, we sought to develop a score to predict disease severity including previous assessment with MSL-COVID-19. Using this Elastic Net Cox Regression, we confirmed that a combination of RR, SpO2 and the MSL-COVID-19 score improved prediction of severe COVID-19 (Fig 1). We fitted the model in the training cohort (n = 1831, 373 outcomes) and identified an  Fig 2).

Comparison of Nutri-CoV with severity scores
Finally, we compared performance of Nutri-CoV compared to MSL-COVID-19, ABC--GOALS, the ROX index, qSOFA, and NEWS to predict COVID-19 severity overall and in the validation cohort (Table 4). We observed a significant and improved performance, as measures by time-dependent AUROC, for Nutri-CoV to predict 15-day risk of severe COVID-19 in comparison to all scores in the validation cohort. When assessing its performance compared to other indexes using decision curve analysis, significant clinical benefit was established for the Nutri-CoV score compared to other indexes (Fig 3).

Sensitivity analyses
We conducted sensitivity analyses for relevant risk categories and to assess methodological decisions during score derivation (Table 5). Notably, Nutri-CoV observed improved performance in cases who were younger than age 60 and had no comorbidities, with decreased performance for use of only hospitalized cases. Notably, we observed a minimal decrease in performance when conducting complete-case analysis compared to multiply imputed data or exclusing cases who had not been discharged.

Discussion
Here, we validated the MSL-COVID-19 score for prediction of inpatient mortality at a COVID-19 reference center in Mexico City. We also demonstrated the role of MSL-COVID-19 in predicting severe and critical COVID-19 and how this estimation can be improved by considering additional clinical data obtained at triage. The repurposed score, which we named Nutri-CoV, includes demographics and comorbidity assessment as well as physical examination in the form of SpO2 and RR assessment, achieved significant discriminative capacity to detect potential cases of severe COVID-19. We deployed both the MSL-COVID-19 and the Nutri-CoV scores onto a web-based tool which could be readily used within a triage setting to identify individuals at the highest risk of developing COVID-19 complications. Our model could be used to make prompt decisions regarding timely admission and treatment initiation in patients at risk of severe and critical COVID-19, as well as resource allocation in the setting of increased healthcare stress during pandemic peaks. Given the remarkable increase in the availability of COVID-19 predictive models, we expect the derivation of our score will be helpful in triage settings with similarities to our institution; however, extensive external validation and calibration studies are required to evaluate its performance in a triage prior to clinical utilization [17]. Given the conducted sensitivity analyses, we do not currently recommend the use of our score for in-hospital clinical deterioration, which could be assessed with other available tools after due external validation [18]. Underlying factors which explain the pathogenesis of severe COVID-19 complications have been extensively studied [19,20]. A role for metabolic comorbidities including type 2  diabetes and its associated glycemic control, obesity, hypertension, older age and male sex has been reported as risk factors for severe COVID-19. In Mexicans, our group developed the MSL-COVID-19score, which attempts to capture the risk of most of these factors with a particular focus on predicting mortality [12]. Despite the relevance of estimating mortality risk in COVID-19, the elevated healthcare stress which has been steadily observed during the pandemic makes the consideration of ICU admission and requirement for invasive intubation relevant, especially for resource allocation and prompt treatment initiation of high-risk patients [21,22]. However, controversy surrounding early intubation for severe COVID-19 could make identification of these cases relevant for close clinical follow-up and evaluation. Previous severity scores developed for COVID-19 consider radiological and laboratory findings in relation to clinical status and comorbidities [10]. Nevertheless, a main concern in low-resource settings is the availability of healthcare resources and a highly sensitive clinical score might be relevant to detect severe disease; whilst the MSL-COVID-19

PLOS ONE
score is useful for quick clinical examination, updating these estimations with current clinical status improves its predictive capacity. Nutri-CoV shows excellent discriminative capacity for low-risk cases, with a low number of eventual false positives which makes it a good screening tool to predict disease severity and resource allocation. Onset of acute respiratory distress syndrome (ARDS) in COVID-19 patients can be attributable to underlying endothelial vascular injury which disrupts the regulation of pulmonary blood flow, leading to a ventilation-perfusion mismatch and eventually reducing oxygenation despite increased lung compliance and thrombogenesis [23]. Low oxygenation can be promptly and objectively assessed using pulse oximetry, allowing identification of COVID-19 related early hypoxia, particularly when no overt dyspnea is present [24]. This pattern of lung injury, related to decreased elastance in the setting of increased lung compliance has been described as Type L COVID pneumonia, which has different implications compared to type H COVID pneumonia, more traditionally linked to ARDS [24][25][26]. Hypoxemia usually leads to increase in minute ventilation by increasing tidal volume, which can be assessed as an increased respiratory rate despite absence of dyspnea in the setting or normal lung compliance. Notably, lung function is impaired in comorbidities linked to increased risk of severe COVID-19 including obesity and type 2 diabetes [27][28][29]; furthermore, most cardio-metabolic comorbidities have been linked to increased expression of the angiotensin converting enzyme (ACE) [30,31]. These factors are jointly evaluated by the Nutri-CoV and MSL-COVID-19 scores, thus indicating its potential to capture pathophysiological components that eventually lead to severe COVID-19 disease and increased mortality. Notably, although dyspnea was associated with increased disease severity, it was not an independent predictor severe disease, probably because of collinearity with variables already included such as pulse oximetry and respiratory rate. The use of our algorithm should accompany clinical judgement as the evaluation of any individual component of the score does not predict the outcome with absolute certainty.
Our study had some strengths and limitations. By assessing patients who were considered to have mild, moderate and severe COVID-19, we were able to capture a wide array of clinical characteristics at triage evaluation. Nevertheless, since all patients were examined from a single center, the possibility of referral or representability bias might reduce performance of Nutri-CoV in other populations. MSL-COVID-19 was developed using nationally representative data, the good performance shown within our institution, validates its implementation in low resource settings. A potential limitation of our approach is that pulse oximetry evaluation was performed in patients who are already on supplementary oxygen; whilst this was controlled in the statistical analysis and considered within the Elastic Net regression framework, the possibility of residual confounding is present. Considering the use of a training and a validation dataset as well as cross-validation for selection of penalization parameters, the model is likely to have adequate performance; however, our model is better suited for adequate performance within our institution or similar institutions within of Mexico and thus requires additional external validation to confirm its applicability in other settings. The web application should be useful to physicians within our institution and for external settings once it is externally validated and could be complementary to decision making in a triage setting.
In conclusion, we validated the MSL-COVID-19 score using data from a COVID-19 reference center in Mexico City. We demonstrated this score could be useful to predict other outcomes related to COVID-19 including ICU admission and requirement for invasive ventilation. Including clinical parameters in the MSL-COVID-19 score, related to COVID-19 ventilatory pathophysiology, increases its performance for prediction of severe COVID-19 and makes it a useful tool for clinical decision making and resource allocation during the healthcare stress of a pandemic scenario. Both the MSL-COVID-19 and Nutri-CoV scores were deployed as interactive webtools for clinical use in a triage setting.