https://orcid.org/0000-0001-8980-7860
Figures
Abstract
Introduction
The decision to intubate and ventilate a patient is mainly clinical. Both delaying intubation (when needed) and unnecessarily invasively ventilating (when it can be avoided) are harmful. We recently developed an algorithm predicting respiratory failure and invasive mechanical ventilation in COVID-19 patients. This is an internal validation study of this model, which also suggests a categorized “time-weighted” model.
Methods
We used a dataset of COVID-19 patients who were admitted to Rabin Medical Center after the algorithm was developed. We evaluated model performance in predicting ventilation, regarding the actual endpoint of each patient. We further categorized each patient into one of four categories, based on the strength of the prediction of ventilation over time. We evaluated this categorized model performance regarding the actual endpoint of each patient.
Citation: Statlender L, Shvartser L, Teppler S, Bendavid I, Kushinir S, Azullay R, et al. (2024) Predicting invasive mechanical ventilation in COVID 19 patients: A validation study. PLoS ONE 19(1): e0296386. https://doi.org/10.1371/journal.pone.0296386
Editor: Samuele Ceruti, Sant Anna Hospital: Clinica Sant’Anna, SWITZERLAND
Received: January 9, 2023; Accepted: December 12, 2023; Published: January 2, 2024
Copyright: © 2024 Statlender et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of Clalit Health Services policy. Data are available from the Rabin Medical Center Institutional Data Access & Ethics Committee for researchers who meet the criteria for access to confidential data. Data can be requested either through the corresponding author, or via Hagit Hendel, Rabin Medical Center CIO, by email at hagithe1@clalit.org.il.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Treatment of respiratory failure includes supplemental oxygen, non-invasive ventilation (NIV), or invasive mechanical ventilation (IMV) [1, 2]. The decision to intubate and invasively ventilate a patient is mainly clinical. A large body of evidence supports the early use of NIV and/or supplemental oxygen in various clinical situations [3] including hypoxemic respiratory failure [4], community-acquired pneumonia [5], and chronic obstructive pulmonary disease (COPD) exacerbation [6]. However, if the patient does not improve, and requires IMV later in his/her disease course, the prognosis might worsen [7–10]. Lately, it has been demonstrated that delaying intubation is harmful also for COVID-19 patients [11, 12]. Recent data regarding respiratory support for COVID-19 patients demonstrated lower risk of tracheal intubation in patients who were treated with continuous positive airway pressure (CPAP) [13], or with high flow nasal canula (HFNC) [14], both compared with low flow nasal cannula. No change was noted in mortality, nor in ventilator-free days. Thus it seems that initial treatment with either CPAP or HFNC is acceptable [15], as reflected in current guidelines [16]. However, a patient’s condition might deteriorate and require IMV, which might cause adverse effects [17], including (but not limited to) ventilator-associated events and ventilator-induced lung injury, and thus its indication should be optimized.
There are several reports regarding machine learning tools that were developed to predict clinical deterioration and need for IMV. A pre-pandemic report by Zeiberg et al. described a model that allows prediction of ARDS development and severity with an area under receiver operator curve (AUC) of 0.81–0.82 [18]. During the COVID-19 pandemic, other machine learning models were developed. Several models used a “snap-shot” of several factors (including comorbidities, vital signs, laboratory results, imaging results) taken at the emergency department or upon ICU admission. The models aim to predict deterioration to IMV, ICU admission, and/or mortality based on the “snap-shot” taken. In most of the models, the prediction relates to the duration of the patient’s hospitalization. The AUC of the models ranges between 0.68–0.94 [19–24]. Other models incorporated repeated measurements of several features (such as vital signs and laboratory results), for prediction of ICU admission with an AUC of 0.79–0.94 [25], IMV onset with an AUC of 0.84 [26], or mortality with an AUC of 0.95 [27].
We recently developed an algorithm to predict respiratory failure and IMV in critically ill COVID-19 hypoxemic patients [28]. The algorithm was developed as a decision-support tool. It is based on four slightly different models, each of which predicts ventilation based on a different calculation, according to its specific development process. The result is given as a risk for ventilation on a scale from 0 to 1. Any result greater than 0 is a positive prediction of IMV. Higher values suggest a higher risk for ventilation onset. The results of each model are updated on an hourly basis, as newer data accumulates (new vital signs and laboratory results are averaged if their measurement is less frequent than once an hour). The AUCs of the models vary from 0.91 to 0.97. Notably, most of the machine learning tools that were developed use only binary results as predictors [19–21, 23, 25–27].
As the Covid-19 pandemic progressed into its second year in 2021, we performed internal validation of the model, by retrospectively applying it to patients who were admitted to Rabin Medical Center with COVID-19 Between March 1st 2021 and February 23rd 2022. Furthermore, we constructed a set of risk-level categories, based on the accumulated hourly model results, trying to mimic a physician deciding whether to intubate. We evaluated these categories’ allocation in terms of intubation prediction.
Methods
Rabin Medical Center IRB approved this work (RMC-0473-21). Informed consent was waived as this is a retrospective study and patient data was anonymized.
Study population
We included all COVID-19 patients who were admitted to Rabin Medical Center hospital between March 1st 2021 and February 23rd 2022 who had hypoxemia (defined as pulse oximetry<88% or PaO2<60mmHg) and who were hospitalized for more than 48 hours. Patients with premature orders of do not resuscitate (DNR) or do not intubate (DNI) were included, as the algorithm predicts the medical need for ventilation, but serves only as a decision-support tool. Ventilated patients upon arrival (whether chronically ventilated or patients who were ventilated by the EMS) were excluded. Most of these patients were admitted to COVID-19-dedicated internal medicine wards. A small percentage were admitted to the ICU. Patients were treated based on COVID-19 treatment guidelines, mainly with steroids and anti-viral drugs as appropriate. Oxygen support was given to hypoxemic patients, mainly with low flow or high flow nasal canula, until an improvement in hypoxemia was noted.
Internal validation of the prediction model
The patient’s data was inserted for prediction into the machine learning models previously described [28]. A brief description of the model is presented here.
Identification of ventilation was achieved both with “operational features” (ventilation settings by physicians as documented in the database) and without them. XGBoost machine learning tool constructed two models (with & without the operational features), with two types of training: one was trained directly on the dataset of COVID-19 patients that were hospitalized in Rabin Medical Center between March 2020 and February 2021, and the other was first trained on MIMIC-III dataset and adapted to Rabin dataset. Hence, we developed four models overall: With/without operational features & with/without MIMIC-III training as the first stage. Therefore, the algorithm uses four different models to predict ventilation. The AUC of the models ranged between 0.91 and 0.97, as reported. Further details for this calculation are found in S1 File.
For internal validation, the algorithm (all four models) predicted the probability of ventilation for each unventilated patient. Prediction is made every hour, starting from the 12th hour of admission. The prediction is made for a future time frame of 4 hours, starting 6 hours from prediction time. We calculated the sensitivity, specificity, positive and negative predictive value, and AUC of each of the four models in relation to the actual endpoint, ventilation status of the patients. Confusion matrices were calculated for a false positive rate of 0.2, as was done in the original paper.
Categorized model development
As clinical decisions are often accepted after several observations across the timeline, we further categorized the models without operational features’ results into four categories. As each of the models might be categorized differently (according to each model’s result), we used the highest risk level from any of the models at any given hour of prediction (Fig 1):
- Zero prediction of ventilation: a patient with a consistent negative risk level of ventilation, or a patient whose highest risk level, at all time points was lower than 0.15.
- Weak prediction: a patient whose highest risk level, at any time point, was 0.15–0.5.
- Moderate prediction: (a) a patient whose highest risk level, at any time point was 0.5–0.8, or (b) a patient whose highest risk level was greater than 0.8 but didn’t fulfill criteria 4 below.
- Strong prediction: a patient whose risk level was consistently high during several hours of observation. Generally, it can be described as a patient who had (a) a single risk level of 1 at any timepoint, or (b) a risk level of 0.8 or greater for more than three hours and an average risk level greater than a threshold that depends on the number of hours with risk ≥0.8. S2 File provides further details regarding this calculation.
Patient a. Zero prediction of ventilation, Patient b. Weak prediction, Patient c. Moderate prediction, Patients d, e, f, g. Strong prediction.
We based the categorized model solely on the predictions of the models without operational features, as records of operational features during non-invasive ventilation were found not to be consistent.
Considering only strong prediction as a positive prediction of ventilation, we calculated the confusion matrix, sensitivity, specificity, and precision, according to the actual clinical course of the patient, whether ventilated or not, for this categorized model.
Since the algorithm was developed as a decision-support tool, when developing this categorized model we counted patients who were not ventilated but died during the hospitalization as if they were ventilated (as it is possible that ventilation was not done due to futility, or a patient’s and/or next-of-kin’s preferences). Practically meaning, we aimed to develop the algorithm to predict ventilation per se, leaving the decision of whether to ventilate or not, to the physician to discern based on the clinical scenario. Of the mentioned machine learning algorithms for COVID-19 patients, Gupta et al. developed a prediction model for a composite outcome including mortality and respiratory deterioration [20]. We could not find in the descriptions of other deterioration models a commentary on the specific situation of mortality without mechanical ventilation, nor a commentary on DNI/DNR status [18, 19, 23–26]. Interestingly, the mortality prediction models, do not mention DNI/DNR status either [21, 27].
Statistical analysis
All statistical analysis was done using Python [sklearn.metrics, scipy.stats]. For the internal validation positive and negative predictions were compared with the actual endpoint of each patient, whether ventilated or not. For the categorized model, positive and negative prediction were compared with the actual endpoint of each patient, whether ventilated/dead or not. Based on all observations, we constructed the confusion matrix, and calculated sensitivity, specificity, precision, and recall. This data allowed drawing an ROC, and its AUC was calculated. AUC p-values calculated using the one-sided Wilcoxon test comparing the ROC with the null hypothesis of the diagonal curve [29].
Results
Eight hundred eighty-one patients positive for SARS-CoV-2 with desaturation were admitted to Rabin Medical Center between March 1st 2021 and February 23rd 2022. 235 of the patients died during hospitalization; 96 of the patients were ventilated. Table 1 details patient demographic and clinical data from the first day of hospital admission; Table 2 groups patients based on their ventilation and mortality status.
The algorithm provided 116,655 hourly predictions. AUC was greater than 0.87 for all the models. Table 3 details AUCs, Youden’s index, sensitivity, and specificity for all models. Fig 2 illustrates the confusion matrix and ROC curves of all models.
(a) Trained on Rabin dataset, with operational features; (b) Trained on MIMIC-III dataset and adapted to Rabin dataset, with operational features; (c) Trained on Rabin dataset, without operational features; (d) trained on MIMIC-III dataset and adapted to Rabin dataset, without operational features.
For the categorized model the number of predictions was 881 (as each patient is categorized, it is equal to the number of included patients). Table 4 shows the confusion matrix of this model. Considering only category 4 (strong prediction) as positive, the sensitivity for ventilation onset was 0.76, specificity 0.972, positive predictive value 0.768, negative predictive value 0.971, and accuracy 0.949. Table 5 details the accuracy of the categories considered as a negative prediction of ventilation.
Sensitivity analysis
To analyze the stability of the results over the categorized model, we varied the high risk level threshold (0.7–0.9), average risk level threshold (0.4–0.6), and the minimal number of hours with high risk (2–4), and calculated all the characteristics of quality in all the combinations of input parameters (Table 6). A slight increase in accuracy and sensitivity was noted, along with a slight decrease in specificity. However, these changes are minimal, and the results are very stable with tight standard deviation and confidence intervals. Therefore, changing the chosen thresholds seems unnecessary.
Discussion
The clinical dilemma regarding invasively ventilating a patient with respiratory failure has no clear solution. Unnecessary delay in intubation might cause self-induced lung injury, and increased mortality [11], however unnecessary ventilation exposes the patient to the adverse effects of ventilation [17]. Several scores were developed trying to address this question, with variable levels of accuracy [30], and several machine learning models were developed to predict COVID-19 patients deterioration to IMV. Most of them use one measurement per feature, taken from a “singular” event (i.e., emergency department visit; first ICU admission day) with AUCs that ranged between 0.68–0.94 [19, 20, 22, 23]. Arvind et al. described a tool that uses multiple measurements per feature as time-series data, with an AUC of 0.84 for IMV prediction (given as a binary prediction) [26].
We have recently published a novel machine learning model, allowing to quantify the risk for IMV onset in hypoxemic COVID-19 patients, which is based on multiple time-series measurements, with AUC greater than 0.9 [28]. In the present study, we ran this model on a new dataset of COVID-19 patients, admitted to Rabin Medical Center, performing internal validation of the model. Development of most of the machine learning tools is based on dividing the patients into training and testing cohorts, with cross-validation of the cohorts [20, 22, 24–26]. Some studies performed prospective validation of the developed models [18, 21, 23].
The AUC of our validation is slightly lower compared to the original research (0.87–0.94 vs 0.91–0.97), Yet, it is still high and indicates accuracy. This AUC is higher compared to the AUC of other machine learning models aimed at predicting deterioration to mechanical ventilation (0.68 [23] & 0.81 [18]).
Transforming the model to the categorized model which incorporates several predictions along the timeline (a “time-weighted” prediction model) is an attempt to mimic clinical decision making. A decision regarding providing a specific treatment is based on the patient’s current status, but also on the clinical course until the present time. Different treatments are chosen when a patient deteriorates or improves. To our knowledge, this categorized model is unique among other machine learning prediction models. Some models base their prediction on several parameters from the day of the first encounter with the patient [18–24]. The model developed by Cheng, et al. allows daily prediction of ICU admission during a hospitalization [25]; Arvind et al. developed a model using a sliding-window approach allowing prediction of intubation within 72 hours of prediction time during hospitalization [26], principally similar to our algorithm [28]. The novel use of the algorithm for the prediction of ventilation onset maintains high accuracy (94.9%).
The categorized model was built without the operational features, as we considered these features might not be available in a prospective use of the model. This might have slightly decreased the accuracy of the categorized model (as the accuracy of the models with the operational features was higher than the models without the operational features). From a clinical point of view, the high accuracy of the categorized model is not high enough to serve as a decider. However, as a decision support tool, the model may have great implications. In the relevant clinical scenario, none of the examined hypoxemic patients can be discharged from medical evaluation. The high negative predictive value (97%) can reassure the physician that the medical course of the disease will very likely not mandate invasive ventilation. The discrimination between several categories of “negative prediction” allows further delineation, as the negative predictive value decreases with higher strength. It is possible that such patients should be hospitalized for observation (and treatment of the primary disease) in a less stressful and intensive setting than a critical care unit. Obviously, as long as respiratory deterioration is a matter of concern, observation should persist, and further evaluations of the model are possible. On the other hand, the positive predictive value is only 65%, meaning that approximately one-third of the patients for whom the model predicts invasive ventilation can be treated without it. It seems reasonable to hospitalize these patients in a critical care unit for further observation (and treatment). It must be emphasized again that the model’s prediction can be taken into consideration, but it is the patient’s characteristics that should dictate the physician’s decision.
This work has several limitations. First, this is a single-center study based on COVID-19 patients, and its generalizability is questionable; Second, this study is retrospective. Both external validation and prospective studies are required to validate this machine learning model to be applicable as a clinical decision support tool. Third, raising the threshold for positive prediction (as was done in the categorized model) decreased its sensitivity and positive predictive values. Interpretation of a positive result should be made cautiously. Fourth, in the categorized model, we considered mortality without ventilation as a positive case of deterioration. This decision might be challenged, especially regarding patients with “do not intubate/resuscitate” orders, or in those whom ventilation seems futile. We believe that this decision improves the model as a decision support tool. Counting mortality without ventilation as positive cases of deterioration allows the categorized model to better predict the clinical deterioration, but leaving the clinical decision itself for the physician to discern.
Conclusion
This is the formal internal validation for the models with a set of patients hospitalized with COVID-19 after the training/testing process of the algorithm [28]. A small degradation in AUC was noted, but AUC remained high and useful. The addition of time-weighted results maintained high accuracy and shows promising results for further research. External validation and prospective studies should be done.
Acknowledgments
The authors would like to thank Dr. Eyal Robinson for the English review of the manuscript.
References
- 1. Hess DR. Noninvasive Ventilation for Acute Respiratory Failure. Respiratory Care. 2013;58: 950–972. pmid:23709194
- 2. Munshi L, Mancebo J, Brochard LJ. Noninvasive Respiratory Support for Adults with Acute Respiratory Failure. Hardin CC, editor. N Engl J Med. 2022;387: 1688–1698. pmid:36322846
- 3. Tomii K, Seo R, Tachikawa R, Harada Y, Murase K, Kaji R, et al. Impact of noninvasive ventilation (NIV) trial for various types of acute respiratory failure in the emergency department; decreased mortality and use of the ICU. Respiratory Medicine. 2009;103: 67–73. pmid:18804357
- 4. Ferrer M, Esquinas A, Leon M, Gonzalez G, Alarcon A, Torres A. Noninvasive ventilation in severe hypoxemic respiratory failure: a randomized clinical trial. Am J Respir Crit Care Med. 2003;168: 1438–1444. pmid:14500259
- 5. Confalonieri M, Potena A, Carbone G, Porta RD, Tolley EA, Umberto Meduri G. Acute respiratory failure in patients with severe community-acquired pneumonia. A prospective randomized evaluation of noninvasive ventilation. Am J Respir Crit Care Med. 1999;160: 1585–1591. pmid:10556125
- 6. Bourke SC, Piraino T, Pisani L, Brochard L, Elliott MW. Beyond the guidelines for non-invasive ventilation in acute respiratory failure: implications for practice. Lancet Respir Med. 2018;6: 935–947. pmid:30629932
- 7. Ozyilmaz E, Ugurlu AO, Nava S. Timing of noninvasive ventilation failure: causes, risk factors, and potential remedies. BMC Pulm Med. 2014;14: 19. pmid:24520952
- 8. Mosier JM, Sakles JC, Whitmore SP, Hypes CD, Hallett DK, Hawbaker KE, et al. Failed noninvasive positive-pressure ventilation is associated with an increased risk of intubation-related complications. Ann Intensive Care. 2015;5: 4. pmid:25852964
- 9. Hraiech S, Alingrin J, Dizier S, Brunet J, Forel J-M, La Scola B, et al. Time to Intubation Is Associated with Outcome in Patients with Community-Acquired Pneumonia. Tang JW, editor.PLoS ONE. 2013;8: e74937. pmid:24069367
- 10. Kang BJ, Koh Y, Lim C-M, Huh JW, Baek S, Han M, et al. Failure of high-flow nasal cannula therapy may delay intubation and increase mortality. Intensive Care Med. 2015;41: 623–632. pmid:25691263
- 11. González J, Benítez ID, de Gonzalo-Calvo D, Torres G, de Batlle J, Gómez S, et al. Impact of time to intubation on mortality and pulmonary sequelae in critically ill patients with COVID-19: a prospective cohort study. Crit Care. 2022;26: 18. pmid:35012662
- 12. Le Terrier C, Suh N, Wozniak H, Boroli F, Giudicelli-Bailly A, Sangla F, et al. Delayed intubation is associated with mortality in patients with severe COVID-19: A single-centre observational study in Switzerland. Anaesthesia Critical Care & Pain Medicine. 2022;41: 101092. pmid:35487412
- 13. Perkins GD, Ji C, Connolly BA, Couper K, Lall R, Baillie JK, et al. Effect of Noninvasive Respiratory Strategies on Intubation or Mortality Among Patients With Acute Hypoxemic Respiratory Failure and COVID-19: The RECOVERY-RS Randomized Clinical Trial. JAMA. 2022;327: 546. pmid:35072713
- 14. Frat J-P, Quenot J-P, Badie J, Coudroy R, Guitton C, Ehrmann S, et al. Effect of High-Flow Nasal Cannula Oxygen vs Standard Oxygen Therapy on Mortality in Patients With Respiratory Failure Due to COVID-19: The SOHO-COVID Randomized Clinical Trial. JAMA. 2022;328: 1212. pmid:36166027
- 15. Nichol AD O’Kane C, McAuley DF. Respiratory Support in the Time of COVID-19. JAMA. 2022;328: 1203.
- 16.
COVID-19 Treatment Guidelines Panel. Coronavirus Disease 2019 (COVID-19) Treatment Guidelines. National Institutes of Health. Available at https://www.covid19treatmentguidelines.nih.gov/. Accessed July 12th 2023. [cited 12 Jul 2023]. Available: https://www.covid19treatmentguidelines.nih.gov/
- 17. Tonetti T, Vasques F, Rapetti F, Maiolo G, Collino F, Romitti F, et al. Driving pressure and mechanical power: new targets for VILI prevention. Ann Transl Med. 2017;5: 286–286. pmid:28828361
- 18. Zeiberg D, Prahlad T, Nallamothu BK, Iwashyna TJ, Wiens J, Sjoding MW. Machine learning for patient risk stratification for acute respiratory distress syndrome. PLoS One. 2019;14: e0214465. pmid:30921400
- 19. Karri R, Chen Y-PP, Burrell AJC, Penny-Dimri JC, Broadley T, Trapani T, et al. Machine learning predicts the short-term requirement for invasive ventilation among Australian critically ill COVID-19 patients. Rashid TA, editor. PLoS ONE. 2022;17: e0276509. pmid:36288359
- 20. Gupta RK, Harrison EM, Ho A, Docherty AB, Knight SR, van Smeden M, et al. Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study. The Lancet Respiratory Medicine. 2021;9: 349–359. pmid:33444539
- 21. Yadaw AS, Li Y, Bose S, Iyengar R, Bunyavanich S, Pandey G. Clinical features of COVID-19 mortality: development and validation of a clinical prediction model. The Lancet Digital Health. 2020;2: e516–e525. pmid:32984797
- 22. Wu G, Yang P, Xie Y, Woodruff HC, Rao X, Guiot J, et al. Development of a Clinical Decision Support System for Severity Risk Prediction and Triage of COVID-19 Patients at Hospital Admission: an International Multicenter Study. Eur Respir J. 2020; 2001104.
- 23. Yu L, Halalau A, Dalal B, Abbas AE, Ivascu F, Amin M, et al. Machine learning methods to predict mechanical ventilation and mortality in patients with COVID-19. Chen RJ, editor.PLoS ONE. 2021;16: e0249285. pmid:33793600
- 24. Heldt FS, Vizcaychipi MP, Peacock S, Cinelli M, McLachlan L, Andreotti F, et al. Early risk assessment for COVID-19 patients from emergency department data using machine learning. Sci Rep. 2021;11: 4200. pmid:33603086
- 25. Cheng F-Y, Joshi H, Tandon P, Freeman R, Reich DL, Mazumdar M, et al. Using Machine Learning to Predict ICU Transfer in Hospitalized COVID-19 Patients. JCM. 2020;9: 1668. pmid:32492874
- 26. Arvind V, Kim JS, Cho BH, Geng E, Cho SK. Development of a machine learning algorithm to predict intubation among hospitalized patients with COVID-19. Journal of Critical Care. 2021;62: 25–30. pmid:33238219
- 27. Yan L, Zhang H-T, Goncalves J, Xiao Y, Wang M, Guo Y, et al. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell. 2020;2: 283–288.
- 28. Bendavid I, Statlender L, Shvartser L, Teppler S, Azullay R, Sapir R, et al. A novel machine learning model to predict respiratory failure and invasive mechanical ventilation in critically ill patients suffering from COVID-19. Sci Rep. 2022;12: 10573. pmid:35732690
- 29. Mason SJ, Graham NE. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Q J R Meteorol Soc. 2002;128: 2145–2166.
- 30. Duan J, Han X, Bai L, Zhou L, Huang S. Assessment of heart rate, acidosis, consciousness, oxygenation, and respiratory rate to predict noninvasive ventilation failure in hypoxemic patients. Intensive Care Med. 2017;43: 192–199. pmid:27812731