Development of a brief scoring system to predict any-cause mortality in patients hospitalized with COVID-19 infection

Nasheena Jiwa; Rahul Mutneja; Lucie Henry; Garrett Fiscus; Richard Zu Wallack

doi:10.1371/journal.pone.0254580

Peer Review History

Original SubmissionMarch 2, 2021
5 May 2021 Decision Letter - Aleksandar R. Zivkovic, Editor PONE-D-21-06850 Development of a Brief Scoring System to Predict Any-Cause Mortality in Patients Hospitalized with COVID-19 Infection PLOS ONE Dear Dr. Jiwa, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Jun 19 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Aleksandar R. Zivkovic Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1) Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2) Thank you for including your ethics statement: "Institutional Board approval was obtained prior to study initiation." Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study. Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”). For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research. 3) PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ 4) Thank you for stating the following financial disclosure: [NO]. At this time, please address the following queries: Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution. State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.” If any authors received a salary from any of your funders, please state which authors and which funders. If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.” Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 5) Thank you for stating the following in your Competing Interests section: [NO]. Please complete your Competing Interests on the online submission form to state any Competing Interests. If you have no competing interests, please state "The authors have declared that no competing interests exist.", as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now This information should be included in your cover letter; we will change the online submission form on your behalf. Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests 6) Please ensure that you refer to Figure 1 in your text as, if accepted, production will need this reference to link the reader to the figure. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer #1: Review “Development of a brief scoring system to predict any-cause mortality in patients hospitalized with COVID-19 infection” First of all, I would like to thank you for the opportunity to read and review your manuscript. I have recently also been involved in research into prediction of short-term mortality in patients with COVID-19. I find the subject of your manuscript very interesting, but there are some mayor aspects that still deserve attention. Please address my comments and suggestions I have listed below. General remarks • As for the methodological aspects of the manuscript, I have some mayor concerns regarding the patient selection (see below). • Abbreviations are not always sufficiently explained throughout the manuscript. For example, in the abstract, the abbreviation AUC is not spelled out. In a general sense, any abbreviation must be written out the first time it is used. Abstract I believe it would be better to present the abstract in the fixed structure of Introduction (or background), Methods, Results and Conclusion). Introduction • In line 3, the word “striking” is used twice in once sentence. I would suggest that it is removed the first time. • The recent systematic review by Wynants et al (Reference 7) describes a relatively large number of (diagnostic) and prognostic prediction models for patients with COVID-19. In a recent publication (Performance of prediction models for short-term outcome in COVID-19 patients in the emergency department: a retrospective study. Annals of Medicine 2021;53(1):402-209), we analyzed and externally validated 11 prediction models. In the introduction of your manuscript it should be made clear why there is a need for another prediction model (i.e. what does this model add?). Methods • I have some mayor concerns about the selection of patients. Out of approximately 1600 patients, you selected 100 patients for the derivation cohort and 148 patients for the validation cohort (i.e. 1450 patients were not analyzed). In what way were the patients selected? How can you rule out selection bias? I think this is a very important aspect of your manuscript. Why only develop the scoring systems in 100 patients (and not more), and why validate the scores in only 148 patients (and not more)? • You performed a retrospective study with two components. For the first component, a sample of 100 patients (out of approximately 1600 patients) was used to develop two scoring systems. In de the second component, those two scoring systems were tested. Instead of naming these samples “sample 1” and “sample 2”, I believe it would be better to refer to these samples as derivation cohort and validation cohort. • In line 5, you state that the scoring systems were tested prospectively. I understand the scoring systems are tested (validated) on retrospective data. The word “prospectively” is confusing and should be avoided here. • In what setting were the patients included? Emergency department or other? • Page 4: for readers not living in the USA (like myself), the surrogate marker for low SES should be explained in more detail (Medicaid, dual Medicare-Medicaid). • Page 6: the abbreviation ABR should be changed to ARB (Angiotensin receptor blocker). • The section on statistical analysis should have its own subheading. Results • As mentioned above, I would suggest the two study samples are referred to as derivation cohort and validation cohort. • The text belonging to subheadings “Sample 1” and “Sample 2” are actually a summary of the data from Table 1. Consider merging the two subheading into one (e.g. “Study samples”) and shortening the text to just the most important data. • Page 6 (Subheading “Sample 2”): in this section you described why you ended up with 148 patients in the validation cohort. I believe this explanation should be moved to the Methods section. In my opinion, you should also clarify whether different patients were analyzed in both cohorts, or whether the two cohort show an overlap. The reason for inclusion or exclusion of patients in the cohorts could also be explained in further detail. • It would greatly improve your manuscript if you could show the patient characteristics of the total population of 1600 patients. This allows the reader to estimate any selection bias. I would suggest adding an extra Table with this information. • Page 6: The subheading “Creating in-hospital mortality prognostic scoring models from Sample 1 cohort” is too long. It could be changed to “Selection of predictors and univariate and multivariate logistic regression analysis”. • Page 7: In line 4, “Hispanic mortality” is misspelled as “Hispanic morality”. The point after 40% should be removed. • Page 8: The subheading “Testing the models using Sample 2 data” could be changed to “Validation of the scoring systems”. • Page 8: How are the two ROC curves compared? This statistical analysis should be explained in more details in the statistical analysis section. • Page 8: Model A could be used in only 107 of 148 patients, because of missing data regarding troponin (27%). Since troponin is not routinely measured, is model A feasible in clinical practice? Discussion • Page 9: You recommend the use of Model B in patients with COVID-19 for predicting in-hospital mortality. However, your analysis shows no statistically significant difference between model A and B (again: the method of comparing the two models is unclear). So, why not use model A? • Page 9: What are the clinical consequences of the use of your scoring systems? In other words, should a low or a high score guide clinical decision-making? • Page 9: The sentence “For reference, an AUC between … considered excellent” belongs to the Methods section and should not be part of the Discussion section. • Page 10: Before being able to compare your results with those from other studies, it is important to estimate the degree of selection bias. See previous comments. • Page 10-11: In addition to the British 4C mortality score (Knight and colleagues), the RISE UP score is also very useful for predicting short term adverse outcome in ED patients with COVID-19 (Annals of Medicine 2021;53(1):402-209 and BMJ Open 2021;11:e045141). [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0254580.r001
Revision 1
27 Jun 2021 Author Response Dear PLOS ONE Editorial Staff, We want to thank you and your reviewers for your thorough review. The text below outlines the reviewers’ comments/questions and our responses. We wish to re-submit the revised manuscript for consideration for publication. Sincerely, Nasheena Jiwa, MD Journal Requirements: Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The Trinity Health of New England Institutional Review Board granted approval prior to study initiation Corresponding Author’s ORCID iD: https://orcid.org/0000-0003-0261-2425 The authors received no specific funding for this work. The authors have declared that no competing interests exist. Please ensure that you refer to Figure 1 in your text as, if accepted, production will need this reference to link the reader to the figure. Figure 1 is now referenced in the text in the last paragraph of the Discussion. Reviewer Comments: General Remarks - First occurrence of area under the curve (AUC) is now spelled out in the Methods section Abstract: This was revised to reflect the comments by the reviewer: Abstract Patients hospitalized with COVID-19 infection are at a high general risk for in-hospital mortality. A simple and easy-to-use model for predicting mortality based on data readily available to clinicians in the first 24 hours of hospital admission might be useful in directing scarce medical and personnel resources toward those patients at greater risk of dying. With this goal in mind, we evaluated factors predictive of in-hospital mortality in a random sample of 100 patients (derivation cohort) hospitalized for COVID-19 at our institution in April and May, 2020 and created potential models to test in a second random sample of 148 patients (validation cohort) hospitalized for the same disease over the same time period in the same institution. Two models (Model A: two variables, presence of pneumonia and ischemia); (Model B: three variables, age > 65 years, supplemental oxygen ≥ 4 L/min, and C-reactive protein (CRP) > 10 mg/L) were selected and tested in the validation cohort. Model B appeared the better of the two, with an AUC in receiver operating characteristic curve analysis of 0.74 versus 0.65 in Model A, but the AUC differences were not significant (p = 0.24. Model B also appeared to have a more robust separation of mortality between the lowest (none of the three variables present) and highest (all three variables present) scores at 0% and 71%, respectively. These brief scoring systems may prove to be useful to clinicians in assigning mortality risk in hospitalized patients. Introduction: - Deleted “striking” - Our prediction model is intended to be a used as a tool for quick interpretation of patient data within the first 24 hours, to predict mortality outcomes Methods: In what way were the patients selected? How can you rule out selection bias? I think this is a very important aspect of your manuscript. Why only develop the scoring systems in 100 patients (and not more), and why validate the scores in only 148 patients (and not more)? Patients in both samples were randomly selected from the pool of patients hospitalized with COVID-19. This is stated in Methods as follows: “For the first component of the study, we reviewed in-hospital records from a randomized sample of 100 COVID-19 patients (out of approximately 1600 records), who had been admitted to our tertiary-care center in Hartford, CT, during the months of April and May, 2020. These patients are known as the derivation cohort. All patient’s had been admitted and hospitalized with a clinical diagnosis and serological confirmation of COVID-19 infection. “A second sample of different patients called the validation cohort of patients was then reviewed. These records were randomly selected from the same population of hospitalized patients in April and May, 2020 (excluding those in the original sample), until an arbitrary number of at least 40 in-hospital deaths from COVID-19 were reviewed.” Because of limitations in research staff (this study was not externally funded) we could not study a larger sample. However, it should be noted that we did achieve statistical significance in several predictor variables that also appear mechanistically sound. Corrected sample 1 = derivation cohort, and sample 2 = validation cohort Removed the word “prospectively” Patient’s included were admitted to the hospital Explained in detail: low SES (Medicaid, dual Medicare-Medicaid). abbreviation ABR was changed to ARB (Angiotensin receptor blocker). Added a statistical analysis subheading Results Corrected sample 1 = derivation cohort, and sample 2 = validation cohort Description of sample 2/validation cohort of patients was moved to methods section Clarified that both cohorts contained different patients It would greatly improve your manuscript if you could show the patient characteristics of the total population of 1600 patients. This allows the reader to estimate any selection bias. I would suggest adding an extra Table with this information. As we stated earlier, we did not have the research personnel to do a chart review on all 1600 patients. However, we could note that the two random samples were similar with respect to patient characteristics, as given in Table 1. The subheading “Creating in-hospital mortality prognostic scoring models from Sample 1 cohort” was changed to “Selection of predictors and univariate and multivariate logistic regression analysis” Corrected: “Hispanic mortality” is misspelled as “Hispanic morality”. “Testing the models using Sample 2 data” was changed to “Validation of the scoring systems”. How are the two ROC curves compared? This statistical analysis should be explained in more details in the statistical analysis section. We clarified this with the following added text: “ROC curves were compared using the SAS logistic procedure and the ROCCONTRAST statement.” - Model A could be used in only 107 of 148 patients, because of missing data regarding troponin (27%). Since troponin is not routinely measured, is model A feasible in clinical practice? We changed the text in the Discussion to reflect this appropriate comment as follows: “Based on performance in the validation cohort, Model B had a slightly higher AUC (0.74 vs. 0.65), although the difference between the two models was not statistically significant. Model B also tended to perform better with respect to mortality separation in logistic regression. Given the small numbers of subjects in our study, the lack of a statistically significant difference in AUC, and the fact that CRP (a component of model B) was not obtained in all patients upon admission, a strong inference on the comparative performance of the two scoring systems would be problematic.” Discussion: You recommend the use of Model B in patients with COVID-19 for predicting in-hospital mortality. However, your analysis shows no statistically significant difference between model A and B (again: the method of comparing the two models is unclear). So, why not use model A? This was addressed with the changes in text above in the Discussion: What are the clinical consequences of the use of your scoring systems? In other words, should a low or a high score guide clinical decision-making The text in the Discussion was changed to reflect the uncertainty of the clinical consequences: “Our study should be viewed as presenting preliminary data which would be needed to be followed by external validation from analysis of a larger dataset across different institutions and over a longer time range, not only to attempt to replicate the findings of the predictive models, but also to determine if the scoring results in meaningful clinical consequences.” Removed “For reference, an AUC between … considered excellent” from the discussion section and moved it to the Methods section Page 10: Before being able to compare your results with those from other studies, it is important to estimate the degree of selection bias. See previous comments. We added the following disclaimer in the Discussion: “Comparing our results with those from other studies of in-hospital mortality is problematic for several reasons, including potential selection biases among the studies, expected regional differences in patient demographics and treatment approaches, and changes over time in therapeutic modalities for this disease.” Page 10-11: In addition to the British 4C mortality score (Knight and colleagues), the RISE UP score is also very useful for predicting short term adverse outcome in ED patients with COVID-19 (Annals of Medicine 2021;53(1):402-209 and BMJ Open 2021;11:e045141). These studies were added to the Discussion and referenced. Thank you. “A retrospective study of 403 adult patients seen in the Emergency Department in a combined secondary/tertiary care center in the Netherlands for the first wave of the pandemic (March through May, 2020) tested 11 prediction models of 30-day mortality as the primary outcome. (18) The investigators identified two prediction models that performed best: 1) RISE-UP (acronym for Risk Stratification in the Emergency Department in Acutely Ill Older Patients) score, which included age, heart rate, mean arterial pressure, respiratory rate, oxygen saturation, Glasgow Coma Scale (GCS), BUN, bilirubin, albumin, and lactate dehydrogenase; and 2) 4-C (Coronavirus Clinical Characterisation Consortium) score, which had been tested previously in the United Kingdom, (19) and included age, sex, co-morbidity, RR, GCS, O2 saturation, BUN, and CRP. With an AUC of 0.83 and 0.84, respectively, both preformed better than ours, but were obviously more complicated in that they required entering more variables.” Attachments Attachment Submitted filename: Response to Reviewers 6-7-21 (1).docx https://doi.org/10.1371/journal.pone.0254580.r002
30 Jun 2021 Decision Letter - Aleksandar R. Zivkovic, Editor Development of a Brief Scoring System to Predict Any-Cause Mortality in Patients Hospitalized with COVID-19 Infection PONE-D-21-06850R1 Dear Dr. Jiwa, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Aleksandar R. Zivkovic Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0254580.r003
Formally Accepted
8 Jul 2021 Acceptance Letter - Aleksandar R. Zivkovic, Editor PONE-D-21-06850R1 Development of a Brief Scoring System to Predict Any-Cause Mortality in Patients Hospitalized with COVID-19 Infection Dear Dr. Jiwa: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Aleksandar R. Zivkovic Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0254580.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .