PONE-D-20-35733

DEVELOPMENT AND VALIDATION OF A PREDICTIVE MODEL OF IN-HOSPITAL MORTALITY IN COVID-19 PATIENTS

PLOS ONE

Dear Dr. Alonso-Dominguez,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Feb 04 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Aleksandar R. Zivkovic

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In the ethics statement in the manuscript and in the online submission form, please provide additional information about the patient records/samples used in your retrospective study, including the date range (month and year) during which patients' medical records/samples were accessed.

3. In your ethics statement in the manuscript and in the online submission form, please ensure that you have discussed whether all data/samples were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data/samples from their medical records used in research, please include this information.

4.In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

5.Thank you for stating the following in the Financial Disclosure section:

"The author(s) received no specific funding for this work."

We note that one or more of the authors are employed by a commercial company: Quironsalud

1. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.  

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In this retrospective observational multicenter study, the authors developed and validated a predictive model for in-hospital mortality of moderate and severe COVID-19 patients, using demographic data, medical history and laboratory parameters. Overall, the study was well executed and the authors achieved its clearly defined objectives. The manuscript was concise, easy to understand and a good read. It contributes to the existing literature on prognostication of outcomes in COVID-19 patients. Following are listed some comments and points to be addressed:

Major issues:

---------------

1) The authors states (p. 19, l. 319) that a predictive model with a high negative predictive value is desirable, since this allows to identify low-risk patients which may be discharged at an earlier stage, and thereby relieve the health care system. However, since the data from the present study is based on patients (both survivors and non-survivors) receiving full treatment, I could fear that mortality will increase in the low-risk patients if they are discharged early solely based on their risk score. The authors should emphasize the need for prospective studies to back up this statement in the conclusions.

2) Was there a sample size analysis or was this a sample of convenience?

Minor issues:

---------------

1) The authors should either combine the first two sentences or rewrite the first sentence in the abstract (p. 2, l. 85-88).

2) In the statistical analysis section (p. 6, l. 190-192), the authors state: “The results were expressed as the mean ± standard deviation and range or number (percentage), wherever appropriate. P values less than 0.05 were considered statistically significant.” It would be more informative and logic to present data as mean ± standard deviation if normal distributed, and as median (25-75 percentiles) if skewed, and numbers (percentage). Even though the authors states that range is listed, I think the authors have instead presented minimum and maximum values, which is of less importance. Also, it should be stated if the P values used are one- or two-sided.

3) Table 1 is very long and difficult to overlook. I would suggest removing ranges, and instead list mean ± SD if normal distributed, median (25-75 percentiles) if skewed, and number (percentage). In the column listing OR (95% CI), please type reference group in each of the univariate analyses involving categorical data, starting with the reference group. It is not clear which variable is associated with a higher OR (e.g. going from male to female, or female to male). Missing data of the individual parameters should be listed as a foot note. Also, the category “hospital stay” is somehow relevant and informative but also misleading, since non-survivors are discharged to the morgue. The authors should remove this parameter in Table 1, and carefully mention the results in the Results section. Finally, “sex” is used in the table, but “gender” is used in the text. Please be consistent.

4) Results are listed in details both in the tables and in the Results section. Data should generally only be listed either in the tables or in the text. Please refine manuscript.

5) Serum ferritin have in several studies been found valuable to determine poor prognosis in COVID-19 patients. The authors state (p. 14, l. 259); “No significant differences were found in categorized ferritin”. Looking at Table 1, non-survivors had significantly higher serum ferritin levels, but when categorized according to elevated ferritin (yes/no), this is no longer significantly different between groups. This detail should be added to the text. Also, please define the term “elevated ferritin” in the Methods section.

6) The authors used area under the ROC curve to evaluate both the development model and the validation model. Even though this is a generally accepted method in medical literature, the AUC only describe the discriminative ability of the model, i.e. correctly allocate patients as survivor or non-survivors according the model. However, adding a statistical analysis to test the predictive ability, e.g. Brier scores, could provide additional strength to the study.

Reviewer #2: Dear Authors;

I have read the manuscript titled “DEVELOPMENT AND VALIDATION OF A PREDICTIVE MODEL OF IN-HOSPITAL MORTALITY IN COVID-19 PATIENTS”. This study retrospectively evaluated 2879 hospitalized COVID-19 patients from four hospitals. Although it is a nice designed study there are several limitations.

1. There are more than 50 studies investigating the in hospital or 7-14 days mortality in patients with COVID-19 infection. Therefore (unfortunately) this manuscript doesn’t add something new to the literature.

2. This new predictive model has a sensitivity of 95.83%, specificity of 41.46%, negative predictive value of 98.01%, and positive predictive value of 24.85%. The authors explain these results however for a predictive model specificity is quite low.

3. doi: https://doi.org/10.1136/bmj.m1328 is a nice review and summary of literature why these predictive models have bias.

4. One selection bias is the inclusion criteria which CURB-65 score is used. Literature showed that CALL or other scores may be used instead of CURB-65 for covid pneumonia.

5. The authors found no difference in lymphocyte count between deceased and survivors. An interesting finding because in most studies lymphocyte count is a predictive factor for mortality.

Thank you.

Reviewer #3: Thank you for the opportunity to review this manuscript, which focusses on an important topic and addresses the important issue of risk stratification for patients with suspected and confirmed Covid-19 infection. With regards to the model presented, I am encouraged to see a relatively large sample size and contemporary data from patients across several centres.

However, I have serious concerns about a number of methodological flaws with the development and internal validation of the model that make it impossible for me to recommend publication at this time:-

1. Inclusion of predictors. Predictors were initially selected to be included in the logistic regression model if a significant association was found between the predictor and the outcome on univariable analysis. This is recognised as an unsuitable approach and may lead to important predictors being excluded. [1] All predictors deemed to be of clinical significance should be included in the logistic regression analysis, regardless of the univariable statistical analysis.

2. Sample size. Although 2070 patients were included in the study, no formal statistical sample size analysis has been undertaken. [2] This is an important step to ensure that the results of the model development can be considered statistically robust.

3. Dichotomisation/categorisation of continuous variables. Transformation of continuous variables into non-continuous variables can reduce the power by the same amount as discarding approximately 1/3 of the data and should be avoided unless absolutely necessary. For variables with recognised cut-offs (i.e. haemoglobin values for anaemia) it can be an acceptable practice, but in this model, you have dichotomised four variables including age. I consider this a serious methodological flaw. [3]

4. Measures of model performance. Whilst you have provided information regarding the discrimination of the model, there are no reported measures of calibration. Whilst discrimination is important in terms of assessing how well the model discriminates between those with and without the outcome calibration is a measure of how closely model predicted outcomes match observed outcomes. Some measure of calibration (observed to expected ratio, calibration plot, calibration slope, calibration-in-the-large) must be included when presenting a model. [4]

5. Internal validation. Whilst an attempt to perform internal validation has been undertaken, the methodological approach is not appropriate. Using a split sample approach (splitting the population into a development and a validation set) is no longer recognised as an acceptable method of internal validation due to a number of drawbacks. Internal validation should be performed using cross-validation or bootstrapping

I think that the multicentre dataset and important nature of the question being asked is worthy of further investigation. However, I think that the entire methodological approach to model development needs to be re-assessed and a new analysis performed. A new paper should be submitted in the future if you are able to do this.

References

[1] Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ 2009;338: b604.

[2] BMJ 2020;368:m441 doi: 10.1136/bmj.m441

[3] Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 2006;25:127–41.

[4] Grant SW, Collins GS, Nashef SAM. Statistical Primer: developing and validating a risk prediction model. Eur J Cardiothorac Surg 2018;54:203–8

[5] Altman DG, Vergouwe Y, Royston P, Moons KGM. Prognosis and prog nostic research: validating a prognostic model. BMJ 2009;338:b605.

Reviewer #4: It is an important topic and interesstung as well. The authors provide a possible predictors of COVID-19 in-Hospital Mortality.

However the current work suffer many Major issues.

Good idea of your work. Relevant study. Introduction of your paper states not clearly the problem but underlines your study purpose.

Material and methods leave behind unanswered questions.

Results are not clearly structured and the table is way too long and confusing

In the discussion you go more into the background of your study (which could be partly mentioned in your introduction). You advice a new prediction tool for in hospital mortality. However

you did a validation in the same paper without background information of the idependent patient sample. ???? that is very much questionable for me….

Major issues:

1. what is the Definition of moderat to Severe COVID-19 Patients?

2. Lines 221 A total of 2879 moderate to severe COVID-19 hospitalized patients were initially

lines 222 evaluated for inclusion in the development cohort. Of these, 809 were excluded: 515

line 223 remained hospitalized at the time of analysis and 294 were on chronic anticoagulant

line 224 treatment before hospitalization. The final sample consisted of 2070 patients (884

line 225 females and 1186 males) with definite outcomes: 1677 (81.01%) patients had been

line 226 discharged (survivors) and 393 (18.99%) patients had died (non-survivors).

Exclusion of a large group of patients. Approximately 28,1 % of patient were excluded.

Follow up time is not clear. (or time to mortality.)?

3. it is not clear how many patients were mechanical ventilated?

4.lines 262- 264 " They were then examined in a multivariate logistic regression model including 1270 patients to identify independent prognostic factors of moderate/severe COVID-19 in-hospital mortality (Table 2)."

Not clear. Sub-analysis of 2070 patients? Why are the 800 patients excluded from this Analysis.

Is that the independent sample of patients about which we read in the abstract.? Not clear.

IF Yes: no characteristics of this patient population present.

5. Line 283 A cut-off of 0.076 was arbitrarily selected.

Please elaborate more why this cut off value and not another ?

6. If a major Revision is possible, the analysis must be checked by a statistician

Minor

1. The language need to be re-edited, there are many typos as well as weak sentences: lines 85-86 To evaluate the risk of demographic data, medical history, and on-admission laboratory parameters in hospitalized COVID-19 patients to predict mortality

(Not well written)

Lines 133-135 Clinicians need better predictors of which patients are prone to deteriorate rapidly or who may go on to die

(“predict Mortality” better) not go die!!!!

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Sebastian Roed Rasmussen

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Development and validation of a predictive model of in-hospital mortality in COVID-19 patients

PONE-D-20-35733R1

Dear Dr. Alonso-Dominguez,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Aleksandar R. Zivkovic

Academic Editor

PLOS ONE