PDIG-D-21-00052

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

PLOS Digital Health

Dear Dr. Singh,

Thank you for submitting your manuscript to PLOS Digital Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Digital Health’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Thanks for your submission. Overall, the comments from our reviewers are very positive. The study tackles an interesting topic and it is well implemented and clearly described. Some issues are noted in the reviews, all of which appear relatively minor. I would be grateful if you could resubmit a revised version of the manuscript after responding to the points raised. Thanks!

==============================

Please submit your revised manuscript by Nov 28 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at digitalhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pdig/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Tom J. Pollard, Ph.D.

Academic Editor

PLOS Digital Health

Journal Requirements:

1. Please expand the acronym “NSF” (as indicated in your Financial Disclosure) so that it states the name of your funders in full.

2. Please provide separate figure files in .tif or .eps format only, and remove any figures embedded in your manuscript file. If you are using LaTeX, you do not need to remove embedded figures.

https://journals.plos.org/digitalhealth/s/figures

3. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does this manuscript meet PLOS Digital Health’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Digital Health does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This study explores the generalizability of training models on one set of data, from a particular hospital or region, and using that model for mortality risk prediction on a different hospital or region. The topic explored here is very relevant to the state of science in this field. I support the effort to develop a better understanding of what contributes to a lack of generalizability from training models in one area and applying them in another. This will help develop better models in the future. However, I do think there are some areas which need to be addressed.

1. Training models from a given region/hospital and then testing those models on a different hospital/region is certainly a good way to explore generalizability. However, it may also be interesting to train on all of the data and then check to see how that generalizes to specific hospitals/regions. Was this approach considered as part of the study design?

2. In the S1 Table, for the electivesurgery variable, in nine of the ten hospitals false is much more common than true but for Hospital ID 458 the opposite it true. Is the result shown for this hospital correct?

3. In the S2 Table, the median age of patients is exactly the same across all four regions. Is this correct?

4. On line #245, the text mentions changes in AUC and CS but numerical results are only provided for AUC. I’d recommend showing the numerical result for CS also since it is mentioned in the text alongside AUC.

5. On line #290, the text references Figure S3 but does not explain what it is showing. If a figure is referenced in the results section it should be accompanied by a brief explanation about what it is showing.

6. On line #308, the term “high-quality care” is used when discussing the direct effects of race on elective surgery. It isn’t clear to me that quality of care is directly related to whether or not elective surgery is chosen. Perhaps this should be explained further or another term should be used for the effect of race on elective surgery.

7. Finally, I’d encourage double-checking the manuscript for grammatical errors. I noticed a number of sentences which were missing words.

Reviewer #2: In this paper, a multi-center cross-sectional study was performed using data electronic health records from eICU database to predict mortality risk, considering the generalizability challenge. A causal discovery algorithm achieved good AUC performance. Other performance metrics which are useful to analyze modeling results were reported. The authors found that the distribution of demographic, vitals and laboratory variables differed significantly across hospitals and regions and that Race mediated differences in the relationship between clinical variables and mortality, by hospital/region.

The paper is in general well written and the findings are interesting.

I only have minor comments.

In the introduction, the sentence in line 73 should be re-written for a clear understanding.

Did the authors consider other missing data imputation techniques for the model covariates?

The values of the covariates in the training dataset should be used as reference to apply the imputation mean in the test set, otherwise results might be biased. The authors mention that the mean is applied to data, in line 153, but do not specify this detail. The difference in mean and standard deviation between the train and the full datasets might be small and not cause an impact on the normalization results, but this should be checked.

In line 166 correct “a CS”.

In the paragraph correspondent to line 216, the authors seem to be describing “bootstrapping”.

When referring to the variables such as the example "urineoutput" in the main text, the authors may write "urine output".

Reviewer #3: Excellent work. The analysis of models scoring metrics is well presented. The research clearly highlights the current imbalance between achieveing ever-higher AUCs despite negletcing overfitting and the generalization ability on different cohorts. The use of causal inference estimation algorithms supports the hypothesis proposed by the authors while helping the reader further grasping the nature of the problem.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PDIG-D-21-00052R1

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

PLOS Digital Health

Dear Dr. Singh,

Thank you for submitting your manuscript to PLOS Digital Health. After careful consideration, we feel that it has merit but does not fully meet PLOS Digital Health's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Mar 19 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at digitalhealth@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pdig/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Harry Hochheiser

Section Editor

PLOS Digital Health

Journal Requirements:

1. Please update your Competing Interests statement. If you have no competing interests to declare, please state: “The authors have declared that no competing interests exist.”

2. We have noticed that you have uploaded supporting information but you have not included a list of legends. Please add a full list of legends for all supporting information files (including figures, table and data files) after the references list.

3. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article's retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

Thanks for your thoughtful response to the comments from our reviewers. The study offers helpful insights into the important question of model generalizability. I have one minor request/comment. The significant skew in the data might be a bit of an issue. Ideally, analyses might include either upsampling or downsampling. Discussion of the potential impact of the skew - either in the discussion or as a limitation - would strengthen this paper.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database

PDIG-D-21-00052R2

Dear Mr. Singh,

We are pleased to inform you that your manuscript 'Generalizability Challenges of Mortality Risk Prediction Models: A Retrospective Analysis on a Multi-center Database' has been provisionally accepted for publication in PLOS Digital Health.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow-up email from a member of our team. 

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they'll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact digitalhealth@plos.org.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Digital Health.

Best regards,

Harry Hochheiser

Section Editor

PLOS Digital Health

***********************************************************

Reviewer Comments (if any, and for reference):