Explainable mortality prediction models incorporating social health determinants and physical frailty for heart failure patients

Zhenyue Gao; Xiaoli Liu; Yu Kang; Pan Hu; Xiu Zhang; Mengwei Li; Yumeng Peng; Wei Yan; Muyang Yan; Pengming Yu; Zhengbo Zhang; Qing Zhang; Wendong Xiao

doi:10.1371/journal.pone.0327979

Peer Review History

Original SubmissionNovember 14, 2023
Decision Letter - Lorenzo Righi, Editor PONE-D-23-36939 Explainable Mortality Prediction Models Incorporating Social Health Determinants and Physical Frailty for Heart Failure Patients PLOS ONE Dear Dr. Gao, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we have decided that your manuscript does not meet our criteria for publication and must therefore be rejected. As an Editor I want to point out that the article is interesting and really has some very positive points: the reported AUROC values indicate good discrimination performance for various mortality timeframes and the use of explainable models is commendable, allowing healthcare providers to understand the contributing factors. At the same time, however, there are major limitations that prevent me from considering it for publication: Inadequate Justification and Contextualization: The introduction lacks sufficient detail on the existing gaps in the literature and how exactly the proposed model improves upon current HF prediction tools. There is a need for a more thorough review of related works. Methodological Flaws: The use of XGBoost is justified, but there is no clear rationale for why this method was chosen over others. Furthermore, while logistic regression and random forests are used as baseline models, there is no discussion on why these specific models were selected. The feature construction process is inadequately explained. The imputation methods for missing data and the creation of new features need more detailed justification and transparency. Model Evaluation and Comparisons: The study claims that their models outperform existing models and clinical scores, yet the comparison lacks robustness. The results are presented with AUROC values, but other important metrics like calibration, clinical usability, and comparison with external validation cohorts are missing. There is also no discussion on potential overfitting, especially given the high number of features relative to the sample size. Explainability and SHAP Analysis: While SHAP values are used to explain the model, the interpretation of these values is superficial. The paper fails to provide insights into how these explanations can be used in clinical practice to improve decision-making. There is a lack of discussion on the limitations and potential biases introduced by the SHAP analysis. Generalizability: The generalizability of the findings to other populations or healthcare settings is not addressed. Given the specific nature of the MIMIC databases, the findings may not be applicable to broader or more diverse patient populations. Presentation and Clarity: The manuscript is poorly organized, with several sections lacking clarity and depth. The flow of information is disjointed, making it difficult to follow the methodology and results. There are numerous grammatical errors and awkward phrasings throughout the text, which detracts from the overall readability and professionalism of the manuscript. I am sorry that we cannot be more positive on this occasion, but hope that you appreciate the reasons for this decision. Kind regards, Lorenzo Righi Academic Editor PLOS ONE [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? Reviewer #1: Partly Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #1: No Reviewer #2: No ****** 3. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #1: Yes Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #1: No Reviewer #2: Yes ****** Reviewer #1: This manuscript aims to develop ML models that can predict heart failure mortality for in-hospital, 90-day and 1-year after admission and determine the importance of SDoH and physical frailty in the predictions. The writing standard needs to be improved to enhance the clarity and help in comprehending the paper. As such the paper is difficult to read and understand with a lot of grammar issues, typos, and missing references. The explanations of the results and findings are highly ambiguous and in many cases it is difficult to understand what specific terms are referring too. In addition, the organization of the paper needs a complete revamp. Specific comments: 1. What is the reason for limiting to only 4 SDOH factors? There has been research that associated SDOH factors such socioeconomic status, and education or health literacy to HF. Why were these factors not considered? 2. Majority of the prior work is discussed in the Discussion section which needs to be moved to the Introduction for better organization and clarity. There are prior studies that explored SDOH and 90-day mortality in HF patients (see below) which needs to be compared with the current work. Sterling MR, Ringel JB, Pinheiro LC, Safford MM, Levitan EB, Phillips E, Brown TM, Goyal P. Social determinants of health and 90‐day mortality after hospitalization for heart failure in the REGARDS study. Journal of the American Heart Association. 2020 May 5;9(9):e014836. 3. The objective of the paper as mentioned in the abstract and conclusion is to develop and validate ML models. Hence it is expected to show these results in the main manuscript and not in the supplements. The authors need to analyze their objectives and main findings and move relevant tables to the main manuscript. 4. “As shown, our prediction models consistently outperformed all models and scores.” I am not sure what scores is referred to here. If this is about the 7 metrics in eTable6, LR is consistently better for specificity, accuracy, F1, etc. 5. “Representative statistical features were calculated based on the type of variable, such as the maximum, minimum, mean, minus, ..” what does minus mean? Are you referring to the difference between the values? 6. “We found both of them could improve the discrimination …” what do you mean by “both of them”? It is not clear from the context. 7. “The potential bias and injustice of HF patients faced by race in care and outcomes has recently been recognized by researchers, and they have been undertaken to mitigate this dual social and biological impact by building the model separately or incorporate race into the model” – This sentence is highly ambiguous and complex. It is better to split this and explain in detail. Also, references are missing. 8. “Survival curves in Figure 1(c) (e) show that survival rates are lower for patients with health insurance and older patients, respectively.” Recheck the statement and order of the figure. 9. Some sentences need to be rephrased as they only convey partial information and is not a complete sentence. e.g., “While most related studies are limited to small cohort sizes.” 10. Many references appear as “ Error! Reference source not found” 11. Some typos – SODH, martial (should be marital) 12. Introduction – “require admitted to intensive care unit (ICU)” – should be "admission" instead of admitted 13. “patients, which has been demonstrated high” – should be " ... which has demonstrated ..." There are too many such grammatical errors that needs to be fixed. It is not feasible to list everything here. Reviewer #2: The authors propose an XAI- eXtreme Gradient Boosting (XGBoost) algorithms to develop predictive models for short- and long-term outcomes. They utilize the SHapley Additive exPlanations (SHAP) method to interpret predictions and assess the importance of different factors. The results show mortality incidences of 14.8% for in-hospital, 7.0% for 90-day, and 13.5% for 1-year mortality. The prediction models achieve high area under the receiver operating characteristic curve (AUROC) scores: 0.836 for in-hospital, 0.790 for 90-day, and 0.789 for 1-year mortality, outperforming baseline ML models and routine clinical scores. Important risk factors identified include age, fall risk, primary language, blood urea nitrogen levels, comorbidities, urine output, insurance type, and lung sounds. The model is interesting, however, I have the following comments: -what is the reason of using 90-days and 1-year intervals, literature or stat justification is required. -what is the p-values for the studied variables between the 2 classes. -what is tbeused CI in the Kaplan-meier plot? -How English vesrus non-Ebglish matters in the prediction? justigicatio, please. ****** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #1: No Reviewer #2: Yes: Abedalrhman Alkhateeb ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] - - - - - For journal use only: PONEDEC3 https://doi.org/10.1371/journal.pone.0327979.r001
Revision 1
2 Dec 2024 Author Response Dear Dr. Lorenzo Righi, I am writing to express my gratitude for your detailed feedback on our manuscript, titled "[Explainable Mortality Prediction Models Incorporating Social Health Determinants and Physical Frailty for Heart Failure Patients," submitted to PLOS One, and to request a reconsideration of our submission. We highly appreciate your positive remarks regarding the use and results of explainable models. We believe our work presents significant innovations, particularly in the incorporation of Social Determinants of Health (SDOH) and physical frailty into machine learning models for predicting both short-term and long-term mortality in heart failure patients. To our knowledge, this approach is unique and fills a critical gap in current research. We also wish to note that similar studies have been published in the PLOS ONE, such as “Target-based fusion using social determinants of health to enhance suicide prediction with electronic health records” and “How attitudes of state and community leaders regarding health equity and social determinants of health are associated with behavioral intentions to improve population health”. I appreciate your feedbacks and would like to discuss the following points: 1. Inadequate Justification and Contextualization: - We acknowledge that the introduction could better detail the existing gaps in the literature. We are prepared to revise this section to thoroughly review related works and clearly articulate how our model advances current HF prediction tools. 2. Methodological Flaws: - While XGBoost was chosen for its superior performance in preliminary tests, we understand the need for a clearer rationale. We will provide a detailed justification for selecting XGBoost over other methods, as well as our choice of logistic regression and random forests as baselines. - The feature construction process, including imputation methods and creation of new features, will be elaborated upon to ensure transparency and robustness. 3. Model Evaluation and Comparisons: - We will enhance our comparison of model performance by including additional metrics such as calibration and clinical usability. Moreover, we will discuss the potential for overfitting and provide results from external validation cohorts where possible. 4. Explainability and SHAP Analysis: - The interpretation of SHAP values will be deepened to provide actionable insights for clinical practice. We will also address the limitations and potential biases of SHAP analysis. 5. Generalizability: - While generalizability is indeed crucial, the primary focus of our study is to explore whether incorporating SDOH and physical frailty can enhance predictive performance, an area that is currently underexplored. Establishing the efficacy of these variables is a necessary first step before broader generalization studies. It is worth noting that several recent articles published in PLOS ONE have also only utilized the MIMIC-IV database, indicating the relevance and acceptance of this dataset for impactful research, such as “U-shaped association between serum triglyceride levels and mortality among septic patients: An analysis based on the MIMIC-IV database” and “Association between red cell distribution width and all-cause mortality in patients with breast cancer: A retrospective analysis using MIMIC-IV 2.0”. 6. Presentation and Clarity: - The manuscript will be thoroughly reorganized to improve clarity and flow. We will also correct grammatical errors and awkward phrasings to enhance readability and professionalism. Attachments Attachment Submitted filename: response to reviewers.docx https://doi.org/10.1371/journal.pone.0327979.r002
Decision Letter - Amir Hossein Behnoush, Editor Dear Dr. Gao, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. The reviewers have mentioned several major issues needed to be addressed. Please submit your revised manuscript by May 11 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols . We look forward to receiving your revised manuscript. Kind regards, Amir Hossein Behnoush Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse. 3. Note from Emily Chenette, Editor in Chief of PLOS ONE, and Iain Hrynaszkiewicz, Director of Open Research Solutions at PLOS: Did you know that depositing data in a repository is associated with up to a 25% citation advantage (https://doi.org/10.1371/journal.pone.0230416)? If you’ve not already done so, consider depositing your raw data in a repository to ensure your work is read, appreciated and cited by the largest possible audience. You’ll also earn an Accessible Data icon on your published paper if you deposit your data in any participating repository (https://plos.org/open-science/open-data/#accessible-data). 4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section. 5. Thank you for stating the following financial disclosure: “The author(s) received no specific funding for this work.” At this time, please address the following queries: a) Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution. b) State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.” c) If any authors received a salary from any of your funders, please state which authors and which funders. d) If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.” Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 6. Please note that in order to use the direct billing option the corresponding author must be affiliated with the chosen institute. Please either amend your manuscript to change the affiliation or corresponding author, or email us at plosone@plos.org with a request to remove this option. 7. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well. 8. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 9. For studies involving third-party data, we encourage authors to share any data specific to their analyses that they can legally distribute. PLOS recognizes, however, that authors may be using third-party data they do not have the rights to share. When third-party data cannot be publicly shared, authors must provide all information necessary for interested researchers to apply to gain access to the data. (https://journals.plos.org/plosone/s/data-availability#loc-acceptable-data-access-restrictions) For any third-party data that the authors cannot legally distribute, they should include the following information in their Data Availability Statement upon submission: a) A description of the data set and the third-party source b) If applicable, verification of permission to use the data set c) Confirmation of whether the authors received any special privileges in accessing the data that other researchers would not have d) All necessary contact information others would need to apply to gain access to the data Additional Editor Comments (if provided): [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author Reviewer #3: All comments have been addressed Reviewer #4: All comments have been addressed Reviewer #5: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions??> Reviewer #3: Yes Reviewer #4: (No Response) Reviewer #5: No ****** 3. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #3: Yes Reviewer #4: (No Response) Reviewer #5: No ****** 4. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #3: No Reviewer #4: (No Response) Reviewer #5: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #3: No Reviewer #4: (No Response) Reviewer #5: No ****** Reviewer #3: The paper use XGBoost model to predict patient mortality rates due to heart failure based on various features, such as social determinants and physical frailty. THe XGBoost model outperform baseline models like GWTG-HF, SOFA, random forest and logistic regression in terms of various metrics and help identify risk factors of heart failure. The paper generally looks good. The writing needs to be polished and significantly improved. Reviewer #4: (No Response) Reviewer #5: the manuscript has a long and monotonic introduction that has mixed up the introduction with the discussion. this will indeed reduce readability and confuse general reader. ****** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #3: No Reviewer #4: No Reviewer #5: Yes: Alireza Ramandi ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0327979.r003
Revision 2
15 May 2025 Author Response Responses to Reviewer #3 Comment 1: The paper use XGBoost model to predict patient mortality rates due to heart failure based on various features, such as social determinants and physical frailty. The XGBoost model outperform baseline models like GWTG-HF, SOFA, random forest and logistic regression in terms of various metrics and help identify risk factors of heart failure. The paper generally looks good. The writing needs to be polished and significantly improved. Response: We sincerely thank the reviewer for the positive evaluation of our study and for acknowledging the contribution of our XGBoost-based model in improving heart failure mortality prediction and risk factor identification. We fully agree with the reviewer’s suggestion regarding the need to polish the manuscript. Accordingly, we have thoroughly revised the entire text for clarity, grammar, and readability. The manuscript has undergone comprehensive language editing to ensure it meets the standards of academic writing. We believe these changes have significantly improved the overall quality and presentation of our work. Responses to Reviewer #5 Comment 1: the manuscript has a long and monotonic introduction that has mixed up the introduction with the discussion. this will indeed reduce readability and confuse general reader. Response: We thank the reviewer for this valuable and constructive comment. We acknowledge that the original introduction was overly long and included elements more appropriate for the discussion section, which may have affected clarity and readability. In response, we have carefully revised the introduction to make it more concise, focused, and engaging. Specifically, we have restructured the content to clearly distinguish the background, research gap, and objectives of the study, and we have removed discussion-like interpretations that were prematurely introduced. We believe the revised introduction now better aligns with academic conventions and improves the reader’s experience. Attachments Attachment Submitted filename: response_to_reviewers_auresp_2.docx https://doi.org/10.1371/journal.pone.0327979.r004
Decision Letter - Amir Hossein Behnoush, Editor Explainable Mortality Prediction Models Incorporating Social Health Determinants and Physical Frailty for Heart Failure Patients PONE-D-23-36939R2 Dear Dr. Zhang, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Amir Hossein Behnoush Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author Reviewer #7: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions??> Reviewer #7: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #7: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #7: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #7: Yes ****** Reviewer #7: (No Response) ****** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #7: No ******** https://doi.org/10.1371/journal.pone.0327979.r005
Formally Accepted
Acceptance Letter - Amir Hossein Behnoush, Editor PONE-D-23-36939R2 PLOS ONE Dear Dr. Zhang, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Amir Hossein Behnoush Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0327979.r006

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .