Development and validation of interpretable machine learning models for triage patients admitted to the intensive care unit

Zheng Liu; Wenqi Shu; Hongyan Liu; Xuan Zhang; Wei Chong

doi:10.1371/journal.pone.0317819

Peer Review History

Original SubmissionNovember 10, 2024
10 Nov 2024 Author Response https://doi.org/10.1371/journal.pone.0317819.r001
13 Dec 2024 Decision Letter - Jerome Baudry, Editor PONE-D-24-51129Development and validation of interpretable machine learning models for triage patients admitted to the intensive care unitPLOS ONE Dear Dr. Liu, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please address comments number 2, number 3 and number 5 made by Reviewer 1. If possible, also try to address comment number 6 (about ethical implications). Please submit your revised manuscript by Jan 27 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols . We look forward to receiving your revised manuscript. Kind regards, Jerome Baudry, Ph.D. Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse. 3. Thank you for stating the following financial disclosure: “This research was supported by the Education Department of Liaoning Province, China. The funding was awarded to Zheng Liu under the grant number LJ232410159024.” Please state what role the funders took in the study. If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." If this statement is not correct you must amend it as needed. Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. Additional Editor Comments : Please address comments number 2, number 3 and number 5 made by Reviewer 1. If possible, also try to address comment number 6 (about ethical implications) [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The manuscript is well motivated and the focus on interpretable ML aligns with growing demand for transparency in medical decision making processes. However, there are areas where the manuscript can be improved to enhance its clarity: 1- Relying only on the MIMIC-IV dataset may limit the applicability of the study to other populations or healthcare systems. Although training-test split is used and worked but it would be beneficial if authors can have some external validation using another datasets 2- About the feature engineering and preprocessing steps, how outliers handled beyond basic thresholds? Were categorial features treated uniformly across algorighms? Removing missing and extreme values might create bias. Adding a sensitivity analysis or some more discussion would be beneficial. 3- Use of random under sampler is not justified 4- Also, it would be beneficial to include a discussion or visualization of how key features influence triage decisions. 5- Improvement of complex models (AUC 0.81) over simpler models (AUC 0.80) is marginal (Table 2). The authors should discuss whether this improvement justifies the added complexity. 6- Ethical implications of using the proposed automated decision making system in clinical should be briefly discussed. Reviewer #2: I appreciate the work represented here. This line of research is crucial for developing reliable AI/ML facilitated predictive models, not only for ICU admissions, but more broadly for more responsive health care across the continuum of care. I would very much like to see this study expanded or extended to include a broader range of data types including mental health, genetic, patient reported outcomes etc., to make it more generalizable. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #1: Yes: Armin Ahmadi Reviewer #2: Yes: Daniel Adamek ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0317819.r002
Revision 1
20 Dec 2024 Author Response Dear Reviewer1, Thank you for your valuable and constructive comments, which have helped us greatly in improving our manuscript. We have carefully reviewed your feedback and appreciate the effort you put into providing such detailed insights. Regarding comments number 2, 3, and 5, as advised by the editors, we have specifically addressed these points in our response. Additionally, we have made efforts to address comment number 6 related to the ethical implications, recognizing its importance. We also acknowledge that comments number 1 and 4 raise important considerations for further improving the study. However, due to current limitations (e.g., data availability or methodology constraints), these issues are challenging to address at this time. Nevertheless, we are deeply grateful for your suggestions and will incorporate them into our future research efforts. In the following, we provide detailed responses to comments 2, 3, 5, and 6. Q2: About the feature engineering and preprocessing steps, how outliers handled beyond basic thresholds? Were categorial features treated uniformly across algorighms? Removing missing and extreme values might create bias. Adding a sensitivity analysis or some more discussion would be beneficial. A: Thank you for raising these important points regarding the feature engineering and preprocessing steps. Your comments have highlighted key aspects critical to the robustness and interpretability of our analysis. We have carefully considered your suggestions and made the following clarifications and revisions to the manuscript: 1.Handling of outliers beyond basic thresholds: As you suggested, we have provided additional details on how outliers were handled in our study. Specifically, beyond the clinically defined thresholds, we further assessed outliers using the interquartile range (IQR) method. For each continuous variable, values outside 1.5 times the IQR below the first quartile (Q1) or above the third quartile (Q3) were flagged as potential outliers. These flagged values were carefully reviewed, and only those deemed clinically implausible were excluded. The updated description can be found in the Feature Engineering and Preprocessing section (lines 142–147) . 2.Treatment of categorical variables: In response to your question about the uniformity of categorical feature processing, we clarified in the manuscript that all categorical variables were uniformly processed across algorithms using one-hot encoding. This approach ensured consistent treatment of categorical features regardless of the algorithm used. For example, binary variables such as sex (male/female) and smoking status (yes/no) were transformed into binary dummy variables, while multi-class variables such as chief complaints were expanded into multiple binary columns. This clarification has also been added to the Feature Engineering and Preprocessing section (lines 147–152) . 3.Potential bias from removing missing and extreme values: We acknowledge your concern regarding the potential bias introduced by removing missing and extreme values, particularly if the data were not missing completely at random. To address this, we have added a discussion in the Limitations section (lines 425–430) . Specifically, we noted that the exclusion of missing and extreme values may introduce selection bias and reduce the representativeness of the dataset. We also highlighted that extreme values could carry clinically important information, and their exclusion might affect model performance. As a potential solution, we suggested that future studies consider applying advanced imputation methods, such as multiple imputation by chained equations (MICE), to address missing values and compare results with models based on complete-case data to evaluate potential biases. Additionally, we recognized that the clinically defined thresholds for extreme values were conservative and might have excluded meaningful data points. We proposed exploring machine learning-based anomaly detection methods or winsorization as alternative approaches in future research. Q3: Use of random under sampler is not justified A: Thank you for your insightful comment regarding the justification of random under sampling. This is indeed a critical methodological consideration that deserves careful explanation. Our decision to employ resampling techniques was primarily driven by the significant class imbalance in our dataset, where ICU patients (minority class) only accounted for 5.21% of the total cases compared to 94.69% for non-ICU patients (majority class). Such severe imbalance could lead to biased model performance if not properly addressed. To systematically justify our sampling strategy, we conducted a comparative analysis of different resampling techniques (SMOTE, SMOTE-Tomek, and RandomUnderSampler) using Logistic Regression as a case study. The results, now presented in Supplementary Table S1, demonstrate that all three resampling methods significantly improved the model's performance metrics for the minority class while maintaining consistent AUC values. We ultimately chose RandomUnderSampler for several critical reasons: 1.It achieved comparable performance improvements to other methods (improving F1-Score from 0.42 to 0.65 in Model 2 and from 0.53 to 0.74 in Model 3) 2.It uses only actual data points, reducing the risk of introducing synthetic sample bias 3.Most importantly, given our large dataset and the need to implement multiple complex models (including Gradient Boosting and Random Forest), SMOTE and SMOTE-Tomek were computationally prohibitive with our available computing resources. RandomUnderSampler offered significantly better computational efficiency while maintaining similar performance benefits, making it the most practical choice for our comprehensive multi-model analysis The use of resampling techniques to address significant class imbalance is well-established in clinical prediction research. For instance, Zahra Rahmatinejad (2024) also faced similar class imbalance challenges in their clinical prediction tasks, where they systematically evaluated different resampling methods including SMOTE-Tomek (doi: 10.1038/s41598-024-54038-4). While their specific choice of resampling method differed based on their unique dataset characteristics and computational requirements, the fundamental approach of addressing class imbalance through resampling aligns with our methodology. We have added these details to both the Methods (lines 181–191) and Results (lines 241–253) sections to provide a clear ratification of our methodological choice. We hope this comprehensive explanation addresses your concern about the use of random under sampling in our study. Q5: Improvement of complex models (AUC 0.81) over simpler models (AUC 0.80) is marginal (Table 2). The authors should discuss whether this improvement justifies the added complexity. A: Thank you for your valuable comment regarding the marginal improvement of complex models (AUC 0.81) over simpler models (AUC 0.80) and whether this justifies the added complexity. We have carefully addressed this point in the revised manuscript as follows (lines 353–368) : Although the difference in AUC between Gradient Boosting and logistic regression appears small, we argue that the added complexity of Gradient Boosting is justifiable in certain contexts because AUC alone does not fully capture a model’s overall performance. As described in the revised discussion, Gradient Boosting demonstrated the highest F1 Score (0.74) among all models, alongside the lowest Brier score (0.044), reflecting superior overall performance, calibration, and probabilistic prediction accuracy. Moreover, Gradient Boosting showed comparable or better clinical usefulness in decision curve analysis (Fig. 4B). These additional metrics highlight the advantages of Gradient Boosting beyond the marginal improvement in AUC. Furthermore, complex models like Gradient Boosting can effectively capture non-linear relationships and interactions between variables, which may be critical in clinical datasets with inherent heterogeneity. Nonetheless, we acknowledge the strengths of simpler models such as logistic regression, including their interpretability, computational efficiency, and practicality in resource-limited or transparent decision-making scenarios. As noted in the manuscript, the choice between simple and complex models ultimately depends on the specific clinical application and the balance between performance gains and trade-offs in interpretability and computational demands. We hope this expanded discussion adequately addresses your concern. Thank you again for your insightful feedback, which has helped us improve the clarity and depth of our manuscript. Q6: Ethical implications of using the proposed automated decision making system in clinical should be briefly discussed. A: Thank you for your thoughtful comment regarding the ethical implications of using the proposed automated decision-making system in clinical practice. This is indeed a critically important aspect of our study, and we have addressed it in the revised Discussion section (lines 412–424) . While such systems can significantly enhance the accuracy and efficiency of triage processes, we emphasized that they should not replace human judgment but rather serve as supportive tools for healthcare professionals. We have also acknowledged potential risks, such as over-reliance on automated systems, which might overlook nuanced clinical contexts that are difficult to quantify or encode into models. Furthermore, we highlighted the importance of addressing algorithmic bias, which may stem from imbalanced datasets or unrepresentative training data, to ensure equitable care across patient populations. In addition, we stressed that transparency in model development, validation, and implementation is essential for fostering trust among healthcare providers and patients. Most importantly, we underscored that the deployment of automated systems should always respect and prioritize patient autonomy, using predictions to inform, rather than dictate, clinical decisions. We hope this addition addresses your concern and further strengthens the ethical considerations of our study. Once again, we sincerely thank you for your thoughtful and constructive feedback. Your comments have been invaluable in improving the quality and clarity of our manuscript. While we have addressed the key points outlined in your review to the best of our current abilities, we recognize the significance of the remaining suggestions and will strive to incorporate them into future research where feasible. We hope that our revisions and responses satisfactorily address your concerns, and we are happy to provide further clarifications if needed. Thank you for your time and effort in reviewing our work. Sincerely, Wei Chong On behalf of all authors Corresponding Author Email: wchong@cmu.edu.cn Attachments Attachment Submitted filename: Response to Reviewers.docx https://doi.org/10.1371/journal.pone.0317819.r003
7 Jan 2025 Decision Letter - Jerome Baudry, Editor Development and validation of interpretable machine learning models for triage patients admitted to the intensive care unit PONE-D-24-51129R1 Dear Dr. Liu, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Jerome Baudry, Ph.D. Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: (No Response) ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #1: Yes: Armin Ahmadi ******** https://doi.org/10.1371/journal.pone.0317819.r004
Formally Accepted
Acceptance Letter - Jerome Baudry, Editor PONE-D-24-51129R1 PLOS ONE Dear Dr. Liu, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Jerome Baudry Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0317819.r005

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .