Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network

Jahan C. Penny-Dimri; Christoph Bergmeir; Christopher M. Reid; Jenni Williams-Spence; Andrew D. Cochrane; Julian A. Smith

doi:10.1371/journal.pone.0289930

Peer Review History

Original SubmissionApril 28, 2022
27 Aug 2022 Decision Letter - Guangyu Tong, Editor PONE-D-22-12461Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network.PLOS ONE Dear Dr. Penny-Dimri, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Oct 11 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Guangyu Tong Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Will the web application be publicly available upon publication? Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse." 3. "Please amend your current ethics statement to address the following concerns: a) Did participants provide their written or verbal informed consent to participate in this study? b) If consent was verbal, please explain i) why written consent was not obtained, ii) how you documented participant consent, and iii) whether the ethics committees/IRB approved this consent procedure. 4. Thank you for stating the following in the Acknowledgments Section of your manuscript: "The ANZSCTS Cardiac Surgery Database Program is funded by the Department of Health (Victoria), the Clinical Excellence Commission (NSW), Queensland Health (QLD), and funding from individual cardiac surgical units participating in the registry. ANZSCTS Database Research activities are supported through a National Health and Medical Research Council Principal Research Fellowship (APP 1136372) and Program Grant (APP 1092642) awarded to C.M. Reid. The Database thanks all of the investigators, data managers, and institutions that participate in the Program." We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: "The author(s) received no specific funding for this work." Please include your amended statements within your cover letter; we will change the online submission form on your behalf. Additional Editor Comments (if provided): Two field experts were invited to review your manuscript. The two major areas for improvement are the unclear description of the methods and the unclear significance/contribution of the paper to the existing knowledge. The writing also needs to be significantly improved in order to meet the publication standard. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly Reviewer #2: Partly ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: I Don't Know Reviewer #2: I Don't Know ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No Reviewer #2: No ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: No Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: • There are many redundant sentences in the abstract, and the logic is also not clear enough. Please refine the content of the abstract again. Author need to make clear the research significance of the article, the research process, the experimental results and conclusions. • The keywords need to be extracted again, and it could be best to reflect the characteristics of this article. • The introduction of the article is complex and tedious about the modern approaches, so it is necessary to highlight the research status at home and abroad, and simplify the background of the article. • In this manuscript, the explanation of flow diagram in Figure 1 is not enough, and the author needs to give a more detailed explanation to facilitate the understanding of readers. • The analysis of the experimental data in Table 3 can be added to improve the uncertainty calibration calculation. • The conclusion of this manuscript needs to be reorganized by highlighting the research results of this research work, not too much introduction of future research methods. • There are grammatical errors in this article, which need to be corrected after careful checking. • Carefully check the references of the article and also ensure the citation is in the right place. • The objectives of your work need to be clearly stated in different sections. • Author needs to highlight the novelty of this research work in clear manner. Reviewer #2: The given manuscript by Penny-Dimri et al. highlights a relevant subject and presents a potentially beneficial and useful method. The results seem convincing. However, some issues need to be addressed before the manuscript can be published. Major issues (in decreasing order of importance): - The UAN approach should be described in detail, as this is not a commonly used method. Please provide details about the architecture, its internals and the training procedure. - In Tables 2 and 3, the best results are written in bold. Does that mean that they are simply the best or does that mean that the bold results are significantly better than other results? I strongly suggest to test for significance and to describe the employed statistical procedure in detail. - It is imperative (otherwise a bias is introduced that renders the results invalid) that the calibration/training of missing value imputation methods is only done on the training set(s). Was that the case in the third strategy mentioned in the manuscript? It did not become entirely clear. Please provide more detail! - Include more recent reference to works dealing with machine learning in cardiac surgery, e.g.: Bodenhofer et al., Eur. J. Cardiothorac. Surg. 60(6), 2021. https://doi.org/10.1093/ejcts/ezab219 Jiang et al., Front. Cardiovasc. Med. 8, 2021. https://doi.org/10.3389/fcvm.2021.771246 Shuhaiber & Conte, Eur. J. Cardiothorac. Surg. 60(6), 2021. https://doi.org/10.1093/ejcts/ezab324 Some of these works indeed demonstrate a general advantage of more complex methods over LR. - There are also some deeper works dealing with interpretations of machine learning models (some limited to tree ensembles, some model-agnostic), e.g.: Lundberg & Lee, NIPS 2017. https://papers.nips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf Sharma et al., arXiv:2012.06734, 2020. https://arxiv.org/pdf/2010.06734.pdf Minor issues: - The figures have very poor resolutions. In particular, the heatmaps in Fig. 2 are illegible. Please fix this! ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0289930.r001
Revision 1
4 Oct 2022 Author Response EDITORIAL COMMENTS: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf Response: Thank you for the above resources. The manuscript has been formatted accordingly. 2. Will the web application be publicly available upon publication? Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse." Response: The code base for generating the web application and running the experiments will be available as a public GitHub repository (https://github.com/jahanpd/UAN). 3. "Please amend your current ethics statement to address the following concerns: a) Did participants provide their written or verbal informed consent to participate in this study? b) If consent was verbal, please explain i) why written consent was not obtained, ii) how you documented participant consent, and iii) whether the ethics committees/IRB approved this consent procedure. Response: The ethics statement has been amended as requested to address the question of consent. The ANZSCTS National Database program data policy and patient information sheet have been included as supplementary materials. See lines 210-213. 4. Thank you for stating the following in the Acknowledgments Section of your manuscript: "The ANZSCTS Cardiac Surgery Database Program is funded by the Department of Health (Victoria), the Clinical Excellence Commission (NSW), Queensland Health (QLD), and funding from individual cardiac surgical units participating in the registry. ANZSCTS Database Research activities are supported through a National Health and Medical Research Council Principal Research Fellowship (APP 1136372) and Program Grant (APP 1092642) awarded to C.M. Reid. The Database thanks all of the investigators, data managers, and institutions that participate in the Program." We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: "The author(s) received no specific funding for this work." Please include your amended statements within your cover letter; we will change the online submission form on your behalf. Response: Thank you for highlighting this apparent discrepancy. It should be noted that the database operates under a grant structure with its own ethics approval, whereas the research of this paper is separate from the operations of the database organization. This work received no specific funding. We have addressed this separation of concerns in our funding statement and removed the funding statement from the acknowledgments. REVIEWER COMMENTS (Tasks Highlighted): 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: • There are many redundant sentences in the abstract, and the logic is also not clear enough. Please refine the content of the abstract again. Author need to make clear the research significance of the article, the research process, the experimental results and conclusions. Response: Thank you for your feedback. The abstract has been rewritten to address these concerns and fulfill PLOS One formatting requirements. • The keywords need to be extracted again, and it could be best to reflect the characteristics of this article. Response: Thank you for your comments. We have revised our keywords. • The introduction of the article is complex and tedious about the modern approaches, so it is necessary to highlight the research status at home and abroad, and simplify the background of the article. Response: Thank you for your feedback. We have amended our introduction to simplify the discussion of alternative approaches and moved some of the method details to the methods section. • In this manuscript, the explanation of flow diagram in Figure 1 is not enough, and the author needs to give a more detailed explanation to facilitate the understanding of readers. Response: Thank you for your feedback. In-text reference to Figure 1 has been made on line 140 in order to provide in-text context to the visualized data processing pipeline in Figure 1. • The analysis of the experimental data in Table 3 can be added to improve the uncertainty calibration calculation. Response: Thank you for your feedback. We have added references to Table 3 at lines 232 and 236. • The conclusion of this manuscript needs to be reorganized by highlighting the research results of this research work, not too much introduction of future research methods. Response: Thank you for your feedback. The discussion and conclusion have been edited and reorganized to focus more on the research presented. • There are grammatical errors in this article, which need to be corrected after careful checking. Response: Thank you for your feedback. We have parsed our work and corrected our grammar. • Carefully check the references of the article and also ensure the citation is in the right place. Response: Thank you for your feedback. We have re-evaluated the references list. • The objectives of your work need to be clearly stated in different sections. Response: Thank you for your feedback. Aims and hypotheses have been altered to be more clear. • Author needs to highlight the novelty of this research work in clear manner. Response: Thank you for your feedback. The novelty of this work has been highlighted on lines 267, 281, 341. Additionally, we hope that the further supplementary materials provided to explain the model in greater detail aid the readers in appreciating the novelty of our approach. Reviewer #2: The given manuscript by Penny-Dimri et al. highlights a relevant subject and presents a potentially beneficial and useful method. The results seem convincing. However, some issues need to be addressed before the manuscript can be published. Major issues (in decreasing order of importance): - The UAN approach should be described in detail, as this is not a commonly used method. Please provide details about the architecture, its internals, and the training procedure. Response: Thank you for your feedback. We have added further details regarding the model design and the training algorithm in the supplemental materials (S1 Appendix) with in-text reference at line 162. - In Tables 2 and 3, the best results are written in bold. Does that mean that they are simply the best or does that mean that the bold results are significantly better than other results? I strongly suggest testing for significance and describing the employed statistical procedure in detail. Response: Thank you for your suggestion. The bold results are simply the best performing. Regarding the application of a statistical test on cross-validation, this is a relatively controversial topic. While cross-validation and other bootstrap methods are widely used to assess the generalization performance of machine learning and statistical models, the process is well known to not produce independent samples. This arises as each fold will share data across training and test sets. As such, applying traditional statistical tests, such as a student's T-test will produce excess type 1 error. While there are corrected tests and cross-validation schemes that purport to improve the power of these tests, we suggest that these come burdened with their own assumptions and do not solve the underlying problem (some references that highlight the problems and approaches include: https://doi.org/10.48550/arXiv.2104.00673, https://doi.org/10.1023/A:1024068626366, https://doi.org/10.1162/neco.1997.9.6.1245, https://doi.org/10.1162/089976698300017197 https://doi.org/10.1007/978-3-540-24775-3_3). Applying a statistical test may give an unreliable p-value and we fear reporting such results would give the reader a false assurance. We also note that it is quite common in ML literature to simply report the raw performance. We have chosen to report the mean of the cross-validation with an accompanying measure of the variance. We believe this gives readers enough information to assess the distribution of performance for each model. - It is imperative (otherwise a bias is introduced that renders the results invalid) that the calibration/training of missing value imputation methods is only done on the training set(s). Was that the case in the third strategy mentioned in the manuscript? It did not become entirely clear. Please provide more detail! Response: Thank you for your feedback. Indeed missing value imputation was performed only on the training set data during cross-validation. This has been specified in-text at line 181, and is verifiable from the provided codebase. - Include more recent references to works dealing with machine learning in cardiac surgery, e.g.: Response: Thank you for your suggestions. We have cited two recent meta-analyses, the latter which includes Bodenhofer et al. We have added the additional citations as requested, however our phrasing remains conservative as the most recent meta-analysis on this topic failed to find a significant advantage of ML over LR (https://doi.org/10.1111/jocs.16842). Lundberg and Lee and Sharme and Conte have been cited where we reference explainability on line 279. Minor issues: - The figures have very poor resolutions. In particular, the heatmaps in Fig. 2 are illegible. Please fix this! Response: Thank you for your feedback. We have endeavored to reformat the images as per PLOS One guidelines and using the PACE online tool to ensure adequate formatting. Please note the appearance of the figures in the compiled PDF do not reflect the quality of the tiff files when downloaded separately. Attachments Attachment Submitted filename: ReviewerComments_Response.docx https://doi.org/10.1371/journal.pone.0289930.r002
28 Nov 2022 Decision Letter - Guangyu Tong, Editor PONE-D-22-12461R1Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network.PLOS ONE Dear Dr. Penny-Dimri, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Jan 12 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Guangyu Tong Academic Editor PLOS ONE Additional Editor Comments: Please carefully address the comments from Reviewer 2 and ensure the claim of this paper is not overstated and biased, as pointed out by Reviewer 2. I also recommend the authors perform the additional test Reviewer 2 requested. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: I Don't Know ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: Most of my comments have properly been taken into account by the authors, so I can recommend the manuscript for publication now. There are two issues, however, that I still want to comment on: 1) I agree that significance of cross validation results are a subtle topic. I acknowledge that p-values might be inflated, so the tests might have a higher type I error rate. It is a fact, though, that such tests are still quite common in literature. Let me make this point clear: I was not so much afraid of insignificant differences being falsely flagged as significant. My concern would rather have been the following: If a result marked in bold would fail the test for significance, then it has no justification to be reported as a serious advancement. So I was rather afraid of falsely announcing an algorithm as better than the rest although it is actually on par with another. Apart from that, I have no doubt - looking at the numbers and the standard deviations in parentheses - that the significance test would be successful. So I suggest the authors either consider performing the test (despite its weaknesses and limitations) or provide a short argumentation in the manuscript why they have not done it (even though it is quite commeon). 2) Regarding the citations on machine learning in cardiac surgery that I suggested: I am not in the position to force the authors to cite particular papers, so I accept that they have not followed my suggestion. However, since they mention their own meta study (which has very much attracted my interest when it appeared), I want to take this opportunity to comment on this work in relation to the present manuscript under review. First of all, I pretty much disagree with their earlier meta-study. I think it deals with an ill-posed question. Nobody every claimed that ML gives a general edge over LR. I fully agree that there might be some situations in which LR is sufficient or even advisable (less overfitting, better interpretability). However, there might also be situations in which ML indeed is advantageous, in particular in situations with complex, non-linear interactions between the input features. I try not to be too polemic about this issue, but the meta-study appears to me like asking whether there is evidence that trucks are better than cars. Well, in many cases, no. But there are some case where the additional power and loading capacity of a truck is of great advantage. It depends on the situation. It is now really ironic/funny/whatever that the authors of exactly this meta-study try to get a paper about ML in cardical surgery published. No matter whether you cite your own meta-study, I think this asks for an explanation. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0289930.r003
Revision 2
27 May 2023 Author Response Additional Editor Comments: Please carefully address the comments from Reviewer 2 and ensure the claim of this paper is not overstated and biased, as pointed out by Reviewer 2. I also recommend the authors perform the additional test Reviewer 2 requested. Response: Thank-you for your feedback. Statistical testing has been performed and is available in the supplemental materials (see response to reviewer 2). Reviewers' comments: Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: Most of my comments have properly been taken into account by the authors, so I can recommend the manuscript for publication now. There are two issues, however, that I still want to comment on: 1) I agree that significance of cross validation results are a subtle topic. I acknowledge that p-values might be inflated, so the tests might have a higher type I error rate. It is a fact, though, that such tests are still quite common in literature. Let me make this point clear: I was not so much afraid of insignificant differences being falsely flagged as significant. My concern would rather have been the following: If a result marked in bold would fail the test for significance, then it has no justification to be reported as a serious advancement. So I was rather afraid of falsely announcing an algorithm as better than the rest although it is actually on par with another. Apart from that, I have no doubt - looking at the numbers and the standard deviations in parentheses - that the significance test would be successful. So I suggest the authors either consider performing the test (despite its weaknesses and limitations) or provide a short argumentation in the manuscript why they have not done it (even though it is quite commeon). Thank-you for your ongoing review. We have provided statistical testing using T-tests and amended the manuscript to reflect this at lines 182 and 240. Tables of p-values for the pairwise T-tests are provided in the supplementary materials. 2) Regarding the citations on machine learning in cardiac surgery that I suggested: I am not in the position to force the authors to cite particular papers, so I accept that they have not followed my suggestion. However, since they mention their own meta-study (which has very much attracted my interest when it appeared), I want to take this opportunity to comment on this work in relation to the present manuscript under review. First of all, I pretty much disagree with their earlier meta-study. I think it deals with an ill-posed question. Nobody every claimed that ML gives a general edge over LR. I fully agree that there might be some situations in which LR is sufficient or even advisable (less overfitting, better interpretability). However, there might also be situations in which ML indeed is advantageous, in particular in situations with complex, non-linear interactions between the input features. I try not to be too polemic about this issue, but the meta-study appears to me like asking whether there is evidence that trucks are better than cars. Well, in many cases, no. But there are some case where the additional power and loading capacity of a truck is of great advantage. It depends on the situation. It is now really ironic/funny/whatever that the authors of exactly this meta-study try to get a paper about ML in cardical surgery published. No matter whether you cite your own meta-study, I think this asks for an explanation. Response: We thank the reviewer for their feedback. Additionally, we appreciate their analysis of our previous work and wholeheartedly agree with their conclusion. In particular, we wish to draw their attention to the discussion of the meta-analysis where we make precisely the same point. This is to say that although ML does not outperform LR as defined by the C-index, there are many reasons to consider using an ML model in non-linear settings such as peri-operative risk prediction. We suggest that ML can add value to risk prediction beyond raw performance. We make explicit the advantages of using explanatory modelling, such as with Shapley values. Furthermore, we believe that our submission under review adds to this argument by showing the flexibility to missing data, and unique explanatory visualization, which our novel ML approach provides. Attachments Attachment Submitted filename: ReviewerComments_Response.docx https://doi.org/10.1371/journal.pone.0289930.r004
31 Jul 2023 Decision Letter - Guangyu Tong, Editor Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network. PONE-D-22-12461R2 Dear Dr. Penny-Dimri, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Guangyu Tong Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: The authors have taken into account my comments properly (or have responded adequately). So I recommend this manuscript for publication. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: No ******** https://doi.org/10.1371/journal.pone.0289930.r005
Formally Accepted
10 Aug 2023 Acceptance Letter - Guangyu Tong, Editor PONE-D-22-12461R2 Paying attention to cardiac surgical risk: An interpretable machine learning approach using an uncertainty-aware attentive neural network. Dear Dr. Penny-Dimri: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Guangyu Tong Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0289930.r006

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .