PONE-D-23-27712Artificial Intelligence Predicts Coronary Artery Bypass Graft Outcomes Using a Single Surgical NotePLOS ONE

Dear Dr. Kilic,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Nov 05 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Amirmohammad Khalaji

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

""Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

4. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

Reviewer #3: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I would like to thank the authors for studying the outcomes following CABG using AI approaches. It is an interesting paper. However, there are some comments as follows:

Title: It is recommended to change the title to “Prediction of Coronary Artery Bypass Graft Outcomes Using a Single Surgical Note: An Artificial intelligence-based prediction model study”

Introduction:

Line 82-83: It is recommended to explain the superiority of AI to conventional analysis (e.g., logistic regression). To the best of my knowledge, AI models did not perform better. Please explain the statistical rational for this approach.

Line 84: In prediction model studies, it is better use predictors instead of risk factors. Please consider it in the whole manuscript, as well.

Methods: This section has been written well. But, according to the recent recommendation, it is recommended to write this section based on TRIPOD writing standard to improve the study replicability.

Please identify that isolated on-pump or off-pump CABG?

Line 195, SHAP value? Please provide its expanded form.

Line 119-120 Performance was measure by AUC. AUC provides information on discriminative ability of the models. It is recommended to provide the Calibration of the models, as an Achill heel in prediction model studies.

Please provide IRB or REC approval.

Results: line 212: median of AUC. It is better to use average instead.

Table 1: It is recommended to write Table 1 in a way to be understand easily by the readers. Please provide the expanded form of abbreviations bottom of the table.

Discussion:

Please discuss the results with previous studies in terms of AUC and predictors.

You can use the following large scale studies:

- Nomali, M., Heidari, M.E., Ayati, A. et al. Risk factors of in-hospital mortality for isolated on-pump coronary artery bypass graft surgery in the northeast of Iran from 2007 to 2016. Ir J Med Sci (2023). https://doi.org/10.1007/s11845-023-03298-6

- Sattartabar B et al (2021) Sex and age difference in risk factor distribution, trend, and long-term outcome of patients undergoing isolated coronary artery bypass graft surgery. BMC Cardiovasc Disord 21(1):1–10

- Also, please provide study limitations and implication.

Reviewer #2: Sherard et al. have conducted a study on designing AI models for the prediction of CABG outcomes in a single center. The findings are interesting; however, there are concerns that should be addressed.

- Features used for training the models should be more explained since they are not clear to the readers. What do the authors mean by surgical notes in the methods and also feature importance in the results section?

- The abbreviations should be mentioned in complete form in their first use (e.g., IQR).

- The headings “ Permutation Tests”, “Embedding Correlations”, and “Feature Importance” in the results section should be further elaborated.

- In the discussion section, a comparison should be made with other AI studies performed on patients undergoing CABG (such as 10.3389/fcvm.2022.977747, 10.3389/fcvm.2022.881881, and 10.1016/j.athoracsur.2021.08.040)

- The study lacks a limitation section which should be mentioned before the conclusion.

Reviewer #3: Summary

The authors have presented two Convolutional Neural Network (CNN)-based models designed to predict Coronary Artery Bypass Grafting (CABG) outcomes utilizing a single pre-operative surgical note. The first model, TextCNN, is a previously known CNN architecture specifically tailored for processing textual input, while the second model (AttnToNumCNN) represents an enhanced CNN architecture that incorporates a novel embedding pipeline for encoding numerical values. Furthermore, the enhanced model incorporates a multi-headed self-attention module. Notably, the authors have demonstrated good predictive performance with Area Under the Receiver Operating Characteristic Curve (AUC) values of 0.79 and 0.82 for TextCNN and the enhanced model, respectively.

Strengths:

* New numerical data embedding pipeline: The introduction of a new embedding layer for encoding numerical data within the medical context is a notable contribution.

* Performance on the primary task: The study has successfully achieved strong predictive performance on the primary task, as evidenced by the AUC values, underscoring the effectiveness of the proposed models in predicting CABG outcomes.

* SHAP Value Utilization: The incorporation of SHAP (Shapley additive explanations) values to explain the model's predictions is a commendable approach, fostering interpretability and transparency in the model's decision-making process.

Weaknesses:

* Lack of Clarity Regarding New Modules: The manuscript could benefit from a more explicit presentation of the specific contributions and impacts of the novel modules introduced in the enhanced model.

* Limited Model Exploration: The authors have primarily explored CNN-based architectures in this study. However, considering the sequential nature of medical notes, it would be advantageous to experiment with alternative models, such as Long Short-Term Memory (LSTM) or Transformer-based architectures, which are designed for effective long-sequence processing.

Issues

Major:

1. The authors have incorporated two novel modules, namely the embedding layer and the self-attention module, into their AttnToNumCNN architecture. They subsequently compare its performance to the baseline TextCNN model. However, a key issue arises in disentangling the distinct contributions of each of these modules to the observed performance enhancement. This lack of clarity raises the possibility that the improved performance may be primarily attributed to the introduction of the self-attention module alone, even if used along with the original embedding layers of the TextCNN model. Furthermore, the introduction of both the embedding layer and self-attention module introduces additional trainable parameters into the enhanced architecture, thereby complicating the fair comparison with the TextCNN model. To address this concern, it is advisable for the authors to report the total number of parameters for both models (it would be great if the number of parameters of the new modules are reported separately as well) and perform an ablation study by introducing only the new embedding layer, subsequently reporting its net impact on performance and then doing the same after adding the self-attention layer. This approach would help disentangle the individual contributions of these modules and enhance the interpretability of the performance improvements observed.

Minor:

1. Utilize Alternative Text Models: While the authors justified their omission of larger models like BERT due to computational costs and literature suggesting comparable performance with smaller models, it is advisable to explore traditional text-processing model families such as RNN and LSTM. These architectures are better suited for processing long-sequences compared to CNNs, which mainly focus on local patterns. Additionally, considering the dominance of Transformer-based architectures in NLP tasks, incorporating a scaled-down Transformer variant with a comparable parameter count to TextCNN could provide valuable insights into their relative effectiveness.

2. Visualize Self-Attention Weights: It is recommended to visualize the self-attention weights for a few input examples to gain visual insights into the behavior of the attention layer. This visualization would enhance the comprehensibility of the attention mechanism's functioning, thereby contributing to a more thorough investigation of its role in model decisions.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Mahin Nomali, Ph.D in Epidemiology

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: CABG-MN.docx

PONE-D-23-27712R1Prediction of Coronary Artery Bypass Graft Outcomes Using a Single Surgical Note: An Artificial Intelligence-based Prediction Model StudyPLOS ONE

Dear Dr. Kilic,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Minor revisions are required according to reviewer 1's comments.

Please submit your revised manuscript by Apr 11 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Amirmohammad Khalaji

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

Reviewer #3: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: (No Response)

Reviewer #3: (No Response)

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I would like to thank the authors for dedicating time to revise the paper accordingly. However, there are comments as follows:

Abstract, methods: Please provide the expanded form of MM at first.

Abstract, results: I could not understand why you provided IQR for ROC instead of 95% CI. Please provide AUC and its uncertainty (95%CI).

Introduction:

Please provide the expanded form of AI, firstly. Please consider it in the whole document.

How about Research ethics approval? And, calibration as a performance measure for prediction model studies?

Results:

You can not report mean and IQR. You can report Mean and SD or Median and IQR. In addition, you must report AUC and 95 % Confidence interval, instead.

This section has been written in a complex way that may be it is interested for statistician or epidemiologist NOT clinicians. It recommended to use graph or table to facilitate its understanding.

Reviewer #2: (No Response)

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Mahin Nomali

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Prediction of Coronary Artery Bypass Graft Outcomes Using a Single Surgical Note: An Artificial Intelligence-based Prediction Model Study

PONE-D-23-27712R2

Dear Dr. Kilic,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at http://www.editorialmanager.com/pone/ and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Amirmohammad Khalaji

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: