Does radiofrequency ablation procedural data improve the accuracy of identifying atrial fibrillation recurrence?

Mingkai Peng; Amit Doshi; Yariv Amos; Liat Tsoref; Mati Amit; Don Yungher; Rahul Khanna; Paul M. Coplan

doi:10.1371/journal.pone.0300309

Peer Review History

Original SubmissionSeptember 28, 2023
10 Dec 2023 Decision Letter - Ibrahim Marai, Editor Transfer Alert This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present. PONE-D-23-31440Does radiofrequency ablation procedural data improve the accuracy of identifying atrial fibrillation recurrence?PLOS ONE Dear Dr. Peng, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Jan 24 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Ibrahim Marai, MD Academic Editor PLOS ONE Journal requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. You indicated that ethical approval was not necessary for your study. We understand that the framework for ethical oversight requirements for studies of this type may differ depending on the setting and we would appreciate some further clarification regarding your research. Could you please provide further details on why your study is exempt from the need for approval and confirmation from your institutional review board or research ethics committee (e.g., in the form of a letter or email correspondence) that ethics review was not necessary for this study? Please include a copy of the correspondence as an ""Other"" file. 3. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section. 4. Thank you for stating the following in the Competing Interests section: “Several authors are employed by Johnson & Johnson or Biosense Webster, a Johnson & Johnson company. However, no Johnson and Johnson or Biosense Webster product was studied in this research project.” Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf. 5. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. In your revised cover letter, please address the following prompts: a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. We will update your Data Availability statement on your behalf to reflect the information you provide. 6. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ. 7. We note that Figure 1 in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright. We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission: 1. You may seek permission from the original copyright holder of Figure 1 to publish the content specifically under the CC BY 4.0 license. We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text: “I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.” Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission. In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].” 2. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: No Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: I Don't Know Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The study evaluates how the addition of procedural data can be used to predict risk of 12 month AF recurrence following radio frequency ablation. The study finds that in addition to AF type duration of ablation and number of lesions in specific regions recorded during the procedure improve prediction of outcome. The study conclusions are consistent with previous studies complete PVI is important and type of AF is important in predicting outcome. Concerns: The use case for this predictive model was not clear. Ideally, prediction of 12 month recurrence would be used to decide to proceed with the ablation. This is the strength of ATLAS and FLAME, which should be recognized as a significant advantage of these approaches. But once the ablation has happened is it expected this could be used for informing advanced monitoring? The expected use case should be identified and commented on in the discussion. The number of ablation cases performed in the hospital network seems to be very low. 300 cases over 3 years for 40 hospitals seems like a very small number. The study should say how many of the AF ablation patients were found overall and why only 300 were studied. There is a risk that this specific group may represent a subset and this is important to consider the results. It was not clear why only CARTO was used for the analysis. None of the indices appeared to be proprietary, this needs to be explained more. The results would be significantly more interesting if they were extended to Abbot, Boston Scientific or Acutus systems It would be good to train the model repeated times with different random seeds/initialization to confirm that the same features are identified. An independent validation data set would significantly improve this study. It was not clear why a train, test and validation set were not used. The analysis could have been done with a cross-fold validation that would give an estimate of the uncertainty in the model AUROC. Is the difference between 0.66 and 0.78 real or spurious based on the specific learning instance. The abstract conclusion on line 36 is not supported by the results and should probably be changed to the discussion line 218: The model…. Are the WACA statistics confounded with the other measurements, how would this impact results? It would be useful to include LA size in the prediction as this is often available and provides good pre-procedure outcome prediction. The large limitation is that this is only applied to a single vendor data set. This needs to be highlighted in the discussion. Minor The ability to deliver force to the right inferior atrium may be dependent on operator experience. It would be nice to add this to the discussion line 214 determine => determines Reviewer #2: Interesting data. Main limitation is the relatively small sample size for a ML study Main limitation is the relatively small sample size for a ML study Main limitation is the relatively small sample size for a ML study ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0300309.r001
Revision 1
13 Feb 2024 Author Response Reviewer #1: The study evaluates how the addition of procedural data can be used to predict risk of 12 month AF recurrence following radio frequency ablation. The study finds that in addition to AF type duration of ablation and number of lesions in specific regions recorded during the procedure improve prediction of outcome. The study conclusions are consistent with previous studies complete PVI is important and type of AF is important in predicting outcome. Concerns: The use case for this predictive model was not clear. Ideally, prediction of 12 month recurrence would be used to decide to proceed with the ablation. This is the strength of ATLAS and FLAME, which should be recognized as a significant advantage of these approaches. But once the ablation has happened is it expected this could be used for informing advanced monitoring? The expected use case should be identified and commented on in the discussion. Author response: Thanks for suggestion. As pointed out by the reviewer, the Female, Long-Lasting, Atrial diameter, Mitral, Extreme (FLAME) score was developed to predict the outcome of non-paroxysmal atrial fibrillation (AF) ablation using patients' demographic and clinical variables. The FLAME score provides clinicians with a tool to analyze individual patients' risk for ablation outcomes. Similarly, ATLAS is designed to estimate the risk of AF recurrence after the first catheter ablation procedure, incorporating patients' age, gender, AF type, left atrial volume, and smoking status. Both FLAME and ATLAS can be used to refine patient selection for ablation procedures and inform patient-physician discussions. The success of the ablation procedure depends on patients' demographic and clinical profiles, as well as the ablation strategies (e.g., power target, lesion duration, ablation index) and techniques (e.g., contact force sensing catheter, catheter irrigation, high power short duration techniques) used during ablation procedures. Our tool can be integrated into the cardiac ablation technology system to provide real-time information to electrophysiologists during the procedure, enabling them to achieve better results. For instance, similar to the advanced technologies in modern automobiles that alert drivers when they deviate from their lane or approach a high-risk collision, our tool is designed to improve ablation effectiveness for electrophysiologists. We aim to develop tools that inform surgeons during cardiac ablation procedures about the probability of recurrence based on the ablation patterns used for patients. Additionally, we seek to offer suggestions for improving ablation locations, the number of lesions, contact force, impedance drop, voltage drop, etc., which collectively contribute to a lower probability of atrial fibrillation (AF) recurrence for patients. Similar predictive models can be developed to guide electrophysiologists on reducing adverse events for patients. These models utilize variables from CARTO data, such as contact force, power, and duration used for lesion formation. Our overarching research goal is to enhance the benefit-risk balance of cardiac ablation procedures for patients by leveraging machine learning models running on cardiac ablation data. Importantly, our objective is not to replace the surgeon's clinical judgment but rather to provide valuable input to guide the radiofrequency ablation workflowIn comparison to ATLAS and FLAME, our predictive model incorporates procedural data during the ablation process to predict AF recurrence. The predictive value of procedural data was demonstrated in this study. The predictive model can inform patients and clinicians about the risk of AF recurrence after ablation. The model also identifies areas for improvement in the ablation procedure and provides directions to enhance ablation strategies. The number of ablation cases performed in the hospital network seems to be very low. 300 cases over 3 years for 40 hospitals seems like a very small number. The study should say how many of the AF ablation patients were found overall and why only 300 were studied. There is a risk that this specific group may represent a subset and this is important to consider the results. It was not clear why only CARTO was used for the analysis. None of the indices appeared to be proprietary, this needs to be explained more. The results would be significantly more interesting if they were extended to Abbot, Boston Scientific or Acutus systems Author response: We agreed with the reviewer that the study would be more comprehensive and interesting if we could include patients using Abbott, Boston Scientific, or Acutus systems. However, we do not have access to a broader array of medical device data. There are two primary reasons for this limitation. First, Mercy Health primarily utilizes Medtronic and Thermocool catheters. Second, Mercy's data use agreement prohibits the sharing of data from one manufacturer to another. Consequently, we were unable to obtain procedural data from devices manufactured by other companies. As a result, only procedural data from the CARTO system are available to us. We acknowledge this limitation in our study and hope that our work will inspire other researchers to conduct similar studies using data from systems produced by different manufacturers. We have included this information in the limitations section. This study focused on patients treated using Biosense Thermocool SmartTouch (ST) and Thermocool ST SurroundFlow (SF) and the CARTO machine with software version 6.0 or above. To link CARTONET data with Mercy Health electronic health records (EHR), nurses from Mercy Health reviewed the cases and assigned the case ID in the procedural case data. The inclusion criterions and data linkage process reduced the number of eligible cases for this study. It would be good to train the model repeated times with different random seeds/initialization to confirm that the same features are identified. Author response: We used the AutoGluon-Tabular framework, a type of automated machine learning (AutoML), to develop predictive models. AutoGluon ensembles multiple models and stacks them in multiple layers. AutoGluon utilizes the random forest, CatBoost, LightGBM, and Neural Net models as the base models and ensembles/stacks them in multiple layers. Bagging and boosting have already been used in the model development process. Because the steps what we have done, incremental benefits from different random seeds/initialization will be minimal. Feature importance was calculated on the test data after the model was finalized. An independent validation data set would significantly improve this study. It was not clear why a train, test and validation set were not used. The analysis could have been done with a cross-fold validation that would give an estimate of the uncertainty in the model AUROC. Is the difference between 0.66 and 0.78 real or spurious based on the specific learning instance. Author response: In this study, we had 306 patients, with 70% used for model training and 30% for model testing. This division allows us to accurately assess the model performance on test data with a large dataset. We used the default settings for the AutoGluon framework. The autogluon.TabularPredictor.fit() function automatically selects 20% of the training data for validation and hyperparameter fine-tuning. Our model development process is straightforward and well-defined, reducing the chance of data leakage in the model development process. The Area Under the Receiver Operating Characteristic curve (AUROC) was calculated based on the test data after the model, both with and without procedure data, was finalized. We acknowledge external validation using data from other health stems should be conducted to validate the performance and generalizability of model. We add this in the limitation section. The abstract conclusion on line 36 is not supported by the results and should probably be changed to the discussion line 218: The model…. Author response: we revised the conclusion following suggestion: The model could be used for identification of patients with high risk of AF recurrence post ablation. Linkage between CARTO procedural data and patient EHR data is valuable to optimize ablation process and improve patient outcomes. Are the WACA statistics confounded with the other measurements, how would this impact results? Author response: Right WACA statistics are calculated as the sum or average of ablation parameters from lesions in the right roof, right posterior, right inferior, and right anterior. Meanwhile, left WACA statistics are calculated as the sum or average of ablation parameters from lesions in the left roof, left posterior, left inferior, left anterior, and ridge. As shown in Table 4, there are no ablation parameters from right and left WACA in the top 10 important features from the final model. Therefore, their absence has minimal impact on the results. It would be useful to include LA size in the prediction as this is often available and provides good pre-procedure outcome prediction. Author response: The left atrium size is not available in the output of CARTONET as it is not recorded as part of CARTO VISITAG. The CARTO VISITAG™ Module records catheter positions along with electrophysiological parameters acquired during RF applications, according to user preferences. The large limitation is that this is only applied to a single vendor data set. This needs to be highlighted in the discussion. Author response: The procedural data comes from CARTONET, which only stored the CARTO system data. The data is from a large health care. Unfortnatley, we can not get data from other manufactures due to data use agreement. We highlighted this limitation in the discussion. Minor The ability to deliver force to the right inferior atrium may be dependent on operator experience. It would be nice to add this to the discussion. Author response: Thanks for the suggestion. We add that in the discussion. Line 214 determine => determines Author response: revised. Reviewer #2: Interesting data. Main limitation is the relatively small sample size for a ML study Author response: This is an observational study that involves linking two different data sources. Procedural data comes from CARTONET, requiring the use of Thermocool ST and STSF catheters, as well as the CARTO system during the ablation process. Nurses from the Mercy Health System review individual cases and assign patient IDs to facilitate linking CARTONET procedural data with Mercy Health electronic health records. The inclusion criteria and data linkage process reduce the number of eligible cases for this study. We allocated 70% of patients for model training and 30% for testing data. This division enables us to accurately assess the model's performance on test data with a large dataset. Our model development process is straightforward, reducing the chance of data leakage during model development. Attachments Attachment Submitted filename: Response letter to reviewer comments.docx https://doi.org/10.1371/journal.pone.0300309.r002
27 Feb 2024 Decision Letter - Ibrahim Marai, Editor Does radiofrequency ablation procedural data improve the accuracy of identifying atrial fibrillation recurrence? PONE-D-23-31440R1 Dear Dr. Coplan, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Ibrahim Marai, MD Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: https://doi.org/10.1371/journal.pone.0300309.r003
Formally Accepted
27 Mar 2024 Acceptance Letter - Ibrahim Marai, Editor PONE-D-23-31440R1 PLOS ONE Dear Dr. Coplan, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Ibrahim Marai Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0300309.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .