PCOMPBIOL-D-25-00198

IonBench: a benchmark of optimisation strategies for mathematical models of ion channel currents

PLOS Computational Biology

Dear Dr. Mirams,

Thank you for submitting your manuscript to PLOS Computational Biology. After careful consideration, we feel that it has merit but does not fully meet PLOS Computational Biology's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript within 30 days Jul 08 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at ploscompbiol@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcompbiol/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. This file does not need to include responses to formatting updates and technical items listed in the 'Journal Requirements' section below.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, competing interests statement, or data availability statement, please make these updates within the submission form at the time of resubmission. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Frédéric E. Theunissen

Academic Editor

PLOS Computational Biology

Stacey Finley

Section Editor

PLOS Computational Biology

Additional Editor Comments :

Dear Authors,

The two reviewers that were assigned to your manuscript agree that it is an appropriate contribution for Plos Comp Biol. However, they raised some minor to less-minor issues that I would like you to address.

Thank you.

Frederic Theunissen

Journal Requirements:

1) We ask that a manuscript source file is provided at Revision. Please upload your manuscript file as a .doc, .docx, .rtf or .tex. If you are providing a .tex file, please upload it under the item type u2018LaTeX Source Fileu2019 and leave your .pdf version as the item type u2018Manuscriptu2019.

2) Please upload all main figures as separate Figure files in .tif or .eps format. For more information about how to convert and format your figure files please see our guidelines: 

https://journals.plos.org/ploscompbiol/s/figures

3) We notice that your supplementary Figures, Tables, and information are included in the manuscript file. Please remove them and upload them with the file type 'Supporting Information'. Please ensure that each Supporting Information file has a legend listed in the manuscript after the references list.

4) Please amend your detailed Financial Disclosure statement. This is published with the article. It must therefore be completed in full sentences and contain the exact wording you wish to be published.

1) State what role the funders took in the study. If the funders had no role in your study, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

5) Please provide a completed 'Competing Interests' statement, including any COIs declared by your co-authors. If you have no competing interests to declare, please state "The authors have declared that no competing interests exist". Otherwise please declare all competing interests beginning with the statement "I have read the journal's policy and the authors of this manuscript have the following competing interests:"

Reviewers' comments:

Reviewer's Responses to Questions

Reviewer #1: Owen & Mirams present a benchmark study on optimization approaches for parameters in cardiac ion channel models, typically described by several coupled ordinary differential equations. This is an original contribution and will be of high importance to researchers in the field, especially given the well-documented and openly available problem definitions. The authors invested great effort in implementing 34 approaches described in literature to rigorously compare them and describe the methodology clearly.

Based on the results of their benchmark, which provides methodological insight, they propose an innovative adaption of existing approaches that consistently performs best on their set of test problems. All data and code are available with documentation and tests.

I have the following comments that could further increase the value of this study_

- Lines 157ff: The choice of benchmark problems is not obvious:

- What's the rationale behind choosing those 4 specific problems?

- Why were problems 1-4 evaluated by comparing currents whereas 5 was compared in terms of summary statistics? (line 176)

- Noise was added to benchmark problems 1+2 but not to 3-5. Why? Would you suspect the results to change if noise was present (as is inevitably the case in any real-world application of these methods)? Previous studies reported that small levels of noise can even lead to better results for gradient-based optimization whereas performance gets worse for higher noise levels.

- The versatility of the provided code could be increased by exposing the level of noise as a configurable parameter.

- Why are the parameter sampling regions for problem 5 smaller (+/-25%) than for the other problems?

- It would be interesting to see the cost values and runtime achieved with all approaches (even if not undershooting the cost threshold considered "successful") as the small number of approaches (compared to the 34 candidates) shown in Fig. 6 is surprising.

- The suggested optimal algorithm is a multi-start approach. Some guidance on the number of required starts would be helpful for future users of this approach as the achievable cost in real-world scenarios will be markedly above 0.

Further points:

- The relation of the described approaches with machine learning approaches for emulating the AP and for parameter optimization could be further elaborated.

- Fig. 2: Should "RMSE: cost" read "RMSE, cost?"

- Fig. 5 and lines 255ff: I have trouble identifying the lower bound 10^-3 in the figure. Please elaborate the caption or adapt the figure.

- Lines 297ff./404: I consider the information on the unidentifiable parameters very valuable. Please provide more guidance on how to read and interpret the plot in the supplement and consider moving key results (which ones were unidentifiable?) to the main manuscript

- Fig. 7 is a bit hard to read with intersecting lines and overlapping symbols. How about providing this information in addition in a supplementary table?

Reviewer #2: Review of article PCOMPBIOL-D-25-00198

The manuscript discusses a developed open source software suite for evaluating optimization methods for inverting model parameters, using data generated from a number of ion channel mathematical models. The authors use this software to benchmark a wide range of optimization approaches from the literature for these chosen model formulations, and from the results, put forth an improved methodology with greater performance in their test suite. The article is clear, well written, and descriptive, and adds a useful tool for researchers working in this space. It is largely appropriate for publication as written.

There are a few comments I would ask the authors to address though, to help add clarity to the results and conclusions.

1. It was somewhat surprising that so many (the vast majority) of the published methods were unable to effectively minimize the cost for any of the chosen formulations. I would ask for some discussion if this finding is consistent with the cited publications on these methods, or if it is a function of the methods chosen in their developed benchmark system? In other words, were these published methods able to solve similar problems elsewhere, but not in your benchmark?

2. The choice of model set-up could be better discussed, in particular the choice of noise. Why was this specific level of noise added, and why to only a few of the model formulations? The choice seems a little arbitrary, and yet could potentially be having a strong effect on the results.

3. All benchmarks are shown in terms of cost and time. I think the results would be well supplemented with additionally showing parameter error, as this in the end is the actual goal of this type of optimization process. This obviously could show poor results in the case where there are so many parameters that are unidentifiable, but I think the reader can contextualize it sufficiently.

4. In the profile curves, I was uncertain of the difference between the ‘optimized’ and ‘unoptimized’ profiles.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

Figure resubmission:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. If there are other versions of figure files still present in your submission file inventory at resubmission, please replace them with the PACE-processed versions.

Reproducibility:

To enhance the reproducibility of your results, we recommend that authors of applicable studies deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Professor Mirams,

We are pleased to inform you that your manuscript 'IonBench: a benchmark of optimisation strategies for mathematical models of ion channel currents' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Frédéric E. Theunissen

Academic Editor

PLOS Computational Biology

Stacey Finley

Section Editor

PLOS Computational Biology

***********************************************************

Dear Matt and Gary,

Thank you for addressing all the reviewers comments. I found your thorough review of published optimization methods and your description of the strengths and weaknesses very useful. I hope that your paper becomes a reference for all researchers modeling ion channels.

Best,

Frederic Theunissen