Dear Mr. Borisov,

Thank you very much for submitting your manuscript "Confidence intervals by constrained optimization – an algorithm and software package for practical identifiability analysis in Systems Biology" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. 

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Daniel A Beard

Deputy Editor

PLOS Computational Biology

Daniel Beard

Deputy Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The article is very clearly written. The authors lay out a timeline of the developments of the field that to me seems fairly complete and leads directly to the new method as a substantial increase in the field. The motivation for focusing on the structural identifiability is made fairly clear. The method is well-described and after reading it, it's clear that it should work. The evidence then demonstrates that it does work. I can easily see this method and this package being used by many researchers in practice.

That said, there are some improvements that should probably be made to the paper before publication. For one, I think that section 3.4 is unnecessary. I think it's fairly clear that this kind of numerical method needs to be computed on some finite support so practically all determinations are going to be made in some box. I don't think that more than a sentence or a paragraph is really required to get that point across. Secondly, the paper itself doesn't seem to have a lot of the validation. One example is used as validation, but the paper needs more. When I look at the package they discuss, I can see 5 clear examples with Binder links that demonstrate the method on more systems: some of this should be in the paper instead of 3.4 in order to more broadly demonstrate the validity of this method. Next, what they established was "structural efficiency", i.e. efficiency in terms of likelihood function evaluations. But it would've been nice to also see "practical efficiency", i.e. raw timings for the MATLAB method and Julia and Python implementation of the new methods, and use this to demonstrate a clear orders of magnitude actual performance improvement. Overall I think it's a really good paper, a good idea, and a strong result with just some touch-ups requires to really hammer home the advance in a more clear way.

Reviewer #2: This article presents a novel method to study practical identifiability of parameters of ODE-based models. The method is innovative and seems to overcome existing methods in terms of computational cost, at least in the presented example. It can definitely be useful for the Research community, especially since the authors have made it freely available either in Julia or in Python. The article is very clear and well written. It cites all relevant literature. I have three minor comments:

-Equation 7 : precise the values for j, to make clearer the fact that this is a system of more than 2 equations.

-Equation 8: it is not obvious how the authors transformed system (7) into system (8). More explanations are needed here since this is key to understand the algorithm. Are the systems strictly equivalent? In the definition of c, the authors need to precise the position of the “1” in the vector.

-The authors claim in the Abstract and Introduction that their method provides more accurate estimation of confidence Interval bounds. However, this is not demonstrated in the article, neither theoretically, nor computationally (On the opposite, they do provide some evidence of the lower computational cost of their algorithm compared to existing ones). Please either add the corresponding evidence or modify the text.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Christopher Rackauckas

Reviewer #2: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, PLOS recommends that you deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see http://journals.plos.org/ploscompbiol/s/submission-guidelines#loc-materials-and-methods

Dear Mr. Borisov,

We are pleased to inform you that your manuscript 'Confidence Intervals by Constrained Optimization – an Algorithm and Software Package for Practical Identifiability Analysis in Systems Biology' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Daniel A Beard

Deputy Editor

PLOS Computational Biology

Daniel Beard

Deputy Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors have addressed my previous concerns and demonstrate a significant improvement to the practical application of practical identifiability analysis with these new results. In addition, I can confirm that their code, timing, and results on the Julia side are easily reproducible.

Reviewer #2: The authors have answered all my comments.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Chris Rackauckas

Reviewer #2: No