Dear Mr. Thomas,

Thank you very much for submitting your manuscript "Model-checking ecological state-transition graphs" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

In this manuscript, the authors outline a methodological approach for studying the dynamics of discrete systems. They present the method in the context of ecosystem state transitions, e.g., recovery from a perturbation event like a fire. Given the longstanding interest in these processes, I suspect the approach will be of interest to the field and the interdisciplinary nature of the work is a strength. However, I strongly agree with reviewer 2 that the work does not fairly characterize its novelty nor its distinct contributions from past efforts. While I do not think novelty is the only criteria for a methods paper (and indeed nothing is truly novel in the strict sense) it's critical that the context surrounding the work be clearly addressed and the specific additions this method brings to the field articulated and defended with evidence. In addition, I believe the paper could be made much stronger by considering a specific data set and/or experimental system that might be studied using this approach. Given the rise in "big data" for ecosystems, demonstrating the utility of this method for such data would be of broad interest.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Samuel V. Scarpino

Associate Editor

PLOS Computational Biology

James O'Dwyer

Deputy Editor

PLOS Computational Biology

***********************

In this manuscript, the authors outline a methodological approach for studying the dynamics of discrete systems. They present the method in the context of ecosystem state transitions, e.g., recovery from a perturbation event like a fire. Given the longstanding interest in these processes, I suspect the approach will be of interest to the field and the interdisciplinary nature of the work is a strength. However, I strongly agree with reviewer 2 that the work does not fairly characterize its novelty nor its distinct contributions from past efforts. While I do not think novelty is the only criteria for a methods paper (and indeed nothing is truly novel in the strict sense) it's critical that the context surrounding the work be clearly addressed and the specific additions this method brings to the field articulated and defended with evidence. In addition, I believe the paper could be made much stronger by considering a specific data set and/or experimental system that might be studied using this approach. Given the rise in "big data" for ecosystems, demonstrating the utility of this method for such data would be of broad interest.

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The submitted manuscript promotes the model checking technique in the area of ecological systems. The article is interesting to read, however some issues are for me not clear. Below I include point-by-point my comments:

1. Line 77: what is “EDIT STMs database”? It should be revised.

2. Line 143 Fig. 3: it should be “whether” instead of “wether”.

3. Line 257: a short technical specification of the computer that has been used should be given (including CPU, RAM, etc.).

4. Line 258 Table 3: how are the control variables embedded? It is not clear from the rest of the article.

5. Line 261: “Thus the STGs drawn below (…)” -> where?

6. In general – section 2: must be extended to include more information. It is now a little bit confusing.

7. In general – section 4: please point out the contributions of the paper.

8. In general – section 4: what are the limitations of applying model checking to ecological STGs? Appropriate information should be added.

Reviewer #2: The authors propose an interesting approach of using model-checking for scenario analysis of ecological systems.

A- General remarks on the positioning of the article:

-----

Several key points regarding the positioning of this work:

1) This approach is not new and has already been published in the following articles:

C. Largouët, M.-O. Cordier, Y.-M. Bozec, Y. Zhao, G. Fontenelle. Use Of Timed Automata And Model-Checking To Explore Scenarios On Ecosystem Models, Environmental Modelling and Software, Elsevier, 2011

M.-O. Cordier, C. Largouët, Y. Zhao. Model-Checking an Ecosystem Model for Decision-Aid. IEEE 26th International Conference on Tools with Artificial Intelligence, 2014

The authors claim that the above mentioned work is only applied in the field of agronomy, which is absolutely not true (since it is indeed about ecosystems as presented here) and that it was used in a "model-checking mindset" (what does that really mean?).

The above mentioned work was done in collaboration with ecologists (co-authors of the publication) using model-checking, query patterns, scenario testing and applied on real applications with validation of the results. This is exactly the approach that the authors wish to develop here.

The authors are asked to situate their own approach in the continuity of these previous works, precursors of the domain, in the introduction section, and to place this new work with respect to the state of the art which is not inexistent contrary to what is stated.

The authors are also asked to clearly specify where the novelty of this work lies in relation to what has already been published (which is currently unconvincing in the actual version of this paper).

2) The approach proposed by the authors is ambiguous. Several times, they defend themselves from using a "computer science" point of view and exempt themselves from giving any formal definitions. However, several concepts are difficult to grasp without more information and globally the proposed approach appears to be too vague and not enough scientifically sound.

Moreover, the aspects related to the application are not sufficiently developed in the results part. The application is only inspired by the study of a previous publication, is described as a "proof of concept" and does not propose any validation.

I therefore question the applicative side claimed by this submission.

B- General remarks on the methodology:

------

1) Choice of the model representation formalism:

Can you please give a formal definition of the STG formalism? Explain if you have defined yourself this formalism, if so how does it differ from existing approaches and what is its interest? If not, give the author's reference. In the source code, this STG seems to be a simple graph, explain if this is the case? Is it a discrete event system formalism? The definition of events is unclear. The non-determinism (L69) stated does not seem to be correct since the events on the arcs are not the same, then it is not a real case of non-determinism as usually defined.

2) Choice of the passage by the formalism of the rules:

On the method, why is it necessary to go through a rule-based system if we already have the functionning graph (Figure 2 left) as input?

- As for the STG model, the formalism of the rule-based system is not given. The article must be self-content without having to refer to another publication. It is not a usual production rule-based system which is disturbing. Otherwise, rule R1 could be triggered directly from state 1 (Gr+, Sh-, Tr-) since the premise Gr+ is consistent and is the only condition for R1 to be invoked. The R2 rule subsumes the R1 rule. How is this handled? From a practical point of view, it seems quite surprising that an expert can state both rule R1 and rule R2 (considering that we don't have the left graph). Please develop all these points to emphasize the interest of using rules in your methodology.

- How are the STG states created from the graph? I don't see how it is possible to create the STG states from these 4 rules without knowing them a priori. The logic of this step must absolutely be explained.

- In conclusion on this part, it seems to me that this rule-based system is more a graph description language than a real functioning description language.

Please explain this aspect more clearly and position yourself in relation to domain-specific languages DSL.

Reference: Fowler M, Parsons R. Domain-specific languages. Upper Saddle River, NJ: Addison-Wesley; 2011.

3) On model-checking:

- If the initial modeling language is a rule-based system why using model-checking approaches and not SAT ones that are relying on logic? Justify the choice of this approach?

Explain what is a "computation tree" even though you are in a symbolic representation (not much explained except in the implementation part when this choice appears - I suppose - as the real choice of your method).

C- General remarks on the results:

-----

This results part is very poor and must absolutely be completed if possible with a real application and at least a validation.

In particular in this part a description of the model is missing. The interest of having a STG is to be able to show a part of the provided model. It is impossible to understand what the model looks like which is very annoying for a paper that proposes to focus on ecological modeling. The "proof of concept" is not sufficient.

- The part related to the obtained model should be better presented.

It would be interesting to have a figure presenting and explaining a part of the obtained model

Where do the 1141 states come from (we simply understood that there were 8 variables and 7 controls) and why are the graphs disconnected?

- "Being fully symbolic, our implementation is 236 highly scalable and can handle models with millions of states." Did you perform scalability analysis before wrtiing such a statement? There is a need for more scientific support.

- Can the answers of model-checking queries be compared with the results obtained by Liao (type of queries, results, response time, etc.)? How can you be sure that the yes/no results are correct? The validation of the approach needs to be investigated.

D- General remarks on the discussion:

-----

The discussion, like the paper, but even more in this section, does not know how to position itself towards its readers. It proposes many concepts that are too difficult to grasp for an audience that is not in the computer science field. For more informed people, it is far too vague and not scientifically sound enough. Please find some examples here that should be explicited :

- What is management policies defined as controls? And what is it "without control"? This point should have been presented before and defined deeper in the methodology part if considered of paramount importance for the authors.

- No, probabilities do not make a system hybrid (and still questionnable for time).

- "Thus they offer fewer analysis capabilities and fewer software tools exist to support them. In general, the more expressive a particular setting is, the less it can be analyzed (because of computational complexity and tools availability), and some questions even become undecidable (i.e. no algorithm can answer them)." Moderate these assertions or support them with references.

Some problems are also undecidable with CTL so the argument is difficult to hold.

This section should be reworked in a more scientific manner.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Mr. Thomas,

We are pleased to inform you that your manuscript 'Model-checking ecological state-transition graphs' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Samuel V. Scarpino

Associate Editor

PLOS Computational Biology

James O'Dwyer

Deputy Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: This is a revised version of the manuscript that I have reviewed before. I appreciate the efforts of the authors to improve the quality of the article. I am satisfied with responses to my previous review as well as with changes in the manuscript.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No