Dear PhD Student Gao,

Thank you very much for submitting your manuscript "Early detection of disease outbreaks and non-outbreaks using incidence data: A framework using feature-based time series classification and machine learning" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

Thanks for the submission! Both reviewers acknowledged that combining machine learning (ML) and dynamical systems approaches to better forecast epidemics is an interesting approach. However, they raised major concerns with the work. Please address the reviewers' comments especially the ones regarding the uncertainty quantification of model forecasts (in my opinion, a very relevant issue in practical use of the model) and the reliability of the model trained on synthetic datasets (i.e. a common problem for ML models due to lack of large real-world datasets).

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Ruian Ke

Guest Editor

PLOS Computational Biology

Thomas Leitner

Section Editor

PLOS Computational Biology

***********************

Thanks for the submission! Both reviewers acknowledged that combining machine learning (ML) and dynamical systems approaches to better forecast epidemics is an interesting approach. However, they raised major concerns with the work. Please address the reviewers' comments especially the ones regarding the uncertainty quantification of model forecasts (in my opinion, a very relevant issue in practical use of the model) and the reliability of the model trained on synthetic datasets (i.e. a common problem for ML models due to lack of large real-world datasets).

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors propose a machine learning framework to predict disease outbreak. They train machine learning classifiers on simulation data from a stochastic SIR model. They generate a bifurcation class, where the model is taken through a transcritical bifurcation, and a null class, where the model does not pass through a bifurcation. The classifiers are trained on the binary classification problem of distinguishing these classes. They perform with very high accuracy on a withheld simulation test set. When evaluated on real data from Covid and SARS, 2 out of the 32 trained models perform with high accuracy.

This paper is written well, has clean figures and a good, logical structure. The subject of the study—combining machine learning and dynamical systems to predict disease outbreaks—is interesting and timely. The combinatorial analysis of the different classifiers/noise/features is thorough, as well as the investigation with respect to rolling vs. expanding windows. Overall, the analysis of the approach using model simulations is well done.

However, I am not convinced that this approach, in its current state, is effective on real data. Figure 4 shows that the majority of the trained classifiers perform no better than random (an accuracy of around 0.5 when combining the empirical datasets to balance the classes). This would suggest that the simulated training set is not sufficiently representative of what is occurring in the real data. The fact that 2 out of the 32 models perform well is not a sufficient indication that generalizable features have been learned from the training set to predict real disease outbreak—-These models may have scored high by chance, given the number of models tested. I would need to see further evidence of the performance of these two classifiers on real data before agreeing with the final statement of the abstract---that there are “statistical features distinguishing outbreak and non-outbreak…in real-world datasets”.

On that note, why not do the Mann-Whitney tests on the features obtained from the real data? Do you observe significant differences between outbreak vs. non-outbreak? Are they as one would expect based on dynamical systems theory? If not, maybe that is the issue here.

Personally, I find the analysis on the synthetic data interesting in itself, and a step forward in developing generalized machine learning approaches for predicting bifurcations. This alone could be an interesting publication in a more specialised journal. If the authors believe that this approach is effective on real data as it stands, then I will need more convincing.

Reviewer #2: The work presented in this manuscript tests the hypothesis that a machine learning model based on features extracted from time series data can be used to forecast whether a transition point in an epidemic will occur. In general, this is an interesting approach which uses machine learning models trained on synthetic data and applies them to real world data. However, my main critique explained in several points below is that the authors train a wide array of models based on choices of features and the machine learning method, but it’s not clear which model should be used in practice. Uncertainty quantification would provide statistical confidence in the predictions, but this is missing from the study.

1) Given that the AUC scores are close to 1, this seems to strongly imply separability of the different classes based on the input features to the machine learning models. If this is the case, it would be useful to visualize this using dimensionality reductions techniques down to 2 or 3 dimensions, e.g., using PCA or t-SNE.

2) I think it would be helpful to plot the noiseless data with the noisy data, e.g., in Figure 1, to visualize how much noise is added to the data.

3) A comparison to noiseless data for the results in Figure 3 would be able to show whether adding noise affects the classification accuracy. I think Figure 3 shows that in some cases, because the AUC is exactly equal to 1, Demographic or White noise doesn’t reduce classification accuracy.

4) Figure 6 shows that the results between classifiers on real world data are heterogeneous. In practice, I don’t think it would be possible to know a priori which classifier would be better to use. Can the authors discuss how can one choose the best classifier?

5) Given that some classifiers achieve 1.0 accuracy, it would be informative if the authors showed the results of training a linear classifier model. In this case, the coefficients of the linear model could be used to interpret which features are driving the high accuracy. Similarly, the authors can report the results of a simple decision tree instead of a gradient boosting model, since the decision tree model will be more interpretable.

6) Can the authors provide uncertainty quantification for the machine learning model predictions? If not, how can one be confident in the prediction?

7) Can the authors provide an explanation/interpretation for why the different types of noise yield different results in figure 7?

8) Model E5L has one of the lowest AUCs in figure 3, and hence one would assume it wouldn’t perform well on real data. However, it has the highest accuracy in figure 6 when testing on real data. How can this be explained?

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Gao,

We are pleased to inform you that your manuscript 'Early detection of disease outbreaks and non-outbreaks using incidence data: A framework using feature-based time series classification and machine learning' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Ruian Ke

Guest Editor

PLOS Computational Biology

Thomas Leitner

Section Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors have addressed my concerns by evaluating their best-performing models on other disease data. I'm happy to recommend publication.

Reviewer #2: Thanks to the authors for addressing all of my previous concerns and comments.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No