Peer Review History

Original SubmissionSeptember 5, 2019
Decision Letter - Roger Dimitri Kouyos, Editor, Virginia E. Pitzer, Editor

Dear Dr Hazelbag,

Thank you very much for submitting your manuscript 'Fitting individual-based models to data in HIV, tuberculosis and malaria: a systematic review' for review by PLOS Computational Biology. Your manuscript has been fully evaluated by the PLOS Computational Biology editorial team and in this case also by independent peer reviewers. The reviewers appreciated the attention to an important problem, but raised some substantial concerns about the manuscript as it currently stands. While your manuscript cannot be accepted in its present form, we are willing to consider a revised version in which the issues raised by the reviewers have been adequately addressed. We cannot, of course, promise publication at that time.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

Your revisions should address the specific points made by each reviewer. Please return the revised version within the next 60 days. If you anticipate any delay in its return, we ask that you let us know the expected resubmission date by email at ploscompbiol@plos.org. Revised manuscripts received beyond 60 days may require evaluation and peer review similar to that applied to newly submitted manuscripts.

In addition, when you are ready to resubmit, please be prepared to provide the following:

(1) A detailed list of your responses to the review comments and the changes you have made in the manuscript. We require a file of this nature before your manuscript is passed back to the editors.

(2) A copy of your manuscript with the changes highlighted (encouraged). We encourage authors, if possible to show clearly where changes have been made to their manuscript e.g. by highlighting text.

(3) A striking still image to accompany your article (optional). If the image is judged to be suitable by the editors, it may be featured on our website and might be chosen as the issue image for that month. These square, high-quality images should be accompanied by a short caption. Please note as well that there should be no copyright restrictions on the use of the image, so that it can be published under the Open-Access license and be subject only to appropriate attribution.

Before you resubmit your manuscript, please consult our Submission Checklist to ensure your manuscript is formatted correctly for PLOS Computational Biology: http://www.ploscompbiol.org/static/checklist.action. Some key points to remember are:

- Figures uploaded separately as TIFF or EPS files (if you wish, your figures may remain in your main manuscript file in addition).

- Supporting Information uploaded as separate files, titled Dataset, Figure, Table, Text, Protocol, Audio, or Video.

- Funding information in the 'Financial Disclosure' box in the online system.

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see here

We are sorry that we cannot be more positive about your manuscript at this stage, but if you have any concerns or questions, please do not hesitate to contact us.

Sincerely,

Roger Dimitri Kouyos

Associate Editor

PLOS Computational Biology

Virginia Pitzer

Deputy Editor

PLOS Computational Biology

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Dear authors,

Please find the attached review.

Reviewer #2: This systematic review used HIV, Malaria, and Tuberculosis studies to provide an overview of the fitting methods used in IBMs modelling infectious disease spread. This is critical as the usage of IBMs becomes more common nowadays than the deterministic models. The calibration of IBMs to data is often a challenge. Overall, this is an interesting and well-written paper. I have some comments that need to be clarified.

1. Line 98: In addition to the stochasticity, the number of parameters to estimate in IBM indicates the complexity of the parameter-search strategy (i.e. estimating only one parameter is the search on one-dimensional real space (1D), estimating two parameters is the search on two-dimensional real space (2D), and so on). The authors may comment in the introduction to show how the complexity of calibration varies between IBMs.

2. Line 144: It is better to provide a reference for the most investigated epidemics.

3. Line 155: Is there any reasons why the authors did the search from 2013?

4. Line 179: “the goal” need to be clarified along this line.

5. Fig 2 lists the different methods of parameter search strategy found, but without minimal explanations. It is better to include in the appendix a table that explains briefly these methods for non-modelers.

6. Line 258: “…while the rest either relied on visual inspection as a GOF (14 articles) or used a quantitative GOF (five articles)...” The authors should discuss why some studies use manual fitting instead of formal parameter-search strategies. This manual fitting is impossible in case of multi-fitting parameters. Another issue of manual fitting is not reproducible due to the stochasticity in IBM.

7. Line 300: A reference along this line is very useful for the simulation tools.

8. This systematic review provides interesting data about the methods of calibration of IBMs, but it did not provide a comparison, which method is best, or what is the strength and limitations of each method. The authors should clarify this point in the limitation section.

9. Minor comment: There is an issue in a cell in Table 1.

Reviewer #3: The authors did a tedious and very valuable work in order to identify articles that used an individual-based model (IBM) to fit to data in HIV, tuberculosis (TB) and malaria, and assess the proportion of them that reported the parameter-search strategy and the type of parameter-search strategy used. This work is particularly important as one aims for more transparency on the optimization methods used in modelling works. However, the authors could go a little bit further in order to answer questions such as:

• Does the proportion of articles reporting the parameter search strategy used vary according to the disease studied (HIV, TB, malaria)?

• What about uncertainty? Is the uncertainty related to the parameter-search strategy (e.g. confidence interval or credibility interval) reported? Does the proportion of article reporting uncertainty depend on the field (i.e. HIV, TB, malaria) or on the parameter-search strategy used (sampling or optimization strategy)?

• How many parameters are estimated in each study? Does this number depend on the parameter search strategy? Are the most complex models (i.e. the ones with the highest number of parameter to estimate) the ones that do not report uncertainty/search strategy?

Answering these questions could help better identify the articles that report searching method less frequently (e.g. assess whether it is related to the field studied). Reporting information about the number of parameters estimated and whether uncertainty is reported could also provide a wider understanding of the issue of lack of transparency that we face in some fields.

In addition, I have a few minor comments:

• Lines 49-54: This is mainly repetition of the results already presented in the previous paragraph. Consider removing this paragraph and replacing by a discussion of the results, e.g. something that looks like the second part of “Author summary” (lines 65-72).

• Line 98-99: To me, it is not clear why a greater complexity could make exact likelihood calculation impossible. I would rather say that greater complexity prevent from identifying the exact maximum likelihood estimator, but it should not prevent the model to calculate the exact likelihood.

• Line 129: What kind of stochasticity do you mention here? Would it not be relevant to report stochasticity in your systematic review as well?

• Line 155: Why only from 2013 and not before?

• Line 166: Maybe you could mention before (in abstract?) that you will focus on studies informing public health policy.

• Line 241: In the “GOF” column of Table 1, there is some formatting problem, as we can not read the whole text in the “Suen et al.” column.

• Line 271: It is not clear how you could obtain a weight (i.e. a number) with the ”inverse of the confidence interval” which are two numbers.

• Lines 309-323: Consider rewriting this paragraph, as, as it is now, it repeats what you already said in the “Results” section. I see the point of summarizing the results, but this could be in a more concise way.

• Line 377: Was it not 48% (40/84) before, instead of 49%?

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: H.H. Ayoub

Reviewer #3: No

Attachments
Attachment
Submitted filename: reviewer.docx
Revision 1

Attachments
Attachment
Submitted filename: Rebuttal.docx
Decision Letter - Roger Dimitri Kouyos, Editor, Virginia E. Pitzer, Editor

Dear Dr. Hazelbag,

Thank you very much for submitting your manuscript "Calibration of individual-based models to epidemiological data: a systematic review" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers raised only one minor issue still requires attention. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. 

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Roger Dimitri Kouyos

Associate Editor

PLOS Computational Biology

Virginia Pitzer

Deputy Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Good job. I have no additional comments.

Reviewer #2: The authors have satisfactorily responded to all my questions and made the necessary changes to the manuscript.

Reviewer #3: All the previously reported issues have been addressed by the authors. The authors might however check the following issue. In Fig3, title (and legend) mentioned that the figure reports the number of target statistics, while the y-axis label mentions the number of calibrated parameters. This must be corrected. Additionally, the authors could present both figures (number of targets and number of calibrated parameters according to the parameter search strategy) side by side in the main manuscript.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, PLOS recommends that you deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see http://journals.plos.org/ploscompbiol/s/submission-guidelines#loc-materials-and-methods

Revision 2

Attachments
Attachment
Submitted filename: Response_to_reviewers.docx
Decision Letter - Roger Dimitri Kouyos, Editor, Virginia E. Pitzer, Editor

Dear Dr. Hazelbag,

We are pleased to inform you that your manuscript 'Calibration of individual-based models to epidemiological data: a systematic review' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Roger Dimitri Kouyos

Associate Editor

PLOS Computational Biology

Virginia Pitzer

Deputy Editor

PLOS Computational Biology

***********************************************************

Formally Accepted
Acceptance Letter - Roger Dimitri Kouyos, Editor, Virginia E. Pitzer, Editor

PCOMPBIOL-D-19-01519R2

Calibration of individual-based models to epidemiological data: a systematic review

Dear Dr Hazelbag,

I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course.

The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript.

Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

Thank you again for supporting PLOS Computational Biology and open-access publishing. We are looking forward to publishing your work!

With kind regards,

Laura Mallard

PLOS Computational Biology | Carlyle House, Carlyle Road, Cambridge CB4 3DN | United Kingdom ploscompbiol@plos.org | Phone +44 (0) 1223-442824 | ploscompbiol.org | @PLOSCompBiol

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .