PCOMPBIOL-D-24-01620

Putting computational models of immunity to the test - an invited challenge to predict B.pertussis vaccination responses

PLOS Computational Biology

Dear Dr. Shinde,

Thank you for submitting your manuscript to PLOS Computational Biology. After careful consideration, we feel that it has merit but does not fully meet PLOS Computational Biology's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript within 30 days Feb 03 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at ploscompbiol@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcompbiol/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. This file does not need to include responses to formatting updates and technical items listed in the 'Journal Requirements' section below.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, competing interests statement, or data availability statement, please make these updates within the submission form at the time of resubmission. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Jessica M. Conway

Academic Editor

PLOS Computational Biology

Rob De Boer

Section Editor

PLOS Computational Biology

Feilim Mac Gabhann

Editor-in-Chief

PLOS Computational Biology

Jason Papin

Editor-in-Chief

PLOS Computational Biology

Journal Requirements:

1) Please ensure that the CRediT author contributions listed for every co-author are completed accurately and in full.

At this stage, the following Authors/Authors require contributions: Pramod Shinde, Lisa Willemsen, Michael Anderson, Minori Aoki, Saonli Basu, Julie G Burel, Peng Cheng, Souradipto Ghosh Dastidar, Aidan Dunleavy, Tal Einav, Jamie Forschmiedt, Slim Fourati, Javier Garcia, William Gibson, Jason A Greenbaum, Leying Guan, Weikang Guan, Jeremy P Gygi, Brendan Ha, Joe Hou, Jason Hsiao, Yunda Huang, Rick Jansen, Bhargob Kakoty, Zhiyu Kang, James J Kobie, Mari Kojima, Anna Konstorum, Jiyeun Lee, Sloan A Lewis, Aixin Li, Eric F Lock, Jarjapu Mahita, Marcus Mendes, Hailong Meng, Aidan Neher, Somayeh Nili, Lars Rønn Olsen, Shelby Orfield, James Overton, Nidhi Pai, Cokie Parker, Brian Qian, Mikkel Rasmussen, Joaquin Reyna, Eve Richardson, Sandra Safo, Josey Sorenson, Aparna Srinivasan, Nicola Thrupp, Rashmi Tippalagama, Raphael Trevizani, Steffen Ventz, Jiuzhou Wang, Cheng-Chang Wu, Ferhat Ay, Barry Grant, Steven H Kleinstein, and Bjoern Peters. Please ensure that the full contributions of each author are acknowledged in the "Add/Edit/Remove Authors" section of our submission form.

The list of CRediT author contributions may be found here: https://journals.plos.org/ploscompbiol/s/authorship#loc-author-contributions

2) We ask that a manuscript source file is provided at Revision. Please upload your manuscript file as a .doc, .docx, .rtf or .tex. If you are providing a .tex file, please upload it under the item type u2018LaTeX Source Fileu2019 and leave your .pdf version as the item type u2018Manuscriptu2019.

3) Please provide an Author Summary. This should appear in your manuscript between the Abstract (if applicable) and the Introduction, and should be 150-200 words long. The aim should be to make your findings accessible to a wide audience that includes both scientists and non-scientists. Sample summaries can be found on our website under Submission Guidelines:

https://journals.plos.org/ploscompbiol/s/submission-guidelines#loc-parts-of-a-submission

4) Please upload all main figures as separate Figure files in .tif or .eps format. For more information about how to convert and format your figure files please see our guidelines: 

https://journals.plos.org/ploscompbiol/s/figures

5) Some material included in your submission may be copyrighted. According to PLOSu2019s copyright policy, authors who use figures or other material (e.g., graphics, clipart, maps) from another author or copyright holder must demonstrate or obtain permission to publish this material under the Creative Commons Attribution 4.0 International (CC BY 4.0) License used by PLOS journals. Please closely review the details of PLOSu2019s copyright requirements here: PLOS Licenses and Copyright. If you need to request permissions from a copyright holder, you may use PLOS's Copyright Content Permission form.

Please respond directly to this email and provide any known details concerning your material's license terms and permissions required for reuse, even if you have not yet obtained copyright permissions or are unsure of your material's copyright compatibility. Once you have responded and addressed all other outstanding technical requirements, you may resubmit your manuscript within Editorial Manager. 

Potential Copyright Issues:

i) Figure 1. Please confirm whether you drew the images / clip-art within the figure panels by hand. If you did not draw the images, please provide (a) a link to the source of the images or icons and their license / terms of use; or (b) written permission from the copyright holder to publish the images or icons under our CC BY 4.0 license. Alternatively, you may replace the images with open source alternatives. See these open source resources you may use to replace images / clip-art:

- https://commons.wikimedia.org

- https://openclipart.org/.

6) Thank you for providing us with your Data Availability Statement. We found that this link "https://www.cmi-pb.org/dataandathttps://www.cmi-pb.org/downloads/cmipb_challenge_datasets/legacy/2nd_challenge/2024-02-02/2nd_challenge/"  reaches a 404 error page. Please amend it to a new link or provide further details to locate the data.

7) Please amend your detailed Financial Disclosure statement. This is published with the article. It must therefore be completed in full sentences and contain the exact wording you wish to be published.

1) State what role the funders took in the study. If the funders had no role in your study, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.".

If you did not receive any funding for this study, please simply state: The authors received no specific funding for this work.

Reviewers' comments:

Reviewer's Responses to Questions

Reviewer #1: In this article, the authors established a community resource to challenge computational models that predict the outcome of booster B.pertussis vaccination from a select group of scientists, evaluate and compare those models in their performance in predicting B. pertussis booster responses. They looked for the features of successful models and identified objects based on the level of difficulty in being predicted. The goal, method and results are well stated and clear. There are a few things in the paper that need to be clarified, for this reason I would suggest this paper to undertake some minor revision.

1. In this ‘invited challenge’, what is the criteria in selecting the scientists and their modeling work?

2. Are all main types of models of predicting immune response covered in this study?

3. The tasks to be evaluated consisted of predicting plasma IgG levels against the pertussis toxin (PT) on day 14 post-booster vaccination and the fold change of the plasma IgG levels against PT between day 14 post-booster vaccination and baseline, predicting the overall frequency of monocytes among PBMCs on day 1 post-booster vaccination and the corresponding fold change and the CCL3 gene expression on day 3 post-booster vaccination and the corresponding fold change values compared to baseline. Is there a reason that those prediction are tested at those specific days (14, 3, 1 post booster vaccine)?

4. Are the evaluation and comparison conducted on the list of ranks for the 21 subjects for each one of the 6 tasks? Is there any evaluation on the values of plasma IgG levels, overall frequency of monocytes among PBMCs or the CCL3 gene expression?

Reviewer #2: Shinde et al. provide a nice overview of the third competition in the Computational Models of Immunity to Pertussis Booster (CMI-PB) challenge series. In their paper, the authors discuss the outcomes of a model development/prediction task undertaken by 20 teams, that also included two "control" models and 22 literature based models. The goal of this competition is noteworthy and raises important questions about the applicability of models developed to understand pertussis vaccination. I have some suggestions for additions that I encourage the authors to consider:

1. Clarifying what "model" means here would be useful for a broader audience. The winning team used machine learning, regression models. As a mechanistic modeller, I was a bit unsure throughout the text what the structure of the models were and were they statistical vs mathematical vs computational etc. The authors do provide a list of the top 5 + "other" model types, but some more discussion would help clarify.

2. Similarly, the control models were unclear to me. Control model 1 used age to rank (?) participants it seems. Is it a regression model with age as the regressor? Maybe a figure panel or equations laying out both control models would be useful.

3. I was interested in the commonalities between the highest ranked models. Were there any biological features of well-ranked (say top 5) conserved across models. If so, what do they help us learn about pertussis booster vaccination. Some overview is given in the discussion. However, given the unique nature of this challenge and the data the team has assembled, it would be nice to also provide key biological take-away messages.

4. For the normalization, was there any removal of background measures when calculating the median baseline concentration?

I also had a few minor comments:

1. On page 4, it may be clearer to move the sentence "The repeat pre-vaccination samples were intended to give a stable estimate of baseline and variability." to after the discussion of the measurement taken from each specimen (otherwise one wonders "baseline and variability of what?").

2. Also on page 4, "such as IL-6 cytokine in the plasma cytokine" could simply be "such as IL-6 in the plasma..."

3. I think there is a missing reference on page 5 ("[38490204]").

4. Page 11: "The detailed description is provided here11." Maybe "is provided in da Silva Antunes et al.11" is better?

Reviewer #3: The manuscript presents an important contribution to the field of systems vaccinology. It effectively demonstrates the value of community-driven benchmarking in computational modeling of vaccine responses and provides a robust framework for evaluating predictive models. The study is well-designed, and its findings offer meaningful insights into the methodologies that improve predictive accuracy and reliability in this context.

The manuscript is well-written, and the dataset and methodology are clearly described. The open science approach, with datasets and code made publicly available, aligns with best practices for transparency and reproducibility in research. The establishment of a community resource and the invitation for future challenges further enhance the relevance and impact of the study.

There are only a few minor points that could be addressed to strengthen the clarity and accessibility of the manuscript:

1. Tables are somewhat dry (i.e. “boring”) in presentation. It would be helpful to visualize some of these results for better engagement and understanding.

2. The manuscript acknowledges the limited size of the challenge dataset, but including a brief discussion on how this limitation could be addressed in future challenges would provide a more balanced perspective. Additionally, it would be beneficial to comment on the dataset specificity of the models and their potential generalizability.

3. Adding a few sentences to highlight how the findings, particularly from a biological perspective, align with or diverge from previous systems vaccinology studies would enhance the contextual significance of the work.

4. While the conclusion is robust, it could be strengthened by outlining how the findings might influence vaccine development or immunological research.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

Figure resubmission:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. If there are other versions of figure files still present in your submission file inventory at resubmission, please replace them with the PACE-processed versions.

Reproducibility:

To enhance the reproducibility of your results, we recommend that authors of applicable studies deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Dr. Shinde,

We are pleased to inform you that your manuscript 'Putting computational models of immunity to the test - an invited challenge to predict B.pertussis vaccination responses' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests. Please also correct the typos flagged by Reviewer 2.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Jessica M. Conway

Academic Editor

PLOS Computational Biology

Rob De Boer

Section Editor

PLOS Computational Biology

***********************************************************

Please correct the typos flagged by Reviewer 2.

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Thank you for your revisions. The concerns raised in my previous review have been adequately addressed, and I have no further comments.

Reviewer #2: The authors have responded to all of my questions and I recommend acceptance.

Just a couple typos that I noticed in the revised manuscript:

-page 9: "For task 2.2, models two models"

-page 15: "Tasks 1.1 and 1.2 (IgG levles)"

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: None

Reviewer #2: None

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

Reviewer #2: No