Dear Prof. Vikkula,

Thank you very much for submitting your manuscript "Excalibur: a new ensemble method based on an optimal combination of aggregation tests for rare-variant association testing for sequencing data" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

The reviewers acknowledged and appreciated the effort to evaluate several state-of-the-art aggregation tests, but several weaknesses in the evaluation and presentation were brought up by multiple reviewers. The omission of SKAT tests from the comparison, the lack of clarity in how correlation between the different aggregation tests is handled, and lack of details related to computation time for an ensemble method such as Excalibur were cited as major weaknesses. The reviews also suggested a summary of the comparisons that could help a user identify some of the tests suitable for various scenarios would add to the value of the manuscript.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Aakrosh Ratan

Guest Editor

PLOS Computational Biology

Jian Ma

Section Editor

PLOS Computational Biology

***********************

The reviewers acknowledged and appreciated the effort to evaluate several state-of-the-art aggregation tests, but several weaknesses in the evaluation and presentation were brought up by multiple reviewers. The omission of SKAT tests from the comparison, the lack of clarity in how correlation between the different aggregation tests is handled, and lack of details related to computation time for an ensemble method such as Excalibur were cited as major weaknesses. The reviews also suggested a summary of the comparisons that could help a user identify some of the tests suitable for various scenarios would add to the value of the manuscript.

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The review is uploaded as an attachment.

Reviewer #2: The authors comprehensively evaluated state-of-art methods and four novel methods performances across different scenarios, and showed the outperformance of Excalibur in controlling type I error while offering the best average power. The authors also investigated each parameter’s effect on the behavior across methods, which provided valuable information for other researchers to choose methods based on different situations in their own research.

One issue that needed be clarified is that the authors claimed that “Excalibur also automates the test selection depending on the genetic region to be explored (e.g. genes, pathways, etc).” However, it is not crystal clear from the construction of the methods on page 13 how Excalibur choose the tests depending on the genes or pathways. Some explanations would be helpful to strengthen this claim.

Also, it would be interesting to see if Exacalibur would also have outstanding performance in the mixed populations with people from different ancestries. The variants effects may differ in size and even in direction in different ancestry populations.

Reviewer #3: The authors try to use an ensemble method, minimum P after multiple test correction, to combine different rare variant tests to achieve a robust and efficient test. However, the comparison design excludes powerful SKAT tests, which is the state-of-the-art methods for rare variant analysis, simply due to strict computation time. The comparison also omits the robust binary SKAT recently developed (Zhao et al., 2020, AJHG). The improved SKAT tests can address the inflated type I error problem when the sample size is small, the allele frequency is too extreme, or the case control balance is too extreme, even though the computation time is increased. This is not a problem because for rare variant test in WES or WGS, we can often scan the whole genomes using a quick version of SKAT and then for interesting genes, we can use a more time consuming resampling version of SKAT to make sure the type I error is controlled. Therefore, I think the comparison is flawed. Here are my suggestions.

1. Include the robust binary SKAT recently developed (Zhao et al., 2020, AJHG) in the comparison.

2. Include those SKAT versions removed due to computing time to have a direct comparison with the proposed ensemble method on type I error, power, and computing time.

3. When comparing power, there is no need to discuss methods that cannot control type I error.

4. Line 473-474: “minimum p-value from the set of tests included in the ensemble method after multiple testing correction using Benjamini-Hochberg”. Is the correction using the Bonferroni correction or Benjamini-Hochberg? The former is commonly used and is robust against correlated test statistics. The latter is used to calculate the false discovery rate (FDR) and in theory it requires independence of test statistics. Whether the FDR adjustment can be used to control type I error in theory is in question.

5. I appreciate that many comparisons have been done. However, it is important to summarize the results that can help users to decide which tests to choose in which scenarios. For example, for each scenario, identify the top 3-5 methods. All methods with obvious inflated type I errors should not be included, which should be removed at the first place.

6. The main figures are all blurred.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Wenan Chen

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Attachments

Attachment

Submitted filename: Review_forauthors.docx

Dear Prof. Vikkula,

Thank you very much for submitting your manuscript "Excalibur: a new ensemble method based on an optimal combination of aggregation tests for rare-variant association testing for sequencing data" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Aakrosh Ratan

Guest Editor

PLOS Computational Biology

Jian Ma

Section Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The review is uploaded as an attachment.

Reviewer #2: The authors revised the manuscript nicely, and explained investigating Excalibur performance on mixed populations is out-of-scope, which is acceptable.

Some comments on line 290: suggest to use "log" instead of "ln" to make it consistent with line 288 where it uses log(MAF). Also suggest to delete "0,402" since the rest cs calculated with different MAFs do not show an associated approximate value. Correct the format for the first c.

Reviewer #3: Here are my future concerns which are important to assess the quality of the manuscript.

1. Table 1 is not complete in the PDF file, so it is hard to check in which scenario which methods are performing well and whether the results make sense.

2. The powers of the proposed methods are much higher than the previous version. The previous proposed methods all have power < 0.6. The suggested Robust binary SKAT is already > 0.6 in the new version. Is it due to the inclusion of the suggested robust binary SKAT recently developed (Zhao et al., 2020, AJHG)? Please add comments or discussions in the paper.

3. The type I error of the robust binary SKAT/SKATO is 0 when the nominal value is 0.05. This is different from the paper (Zhao et al., 2020, AJHG) which shows that the type I error is very close to the nominal level. Can the authors explain? Is it because the sample size or other settings are different? It will be more convincing if the authors can show that when the simulation setting parameters are changed to that used in (Zhao et al., 2020, AJHG), then the type I error is close to the nominal.

4. Include those SKAT versions removed due to computing time to have a direct comparison with the proposed ensemble method on type I error, power, and computing time. Here is the list to consider because these methods is likely to be more powerful even though more time is needed.

pBin_linear_SKATO_ER

pBin_linear_weighted_SKATO_ER

pBin_linear_SKATO_ERA

pBin_linear_weighted_SKATO_ERA

pBin_linear_SKATO_Hybrid

pBin_linear_weighted_SKATO_Hybrid

5. Besides the proposed methods, it will add more value to the paper to summarize the results to identify the top 3-5 individual methods for each scenario because users may only consider running one individual method instead of an ensemble method.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References:

Review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Attachments

Attachment

Submitted filename: Excalibur_R1_Review.docx

Dear Prof. Vikkula,

We are pleased to inform you that your manuscript 'Excalibur: a new ensemble method based on an optimal combination of aggregation tests for rare-variant association testing for sequencing data' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Aakrosh Ratan

Guest Editor

PLOS Computational Biology

Jian Ma

Section Editor

PLOS Computational Biology

***********************************************************