Dear Dr. Vogelstein,

Thank you very much for submitting your manuscript "Eliminating accidental deviations to minimize generalization error and maximize reliability: applications in connectomics and genomics" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Blake A Richards

Associate Editor

PLOS Computational Biology

Jian Ma

Deputy Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: My review is uploaded as attachment file.

Reviewer #3: Spurred by the reproducibility issue, and motivated by classical works in statistics (Spearman, Fisher), the authors expand on a statistic called discriminability, which quantifies the degree to which an individual’s samples are similar to one another. (Some of the authors proposed this index earlier.) The aim is to minimize generalization error and maximize reliability. Discriminability is a nonparametric index that is appropriate for a wide range of data types (text, images). By using simulations, proving theorems (for additive noise), and demonstrating its use on neuroimaging and genomic data, they show that their index outperforms intraclass correlation (ICC), fingerprinting index, and kernel methods.

Comment.

The authors say that the ICC’s Gaussian roots are a problem. However, their theorems deal with Gaussian errors (the mixture of Gaussians seems fine, for they are flexible). What happens if the errors are not Gaussian ? Many homogeneous populations are skewed, so skew-normal or related distributions may be more appropriate.

The paper is very well written; I spotted a few awkward phrasings

p 4

why SUM i * s_{i} ? why not SUM s_{i} ?

indentify —> identify

for the below definition —> for the definition below

p 24

to overcome to requirement of ICC —> to overcome the requirement of ICC

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Stephen C. Strother

Reviewer #3: No

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References:

Review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Attachments

Attachment

Submitted filename: Strother_ Review_PCOMPBIOL-D-21-00089.pdf

Dear Dr. Vogelstein,

We are pleased to inform you that your manuscript 'Eliminating accidental deviations to minimize generalization error and maximize replicability: applications in connectomics and genomics' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests. Please also be sure when producing the final version of the paper for publication to address the remaining few comments from Reviewer 1, particularly the note regarding NPAIRS.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Blake A Richards

Associate Editor

PLOS Computational Biology

Jian Ma

Deputy Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Response to Rev1, Comment 3: I appreciate the authors expanding their list of prior work to include NPAIRS. This is a minor issue that should not distract from their contributions in the rest of the manuscript, but NPAIRS is and has been applied without SPMs beyond the Neuroimaging domain. One only needs to modify the similarity distance metric as shown using mutual information in the original companion paper with sensitivity maps (Kjems et al., 2002: https://pubmed.ncbi.nlm.nih.gov/11906219/) and then illustrated with Raman spectroscopy (Sigurdsson et al., 2004: https://pubmed.ncbi.nlm.nih.gov/15490825/), which is well beyond the Neuroimaging domain. Furthermore, NPAIRS has also been modified to apply to the proteomics domain (https://pubmed.ncbi.nlm.nih.gov/20531955/) and application to images of GWAS SNP expression arrays is relatively trivial. Finally, NPAIRS can readily also be applied in connectomics by using a graph similarity metric and some form of sensitivity analysis on graph ensembles, although to my knowledge this has not been published.

I think the authors important contributions in their manuscript will stand on their own right without unnecessarily claiming limitations of other approaches that are easily refuted by the literature.

Response to Rev 1, Comment 9: I think the English you want is: Illustrates […] (i) the preprocessing pipelines that maximise … . You could also use (i) which preprocessing pipelines maximise … . The first is better Strunk and White English but the second also captures the same meaning.

Response to Rev 1, Comment 14: Supp. S7 is an interesting and useful addition to the paper. I look forward to further work from this group following these ideas.

Reviewer #3: The authors have addressed my concerns.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Stephen C. Strother

Reviewer #3: No