Dear Dr. Peng

Thank you very much for submitting your manuscript "ProInfer: An interpretable protein inference tool leveraging on biological networks" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

In particular I would suggest to focus on the issue of false positives raised by the reviewers and the performance comparison with other methods available. In addition is of paramount importance that your tools are accessible and usable without errors, therefore your github repository needs to be updated and kept up to date and flawless.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Franca Fraternali

Guest Editor

PLOS Computational Biology

Lucy Houghton

Staff

PLOS Computational Biology

***********************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors developed a protein inference tool named ProInfer. The proposed method makes use of prior knowledge from biological networks to improve the performance of protein assembly based on MS data. ProInfer showed superior performance compared to several other publically available tools. The tool was written in Python, and the source code is openly available. The work is of value, and ProInfer can be a valuable tool for proteomic data analysis.

To make the tool more accessible, especially to colleagues without computational background, I suggest, in addition to releasing the source code, providing a stand-alone version or a web-based application.

Reviewer #2: ProInfer is a computational analysis framework which infers the identification of proteins from ambiguous peptides identified in mass spectrometry data. The conceptual advance ProInfer provides is incorporating information from protein interaction networks such as CORUM (i.e. protein complexes) to improve the identification of proteins in a given sample. The authors’ compare their method against other protein inference algorithms on a benchmark of Human Protein Atlas validated/unvalidated proteins and show superior recall although limited relative precision. The authors’ also include an analysis of differentially expressed proteins and compare their results to other popular algorithms showing mostly positive performance.

Major:

1) Integration of protein complex information to improve protein identification may also introduce bias in the true positives and false positives proteins identified by the method. The authors’ alluded to some of the reasons in their discussion for true positive bias, in particular the incompleteness of biological networks. This however is not the only reason for bias, and I feel a full analysis of the false positives is necessary to give a complete picture.

One possible source of false positives is partners of proteins that participate in many complexes. It is unclear how these cases are handled in the ProInfer framework. This is a major concern due to the use of maximum probability values. Calmodulin (CALM1), for example, is found in 11 different Human Corum complexes many of them with non-overlapping subunits and unlikely to be expressed all together. If I understand the algorithm correctly, in a scenario where CALM1 is highly (or moderately) abundant in a sample, step 2 will assume all 11 CALM1 containing complexes are present regardless of if the complexes are actually formed, expressed, or seen previously in that cell type/condition. Further in Step 4, using the maximum will transfer the probability of the most confident protein (CALM1 in this case) to all other interaction partners in the 11 complexes regardless of if the individual subunits are expressed or not. This potentially is one source of false positives the method is producing.

An additional issue that is unclear is how the framework handles paralogous proteins. Paralogous proteins often share the same peptides and can participate as mutually exclusive partners in protein complexes. This method suggests if a partner in a shared complex is present, both paralogous proteins would be identified.

Due to the above issues and possible other unknown biases, a more complete analysis of the false positives generated by the method is needed and a description of the biases introduced into the identified proteins due to the use of protein complexes as a source of integrated information.

2) It is unclear how the decoy protein complexes are utilized in the method. It is mentioned but not fully described.

3) The Human Protein Atlas is used as a benchmark. It is unclear which dataset within the HPA was used. A link to the downloads page would aid in replicability. Further, the description of “validated” vs “non-validated” is vague. What do the authors’ mean by “different tools”? Is it only antibody based methods? Specifics are required for a full evaluation of the method. Publishing the full benchmark in the supplemental along with ProInfer’s identified proteins is necessary as well.

4) The text states “… literature proofs were searched to prove they really associate with lung cancer.” It is unclear what the criterion is for inclusion here. Many studies have extremely weak associations between genes and diseases and rigorous review of the literature is required to ensure strong evidence of associations. The authors should detail their method and include a supplemental table for the reader to evaluate.

5) A fresh clone of the github repository does not produce executable code. I received an error when trying the example command: python ProInfer.py ./DDA1.tsv

NameError: name 'argparse' is not defined

When I added ‘import argparse’ to the file, I received a different error:

ProInfer.py: error: unrecognized arguments: ./DDA1.tsv

Minor:

Some of the axes in the figures are not labeled properly. For example in Figure 3A-C, the x-axis requires a label.

Reviewer #3: In this article, the authors present ProInfer, a method that performs protein identification by using an inference method harnessing protein complex information.

The methods are interesting, and well explained, and the results appear convincing. I have a few comments that I believe should be addressed:

- the independence assumption in equation 6 seems entirely unjustified, especially if overlapping peptides are considered. Can the authors test that assumption? This directly relates to the next point, since this formula is used for the computation of the FDR:

- The flat precision curve for ProInfer in figure 2B is puzzling: the precision should go down as the FDR increase. How do the author explain this behaviour?

- It seems to me that the "reliable" complexes are implicitely given a prior of 1, and the same weight. But different complexes have widely varying concentrations, and confidence. Are they all given the same prior likelihood ?

- how useful is proinfer when pre-existing identifications of proteins are weaker? for example with less identified proteins in the reference? and how does it affect ProInfer's precision and recall? It seems hard to tell whether ProInfer is efficiently using weak peptide identifications, or using noisy peptide spectra to assign proteins from the network.

- it would be useful to see a comparison between proInfer and some of the other tools as a function of psm threshold: does proInfer do better than other tools at smaller psm thresholds?

- It would be nice to have in supplementary some peptide spectra, to illustrate what type of evidence that proInfer uses to draw its inference, that would otherwise be ignored.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No: Code is available but does not run without errors. Full results of the code is not provided in the supplementary tables.

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Tristan Cragnolini

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Dr Peng

Thank you very much for submitting your manuscript "ProInfer: An interpretable protein inference tool leveraging on biological networks" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

In particular you should carefully address the requested definition of your approach to defining false positive. Please respond carefully to the suggestions of the reviewer both in the first and second revision,

Looking forward to your revised manuscript,

with my best regards

Franca Fraternali

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Franca Fraternali

Guest Editor

PLOS Computational Biology

Lucy Houghton

Staff

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #2: I thank the authors for clarifying some of the points in their manuscript.

1. I do still have a concern about certain proteins found in many complexes. As mentioned before, some complexes may not be expressed even though a member is highly abundant (CALM1 as an example). This may lead the algorithm to improperly “enhance” the confidence score of a protein associated to a complex that is not expressed. This may be a source of false positives. In the authors’ response, they describe a scenario of two proteins P1 and P2 which have small confidence scores which map to two different complexes associated to CALM1, c1 and c2. P1 and P2 are both enhanced due to their association to c1 and c2. If we additionally know that c2 is not expressed in the tissue of study, P2’s confidence score was then erroneously enhanced. In the described algorithm, this scenario is only problematic with proteins that are found in many complexes such as CALM1.

A potential test would be to remove all proteins and their complexes from CORUM seen in > 3 complexes and rerun ProInfer. As a control, remove the same number of proteins/complexes that match the average abundances in the first set. This may show if the many complex proteins are having an effect.

2. The use of the terms “validated” and “non-validated” is confusing as it suggests using additional experimental evidence for validation. Using terms “true-positives” and “false-positives” seems sufficient.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References:

Review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Dear Dr Peng,

We are pleased to inform you that your manuscript 'ProInfer: An interpretable protein inference tool leveraging on biological networks' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Franca Fraternali

Guest Editor

PLOS Computational Biology

Lucy Houghton

Staff

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #2: The new analysis addresses my concerns regarding false positives introduced by proteins common to multiple complexes. I also feel the revised manuscript is clearer with the additional edits.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: None

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No