PONE-D-20-04130

Survey of metaproteomics software tools for functional microbiome analysis.

PLOS ONE

Dear Dr Jagtap,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The overall assessment of this work is positive but reviewers have raised several issues that need to be addressed. Regarding the comparison and presentation of the results, the choice of Galaxy should be better justified. Including the run time would also enhance the results. In-depth arguments are also required in particular regarding the discussion raised by reviewer1 on the use of GO terms. Careful accounting of the reviewers' comments can definitely make this paper a flagship for users of metaproteomics analysis pipelines.

We would appreciate receiving your revised manuscript by April 30, 2020. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Frederique Lisacek

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.Please ensure that you refer to Figure 4 in your text as, if accepted, production will need this reference to link the reader to the figure.

3. We note that you have included the phrase “data not shown” in your manuscript. Unfortunately, this does not meet our data sharing requirements. PLOS does not permit references to inaccessible data. We require that authors provide all relevant data within the paper, Supporting Information files, or in an acceptable, public repository. Please add a citation to support this phrase or upload the data that corresponds with these findings to a stable repository (such as Figshare or Dryad) and provide and URLs, DOIs, or accession numbers that may be used to access these data. Or, if the data are not a core part of the research being presented in your study, we ask that you remove the phrase that refers to these data.

4. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript aims to benchmark six available metaproteome annotation tools, eggNOG-mapper, MEGAN6, MetaGOmics, MetaProteomeAnalyzer (MPA), ProPHAnE, and Unipept.

To benchmark the tools, an “oral dysbiosis” dataset was selected which includes the samples of plaque bacteria that were grown with and without sucrose. For both of these samples, the predicted peptides were normalised and mapped to functional groups/terms using the benchmarked tools. Subsequently the differential expression of the terms were calculated for each of the tool. The core of the benchmark is the comparison of these differentially expressed terms.

The manuscript also extensively compares inputs and outputs of the benchmarked tools. As they vary widely in that respect, the presented methodology generates specific inputs for each of the tools and aims to synthesise comparable output datasets.

The authors find that although the differentially expressed annotated terms are all significantly correlated between the tools, they exhibit a large degree of variability. At the end two of the tools (one which exhibits high specificity and sensitivity, and one that mapped the largest amount of terms) were then incorporated into the Galaxy platform.

Growing interest in and availability of metaproteomics data will result in significant need for annotation tools and established annotation pipelines. The presented manuscript could be of significant interest to metaproteomics community and have immediate impact on metaproteomics analysis methods. Therefore it is important to assure that the analysis presented in the manuscript is correct and interpretable.

The core aspect of the presented methodology is assigning the differential expression to each GO term and making the sets comparable between the tools. Unfortunately this is also the weakest part of the manuscript.

For the methods to be comparable, each GO term, for each peptide assigned by the method, should be projected to the root of its ontology and the differential expression should be calculated at each of the GO terms after that projection. It is not clear if this is done here. As such, without this projection, the methods that assign more granular terms will have diminished chance of picking up the signal, as the differential expression signal would be divided onto the term’s children whereas less granular method could incorporate the signal from all the term’s children. GO prediction benchmarking should reward the methods that assign more granular terms, as the GO terms could always be projected. Instead it seems that described method of the comparison effectively punishes for it. This is such a critical weakness, that lets me to believe I did not understand the methodology correctly. However the only part of the manuscript that describes the implementation of GO ontology traversal is found in the analysis of the 20 random peptides which is not related to the core of the comparison.

To partially mitigate the above point GO terms were translated into higher level GO slim subset, however this subset consists of only 147 terms, which may be too broad to generate the interpretable differential expression signal. I say potentially, as authors did not include in the manuscript any results from GO slim comparison except calculating Jaccard index between the tools’ outputs, which is not particularly informative.

All differentially expressed list of terms (projected to root) from all the tested methods, for all three GO ontologies should be included as a part of supplementary data. This will allow readers a way to judge the validity of the method by themselves. The overlap comparisons and “Top ten” terms for each method not sufficient for a meaningful interpretation.

Another point that should be addressed is a need for a biological interpretation of the results generated by the methods. There must be some processes that we should expect to see being differentially expressed between sugar rich and a sugar deprived environment. For example this may include the GO terms associated with energy metabolism, biofilm creation, growth etc. Expert curators with the knowledge of plaque bacteria could highlight few higher level GO terms that have a potential to be differentially expressed between the two conditions. If such term, selected by curators, does not exhibit metaproteome level differential expression, then we should not expect it to be picked up by any method by chance, conversely if at least one method finds such term (or its descendant) to be differentially expressed it probably won’t be by chance alone. The ranking of these curated term in the methods output could potentially highlight which of the tools is more suitable for finding biological signal in this particular metaproteome data. Of course it is important to note that the conclusions derived from this test cannot be trivially extended to all other biological datasets, nonetheless there is an insight to gain from assessing distinct aspects of the differences between the tools’ outputs.

Other issues:

The authors state in the abstract that the “A BLAST analysis against the Universal Protein Knowledgebase revealed that the sensitivity and specificity of functional annotation differed between tools”. Highlighting this measure in the abstract is debatable. Primarily because there is no reason why simple BLAST against UniProt would be regarded as gold standard. If it is, there is no need for more sophisticated tools. It’s a useful measure to some extent, but it’s not clear what that measure actually conveys as it seems to me that it depends heavily on the degree of overlap between UniProt and the peptide search database (HOMD in this case). In such case almost by definition Unipept would have to generate the most precise GO, as it is directly based on the peptide matches to UniProt database. This is also what we see.

Execution time of the tools, is always an important factor in the decision which of the available tools to use. For large scale analysis it is often the deciding factor. It would be helpful to point out the total computation time of each of the pipelines that were used to generate final GO terms. If a database only starts with peptides the Mascot search should be added to the computation time, but the execution time of the tool itself should also be highlighted.

The authors state that DIAMOND has “similar sensitivity to BLAST”. There is a price for the speed-up of search and it seems to be sensitivity [1, Fig. 2b]. In my experience this is especially true for a very diverged proteins (identity <30%), where DIAMOND significantly loses sensitivity. The exact numbers will depend on the testing methodology.

Figure 3. Legend should be corrected. Pink in “Type of term” legend relates to “Extraneous” terms, which are not present in the bar chart. However, in the same figure pink is used also as a color of eggNOG terms that are not part of the MetaGOmics output. This is misleading as only only one of the eggNOG terms (Formylmethionine deformylase activity”) could be considered as “extraneous” in relation to the gold terms.

[1] Steinegger, Martin, and Johannes Söding. "MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets." Nature biotechnology 35.11 (2017): 1026-1028.

Reviewer #2: This is an interesting manuscript comparing different tools used for metaproteomic data processing. The manuscript has some limitations that need to be addressed.

-This seems to be a combination of comparing tools and an add-on to the Galaxy platform. Sufficient details are provided for the comparison of the tools. However, it is not possible to review the add-on to the Galaxy platform as limited details are provided. It is a bit strange to have both in one manuscript. I would suggest to focus the manuscript on comparing the tools.

-Table 3. Comparing ranks is really not useful and somewhat misleading. Instead, the results from each GO term should be analyzed (# of peptide mapped and quantification) and the results statistically compared.

-Figure 4 should either be removed or moved as supplementary.

-The authors did a good job at highlighting the limitations of their analysis.

-A few typos throughout the manuscripts.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-20-04130R1

Survey of metaproteomics software tools for functional microbiome analysis.

PLOS ONE

Dear Dr. Jagtap,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Mainly, the remaining inconsistencies spotted by the reviewer should be attended to.

Please submit your revised manuscript by October 26, 2020. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Frederique Lisacek

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: The authors describe in their manuscript a comparison of tools providing functional annotations, applied here on metaproteomics data. A preprocessed common dataset is used for all six tools. The functional annotation is investigated through directly predicted or secondarily obtained gene ontologies to have a common framework to do comparisons.

As clearly stated in the title and acknowledged by the authors, it is more a survey than a well-controlled benchmarking and it is limited to providing an overview of the available methods while demonstrating that their predictions vary substantially. The authors discuss the cause of these discrepancies as being for instance the variety of reference database, and advocate for the improvement and curation of these resources. Since the field lacks any benchmarking for that kind of aims, this study is a first step and therefore valuable.

While I appreciate the design of the different evaluations and the discussion, I find the overall description of the tools and their evaluation rather confusing, which in my opinion impair the understanding of the results if you are not a user of each of these six different pipelines. I am unsure whether this is caused by multiple errors in the manuscript or a lack of rigor in the systematic description of the inputs/outputs/databases. Different sections, figures, and tables seem to contradict or to consider the features of each tool differently. I would advocate for polishing and clarification. The language is clear.

- In general, page5:"Standard procedures were used for each tool" is not sufficient to identify when methods overlap regarding their data sources. Databases underlying the tool should be listed explicitly for each of the six methods, and not scattered across different figure and tables, no matter bundled with the tool or to be provided externally.

- For instance, eggNOG db is used by multiple tools, probably for different purpose. Figure 1 does not connect eggNOG db to MEGAN, while methods state page5:"the analysis was performed using eggNOG ". Table 2 states that eggNOG obtain functional annotation of type "proteins" while MEGAN obtain them from eggNOG groups. All together, this prevents the reader from understanding whether MEGAN and eggNOG get part of their functional annotation from a common source or not at all.

- Figure 1 is not that helpful as it does not reflect the difference between tools that matter during the evaluation. Especially, it does not clearly separate and define the databases used for classification of the input and their functional annotation. In some part of the manuscript, other db (page5: "For MetaGOmics, a list of peptides and the HOMD were uploaded (...)") are mentioned but not present on Figure 1.

- Two tools are described as natively predicting GO, vs four which do not (page8:"tools that natively output GO terms (i.e., Unipept and MetaGOmics)"). But on Figure 1, five out of six tools output GO, so this seems to be a different interpretation of natively predicting GO. Moreover, methods states that page6:"GO terms (...) are common annotation type throughout most of the tools (i.e., eggNOG-mapper and Unipept) (...) However, MEGAN, ProPHAnE, and MPA do not provide direct GO term outputs". Only five tools mentioned, most of the tools being two of them (!), and not matching previous statements or Figure1.

- I am unsure whether MPA uses the output of XTandem and Peptideshaker since the "input" folder in the supplementary archive contain a fasta that contains cRAP protein, which resemble a database use for search and filtering (277K_for_737_MPA_with_cRAP.fasta). I suppose this could indicate the actual input is the initial spectrum files identical to what is given to XTandem (in methods: page5:"For MPA, the same parameters as for SearchGUI were used"). This would contradict the statement in the discussion page21:"While using the same post-search inputs from the same dataset". Or there is something else missing (cRAP-filtereds spectrum files?) as input for MPA in the supplementary archive.

- To clarify all the points above, a visual summary of the workflow from the first difference appearing in any tool to the end of the process with the translation of GO would be beneficial, either as supplementary or as replacement of Figure 1.

- The definition of peptide-centric / peptide-level / protein-centric / protein-level could be clarified, if these terms are necessary to understand the input and outputs that lead to GO predictions attached to individual proteins, OG, or peptides. These terms seems to be used in contradictory ways and are confusing.

page9: "Of the tools that used peptide-level information (see Figure 1)". => Figure 1 contains "protein-level information" as possible output, and "peptides" as possible input. The reader could assume that "peptide-level information" are those without the checkmark for "protein-level information" outputs in Figure 1, but "used" does not mean "outputed".

Page9: "tools that provided peptide-level outputs were used, thus excluding MEGAN, MPA, and ProPHAnE" This does not match any of the possibilities depicted in Figure 1, but does match page10:"analysis level" in the results section.

page17: " protein-level tools is highlighted here (MEGAN and ProPHAnE) " Again, this is not the same list... the meaning might be "some of the protein-level tools" but the general confusion about these terms cast doubts.

page18: "a peptide-centric (eggNOG-mapper, MEGAN, MetaGOmics and Unipept) or protein-centric approach (MPA and ProPHAnE)." This matches the grouping in figure1 and seems correct, but "centric" does not tell whether to focus on inputs or outputs and MPA and ProPHAnE do not have the same input type according to figure 1.

- page16: "The single- peptide analysis (Figure 3, Supplement S5 and Supplement S6) was carried out for Unipept, MetaGOmics and eggNOG-mapper since they provide GO term annotations at a peptide level. As a result, we do not have results for the protein-level tools MPA and ProPHAnE." Related to previous comments, MEGAN is not included, no justification here... is it because of protein-level vs peptide-level? Or inhability to provide GO term ?

minor:

- the Jaccard Index is only mentioned in the discussion while it was removed elsewhere during refactoring. Please check whether it is still intended to refer to this or prefer something else to match figure 2

- MEGAN5 and MEGAN6 are mentioned as being the evaluated version. Please check whether this is a typo.

- Supplementary figure S5: one color is not labeled (pink)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Survey of metaproteomics software tools for functional microbiome analysis.

PONE-D-20-04130R2

Dear Dr. Jagtap,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Frederique Lisacek

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: