Dear Dr Zhang,

Thank you very much for submitting your manuscript 'Maximizing the Reusability of Gene Expression Data by Predicting Missing Metadata' for review by PLOS Computational Biology. Your manuscript has been fully evaluated by the PLOS Computational Biology editorial team and in this case also by independent peer reviewers. The reviewers appreciated the attention to an important problem, but raised some substantial concerns about the manuscript as it currently stands. While your manuscript cannot be accepted in its present form, we are willing to consider a revised version in which the issues raised by the reviewers have been adequately addressed. We cannot, of course, promise publication at that time.

Please especially address the criticism of reviewer 2 which presents a major hurdle for publication at this time.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

Your revisions should address the specific points made by each reviewer. Please return the revised version within the next 60 days. If you anticipate any delay in its return, we ask that you let us know the expected resubmission date by email at ploscompbiol@plos.org. Revised manuscripts received beyond 60 days may require evaluation and peer review similar to that applied to newly submitted manuscripts.

In addition, when you are ready to resubmit, please be prepared to provide the following:

(1) A detailed list of your responses to the review comments and the changes you have made in the manuscript. We require a file of this nature before your manuscript is passed back to the editors.

(2) A copy of your manuscript with the changes highlighted (encouraged). We encourage authors, if possible to show clearly where changes have been made to their manuscript e.g. by highlighting text.

(3) A striking still image to accompany your article (optional). If the image is judged to be suitable by the editors, it may be featured on our website and might be chosen as the issue image for that month. These square, high-quality images should be accompanied by a short caption. Please note as well that there should be no copyright restrictions on the use of the image, so that it can be published under the Open-Access license and be subject only to appropriate attribution.

Before you resubmit your manuscript, please consult our Submission Checklist to ensure your manuscript is formatted correctly for PLOS Computational Biology: http://www.ploscompbiol.org/static/checklist.action. Some key points to remember are:

- Figures uploaded separately as TIFF or EPS files (if you wish, your figures may remain in your main manuscript file in addition).

- Supporting Information uploaded as separate files, titled Dataset, Figure, Table, Text, Protocol, Audio, or Video.

- Funding information in the 'Financial Disclosure' box in the online system.

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see here. 

We are sorry that we cannot be more positive about your manuscript at this stage, but if you have any concerns or questions, please do not hesitate to contact us.

Sincerely,

Ilya Ioshikhes

Associate Editor

PLOS Computational Biology

Thomas Lengauer

Methods Editor

PLOS Computational Biology

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: attached

Reviewer #2: Although the paper is described as the prediction of missing data in public databases, in essence it is addressing a specific problem of whether specific sample attributes (variables), such as ethnicity, estrogen receptor (ER) status, progesterone receptor status, and similar can be predicted from gene expression data and to what extent. The link to the public data is that if, for instance, a breast cancer gene expression data contains data from patients some of which are ER positive and some ER negative, but the sample status is not given, one can ask if this is possible to derive from the data itself?

One can ask this question within a particular dataset and across datasets. In the first setting one would take a dataset where this variable (e.g., ER status) is included in the annotation, hide it, train a machine learning model and then use cross-validation to assess the accuracy of the model. In the second setting, one would train the model on one dataset, and then test on a different dataset. The second approach makes sense, the first really does not. One could also think of an approach where several datasets are merged, jointly normalised and then model is trained on the merged dataset, but I cannot imagine why one would do this, rather than the second – effectively meta-analysis approach. Unfortunately, I was not able to understand from the paper, which of these approaches the authors take (possibly this can be deduced by studying the supplementary material, but I did not do this, nor should a reader). I think that if the authors are invited to resubmit the paper, they should clarify this.

In addition, I am worried that instead of using well accepted methods for assessing the accuracy of machine learning algorithms, the authors have designed their own methods. I did not fully understand the rationale of this.

Finally, one has to ask what is the utility of this? The authors spend significant space in the manuscript discussing comparisons of various ML methods but is this really interesting? It would be more interesting to choose a wider range of variables (e.g., sex, age bracket, tumour stage, etc) and compare how well each can be predicted (cross datasets). The problem of course would be that most variables are specific to particular diseases, or particular studies.

The authors claim that their missing value imputation approach help in downstream analysis, particularly in differential gene expression. It is not surprising that it does (though I was not able to easily understand the author proof of this), but one could just as well use a latent variable approach (i.e., assume that there are latent variables in the dataset, which need to be factored out) for this problem, an approach which is well familiar to statistical geneticists. Computationally imputing missing values in the public databases would be rather dangerous, as this might easily lead to circularity in the downstream analysis by other authors.

To summarise, I do not think that the paper is written clearly enough, I think the manuscript contains too much philosophy around, e.g., FAIR principles, which are not really relevant to the main line of the presented research, and in the current form it does not represent a clear advancement of science.

Reviewer #3: In this study, we develop a framework to predict missing metadata of gene expression datasets to maximize their reusability. We propose a new metric called Proportion of Cases Accurately Predicted (PCAP), which is optimized in our specifically-designed machine learning pipeline.

The problem is extremely important and relevant of ensuring high quality metadata and with the increasing amount of data, methods are required to tackle it. The paper proposes a method to predict missing metadata using machine learning methods and show that their predictions can be used in downstream analysis. However, I have concerns about the main research question, methodology design and evaluation.

Here are my detailed comments:

- “However, the majority of public genomic data do not contain enough metadata, which severely limits their reusability.” Please provide evidence for this.

- “It may seem quite straightforward to infer missing metadata using gene expression profiles, given the abundance of previous research works.” Claims like these must be supported by evidence.

- “In this study, we investigate the above issues to infer missing metadata using multiple gene expression datasets with a wide variety of machine learning methods.” This is too broad and vague. Which machine learning methods? Why those methods? Why those specific datasets?

- Overall, the main research question/hypothesis is missing.

- Another major concern is that there is no discussion of related work. What is the gap that the authors are trying to fill? There have been studies that have looked at methods to predict metadata:

-- Predicting biomedical metadata in CEDAR: A study of Gene Expression Omnibus (GEO) https://www.sciencedirect.com/science/article/pii/S1532046417301405

-- Predicting structured metadata from unstructured metadata https://academic.oup.com/database/article/doi/10.1093/database/baw080/2630448

-- massiR : a method for predicting the sex of samples in gene expression microarray datasets https://academic.oup.com/bioinformatics/article/30/14/2084/2390865

- This sentence “For each variable, the subset of samples with missing values was not considered in the analysis of the corresponding variable, since we cannot evaluate the performance of our method if we use these samples.” puzzled me because I thought that the aim of the project was to to infer missing metadata.

- There doesn’t seem to be any novelty of the method that is used apart from an evaluation measure that they propose (PCAP - Proportion of Cases Accurately Predicted), which is also not evaluated. Only PCAP95 and PCAP90 seemed to have evaluated. How were these values determined? What is the threshold at which the accuracy decreases?

- In the “Normalization of gene expression” section, three normalization techniques are mentioned, whereas in the results sections only the effect of Rank Normalization is discussed. The other two methods’ results should also be discussed.

- I miss details on the “Model selection pipeline”, how does that work? There is no explanation of the pipeline.

- The goal of predicting missing metadata is to use the corresponding gene expression data in other analyses.

- The example presented about the “differential gene expression analysis (DGEA) using predicted race” is a bit unclear. I would suggest to create a graphic to help understand it.

- Additionally, it is insufficient to show that the predicted metadata can be useful in only one use case

- “recover a high percentage of samples with the desired accuracy” - please add more details for this

- Also, it is unclear whether the method can be generalized to other metadata

- Additionally, the authors do not provide their code which makes it even more difficult to understand the implementation of the methodology.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No: The cleaned metadata is not made available and nor is the code.

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Shannon E. Ellis

Reviewer #2: No

Reviewer #3: Yes: Amrapali Zaveri

Attachments

Attachment

Submitted filename: PLOSCompBio_Review.pdf

Dear Dr. Zhang,

Thank you very much for submitting your manuscript "Maximizing the Reusability of Gene Expression Data by Predicting Missing Metadata" for consideration at PLOS Computational Biology.

As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Ilya Ioshikhes

Associate Editor

PLOS Computational Biology

Thomas Lengauer

Methods Editor

PLOS Computational Biology

***********************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Generally, I would love exploration here and clarity in explanation/display of testing and am unsure how I would *use* this in the future if I found the results compelling. The two main conclusions seem to be:

1. Larger sample size not always the answer after metadata predictions used to supplement these points

2. PCAP a better metric

From the text and figures, I'm not convinced of these. I'm not saying they're untrue...I just struggle to draw these conclusions from the information provided. If these are not to be the takeaways, I'd revisit the text. If these are the main takeaways, I'd revisit the clarity of explanations provided and figures.

Major:

- The manuscript has not become clearer in this round of revision. I'm not convinced that PCAP is a better approach and I'm even less clear about what predictions are being made, what cutoffs to use, how to determine ideal sample size after prediction, or what metrics to use to decide this. I think taking a hard look at the organization of these explanations and clarity in display of results would go a long way.

- Figure 4 is very unclear to me. PCAP90 (the metric used in the manuscript) looks pretty darn close to random to me (on median); What about other metrics that you claim PCAP is better than? Why not included here?

- Figure 6 - without knowing what each point is, it's hard to know what variables you can predict with this method and if a different method would be better

- Thanks for sharing the URL to the code in response to reviewers. Please include this in the manuscript directly.

Minor:

- First paragraph of introduction: "The availability of..." is missing a citation

- Bottom of Page 3 (newly added material) - "It is unknown that whether" <- I think that is an additional word. If not, I'm not sure what this sentence is trying to say. Also, that sentence needs a citation or better wording. Also DGES is the initialism used here, but I think it's supposed to be DGEA

- Page 7 - new content; "Our pipeline can be easily extended to multi-class predictions" <- either demonstrate this directly in manuscript or remove this statement

- Top of pg 10 (new content) - unsure what the sentence "For variable," is conveying - have variables A,B,C, and D been introduced in text outside of figures yet?

- Figure 1 colors - hard to compare between plots as negative and positive and the same color in (a) and (b)

Reviewer #2: The authors have significantly extended the study and have addressed my concerns.

Reviewer #4: Attached file

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #4: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #4: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, PLOS recommends that you deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions, please see http://journals.plos.org/compbiol/s/submission-guidelines#loc-materials-and-methods

Attachments

Attachment

Submitted filename: Review-PLOS-Comput.docx

Dear Dr. Zhang,

We are pleased to inform you that your manuscript 'Maximizing the Reusability of Gene Expression Data by Predicting Missing Metadata' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Ilya Ioshikhes

Associate Editor

PLOS Computational Biology

Thomas Lengauer

Methods Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Thank you to the authors for their thoughtful revisions and updates to the manuscript. I'm happy with the manuscript in its current form and have no further suggestions or requests.

Reviewer #4: No more comment.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #4: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #4: Yes: Hulin Wu