Dear Dr. Goecks,

Thank you very much for submitting your manuscript "Galaxy-ML: An Accessible, Reproducible, and Scalable Machine Learning Toolkit for Biomedicine" for consideration at PLOS Computational Biology. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. The reviewers appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Mihaela Pertea

Software Editor

PLOS Computational Biology

Mihaela Pertea

Software Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: Overview: Well written paper. Clear and concise. The Galaxy-ML toolkit and it's accessibility through Galaxy will be a powerful tool available to Biomedical scientists. The paper clearly demonstrates multiple examples of how the toolkit can be used by reproducing existing ML data models with accuracy.

Major Comments: None

Minor Comments: The Galaxy platform itself has some limitations in terms of reproducibility. Although, the platform is openly available and hence the ML toolkit, it would be nice to see a more interoperable way of representing Galaxy workflows so they are repeatable in other cloud or local compute environments.

Extensibility is not necessarily clear or demonstrated in the publication. An potential minor addition, would be to address how additional ML libraries can be added to the existing toolkit and whether customization of those libraries is possible.

Conclusion: The paper is well written and the power of the toolkit as well as it's accessibility through Galaxy is successfully demonstrated as a novel addition to the growing resources available to the Biomedical community.

Reviewer #2: The authors describe Galaxy-ML, an extension to the Galaxy computational workbench that is designed to assist users with running supervised machine learning experiments in the cloud.

The Galaxy interface leverages popular machine learning frameworks including Scikit-learn and Keras to prepare, model, and evaluate supervised classifiers in the Galaxy setting.

The authors performed analyses and meta-analyses on example datasets, accurately describing rationale and methods for feature selection, as well as training and evaluating models.

The most valuable content in this paper comes from the methods section, which serves as an outstanding introductory guide for using the Galaxy-ML toolkit for exploratory machine learning analyses on diverse datasets. The provided workflows are an immense value add, as is the extensive community documentation cited in this manuscript at https://galaxyproject.org/community/machine-learning/.

Overall, this manuscript represents a valuable addition to the field of open-source and community-accessible tools for machine learning in the biomedical domain. The introduction to the tools and the provided methods and documentation are outstanding. I have only a couple concerns over specific statements made in the paper, which I would like to see the authors address before having this manuscript published in PLOS Computational Biology.

Concerns:

The statement “Galaxy-ML tools have also been run more than 12,000 times on the public U.S. (https://usegalaxy.org) and European Union Galaxy servers (https://usegalaxy.eu)” describes this toolkit as being run on both the US and EU servers, but I was unable to locate Galaxy-ML on the US servers with a moderate effort. The machine learning extensions for the EU server are located at https://ml.usegalaxy.eu, which I think is a useful URL the authors should consider adding to the manuscript, perhaps in lieu of the references to the primary server interfaces in this statement (which reference a broader set of tools for the EU server, and do not reference the Machine Learning tools at all on the US server). I would recommend the authors work with the US server team to incorporate the Galaxy-ML tools on that site as well, or provide some discussion in the manuscript on why they are not available on the US server. If the Galaxy-ML tools are not publicly available on the US server, the above statement should also be adjusted to reflect the reality of the situation (it is not useful to know that the authors somehow ran these tools on the US server if that is not available to the readers of this article).

The statement “To the best of our knowledge, this is among the most comprehensive machine learning experiments on cancer cell line RNA-seq and drug response datasets” is underqualified (to the best of our knowledge? among the most? how is comprehensiveness measured?), unnecessary (this does not add value to the paper or the useful work described there), and difficult to prove (large dataset ML experiments are a common research activity, particularly in cancer cell line data; many remain unpublished). This statement should be removed.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: None

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Alex Handler Wagner

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References:

Review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Dear Dr. Goecks,

We are pleased to inform you that your manuscript 'Galaxy-ML: An Accessible, Reproducible, and Scalable Machine Learning Toolkit for Biomedicine' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Mihaela Pertea

Software Editor

PLOS Computational Biology

Mihaela Pertea

Software Editor

PLOS Computational Biology

***********************************************************