Peer Review History

Original SubmissionMay 13, 2022
Decision Letter - Gregg Roman, Editor
Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

PONE-D-22-14030The Image Data Explorer: interactive exploration of image-derived dataPLOS ONE

Dear Dr. Heriche,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Both reviewers found the idea and work to have value for the analysis of imaging data.   Both reviewers had some constructive comments that I believe would help improve.  Reviewer #1 had four specific comments and questions that I believe are specific and helpful.  Please respond to each in your response.  Also, please improve the WIKI as suggested and answer the questions in comment #4 and  in #1.  This would be necessary for acceptance of the manuscript for publication in PLoS One.        

Reviewer #2  also had several useful suggestions.  This reviewer listed two drawbacks.  Even if you decide not to directly address them in the manuscript (something I suggest, but will not require), please address/answer them in your response. This reviewer also sought a way to make the use of the software easier by suggesting a video. Please be thoughtful in how you can make the use of this tool easier, which has direct befits for the paper and the tool's adoption by the imaging community.    

Please submit your revised manuscript by Aug 06 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Gregg Roman, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please update your submission to use the PLOS LaTeX template. The template and more information on our requirements for LaTeX submissions can be found at http://journals.plos.org/plosone/s/latex.

3. Thank you for stating the following in the Funding Section of your manuscript: 

"Work by BSS and YS was supported by EOSC-Life under grant agreement H2020-EU.1.4.1.1. EOSC-Life 824087."

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. 

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: 

"Work by BSS and YS was supported by EOSC-Life under grant agreement H2020-EU.1.4.1.1. EOSC-Life 824087.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

********** 

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This Manuscript introduced a tool for interactive image data visualization. The image data explorer (IDE) can be used as a web application or installed on local computers. IDE integrates interactive linked visualization of images and derived data points with exploratory data analysis methods, annotation, classification and feature selection functionalities. The manual of the Software needs to be optimized for new users.

1. How to input the images to your data base if users do not have a S3-compatible object store? What is the address of the ‘one common root directory accessible by the server component of the IDE’?

2. The IDE wiki need to be updated and optimized. I tried RStudio installation based on the wiki, but still could not get the code running. I would suggest to provide corresponding compatible RStudio download links and step-by-step installation and screenshots.

3. With all these plotting, machine learning and interactive functions, it is better to provide some statistical tools for better data analysis and exploring. For example, incorporating the ANOVAs and Post-hoc test would be very helpful.

4. In the Results session, In this case, how do you conclude ‘the resulting classifier has an accuracy of 74% which is above the no information rate and the most significant feature corresponds to nucleolus size’? How do you define the no information rate? In figure 3, what is the range (x-axis) of feature importance? The most important feature is the Nucleoli size and is less than 0.2, what does this mean?

Reviewer #2: This paper presents a tool (more precisely a web application also available as a Desktop application) that permits dynamic visualization of images and their (numerous) features and offers a set of statistical tools to apply to these features. This goes from exploratory data analysis to classification and seems very flexible, so that you could integrate your own algorithms.

Acquisition of big data in microscopy becomes more and more frequent (e.g. high content screening or mosaic acquisitions). The community definitely needs tools to be able to visualize and classify features obtained by any analysis workflow on this amount of data. This is offered by this software, with very simple inputs: the images and the associated features stored in a table (CSV type e.g.), generated by any workflow beforehand.

With no doubt, a specific attention has been paid to the interface: it seems very user-friendly and the dynamic interaction between the results table and the corresponding structures in the image is really effective; it can help detect anomalies in the workflow (e.g. bad segmentations) and then create more accurate results.

However, two drawbacks:

- the limitation to 3-dimensions for analyzed data (today data acquired on the microscope are often 5D); I do not understand this limitation since color-z-time information could be included in the table and the application already reads TIFF images (of course one would be limited by its own computer memory)

- the ROIs are only represented by one point (if I understood well) and not by the "real" shape; I understand that it is due to the fact that the analysis is performed beforehand but maybe could be interesting to find a way to display them as Region of Interest

Some (minor) remarks:

- The authors could also cite the Plugin BAR of ImageJ; even if it does not implement any statistical tool, the ROI color-coding can really help the user to define (instinctively) which parameter can describe the best, for instance, the difference between structures morphology

- I understand the choice of R and shiny framework that offers the adequate tools for this project, but I suggest to find a way (if possible of course) to launch it from another software in order to avoid a workflow using several softwares separately

- a video showing how the software works could also be a bonus

********** 

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Revision 1

We would like to thank the reviewers for their useful feedback. Below are our point by point responses to the reviewers' comments.

Reviewer #1:

1- How to input the images to your data base if users do not have a S3-compatible object store? What is the address of the ‘one common root directory accessible by the server component of the IDE’?

When the IDE runs on a user’s computer, the images can be on any file system the user has access to from that computer. This also applies when the IDE is run on a server, i.e. the images can be on any file system the server has access to. The only requirement is that the images must be under one common top-level directory (as opposed to, for example, being distributed across multiple file systems). We refer to this common top level directory as the image root. S3 storage is an alternative for remote access, i.e. when the images are not on a file system accessible to the computer the IDE is running on. We have now clarified this in the text and in the documentation.

2- The IDE wiki need to be updated and optimized. I tried RStudio installation based on the wiki, but still could not get the code running. I would suggest to provide corresponding compatible RStudio download links and step-by-step installation and screenshots.

We’re sorry for that. We realized that the instructions made some assumptions about requirements being satisfied on the user’s system. We’ve now written step-by-step instructions to include these requirements and added screenshots where relevant. As part of the process we also uncovered a bug in the latest version of one of the required R packages that prevented its installation and thereby preventing the IDE from running. Following our report, the package maintainer has now fixed this bug.

3- With all these plotting, machine learning and interactive functions, it is better to provide some statistical tools for better data analysis and exploring. For example, incorporating the ANOVAs and Post-hoc test would be very helpful.

We’re happy to add functionalities to support new use cases. In fact a statistics workspace had been planned but wasn’t implemented because none of the projects that started using the IDE needed it. However, we’ve now added the statistics workspace with one-way ANOVA and post hoc tests and describe it in the manuscript.

4- In the Results session, In this case, how do you conclude ‘the resulting classifier has an accuracy of 74% which is above the no information rate and the most significant feature corresponds to nucleolus size’? How do you define the no information rate? In figure 3, what is the range (x-axis) of feature importance? The most important feature is the Nucleoli size and is less than 0.2, what does this mean?

We apologize for the lack of details. The no information rate is the best performance a naive classifier could reach by always assigning the label of the most abundant class and is therefore the proportion of the most abundant class.

The measure of feature importance used in the IDE is called the gain. The gain quantifies the improvement in accuracy obtained when the corresponding feature is included in a branch of the (tree-based) classifier. The values are relative and sum to 1 over the features such that when comparing two features, the one with the highest value is more important for the classifier performance.

We have now expanded the corresponding section of the text with more explanations.

Reviewer #2:

- the limitation to 3-dimensions for analyzed data (today data acquired on the microscope are often 5D); I do not understand this limitation since color-z-time information could be included in the table and the application already reads TIFF images (of course one would be limited by its own computer memory)

Images of higher dimensions are supported, i.e. they can be read, but the viewer itself is limited to 3 dimensions. So in the case of higher dimension images, the IDE merges the channels into a colour image and the user has to decide which of depth or time should be displayed. We currently don’t have a 5D viewer that would allow for the kind of integrated web-based three-way interactivity the IDE is offering. However, in the future we plan to leverage the development of the next generation image file format ome.zarr for image viewing, for example by modifying the viv library (https://github.com/hms-dbmi/viv) to support the kind of interactivity needed by the IDE.

- the ROIs are only represented by one point (if I understood well) and not by the "real" shape; I understand that it is due to the fact that the analysis is performed beforehand but maybe could be interesting to find a way to display them as Region of Interest

We agree that representing ROIs as shapes on the original image would be an interesting feature and considered it earlier in the project. We decided to forgo its implementation for two reasons: it is technically more challenging to implement (i.e. drawing shapes on the fly on the original image without delaying display to the user) and, since we rely on pre-computed data, there is no simple and standard representation in tabular form (as opposed to a point which only needs x, y, z, t coordinates and which most analysis software already associate with objects, e.g. center of mass, brightest point). However as many image analysis workflows produce a label mask image, we added a second interactive image viewer to allow viewing the mask on the same screen as the original image. This being said, discussions are also ongoing on the topic of ROI representation in the ome.zarr format which as mentioned above we plan to use in the future.

- The authors could also cite the Plugin BAR of ImageJ; even if it does not implement any statistical tool, the ROI color-coding can really help the user to define (instinctively) which parameter can describe the best, for instance, the difference between structures morphology

We have added mention of the BAR plugin as example of data exploration that’s possible in ImageJ.

- I understand the choice of R and shiny framework that offers the adequate tools for this project, but I suggest to find a way (if possible of course) to launch it from another software in order to avoid a workflow using several softwares separately

The IDE can be started from other software either as an R script (for example using the command-line shown in the installation wiki) or in the remote case by linking to the appropriate URL. However, automatically transferring data from an external software to the IDE is currently not possible and would require the development of a complex API that the external software would need to use.

- a video showing how the software works could also be a bonus

We’ve added a short (less than 2 min) introductory video to the repository at https://git.embl.de/heriche/image-data-explorer/-/raw/master/videos/intro.mp4

It focuses on how to get started with the data input area and we now plan to follow up with a series of short videos each dealing with one of the IDE workspaces.

Attachments
Attachment
Submitted filename: Response to Reviewers.pdf
Decision Letter - Gregg Roman, Editor

The Image Data Explorer: interactive exploration of image-derived data

PONE-D-22-14030R1

Dear Dr. Heriche,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Gregg Roman, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Formally Accepted
Acceptance Letter - Gregg Roman, Editor

PONE-D-22-14030R1

The Image Data Explorer: interactive exploration of image-derived data

Dear Dr. Heriche:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr Gregg Roman

Academic Editor

PLOS ONE

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .