Peer Review History

Original SubmissionSeptember 19, 2019
Decision Letter - Ilya Ioshikhes, Editor, William Stafford Noble, Editor

Dear Dr Zook,

Thank you very much for submitting your manuscript 'SVCurator: A Crowdsourcing app to visualize evidence of structural variants for the human genome' for review by PLOS Computational Biology. Your manuscript has been fully evaluated by the PLOS Computational Biology editorial team and in this case also by independent peer reviewers. The reviewers appreciated the attention to an important problem, but raised some substantial concerns about the manuscript as it currently stands. While your manuscript cannot be accepted in its present form, we are willing to consider a revised version in which the issues raised by the reviewers have been adequately addressed. We cannot, of course, promise publication at that time.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

Your revisions should address the specific points made by each reviewer. Please return the revised version within the next 60 days. If you anticipate any delay in its return, we ask that you let us know the expected resubmission date by email at ploscompbiol@plos.org. Revised manuscripts received beyond 60 days may require evaluation and peer review similar to that applied to newly submitted manuscripts.

In addition, when you are ready to resubmit, please be prepared to provide the following:

(1) A detailed list of your responses to the review comments and the changes you have made in the manuscript. We require a file of this nature before your manuscript is passed back to the editors.

(2) A copy of your manuscript with the changes highlighted (encouraged). We encourage authors, if possible to show clearly where changes have been made to their manuscript e.g. by highlighting text.

(3) A striking still image to accompany your article (optional). If the image is judged to be suitable by the editors, it may be featured on our website and might be chosen as the issue image for that month. These square, high-quality images should be accompanied by a short caption. Please note as well that there should be no copyright restrictions on the use of the image, so that it can be published under the Open-Access license and be subject only to appropriate attribution.

Before you resubmit your manuscript, please consult our Submission Checklist to ensure your manuscript is formatted correctly for PLOS Computational Biology: http://www.ploscompbiol.org/static/checklist.action. Some key points to remember are:

- Figures uploaded separately as TIFF or EPS files (if you wish, your figures may remain in your main manuscript file in addition).

- Supporting Information uploaded as separate files, titled Dataset, Figure, Table, Text, Protocol, Audio, or Video.

- Funding information in the 'Financial Disclosure' box in the online system.

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see here

We are sorry that we cannot be more positive about your manuscript at this stage, but if you have any concerns or questions, please do not hesitate to contact us.

Sincerely,

Ilya Ioshikhes

Associate Editor

PLOS Computational Biology

William Noble

Deputy Editor

PLOS Computational Biology

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

[LINK]

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors present a tool called SVCurator, which allows users to view images of sequencing reads from multiple different platforms for the same individual and answer questions about the presence of structural variants. The authors used this tool to interrogate putative SVs for HG002 via crowd sourcing and via expert review. The resulting labels were compared across the expert reviewers, the crowd, and compared to the GIAB benchmark calls.

Major Comments:

SVCurator is a valuable tool for visualizing sequence level evidence for structural variants across sequencing platforms. I think the community will benefit from this tool and the results of the study. I have no major suggestions.

Minor Comments:

- I’m curious about the events that the experts commonly agreed upon or disagreed most on. Can you describe these in terms of their qualitative features? Are they similar to what the non-experts agreed or disagreed upon?

- In Figure 1, Q1 shows a question about the size range for this variant – how are those ranges generated?

- Did you include any controls (e.g. images of regions where no putative SV call existed)?

- Do you have any information about experts and the crowd make their decisions? Do they weight the data in the same ways or look for particular features?

- Were there any individual SVs (e.g. rather than classes of SVs) that the crowd performed better than the experts? If so, it would be interesting to describe these.

- Figure 3 – These may be easier to read as a violin plot.

- Can you explain why the variants in the “another_var” category have such high discordance levels?

- When you say that on page 16 that the events were manually inspected – do you mean by the Authors? Or can you explain how this manual inspection is different than what has been described in other places in the manuscript?

- Figure 9 is really helpful. I think it would help the reader understand the methods and definitions of the various categories if you create something like this to include earlier on (e.g. a Figure 1A).

- Overall I would like to hear more about SVCurator – what parameters or visualizations can be tuned or varied for future studies. I imagine other groups may not have quite so much data on the same sample, but clinical groups, core labs, and SV detection tool developers may be interested in using SVcurator to review data. How flexible will SVCurator be in other studies when not all data types are present? Can you describe the API or general approach for a future user to use this tool?

- Are there any limitations of SVCurator? For example, how much depth can it reasonably display? Does it randomly downsample, etc. Are the images static (e.g. screenshots from IGV) or are users able to use IGV features to scroll, group or color reads, etc?

Reviewer #2: The manuscript entitled “SVCurator: A Crowdsourcing app to visualize evidence of structural variants for the human genome” by Dr. Zook and colleagues described an experience of crowdsourcing manual curation of germline structural variations (SV) of a Genome In A Bottole (GIAB) sample by “recruiting” 61 curators to manually review 1235 putative insertions and deletions. The user interface displays all the relevant information generated by diverse technology together on the same page (e.g. PacBio long read, 10X Genomics reads and Illumina reads), which is useful when evaluating as the user can zoom and scroll pretty easily. Overall, having a review strategy of crowed sourced evaluations to generate a reference SV data is an interesting approach and binning the curators into experts (threshold 1 and non-experts (threshold 2) is a good strategy to avoid reducing the quality of the results due to the input from an inexperienced reviewer.

Major issues:

1) The interface appears to be designed for the specific task of reviewing the GIAB samples. There is no way to upload a custom set of SVs. Without an interface for uploading the custom data, it is unclear how the publication of the tool can benefit research community was. The authors also need to provide tutorial on how to upload raw SV data/images, recruit reviewers, and collect and assess results.

2) The original SVs were generated from 30 variant callers. Can the author indicate the accuracy and sensitivity of each caller based on the curation consensus?

3) In Discussion, the authors mentioned that the crowdsourcing results could be further validated by PCR. Given the long-read sequencing support, would PCR validation still be required? It would be important to present how many of the crowdsourcing results were corroborated by at least one long-read technology (i.e. PacBio, 10X or long-read library). This information was not presented in Figure 8. It is intriguing to see that a small proportion of the SVs were supported only by <=1 technology (Figure 8A). It will be important to explain why these events are considered real.

4) The quality of the manuscript needs to be improved. The text label of Figure 1 is not readable, Figure 7 was of poor quality; Figure 8 used different width to show the bar plot and Figure 9 was first referred on page 15, ahead of Figure 6. A number of paragraphs describe the performance of a specific group of curators working with this specific dataset. This content may need to be removed as it is of little scientific interest and yields little insight. The phrase "python flask web based app" appears throughout the manuscript which does not make sense for people who are not in the field of software development.

Minor issues

1) The “help” tab on the web page is broken, and there is another dead link on the page which raises the concern that an additional image isn’t being rendered. Also, I’m not seeing any that shows where the soft clipped bases are located, which can be important for SV review. They do show realignment, but without checking the softclip orientations and locations, this can be misleading.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Revision 1

Attachments
Attachment
Submitted filename: SVCurator Reviewer Response 191126.docx
Decision Letter - Ilya Ioshikhes, Editor, William Stafford Noble, Editor

Dear Zook,

We are pleased to inform you that your manuscript 'A crowdsourced set of curated structural variants for the human genome' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

Ilya Ioshikhes

Associate Editor

PLOS Computational Biology

William Noble

Deputy Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The authors addressed all of my comments; I recommend this manuscript for publication.

Reviewer #2: The authors have addressed all the concerns raised in the previous review.

**********

Have all data underlying the figures and results presented in the manuscript been provided?

Large-scale datasets should be made available via a public repository as described in the PLOS Computational Biology data availability policy, and numerical data that underlies graphs or summary statistics should be provided in spreadsheet form as supporting information.

Reviewer #1: Yes

Reviewer #2: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Formally Accepted
Acceptance Letter - Ilya Ioshikhes, Editor, William Stafford Noble, Editor

PCOMPBIOL-D-19-01608R1

A crowdsourced set of curated structural variants for the human genome

Dear Dr Zook,

I am pleased to inform you that your manuscript has been formally accepted for publication in PLOS Computational Biology. Your manuscript is now with our production department and you will be notified of the publication date in due course.

The corresponding author will soon be receiving a typeset proof for review, to ensure errors have not been introduced during production. Please review the PDF proof of your manuscript carefully, as this is the last chance to correct any errors. Please note that major changes, or those which affect the scientific understanding of the work, will likely cause delays to the publication date of your manuscript.

Soon after your final files are uploaded, unless you have opted out, the early version of your manuscript will be published online. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

Thank you again for supporting PLOS Computational Biology and open-access publishing. We are looking forward to publishing your work!

With kind regards,

Laura Mallard

PLOS Computational Biology | Carlyle House, Carlyle Road, Cambridge CB4 3DN | United Kingdom ploscompbiol@plos.org | Phone +44 (0) 1223-442824 | ploscompbiol.org | @PLOSCompBiol

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .