PONE-D-19-34960

Study of low-dose PET image recovery using supervised learning with CycleGAN

PLOS ONE

Dear Dr. Ye,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

ACADEMIC EDITOR:

The two reviewers are in agreement that the paper is well written and interesting, but they also raise several important concerns, including i) methodological details missing, ii) additional experiments to conduct in order to increase the confidence in the conclusions (currently not entirely supported by the results).

I also suggest the authors to evaluate the produced images using other, more advanced metrics than SUVmax / mean. For instance, radiomic features have been used increasingly over the last decade to characterize PET images further than basic SUV measurements.

==============================

We would appreciate receiving your revised manuscript by Jun 14 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Mathieu Hatt, MSc, PhD, HDR

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

3.  Thank you for stating the following in the Financial Disclosure section:

"The author(s) received no specific funding for this work." 

We note that one or more of the authors are employed by a commercial company: "Zhejiang Minfound Intelligent Healthcare Technology Co., Ltd. and MinFound Medical Systems Co., Ltd.,"

a) Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

b) Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc. 

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

4. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 2 in your text; if accepted, production will need this reference to link the reader to the Table.

5. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: REVIEW: “Study of low-dose PET image recovery using supervised learning with CycleGAN”

Summary:

- In this paper, the authors tackle the problem of recovering FDPET images from LDPET images.

- They introduce a new architecture “S-CycleGAN” to learn a nonlinear mapping between LDPET and FDPET images. It is based on the well-known CycleGAN architecture but trained with an additional supervised learning loss.

- The system is compared with two other deep learning methods (RED-CNN and 3D-cGAN) with five different metrics including SUVmean and SUVmax.

Key strengths:

- The method is evaluated quantitatively for lesion SUVs with SUVmax and SUVmean. They show experimentally that it is better at preserving the SUVmax values than the two other methods.

- The paper is easy to read and understand.

Key weaknesses:

- The authors do not substantiate the choice of the CycleGAN architecture (designed for unpaired datasets) on a paired dataset. The benefit of adding a second generator/discriminator pair to transfer an image from the FDPET domain to the LDPET domain is unclear.

- This claim is unproven: “the proposed method is better at preserving image details than the other two methods”. The samples shown in figures 7 and 8 seem to support this claim, but a quantitative justification should be given with the proper metrics. Indeed, the NRMSE, SSIM, and PSNR are not adequate perceptual metrics (https://arxiv.org/abs/1801.03924). A perceptual loss (LPIPS) or a mean opinion score measure (MOS) could be used to support this statement.

Miscellaneous remarks:

- Typo in the abstract: “S-CylceGAN”

- Typo line 78: “supervise”

Reviewer #2: This paper investigates a supervised, cycle-consistent, conditional GAN to map low-dose PET brain images to full-dose images. Reported performance is promising compared to other deep learning frameworks. However, the study has two main limitations that need to be addressed before possible publication.

* A training set (99 samples) and a test set (10 samples) were defined. However, no validation set is mentioned, which is a severe limitation. In particular, it is not clear how the numerous hyperparameters (the alpha, beta and gamma loss weights, the learning rate, Adam optimizer's beta1 and beta2 parameters, the patch size, network hyperparameters, etc.) were set: if the number of hyperparameters is close to the number of test samples (10), then the selection criterion for these hyperparameters is very critical. Also, it is not clear how the stopping criterion was defined. Training is generally stopped when the validation loss is minimal. When is it stopped if no validation set was defined? When the test loss is minimal? Given the small number of samples (109), I recommend to follow a cross-validation strategy on the 99 development samples: the value of hyperparameters (including the number of training epochs) could be defined as those maximizing the cross-validation score. A final model would then be trained on the entire development set using the optimal hyperparameter values.

* Each of the four training losses (Adversarial Loss, Cycle Consistency Loss, Identity Loss, Supervised Learning Loss) seem relevant. However, it is not clear how each of them contribute to the overall performance. Note that the alpha, beta ang gamma weights are not reliable indicators in themselves. The added value of this paper would be much higher if the impact of each loss function was analyzed independently: this could be done by retraining the system without the loss function to analyze.

Minor comments are listed below:

* Other papers address the same task using CycleGANs (for instance "Yang Lei et al. Low dose PET imaging with CT-aided cycle-consistent adversarial networks, Proc. SPIE 11312, Medical Imaging 2020: Physics of Medical Imaging, 1131247 (16 March 2020); " ext-link-type="uri" xlink:type="simple">https://doi.org/10.1117/12.2549386"). Differences with these papers should be highlighted.

* The expression of the Wasserstein distance should be provided.

* Training details for RED-CNN and 3D-cGAN solution should also be provided. Moreover, methodological differences between these solutions and the proposed solution should be better highlighted.

* Why providing hardware details (NVIDA TITAN GTX GPU) if no computation times are given? Training times and inference times should be provided.

Typos:

* applicaions - applications

* an Machine Learning - a Machine Learning

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Study of low-dose PET image recovery using supervised learning with CycleGAN

PONE-D-19-34960R1

Dear Dr. Ye,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Mathieu Hatt, MSc, PhD, HDR

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Only a few minor typos need to be corrected when preparing the final version.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All concerns and weaknesses pointed out during the first round of peer review have been adequately addressed.

Typos:

- "CylcleGAN" line 253

- "Cycle-consistence loss" line 258 - "Cycle-consistency loss"

- "All experiments were conducted using the Keras with Tensorflow backend" line 188 - remove "the" and cite Keras/Tensorflow (https://keras.io/getting_started/faq/#how-should-i-cite-keras)

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No