PONE-D-22-28394Towards 3D Deep Learning for neuropsychiatry: predicting Autism diagnosis using an interpretable Deep Learning pipeline applied to minimally processed structural MRI data

5946 Words, 2 Figures and 1 TablesPLOS ONE

Dear Dr. Garcia,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 13 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Francesca Benuzzi, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Our internal editors have looked over your manuscript and determined that it is within the scope of our Reproducibility and Replicability in Neuroscience and Mental Health Research Call for Papers. The Collection will encompass a diverse and interdisciplinary set of protocols and research articles adhering to transparent and reproducible reporting practices in the areas of clinical psychology, psychiatry, mental health, and neuroscience. Additional information can be found on our announcement page: https://collections.plos.org/call-for-papers/reproducibility-and-replicability-in-neuroscience-and-mental-health-research/. If you would like your manuscript to be considered for this collection, please let us know in your cover letter and we will ensure that your paper is treated as if you were responding to this call. If you would prefer to remove your manuscript from collection consideration, please specify this in the cover letter.

3. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical.

4. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well. 

5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 

Additional Editor Comments:

Dear authors,

I have sent your paper to three reviewers, and two of them have recommended major revisions but, I firmly believe that the issues raised can be addressed.

In particular, Reviewer 2 recommends greater transparency and accuracy in describing the methods, so that even those who are not familiar with the subject matter can fully understand the methodological aspects of the work.

Reviewer 3, on the other hand, emphasizes the need for a thorough revision of the text and figures.

Best regards,

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: I Don't Know

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors propose an AI method, based on 3D Deep learning algorithms, for the classification of ASD vs typical subjects. They used public datasets such as ABIDE I and ABIDE II together with ADHD200.

There are in my opinion some important issues:

-the authors explicitly avoid to perform spatial normalization to a common template. The reason for this choice is that additional post processing steps may introduce biases. However, a good spatial normalization is a powerful way to favor 3D algorithms convergence and to make importance features mapping interpretable.

Indeed, the authors process the gradCAM importance maps to be interpretable by adding an additional segmentation step and looking for correspondences between thresholded maps and segmentation which is quite complicated (and prone to biases such as those they tried to avoid).

In addition, images acquired in the same site may be acquired with a standard positioning protocol which will be very peculiar for 3D classifier such those trained by the authors. In the case of not well-balanced samples (for example with a predominance of ASD subjects in an acquisition site, this would imply a strong site biases which may be confused by an ASD effect.)

-the performances on the test sets are very low and almost comparable with the chance level. Since the test sets are the only data on which a classifier demonstrates its capability to generalize, the throughout discussion on the importance features does not reflect the real brain areas by which the classifier is guided to distinguish ASD vs TD but the areas by which the classifier managed to optimize its performance on the training and validation data.

-In S10 figure 8 it is clear that, despite the ASD gravity of deficit the performance of the classifier is completely dominated by the effect of site.

-The authors did not show any MR brain images before and after brain extraction and intensity normalization. I suggest to show for at least 3 examples taken from different acquisition sites the raw images, the “brain extracted” images and the intensity normalized images to better understand the real input of the 3D deep classifiers.

-The authors did not show any example of gradCAM maps and brain segmentation. This is important to understand how much focused are the maps of importance in comparison with the ROI segmentations.

-There are some parts in the results section for which the correspondent description in the methods section is missing (the effect of gender, the effect of age, the multi-site effect…).

-S1 table 1 and 2. Please reduce the significant figures.

- S10 Fig 7, S10 Fig 8, and S10 Fig 9 please limit the x axis to 1.

Reviewer #2: This is a nicely written paper but the emphasis does not seem quite right. I am not a computational scientist but I do understand this technique in a broad way in line with PLOS 1's wide scope. So some issues may reflect my lack of knowledge on what is standard in the literature on machine learning.

1, I understand this approach takes a lot of computational resources but computer time used is not the same burden as human time used. So I don't buy that saving processing time by not morphing the brains into MNI space is a major +. It makes more sense to do it for substantive reasons (i.e, capture anomalies about brain structure in ASD - but what? gyrification in temporal lobe?).

2. One take home I got was - these machine learning approaches on structural data including deep learning do not do a great job predicting autism. Site seems to matter - so maybe that is a real thing as culture does seem to affect behavioral differences in autism (koh & Milne).

3. A major plus is going back to identify the relevant structures. However,

a. you do not give many references or ideas why these structures are important except for language, which is only true of some of them (for instance, TPJ = mentalizing), or whether/how they differ in ASD (based on references).

b. you do not tell us how these structures differ in your data, but it seems that you cannot, right? COuld you test one theoretical prediction to examine this? If you have an idea of how they might differ (ie thicker frontal cortex).

4. You present too many options for models, how to identify relevant areas etc. Decide which you think is (most) correct, and minimize discussion of the other options. Can you identify if the differences in specificity and sensitivity between models are significantly different and identify the best one? This also makes the figures too busy.

5. A big question - why would you want to use expensive, hard to collect neuroimaging data to identify autism? you are likely to get better, cheaper predictive models from eye movements to a set of pictures or other behavioral measures. So I am not convinced we would ever use fMRI to identify autism, but if you can go back and identify the relevant regions and how they differ - that does seem useful.

6. Another major plus are the age and gender analyses. Perhaps expand a bit further, illustrate age, and clarify if they interact at all. why do predictive regions change with age? Are there typical changes in these age ranges that differ in ASD? of course more examination of the effect of IQ and comorbid conditions (like ADHD) would also be great.

7. I found these items jargony/ did not understand the concept fully:

a. Guided Grad-CAM

b. not sure why you would ever use a 2-D program for 3-D brain data?

c. would like more details in why gyrification differs in autism and how, since this is a good reason to not morph brain data (i.e., can change side of gyrus activation is evident on).

d. differences/similarities between Machine learning vs Deep learning vs traditional multivariate approaches

e. not sure what cropping and padding is - just adding on to the edges?

f. I liked the discussion of false positives, negatives, etc but it gets a bit confusing as to the actual implications? Perhaps you could clarify with a table or just clarify writing?

g. section 4.4 does not make sense to me, in light of already using BET software (removing info outside the brain).

f. not sure what the inference step is?

Nice job in transparency and using large datasets.

Reviewer #3: In summary, this manuscript aims to address the strong data heterogeneity and interpretability of deep learning in predicting Autism diagnosis based on structured MRI data and then proposes an interpretable predictive pipeline for inferring Autism diagnosis using 3D Deep Learning applied to minimally processed structural MRI scans. The design of the solution looks like reasonable and the analyses on experiments results are justified. However, the manuscript must be significantly revised and address the following issues.

1. The manuscript needs to be carefully revised from beginning to end. There are language errors and it needs to be polished by professional language experts.

2. The authors claim that one contribution of the study is the adoption of the minimal preprocessing pipeline , i.e. no transformation to template space, to avoid any impact of brain normalization on the detecting of Autism-related alterations in brain structure. However, the manuscript lacks a comparison with the relevant methods of using spatial template matching, which is not rigorous enough.

3. The resolution of the graphics in the current manuscript is too low, please replace it with high-quality graphics.

4. A large number of figures, tables, and experimental results are attached to the supporting material, which are important for method demonstration and should be included in the body, such as Table S5, S6, etc.

5. The proposed solution to the problem of data heterogeneity in the manuscript was achieved by training large data sets, which, in my opinion, did not completely or even alleviate the relevant problems, and therefore the description of this contribution should be removed. At the same time, to alleviate the problem of data heterogeneity, the author can explore the direction of reducing the distribution difference between the source domain and the target domain.

6. The performance of the autism diagnostic task in the manuscript was modest and lacked comparison with existing and conventional methods.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-22-28394R13D CNN for neuropsychiatry: predicting Autism with interpretable Deep Learning applied to minimally preprocessed structural MRI dataPLOS ONE

Dear Dr. Garcia,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jun 24 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Yangsong Zhang, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

The authors should address the comments from the reviewers.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Partly

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: the brain bases of asd is a messy confusing topic, although there are some consistencies in the literature. This paper is a good attempt but I am not sure that the authors have convinced us that looking for a single, brain signature makes sense for diagnosing a behaviorally-defined and heterogeneous disorder with many genetic bases

Reviewer #3: All concerns raised previously have been properly addressed. The paper is now in a better shape and ready for acceptance.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

3D CNN for neuropsychiatry: predicting Autism with interpretable Deep Learning applied to minimally preprocessed structural MRI data

PONE-D-22-28394R2

Dear Dr. Garcia,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Yangsong Zhang, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

The manuscript was significantly improved after two round revision, and can be accepted for publication.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thanks for addressing my concerns. I still wonder how to address the diversity in autism but it is a difficult topic.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

**********