PONE-D-21-31247Machine learning outperformed logistic regression classification even with limit sample size: A model to predict pediatric HIV mortality and clinical progression to AIDSPLOS ONE

Dear Dr. Tagarro,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Mar 04 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Zaher Mundher Yaseen

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include a complete copy of PLOS’ questionnaire on inclusivity in global research in your revised manuscript. Our policy for research in this area aims to improve transparency in the reporting of research performed outside of researchers’ own country or community. The policy applies to researchers who have travelled to a different country to conduct research, research with Indigenous populations or their lands, and research on cultural artefacts. The questionnaire can also be requested at the journal’s discretion for any other submissions, even if these conditions are not met.  Please find more information on the policy and a link to download a blank copy of the questionnaire here: https://journals.plos.org/plosone/s/best-practices-in-research-reporting. Please upload a completed version of your questionnaire as Supporting Information when you resubmit your manuscript.

3. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. 

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

5. Thank you for stating the following in the Acknowledgments Section of your manuscript: 

This work has been supported within EPIICAL project by through an independent grant to the PENTA (Paediatric European Network for Treatment of AIDS) Foundation. The funders had no role in study design, data collection, analysis, and interpretation, or manuscript preparation.

We note that you have provided additional information within the Acknowledgements Section that is not currently declared in your Funding Statement. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. 

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: 

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 

6. Please note that in order to use the direct billing option the corresponding author must be affiliated with the chosen institute. Please either amend your manuscript to change the affiliation or corresponding author, or email us at plosone@plos.org with a request to remove this option.

7. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

8. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

9. One of the noted authors is a group or consortium [EPIICAL Consortium]. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for asking my opinion about the manuscript entitled ‌‎"Machine learning outperformed logistic regression classification even with limit sample size: A model to predict pediatric HIV mortality and clinical progression to AIDS" This manuscript ‎aims to compare the performance of 7 algorithms in the prediction of 1-year mortality and clinical progression to AIDS in a small cohort of infants living with HIV from South Africa and Mozambique. I believe that this manuscript should ‎be minor revision:‎

The aim of this study (Page 1- Abstract) was there explore compare the performance of 7 algorithms in the prediction of 1-year mortality and clinical progression to AIDS in a small cohort of infants living with HIV from South Africa and Mozambique. There were several good things about the paper, such as ‎aim good. But the abstract should be reformulated and the ‎objective of the study should be well highlighted.‎

‎1. The abstract is very long.‎

‎2. In the introduction, include the significance of the study as ‎well as novelty. What makes the study different from the rest and ‎what does it add to the current knowledge?‎

‎2.1. In the introduction, the authors should have explained the ‎purpose of this study and the existing gaps in this field and ‎explain why this study was conducted.‎

‎3. In the material and methods section, include sample size justification. ‌‎

‎4. Conclusions are too short and not supported by results.‎

‎5. References are relevant, correct, and not recent. The number of ‎references should be increased.‎

‎6. There are a lot of grammatical errors. This must be taken care ‎of and addressed.‎

Reviewer #2: I find out this article very interesting. It construction is very good and language fulfills scientific research standards.

Selected methods from ML for comparison are general enough and popular enough. I recommend this article for publication.

However I want to authors check if the the sigma value in the SVM method is accurate. If the sigma is very small the SVM tends to make a local classifier, larger sigma tends to make a much more general classifier. Please check if the optimization of this parameter was done right.

The second thing, which will make this article more clearer to the reader, would be strict references to the Project R packages. I am familiar with this software, but I think that if someone is not, it would be hard to him to reproduce the methods used in results discussion.

Reviewer #3: Summary:

The authors compared seven well-understood machine learning (ML) approaches to the problem of predicting mortality and progression to full AIDS on infants with HIV from South Africa and Mozambique. They compared prediction accuracy, sensitivity, and specificity of these methods to highlight the relative benefits of some ML methods and especially against logistic regression, a very common prediction method in medicine. Developing a more accurate predictive model for health outcomes of HIV-infected infants is a very worthwhile task.

Technical Soundness:

The largest obvious challenge with this study, and one that the authors openly recognize, is the small dataset to be used for training and validation. With a 70/30 test-train split on n=100 observations, I believe the authors are challenged to convincingly demonstrate the generalizability of their comparative study. Some ML methods intrinsically perform better than others with smaller training datasets (e.g. Random Forest vs. ANN), but I remain unconvinced the the relative performance of the algorithms described herein would still hold true if one could train on more data. In other words, are the reported relative results really indicative of the true efficacy of the methods or are they significantly influenced by data sparsity alone?

I think it would benefit this comparative study to increase the number of observations for training and validation by at least an order of magnitude. If that can't be done in a practical way with actual observational data due to real-world constraints, I would recommend that the authors look at tabular data augmentation methods to synthetically increase the size of the training and test sets. Tabular data augmentation such as Test-Time augmentation (TTA) can synthetically add new observations by adding small amounts of gaussian noise to duplicated data rows. Libraries such as DeltaPy (https://github.com/firmai/deltapy) also assist with meaningful tabular data augmentation and Generative adversarial networks (GANs) and other methods have also been used for this (https://arxiv.org/abs/2010.00638). I request the authors to find a way to train and test their algorithms on more data to at least quantify/estimate the effect of small dataset size on the relative performance of the tested results.

I would have liked to see some attempt at feature importance estimation under all 7 different algorithms. Which independent variables have the most influence on the balanced accuracy of various ML models? While a bit more challenging for some methods such as ANN, determining relative feature importance under logistic regression and Random Forest is direct and straightforward. Even with ANN, you should be able to do Permutation Feature Importance estimation. Ideally, I would like to see a table of algorithms vs. features/variables ranked by importance. In other words, which features contribute the most to prediction accuracy across the various ML methods? And is there consistency in feature importance across ML methods?

Figure 2 shows the AUC under the ROC curve and is very telling. An AUC of 50% indicates a totally random classifier, and several of the reviewed algorithms approximated a random classifier with an AUC of close to 50%: Logistic (51.7% AUC), SVM (46% AUC), ANN (52.9% AUC). Really, only Random Forest performed acceptably (73.2%) by the AUC metric, probably because RF had the highest sensitivity. Still, an AUC of 73% is not great. I would love to see how these AUC might change/improve as a function of additional training data.

Given the known class imbalance in the data, the authors chose to randomly subsample the already limited training data so that the class frequencies matched the least prevalent class. Another way around this might have been to use a confusion matrix, but report balanced accuracy [(sensitivity + specificity)/2] instead of simple accuracy.

Table 2 shows the tuned hyperparameters discovered by doing a gridded 10-fold cross validation. These are not comprehensive hyperparameters for these algorithms, so it would be helpful to understand how these were chosen. For example, did you only optimize mtry for RF? How did you chose this hyper parameter instead of ntree, sampsize, maxnodes, etc.? Can you include justification for which hyperparameters you chose to optimize for each method?

The data is well-described in the manuscript, but the data is not publicly available. The authors stated that the data is protected by GDPR. I suspect that a sufficiently anonymized version of these data could have been made public in a way that did not conflict with GDPR. If not, please include the actual URL or DOI to the REDCap instance holding these data with better instructions for requesting access.

It would be helpful to review your R source code. Can the authors please publish this source code, perhaps in a public GitHub repository and share the URL or DOI to access to the source code?

Minor editing: There are various misspellings such as "enrolment" instead of enrollment on lines 46-51. Or "tunning" instead of tuning in Table 2. Lines 181-184 have some english word choice problems such as "These" instead of "This" or "will death" instead of "will die".

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Luke Sheneman

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-21-31247R1Machine learning outperformed logistic regression classification even with limit sample size: A model to predict pediatric HIV mortality and clinical progression to AIDSPLOS ONE

Dear Dr. Tagarro,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

============================== Please submit your revised manuscript by Oct 28 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Zaher Mundher Yaseen

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thanks to the corresponding author, he was successful in ‎answering our questions.

Thank you for asking my opinion about the ‎manuscript ‎entitled ‎‎"Machine learning outperformed logistic regression classification even with limit sample size: A model to predict pediatric HIV mortality and clinical progression to AIDS". This ‎manuscript ‎aims to compare the performance of 7 algorithms in the prediction of 1-year mortality and clinical progression to AIDS in a small cohort of infants living with HIV from South Africa and Mozambique. I believe that this manuscript should ‎be ‎minor revision.

Reviewer #3: The authors have revised their manuscript wherein they compared seven well-understood machine learning (ML) approaches to the problem of predicting mortality and progression to full AIDS on infants with HIV from South Africa and Mozambique. The authors should be congratulated for tackling such meaningful and important work.

The authors responded to all reviewer comments and made corrections or additions where possible to address reviewer feedback.

The largest problem with this study is the size of the dataset (n=100), as the authors fully recognize. It is not convincing to compare/contrast the performance of machine learning methods against only one extremely small dataset. There is not enough evidence here to demonstrate that Random Forest will generally outperform other methods for these kinds of data. All one can say is that RF outperformed other methods for this single, small dataset. That is an interesting datapoint, but I'm afraid I still remain unconvinced due to lack of sample size and/or replicates. If real-world data was not available (for practical reasons), I would have liked to see some kind of data augmentation via simulation to increase the length of this dataset OR simulate a number of small synthetic n=100 datasets with similar feature distributions. The authors mentioned that their goal was to compare ML methods specifically under sample-size constraints. That is fine, but the investigators should have more replicates (i.e. more n=100 datasets) to provide a convincing case. Comparing these ML methods against real + simulated data would instill more confidence in their comparative study. The authors mentioned that they will seek to explore simulation in future work, but they did not do so for this manuscript. Small sample size with one dataset is still a problem here and the authors have not offered a solution in their response to the reviewers.

I am satisfied that the authors have done feature importance estimation as now mentioned in the results section and supplemental material. I am curious how this estimation was done for methods like ANN, where is it not always clear how features are used in hidden layers. It would be helpful if an extra sentence or two describing how feature importance estimation was done would be helpful. The authors now included Fig 1 in the supplemental material, but I don't have access to that as a reviewer so I cannot comment on that supplemental figure.

The reviewers did a good job at better describing their tunable parameters for all tested algorithms in the Methods section and Table 2. This increases the interpretability and reproducibility of this work.

The authors have made their R code publicly available and have provided a DOI link.

The lack of openly publishing these data may still be an issue. The authors cannot openly and freely share data for a variety of very good reasons, and they have provided alternative methods for investigators to request data on a case-by-case basis. Whether this meets the PLOS ONE standards and requirements for open data is up to the Editor and PLOS ONE.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #3: Yes: Lucas Sheneman

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Machine learning outperformed logistic regression classification even with limit sample size: A model to predict pediatric HIV mortality and clinical progression to AIDS

PONE-D-21-31247R2

Dear Dr. Tagarro,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Zaher Mundher Yaseen

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: We feel that the manuscript has extensively improved. ‎Thanks to the corresponding author, he was ‎successful in ‎answering our questions. (ACCEPT).‎

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********