PONE-D-21-15650

Using machine learning to determine the shared and unique risk factors for marijuana use among child-welfare versus community adolescents

PLOS ONE

Dear Dr. Negriff,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I apologize for the unusually long time that the review process took, but I needed to secure good reviews both from the theoretical and methodological points of view. Both reviewers see potential in your manuscript and I agree, but there are several aspects that require improvement. Nonetheless, they don't identify unfixable flaws, hence, I am confident you can address all their comments. I will send your review to the same reviewers.

Please submit your revised manuscript by Apr 02 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Carlos Andres Trujillo, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following financial disclosure: "Funding acknowledgements: National Institutes of Health Grants R01HD39129 and R01DA024569 (to P.K. Trickett., Principal Investigator)."

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." 

If this statement is not correct you must amend it as needed. 

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. 

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well. 

5. We note that you have referenced "Margolin G. The Youth Symptom Survey Checklist. Los Angeles, CA: Unpublished manuscript; 2000" which has currently not yet been accepted for publication. Please remove this from your References and amend this to state in the body of your manuscript: "Margolin G. The Youth Symptom Survey Checklist. Los Angeles, CA: Unpublished manuscript; 2000" as detailed online in our guide for authors

http://journals.plos.org/plosone/s/submission-guidelines#loc-reference-style.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The study proposes a statistical novel approach to understand the interplay of risk factors of marijuana use, comparing a child welfare population and a non-welfare population. The authors argue that there is little research addressing the specific risk factors that affect welfare children, and that they could differ from community children. The manuscript is well written and explains clearly step by step the LM which helps understand the process for the analysis

When the introduce variables such as depression, anxiety and PTSD as risk factors, the age of onset of such psychological issues should be addressed, because those problems could be the result of drug use (and marijuana not being the onset drug it may be more probable), then they will no be risk factors, but outcomes. If the age of onset is not clear or the authors do not have it, then it should be addressed in the limitations section.

The researchers included the use of alcohol in their variables, but it is more abuse than use, which is different. They address the alcohol use by asking how many times in the past 12 months have your passed out drunk. Even if the answer is never, the adolescent may be drinking a lot, but he is not getting drunk, or he could have been drunk but not necessarily passed out. It is clear why they have dichotomized the dependent variable (marijuana use), because of the model they are using, but it should also be reviewed what happen when the variable is maintained with its full scope of categories.

For better understanding, it is important also to explain how was measured peer drug use. The authors said it was measured through one item, but they do not explain how. It is important that at the introduction, the researchers clarify if they are really measuring self-esteem, or they are just measuring competence and body image. They should support this notion of self-esteem by literature that argues that this is self-esteem, and not self-efficacy or just self-competence. It is also important to discuss why this kind of variables have no impact on the dependent variable in either of the two groups.

It is really interesting the use of ML to analyze the risk and protective factors, because it reveals new ways in which the risk and protective factors interact. The study of this factors through LM can be an ideal analytic approach to studying multiple variables such as risk and protective factors. The study has several strengths, including its data-driven approach to study many factors that could influence marijuana use in both samples. One of the things that I could question about the paper is the way use of marijuana was measured. The possible answers were some, a lot, none. I think this type of categorization can diminish the possible variance that could be found in this kind of behaviors. It is also difficult to understand what the difference could be, for the adolescent, between a lot and some use.

It is really surprising the result that parental closeness is a risk factor for children in welfare. Even though the authors give some hypothesis, it could also be interesting to explore, or to propose for future research if, maybe based on the social development model (Hawkins & Catalano), the family, in certain circumstances could become a risk factor, because the child or adolescent has an affective attachment, so she follows her family behaviors, and parents could be using drugs themselves. It could also be a way the adolescent reacts depending on her placement. The authors, in their limitations said that they did not account for placement as a variable so they could pair both samples, but maybe that is one of the reasons of this result. I am not sure that the dichotomization of the race variable is accurate.

Another surprising result is that peer delinquency reduces marijuana use in community sample. I the researchers suggest it was a variable that did have issues in the weight it has, but I think that the problem could be how the adolescent sees his peers, or the dichotomization of the dependent variable. The authors should explore a little farther this result.

Finally, I don’t agree that black and Latino should necessarily share the same backgrounds or contextual variables.

Reviewer #2: PLoS One Review:

Using machine learning to determine the shared and unique risk factors for marijuana use among child-welfare versus community adolescents

1. I suggest more extension and detail in how decision trees work in ML techniques. No reference in lines 56-58: “Not only can ML provide potentially more accurate predictive models, but techniques such as decision trees can provide new insights into non-linear relationships.”.

2. Lines 82-83: “there are no studies delineating both the shared and unique risks” is the italic necessary?

3. Why did CW=child welfare and non-CW turn into Line 117: maltreated and comparison groups? Please have uniformity.

4. What does a “biracial” participant’s race mean?

5. According to data analysis, why do not present results before and after multiple imputations? The missingness proportion is less than 2%, but if you apply multiple imputations, results sure present how data and results look before imputation.

6. Here is one serious concern with the analytical approach: Lines 242-243 describe how “generally a value higher than 0.7 designates a good model, and higher than 0.8 a strong model”. Table 3 presents AUC for all models between 0.79 and 0.83. That means all models (i.e., Logistic Regression, Lasso and SVM) are good or strong. Why do you opt for different models under small and marginal differences?

7. Following the previous comment in Line 237- 239: In linear SVM, one can interpret the magnitude and sign of the coefficient in the linear hyperplane similarly to the coefficients in Logistic Regression and Lasso. What references do you have for this affirmation? Also, it is unclear whether your study is a comparison (e.g., CW=child welfare and non-CW), but you might use non-comparable models (e.g., Logistic Regression vs SVM). Unusually, two different models are recognized for two independent samples under the same analysis. For CW, it is SVM, and for N-CW, it is the Logistic Regression. It is necessary to have a better explanation and justify with technical background why a model may be more applicable to one group and not to the other. This point needs a deep rationale for your data analysis proposal.

8. Line 272 It is “perturbed” or “permuted”?

9. Why do you use the restrictions of the Lasso model and the technique called Backward Feature Selection? It seems like there are too many restrictions for the exploration of Line 46: “further work to delineate the relative importance of known predictors of substance use /…/”.

10. How sensitive is the Permutation Feature Importance (PFI) to the SVM and the logistic regression results?

11. It is not easy to interpret the values from the Permutation Feature Importance (PFI) analysis. However, if the arbitrary selection of models we asked about in point 7 is justifiable, at least all results in all models should be presented for a fair comparison.

12. Since the authors claim that this is a study “Line 396-398: to provide evidence regarding the comparability of risk factors for substance use among CW-involved versus non-CW-involved youth, but the first to so within one study and using machine learning approaches.”, it is capital to clarify the analytic concerns mentioned before. The richness of their approach will reside in how well the points 5 to 11 are corrected and rationalized.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Juan J Giraldo-Huertas

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: PLoS One Review_PONE-D-21-15650.docx

Using machine learning to determine the shared and unique risk factors for marijuana use among child-welfare versus community adolescents

PONE-D-21-15650R1

Dear Dr. Negriff,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Carlos Andres Trujillo, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I think the authors have adressed all my concerns. I think it has improved and is ready for publication. I really like the research.

Reviewer #2: I appreciate the effort and clarity of the authors in every answer to the comments. Also, I have no additional comments for the authors or concerns about dual publication, research, or publication ethics.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Angela Trujillo

Reviewer #2: Yes: Juan Jose Giraldo-Huertas

**********