PONE-D-20-18734

Executive functions form a single construct and are associated with schooling: evidence from three low- and middle- income countries

PLOS ONE

Dear Dr. Wray,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The manuscript has been evaluated by three independent Reviewers, please see their comments appended at the bottom of this letter. As you will see, there were several major concerns that should e addressed in a reviewed version of your study.

Please submit your revised manuscript by Sep 18 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Angel Blanch, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

3. Please note that in order to use the direct billing option the corresponding author must be affiliated with the chosen institute. Please either amend your manuscript or remove this option (via Edit Submission).

4. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 5 in your text; if accepted, production will need this reference to link the reader to the Table.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors indicate that the instruments to measure Executive Functions have been validated mainly in WEIRD samples. The goal of the paper is to address this limitation by employing an impressive database of individuals from Guatemala, the Philippines, and South Africa. The article is in general very well written. The main problem I see is that the statistical methods used are not completely suitable. Below are some suggestions for improvement that I hope the authors will find useful.

1. The statistical procedure that allows the evaluation of the generalizability of the instruments across cultures, language or other variables is the measurement invariance analysis. The works of Bach et al. (2018) and Marsh et al. (2009) illustrate how this can be done using an exploratory factor analysis approach. The separate analysis in each database does not allow us to say that the measured construct does not vary. The analysis provided by the authors could be understood in any case as a modest first contribution to the study of the measurement invariance of this test.

Work cited:

Bach, B., Sellbom, M., & Simonsen, E. (2018). Personality inventory for DSM-5 (PID-5) in clinical versus nonclinical individuals: Generalizability of psychometric features. Assessment, 25(7), 815-825.

Marsh, H. W., Muthén, B., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J., & Trautwein, U. (2009). Exploratory structural equation modeling, integrating CFA and EFA: Application to students' evaluations of university teaching. Structural equation modeling: A multidisciplinary journal, 16(3), 439-476.

2. With respect to the exploratory factor analysis included, it would be advisable to include all 9 items in the same analysis. The degrees of freedom would be positive and the fit of the model could be assessed. In addition, the correlations between the scores obtained with oblique rotation could be reported. It is also necessary to indicate which estimation method was used (e.g., ML, ULS). Finally, the use of the scree plot has been criticized because its interpretation is highly subjective. Authors should use the parallel analysis method, as it is the gold standard method for dimensionality assessment (Auerswald & Moshagen, 2019; Finch, 2020; Lim & Jahng, 2019).

Work cited:

Auerswald, M., & Moshagen, M. (2019). How to determine the number of factors to retain in exploratory factor analysis: A comparison of extraction methods under realistic conditions. Psychological Methods, 24, 468–491.

Finch, W. H. (2020). Using fit statistics differences to determine the optimal number of factors to retain in an exploratory factor analysis. Educational and Psychological Measurement, 80, 217–241.

Lim, S., & Jahng, S. (2019). Determining the number of factors using parallel analysis and its recent variants. Psychological Methods, 24, 452–467.

3. In Table 3 the authors calculate the correlation of the total test score (i.e. considering the three factors) and several criterion variables. However, this factorial model has not been tested. For it to make sense to use the total factor scores it would be necessary to show 1) that a unidimensional for all items obtains good fit, 2) that a three-factor model with all items obtains good fit and the factors are highly correlated. Without doing so, there is some inconsistency between this analysis and the factor analysis presented before.

4. I appreciate that the authors report the test-retest reliability statistic, but it would be necessary to also include Cronbach's alpha and, the most accepted, Omega coefficient (Viladrich, Angulo-Brunet, & Doval, 2017).

Work cited:

Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). A journey around alpha and omega to estimate internal consistency reliability. Annals of Psychology, 33(3), 755-782.

5. An effort should be made to clarify the goal of the study. For example, the title seems to indicate that the relationship with schooling will be paramount, in the introduction it is important to determine whether the factor structure in non-WEIRD samples is similar to that obtained with WEIRD samples, and in the results section much of the text is devoted to commenting on the results for practice effects. Moreover, it is necessary to discuss the results in relation to those obtained with WEIRD samples. What previous results on factor structure can be commented on? What has been the correlation obtained with schooling?

Reviewer #2: The paper focuses on a new area of investigation, the executive functions and schooling in three low- and middle- income countries. The topic is interesting, but the analyses are poor, the design of the study is not clear and the results appear limited.

It is not clear the real objectives of this study and the valued added of it.

I could give some suggestions to ameliorate it.

1. It is not clear the number of control group participants. How did you decide this number? Have you run power analysis? Not clear why you showed these control groups and what is their function in your study.

2. You didn’t show in a table the participants’ characteristics

3. You also noted that It could be present a memory effect in the test-retest of the FE tasks with a so less number of days between the two assessments. Why did you decide to have this measure? Clarify it.

4. What about possible gender or age differences? It would be interesting to have also this information to have a complete assessment of this EF measure in these LMI countries

5. I don’t found the tables with participants’ characteristics in each country. You stressed the schooling, but you didn’t report it in the tables and you didn’t stress the schooling situation in these countries and why it is important for the aim of the study.

6. The discussion could be more interesting adding the new suggested analysis and with the revisions suggested below.

Reviewer #3: Review of “Executive functions form a single construct and are associated with schooling: Evidence from three low- and middle- income countries” by Wray et al. for PLOS ONE PONE-D-20-18734

The present research reports a study conducted in Guatemala, the Philippines and South Africa measure the executive function (EF) in three large adults samples. The goal of the research is to demonstrate that the three target measures (a flanker task, the DCCS, and a list sort task) cohere well in assessing EF. The work does demonstrate this; it also illustrates that EF assessment in low and middle-income countries can be achieved without highly trained personnel, but with the help of portable technology (tablets). The data show good convergent validity, good to reasonable test-retest reliability, but also modest training effects.

Overall, this paper does provide good evidence for what it sets out to demonstrate, and will be solid evidence that EF assessment is feasible in low to middle-income countries. Whereas some researchers may have taken this for granted all along, the present research certainly will assuage the concerns of those who have doubted this. In this sense, this paper will certainly be useful to the field.

There are few observations that would be helpful to have the authors address, though.

1. The Guatemala sample is older and more heterogeneous than the other two samples in terms of age, and the South African samples appears to be most homogeneous (no age variation). In Guatemala the age of participants ranges from 38 to 58 years, a time during which much can happen in the development of a society. This might be in part responsible why the correlation coefficient for Guatemala is highest for schooling and lowest for South Africa. Do findings hold when the Guatemalan samples is divided in an older and a younger half? This also gets at possible age effects that one might expect within the Guatemala sample.

2. Ideally, when the factorial structure is to be compared across different samples, one would expect the authors to conduct a confirmatory factor analysis, demonstrating the convergence across the three country samples. In the present scenario, one would expect that the model would not fit very well, simply because with large samples even small differences in factor loadings would undermine the model fit (such as the fact that the loading of working memory task is much higher in the Guatemalan sample than the other two samples). Doing this may not be too critical. However, the reader might want to see one additional decimal for the factor loadings in Figure 1 (for a total of two decimals), as the fit might be reasonably good. Yet, coefficients might not be identical for the flanker task and DCCS, as currently implied.

3. Small issue: In Table 5, the performance decreases between T1 and T2 for the DCCS, and between T2 and T3 for the Raven’s progressive matrices. The corresponding t- and d-values should be negative.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Marta Tremolada

Reviewer #3: Yes: Markus Kemmelmeier

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Executive functions form a single construct and are associated with schooling: evidence from three low- and middle- income countries

PONE-D-20-18734R1

Dear Dr. Wray,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Angel Blanch, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: I found the paper really ameliorated following the reviewer's suggestions. I appreciated your hard work.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Marta Tremolada