PONE-D-23-10434

Is it cognitive effort you measure? Comparing three task paradigms to the Need for Cognition scale?

PLOS ONE

Dear Dr. Pfuhl,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I and two experts in the field have carefully reviewed your manuscript. We agree that that the study you report on is important and that the paper has promise. One reviewer recommends a minor revision, the other a major revision. I am satisfied that the reviewer concerns can be addressed in a minor revision.

R1 considers the paper to be outstanding, but makes a number of suggestions to improve the paper for its readers. Please address these points.

R2 raises some concerns about the COGED paradigm and I invite you to respond to these and discuss the points raised in your discussion section. R2 also asks for more details in you methods section, sufficient to allow a replication of your analysis. This seems a very reasonable request.

Please also note that this journal does not provide a proofreading service. So please do a careful proof read before submitting a revised version.

Please submit your revised manuscript by Jul 20 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Mark Fenton-O'Creevy, PhD

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

********** 

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

********** 

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

********** 

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Let's keep it short: This manuscript is outstanding and deserves utmost recognition. Not only does it combine six studies with reasonable sample sizes, not only did the authors use Bayesian statistics in addition to conventional hypothesis testing; not only is the manuscript well-written and easily comprehensible and comes with a thorough and well-balanced discussion of the empirical results: It also provides very important information to the growing number of researchers who investigate cognitive effort. The (non)correlation of established effort-related tasks with each other on the one hand and the correlation of two of these tasks with Need for Cognition on the other hand is per se a finding of great importance and will aid other researchers in planing their studies on effort investment and effort discounting (not only, but also, because the updated effect sizes will allow for appropriate power calculations for future studies). And all this comes with open data and code, which may stimulate re-using these data sets and trying new approaches to examine them.

There are only few remarks from my side:

I missed a section on Statistical Analyses where the software used is cited and the procedures employed are explained. Of course, this can easily be inferred from the *.jasp files, but i f one does not know or use JASP, this information is important to reproduce the results with some other software.

In the context of such a Statistical Analyses section, the power issue should be addressed (e.g., "with the presents sample sizes, we were able to detect correlations of r = .xx - .yy at an alpha level of .05 and a power of .80")

Also, I suggest to add codebooks to the data on the OSF repository. Also, some README file on how to best re-use these data (including where to download JASP) might be helpful to get the most out of these great data sets.

As one final remark: The authors might want to discuss or at least cite the study of Kramer et al. (2021), https://doi.org/10.1016/j.cogdev.2020.100978 who also investigated the relation of the COG--ED paradigm and Need for Cognition.

Overall, I regard this manuscript as an extremely important contribution the field of research on effort discounting including individual differences in this regard. I am happy that I was asked to review this manuscript (so I already now know about these important findings) and hope to see it published very soon (so that others will know about these finding, too).

Signed Review,

Alex Strobel

Reviewer #2: This manuscript describes the results of an analysis of three different cognitive effort tasks across six different studies. The research question is justified and relevant. The introduction and discussion are well written. I have two major points of criticism, one regarding the validity of the conclusions on COGED and Need for Cognition, and one on the level of detail, as well as several minor points.

Major points:

1. The COGED paradigm assumes that all participants inherently prefer the easiest level. It assigns the highest subjective value to 1-back and compares each more difficult level with 1-back in a fashion that does not allow the subjective value for any other level to exceed that of 1-back. This is highly problematic for drawing conclusions about associations with traits such as Need for Cognition. Since Need for Cognition describes the tendency to seek out and enjoy effortful cognitive activities, it would be misguided to assume that participants high in Need for Cognition prefer 1-back over any other level. 1-back is so monotonous and has such a low cognitive demand that it is highly aversive to those who enjoy effort. The subjective values returned by the COGED paradigm are therefore only able to reflect the true preference pattern of anyone who does indeed prefer 1-back over any other level. The “average indifference points” that were used here as an operationalisation of effort discounting are confounded by this distortion: Since the subjective values of participants with high Need for Cognition are artificially high for 1-back, even though they might not like 1-back at all, the “average indifference points” are also artificially high. Seen as a curve of subjective values across n-back levels, the area under the curve might in reality not even be larger for those with high Need for Cognition, it’s just that the peak of the curve is shifted to the right for them. There is a preprint that raises that concern and replicates the findings of Westbrook et al. (2013) with an adaptation of the COGED paradigm (Zerna, J., Scheffel, C., Kührt, C., & Strobel, A. (2022, March 24). When easy is not preferred: A discounting paradigm to assess load-independent task preference. https://doi.org/10.31234/osf.io/ysh3q). The results of the present manuscript should be interpreted with substantial amounts of caution when it comes to individual differences and traits like Need for Cognition.

2. The level of detail in reporting the methods is not sufficient. Especially since the results were not preregistered (or preregistration has not been stated in the manuscript), the description of the analyses is lacking clarity and specificity. A replication would not be possible with the current state of the methods/results section. I highly recommend that the authors provide explicit information in the methods/results section regarding outlier exclusion, handling of missing data, violations of distribution assumptions, transformations, formulas to compute scores such as d’ and subjective values, control variables, DVs and IV in the regressions, estimators, potential confounds, etc.

Minor points:

1. It seems like the authors accidentally added a question mark at the end of the title during submission. The title in the pdf is correct, but the manuscript name contains the question mark.

2. I do not think that stating the results in lines 62 to 66 is helpful for the reading flow. The authors might want to consider removing or replacing these lines with a very brief overview of the study design, e.g. population, online or lab, between or within.

3. Line 79, line 151, line 258, line 400+6 (unlabelled), and line 431 contain numbers for in-text citations, please replace them with the author names of the corresponding reference.

4. The grammar in line 80—81 is off (“However, recent work has questioned […] and propose […]”).

5. The brackets don’t add up in line 88 and in line 130.

6. The original paper on the COGED paradigm by Westbrook et al. (2013) referred to the levels by different colours rather than by their names to avoid anchoring effects in the effort discounting. Please justify briefly why you deviated from this method.

7. The introduction mentions the challenge of measuring effort, but remains quite superficial in its criticism. I recommend including insights from the following paper: Thomson, K. S., & Oppenheimer, D. M. (2022). The "Effort Elephant" in the Room: What Is Effort, Anyway?. Perspectives on psychological science : A journal of the Association for Psychological Science, 17(6), 1633–1652. https://doi.org/10.1177/17456916211064896

8. The description of people high in Need for Cognition as people who “engage with information to make sense of things and events” on lines 132—133 is very colloquial. The authors might want to consider changing this phrase and briefly explaining the different ways in which the elaborate information processing style affects task performance, academic achievement, etc.

9. The “than” in line 137 should be a “then”, I believe.

10. The citation of Strobel et al. (2020) in line 141 is not in the number-citation style.

11. Lines 146—149 contain four different versions of where to put the hyphen in “test-retest reliability”, please be consistent throughout the manuscript.

12. I believe the second half of the sentence is missing in line 164.

13. The reported age parameters are not consistent between studies on page 8. For some studies, the authors report mean and standard deviation, for some only the range. Please indicate whether this information is not available for these studies or, if it is, you might want to consider presenting the sample characteristics in tabular rather than text form. That would save room and make it easier to draw comparisons between samples.

14. Since some of the included studies are 10 years old, I assume that there are other publications based on the data of those studies. Please indicate those as references in the descriptions of the studies on page 8.

15. The stimulus durations of the n-back task differ greatly between studies (1.5 seconds in study 1 and 2 versus 0.5 seconds in study 3 to 6), as do the response time windows (1.5 seconds in study 1 and 2 versus 2 seconds in study 3 to 6) and response patterns (not reported for study 1 and 2 versus pressing one key for a target in study 3 to 6). Please mention briefly in what way these differences might influence the comparability between studies and whether or not you incorporated this into the analyses.

16. The methods section does not describe how the subjective values are computed. Please add this information.

17. Please include the exact formula of how you computed d’.

18. The NASA-TLX in its analogue form is measured on a scale from 1 to 20, because each line indicates a step of 5 points, 100 in total. If “21” is a typo, please correct it. If not, please justify why you deviated from the original NASA-TLX scaling.

19. If you have preregistered your analyses, please include a link to the time-stamped preregistration in the manuscript. I you have not, please justify why, and note that the analyses are exploratory rather than confirmatory.

20. Please increase the font size in figure 4, it is impossible to read.

21. The formatting of “BF10” is inconsistent throughout the manuscript, sometimes subscript, sometimes not. Please make sure it is consistent.

22. The BF10 in the fourth row of table 1 should be written in bold.

23. Please be more specific in the description of your analyses on line 354—357. These sentences do not sufficiently describe the operationalisations and regression formulas, i.e. which variables were the IV and the DV, which variables were controlled for, etc. The descriptions of the results on the following lines should be adjusted accordingly.

24. Since participants received feedback on their accuracy after each n-back level, the association of accuracy and effort discounting might be confounded by the fact that participants used the feedback as an anchor during effort discounting, preferring levels in which they knew they performed well. Please discuss this possibility briefly in the discussion section.

25. It is fantastic that the authors share their data and analysis scripts. However, looking into the subfolders of the studies on OSF, I think other researchers will find it hard to navigate the csv-files based on their name. Some names contain an author name and a number, others only construct acronyms, which then reappear in multiple file names with other constructs. A clear naming convention would be helpful, as well as meta-data (tags) to increase the discoverability of the project.

********** 

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Alexander Strobel

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

“Is it cognitive effort you measure? Comparing three task paradigms to the Need for Cognition scale”

PONE-D-23-10434R1

Dear Dr. Pfuhl,

Thank you for your careful response to comments from myself and reviewers. I am delighted move this very useful contribution to the study of cognitive effort into the publication process.

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Mark Fenton-O'Creevy, PhD

Academic Editor

PLOS ONE