PONE-D-21-14391

Does Eye-tracking have an effect on economic behavior?

PLOS ONE

Dear Dr. Palma,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

As you will see below I have two very different reports. The first arises from an economist who did not like the paper very much and the second from a psychologist who liked a lot. Since I am economist myself I can understand (and agree with many of them) the critical view of the first report. 

Personally I feel that the paper is interesting. Yes, it is. But given that the potential audience of this piece of research in a journal like PLOS ONE might be diverse (including economists, game theorists, etc...) I would strongly suggest to revise the paper according to the suggestions provided by Referee #1. Honestly, I feel that making the paper for a wider audience is better.

Please submit your revised manuscript by Jul 09 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Pablo Brañas-Garza, PhD Economics

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.Thank you for including your ethics statement: 

"This RCT was registered in the American Economic Association Registry for randomized control trials under trial number AEARCTR-0004174. This study was approved by the IRB, protocol IRB2018-1602.".   

Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.

Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

3. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

4. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors' self-declared goal is to "investigate whether the use of an eye-tracking device induces changes in the economic behavior in incentivized laboratory experiments" (lines 25-26).

They report statistical findings from a laboratory experiment organized around 8 tasks that are well-known in the experimental-economics literature and two treatments (eye-tracking vs no eye-tracking). The results do not show statistically significant differences in behavior across treatments except for risk attitudes measured through the Holt-and-Laury task. The authors argue that those differences are due to difficulties experienced by subjects in the eye-tracking calibration process.

My main concern with the paper is its lack of focus.

I do not see why a "non-invasive" and practically imperceptible eye-tracking technique would cause any change in behavior. The authors motivate their research by stating that "external validity would be limited to situations in which 27 participants know they are being observed" (lines 27-28). But is it not always the case the experimental participants are being observed and that they are well aware of that? Having one's eye-movements observed is not the same as being "watched" by some observing eyes.

It is very much misleading to refer to "watched by eyes" effects detected by other researchers when subjects saw computer-generated eyespots (or an image of eyes) and acted in a more socially-desirable manner. Note that those eyespots (and eyes) were actually blind. The other main reference using eye-tracking glasses (and provocative swimsuit calendars) is also irrelevant here, not only because it used deception, but because it is not related to economic decision-making.

In spite of the above, let us suppose that eye-tracking might alter behavior. The related experiment should then control for the effect of calibration of the eye-tracking device (given that calibration is necessary for eye- tracking). We would need a baseline treatment with no calibration and no eye-tracking, a treatments with calibration and eye-tracking, and also one with calibration and no eye-tracking. The fourth combination of no calibration and eye-tracking is technically impossible. The paper should be refocused. Instead of putting the experimental tasks in the spotlight, it should pay more attention to the fine details of the eye-tracking process. It should not only report technical details of the device and explain the calibration process and whether all observations were included in the statistical analysis or not, but also control for all those details in the statistical analysis.

Right now, although the paper talks about eye-tracking effects, I find that the data and the statistical results are more about calibration effects.

My remaining concerns are related to the statistical analysis.

* When analyzing acceptance decisions from the ultimatum game, the authors should control for the size of the initial offer.

* Similarly, data were collected over several periods in the public-goods game and the double auction. Both game are known to produce time trends, so the authors should control for that.

* Please avoid the term "marginally significant" and refrain from interpreting "differences" with a p-value larger than 0.05.

* As a robustness test, check the OLS findings with a regression model that is more appropriate for count data (Holt-and-Laury task, Eckel-and-Grossman task).

Reviewer #2: The present work aims to explore whether the use of eye-tracking measures influences the outcome in eight different tasks frequently used in the field of experimental economics (e.g., through an increase in social desirability). From all the tasks, only the Holt and Laury risk assessment task shows differences when control and treatment (eye-tracking) conditions are compared. More specifically, they found that the individuals with a greater number of calibration attempts presented more risk aversion behavior. When removed, there were no differences between conditions. The authors concluded that the use of eye-tracking does not impact the quality of the data obtained in these tasks.

The manuscript is well-written and methodologically sound; besides, the results obtained support the use of a methodology with the potential to make significant contributions in experimental economics. While I recommend it for publication, I have few comments that I include below.

Lines 29 to 33. The authors employed tasks that have several levels of socially desirable choices. I wonder whether they examined differences (for example, in effect size) between tasks with no socially preferred choice and those that have.

Lines 48 to 69. Because earlier studies revealed a short-lived social-desirability effect associated with the use of eye-tracking devices and the authors reviewed such literature in the introduction, an analysis including, for example, two halves of each task could share some light to such effect in the domain of experimental economics.

Lines 610 to 612. Differences in the number of calibration attempts across tasks should be included, even if they are not expected—also, an explanation of why calibration attempts affect the HL but not the others.

Line 626. Could they explain a bit further how the estimation of the emotional states works?

Minor:

Line 71-> Name the Hawthorne effect earlier when including the impact of being observed in other studies so that a reader that is not familiar with the term can fully understand its relevance (for example, between lines 39 to 45).

Typos

Line 284 -> a “T” in theoretically

Line 287-> Frey and Bohnet find(without the final “s”)

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Does Eye-tracking have an effect on economic behavior?

PONE-D-21-14391R1

Dear Dr. Palma,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Pablo Brañas-Garza, PhD Economics

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I have no further comments at this point, and I do now wish to insist on the points raised earlier.

Reviewer #2: Thank you for addressing the comments.

Just a couple of very minor things:

Page 2, line 57-60: The clarification in parenthesis doesn't seem necessary.

Please, check typos and consistency across references (for example, keep all journals' names abbreviated)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No