Overcoming the threat of anti-bias interventions: Combining self-report and psychophysiological measures to capture the process of change

Félice van Nunspeet; Esmee M. Veenstra; Beatriz Monteiro Graça Casquinho; Naomi Ellemers; Daan Scheepers; Miriam I. Wickham; Elena A. M. Bacchini; Jojanneke van der Toorn; on behalf of The Organizational Behaviour Group

doi:10.1371/journal.pone.0314813

Peer Review History

Original SubmissionJanuary 25, 2024
14 Aug 2024 Decision Letter - katsuya oi, Editor PONE-D-24-01387Overcoming the threat of anti-bias interventions: Combining self-report and psychophysiological measures to capture the process of change. PLOS ONE Dear Dr. van Nunspeet, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. ============================== My apologies for the delay in reaching a decision. I have carefully read the manuscript, along with the review we received, and I found both to be highly insightful. I agree with the reviewer’s constructive feedback and encourage the authors to address their concerns thoroughly. In my assessment, I found the study to be scientifically sound and comprehensive. While we do not evaluate submissions based on their significance alone, I would like to highlight that this paper is both innovative and important. It challenges the conventional assumptions around anti-bias training by offering a fresh perspective, emphasizing the significance of the delivery method and the psycho-physiological reactions of participants. Without careful attention to these aspects, such training could result in underwhelming or even counterproductive outcomes. I am pleased that the authors have chosen PLOS ONE as the venue for this valuable work. ============================== Please submit your revised manuscript by Sep 28 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, katsuya oi, PhD Academic Editor PLOS ONE Journal requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. When completing the data availability statement of the submission form, you indicated that you will make your data available on acceptance. We strongly recommend all authors decide on a data sharing plan before acceptance, as the process can be lengthy and hold up publication timelines. Please note that, though access restrictions are acceptable now, your entire data will need to be made freely accessible if your manuscript is accepted for publication. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If you are unable to adhere to our open data policy, please kindly revise your statement to explain your reasoning and we will seek the editor's input on an exemption. Please be assured that, once you have provided your new statement, the assessment of your exemption will not hold up the peer review process. 3. One of the noted authors is a group or consortium [The Organizational Behaviour Group]. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address. 4. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. Additional Editor Comments: My apologies for the delay in reaching a decision. I have carefully read the manuscript, along with the review we received. I agree with the reviewer’s constructive feedback and encourage the authors to address their concerns thoroughly. In my own assessment, I found the study to be scientifically sound and comprehensive. While we do not evaluate submissions based on their significance alone, I would like to highlight that this paper is both innovative and important. It challenges the conventional assumptions around anti-bias training by offering a fresh perspective, emphasizing the significance of the delivery method and the psycho-physiological reactions of participants. Without careful attention to these aspects, such training could result in underwhelming or even counterproductive outcomes. I am pleased that the authors have chosen PLOS ONE as the venue for this valuable work. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: This manuscript attempt to test for physiological responses to different interventions to minimize implicit biases. It uses an example of biases in evaluations of teaching and uses a 2x2 design to test how the implication that the respondents were biased and the framing of solutions changed these responses. The authors find some evidence that the type of intervention does seem to matter with the self implied evidence of bias increasing a threat response while a promotion focused frame increases the challenge response. This is a technically sound manuscript. The experiment is well done and the results are pretty clear. I would like to see a little more information about the evaluations that the respondents were accused of being biased about. Were they connected to the professor running the experiment? I can imagine that a respondent who was told that they were biased in their evaluations of a professor by that professor may see this as a threat in ways that they would not have if it was by a more neutral arbiter. I am guessing that the researchers did create some space between the person running the experiment and the target of the evaluations, about it should be clear. The choice of teaching evaluations as the target here is dicey. There is some evidence that interventions to reduce implicit biases can improve the scores for female faculty (Peterson et al 2019 “Mitigating gender bias in student evaluations of teaching” PLoS ONE), but a different widespread test found no effect of these interventions (Genetin et al 2021 “Mitigating implicit bias in student evaluations: A randomized intervention” Applied Economic Perspectives and Policy). Does the effectiveness of the intervention matter for the results? The authors have some self reports, but both of those other papers have experimental evidence about the effectiveness on actual evaluations. If, for instance, the Generin paper is correct and the interventions do not actually work, then some kind of negative reaction seems appropriate from the respondents. In contrast, if the intervention would have actually reduced the implicit biases, the effects may have been different. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0314813.r001
Revision 1
30 Oct 2024 Author Response Dear Dr. Katsuya Oi, dear Academic Editor of PLOS ONE, Enclosed we submit the revised version of our manuscript (Manuscript ID PONE-D-24-01387 “Overcoming the threat of anti-bias interventions: Combining self-report and psychophysiological measures to capture the process of change.”) to be considered for publication in PLOS ONE. We want to thank you for handling our submission to PLOS ONE, and for the opportunity to revise and resubmit our manuscript. We are very thankful for the positive evaluation and constructive comments. We have noted each comment in response to our initial manuscript, and at your invitation to prepare a revision, we have addressed all the issues that were raised. We feel that we were able to successfully deal with the outstanding questions and requirements. Below, we indicate how we addressed these points one by one. Reviewer’s comments: A) The Reviewer indicated they wanted “to see a little more information about the evaluations that the respondents were accused of being biased about.” The Reviewer wondered whether participants were connected to the professor who run the experiment, as it could be expected that “a respondent who was told that they were biased in their evaluations of a professor by that professor may see this as a threat in ways that they would not have if it was by a more neutral arbiter.” Consistent with the Reviewer “guessing that the researchers did create some space between the person running the experiment and the target of the evaluations”, we ensured this in the experiment. Moreover, in line with the Reviewer’s remark that we should be clearer about this, we included the following information in the method section of our revised manuscript, on page 12: “Importantly, none of the researchers (i.e., the experimenters and the university professor who presented in the webinar), was included in participants’ teaching evaluations as these evaluations concerned courses outside of the researchers’ teaching responsibilities. Independence between the researchers and the research participants was thus assured.” Additionally, we expanded our explanation of how the information about the self-implying evidence was presented (i.e., not in the webinar, or by the professor). On page 12, we added the last sentence below: “Participants in the self-implied condition were also presented with a brief summary indicating they and their fellow students had shown a gender bias in their teacher evaluations – allegedly based on the online questionnaire they had filled out before they had come to the lab. This feedback was not presented within the webinar (i.e., by the professor), but rather provided as a separate (additional) section of the experiment.” B) The second, and last, comment of the Reviewer concerns the fact that whereas we focused on teaching evaluations as the target of our gender bias intervention, we did not test whether our intervention affected such teaching evaluations. Furthermore, the Reviewer poses that the potential results of such a test would have implications for the interpretations of our findings. More specifically, the Reviewer mentions that “there is some evidence that interventions to reduce implicit biases can improve the scores for female faculty (Peterson et al 2019 “Mitigating gender bias in student evaluations of teaching” PLoS ONE), but a different widespread test found no effect of these interventions (Genetin et al 2021 “Mitigating implicit bias in student evaluations: A randomized intervention” Applied Economic Perspectives and Policy).” The Reviewer then wonders whether “the effectiveness of the intervention matter for the results?” We thank the Reviewer for alerting us to this matter and for the references to the different findings regarding the effectiveness of gender bias interventions in teaching evaluations. Although we consider these questions as highly relevant in a broader sense, with regard to the more concrete aims of the current contribution, we note that we intentionally focused on a somewhat different question, namely the processes underlying the potential (in)effectiveness of such interventions, as we describe on pages 2 and 3 of the original (as well as the revised) manuscript (“More specifically, we measure how psychophysiological responses unfold during the intervention to capture the process of change, and we combine this with self-reports revealing how participants explicitly reflect on their experience. This approach increases our understanding of psychological mechanisms that may either undermine or enhance the likelihood that people engage with the intervention and adapt their behaviour.”). In other words, the aim of our study was not to test the effectiveness of (the parts of) the intervention that was presented in our experimental design (i.e., the confrontation with self-implying vs. not self-implying evidence of bias, and the reflection on potential solutions in terms of ideals vs. obligations) on the elimination of biased judgments in subsequent course evaluations by these students. Instead, our measure of ‘willingness to take action’ to capture participants’ intentions towards reduction of bias in teaching assessments was framed more broadly. Notably, such action might also include other types of initiatives, such as taking action to change current methods to evaluate teaching quality, due to the realization that these are subject to bias. For future research, we see the potential value of additional measures to examine the impact of our intervention more specifically. We have therefore included this reflection in the revised version of our manuscript (including the references mentioned by the Reviewer), in the paragraph on future research directions. Specifically, on page 34, we now write: “Relatedly, decided not to include a post-measure of participants’ teaching evaluations to test the effectiveness of our intervention, as has been done in prior work on gender bias in teaching evaluations [86, 87]. This was outside the scope of the current research, as we intentionally focused on investigating the underlying psychological mechanisms that may contribute to the varying outcomes of such interventions (as well as the mixed results of anti-bias interventions in various other contexts [e.g., 14, 15, 88]. Further, we note that the impact of our intervention could be broader than that (e.g., by motivating students to take action against survey methods to evaluate teaching quality due to the realization that these are subject to bias). Nevertheless, future work could include a post-measurement of bias to more specifically assess whether and how these types of awareness raising interventions help to eliminate bias.” In addition to the comments made by the Reviewer, we also addressed the additional requirements you listed to prepare our manuscript for publication in PLOS ONE. We detail the changes that were made below. 1) We carefully checked the PLOS ONE style templates and have ensured that our manuscript meets PLOS ONE's style requirements, including those for file naming. We also made sure to refer to these files using the correct names in the revised manuscript. Please note that this also implies that we have newly uploaded our Figures and Supporting Information documents, to adhere to the correct resolution requirements and file naming respectively. In terms of content, these files did not change. 2) Consistent with our prior indication that we will make our data available on acceptance, we still aim to adhere to your open data policy. We have prepared a folder to be uploaded in OSF upon publication, that contains 1) the dataset and syntax for all the analyses reported in the main manuscript, and 2) the supporting information documents as well as the syntax for the analyses reported here (and which can be applied to the dataset mentioned under 1). 3) In line with your request about our research group as one of the authors, as well as information in the formatting requirements (i.e., "If there is a consortium or group author on your manuscript, please provide a note that describes where the full membership list is available for the readers."), we have listed all the contributors to the manuscript from The Organizational Behavior Group in the acknowledgments section of our manuscript. In addition, we added a note that specifies where the full membership list is available for the readers, and we have added the lead author for this group along with a contact email address. This can be found in the Acknowledgments section on page 36 of the revised manuscript. We hope these changes meet your request, but please inform us in case additional adjustments need to be made with regard to this issue. 4) We have, once again, reviewed our reference list to ensure that it is complete and correct. Compared to the initial manuscript, we did add three references in our revision, but we have made sure to correctly renumber all references as well as the corresponding reference list. Finally, we noticed that we initially included a URL when mentioning a DOI. Consistent with the criteria and guidelines for formatting references, we now provide DOI’s while adhering to the format with both the label and full DOI included at the end of the reference (and not providing the URL). Once again, we want to thank you and the Reviewer for the positive and constructive comments. We did our best to incorporate the suggestions and requests in the revised manuscript and we feel that the suggestions have further improved the quality of our paper. We look forward to your reply. Sincerely, also on behalf of Esmee Veenstra (shared first author), and our co-authors: Beatriz Casquinho, Naomi Ellemers, Daan Scheepers, Miriam Wickham, Elena Bacchini, and Jojanneke van der Toorn, Félice van Nunspeet Corresponding author Attachments Attachment Submitted filename: Response to Reviewers.docx https://doi.org/10.1371/journal.pone.0314813.r002
18 Nov 2024 Decision Letter - katsuya oi, Editor Overcoming the threat of anti-bias interventions: Combining self-report and psychophysiological measures to capture the process of change. PONE-D-24-01387R1 Dear Dr. van Nunspeet, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, katsuya oi, PhD Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The authors have addressed all of my comments and I think the paper should be accepted for publication. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No ******** https://doi.org/10.1371/journal.pone.0314813.r003
Formally Accepted
2 Jan 2025 Acceptance Letter - katsuya oi, Editor PONE-D-24-01387R1 PLOS ONE Dear Dr. van Nunspeet, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. katsuya oi Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0314813.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .