Testing for reviewer anchoring in peer review: A randomized controlled trial

Ryan Liu; Steven Jecmen; Vincent Conitzer; Fei Fang; Nihar B. Shah

doi:10.1371/journal.pone.0301111

Peer Review History

Original SubmissionAugust 5, 2023
16 Oct 2023 Decision Letter - Stephan Leitner, Editor PONE-D-23-21814Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled TrialPLOS ONE Dear Dr. Liu, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Nov 30 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Stephan Leitner Academic Editor PLOS ONE Journal requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Please update your submission to use the PLOS LaTeX template. The template and more information on our requirements for LaTeX submissions can be found at http://journals.plos.org/plosone/s/latex. 3. Thank you for stating the following in the Acknowledgments Section of your manuscript: “This work was supported in part by NSF CAREER Award 1942124 and NSF 2200410.” We note that you have provided funding information that is currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: “NS received the National Science Foundation CAREER Award 1942124 (https://www.nsf.gov/). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. NS and FF received the National Science Foundation Communications and Information Foundations 2200410 (https://new.nsf.gov/funding/opportunities/ccf-communications-information-foundations-cif). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.” Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 4. We note that Figure 2 in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright. We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission: a. You may seek permission from the original copyright holder of Figure 2 to publish the content specifically under the CC BY 4.0 license. We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text: “I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.” Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission. In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].” b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only. 5. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. Additional Editor Comments: Dear authors, We have received two reviews for your paper from expert reviewers. While one is very positive, the other is slightly more critical. I would like to offer you the chance to address these issues in a revision, particularly by extending the discussion of results and limitations as suggested by the more critical reviewer. Regards, Stephan Leitner [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Partly ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: Thanks for this nicely designed and conducted experiment. The paper is innovative and the mild deception you use tolerable. I like the paper and results and thus support publication. Here just a few minor points you may change in the paper: - page 1, second line from bottom: replace "change" with "adapt" - page 11, line under Section 4 - delte this unnecessary sentence "In this section..." Reviewer #2: In this paper, the authors investigate an intriguing phenomenon in peer reviews, known as "Anchoring", where reviewers simply do not adjust their scores as much as they should. Through a randomized controlled trial, the paper studies the anchoring effect in peer reviews. The experiment designed by the authors involved 108 researchers with varied research backgrounds. The main findings of the authors include: 1) paper quality was perceived differently between the two groups; 2) no evidence of anchoring bias was found. The paper is meticulously designed, addressing some challenges, which showcases the substantial effort put forth by the authors. The writing is clear, the results are presented succinctly, and the overall presentation is good. Additionally, the authors have engaged in a degree of discussion regarding the limitations of their method. Overall, the article makes a certain contribution to anchoring in peer review. Pros: 1. The research questions are interesting and novel. 2. The experiments are well designed. The experiment design of the article is commendable and incurred certain costs, including the recruitment of trial participants and data collection, etc. 3. The writing of the paper is very good, including sections like the abstract, introduction, and conclusion. The authors have highlighted their main conclusions and also discussed other findings like the involvement of junior participants. Cons: 1. The experimental setup of the article has certain limitations, such as the construction of only one fake paper. 2. The data provided by the authors is somewhat scant, although they have discussed the limitations. 3. The gap between the authors' experiments and the real world is considerable, making it challenging to assure the validity of the findings on real-world data. My primary concerns regarding the article are as follows: As mentioned in the paper, the experimental design is overly idealized and deviates significantly from real-world scenarios. Firstly, in real-world situations, especially in computer conference papers, a reviewer typically reviews multiple papers, and a paper receives reviews from multiple reviewers. This introduces elements of peer effect and peer pressure, which are considered crucial factors [1]. Furthermore, the revision of scores by reviewers may also depend on the authors' rebuttal skills or rebuttal politeness [2,3,4]. These aspects are not reflected in the experiments conducted in the paper. While considering these factors is challenging, if the authors claim to study the anchoring phenomenon in peer reviews, then these factors cannot be overlooked. Other minor suggestions: Please revise some arxiv paper reference into their proceedings versions. References: [1] Gao, Yang, et al. "Does My Rebuttal Matter? Insights from a Major NLP Conference." Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019. [2] Huang, Junjie, et al. "What makes a successful rebuttal in computer science conferences?: A perspective on social interaction." Journal of Informetrics 17.3 (2023): 101427. [3] Rogers, Anna, and Isabelle Augenstein. "What Can We Do to Improve Peer Review in NLP?." Findings of the Association for Computational Linguistics: EMNLP 2020. 2020. [4] Bharti, Prabhat Kumar, et al. "PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews." Language Resources and Evaluation (2023): 1-23. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0301111.r001
Revision 1
2 Mar 2024 Author Response Reviewer #1: Thank you for your careful review of our manuscript. We have made changes to the paper according to your following suggestions: - page 1, second line from bottom: replace "change" with "adapt" - page 11, line under Section 4 - delte this unnecessary sentence "In this section..." You can find these changes in the “track changes” version of the revision. Reviewer #2: Thank you for your detailed comments and suggestions for our manuscript. In particular, we appreciate your careful consideration the various facets of our experiment design. We would like to briefly discuss your concerns regarding our paper: 1. The experimental setup of the article has certain limitations, such as the construction of only one fake paper. Response: We agree with the reviewer that this is indeed a limitation to our work. We have added a paragraph in the discussion specifically discussing this limitation. We reproduce the paragraph here for the reviewer's convenience: “Another limitation of our work is that we run our experiment with only one paper, which could lead to our findings to be less generalizable. There is precedence of research involving reviewers reviewing fake papers, and in each of these only 1 to 3 papers are constructed [1–5]. Due to the high sample size determined from the power analysis (see S2 Appendix) and the limited pool of eligible participants (see S3 Appendix), we chose to have one paper to reduce the sample size needed in order to test for statistical significance, as having multiple papers would require an additional random effect to be modeled. Future work may also include papers from multiple domains to bolster the generalizability of the study.” 2. The data provided by the authors is somewhat scant, although they have discussed the limitations. Response: We provide the full anonymized numerical responses and participant institution and year in the data folder of https://github.com/theryanl/ReviewerAnchoring. Our IRB approval and the consent of participants do not permit us to include participant comments in the dataset. This was due to potential concerns of anonymity breach (e.g., “Since I work in the neighboring field of xxx, …”). The participant pool, mainly CS or CS-related field PhDs, as well as their institution and year are provided. If additional information regarding their area of study can be inferred from their responses, this may cause certain participants’ identities to be identifiable. 3. The gap between the authors' experiments and the real world is considerable (e.g., peer effect, peer pressure, rebuttal skills, rebuttal politeness), making it challenging to assure the validity of the findings on real-world data. While considering these factors is challenging, if the authors claim to study the anchoring phenomenon in peer reviews, then these factors cannot be overlooked. Response: As you mention, there are many other peer effects in the process of rebuttals. The design of this experiment deliberately avoids adding these additional effects in order to isolate the effect we wish to understand – anchoring in its base form (as defined by [6]) without social or author effects. We believe this base form is important to analyze, as an existence of anchoring in this setting would mean that the re-reviewing paradigm itself causes anchoring bias to happen, meaning that altering social pressures and other effects would be insufficient to avoid bias. This setting is also consistent with anchoring bias studies in psychology which did not contain external social pressures or author skills or politeness [6]. In this case, adding other peer effects or author skills and politeness can confound any observations that we make. These factors are orthogonal to anchoring, and our objective in the experiment design is to single out anchoring and avoid these other confounders. This is actually a flaw with the study by Gao et al. you cited – the paper claims "peer pressure" as a reason for their observation, but their study does not isolate such effects and hence the claim can be confounded by various factors such as program chair instructions to arrive at a consensus. Our study, in contrast, asks a specific question (of anchoring), and the experiment design deliberately and carefully avoids other such confounders in order to answer this specific question. By coming to a conclusion on just anchoring, this also allows us to better evaluate the existence of other effects in future studies, and possibly help inform the design of future peer review paradigms. 4. Please revise some arxiv paper reference into their proceedings versions. As you suggested, we have made changes to the references, replacing arxiv paper references to their proceedings versions. They should be up-to-date with Google Scholar as of 2/9/2024. [1] W. G. Baxt et al. “Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance”. In: Annals of emergency medicine (1998). [2] G. B. Emerson et al. “Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial”. In: Archives of internal medicine (2010). [3] F. Godlee, C. R. Gale, and C. N. Martyn. “Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial”. In: JAMA (1998). [4] S. Schroter et al. “Effects of training on quality of peer review: randomised controlled trial”. In: BMJ (2004). [5] S. Schroter et al. “What errors do peer reviewers detect, and does training improve their ability to detect them?” In: Journal of the Royal Society of Medicine (2008). [6] A. Tversky and D. Kahneman. “Judgment under uncertainty: Heuristics and biases.” In: Science (1974). Attachments Attachment Submitted filename: Reviewer Letter for Anchoring Submission - Google Docs.pdf https://doi.org/10.1371/journal.pone.0301111.r002
12 Mar 2024 Decision Letter - Stephan Leitner, Editor Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial PONE-D-23-21814R1 Dear Dr. Liu, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at http://www.editorialmanager.com/pone/ and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Stephan Leitner Academic Editor PLOS ONE Additional Editor Comments (optional): We have now received two reviews from the experts who evaluated your manuscript in the previous round. I am very pleased to inform you that, based on the reviewers' assessments and my own evaluation of your manuscript, we can now accept it for publication. Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed Reviewer #2: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: (No Response) Reviewer #2: Thanks to the author for the reply, although I still think that peer pressure must be considered when analyzing the anchoring effect of peer review (peer review opinions are obviously given by multiple people together, and few papers are reviewed by only one reviewer). The reference [6] is not the definition of anchoring effect in peer review. However, I agree that the author used randomized experiments to verify the anchoring effect. While PLOS ONE does not attempt to use the peer review process to determine whether or not an article reaches the level of 'importance' required by a given journal, PLOS ONE uses peer review to determine whether a paper is technically rigorous and meets the scientific and ethical standards for inclusion in the published scientific record. So I am OK with the current revisions. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No ******** https://doi.org/10.1371/journal.pone.0301111.r003
Formally Accepted
17 Jul 2024 Acceptance Letter - Stephan Leitner, Editor PONE-D-23-21814R1 PLOS ONE Dear Dr. Shah, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Stephan Leitner Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0301111.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .