Did people really drink bleach to prevent COVID-19? A guide for protecting survey data against problematic respondents

Leib Litman; Zohn Rosen; Rachel Hartman; Cheskie Rosenzweig; Sarah L. Weinberger-Litman; Aaron J. Moss; Jonathan Robinson

doi:10.1371/journal.pone.0287837

Peer Review History

Original SubmissionAugust 5, 2022
23 Nov 2022 Decision Letter - Xin Shen, Editor PONE-D-22-20912Did People Really Drink Bleach to Prevent COVID-19? A Tale of Problematic Respondents and A Guide for Measuring Rare Events in Survey DataPLOS ONE Dear Dr. Rachel Hartman, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Dec 16 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Xin Shen Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information. 3. Please note that the name of the ethics committee that approved your study seems to be misspelled in the manuscript (InteRreview instead of InteGreview). 4. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: No ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: I have read this interesting manuscript and it provides a great insight into a major issue with public health surveys. I was wondering why the author(s) opted to split the study into to and report them somewhat separately. The methods sections for each arm of the study could have been merged in a single section with clear descriptions of the different components. Similarly the results could have been presented in a composite table. A composite summary of the results could have better presented the results more than the narrative description. The use of the flow diagram (although not legible) is also a good way to represent how the problematic respondents were isolated in each arms of the study. Find additional comments in the text. Reviewer #2: I recommend publication. Researchers and journalists are far too quick to highlight literally incredible findings from surveys which make no sense and are just plain wrong. The authors' very plausible explanation is that there are wayward (problematic) respondents and once their responses are removed the results become realistic. The paper is, at the very least, a cautionary tale. I have a couple of suggestions for the revised version. The paper reads as two papers instead of one. The reader reaches a discussion section and then the second study is introduced. There is a second, much longer, discussion section later. Please integrate the two studies, so the paper reads as a conventional research article rather than unexpectedly, as two. The paper is too long. It could be a short, sharp interesting piece. I got bored by the second discussion section. The discussion in the second Discussion section on the numbers of calls to poison centers should be in the introduction as indicative evidence that COVID probably did not cause people to ingest disinfectants etc. Similarly, the discussion on the responses to the open-ended questions do not belong in the discussion but in the results section or in an appendix. It is an interesting topic so let's make an easy read. The estimate of 23% for problematic respondents is very high and over 30% for the second study is even higher. If representative, these estimates undermine the legitimacy of most estimates from survey research. I would be interested to know if the authors consider that these estimates are realistic for survey research, in general, or are far too high for reasons to do with their studies. Some commentary on the proportion of problematic respondents would be appreciated. The authors need to acknowledge that the issue of people ingesting household bleach was highly politicized, because of Donald Trump's injudicious and widely misinterpreted comments. I think that if a senior politician from the other side of politics made those comments, there wouldn't have been the media meltdown and maybe the CDC would not have written that report which although didn't mention Trump, was in response to the political climate and had an underlying political message. I realize that we live in high politicized times, but the issue is no longer on the political agenda. I think it is important that the political context of the issue should be acknowledged. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. Attachments Attachment Submitted filename: PONE-D-22-20912_reviewer_comments.pdf https://doi.org/10.1371/journal.pone.0287837.r001
Revision 1
16 Dec 2022 Author Response Miscellaneous: • Fixed all the headings • Fixed Table 1 • Fixed title page, changed the author order, and changed the title • Fixed integReview typo • Added more details about consent • Rewrote large portions of the paper for clarity and concision • We reviewed the reference list and did not see any that have been retracted. We added the following citations: o 52. Harrison KL, Zane T. Focus on science: Is there science behind that? Bleach therapy. Sci Autism Treat. 2017;14(1):18-24. o 54. Khadse PA, Murthy P. Accidental deaths from hand sanitizer consumption among persons with alcohol dependence during the COVID-19 lockdown in India: Analysis of media reports. Asian J Psychiatry. 2021;63:102794. doi:10.1016/j.ajp.2021.102794 o 55. Kochgaway L, Nair AG, Mitra A, Bhargava S, Singh M. COVID casualty: Bilateral blindness due to ingestion of spurious sanitizer. Oman J Ophthalmol. 2020;13(3):164-166. doi:10.4103/ojo.OJO_277_2020 o 63. Aronow PM, Kalla J, Orr L, Ternovski J. Evidence of Rising Rates of Inattentiveness on Lucid in 2020. Published online September 13, 2020. doi:10.31235/osf.io/8sbe4 Reviewer #1: I have read this interesting manuscript and it provides a great insight into a major issue with public health surveys. I was wondering why the author(s) opted to split the study into two and report them somewhat separately. The methods sections for each arm of the study could have been merged in a single section with clear descriptions of the different components. Similarly the results could have been presented in a composite table. A composite summary of the results could have better presented the results more than the narrative description. The use of the flow diagram (although not legible) is also a good way to represent how the problematic respondents were isolated in each arm of the study. Thanks for the suggestion (also echoed by reviewer 2)—we agree that combining the studies makes for a better read! We have done as you recommended, combining studies 1 and 2 and inserting a table with a composite summary (and comparison to the CDC study). Find additional comments in the text. These comments were super helpful, thank you! I believe we addressed all of them in the revised text. We briefly summarize the changes here: 1. Added a citation for “Despite the widespread use of survey research, self-report data has come under increasing scrutiny over the last ten years due to data quality concerns.” 2. Fixed typo on page 3 (extra word) 3. On page 6, you noted that we didn’t spell out the CDC. However, this was not the first mention (see page 3). 4. Fixed some of the tenses to past tense where appropriate 5. Fixed typo on page 8 6. Re-did the demographics table to more clearly convey the information 7. Rephrased the sentence on page 11 8. Relabeled the hypotheses as H1, H2, etc. 9. We added the word “moderate” to accompany the .44 correlation on page 14. Based on Cohen’s conventions, correlations between .3 and .5 are considered moderate. 10. We removed the rhetorical question on page 24. 11. We added a reference to “There have been several documented cases of blindness and death due to drinking hand sanitizer since the start of the Covid-19 pandemic” 12. We recreated Figure 3 (now Figure 2). Reviewer #2: I recommend publication. Researchers and journalists are far too quick to highlight literally incredible findings from surveys which make no sense and are just plain wrong. The authors' very plausible explanation is that there are wayward (problematic) respondents and once their responses are removed the results become realistic. The paper is, at the very least, a cautionary tale. I have a couple of suggestions for the revised version. The paper reads as two papers instead of one. The reader reaches a discussion section and then the second study is introduced. There is a second, much longer, discussion section later. Please integrate the two studies, so the paper reads as a conventional research article rather than unexpectedly, as two. Thanks for this great suggestion (also mentioned by Reviewer 1). We combined the two studies. The paper is too long. It could be a short, sharp interesting piece. I got bored by the second discussion section. We have shortened the paper considerably. The discussion in the second Discussion section on the numbers of calls to poison centers should be in the introduction as indicative evidence that COVID probably did not cause people to ingest disinfectants etc. We agree with this suggestion, and have moved the relevant paragraphs to the introduction. Similarly, the discussion on the responses to the open-ended questions do not belong in the discussion but in the results section or in an appendix. It is an interesting topic so let's make an easy read. Good point. We moved the discussion to the results section. The estimate of 23% for problematic respondents is very high and over 30% for the second study is even higher. If representative, these estimates undermine the legitimacy of most estimates from survey research. I would be interested to know if the authors consider that these estimates are realistic for survey research, in general, or are far too high for reasons to do with their studies. Some commentary on the proportion of problematic respondents would be appreciated. We commented on this in the introduction, but have also added a section in the discussion. Here are the two sections: Introduction: Problematic respondent bias is a ubiquitous problem that requires mitigation in any type of survey20,28,28, regardless of the modality or demographic population. Thus, the inclusion of rigorous methodology to support the validity of estimates drawn from survey data is critical 37. To this end, researchers have developed data validity screening instruments to combat problematic responses. These instruments can be added before the survey to prevent problematic respondents from participating28, or they can appear within the survey to identify problematic respondents to be excluded from the analytic sample15,29–34. Using such instruments has helped reveal that problematic respondents can drastically attenuate results, at times leading researchers to conclude that previously established findings lack validity17,35,36. This is especially important for studies with direct implications for public health and public policy13. One popular modality of collecting survey responses is via online opt-in panels, which constitute more that 80% of currently conducted public opinion polls25, and are increasingly used in public health, political science, and social and behavioral sciences28,38. A large literature on opt-in panels indicates that the percentage of problematic respondents on such panels is substantial6,14,25,28,39–44. Estimates of the magnitude of problematic respondent bias in online opt-in panel platforms vary between 4-7%25 and 30%28, although in some studies the magnitude of inattention has been as high as 50%14. Discussion: Across both samples, we revealed high levels of problematic responding (23% and 33% across Samples 1 and 2, respectively). These figures are in line with estimates of problematic responding in the literature, ranging from 7%25 to 50%14. Reports of problematic respondents on online surveys dramatically increased during the first few months of the Covid-19 pandemic. In the Spring of 2020, multiple teams who had collected over 100,000 survey responses on Lucid, reported that rates of problematic responding were between 30% and 84%63. This was the same platform from which the CDC had collected their data, and also covers the same time period. The authors need to acknowledge that the issue of people ingesting household bleach was highly politicized, because of Donald Trump's injudicious and widely misinterpreted comments. I think that if a senior politician from the other side of politics made those comments, there wouldn't have been the media meltdown and maybe the CDC would not have written that report which although didn't mention Trump, was in response to the political climate and had an underlying political message. I realize that we live in high politicized times, but the issue is no longer on the political agenda. I think it is important that the political context of the issue should be acknowledged. We agree that the toxic political environment likely contributed to the existence of the CDC report. We think there is a cautionary tale to be told, for both scientists and journalists, about the eagerness to accept and promote findings that align with one’s political ideology. We hope that readers of our paper will take note of these implications. However, we’re reluctant to comment on the political aspect of this issue within the paper, because to some readers this might make us appear politically biased or not impartial, and it could lower our credibility. For the paper to remain as objective as possible, we’re happy for people to draw their own conclusion. We think after our paper is published, there’s room for other papers (e.g., op eds) to be published, that expand on the role of politics. We’d be happy to collaborate with anyone interested in pursuing that angle in alternative avenues, but this paper doesn’t seem like the best place for it. Attachments Attachment Submitted filename: Response to Reviewers.docx https://doi.org/10.1371/journal.pone.0287837.r002
14 Jun 2023 Decision Letter - Xin Shen, Editor Did people really drink bleach to prevent COVID-19? A guide for protecting survey data against problematic respondents. PONE-D-22-20912R1 Dear Dr. Hartman, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Xin Shen Academic Editor PLOS ONE Additional Editor Comments (optional): Dear authors, In order to expedite the publication of the qualified manuscript, I checked your response to the one of last reviewers who had no response. I noticed your response to the reviewer and you have addressed all concerns. Please note that I have acted as a reviewer for this manuscript, and you will find my comments below, under Reviewer #3. For the expedition of publication, please follow our journal style and double-check the format. Best wishes, Xin Shen Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: All comments have been addressed Reviewer #3: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes Reviewer #3: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: Yes Reviewer #3: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes Reviewer #3: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes Reviewer #3: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: Thet have addressed my concerns and I am happy to recommend publication. The authors should check and recheck the text. Reviewer #3: (No Response) ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: Yes: Gary N. Marks Reviewer #3: No ******** https://doi.org/10.1371/journal.pone.0287837.r003

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .