Dear Dr Bakker,

Thank you for submitting your manuscript entitled "Ensuring the quality and specificity of preregistrations" for consideration as a Meta-Research Article by PLOS Biology.

Your manuscript has now been evaluated by the PLOS Biology editorial staff as well as by an Academic Editor with relevant expertise. I apologize for the delay while we discussed the information provided. I am writing to now let you know that we would like to consider this manuscript further. While we will take the previous reviews from Advances in Methods and Practices in Psychological Science into consideration, unfortunately since two of the reviewers do not reveal their identity we are not able to fully utilize those reviews and will need to get 1-2 reviews of our own. If you are happy with this, please proceed with the steps below.

Before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Please re-submit your manuscript within two working days, i.e. by Mar 04 2020 11:59PM.

Login to Editorial Manager here: https://www.editorialmanager.com/pbiology

During resubmission, you will be invited to opt-in to posting your pre-review manuscript as a bioRxiv preprint. Visit http://journals.plos.org/plosbiology/s/preprints for full details. If you consent to posting your current manuscript as a preprint, please upload a single Preprint PDF when you re-submit.

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Feel free to email us at plosbiology@plos.org if you have any queries relating to your submission.

Kind regards,

Hashi Wijayatilake, PhD,

Managing Editor

PLOS Biology

Dear Dr Bakker,

Thank you very much for submitting your manuscript "Ensuring the quality and specificity of preregistrations" for consideration as a Meta-Research Article at PLOS Biology. Your manuscript has been evaluated by the PLOS Biology editors, an Academic Editor with relevant expertise, and by two PLOS Biology reviewers. Firstly, let me sincerely apologize for the unusual and lengthy delay getting you this decision. As I had explained in an earlier email, we had a reviewer signed up who, despite assurances, never delivered his/her comments. We then had to get new reviewers signed on at that late stage. We had two reviewers signed on with the aim of getting 1-2 additional reviewers, as mentioned, and luckily both have now delivered on schedule. The good news is that both are quite positive about the study and find the research question interesting. Considering these reviews, along with the reviews from Advances in Methods and Practices in Psychological Science, especially Reviewer 3 (who had signed his review), we will be happy to pursue the manuscript at PLOS Biology. However, the reviewers do raise important points that will need to be thoroughly addressed in a revision. Primarily, Reviewer 2 has concerns about the analysis and power calculations, which will need to be fully and satisfactorily addressed. Both reviewers also point out that, given the broad potential interest in this article across fields, it needs to be made much more understandable. The editorial team and Academic Editor all agree that a major rewrite is needed for this to be suitable for our broad readership. Overall, while we will not be able to accept the current version of the manuscript, we would welcome re-submission of a revised version that takes into account the reviewers' comments.

Please note that we cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent for further evaluation by the reviewers.

We expect to receive your revised manuscript within 2 months.

Please email us (plosbiology@plos.org) if you have any questions or concerns, or would like to request an extension. At this stage, your manuscript remains formally under active consideration at our journal; please notify us by email if you do not intend to submit a revision so that we may end consideration of the manuscript at PLOS Biology.

**IMPORTANT - SUBMITTING YOUR REVISION**

Your revisions should address the specific points made by each reviewer. Please submit the following files along with your revised manuscript:

1. A 'Response to Reviewers' file - this should detail your responses to the editorial requests, present a point-by-point response to all of the reviewers' comments, and indicate the changes made to the manuscript.

*NOTE: In your point by point response to the reviewers, please provide the full context of each review. Do not selectively quote paragraphs or sentences to reply to. The entire set of reviewer comments should be present in full and each specific point should be responded to individually, point by point.

You should also cite any additional relevant literature that has been published since the original submission and mention any additional citations in your response.

2. In addition to a clean copy of the manuscript, please also upload a 'track-changes' version of your manuscript that specifies the edits made. This should be uploaded as a "Related" file type.

*Re-submission Checklist*

When you are ready to resubmit your revised manuscript, please refer to this re-submission checklist: https://plos.io/Biology_Checklist

To submit a revised version of your manuscript, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' where you will find your submission record.

Please make sure to read the following important policies and guidelines while preparing your revision:

*Published Peer Review*

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. Please see here for more details:

https://blogs.plos.org/plos/2019/05/plos-journals-now-open-for-published-peer-review/

*PLOS Data Policy*

Please note that as a condition of publication PLOS' data policy (http://journals.plos.org/plosbiology/s/data-availability) requires that you make available all data used to draw the conclusions arrived at in your manuscript. If you have not already done so, you must include any data used in your manuscript either in appropriate repositories, within the body of the manuscript, or as supporting information (N.B. this includes any numerical values that were used to generate graphs, histograms etc.). For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5

*Blot and Gel Data Policy*

We require the original, uncropped and minimally adjusted images supporting all blot and gel results reported in an article's figures or Supporting Information files. We will require these files before a manuscript can be accepted so please prepare them now, if you have not already uploaded them. Please carefully read our guidelines for how to prepare and upload this data: https://journals.plos.org/plosbiology/s/figures#loc-blot-and-gel-reporting-requirements

*Protocols deposition*

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosbiology/s/submission-guidelines#loc-materials-and-methods

Thank you again for your submission to our journal. We hope that our editorial process has been constructive thus far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Hashi Wijayatilake, PhD,

Managing Editor

PLOS Biology

*****************************************************

REVIEWS:

Reviewer #1 (Matthew Page):

This is a really interesting study addressing an important question: does a more detailed template for registering study plans lead to a more transparent plan? The methods are rigorous and documented in sufficient detail to allow others to replicate the study. The authors did a great job responding to the reviewer comments at the other journal, but I have a few other comments that I hope will improve the manuscript.

General comment: The term preregistration is used throughout. I am from medicine where we have used the terms prospective registration or retrospective registration for decades, so I find the "pre" added to this term confusing. The use of "pre" implies some act done before registering a study, but I'm not sure what that act is. A clarification about terminology would be great, particularly for health and medical researchers who would also benefit from reading this work.

Abstract: The authors claim that "Opportunistic use of 'researcher degrees of freedom' aimed at obtaining statistical significance increases the likelihood of obtaining false positive results and overestimated effect sizes, and lowers the replicability of published results." I agree with the first part of the sentence, but not with the second (i.e. "…lowers the replicability of published results"). Incomplete reporting and not sharing study materials lowers the ability for others to replicate a study, not use of 'researcher degrees of freedom'. But perhaps I am interpreting "replicability" differently to the authors (given the definition varies so much!)

Abstract: The authors claim that "Results of comparing random samples of 53 preregistrations from each format indicate that neither format restricted all researcher degrees of freedom and

the Prereg Challenge format performed better on restricting degrees of freedom than the "Standard" format." The latter half of the sentence (re the Prereg Challenge format performing better) is not very clear (as a reader am I to interpret that this format was better for ALL researcher degrees of freedom, and how much better?). It also appears inconsistent with how you interpret the results in the Discussion.

Abstract: The authors claim that "We also found a very low concordance rate among coders about the number of hypotheses (about 15%), suggesting that preregistration is difficult and an acquired skill…" Sorry but I don't follow the logic here. Low concordance among raters appraising preregistrations implies that it is difficult for readers to understand the content of preregistrations or that the coders needed more training, not that that the act of preregistration itself is difficult, in my view. Perhaps clarification on this point would be useful.

Introduction: The authors state that "Specification of the design and the analysis plan before observing the outcomes prevents the outcomes from affecting design and analysis decisions." However, no empirical evidence to support this claim is provided, despite there being a vast literature on reporting biases in medicine that has examined this issue directly that should be acknowledged. Also the paragraph starting, "The potential value of preregistration has been known for a long time…" is also rather limited in my view as it makes no mention of the vast literature on the potential value and observed limitations of registration of clinical trials in medicine. Why not build on that past literature to frame your arguments in the current study? For example, we have known for many years via meta-research studies in medicine that basic registration isn't enough to prevent researcher degrees of freedom, and that detailed protocols and statistical analysis plans are essential (for which a reporting guideline by Gamble in JAMA exists). I realise you may want to focus only on psychology, but I just think that reflecting on this literature could make the rationale for the study stronger.

FOLLOW-UP COMMENT: Oh, I see you refer to this evidence in the Discussion. Hmm, seems a bit odd not to allude to this evidence in the Introduction.

Methods - Variables of interest: I wonder if an alternative to "Restriction Score" could be used. It has a pretty negative connotation I think, with a possible interpretation of this study being that "preregistration restricts researchers!". Why not something more positive like a "Transparency Score", thus leading to the more positive conclusion that "preregistration improves transparency!".

Results: Minor quibble - there are too many abbreviations in the Results section (e.g. DF, PCR, SPR, DV, IV) which makes it hard to follow. A case in point is the following sentence, which I find dizzying to read: "Higher scores were obtained by PCR than by SPR on DFs that pertained to the operationalization of the variables in the study: D1 ('multiple manipulated IVs', D = 0.22), D3 and A5 ('multiple measures DV' and 'select DV measure', D = 0.19), and D5 and A12 ('additional IVs exclusion' and 'in/exclusion criteria', D = 0.24).". I see no problem with spelling everything out in full. Also, for a medical researcher like me, PCR means something else entirely!

Table 2. I'm sure this is reported elsewhere but from just reading the table it's unclear to me what the range of possible scores is in the Mean (SD) columns (e.g. is a mean of 1.98 large or small?). A footnote would help

--

Reviewer #2 (Ulf Toelch, QUEST Center for Transforming Biomedical Research):

The study by Vanderkamp et al. addresses an important and timely subject namely how current practices in preregistration influence the depth and detail of preregistrations. I also appreciate the openness and transparency to publish the previous reviewer comments. The material on the OSF repository seems complete, I did however not review the code. The manuscript is not an easy read and could still need some editing towards readability. Particularly some procedures are only clear if you read the R code or the supplementary coding scheme. Beyond this I have some major concerns particularly about the analysis that I would like to see clarified.

1. This is reflecting on a point one of the previous reviewers as to the transition from preregistration

to manuscript. A recent preprint at least hints at the possibility that adherence to protocols may not be optimal (10.31234/osf.io/d8wex). It is thus even more important to have high standards for preregistration. It will be harder to adhere to a preregistration with full analysis plan than to a preregistration stating only a hypothesis. It is, however, a lot more work for reviewers to connect a preregistration to a manuscript if the preregistration is long and exhaustive. I feel that the last part of the discussion puts little emphasis on the fact that a. writing good preregistrations is hard b. reviewing them puts a considerable additional burden on reviewers. A more balanced view on preregistration would be warranted and cases where preregistration may not be applicable could be stated.

2. The list of DFs by Wicherts et al. is at least partly targeted specifically at psychology. As PLOS Biology has a broad readership it will be interesting from which fields the preregistrations stem. My guess is that preregistrations will be mainly from psychology. This could be another limitations as e.g. animal experiments may need a different focus for preregistration (https://www.animalstudyregistry.org)

3. I do not find the analysis very compelling.

A. The power analysis is done on a Cohens d of .5 You state that as you do not have any prior knowledge about the ES. One than must assume that you think that .5 will be a meaningful effect size. This corresponds to a Cliffs d of .33. This means that only effect sizes equal/larger should be considered as meaningful difference (not only statistically significance). How will smaller effect sizes make a difference between the two preregistration schemes? And if they do why did you not power for them? See my summary point below also.

B. It is not clear whether power calculation was conducted on two or one sided test (perhaps I did not read this properly). Your main results is one sided (summary restriction scores) the single DFs are two sided. This needs clarification.

C. Your main result (summary restriction scores) is calculated on the mean of median scores. This is not reflected in the power analysis. Some DFs can only have values of 0, 2, and 3, some can have values of 0, 1, 2, 3. As this is your main result your choice to first calculate the median and then the mean needs more motivation. Perhaps you could conduct a non parametric bootstrap to confirm your result. It is also not clear to me what the calculated effect size here means. What would be a viable interpretation?

D. Table 2. Your data has a clear hierarchical structure (different categories) with a lot of depended data points (within study). This is not reflected in your simple multiple test two group comparison scheme. The preregistered analysis is simplistic and neglects a lot of these effects. What are correlations within study? Will a preregistration that is underreporting in one area also do this in another (as additional material)?

Additionally you state: "We then conducted exploratory follow-up analyses to investigate on which DFs the two preregistration types differed." in Material and Methods

And: "As preregistered, we conducted follow-up analyses to examine on which of the DFs there were statistically significant differences in the distribution of scores between SPR and PCR." in Results

Does this mean that the whole Table 2 is exploratory and not confirmatory? But is it preregistered? This could be an error in my reading but as it is this is confusing and does not allow a critical appraisal of the results in Table 2. The results also showcase the pitfalls of prespecifying analyses where the data structure is complicated. If I prespecify a suboptimal analysis scheme in a preregistration and reviewers want significant changes will this then result in entirely exploratory analyses in the final paper? I am unsure about the last point myself but want to mention it here to stimulate discussion.

To summarise, I would have wished for a result reporting and discussion based on the effect sizes. What does it mean if there is a difference between the two scores? What is the consequence of such a difference for the quality of the pre-registration. For ordinal variables an argumentation based on the mean values should be avoided. I find the percentages per ordinal level quite illustrative and a Figure for cases like randomisation and missing data, etc. could be helpful. In this context I miss the connection to survey data on QRPs. It would benefit the paper if you could relate the base rates of QRPs that you find for preregistrations to findings from the literature on self reported QRPs (Loewenstein and others). This would also address some of the concerns of previous reviewers and contextualise your results better. I know that space is limited but you could point out where preregistrations are particularly successful and where there is additional effort necessary.

I sign my reviews

Ulf Toelch, QUEST Center for Transforming Biomedical Research

Dear Dr Bakker,

Thank you for submitting your revised Meta-Research Article entitled "Ensuring the quality and specificity of preregistrations" for publication in PLOS Biology. I have now obtained advice from the original reviewers and have discussed their comments with the Academic Editor.

We're delighted to let you know that we're now editorially satisfied with your manuscript. However before we can formally accept your paper and consider it "in press", we also need to ensure that your article conforms to our guidelines. A member of our team will be in touch shortly with a set of requests. As we can't proceed until these requirements are met, your swift response will help prevent delays to publication. Please also make sure to address the data and other policy-related requests noted at the end of this email.

To submit your revision, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' to find your submission record. Your revised submission must include the following:

- a cover letter that should detail your responses to any editorial requests, if applicable

- a Response to Reviewers file that provides a detailed response to the reviewers' comments (if applicable)

- a track-changes file indicating any changes that you have made to the manuscript.

*Copyediting*

Upon acceptance of your article, your final files will be copyedited and typeset into the final PDF. While you will have an opportunity to review these files as proofs, PLOS will only permit corrections to spelling or significant scientific errors. Therefore, please take this final revision time to assess and make any remaining major changes to your manuscript.

NOTE: If Supporting Information files are included with your article, note that these are not copyedited and will be published as they are submitted. Please ensure that these files are legible and of high quality (at least 300 dpi) in an easily accessible file format. For this reason, please be aware that any references listed in an SI file will not be indexed. For more information, see our Supporting Information guidelines:

https://journals.plos.org/plosbiology/s/supporting-information

*Published Peer Review History*

Please note that you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. Please see here for more details:

https://blogs.plos.org/plos/2019/05/plos-journals-now-open-for-published-peer-review/

*Early Version*

Please note that an uncorrected proof of your manuscript will be published online ahead of the final version, unless you opted out when submitting your manuscript. If, for any reason, you do not want an earlier version of your manuscript published online, uncheck the box. Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us as soon as possible if you or your institution is planning to press release the article.

*Protocols deposition*

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosbiology/s/submission-guidelines#loc-materials-and-methods

Please do not hesitate to contact me should you have any questions.

Sincerely,

Roli Roberts

Roland G Roberts, PhD,

Senior Editor,

rroberts@plos.org,

PLOS Biology

------------------------------------------------------------------------

DATA POLICY:

You may be aware of the PLOS Data Policy, which requires that all data be made available without restriction: http://journals.plos.org/plosbiology/s/data-availability. For more information, please also see this editorial: http://dx.doi.org/10.1371/journal.pbio.1001797

Note that we do not require all raw data. Rather, we ask that all individual quantitative observations that underlie the data summarized in the figures and results of your paper be made available in one of the following forms:

1) Supplementary files (e.g., excel). Please ensure that all data files are uploaded as 'Supporting Information' and are invariably referred to (in the manuscript, figure legends, and the Description field when uploading your files) using the following format verbatim: S1 Data, S2 Data, etc. Multiple panels of a single or even several figures can be included as multiple sheets in one excel file that is saved using exactly the following convention: S1_Data.xlsx (using an underscore).

2) Deposition in a publicly available repository. Please also provide the accession code or a reviewer link so that we may view your data before publication.

Regardless of the method selected, please ensure that you provide the individual numerical values that underlie the summary data displayed in the following figure panels as they are essential for readers to assess your analysis and to reproduce it: Fig 1. NOTE: the numerical data provided should include all replicates AND the way in which the plotted mean and errors were derived (it should not present only the mean/average values).

Please also ensure that figure legends in your manuscript include information on where the underlying data can be found, and ensure your supplemental data file/s has a legend.

Please ensure that your Data Statement in the submission system accurately describes where your data can be found.

------------------------------------------------------------------------

REVIEWERS' COMMENTS:

Reviewer #1:

[identifies himself as Matthew J Page]

The authors have done an excellent job addressing the reviewers' comments and I have no further suggestions

Reviewer #2:

[identifies himself as Ulf Toelch, QUEST Center for Transforming Biomedical Research]

The authors have dealt with all comments in a detailed and thorough manner. Ambiguities particularly in the analysis were clarified and valuable additional analyses have been conducted. This clearly added to to the transparency of the manuscript. Together with the open code and data it will allow readers to critically assess the evidence presented in the manuscript. Importantly, additional limitations have been pointed out in the discussion. Overall, I think this manuscript will be of great interest to the scholarly community and definitively provoke fruitful discussions.

Dear Dr Bakker,

On behalf of my colleagues and the Academic Editor, Lisa Bero, I am pleased to inform you that we will be delighted to publish your Meta-Research Article in PLOS Biology.

PRODUCTION PROCESS

Before publication you will see the copyedited word document (within 5 business days) and a PDF proof shortly after that. The copyeditor will be in touch shortly before sending you the copyedited Word document. We will make some revisions at copyediting stage to conform to our general style, and for clarification. When you receive this version you should check and revise it very carefully, including figures, tables, references, and supporting information, because corrections at the next stage (proofs) will be strictly limited to (1) errors in author names or affiliations, (2) errors of scientific fact that would cause misunderstandings to readers, and (3) printer's (introduced) errors. Please return the copyedited file within 2 business days in order to ensure timely delivery of the PDF proof.

If you are likely to be away when either this document or the proof is sent, please ensure we have contact information of a second person, as we will need you to respond quickly at each point. Given the disruptions resulting from the ongoing COVID-19 pandemic, there may be delays in the production process. We apologise in advance for any inconvenience caused and will do our best to minimize impact as far as possible.

EARLY VERSION

The version of your manuscript submitted at the copyedit stage will be posted online ahead of the final proof version, unless you have already opted out of the process. The date of the early version will be your article's publication date. The final article will be published to the same URL, and all versions of the paper will be accessible to readers.

PRESS

We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with biologypress@plos.org. If you have not yet opted out of the early version process, we ask that you notify us immediately of any press plans so that we may do so on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

Thank you again for submitting your manuscript to PLOS Biology and for your support of Open Access publishing. Please do not hesitate to contact me if I can provide any assistance during the production process.

Kind regards,

Alice Musson

Publishing Editor,

PLOS Biology

on behalf of

Roland Roberts,

Senior Editor

PLOS Biology