Dear Dr. Walleczek,

Editor comment

Please submit your revised manuscript by Apr 14 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Michael B. Steinborn, PhD

Section Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1.Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously? -->?>

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available??>

The PLOS Data policy

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English??>

Reviewer #1: Yes

**********

Reviewer #1: The manuscript, “Metascientific replication project with the advanced meta-experimental protocol of the transparent psi project procedures for testing the precognitive effect claimed by Bem,” investigates the replicability of claims about precognitive effects reported by Bem (2011) using advanced methodologies, including the AMP-TPP protocol, across three large-scale studies. While Study 2 replicated an exploratory finding suggesting a slight below-chance effect, neither the original precognitive effect nor the replicated anomaly was confirmed in Study 3. The results highlight methodological advancements for rigor in psychological research and suggest that observed results are better explained by method-derived anomalies rather than psi-theory.

The manuscript features a well-powered design and demonstrates commendable adherence to open-science and transparency practices, significantly enhancing its credibility and reproducibility while offering well-founded criticism of psi-theory. However, several areas warrant improvement, including clarifying key methodological details, improving the comprehensibility of the discussion, and addressing the results more thoroughly with critical consideration of specific psi concepts. Overall, this study is a valuable contribution to the field, and the authors’ commitment to rigorous research practices is evident. I believe that addressing the issues outlined above in a major revision would greatly enhance the manuscript’s quality.

For detailed comments, see the attached document.

**********

what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy

Reviewer #1: Yes:  Julian Gutzeit

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: Review.pdf

Dear Dr. Walleczek,

Please submit your revised manuscript by Aug 30 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Michael B. Steinborn, PhD

Section Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions??>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously? -->?>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available??>

The PLOS Data policy

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English??>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

Reviewer #1: I want to thank the authors for their thorough response and revision of the manuscript. I think they have successfully improved the clarity and the overall rigor and coherence of their work. I do not require any further changes and recommend accepting the manuscript as it is. Nice work!

Reviewer #2: see my comments below or attached pdf.

General Comments:

As a statistical methodologist and meta-researcher, rather than a domain expert in psi or precognitive effects, my review focuses primarily on the methodological rigor and transparency of the submitted manuscript. I commend the authors for their ambitious effort in designing and conducting three registered replication studies to test the precognitive effect claimed by Bem (2011). The manuscript is well-written, with a clear structure and a commendable commitment to open science practices, including preregistration, data transparency, and the use of advanced methodological protocols such as the Advanced Meta-Experimental Protocol (AMP) and Transparent Psi Project (TPP) procedures. These practices align with the growing emphasis on replicability and reliability in scientific research. Plos One should be proud to consider publishing work that prioritizes such rigor, particularly in a field where replication studies remain underappreciated. The authors’ efforts to address methodological challenges in psi research, such as confirmation bias and intrinsic measurement bias, are noteworthy and contribute to advancing metascience in this domain. However, I have two major concerns regarding the justification for the replication studies and the generalizability of the findings, which warrant major revisions to strengthen the manuscript.

1. The authors provide a critique of the limitations of meta-analyses in parapsychology, noting that the supportive claims for psi phenomena may be compromised by the methodological weaknesses of individual studies included in these analyses (see refs. 9, 32–34). This is a valid concern, given the prevalence of exploratory designs, lack of preregistration, and potential for questionable research practices in the field. However, the rationale for replicating Bem’s Experiment 1 specifically is less robust. The authors state, “The present work investigated the most-widely discussed claim for an experimental psi effect in recent years, namely the precognitive effect claimed with Experiment 1 in the report by Bem. One major reason for choosing the paradigm of Bem Experiment 1 was the fact that a meta-analysis of 90 replication studies claimed an overall significant precognition effect—seemingly confirming the original claim of precognition”. This justification is insufficiently developed, as it does not critically engage with the quality of the 90 replication studies included in Bem et al. (2016; F1000Research, 4, 1188).

Meta-analyses of replication studies, if conducted rigorously, represent high-level evidence, as they aim to synthesize findings from studies explicitly designed to test the same phenomenon, theoretically minimizing selective reporting or publication bias. Questioning the conclusions of such a meta-analysis requires a stronger empirical basis than the general methodological concerns raised in the Introduction. To strengthen their rationale, I recommend that the authors systematically assess the risk of bias in the 90 replication studies included in Bem et al. (2016). This could involve a structured evaluation using established tools (e.g., RoB 2.0) or a custom risk-of-bias framework for replication studies to examine aspects such as preregistration status, sample size adequacy, blinding, and handling of exploratory analyses. The results of this assessment should be reported as an initial section in the Results, providing a clear justification for the need for the current replication studies. Additionally, the authors could explore whether the meta-analytic conclusions depend on study quality by conducting subgroup analyses (e.g., comparing high- vs. low-quality studies) or meta-regression to examine the influence of methodological rigor on effect sizes. Such analyses would provide a more robust foundation for the replication effort and clarify whether the claimed precognitive effect warrants further scrutiny due to methodological flaws in prior studies.

2. The authors have implemented rigorous protocols, including the AMP and TPP, to enhance the validity and transparency of their replication studies. The use of counterfactual meta-experiments (e.g., REG-based trials) and diagnostic calibration routines to detect systematic biases is particularly commendable, as these approaches address potential sources of error that are often overlooked in psychological and parapsychological research. However, the replicability of the authors’ findings—particularly the anomalous below-chance effect observed in Study 2—requires verification by independent laboratories using the same experimental protocol, population characteristics, and analytical methods. The field of metascience has increasingly emphasized the value of multi-laboratory replication projects, such as Registered Replication Reports (RRRs) or Many Labs initiatives, which provide robust evidence of replicability by accounting for laboratory-specific variations and reducing the risk of idiosyncratic findings (e.g., Klein et al., 2014, Perspectives on Psychological Science). While the authors’ three studies are high-powered and methodologically sophisticated, their execution within a single research framework limits the generalizability of the results.

I recommend that the authors acknowledge this limitation explicitly in the Discussion section and tone down their claims about the replicability (or lack thereof) of the precognitive effect. The authors should also discuss the feasibility of a multi-laboratory replication effort to test the robustness of their findings, particularly the unexpected below-chance effect in Study 2, which was not replicated in Study 3. This discussion should address potential sources of heterogeneity (e.g., participant characteristics, online vs. lab settings) that could influence replicability and outline how future studies could address these factors. By moderating their claims and emphasizing the need for independent replication, the authors can strengthen the manuscript’s credibility and align it with best practices in metascience.

Minor comments:

While the statistical analyses are well-described, the rationale for choosing specific priors in the Bayesian proportion tests (e.g., BUJ prior, replication prior) could be clearer. The authors should briefly explain why these priors were selected and how they influence the interpretation of the Bayes factors, particularly for readers unfamiliar with Bayesian methods in parapsychology.

Please make raw data and code publicly available. There are no data and code archived in the repository provided by the authors (https://osf.io/8jteq/)

Sincerely,

Yefeng Yang. Note that I sign all my review comments

Reviewer #3: The article is detailed and of certain reference value, but there are still some minor issues to be addressed, as follows:

1. Description of experimental procedures is not rigorous

"the only substantial differences... participants completed the trials online... minor technical adjustments were made to the software."

The specific content of "minor technical adjustments" is not explained. Please provide an explanation.

2. Discussion section

Sections 4.3-4.7 show a slight imbalance in the comparison between the psi interpretation and the traditional interpretation. Some paragraphs are piled with technical jargon, resulting in insufficient readability.

It is recommended to use a table to compare the core viewpoints, evidential support, and limitations of the two interpretations, so as to enhance visualization.

Simplify the discussion on the quantum mechanics part, focus on the contradiction between the "non-signal transmission principle" and the precognitive effect, and avoid deviating from the theme.

3. In Section 4.8 (Conclusions), add specific suggestions for future research to enhance the forward-looking nature of the study.

4. Please supplement the limitations of this study.

**********

what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy

Reviewer #1: Yes:  Julian Gutzeit

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: comments on PONE-D-24-43107.pdf

Metascientific replication project with the advanced meta-experimental protocol of the transparent psi project procedures for testing the precognitive effect claimed by Bem

PONE-D-24-43107R2

Dear Dr. Walleczek,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager®  and clicking the ‘Update My Information' link at the top of the page. For questions related to billing, please contact billing support .

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Michael B. Steinborn, PhD

Section Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewer #2:

Reviewer #3:

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions??>

Reviewer #2: Partly

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously? -->?>

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available??>

The PLOS Data policy

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English??>

Reviewer #2: Yes

Reviewer #3: Yes

**********

Reviewer #2: The author team has done a great job in addressing my comments and criticisms, especially in terms of clarifying the motivation of this replication study to properly credit prior work and avoid mis-criticising prior work, and avoid exaggerating findings (generalizability of their findings and conclusions should be interpreted in-context and along with limitations). To clarify, I am not the subject expert, and I am unable to judge the significance of this work. As a statistician, I think this replication study is well-registered, and conducted. I am also a meta-researcher and advocate for open science and encourage more replication studies verifying prior findings. I am delighed to see the publication of this work.

Yefeng Yang (signed)

Reviewer #3: I really appreciate the efforts made by the authors for this. The article has been revised very well. I suggest it be accepted for publication.

**********

what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy

Reviewer #2: Yes:  Yefeng Yang

Reviewer #3: No

**********