PONE-D-25-08411Meta-regression to explain shrinkage and heterogeneity in large-scale replication projectsPLOS ONE

Dear Dr. Heyard,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by May 31 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Yazhou He

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Review of the Manuscript ID PONE-D-25-08411 Title: “Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects” for the Plos One Journal.

General Comments

From my point of view, it is a very interesting topic and simultaneously it seems that to the best of my knowledge is an empirical research aims to use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and the heterogeneity variance (represented by the scale). This could also provide valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualise results of PRs. The proposed methodology is illustrated using data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online.

The paper consists of following sections: Introduction, Methods, Results, Discussion, Implications for Nursing Education, Limitations, Conclusion.

However, I find some recommendations:

1. The Manuscript needs careful English proofreading because there are some shortcomings. For instance, the article “the” is sometimes missing in front of nouns, the message in some paragraphs is not clear enough. It looks like the first part was written by one author with a greater command of the English language, and the rest of the paper was written by someone else. The numerous grammar errors made this a difficult paper to read. It was strange to see the authors refer to tables that were not submitted. I was unable to find any supplementary material to the submission, so I think this was truly omitted by the authors. Please read the manuscript carefully.

2. It would be very useful to add in the "Introduction" section the purpose, objectives and hypothesis of the research. I consider that a weak point of the paper is that the authors did not show the novelty of the paper compared to other works. That is why, I consider that the introduction should specify the novelty of the paper compared to other papers published in this area.

3. The authors must also show the values of the adjusted R-square, which is more relevant in the methods used in this paper.

4. Authors must present the results of the processing in tabular form and interpret the results. The paper cannot be accepted in this form.

5. The conclusions at the end of the paper should be expanded showing the policy implications of the research results.

In conclusion, the article should be improve. It should also be enhanced with a review of the literature adequate to the subject and a broader interpretation and commentary of the research results.

Reviewer #2: Review of the Manuscript ID PONE-D-25-08411 Title: “Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects” for the Plos One Journal.

General Comments

From my point of view, it is a very interesting topic and simultaneously it seems that to the best of my knowledge is an empirical research aims to use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and the heterogeneity variance (represented by the scale). This could also provide valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualise results of PRs. The proposed methodology is illustrated using data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online.

The paper consists of following sections: Introduction, Methods, Results, Discussion, Implications for Nursing Education, Limitations, Conclusion.

However, I find some recommendations:

1. The Manuscript needs careful English proofreading because there are some shortcomings. For instance, the article “the” is sometimes missing in front of nouns, the message in some paragraphs is not clear enough. It looks like the first part was written by one author with a greater command of the English language, and the rest of the paper was written by someone else. The numerous grammar errors made this a difficult paper to read. It was strange to see the authors refer to tables that were not submitted. I was unable to find any supplementary material to the submission, so I think this was truly omitted by the authors. Please read the manuscript carefully.

2. It would be very useful to add in the "Introduction" section the purpose, objectives and hypothesis of the research. I consider that a weak point of the paper is that the authors did not show the novelty of the paper compared to other works. That is why, I consider that the introduction should specify the novelty of the paper compared to other papers published in this area.

3. The authors must also show the values of the adjusted R-square, which is more relevant in the methods used in this paper.

4. Authors must present the results of the processing in tabular form and interpret the results. The paper cannot be accepted in this form.

5. The conclusions at the end of the paper should be expanded showing the policy implications of the research results.

In conclusion, the article should be improve. It should also be enhanced with a review of the literature adequate to the subject and a broader interpretation and commentary of the research results.

Reviewer #3: The introduction is a situational context of replication studies (RPs) in psychology and allied sciences. It reports low reproducibility rates and systematic shrinkage of effect sizes. The authors support the use of meta-regression techniques, specifically location-scale models, for exploring drivers of heterogeneity and shrinkage between replication study-pairs.

• Introduction would benefit from a clearer definition of how existing meta-analytic approaches lack an adequate explanation of heterogeneity and how location-scale meta-regression adds to the debate.

• A very concise summary of the data and context (RPP and RPEE) would improve background to aims flow.

• Establish key terms like shrinkage and heterogeneity at the outset for clarity to a multidisciplinary readership.

Literature Review

The article discusses pertinent replication studies (e.g., RPP, RPEE) and presents key statistical techniques from the meta-analysis literature (e.g., additive vs. multiplicative models, location-scale meta-regression).

• Literature review is discontinuous and interrupted by introduction and methodology sections. Perhaps consolidate it all in one section to:

• Strongly distinguishes conceptual contributions of previous meta-regression applications (e.g., RPP supplement, Altmejd et al.).

• Highlights any shortfall in the use of scale modeling or quantification of residual heterogeneity by past studies.

• Merge the rationale for using multiplicative vs. additive heterogeneity models with citations to past applications, explaining when to use each in RP contexts.

Methodology

The authors employ location-scale meta-regression models to replication-minus-original differences in effect sizes across 87 study-pairs in two large-scale replication projects (RPP and RPEE). Both multiplicative and additive heterogeneity models are estimated.

• Make assumptions explicit and test them: While the modeling framework is accurate, the paper can be enhanced by making important assumptions (e.g., normality of residuals, independence of study-pairs) explicit and testing them.

• Residual diagnostics (e.g., Q-Q plots, heteroscedasticity tests) should be described or displayed, particularly for the models in the end.

• Model selection transparency: While model selection using AIC is treated, it would be helpful to justify the specific criteria used (e.g., smallest AIC or parsimony). Cross-validation or leave-one-out procedures can be referenced as checks for robustness.

• Details regarding how covariates were chosen (apart from "subject knowledge") would be valuable—did authors use pre-registered selection procedures, and were there any variables dropped due to multicollinearity?

• Covariates such as "number of authors" and "citation count" can probably be correlated; a VIF analysis or correlation matrix would be added to address multicollinearity.

• The weighted regression approach is appropriate, but independent errors are specified, which would be violated with the pairing of original and replication studies. The article could investigate cluster-robust standard errors or even a multilevel model for improved management of the nested structure.

• Provide further details about how to diagnose the distribution of residuals—especially since inference relies on normality.

• Consider adding goodness-of-fit statistics or R²-type measures, where available, to supplement AIC and facilitate interpretation.

• There could be value in exploring Bayesian meta-regression or reporting bootstrap confidence intervals, especially given the comparatively modest sample size and model complexity.

Results

The results reveal robust evidence of shrinkage (original effects > replication effects) and between-study-pair heterogeneity. Some covariates such as original standard error, study discipline, author gender ratio, and replication language are associated with shrinkage or heterogeneity.

• Coefficient interpretation (e.g., in log-heterogeneity scale) may be difficult to apply for readers. Clarify scale predictors' implications in easy language, maybe using pictorial examples.

• It would help in discussing more narrative around significant results (e.g., the gender makeup finding, the language of replication) and to discuss potential mechanisms or theoretical implications.

• Incorporate a sensitivity analysis, for instance, excluding outliers or re-running analyses with a subset of more similar studies.

• The argument would be assisted by clearer differentiation between correlation and causation. For example, the correlation between proportion of author gender and shrinkage is intriguing, but caution should be taken in attributing such effects to causality.

• Some conclusions (e.g., about precision or experience of authors) would be bolstered by supportive references or theoretical rationale.

• The authors acknowledge the exploratory nature of the study, but this could be even more emphatically stated, particularly with regard to the data-driven model choice.

Reviewer #4: Thank you for the opportunity to review your manuscript titled "Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects." Your study addresses a timely and important issue within the metascientific domain—specifically, how effect size shrinkage and heterogeneity manifest in large-scale replication efforts. The use of meta-regression to explore predictors across multiple replication datasets is both novel and meaningful, and the overall structure of the manuscript is logically organized and well-motivated.

Reviewer #5: The article represents a significant methodological contribution to the field and has great potential for publication. However, there are some improvements that could be made before publishing it.

1) Although the objectives are implicit throughout the text, they could be presented in a more explicit and structured way in the introduction, making it easier to read for a wider, interdisciplinary audience.

2) Although the article compares the two heterogeneity models (additive and multiplicative), the theoretical and practical justification for choosing between them could be better substantiated. I recommend the authors include a section that discusses more clearly in which contexts each model is more appropriate and how the results would change if only one of the models were used.

3) Despite the fact that the coefficients of the models are presented with their confidence intervals, a clearer interpretation of the practical impact of each covariate is lacking. For example: What is the expected magnitude of shrinkage for an increase of 1 unit in the average number of citations?

4) I suggest the authors point out some limitations. The absence of variables on methodological bias or questionable practices. The relatively small sample size (n=87), which may affect the stability of models with many covariates.

5) Suggestions: a future article could compare the different model selection criteria.

Reviewer #6: At the outset, let me to highlight that the research study has already been made available over two online pages:

a. https://ideas.repec.org/p/osf/metaar/e9nw2_v1.html

b. https://osf.io/preprints/metaarxiv/e9nw2_v2

The comments pertaining to this research study are listed below:

1. The problem statement has been well-defined. It clearly identifies the gaps in scientific knowledge and provides strong justification for the current research.

2. The Literature Review has been done effectively. Most of the previous works have been appropriately referenced. It justifies the scope of work undertaken in this research article.

3. A clear understanding of the work undertaken. The researchers have well conceptualized and summarized the entire research work undertaken by them.

4. The researchers have well demonstrated the methodology. The methodology both for shrinkage and heterogeneity used, clearly identifies relevant strengths and weaknesses of the methods.

5. The results are interpreted in light of the proposed research problem and existing literature. This includes explanations and instructional tables and figures as well as supplementary information provided in the appendix. The analysis and Interpretations based on the results are quite convincing.

6. The references are correctly mentioned and are quite adequate in number. The latest references have been provided.

7. There is one suggestion which can somehow ensure the robustness of results pertaining to model selection criterion. In addition to utilizing Akaike information criterion (AIC), other criteria such as Schwarz/Bayesian information criteria or/and scoring rules combined with leave-one-out cross-validation may also be used for presenting respective tables and figures. By doing this, it will also overcome one of the limitation highlighted in the research article.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Stefanos Balaskas

Reviewer #4: Yes: Baranidharan Subburayan

Reviewer #5: Yes: Raul Afonso Pommer-Barbosa

Reviewer #6: Yes: Muhammad Abdus Salam

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: reviewer Report.docx

Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects

PONE-D-25-08411R1

Dear Dr. Heyard,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Yazhou He

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

Reviewer #5: All comments have been addressed

Reviewer #6: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #5: Yes

Reviewer #6: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have modified the paper according to the reviewer's recommendations. I agree to accept the paper for publication.

Reviewer #2: All the recommendations have been applied. The paper substantialy improved. I agree to accept the paper for publication.

Reviewer #3: The authors have adequately addressed all my comments and concerns. I believe the paper is ready for publication.

Reviewer #5: After a careful reading of the manuscript, I consider that the authors have responded satisfactorily to the indicated revisions, so I recommend the publication of this manuscript.

Reviewer #6: The authors have duly incorporated the reviewers’ comments where appropriate, ensuring that key concerns and suggestions have been adequately addressed. The manuscript is now in a well-developed form and is suitable for publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Stefanos Balaskas

Reviewer #5: Yes: Raul Afonso Pommer Barbosa

Reviewer #6: Yes: Muhammad Abdus Salam

**********