Dear Natasha,

Thank you for submitting your manuscript entitled "Scientists should not increase the overall n by default when including both sexes in in vivo studies: evidence from statistical simulations" for consideration as a Meta-Research Article by PLOS Biology.

Your manuscript has now been evaluated by the PLOS Biology editorial staff, as well as by an academic editor with relevant expertise, and I'm writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. After your manuscript has passed the checks it will be sent out for review. To provide the metadata for your submission, please Login to Editorial Manager (https://www.editorialmanager.com/pbiology) within two working days, i.e. by Feb 13 2023 11:59PM.

If your manuscript has been previously peer-reviewed at another journal, PLOS Biology is willing to work with those reviews in order to avoid re-starting the process. Submission of the previous reviews is entirely optional and our ability to use them effectively will depend on the willingness of the previous journal to confirm the content of the reports and share the reviewer identities. Please note that we reserve the right to invite additional reviewers if we consider that additional/independent reviewers are needed, although we aim to avoid this as far as possible. In our experience, working with previous reviews does save time.

If you would like us to consider previous reviewer reports, please edit your cover letter to let us know and include the name of the journal where the work was previously considered and the manuscript ID it was given. In addition, please upload a response to the reviews as a 'Prior Peer Review' file type, which should include the reports in full and a point-by-point reply detailing how you have or plan to address the reviewers' concerns.

During the process of completing your manuscript submission, you will be invited to opt-in to posting your pre-review manuscript as a bioRxiv preprint. Visit http://journals.plos.org/plosbiology/s/preprints for full details. If you consent to posting your current manuscript as a preprint, please upload a single Preprint PDF.

Feel free to email us at plosbiology@plos.org if you have any queries relating to your submission.

Kind regards,

Roli

Roland Roberts, PhD

Senior Editor

PLOS Biology

rroberts@plos.org

Dear Natasha,

Thank you for your patience while your manuscript "Scientists should not increase the overall n by default when including both sexes in in vivo studies: evidence from statistical simulations" was peer-reviewed at PLOS Biology. It has now been evaluated by the PLOS Biology editors, an Academic Editor with relevant expertise, and by three independent reviewers.

Based on the reviews, we are likely to accept this manuscript for publication, provided you satisfactorily address the remaining points raised by the reviewers and the following data and other policy-related requests.

IMPORTANT - please address the following:

a) Please attend to the requests from the reviewers.

b) Regarding reviewer #1's comment "Perhaps another Article Type is more appropriate?," the answer is "yes" - I hadn't noticed that this was submitted as a regular Research Article; I have now switched it to a Meta-Research Article, which we feel should address this reviewer's valid concern - the advance here is in meta-research, not statistics.

c) Please change the Title to "Statistical simulations show that scientists need not increase overall sample size by default when including both sexes in in vivo studies"

d) Please address my Data Policy requests below; specifically, we need you to supply the numerical values underlying Figs 1BCD, 2BCD, 3BCDE, 4BCD, S1ABC, either as a supplementary data file or as a permanent DOI’d deposition. I note that you already have a Zenodo deposition, but access is currently restricted, so I can’t check compliance; please rectify this.

e) Please cite the location of the data clearly in all relevant main and supplementary Figure legends, e.g. “The data underlying this Figure can be found in S1 Data” or “The data underlying this Figure can be found in https://doi.org/10.5281/zenodo.7629021

As you address these items, please take this last chance to review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the cover letter that accompanies your revised manuscript.

We expect to receive your revised manuscript within two weeks.

To submit your revision, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' to find your submission record. Your revised submission must include the following:

- a cover letter that should detail your responses to any editorial requests, if applicable, and whether changes have been made to the reference list

- a Response to Reviewers file that provides a detailed response to the reviewers' comments (if applicable)

- a track-changes file indicating any changes that you have made to the manuscript.

NOTE: If Supporting Information files are included with your article, note that these are not copyedited and will be published as they are submitted. Please ensure that these files are legible and of high quality (at least 300 dpi) in an easily accessible file format. For this reason, please be aware that any references listed in an SI file will not be indexed. For more information, see our Supporting Information guidelines:

https://journals.plos.org/plosbiology/s/supporting-information

*Published Peer Review History*

Please note that you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. Please see here for more details:

https://blogs.plos.org/plos/2019/05/plos-journals-now-open-for-published-peer-review/

*Press*

Should you, your institution's press office or the journal office choose to press release your paper, please ensure you have opted out of Early Article Posting on the submission form. We ask that you notify us as soon as possible if you or your institution is planning to press release the article.

*Protocols deposition*

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please do not hesitate to contact me should you have any questions.

Sincerely,

Roli

Roland Roberts, PhD

Senior Editor,

rroberts@plos.org,

PLOS Biology

------------------------------------------------------------------------

DATA POLICY:

You may be aware of the PLOS Data Policy, which requires that all data be made available without restriction: http://journals.plos.org/plosbiology/s/data-availability. For more information, please also see this editorial: http://dx.doi.org/10.1371/journal.pbio.1001797

Note that we do not require all raw data. Rather, we ask that all individual quantitative observations that underlie the data summarized in the figures and results of your paper be made available in one of the following forms:

1) Supplementary files (e.g., excel). Please ensure that all data files are uploaded as 'Supporting Information' and are invariably referred to (in the manuscript, figure legends, and the Description field when uploading your files) using the following format verbatim: S1 Data, S2 Data, etc. Multiple panels of a single or even several figures can be included as multiple sheets in one excel file that is saved using exactly the following convention: S1_Data.xlsx (using an underscore).

2) Deposition in a publicly available repository. Please also provide the accession code or a reviewer link so that we may view your data before publication.

Regardless of the method selected, please ensure that you provide the individual numerical values that underlie the summary data displayed in the following figure panels as they are essential for readers to assess your analysis and to reproduce it: Figs 1BCD, 2BCD, 3BCDE, 4BCD, S1ABC. NOTE: the numerical data provided should include all replicates AND the way in which the plotted mean and errors were derived (it should not present only the mean/average values).

IMPORTANT: Please also ensure that figure legends in your manuscript include information on where the underlying data can be found, and ensure your supplemental data file/s has a legend.

Please ensure that your Data Statement in the submission system accurately describes where your data can be found.

------------------------------------------------------------------------

DATA NOT SHOWN?

- Please note that per journal policy, we do not allow the mention of "data not shown", "personal communication", "manuscript in preparation" or other references to data that is not publicly available or contained within this manuscript. Please either remove mention of these data or provide figures presenting the results and the data underlying the figure(s).

------------------------------------------------------------------------

REVIEWERS' COMMENTS:

Reviewer #1:

Phillips et al. use simulations to show that including animals of both sexes and using an appropriate statistical analysis does not reduce the power to detect treatment effects, compared with using animals of one sex. The authors identify two misconceptions held by many scientists and three errors that are frequently observed in published papers. They then use simulations to demonstrate why the misconceptions are incorrect and provide better ways to analyse data.

This paper has been submitted as "Research Article" and I have been asked to comment on the "novelty and significance of the findings, its technical merit, the and experimental design". All the results are expected based on statistical theory and are well known to statisticians, and so are not novel or significant. However, demonstrating the results with simulations may be more convincing to many scientists. Having such a paper to reference when discussing these issues with colleagues and collaborators would be valuable. Perhaps another Article Type is more appropriate?

The statistical analyses are appropriate and the supplementary information provides a useful case study with real data. I cannot access the scripts and data on Zenodo as access is restricted.

I only have a few minor comments:

1) The font size is too small in all the figures and difficult to read. Adding

+ theme(text = element_text(size = 14))

to the plots should fix this.

2) Line 210: It would be useful to add Gelman and Stern's paper to the references here (https://www.tandfonline.com/doi/abs/10.1198/000313006X152649). I believe they were the first to draw attention to this problem.

3) Line 419-421: I think this sentence is missing some words. Is the intended meaning: "There is no requirement to perform a power calculation to detect …"?

4) Panel A in Figure 2 is referenced multiple times (lines 191, 193, 198, 200) but it's unclear where we should be looking as there are four graphs. Adding a more specific pointer would make it clearer (e.g. Fig 2A, top right).

Reviewer #2:

This is another outstanding and immensely practical contribution from this group. The paper does an excellent job in identifying the main issues, myths and all-too-common procedural errors involved in SABV studies. They provide transparent and readily reproducible methodology and code. I especially loved the explanatory boxes which made the methods clear without interrupting the flow of the text.

Ethical oversight assurances are provided, and most animal-based procedures reported in accordance with ARRIVE 2.0. However, there was no statement as to the eventual disposition of the mice, and how they were euthanased prior to collecting kidney tissue samples. Please provide description of these methods.

A few other minor comments:

1. The worked example is a really good idea. It could be made more practical for the average reader by streamlining the text quite a bit. The text is a bit clunky and focuses a lot on the results and less on the actual 'pipeline'. The text could be reduced by about 1/3rd & more bullet points added so it is easier to follow. Could the authors also consider placing this more upfront & incorporate something in the title & intro that states that the authors provide step by step methods on how to implement these features in actual studies (e.g. "tutorial" or "primer" into the title?) Or maybe consider a second companion paper? I am a bit worried that researchers who really need to see this will be put off by the heavy emphasis on the simulations & lessen the value of the practical implementation tips.

2, Although the authors incorporated bias minimisation methods such as allocation concealment and randomisation in their own studies, there is almost no mention up front. These are essential components of experimental design.

3. There is the added effect of cage. Was cage included as a block or were cage effects assumed to be zero?

4. The sample size justification statement is far too vague to be verifiable. A few sentences could be added to explain. Not saying that it is, but as it stands, it seems to reinforce the all-too-common practice of choosing 5 per "group" because that is the usual per-cage stocking density.

5. Mention should be made of sample size estimation in general for factorials. Powering factorials is different from the usual approach taken for two-group or single-factor completely randomised designs. As you know, main effects are estimated by comparing the means of 'sets of conditions' or a combination run, not by directly comparing the means of individual "groups". This needs to be mentioned upfront as well. Much of the resistance to adopting factorial designs is when researchers try to power them from G*Power which suggests far larger samples sizes are required. Sample sizes will also be too small for pairwise comparison of "groups". it is far more fruitful and less confusing to clarify that with factorials what is replicated are experimental runs (factor combinations) & that the researcher needs to think about total sample size, not group size. This is mentioned (sort of) in the text but it needs to be stated much more clearly. mention might be made of the Lakens and Caldwell paper (2021) on sample size for factorials. DOI:10.1177/2515245920951503

6. If the factorial is conducted for screening purposes, emphasis should be on how "relative promise" of the sex x trt interactions are evaluated e.g in terms of precision of the estimate. not statistical significance. mention of practical effect size could be added up front.

Reviewer #3:

Summary: In this manuscript, Philips and Colleagues conduct statistical simulations to address common misconceptions surrounding the use of both sexes in in vivo studies. It is commonly thought that in order to design an experiment which is statistically powered to analyze data by sex, the sample size (n) must double. To address this, they conduct power analyses which mirror a variety of experimental conditions with varied effects of treatment and sex. They provide clear explanations surrounding the power analyses and the results. Their simulations highlight the fact that in most scenarios, when sexes are split across treatment groups, the sample size is enough to sufficiently power the study.

Added Value of the Study: Several other articles have discussed how to design and analyze experiments involving both sexes. Most articles are written as reviews and provide overviews and/or tutorials of statistical methods. This manuscript expands upon that work by including the aforementioned power analyses and provides a case study which can be used for training purposes. It also reiterates the importance of factorial vs. pooled analyses, demonstrating that their incorrect use could lead to misleading results that ultimately detract from our understanding of how sex informs health and disease.

Overall, I find this article to be well written and easily accessible to a broad audience. However, I acknowledge that I do not have expertise in statistical modeling and hope that other reviewers can comment on strengths and/or weaknesses that pertain to the methods used here.

Dear Natasha,

Thank you for the submission of your revised Meta-Research Article "Statistical simulations show that scientists need not increase overall sample size by default when including both sexes in in vivo studies" for publication in PLOS Biology. On behalf of my colleagues and the Academic Editor, Marcus Munafò, I'm pleased to say that we can in principle accept your manuscript for publication, provided you address any remaining formatting and reporting issues. These will be detailed in an email you should receive within 2-3 business days from our colleagues in the journal operations team; no action is required from you until then. Please note that we will not be able to formally accept your manuscript and schedule it for publication until you have completed any requested changes.

Please take a minute to log into Editorial Manager at http://www.editorialmanager.com/pbiology/, click the "Update My Information" link at the top of the page, and update your user information to ensure an efficient production process.

PRESS: We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with biologypress@plos.org. If you have previously opted in to the early version process, we ask that you notify us immediately of any press plans so that we may opt out on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

Thank you again for choosing PLOS Biology for publication and supporting Open Access publishing. We look forward to publishing your study. 

Sincerely,

Roli

Roland G Roberts, PhD, PhD

Senior Editor

PLOS Biology

rroberts@plos.org