Authorization of Animal Experiments Is Based on Confidence Rather than Evidence of Scientific Rigor

Accumulating evidence indicates high risk of bias in preclinical animal research, questioning the scientific validity and reproducibility of published research findings. Systematic reviews found low rates of reporting of measures against risks of bias in the published literature (e.g., randomization, blinding, sample size calculation) and a correlation between low reporting rates and inflated treatment effects. That most animal research undergoes peer review or ethical review would offer the possibility to detect risks of bias at an earlier stage, before the research has been conducted. For example, in Switzerland, animal experiments are licensed based on a detailed description of the study protocol and a harm–benefit analysis. We therefore screened applications for animal experiments submitted to Swiss authorities (n = 1,277) for the rates at which the use of seven basic measures against bias (allocation concealment, blinding, randomization, sample size calculation, inclusion/exclusion criteria, primary outcome variable, and statistical analysis plan) were described and compared them with the reporting rates of the same measures in a representative sub-sample of publications (n = 50) resulting from studies described in these applications. Measures against bias were described at very low rates, ranging on average from 2.4% for statistical analysis plan to 19% for primary outcome variable in applications for animal experiments, and from 0.0% for sample size calculation to 34% for statistical analysis plan in publications from these experiments. Calculating an internal validity score (IVS) based on the proportion of the seven measures against bias, we found a weak positive correlation between the IVS of applications and that of publications (Spearman’s rho = 0.34, p = 0.014), indicating that the rates of description of these measures in applications partly predict their rates of reporting in publications. These results indicate that the authorities licensing animal experiments are lacking important information about experimental conduct that determines the scientific validity of the findings, which may be critical for the weight attributed to the benefit of the research in the harm–benefit analysis. Similar to manuscripts getting accepted for publication despite poor reporting of measures against bias, applications for animal experiments may often be approved based on implicit confidence rather than explicit evidence of scientific rigor. Our findings shed serious doubt on the current authorization procedure for animal experiments, as well as the peer-review process for scientific publications, which in the long run may undermine the credibility of research. Developing existing authorization procedures that are already in place in many countries towards a preregistration system for animal research is one promising way to reform the system. This would not only benefit the scientific validity of findings from animal experiments but also help to avoid unnecessary harm to animals for inconclusive research.


Questions about internal validity
Allocation concealment

Preparation of the animals, induction of disease or animal model
Evaluation for allocation concealment assesses whether the experimenter preparing the animals (e.g. entry health check, inclusion criteria) or inducing the disease or animal model is blind with respect to the future allocation of the animals to the treatment groups of the experiment.

NA:
 If no preparatory activities are performed or animals are not assigned to treatment groups (e.g. study on strain differences where animals are already "assigned" to treatment groups). YES:  If the manipulation (preparation, induction of disease or animal model) on the animals is performed prior to their assignment to treatment groups. NO:  If the manipulation (preparation, induction of disease or animal model) on the animal is performed after their assignment to treatment groups.
 If the experimenter who performs the manipulation (preparation, induction of disease or animal model) on the animals is aware of their future allocation to treatment groups.
 If no information is provided as to whether the experimenter who performs the manipulation (preparation, induction of disease or animal model) on the animals is aware of their future allocation to treatment groups.

Conduct of study
Refers to the application of treatments to, and manipulations of, the animals in the course of the study.

NA:
 If blinding is impossible; i.e. if treatment groups differ visibly (e.g. different strains of mouse). YES:  If it is clearly stated that the investigator is blind with respect to the treatment group of the animals when manipulating them throughout the course of the experiment. NO:  If the investigator is not blind with respect to the treatment group of the animals when manipulating them throughout the course of the experiment.
 If no information is provided indicating that the investigator is blind with respect to the treatment group of the animals when manipulating them throughout the course of the experiment.

2.2
Outcome assessment Refers to the collection of data from the animals in the course of the study.

NA:
 If blinding is impossible; i.e. if treatment groups differ visibly (e.g. different strains of mouse). YES:  If it is explicitly stated (or unmistakable by the described procedure) that the investigator is blind (unaware) with respect to the treatment group of the animals when collecting data from them. NO:  If the investigator is not blind with respect to the treatment group of the animals when collecting data from them.
 If no information is provided indicating that the investigator is blind with respect to the treatment group of the animals when collecting data from them.

Treatment allocation
Refers to whether or not the animals are allocated randomly to treatment groups.

NA:
 If there are no treatment groups. YES:  If the animals are allocated randomly to treatment groups.  If a study design of the category of "Randomized Block Designs" (e.g. Latin Square, Cross-over, Double-Crossover) is used. NO:  If no information is provided indicating that the animals are allocated randomly to treatment groups.

Conduct of study
Refers to whether or not the order in which animals are manipulated and data collected is randomized or counterbalanced among treatment groups.

NA:
 If the study design does not allow randomization or counterbalancing. YES:  If the order in which animals are manipulated and data collected is randomized or counterbalanced among treatment groups.
 If a study design of the category "Randomized Block Designs" (e.g. Latin Square, Cross-over) is used.  If the order in which animals are manipulated or data collected is determined by the randomization protocol used for treatment allocation. NO:  If the order in which animals are manipulated or data collected is not randomized or counterbalanced among treatment groups.
 If no information is provided indicating that the order in which animals are manipulated or data collected is randomized or counterbalanced among treatment groups.  If the method of sample size calculation is not stated explicitly or relevant key parameters are missing.

4.2
Based on some reference Refers to reference to previous studies either by the same or other experimenters (e.g. own data, own experience, pilot studies, publications) or to advice from third parties (e.g. statistical service, biostatisticians).

NA:
 If the study is a single case study.

YES:
 If any reference is provided as to how the sample size was determined ("sample size was chosen so as to reach significant results" is not enough).
 If any reference to a sample size calculation is mentioned but key parameters are missing (e.g. "we have done a power analysis") NO:  If no reference is provided as to how the sample size was determined.

Inclusion and exclusion criteria
Refers to criteria determining as to whether or not an animal is a suitable subject for the planned study (e.g. health status, body weight or size, performance). These criteria must be assessed prior to the study onset and need to be taken into account when determining sample size.

YES:
 If inclusion and/or exclusion criteria have been specified (explicitly or implicitly). NO:  If no inclusion and/or exclusion criteria have been specified.

5.2
Termination criteria Refers to criteria for excluding animals in the course of a study.

YES:
 If termination criteria have been specified (explicitly or implicitly). NO:  If no termination criteria have been specified.

6
Primary outcome variables 6.1 Specification of primary outcome variable Refers to the main variable considered in the outcome assessment, i.e. to what is being measured. One primary variable (and possibly several secondary variables) need to be determined prior to the study onset and need to be taken into account when determining sample size.

YES:
 If one primary outcome variable is specified or only one is measured.  If one of the investigated variables is linked to sample size calculation or statistical analysis. 1 Specification of statistical model Refers to whether or not the method by which the data are analyzed has been specified and explained in detail. Whether or not the suggested statistical method is appropriate with respect to the study design will be evaluated for a subset of applications for which sufficient detail is available (with support by Christina Nathues). Whether or not an application qualifies for such further evaluation is noted in the comment section of the checklist.

NA:
 If the study is a single case study. YES:  If full detail on statistical model is provided (incl. dependent, independent, and random factors).  If it is stated that descriptive statistics, but no inferential statistics, will be performed, and a justification is provided for this. NO:  If the method of statistical analysis is not specified or not enough detail is provided.

7.2
General statistical method Refers to whether or not the methods by which the data are analyzed is stated in general terms, meaning that only general concepts of statistics are mentioned but no explicit details provided (i.e. statistical analysis without further explanation).

NA:
 If the study is a single case study. YES:  If a statistical method is mentioned (e.g. ANOVA, t-test, parametric statistics, "data will be statistically analyzed" is not enough). NO:  If no mention is made of the method of statistical analysis or not enough detail is provided (e.g. "statistical tests will be performed").  If a reference is provided (e.g. "pers. comm.", "own data", "standard method", citations, etc.) supporting the method for the purpose it is intended to be used. NO:

Questions about the accuracy of Form A
 If no reference is provided supporting the use of the method.

9
Individual identification 9.1 Marking and tagging methods Refers to information on the marking or tagging method as required on the application Form A under point 52.

NA:
 If the experimental unit is a group of animals (e.g. cage, aquarium) rather than the individual animal or if animals are singly housed.
 If the animals are farm animals, pet animals, or zoo animals that are either marked by law (e.g. ear tags, chips), or can be identified by individual characteristics. YES:  If the marking method is clearly stated (e.g. ear clipping, staining, fur marks, etc.) NO:  If no information on marking or tagging is provided.

Number of animals
Refers to information on the total number of animals as required on the application Form A under point 54.3, in particular whether the total no. of animals (table 33)   If contradictions occur between the number(s) of animals mentioned at different places on Form A.

Number of groups
Refers to information on the number of treatment groups as required on the application Form A under point 54.3, in particular whether the number of groups (incl. control group(s)) is correctly stated.

NA:
 If there are no treatment groups (e.g. monitoring studies). YES:  If the number of groups being studied is explicitly and consistently stated. NO:  If the number of groups is not stated.  If contradictions occur between the numbers of groups mentioned at different places on Form A.

Number of animals per group
Refers to information on the number of animals per group as required on the application Form A under NA:  If the animals are wild animals. YES:  If detailed score sheets are provided.  If the variables and scores and the interval at which they are assessed is described in detail (i.e. "the animals will be checked daily by animal care personnel" is not enough). NO:  If no or no clear information is provided on the variables and scores and/or the interval at which they are assessed.

Degree of severity
Refers to information regarding the number(s) of animals per degree of severity as required on the application Form A under point 56.4.

YES:
 If the number(s) of animals per degree of severity is clearly listed and add up to the number(s) of animals listed in table 33. NO:  If no information is provided on the degree(s) of severity of the animals used or if the information is incomplete (i.e. does not include all animals listed in  If the fate of all animals is clearly stated (e.g. method of euthanasia, rehoming, re-use in following studies). NO:  If the fate of the animals is not stated or not for all animals listed in table 33.

Comments
In this section specific comments can be made that may be relevant for the analysis of the evaluations. Moreover, a comment should be made if the application is a potential candidate for a more thorough assessment of the statistical methods.