a. Animals.

All experiments were performed with C57BL/6J mice. The details regarding the age, sex, and origin of the animals are summarized in Table 8 as they differed across stages. Likewise, and as part of the 3-stage experimental approach animal numbers, housing, husbandry, and experimental conditions also varied across stages (see Table 8). Note that the animal numbers in Table 8 represent the result of the power calculation; however, some sites included more animals to, for example, even out the total number of animals per group. Thus, the number of observations differs from the one required on Table 8. The animals used in each protocol were experimentally naïve and came from independent batches at all sites. The use of different animal providers among laboratories served as a representation of common differences between laboratories and study populations to test the performance of the different protocols.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Variables and corresponding values across the different stages and their respective protocol(s) followed by all sites.

https://doi.org/10.1371/journal.pbio.3001886.t008

b. Design.

The readouts of distance traveled in the open field were collected during 3 consecutive stages with around 1 year apart, in 7 different laboratories. In stage 1, all sites performed the study with minimal alignment (strain, age, drug treatment, vehicle, primary outcome measure, and test duration) using their “in-house” standard operation procedures (SOPs) under the local conditions at each site (light intensity, arena size, husbandry conditions, etc.); this is referred to as local protocol and was intended as a baseline measurement of the variability between a “random” sample of laboratories. In stage 2, specific husbandry and experimental conditions were harmonized across laboratories. In addition, besides a standardized cohort of the harmonized protocol, with all factors standardized within laboratories, a heterogenized cohort of the harmonized protocol was used; this cohort aimed to increase within-site variability of the data by systematically varying 2 selected “heterogenization” factors using a 2 × 2 factorial design. Finally, the stage 3 goal was to challenge the sensitivity of the standardized cohort of the harmonized protocol from stage 2 by using the same protocol but with a different drug treatment than previously used (Table 8). The local protocol used in stage 1 was replicated in stages 2 and 3 as a control condition for the harmonized protocols, and to assess the replicability of results obtained with the local protocol across stages within each of the laboratories.

An a priori power analysis (G*Power v3.1.9.2) was performed based on effect sizes estimated from literature [25] and a previous study with the NMDA receptor antagonist MK-801 performed by one of the partners. The study was powered so that each treatment for each sex and each laboratory is treated as a stand-alone test. Alpha was set to 0.05 and power to 0.9 for a t test for means-difference for 2 independent means. The calculated effect sizes were considered large. To evaluate the required n numbers in the case of a “medium” effect size, the effect size was halved and corresponding n numbers were again estimated. The required n numbers for “MK-801 0.3 mg/kg” and “3 mg/kg diazepam” were considered too low and were therefore increased to 6. Recommendations of a minimum number of animals to enroll per drug and dose treatment are shown in Table 8. This power calculation was used for the stage 1 protocol. The number of animals used per stage was adapted according to the compounds used and the number of animals available in the animal use licenses.

c. Pharmacological compounds.

In each of the stages, the spontaneous locomotor activity after acute administration of a compound was compared across treatment groups and/or dosages. While the compounds and dosage used varied across stages (Table 8), the pretreatment time was kept constant with administration 30 minutes before the start of the test. Diazepam (Duchefa Biochemie, BUFA, Roche, Merck, Sigma Aldrich, TEVA) was used in stage 1 together with MK-801 (Sigma Aldrich); the latter was also used in stage 2. Clozapine (Sigma Aldrich cat# C6305) was administered in stage 3. Data from the different drug treatments were compared with the control group that received the respective vehicle (i.e., drug dose = 0 mg/kg). The vehicle was different across treatments according to the compound solubility (see Table 8).

The dose range of each compound was selected according to the goal of each stage as follows. Stage 1: At the localization stage, we aimed for a dosage with a strong effect and a dosage with a medium effect, assuming replicability would be lower with a subtler effect. Doses were based on previous data collected from one of the partners. However, from this, we expected 0.3 mg/kg to have a stronger effect (hence the smaller sample size), but it turned out that it was the other way round. Therefore, we used the smaller dose in stage 2.Stage 2: At stage 2, the focus was on comparing localization with harmonization (primary aim) and standardization with heterogenization (secondary aim); because this increased the number of treatment groups considerably, we limited the study to a single dose against saline control. For the same reason, we limited the study to a single sex (choosing females to minimize the risk of injury and aggression, which is more frequent in males). Thus, stage 2 is a kind of proof-of-principle study to inform stage 3.Stage 3: Similar to stage 1, we wanted a dose with a strong effect and a dose with a weaker effect.

d. Protocols.

Local protocol: A predefined minimum set of requirements were aligned across sites (Table 8). Variables not addressed in these minimum requirements were to be handled according to the SOP of each site. All variables, both aligned and nonaligned, were reported post hoc following stage 1 tests to generate an inventory of the different environmental variables that may have influenced the between-laboratory variability. This protocol represents the most common scenario in preclinical biomedical animal research, where independent studies are standardized within laboratories but conditions and procedures vary between laboratories.

The local protocol was replicated in all stages to test the replicability of results within each laboratory, and as the control protocol to compare the other protocols at each stage. Therefore, the number of animals, drug treatments, and vehicles differed across stages according to Table 8.

Harmonized protocol–Standardized cohort: From the local protocol inventories, variables that differed between sites were identified and chosen based on their biological relevance and feasibility to be modified at all sites; these were further harmonized across sites to test whether the variability of effect sizes across sites observed in the local stage could be reduced by controlling for these variables. This cohort aimed to assess how much the lab-specific differences in the standardized protocol contribute to between-lab variation. As indicated in Table 8, this protocol was used in stages 2 and 3; the treatment, vehicle, and treatment dosage differed between stages as well as the inclusion of male mice in stage 3.

Harmonized protocol–Heterogenized cohort: This protocol was identical to the standardized cohort, with the exception that 2 factors were systematically varied within sites to account for the variability between sites, namely light intensity in the experimental arena was set to either dim (20 Lux) or bright (80 Lux) and the window time of testing concerning the light–dark phase was varied between early (2 to 4 hours after light on) or late (8 to 10 hours after lights on).

The standardized and heterogenized cohorts of the harmonized protocol were tested in parallel to assess whether simple heterogenization of environmental factors would further reduce the between laboratory variability compared to the local protocol.

a. Behavioral assessment.

The Open Field (OF) test was used throughout the study to measure the effect of different pharmacological compounds (Table 8) on locomotor activity. Mice were placed in the center of an empty open arena and horizontal activity was recorded for 15 minutes; afterwards, the animal was taken back to its home cage. The outcome measure was the total distance traveled in the OF arena. Details on the open field arenas, recording, and scoring methods are summarized in Table 8.

Animal handling and drug administration, scoring, and analyses were performed blinded to the treatments unless stated otherwise. This means that the person handling and dosing the animals was not aware of the allocation of animals into experimental groups nor about which of the treatments was being administered. Animals were block randomized into groups by various methods (Blindr; random number generator from Mathematica v11, Wolfram; R script provided by one of the partners or developed in-house as part of their Data Management software) except for site 5, which used even distribution of animals into groups. The outcome measure was the total distance traveled for 15 minutes in the Open Field.

There were no predetermined exclusion criteria unless animals presented health issues. However, some sites were able to add animals to, for example, even out the number of animals in each treatment group. Therefore, the raw data has more animals than the ones mentioned in Table 8 for some of the sites.

b. Data management.

Once data were acquired, each site transferred its raw data and metadata to an Excel structure shared across sites, which was then shared for centralized analysis. Each site was responsible for checking the soundness of the data (i.e., quality check for the video length, accurate scoring, correct group coding, etc.). Averaged data from each treatment group per site across stages can be found in the S1, S2, and S3 Supplementary Stage files.

c. Statistical methods.

Before the analysis, a log transformation was performed to the outcome variable (i.e., total distance traveled) because data are naturally bounded between 0 and + infinity. The log transformation changes the bound and sets it between −infinity and +infinity, which is more aligned with the assumptions of linear modeling. Moreover, we are sure that the model accounts for those natural bounds when estimating the effects. Otherwise, it could be that some effects have a 95% CI lower bound lower than 0, which would be uninformative given the outcome variable analyzed.

Within stages and protocols analysis.

To study the lab-to-lab variation by stage and protocol, 2 types of models were used. The first one explored the differences in dosing effects by the laboratory. This analysis reflects a situation where each laboratory would perform the comparisons internally and aims to highlight the variability of estimated differences between laboratories. It was expected that the Harmonized protocol provides more consistent results than the Local protocol. A simple linear regression was fitted to the natural logarithm transform of total distance travelled by laboratory with drug treatment as a unique fixed effect: (1) where Y_id is the natural logarithm transform of total distance travelled i for drug treatment d,

β₀ is the intercept of the model (the expected Y_id for drug treatment d of reference),

β_d is the effect of drug treatment d on Y_id (the expected change in Y_id when drug treatment d is considered), and

ε_id is the random error associated with Y_id: ε_id~N(0, σ²_ε) where is the residual or biological variance.

The drug treatment effects and their contrasts were estimated using the R package emmeans.

Note that for the standardized protocol in stage 3, Y_id is the natural logarithm transform of total distance travelled plus 1 because of some zero values for which the natural logarithm would not be defined. Moreover, also for the standardized protocol in stage 3, huge discrepancies were observed between variances of drug treatment. Hence, 1 variance per drug treatment was modelled instead of 1 pooled variance: . This specific model with individual variance per drug treatment was fitted with the R package glmmTMB.

The second model explores the differences in dosing effects overall laboratories accounting for the lab-to-lab variability. It aimed to directly estimate the variance associated with differences between laboratories and assess the percentage of total variance it represented. It was expected that the Harmonized protocol provides lower lab-to-lab variance than the Local protocol while having similar residual variances. A linear mixed model was fitted to the natural logarithm transform of total distance travelled with drug treatment as a unique fixed effect and laboratory as well as the interaction between laboratory and drug treatment as random effects: (2) where Y_idl is the natural logarithm transform of total distance travelled i for drug treatment d and lab l,

β₀ is the intercept of the model (the expected Y_idl for drug treatment d of reference),

β_d is the effect of drug treatment d on Y_idl (the expected change in Y_idl when drug treatment d is considered),

b_l = the random intercept of laboratory l: ,

d_dl = the random intercept of drug treatment d and laboratory l: , and

ε_idl is the random error associated with Y_idl: ε_idl~N(0,σ²_ε).

σ²_b, σ²_d and σ²_ε are referred as the variance components in this model. In linear mixed model, variance is decomposed in several terms of interest to understand which ones are the main source of variability in the data. In this specific case, σ²_b is the lab-to-lab variability, σ²_d is the variability in differences between drug doses observed between lab, and σ²_ε is the residual or biological variability.

The model was fitted with R package lmer. The drug treatment effects and their contrasts were estimated using the R package emmeans. Note that for the standardized protocol in stage 3, the same modifications were applied as for the simple linear model.

Between stages analysis.

To study the stage-to-stage variation for 1 protocol, 2 types of approaches were used. The first one compared the within-stage results by stage for common dosing groups. It assesses if effects observed in laboratories and if the variance components are similar from one stage to another. This would indicate that the results are replicable. Local protocols of stages 1 and 2 are compared using the common control and MK-801-0.2 mg/kg groups, whereas Harmonized protocols (standardized cohort) stages 2 and 3 are compared using the common 2.5 mg/kg clozapine treatment. Note that the models presented simplify for the standardized cohort of the Harmonized protocol because there is only 1 dose.

The second model explored the effects over all laboratories accounting for lab-to-lab and stage-to-stage variability. It aimed to estimate the proportion of the total variance attributable to between-stage differences. A linear mixed model was fitted to the natural logarithm transform of the total distance travelled with drug treatment as unique fixed effect and laboratory as well as interaction between laboratory and drug treatment, stage and the interaction between stage and drug treatment as random effects: (3) where Y_idls is the natural logarithm transform of total distance travelled i for drug treatment d,

lab l and stage s,

β₀ is the intercept of the model (the expected Y_idls for drug treatment d of reference),

β_d is the effect of drug treatment d on Y_idls (the expected change in Y_idls when drug treatment d is considered),

b_l = the random intercept of laboratory l: ,

d_dl = the random intercept of drug treatment d and laboratory l: ,

s_s is the random intercept of stage s: ,

g_ds is the random intercept of drug treatment d and stage s: , and

ε_idls is the random error associated with Y_idls: ε_idls~N(0, σ²_ε), where is the residual or biological variance.

Influence of external factors.

Additional models were performed to study the effects of heterogeneous factors introduced for the Harmonized protocol–Heterogenized cohort (the light intensity and the time of testing) in stage 2 and the blocking factor for the Harmonized protocol in stage 3 (sex) on the data. Two approaches were considered, first a by-laboratory analysis, then an across laboratory analysis, both by factor of interest. The models were based on the ones used in the within stages and protocols analysis. Two fixed effects were added each time, the factor and the interaction between the factor and the drug treatment. The models were fitted with R package lmer. Statistical significance of those effects was tested with F tests (Type III) using the R package lmerTest. The different effects and their contrasts were estimated using the R package emmeans.

The boxplots were computed with the raw data that can be found in https://osf.io/8f6yr/, while the treatment effect differences are reported in the S1, S2, and S3 Supplementary Stage files. See each figure for specifics. The analysis codes can be found in the OSF repository (DOI: 10.17605/OSF.IO/8F6YR).