Initialisation.

The key parameters to initialise are the S covariance matrices Σ_1:S. We initialise Σ_s at the sample covariance matrix calculated using a specified subset of samples, denoted : (8) where is the P×G matrix of UV results; any missing data in are zero-filled just for the purposes of the above calculation. Should this yield a positive semidefinite , we add εI to ensure positive definiteness at initialisation; for results shown here, we use ε = 0.05.

Our main results are based upon the case of a single covariance matrix (S = 1) in which case comprises all (nonsynthetic null) samples in the training set. When the model is specified to have more than 1 covariance matrix (S>1), we choose the subsets to partition the training set via model-based clustering of the zero-completed version of using the function Mclust() in the R package mclust with default parameter settings.

The likelihood of the MV model (2)–(4) is multimodal, and, hence, convergence of the EM algorithm is sensitive to initialisation. This sensitivity is investigated in a number of ways as part of our model checking section in Methods–Model checking and sensitivity analyses. By repeating our entire analysis for data subsets, e.g., of size 500, we capture variation in empirical initialisation (since the are based only on the training data) as well as variation in the likelihood surface from data subsampling; we demonstrate that our results are robust to the combination of these 2 types of variation.

The initialisation at the empirical covariance matrix (8) is helpful for enabling the EM algorithm to target the global optimum. To demonstrate this, we investigate random, vanilla initialisation of Σ, setting (9) We perform this random initialisation for 10 data subsets of size 2,000 in the single-covariance matrix case S = 1 (these are the same data subsets used for the main cross-validated analysis). Then, in each of these cases, we examine the value of the cross-validated likelihood fit and compare it to the sample covariance initialised cross-validated likelihood fit (S10 Fig, Methods–Cross-validation and model averaging and Methods–Cross-validated likelihood for IMPC data). Across 10 folds, the randomly initialised fits perform systematically worse in all cases in terms of CV likelihood, illustrating the benefits of using a supervised initialisation in this context to mitigate the nonconvexity of the optimisation.

There are potential enhancements to the EM algorithm to increase the probability of convergence to the global maximum, such as the split and merge algorithm of [23]. While our basic EM implementation appears to provide good performance for the datasets considered here, particularly with reasonable initialisation, it could usefully be extended to incorporate such enhancements in future.

Convergence.

The EM algorithm is deemed to have converged when the change in objective function between consecutive iterations falls below a tolerance threshold. We choose the tolerance threshold adaptively, with reference to variation in log likelihood contribution across samples. Specifically, denoting the contribution of the gth sample to the log likelihood at the tth iteration by , the tolerance is set to , where MAD() denotes the median absolute deviation, N_tra the number of training samples, and ε_tol a user-specified constant (we used ε_tol = 10⁻⁴).

Synthetic null data.

We define a synthetic null line to be a subsample of typically 10 to 20 WT animals chosen at random so as to reflect the experimental design properties of an actual KO line. Synthetic nulls play important methodological roles in our inference: most importantly in permutation-based control of the Fdr when calling phenotype hits, but also in estimation of the experimental correlation matrix R in the MV model at (2). Synthetic null lines are generated by randomly selecting groups of WT animals from a single centre so as to match the experimental design characteristics of a particular true KO line at that centre. Specifically, for each litter of the true KO line, we sample from a computationally matched WT litter at the same centre. For a KO litter with l animals that was first phenotyped on day d, we sample a WT litter from all possible WT litters at the same phenotyping centre having at least l animals and randomly select l animals from that litter. Litters that were measured closer in time to day d are selected with higher probability [11].

Hypothesis testing and Fdr.

A vital output of the IMPC is the data-driven compilation of a list of (phenotype, KO gene) pairs at which there is evidence for the phenotype being perturbed by the gene KO. This leads us to the analytical goal of testing the null hypothesis H⁰: θ_pg = 0 with high statistical power under a controlled false positive rate. The IMPC data have many levels of complex structure, resulting in potential for model misspecification and inflated false positive rates for parametric tests. Further, the IMPC’s massive number of often strongly correlated tests calls for an effective power-preserving multiple testing correction. We address these challenges by controlling the Fdr using the Westfall–Young permutation approach, which provides robustness to model misspecification in combination with high statistical power when tests are correlated [32,48]; the synthetic null lines serve as the set of true null hypotheses.

To test the null hypothesis of no perturbation of phenotype p in KO line g, we use a z-statistic defined as the ratio of posterior mean to posterior SD, i.e., (10) with the corresponding definition for the UV model output. We choose a significance threshold, denoted τ, so that if (11) then line g is called as significantly perturbed at phenotype p, with directionality determined by the sign of z_pg.

We choose τ so as to control the Fdr. We use the “Bayesian” Fdr definition [30]: (12) where H⁰ is a null hypothesis, T a test statistic, and a critical (rejection) region, which is chosen to control the corresponding empirical form [30] (13) The definition of Fdr in (13) is conservative in the sense that our control of implies similar control of the Benjamini–Hochberg FDR [30,50]. Our choice of (13) is primarily motivated by convenience: Synthetic null data allow the term in the numerator of (13) to be estimated and controlled. The other terms in (13) can be dealt with straightforwardly: The denominator is known, and the prior ℙ(H⁰ true) can be specified, conservatively at 1 as we do here, or informatively when prior information is available. We estimate Fdr at 2 levels of granularity: the phenotype–gene pair, and the gene. At the phenotype–gene pair level (i.e., for a single test), Fdr_single is the Fdr resulting from rejecting the null hypothesis, , at each phenotype p, gene g pair for which |z_pg|≥τ: (14) We estimate Fdr_single(τ) by (15) where z_pg from synthetic null lines are included in the numerator to estimate the second term in the numerator of (13).

At the gene level, Fdr_complete is the Fdr resulting from rejecting the complete null hypothesis across all phenotypes at each gene for which max_p{|z_pg|}≥τ, (16) with its empirical form defined analogously to (13). We estimate by (17) where again z_pg from synthetic null lines are included in the numerator.

We monitor both and as related but distinct estimates of Fdr. However, it is that we use to select τ to control while maximising power, via the optimization (18) Control of via the permutation based (i.e., synthetic null based) is an implementation of the Westfall–Young permutation procedure [31,32,48].

Replicability and false sign rates.

When a KO line is phenotyped in multiple laboratories, calling hits (identifying significant perturbations) in the same direction in both laboratories is supportive of a method’s replicability. In Fig 6, concordant hits correspond to points in the blue shaded regions. In contrast, hits presenting an increasing phenotype in one laboratory and a decreasing one in the other imply that at least one of the 2 hits is a false positive (indicated by the red regions in Fig 6). In our analyses, we examine such concordance across pairs of contexts (across laboratories as just introduced, and also to compare heterozygotes versus homozygotes). It is useful to be able to relate the degree of observed replicability to an underlying error rate, as this provides extra validation of effective error rate control. We therefore develop a method for quantifying (dis)agreement: a replicability-based estimate of the Fsr, which we denote and derive below. The method maps a contingency table of signed annotations to a compatible Fsr.

The general situation of interest has pairs of signed significance calls outputted in 2 conditionally independent contexts. We represent calls in this section by {−1,0,1} with 1 and −1 denoting significant positive or negative phenotypic perturbations, z>τ and z<−τ, respectively, and zero denoting the nonsignificant result |z|<τ. The general set of concordance data can be represented by counts as in Table 4 where the number of points in the blue and red regions of Fig 6 are denoted by n₋₋+n₊₊ and n₊₋+n₋₊, respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Replicability table between methods A and B.

https://doi.org/10.1371/journal.pbio.3001723.t004

We will show that, while Table 4 provides little information about Fdr, the probabilities underlying Table 4 can be usefully related to the Fsr, defined as the probability of incorrectly estimating the sign of an effect (making a “type S error”) given that the null hypothesis of zero effect is rejected [49]: (19) We motivate our derivation of an estimator for Fsr by considering the following ratio, , which increases with the level of discordance in Table 4: (20) We note that where A,B∈{−1,0,1}, and we express (21) (22) in which we have defined ψ≔ℙ(θ = 0|AB≠0), where ψ is interpretable as a “double false discovery rate,” i.e., the probability of the null (θ = 0) being true given it has been rejected in 2 conditionally independent tests, e.g., on data sets gathered in 2 different laboratories (note that ψ = O(Fdr²) is small under reasonable control of Fdr). Further, in the step from (21) to (22) we have assumed i.e., that false positives are equally likely to be in the positive or negative direction. We also used the following in the step from (21) to (22):

Solving (22) for Fsr gives: (23) The right-hand side of (23) is a decreasing function of ψ, so we define a conservative (slightly upwardly biased) estimator of Fsr by setting ψ = 0: (24) where q has been replaced by the estimator defined at (20). We obtain an approximate confidence interval for Fsr by substituting (in place of in (24)) exact binomial confidence interval bounds for q derived under a model where the number of disagreements follows a binomial distribution with success probability q:

Cross-validation and model averaging.

Highly parameterised statistical models can overfit data, resulting in poor out-of-sample performance. This overfitting concern applies to the MV model here, as it has a flexible and high-dimensional covariance matrix parameterisation, which is learned empirically, although it is somewhat mitigated by the structural regularisation via a factor model representation at (4). To protect against overfitting, all MV results are inferred within a cross-validation framework whereby we split the data set into “training” and “test” sets C times, and then combine test set results across splits using Bayesian model averaging.

We denote each line g as either being in the set of true KO lines, or in the set of synthetic null lines; each of these 2 sets comprises N_tot = 4,548 lines (since each synthetic null line matches the design of a true KO line).

For each cross-validation split c of 1,…,C, we randomly partition into a training set of size N_tra and a test set of size N_tot−N_tra. We randomly partition similarly and independently into and . We proceed to estimate using training genes, i.e., . We then estimate conditional on using test genes, i.e., . Test set estimates of are combined across cross-validation splits using Bayesian model averaging, i.e., (25) which represents the combined posterior as a mixture of the split-specific posteriors, , each of which is a Gaussian mixture. We define posterior MV estimates as the mean and standard deviation of the combined posterior in (25), and these estimates are taken forward to phenotype calling under a controlled Fdr. Our framework for cross-validated empirical Bayes inference is laid out in Table 5.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Cross-validated empirical Bayes inference.

https://doi.org/10.1371/journal.pbio.3001723.t005

Additionally, we calculate a pooled covariance estimate by Bayesian model averaging across the C split-specific models using test set data, i.e., (26) taking forward from (26) to factor analysis (Methods–Factor model).

Sensitivity analysis.

We verify that our downstream factor analysis of is robust by comparing results across different cross-validation folds. Specifically, we compare the varimax-rotated factor loadings from our final estimate defined at (26) to those from each fold c and select the showing highest discrepancy based on the symmetrized KL divergence (), (27) The loadings plots for and are compared in S11 Fig and appear qualitatively similar with differences only at a small number of factors. These limited differences are due to merging or splitting of factors across the 2 decompositions. Our conclusion here is that the factor analysis is relatively insensitive to data subsampling: Variation across factor decompositions should occur at only a small number of factors in the worst case scenario.

Data subsampling.

Here, we examine the stability of results to potential MV heteroscedasticity across KO lines that may not be captured by our MV mixture model. We perform a sensitivity analysis in which we subsample 500 of the total 4,548 lines at random and use them as the training set to refit the MV model. We perform this subsampling c = 1,…,10 times, each time estimating from the training set and retaining the MV model phenotype calls from the test set. We find the fold, c′, with the greatest symmetrized KL divergence between to as at (27).

In Table 6, we compare fold c′’s signed phenotype calls to the corresponding calls in the full analysis (i.e., the cross-validated and model-combined analysis at (26)). The level of discordance is low with a total of 37 disagreements across 8,316 instances where both models call a hit. (Note that we cannot estimate Fsr effectively from Table 6, because of the conditional dependence between test results from the full and subsampled analyses.) It is a reassuring quality control check that picking the most discrepant subsample of size 500 leads only to this small level of discordance, suggesting that a reduced sample size, while reducing power, should not lead to any qualitative disagreement with the conclusions of the full-data analysis.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Comparison of signed phenotype hits for the MV model applied to the most KL-divergent subsampled data set of training size N = 500 (left) compared to the full data set of training size N = 2000 (top).

We represent calls by a number in {−1,0,1}, with 1 and −1 denoting significant positive and negative phenotypic perturbations, respectively, and 0 denoting a lack of statistical significance.

https://doi.org/10.1371/journal.pbio.3001723.t006

Predicting masked data.

It can be seen from Fig 2 that data are often missing across an entire procedure for a KO line. To check missing data inference, we perform the following “mask-predict-compare” algorithm: (i) for each KO line in the test set, artificially mask data from each of its measured procedures in turn; (ii) predict the perturbations underlying the masked data; and (iii) compare the predicted perturbations to those estimated by the UV model on the unmasked data. We refer to the inference on masked data as leave-one-procedure-out MV (LOO-MV). The level of discordance between the LOO-MV and UV results is low (S9 Fig) and is compatible with = 0.4% (95% CI: 0.2% to 0.7%). This is consistent with the false positive rate being well calibrated even when inference is performed in the presence of missing data.

Examining discordance between the MV model and the IMPC database.

Referring back to Results–Comparison with IMPC database and Table 1(B), here, we inspect these 3 cases of disagreement more closely by examining which directionality (our MV model or the IMPC database) is more biologically sensible. We use empirical Bayes to quantify prior beliefs about hit directionality at any particular phenotype p as a probability: ℙ_prior(θ_pg>0|θ_pg≠0). This involves aggregating information on directionality from that phenotype’s hits across all genes, which can be done via a simple average: (28) Since there is no disagreement between our UV model’s calls and those in the IMPC database, we include hits from both these methods in (28), but we do not include calls from our MV model. We use the prior defined in (28) to analyse the 3 signed phenotype hits showing disagreement between our MV model and the IMPC database, yielding a Bayes factor of 1.45 supportive of the MV model, but this is a weak Bayes factor that does not provide substantial evidence either way on which model’s outputted directionality is most sensible on these instances of disagreement; this negative result makes sense given the small sample size of 3 leading to low power.

Gene ontology analysis.

We use the R package GOfuncR to test for co-enrichment between GO terms and IMPC phenotypes. An important feature of this package is that it corrects for multiple testing and interdependency of the tests, using random permutations of the gene-associated variables to control family-wise error rates. We create IMPC–phenotype gene sets comprising genes that are not only significantly perturbed, but also exhibit an effect size of at least 2 times the SD of effect sizes across all genes; in the mathematical notation we have introduced, our IMPC gene set for phenotype p is defined as: (29) (30) We apply an additional filter to focus only on homozygous KOs. We use a family-wise error rate threshold of 5% (the probability of one or more false positives when testing a single IMPC gene set for co-enrichment against all BP GO terms is constrained to be less than 5%).

We perform 1,000 permutations of the GO graph for each IMPC phenotype. The background gene set (also known as the gene universe) is defined for the MV model to be all homozygote–KO genes at which some phenotype measurements are available (a total of 2,628 genes); for the UV model, the background gene set comprises all homozygote–KO genes at which this particular phenotype is available. The basic inferential tool is a Fisher exact test for independence of row and column classifications in a 2-by-2 contingency table, such as those shown in Fig 7. Implementing this basic test within GOfuncR ensures that error rates, here family-wise error rates, are controlled appropriately.

Factor model.

In order to facilitate interpretation of phenotypic perturbations, we calculate the eigendecomposition of the correlation matrix underlying , i.e., (31) in the varimax() function with default parameters in R [51].

Denoting the rotated sparse loadings P-vectors by λ_l, l = 1,…,20, the lth factor score for the gth KO gene is .

Hypothesis testing is performed to identify significant perturbations in factor scores. Denoting and , we form test statistics (32) and control Fdr analogously to Methods–Control of error rates, where instead of phenotypes p = 1,…,P we now have factor scores l = 1,…,20.

Extreme deconvolution (XD).

The model used in [23] and which underlies the accompanying software package Extreme Deconvolution (XD) is similar to the likelihood we use (2)–(3) but has a few differences. XD has the constraint M = 1, i.e., does not have the multiple scaling parameters ω_1:M introduced in [22]. XD generalises to underlying mixture components with nonzero means μ_1:S that are themselves estimated, i.e., N(θ_·g|μ_s, Σ_s). For the purposes of method comparison, XD is run with μ_s≡0, as the zero-mean model is appropriate for the data sets we analyse here.

XD uses a similar EM algorithm to ours to maximise the likelihood jointly with respect to Σ_1:S and π (and μ_1:S more generally). Throughout the XD EM algorithm optimisation, the rank of each Σ_s remains the same as the rank of its initialised value [22]. We initialise XD at the same values Σ_1:S and π as we initialise our own model (except when we are running XD to generate data-driven matrices for mash, in which case we follow the directions in [22]). In summary, any differences between the fit of our model and the fit of XD are driven primarily by the absence of scaling parameters ω_1:M in XD and our factor-model regularisation of Σ_1:S.

Multivariate adaptive shrinkage (mash).

Our method’s model likelihood (2)–(3) is the same as was introduced in [22] and which is the basis for the software package mash. A particularly important insight in [22] is the introduction of a multiscale mixture across a ladder of scales denoted by ω_1:M. The authors note the utility of this approach in the context of multi-tissue eQTLs, and we find it also to be useful for MV mouse phenotyping data. We believe the multiscale covariance model form of [22] has the potential to enhance MV inference across a broad range of scientific disciplines.

The key difference between our approach and mash is in how the covariance matrices Σ_1:S are defined and estimated. We parameterise our model with a small number of regularised covariance matrices (we consider S = 1,2 here) and optimise Σ_1:S and π collectively as part of model fitting. In contrast, mash generates and fixes a larger number (S = P+10) of covariance matrices Σ_1:S, in advance of optimising (2)–(3) with respect to π.

In more detail, mash generates 2 distinct types of covariance matrices: 8 data-driven and P+2 canonical. The 8 data-driven covariance matrices inputted into mash are a low-rank representation of the empirical phenotypic covariance among those samples exhibiting largest phenotypic effects. Three of the data-driven covariance matrices are generated using the XD software [23]. In addition, P+2 canonical P×P covariance matrices are generated, comprising the identity matrix, a matrix of ones, and for p = 1,…,P where e_p is a P-vector with zeros everywhere except for the pth element which is set to 1.

While an elegant aspect of mash is that the optimisation with respect to π given Σ_1:S is convex, its generation of covariance matrices involves nonconvex optimisation within the XD software, so there is potentially some sensitivity to initialisation [22]. Our EM algorithm’s MAP optimisation with respect to Σ_1:S and π is nonconvex, and we investigate sensitivity to initialisation in Methods–Initialisation.

Hit rates, error rates, and model fit.

We compare the power (hit rates) and error rate (estimated Fdr or Fsr) of the different methods on the IMPC data. We consider a number of hypothesis testing frameworks, determined by their test statistics and critical regions (rejection criteria).

Test statistics for testing for a perturbation of phenotype p at gene g:

1. the z statistic
2. the local false sign rate [27]:

(33)

Critical regions:

1. controlling Fdr_complete<5% via a permutation-based test-statistic threshold τ, i.e., with critical region either |z_pg|>τ or lfsr_pg<τ (Methods–Control of error rates and [32,48]);
2. nominally controlling the local Fsr lfsr_pg<5% [22].

Table 7 shows hit rates and error rates under 3 different methods of error rate control, with the subtables corresponding to the test statistics and critical regions defined above: Table 7A presents A1 (test statistic A with critical region 1); Table 7B presents B1; and Table 7C presents B2. Hit rates are shown stratified according to whether the raw data are measured or missing (with 95% nonparametric bootstrap CIs). We display error rate estimates , , and as defined in Methods–Hypothesis testing and Fdr.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Hit rates and error rates compared across models.

The row showing the model and error rate control used in the main analyses in the paper, , is highlighted in bold. The highest hit rates are underlined.

https://doi.org/10.1371/journal.pbio.3001723.t007

Cross-validated likelihood for IMPC data.

Within the inferential framework described in Methods–Cross-validation and model averaging, we calculate the likelihood of the test set data under a model fitted using only the training data. With reference to (2)–(3), the per-sample log cross-validated likelihood for fold c is (34) where is the set of KO genes in the test set of fold c (Methods–Cross-validation and model averaging). Table 8 shows benchmarking results where we present the mean of (±2×SEM) taken across 10 cross-validation folds.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Comparison of cross-validated log likelihood across MV models on the IMPC data.

The row showing the model used in the main analyses in the paper, , is highlighted in bold. The largest CV log likelihood is underlined. The results shown are the per-sample log likelihood of (34) averaged across folds c = 1,…,10, along with ±2 SEM intervals.

https://doi.org/10.1371/journal.pbio.3001723.t008

Cross-validated likelihood on additional data set.

We compare the various MV methods on an additional data set to examine if the same qualitative performance of the various methods persists. A natural data set to use is the multi-tissue eQTL study of Urbut and colleagues [22] on which mash was first developed. The data set comprises 16,069 samples, each of which corresponds to a (gene, single nucleotide polymorphism) pair. These expression quantitative trait loci (eQTLs) are measured in 44 tissues. So, N_tot = 16,069 and P = 44, in contrast to N_tot = 4,584 and P = 148 for the IMPC data.

We analyse the eQTL data using the empirical Bayes cross-validation framework laid out in Methods–Cross-validation and model averaging. We follow the methods of [22] for data preprocessing and estimation of R. The size of our training folds on the eQTL data are 5,000 (in contrast to 2,000 for the IMPC data), but otherwise, parameter settings are the same. One important difference is that there are no missing data in the eQTL study. Table 9 shows the CV log likelihood comparison on the eQTL data set.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Comparison of cross-validated log likelihood across MV models on the eQTL data from Urbut and colleagues [22].

The largest CV log likelihood is underlined. The results shown are the per-sample log likelihood of (34) averaged across folds c = 1,…,10, along with ±2 SEM intervals.

https://doi.org/10.1371/journal.pbio.3001723.t009

Discussion of benchmarking results.

The main results we have presented our based upon the MV model (ComposeMV) with S = 1 and K = 20 (notation introduced in Results–Multivariate model) while controlling as described in Methods–Control of error rates. We refer to this model as in this section. We now briefly discuss the results of benchmarking across 12 models under various means of error rate control (Tables 7 and 8).

Focusing on Table 7A, where we control , the hit rate on measured data for is the largest across all benchmarked models at 10.5% (10.2, 10.9). This hit rate is also optimal when compared to the other 2 considered methods of error rate control (Table 7B and 7C). It’s worth noting, when comparing hit rates to Table 7C, that the monitored error rates in Table 7C are generally higher than in Table 7A and 7B, attributable to using a different method for error rate control, nominally controlling lfsr<5%.

In terms of optimal hit rates when data are missing, we see that other models (e.g., ComposeMV with S = 2 and K = 15) performed slightly better with 2.0% (1.8, 2.1) compared to 1.3% (1.2, 1.4) for , but this comes with a higher estimated error rate compared to .

Turning to the cross-validated likelihood comparison in Table 8, we see that the mean per-sample cross-validated log likelihood for is = −53.4 (−53.6, −53.1). This is marginally improved upon by ComposeMV with S = 2 and K = 15 having = −53.0 (−53.2, −52.7). , and ComposeMV generally, compare favourably with existing methods, performing somewhat better than mash which has = −54.2 (−54.4, −53.9) and considerably better than XD with S = 2, which has = −60.0 (−60.7, −59.3).

For the benchmarking on an additional eQTL data set, shown in Table 9, a similar pattern emerges—the current paper’s MV model, labelled ComposeMV, performs somewhat better in terms of CV likelihood compared to mash, while performing substantially better than XD. Interestingly, for the eQTL data benchmarking, the ComposeMV models with S = 2 perform best of those in the table, which suggests that having a mixture of multiple learned covariance matrices (in addition to the multiscale ladder of the ω_m) may be particularly useful in certain contexts.