Model and implementation

We model the LFC of genes as a mixture of three Gaussian components, corresponding to non-DE, up-regulated and down-regulated genes and make use of random effects meta-analysis to incorporate the variance in the gene-level LFC estimates in parameter estimation (see Methods for details). The statistical tests at the gene set level consist of comparison of the fit of models in which the LFC distribution is the same (null hypothesis) or different (alternative hypothesis) for genes in the inset (the gene set of interest) and outset (the remaining genes in the background set). We evaluated three forms of model comparison, corresponding to different kinds of GSA. The first consists of comparing the fits of a model in which all parameters of the LFC mixture distribution are shared between genes in the inset and outset to a model that allows the proportion of DE genes to be larger in the inset. We refer to this as the one degree of freedom (1DF) test. It evaluates whether there is a greater proportion of DE genes in the inset than in the outset. The second kind of test evaluated allows both the proportion of DE genes and the fraction of the DE genes that are upregulated/downregulated to differ between the inset and the outset, making it a two degrees of freedom (2DF) test. In the third kind of test, we compared the fit of a model in which all parameters of the mixture distribution are allowed to differ between genes in the inset and outset to a model in which all parameters are shared. This test has six degrees of freedom (referred to as the 6DF test) and is sensitive to any differences in the mixture distribution between the inset and outset (e.g. a difference in the size of the DE effect even if the proportion of DE genes is the same). In all cases we used the Likelihood Ratio test (LRT) for the comparison of model fit (details of all of the tests are provided in Methods).

Performance on simulated data

We first used simulations to assess method performance relative to existing methods. In the simulations the group effect was assumed to be normally distributed in the inset and outset, but with a larger variance in the inset, resulting in a larger number of DE genes with LFC above the threshold shown (Fig 1A). This is deliberately far-removed from our model, which fits a mixture of three normal distributions. The simulations illustrate a key feature that distinguishes our approach to GSA from many existing methods. GSA methods designed to identify gene sets with a higher proportion of DE genes typically perform statistical tests on the proportions of genes falling below an arbitrary p-value threshold. This proportion, however, will depend on the number of samples in an experiment. Indeed, it is likely to be the case that all genes are DE to some extent between any two biologically distinct groups of samples, with this difference becoming statistically significant, given sufficient sample numbers. Instead, we define DE by setting a threshold on the absolute value of the LFC and our method compares proportions of genes for which the true (but unknown) absolute value of the LFC is above this threshold.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Power on simulated data with large effect sizes.

A) LFC distributions in the inset (red) and outset (blue) for an example simulation. The variance of the inset distribution was allowed to change, leading to simulations with different levels of enrichment for DE genes in the inset. The dashed lines show the location of the threshold (fold-change of 1.5) used to define differential expression for methods that apply such a threshold. B, C) Comparison of power of different methods as a function of the level of enrichment in the inset for simulations consisting of 20 (B) or 40 (C) samples. The x-axis indicates the relative enrichment in the inset of the proportion of the LFC distribution beyond the upper or lower thresholds.

https://doi.org/10.1371/journal.pcbi.1010278.g001

Our method provided a significant improvement in power, defined as the proportion of enriched gene sets detected, compared to ORA, and GSEA on the simulated data (Figs 1B, 1C, 2B and 2C). As expected, the improvement in power was sensitive to the threshold used to define differential expression (S1, S2 and S3 Figs). The undirectional version of GSEA, using the absolute signal to noise ratio (SNR) as the ranking metric, had low power with low sample numbers, but its power increased with increasing sample numbers, due to the increased number of sample permutations available to derive the p-value. Accounting for the uncertainty in the gene-level effect size estimates in our method resulted in a significant improvement in power, relative to a version of our method that did not take the standard error of the effect-size estimates into account (Fig 3). As expected, this improvement was greater for lower sample numbers, for which there is greater uncertainty in the effect size estimates. The false positive rate of our method was similar to that of existing methods (S4 Fig) and it remained below the significance threshold whether or not the standard error of the effect size estimate was included in the model (S5 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Power on simulated data with small effect sizes.

A) LFC distributions in the inset (red) and outset (blue) for an example simulation. As in (Fig 1) the variance of the inset distribution was allowed to change, leading to simulations with different levels of enrichment for DE genes in the inset. The dashed lines show the location of the threshold (in this case a fold-change of 1.1) used to define differential expression. B, C) Sample sizes of 20 (B) or 40 (C) were simulated. The x-axis indicates the relative enrichment in the inset of the proportion of the LFC distribution beyond the upper or lower thresholds.

https://doi.org/10.1371/journal.pcbi.1010278.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Incorporating uncertainty boosts power.

Power obtained with and without taking account of the standard error of the gene-level effect size estimates is shown in blue and red, respectively for simulations based on 20 (A) or 40 (B) samples. The parameter space of the simulations was the same as in Fig 1.

https://doi.org/10.1371/journal.pcbi.1010278.g003

Our method requires researchers to specify upfront the magnitude of the DE effects of interest. Unsurprisingly, the power of our method to detect the difference in the proportion of genes above the threshold in the simulated data depended on the relationship between the DE threshold and the LFC distributions in the inset and outset (Figs 1 and 2 and S1–S3 Figs). The highest power is achieved when the threshold defines a region in the effect size distribution in which there is a substantial difference between the inset and outset. No such fold-change threshold is specified for existing enrichment-based methods and, instead, such methods often rely on p-value thresholds from statistical tests at the individual gene level. The lack of an explicit definition of differential expression can lead to statistical inconsistencies. For example, the power of ORA decreased with increasing sample numbers in some simulation settings (Fig 1 and S6 Fig). This resulted from an increase in the power to detect the smaller expression changes in the outset as the sample number increases, causing a reduction in the difference in proportions of significantly differentially expressed genes. A threshold can also be applied in the original differential expression analysis when using ORA, with the test for enrichment applied to the proportion of genes that are statistically significant and have absolute LFC above the threshold. This resulted in a consistent increase in power in ORA with increasing sample numbers (Figs 1 and 2 and S1–S3 Figs), highlighting the importance of explicitly defining the effect sizes that are of interest.

The improvement in power over established methods was maintained in simulations that included unequal numbers of upregulated and downregulated genes (S7 Fig). The power of the more constrained undirectional 1DF test and ORA was unchanged as the proportion of upregulation among the DE genes varied. As expected, the directional GSEA achieved high power once a majority of DE genes in a set were upregulated. Our 2DF test and 6DF test also increased in power in these scenarios, as they had greater flexibility to fit the differences in the effect size distributions between the inset and outset. These simulations consisted of 10,000 genes, 100 of which were in the enriched gene set. We also assessed the effect of varying the gene set size on power. Unsurprisingly, power decreased for all methods as the gene set size was reduced (S8 Fig). When the size of the enriched set was increased to 200 genes, power improved for all methods. In all scenarios investigated our approaches maintained a power advantage over established methods. Unlike other methods, the 6DF test is specifically designed to identify differences in DE effect size distributions between genes in the inset and outset, regardless of whether the proportion of DE genes is different. In simulations consisting of a difference in effect sizes, but not in the proportion of DE genes between the inset and outset, the 6DF test was sensitive to this difference, whereas the other methods were not (S6 Fig).

Real data analysis

We tested 347 KEGG gene sets for enrichment of DE between tumour and normal samples using data from 15 cancer types obtained from TCGA [11]. The differential expression analysis and benchmarking of our results was carried out using GSEABenchmarkeR [3]. Our approaches had a far lower runtime than the permutation-based GSEA, while the runtime of ORA was very low in comparison to our approach (S9 Fig). All of our approaches found a higher proportion of significant gene sets than either GSEA, directional GSEA or ORA, before and after correcting for multiple testing (S10 Fig). The low proportion of significant gene sets, after correcting for multiple testing is consistent with results reported in previous benchmarking studies [3]. In order to evaluate the power of our approach we compared the distribution of the nominal p-values of cancer associated gene sets across all cancer types, with the p-values found by established methods (Fig 4). Our approach compared favorably (i.e. tended to have lower p-values for cancer-associated gene sets) with established methods across the four cancer-associated gene sets. We also obtained the rank for each gene set (among the 347 gene sets tested) in each cancer type. All of the gene sets evaluated tended to be ranked higher using our approach than by existing methods (Fig 4A, 4B and 4C), with the exception of the miRNAs in cancer gene set (Fig 4D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Enrichment for cancer-associated gene sets in TCGA data.

Boxplots of log-transformed nominal p-values across 15 cancer types are shown on the left column (lower values indicate more significant results). The rank of the same cancer-associated gene sets is shown in the right-hand column (lower values indicate higher rank). Each row shows the results obtained for the gene set named above the row. A fold-change threshold of 1.5 was used for the MREMA tests.

https://doi.org/10.1371/journal.pcbi.1010278.g004

We examined the ranking of specific disease-associated gene sets in 38 microarray datasets corresponding to these diseases. These datasets have previously been compiled and used to evaluate gene set analysis tools [12] [13] [3]. Our methods performed well on these data, particularly for very highly ranked gene sets (Fig 5). For example 21% of these gene sets were ranked in the top ten using our 1DF and 2DF tests and 16% for the 6DF test; however, only 5% and 8% were in the top 10 using GSEA and and ORA, respectively. The 1DF maintained an advantage over other methods for a wider range of values of the number of top-ranked gene sets considered, but the performance of all methods became closer as the number of top-ranked genes considered became very large (S11 Fig). This was expected, as increasing the number of top-ranked genes to consider allowed more methods to recover the relevant disease-associated gene set, even when it was not highly ranked.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Ranking of disease-associated gene sets in corresponding gene expression experiments.

The proportion of times the disease-associated gene set was included in the x top-ranked gene sets as a function of x. The MREMA tests had higher values of this proportion than established methods when only the most highly ranked gene sets were considered (i.e. low values of x, corresponding to the left-hand side of the plot). The 1DF test maintained this advantage for longer than the other two tests, with GSEA matching the performance of the 1DF test by the right-hand side of the plot. A fold-change threshold of 1.5 was used for the MREMA tests. The position of the disease-associated sets in the full ranking is available in supplementary data (S1 Table).

https://doi.org/10.1371/journal.pcbi.1010278.g005

Gaussian mixture model (GMM)

We model the LFC using a mixture of Gaussian distributions, one with a mean of zero, for non-DE genes, and two with positive and negative means to capture positive and negative DE genes respectively. The LFC distribution is then given by: (1) where μ_l, and π_l are the mean, variance and weight, respectively, of component l. The weights of all components sum to one. As a null hypothesis, we use a shared mixture distribution for all genes, regardless of gene set membership. In the alternative models gene set membership is included in the LFC distribution as follows: (2) where z_i is a binary variable indicating the gene set membership of gene i. This allows the LFC distribution to differ between the inset and the outset. In the alternative model for the 1DF test (S1 Appendix Eq. 29), all genes contribute to the estimate of the means, variance and a parameter, c, which determines the proportion of DE genes that are upregulated in both the inset and outset (S1 Appendix Eq. 30). The weight assigned to the non-DE genes in the inset is estimated using only genes in the set and the weight assigned to the non-DE genes in the outset is estimated using only genes outside the set. This means that while the proportions of DE genes can differ between the inset and the outset the fraction of those DE genes that are upregulated/downregulated is the same. In the alternative model for the 2DF test both the proportion of DE genes and the fraction of those genes that are upregulated/downregulated are estimated separately for the inset and outset (S1 Appendix Eq. 21). In the alternative model for the 6DF test all parameters for the inset distribution are estimated using the genes in the set while all parameters in the outset distribution are estimated using genes outside the set (S1 Appendix Eq. 3).

Our method uses a soft threshold, τ, to define biologically relevant DE. We place the constraint on the parameters of the mixture components corresponding to upregulated and downregulated genes such that at most 0.25 of the area of either component falls in the interval [−τ, + τ]. In each M-step of the EM algorithm a minimum value is set for the mean of the upregulated component to satisfy this condition. If the mean of the upregulated component is estimated to be below the minimum value, the minimum value replaces the estimate. This minimum value is then updated in the next iteration of the EM algorithm. A dynamic maximum value for the mean of the downregulated component is likewise used to replace any estimated mean that leads to 0.25 of the area of that component above τ. The variance of the non-DE component is fixed such that 95% of the area of the component lies within the interval [−τ, + τ].

Mixture random-effects meta-analysis

The GMM described above does not consider the standard error associated with the LFC estimate for each gene. Each estimated LFC comes with uncertainty, in other words each estimated LFC is the sum of the true, but unknown, LFC, and an error term. We assume normally distributed error with mean zero and standard deviation given by the standard error of the estimated LFC (σ_i, for gene, i). Compounding the GMM and the error we obtain the following LFC distribution: (3)

Similar to the previous section, we can build the alternative model by adding the binary indicator for the set membership: (4)

Considering each gene as a single study and interpreting as the within study variance and as between study variance, we see that our model is similar to standard random effect meta analysis except that the underlying distribution of LFC is a GMM instead of a single distribution. To estimate the parameters of the model we use the Expectation Maximization algorithm. In the M-step we use an iterative approach, optimizing successively over the mean and variance parameters of each Gaussian distribution in the mixture model. This is because unlike a simple GMM, the contribution of each gene to the parameters is weighted not only by the posterior probability (calculated in the E-step) but also by a weight whose formula is given by [10]. Additional model and implementation details are provided in S1 Appendix.

Simulations

We simulated RNA-Seq gene expression data for N individuals in two equally sized groups. The genes were divided into S non-overlapping gene sets of equal size. We sampled gene expression counts for each gene, i, from the negative binomial distribution NB(μ_i, φ_i). In order to choose realistic parameters for this distribution, we used a real data set. The mean and the dispersion parameters for all genes in this data set were estimated using DESeq2 and the BRCA-TCGA data. For each gene in the count data matrix, we randomly chose a mean and a dispersion parameter from these values. The expression values for non-DE genes were again sampled from NB(μ_i, φ_i) and expression values for DE genes were sampled from NB(μ_i, φ_i) for the control group. The expression values for the case group was sampled from NB(FCμ_i, φ_i) distribution, with FC the fold change value.

For the main power simulations (Figs 1 and 2), LFC values for the inset and the outset were drawn from a normal distribution with a mean of zero and the standard deviation differing between the inset and the outset. Equal numbers of DE genes were up- and down-regulated in the inset and outset. In each power simulation there was 10,000 genes split into 100 gene sets of 100 genes each. All gene sets were tested for enrichment and p-values were corrected for multiple testing using the Benjamini-Hochberg procedure. We applied a significance threshold of 0.05 on the adjusted p-values. In order to highlight significantly enriched gene sets we imposed a second condition, as well as statistical significance, the estimated proportion of DE genes had to be higher in the gene set than in the background. This ensured that we avoided identifying gene sets that were significantly depleted for DE genes. For the null simulations (S4 Fig) the LFC values of all sets were drawn from the same normal distribution. Each of the 100 gene sets were tested for enrichment. This was repeated 10 times to give 1,000 tests of enrichment where none was present. We also performed simulations with unequal fractions of upregulated/downregulated genes in the inset and outset (for the results shown in S6, S7 and S8 Figs). The LFC values for these simulations were drawn from a mixture of three Gaussian distributions (corresponding to upregulated and downregulated genes and a component centred on zero for non-DE genes). Using DESeq2 [14] we obtained an estimate of the LFC and the standard error of this estimate. Established methods were implemented using the GSEABenchmarkeR package in R [3], with the exception of GSEA(absolute) which was run using the absolute signal to noise ratio as the ranking metric [15]. Both GSEA and GSEA(absolute) were run with 1,000 permutations.

Real data

Gene count data for 15 TCGA paired tumour and normal datsets, and KEGG gene sets were retrieved using the GSEABenchmarkeR package [3]. Differential expression analysis was again carried out using DESeq2. The established methods were implemented as in the simulations. A fold-change threshold of 1.5 was used for the 1DF, 2DF and 6DF tests. The same approach was used for the microarray data, with the exception that the differential expression analysis was carried out using limma [16]. The results of the MREMA tests were ranked as follows for both the TCGA and microarray data. For all gene sets where the estimated proportion of DE genes was higher in the set than the background the ranking was according to p-value (in ascending order). This placed sets with the highest statistical significance for enrichment at the top. Below the enriched sets, gene sets that were assigned a lower proportion of DE genes than the background were ranked by p-value in ascending order. This meant that gene sets that were significantly depleted were ranked towards the bottom.