1. Using uncorrected p-values for statistical significance

Enrichment tests generate p-values (probability values) between 0 and 1. The p-value estimates the probability of an observed enriched category occurring by random chance. A low p-value (e.g., p < 0.05) suggests the observed result would be unlikely from random data, suggesting a real effect. However, as gene set libraries can contain thousands of categories, we could expect 5% of these to meet the p < 0.05 threshold with random data. Therefore, we almost always get many “significant” results just by chance [4]. Our previous literature study showed that this problem was prevalent in 43% of FEA articles [16].

There are a variety of p-value correction methods to reduce the risk of false positives [12], including approaches from Sidak [17], Holm-Bonferroni [18] and Benjamini–Hochberg [19]. The Benjamini–Hochberg method, also called the false discovery rate (FDR) method, appears to be the most widely used in genomics to adjust p-values, but it has been critiqued as being overly conservative when a larger fraction of tests are not null [20].

Our simulation analysis identified a mean of 10.4 Reactome pathways as significant (p < 0.01) from randomly selected foreground genes when p-value correction was not implemented; this was reduced to 0.02 pathways, after FDR correction (S1A Fig). Our example analysis indicated that ~45% of pathway enrichment results could be false if correction of p-values is not conducted (S1B Fig).

To avoid unacceptable false positives, use a tool that provides adjusted significance values like FDR. P-value adjustment can also be done separately with other tools like Stata, SPSS, GraphPad and R. A stricter FDR significance threshold, like 0.01, has been shown to be effective in reducing false positives as compared to 0.05 [21].

2. Wrong background gene list

Defining a background list (a.k.a. universe or reference) is crucial because genes that have no chance of being part of the foreground, i.e., undetected genes, should not contribute to enrichment calculations as they will inflate significance values [4,12].

Every type of omics analysis has its limitations. Microarrays can only measure genes they are designed to assay. RNA-seq has certain genes that are poorly detected as a result of sequence similarity or GC bias. Biological differences also play a big part in what is detected [15]. From 78k human genes annotated in Ensembl’s latest release (v115), typically only 12–20k are expressed at detectable levels with RNA-seq in any one tissue.

The severity of this problem is contentious, but our previous analysis of seven RNA-seq studies suggested that using the wrong background could lead to false positive rates of 60%–80% (FDR < 0.05 was used) [16].

The example analysis with a background of detected genes identified 207 significant pathways, but a background of all annotated genes led to an additional 601 false-positive pathways (S2A Fig).

Recommendations for selecting a detection threshold and defining a background list are given in Box 1.

Microarray gene expression and DNA methylation: Discard known problematic probes. Include all genes with probes that pass quality control filtering.

Genomics (e.g.,: variant searches by whole genome or exome sequencing): All annotated genes could be used unless there are reasons to believe that some are not detected (e.g., some genes might not be included in the exome enrichment process, extreme GC content, etc.). This can be checked using sequence depth tools.

3. Using a tool that does not report enrichment scores

FDR values can tell us whether an observation is statistically significant, but it does not inform whether it is biologically impactful [22,23]. For that, we need some measure of effect size. In FEA, we can use an enrichment score as a surrogate measure of effect size. For rank-based tools like GSEA, the enrichment score varies from -1 to +1, denoting the distribution of genes in a gene set relative to all other genes [10]. For a gene set composed of 15 genes, a score of 1.0 would mean that these 15 genes are the top 15 upregulated, while if a value is close to 0, it means the distribution of genes is close to what you might get by random chance. For over-representation methods like DAVID, the fold enrichment score is often quoted, which is the odds ratio of genes of a gene set in the foreground list as compared to the background [7]. Unfortunately, many common tools do not provide enrichment scores (e.g., clusterProfiler and g:Profiler), which leaves researchers with no information about their effect sizes. Tools that provide enrichment scores include ShinyGO (web) [24], GSEA [10] and fgsea (fora) [11].

4. Prioritising results solely by p-value

FEA can return hundreds of significant results, which can be confusing to interpret. Many tools by default will sort the results by significance, but this can result in missing the most interesting results. As p-value prioritisation emphasises generic functions with large gene sets and moderate fold changes, there is a risk of overlooking smaller gene sets with larger fold changes (contrast Tables B and C in S1 Supporting Information). Smaller and more specific gene sets with larger magnitude enrichment scores are generally better candidates for downstream validation due to their explanatory power.

To avoid this problem, end users should also do enrichment score prioritisation, by first removing pathways above the FDR threshold (e.g.,: 0.05 or 0.01) and then sorting by enrichment score magnitude.

5. Foreground lists that are too large or too small for ORA

It is a common misconception that only differentially expressed genes that meet the FDR threshold should be submitted to an enrichment test. Tarca and colleagues suggest a heuristic that selects the top 1% of genes if there are none that meet the standard significance cut-off [25]. If proper FDR control of enrichment results is applied (See #1 above), then gene selection criteria can be flexible. The caveat is that enrichment tests (like the hypergeometric method) have size ranges of input gene lists that work best. If the number of foreground genes is too large, then the enrichment scores will not be as large or interesting, but if the foreground is too small, then the overlap with pathways will be small and fail to reach statistical significance.

Our testing suggests that a gene list size of 700–900 genes, or 5%–9% of all those detected would be optimal for a differential expression study (S4 Fig). To achieve this number, thresholds for significance or fold change filtering can be fine-tuned. Nevertheless, some users may want to avoid setting seemingly arbitrary thresholds—in that case, using an FCS method like GSEA instead that calculates enrichment from all detected genes is recommended.

6. Not running ORA separately on up- and down-regulated genes

In some articles, we noticed that authors did not conduct separate ORA tests for up- and down-regulated gene lists, instead opting to submit the combined list for ORA. This is not necessarily an error, as it tests the hypothesis that some pathways are “dysregulated”; a mix of up- and down-regulated genes which appear at an elevated rate. However, the results from the “combined” and “separate” approaches are very different.

The example dataset showed the combined approach identified 82% fewer significant results as compared to the separate approach (S5 Fig). There were no enriched pathways specific to the combined test.

The reason behind this is 2-fold. Firstly, we know that genes in the same pathway are typically correlated with each other [26]. Consider cell cycle genes, or genes responding to pathogens, which are activated in unison to coordinate a complex biological process. In a typical differential expression experiment after a stimulus, this results in pathways that are predominantly up- or down-regulated, but rarely a mix of up and down. Due to this phenomenon, the up and down lists each have relatively strong enrichments, but they are diluted when combined [27]. Based on this, ORA users should use both the combined and separate approaches if directional information is available (some omics types do not).

7. Using shallow gene annotations

One of the most important decisions for FEA is selecting the pathway or ontology database to query. There are many options to consider, both proprietary and open source. When choosing, users should consider whether the database contains the gene sets that they a priori suspect will be altered. Secondly, consider the breadth and depth of the pathway library; this will be where the unexpected discoveries may occur and it pays to use a comprehensive library to capture as many aspects of the dataset as possible.

The example analysis showed that using a larger pathway database like Reactome or Gene Ontology Biological Process results in richer results as compared to smaller databases like KEGG (Table D in S1 Supporting Information).

Using a smaller database like KEGG may be justified based on a priori hypotheses, but in most cases where the goal is discovery of novel themes, a larger pathway database would be recommended. Users should be aware that these larger databases have some degree of redundancy which can be confusing to interpret [28].

8. Using outdated gene identifiers and gene sets

Data repositories like Gene Expression Omnibus (GEO) [29] contain thousands of previously published data sets that we can reanalyse with new pathway databases and software tools to gain further insights. However, when the data is several years old, we should use it with caution, as many gene names may have changed. For example, Illumina’s EPIC DNA Methylation microarray was released in 2016, and in the following eight years, 3,253 of 22,588 gene names on the chip changed (14.4%) [30]. Therefore, these genes would not be recognised by the FEA software. To update defunct gene symbols, the HGNChelper R package can help [31], also having the benefit of fixing gene symbols corrupted by Excel autocorrect, which are unfortunately common in GEO [32]. Persistent gene identifiers like Ensembl (e.g., ENSG00000180096) and HGNC (e.g., HGNC:2879) are less likely to change over time and are therefore preferable over gene symbols (e.g., SEPTIN1) for FEA.

FEA users should also understand how well updated their preferred pathway databases are. Actively updated databases like Reactome [33] constantly increase in size as annotation consortia continue assimilating functional information from the literature (See S6 Fig). The version of KEGG available on MSigDB has not changed since 2010, but versions available through the KEGGREST API and DAVID Knowledgebase appear to be regularly updated. A regularly updated database is likely to lead to richer and more relevant FEA findings [34].

9. Bad presentation

Bad presentation of data is not exclusive to pathway enrichment, but there are a few key mistakes that should be avoided:

1. The number or proportion of selected genes in a category is sometimes shown as evidence of enrichment, but this can be misleading because it does not take into consideration the frequency of these genes in the background list. Enrichment scores and adjusted p-values are better for this purpose.
2. Presenting enrichment results as a pie chart is not recommended because it is not possible to show enrichment scores and significance values in this form. Bubble or bar charts are better alternatives.
3. Sometimes a network of genes or pathways are shown, but the significance of nodes and edges are not described.
4. Figures missing key elements such as axis labels.
5. FEA mentioned in the abstract but no data shown in the main article or supplement.
6. Confusion around which tool was used for each figure and panel.

Such misinterpretation and data presentation problems can also occur when a tool is used without understanding the statistical basis of inference [35], so it is crucial that users take the time to familiarise themselves with the tool’s documentation and recommendations.

10. Neglecting methods reproducibility

According to Goodman and colleagues [36], methods reproducibility is:

“the provision of enough detail about study procedures and data so the same procedures could, in theory or in actuality, be exactly repeated.”

There are several crucial pieces of information required in order to enable reproducibility of FEA including;

• how genes were selected or scored—especially whether up- or down-regulated genes were considered separately or combined,
• the tool used, and its version,
• the options or parameters applied,
• the statistical test used,
• the gene set/pathway database(s) queries, and their versions,
• for ORA, how a background list was defined, and,
• how p-value correction was done [14,37].

A systematic literature analysis published in 2022 found insufficient background list description in 95% of articles describing ORA tests, and p-value correction was insufficiently described in 43% of articles, suggesting that FEA generally suffers from a lack of methods reproducibility [16]. Examples of poor and good methods reproducibility are provided in S1 Supporting Information, together with an AI prompt that users could use to assess their Methods sections.

In addition to including the methodological details mentioned above, authors could also provide gene profile data and/or gene lists used for FEA as supplementary files, or better still, provide full reproducibility and transparency with the five pillars framework [38].