Compositionality in sequence count data

Compositionality refers to the nature of sequence count data as containing relative abundance information only. In the differential abundance setting, several authors [1–3] have described the problem this poses: whereas researchers would like to interpret change in absolute abundances, compositional effects mean using change in relative abundances as a proxy can lead to false discoveries. A few authors have cited instances of these false discoveries in real data. Coate and Doyle [4, 5] discussed the issue of transcriptome size variation in plants and other systems and the impact of this on accurate transcriptome profiling. Nie et al and Lin et al [6, 7] documented the phenomenon of widespread “transcription amplification” by the transcription factor c-Myc and Lovén et al [8] used c-Myc data and parallel RNA quantification assays to show that substantial differences in total abundance between control and elevated c-Myc conditions resulted in very different interpretations of apparent differential expression.

Common to these studies of transcriptomes is a recommendation that, where feasible, researchers leverage RNA spike-ins as controls against which changes in observed abundance can be scaled [9, 10]. But this practice has fallen short of widespread adoption. While several papers have expressed confidence in the utility of spike-ins [11–15], the doubt cast by reports of widespread batch effects [16] and technical noise [17] have had the effect of reducing researcher confidence in their use. Further, the introduction of spike-ins is not practical on all platforms.

Where approaches that rely on spike-ins are undesirable or infeasible, sample rescaling procedures have proliferated. These methods typically assume the existence of a stable set of features and attempt to normalize compositions in such a way as to recover this stable set across samples. In fact, in transcriptomics, these methods predominate.

In the setting of microbial community profiling, the prevailing assumption is that typical compositions are too simple for rescaling methods to work well (although results in benchmarking studies have been mixed [19, 20]). Competing approaches have been developed for dealing with compositionality in microbial sequence count data. Quantitative microbiome profiling [21] and similar approaches combine relative abundances with complementary measurements of microbial load to reconstruct absolute abundances. In contrast, so-called compositional methods are also utilized. These involve log relative representations which can give approximate log-normality, such that workhorse statistical methods for continuous data may be applied. However, interpretation of these quantities can be challenging (e.g. as with the isometric logratio [22]).

Though there is evidence from simulated and real data that scale—i.e. increasing complexity of composition in terms of numbers of genes, transcripts, or bacterial sequence variants—mitigates the problem of compositionality [2, 20], it remains unclear whether there are instances where it is reasonable to substitute relative abundances for absolute abundances and several fields could benefit from clarity on this issue. In this work, we quantify the discrepancy in differential abundance calling on simulated and real data sets representative of 16S metabarcoding, bulk RNA-seq, and single-cell RNA-seq experiments.

In particular, we evaluate a set of methods which rescale observed total abundances and show that all such rescaling strategies outperform a simple library size normalization in our simulated data. The methods selected are a diverse set meant to be representative but not exhaustive and draw from the fields of transcriptomic and metagenomic analysis: ALDEx2 [23], ANCOM-BC [24], DESeq2 [25], edgeR [26], and scran [27]. Each of these methods rescales the observed counts against a reference quantity, implicit or explicit—typically, a subset of putatively stable features. Where such a stable reference exists, differences in these rescaled counts should approximate differences in absolute counts. In all cases, baseline or “true” differential abundance for each feature in the absolute count data was determined by the use of a simple generalized linear model. This yielded a common standard against which to compare differential abundance calls made by ALDEx2, ANCOM-BC, DESeq2, edgeR, and scran. Details on the simulation procedure and these differential abundance calling algorithms are given in Methods.

In simulations exploring a broad range of differential abundance scenarios, we find median false positive rates of differential abundance calls made on absolute versus relative counts are 11.5% and that false positive counts increase with the proportion of differentially abundant features. The number of features (e.g. genes or bacterial sequence variants) plays little role in observed outcomes. An exploration of sequencing data collected from twelve external studies reveals similar trends. Further, we show that summaries of sparsity and the prevalence of apparent feature-level change can provide reasonable predictions of the amount of discrepancy in differential abundance calls in simulated data and may be useful for bounding expectations of discrepancy in real data sets.

Simulated data

All rescaling methods have higher accuracy than edgeR with library size normalization.

Library size—or total count—normalization represented an anticipated worst case for accuracy in differential abundance testing. In order to test this, we employed edgeR, which allows for either total count or trimmed mean of M-values (TMM) normalization [26]. The difference in accuracy outcomes with and without TMM normalization was substantial. Median specificity was only 0.35 when the observed per-sample totals were normalized by total counts, versus 0.88 when using TMM normalization on the same collection of simulated data sets. False positive counts were over four times higher in the absence of TMM normalization, demonstrating that, for edgeR, a very widely used differential abundance calling method, accuracy is greatly improved by this heuristic per-sample rescaling. These results are illustrated in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. False positive counts as a function of percent differentially abundant features in simulation.

Counts of false positives as a function of the percent of simulated differentially abundant features for six methods. edgeR with no heuristic rescaling is shown at top. Columns segregate simulations with increasing numbers of features. All rescaling methods have higher overall accuracy than edgeR without rescaling. Counts of false positives tend to be highest where a majority of features are differentially abundant.

https://doi.org/10.1371/journal.pcbi.1010284.g002

Moderate specificity was typical but cases of low specificity were observed in all settings for all methods.

Median specificity was moderately high at 0.885—lowest for ANCOM-BC in the Cell Transcriptomic setting at 0.835 and highest for ALDEx2 in the same setting at 0.937 (see Table 1). However, a minority of simulated data sets yielding very large false positive rates were observed in every setting. Data sets with very low specificity (< 0.5) made up less than 10% of simulated cases in the Transcriptomic settings and more than 14% in the Microbial setting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Median and thresholded specificity by simulated setting, FDR ≤ 0.05.

Low specificity is a common outcome in simulated data. A majority of simulated data sets have false positive rates in excess of 5%. A minority of data sets have false positive rates in excess of 50%.

https://doi.org/10.1371/journal.pcbi.1010284.t001

Though scran was the top performer, outcomes were similar across methods.

Scran had the highest overall accuracy with a median sensitivity of 0.93 and specificity of 0.91. Sensitivity was lower in ALDEx2 (median = 0.78) and specificity was lower in DESeq2 (median = 0.86). All in all, methods exhibited similar performance in terms of their distributions of sensitivity and specificity across settings. See Figs 1 and 2 and Tables 1 and 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Median and thresholded specificity by simulated setting, FDR ≤ 0.01 plus effect size thresholding.

Specificity in simulated data under more stringent conditions: here differential abundant is called using FDR ≤ 0.01 and an effect size threshold (a fold change of at least 2-fold).

https://doi.org/10.1371/journal.pcbi.1010284.t002

Next, we explored the characteristics of relative abundances associated—either positively or negatively—with observed sensitivities and specificities in simulated data. It is possible to imagine characteristics which might indicate the presence of distortion by compositional effects, for example, an increase in the percent of rare features from one condition to the next (in effect, dropouts). While we might not expect any single such characteristic to predict the sensitivity or specificity of calls made on relative versus absolute abundances, composites of such characteristics might be informative. We describe the strongest associations we observed below.

Specificity is strongly anti-correlated with the estimated proportion of differential features.

The proportions of features undergoing large fold decreases or large fold increases in abundance relative to the mean were highly informative with respect to specificity (ρ = −0.61 and ρ = −0.43 respectively). The standard deviation of the change in log counts between simulated conditions was also anti-correlated with specificity (ρ = −0.56). In each case, these characteristics supplied evidence of the existence (or lack) of widespread change in composition. Methods which rescale observed abundances rely on a pool of stable reference features against which to estimate per-sample “scaling factors.” The larger the number of components in the composition apparently changing, the greater the extent to which this rescaling is impaired.

We utilized these and over 50 additional features derived from relative abundance data to train per-method models of both sensitivity and specificity, with the aim of predicting the discrepancy in relative versus absolute differential abundance calls from features of observed data alone, reasoning that these “signatures” might be highly informative. All model features are outlined in S2 Table and feature importance is explored in S3 and S4 Tables. Predictive models were trained on 80% of our simulated data and their performance was evaluated on the held-out 20% of simulated data sets. Predictions of specificity were more accurate than predictions of sensitivity (mean specificity R² = 0.70; mean sensitivity R² = 0.59). Prediction of outcomes was most successful for ALDEx2, where the R² values for sensitivities and specificities on held-out data were 0.74 and 0.72, respectively. Full results are given in Table 3. These results suggested it might be reasonable to predict discrepancy between differential abundance calls made on absolute versus relative abundances using characteristics of the relative abundance data alone. However, the study of some representative real data sets will suggest prediction may be much more difficult in this setting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Predictive model performance summaries.

Performance of per-method random forest predictive models of sensitivity and specificity on held out data sets.

https://doi.org/10.1371/journal.pcbi.1010284.t003

Real data

Next, we examined a variety of real data sets across many experimental settings in order to explore outcomes in real data. We collected publicly available data from twelve studies [28–39] and attempted to reconstruct absolute abundances by normalizing observed total sample abundances against reference quantities provided in the same published materials. In most cases, these reference quantities were external RNA spike-in sequences. In others, reconstructed absolute abundances had already been estimated, as in [28] through quantitative microbiome profiling (QMP [21]). In one case [38], we normalized against the housekeeping gene gapdh [40] and in another [33], against paired measurements of cell mass. Visual and textual summaries of these data sets are available in Figs 3 and 4 and Table 4. While not exhaustive, these studies illustrate a wide range of experimental conditions. We acknowledge the difficulty in reconstructing absolute abundances and caution that these estimates are likely noisy. Rather than call them absolute abundances, we will adopt the helpful terminology of [9] and call these reconstructions “nominal abundances” to distinguish them from some theoretical ground truth. While these quantities are an abstraction, since they derive from real experiments, we believe they capture some of the real variation in composition and scale we could expect to see in data from typical experiments and should serve to ground expectations. As with simulated data, we proceed by comparing differential abundances called on the relative count data (via ALDEx2, ANCOM-BC, DESeq2, edgeR, or scran) with differential abundances called on the nominal abundances—our proxy for “true” differential abundance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Visual summaries of real data sets 1–6.

Visual summaries of nominal abundances (left panels) and relative abundances (right panels) for a) Hagai et al. b) Hashimshony et al. c) Song et al. d) Monaco et al. e) Vieira-Silva et al. f) Barlow et al. Features (genes or bacterial sequence variants) with at least 1% relative abundance across all samples are colored; all other features are gray.

https://doi.org/10.1371/journal.pcbi.1010284.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Visual summaries of real data sets 7–12.

Visual summaries of nominal abundances (left panels) and relative abundances (right panels) for g) Grün et al. h) Muraro et al. i) Kimmerling et al. j) Yu et al. k) Owens et al. m) Klein et al. Features (genes or bacterial sequence variants) with at least 1% relative abundance across all samples are colored; all other features are gray.

https://doi.org/10.1371/journal.pcbi.1010284.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Overview of real data sets. Real 16S metabarcoding, bulk RNA-seq, and single cell RNA-seq data sets corresponding to the abundances shown in Figs 3 and 4. The estimated percent differential features are those significantly differential to a negative binomial GLM in the nominal abundances.

https://doi.org/10.1371/journal.pcbi.1010284.t004

We have endeavored to include among the real data sets some of the most challenging possible cases for differential abundance calling from relative count data. Many of the data sets we have selected involve a large amount of absolute and/or compositional change across experimental or biological conditions. These data sets help us answer the questions: What is the scale of discrepancy in the worst cases? How useful are summaries derived from relative abundances in these cases? Could a researcher reasonably predict error from relative abundance data alone?

Accuracy outcomes for these real data sets are given in Figs 5 and 6 and S5 and S6 Tables. These outcomes can be broadly classified on the basis of sensitivity into low, moderate, and high sensitivity cases, described below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Results for real data sets 1–6.

Results summaries for a) Hagai et al. b) Hashimshony et al. c) Song et al. d) Monaco et al. e) Vieira-Silva et al. f) Barlow et al. Barplots on the left indicate counts of true and false positives (TP, FP) and true and false negatives (TN, FN). Boxplots on the right indicate the 50% (box) and 90% (line) intervals of prediction and points, the observed values of either sensitivity or specificity.

https://doi.org/10.1371/journal.pcbi.1010284.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Results for real data sets 7–12.

Results summaries for g) Grün et al. h) Muraro et al. i) Kimmerling et al. j) Yu et al. k) Owens et al. m) Klein et al. Barplots on the left indicate counts of true and false positives (TP, FP) and true and false negatives (TN, FN). Boxplots on the right indicate the 50% (box) and 90% (line) intervals of prediction and points, the observed values of either sensitivity or specificity.

https://doi.org/10.1371/journal.pcbi.1010284.g006

Mitigation strategies

We pursued two mitigation strategies which we believed could improve the accuracy of differential abundance estimates from relative counts: 1) increased stringency of differential abundance testing and 2) normalization against stable reference features with DESeq2.

In the first case we both increased the significance threshold for differentially abundant features (FDR ≤ 0.01) and enforced a fold change (effect size) threshold for differential features. In essence, we now ask about the agreement between absolute and rescaled relative data in terms of big, unambiguous differences. In this setting, median specificity was improved from 0.90 to 0.95 and from 0.63 to 0.85 in the subset of data sets with a majority of differentially abundant features. For full results, see Table 2 and compare Figs 1 and 2 with S7 and S8 Figs respectively.

In a second analysis, we briefly explored the effect of normalization by stable reference features. DESeq2 allows a user to specify a set of reference features against which samples will be normalized using a “control_genes” option. We utilized this feature in three settings: 1) normalizing simulated data sets against low-variance features identified in absolute counts, 2) normalizing real data sets against low-variance features identified in absolute counts, and 3) normalizing real data against a set of 13 “housekeeping genes” pulled from the literature including gapdh and hprt1 [40–43].

Results derived from low-variance gene normalization show mostly modest improvements in specificity in both simulated and real data (S7 and S8 Tables; S9 Fig). For example, median specificity was increased by over 15% in data sets with an initial “high” FPR of greater than 10%. This improvement was progressive: real data sets with the worst specificities showed the greatest improvement, from 0.653 to 0.902 (Song et al.), 0.830 to 0.896 (Monaco et al.), and 0.853 to 0.906 (Muraro et al.; S8 Table). The housekeeping gene set was less effective and even increased discrepancy in two data sets, suggesting that, although normalization by control features can improve accuracy substantially, caution should be employed here: strong biological prior knowledge or independent validating measurements may be required if researchers wish to propose stable genes for sample normalization.

Simulation model

We designed a simulation framework to generate count data corresponding to two arbitrarily different conditions, denoted by superscripts in the equations below. First, for p = 1, …, P features in the first condition, a set of log mean abundances was drawn as (1) where hyperparameters m and S tune the mean and standard deviation of baseline log abundances. A correlation matrix was drawn as (2) where scale matrix Q was supplied as either the identity matrix (for a minority of simulations) or a dense correlation matrix with net positive elements. The matrix Ω is subsequently re-scaled to a correlation matrix and used to draw correlated feature perturbations in a second condition as (3) where the hyperparameter a exists in order to tune the overall scale of the correlated log perturbations. Mean abundances on the scale of sequence counts for each condition are calculated as (4)

Differentially abundant features in some desired proportion, c, are obtained as follows: features are selected as differentially abundant with probability c. For those selected features only, the perturbed serves as the mean abundance in the second condition; for all other features, the mean abundance in both the first and second conditions is given by . Let these new vectors be . These represent mean the abundances of P features in two conditions, some of which differ across conditions, others of which are identical. Replicates i = 1, …, 10 are then generated for each condition as follows. A fixed dispersion parameter for absolute counts is defined as d_abs = 1000 and those counts are drawn as (5) where (6) (Note that the dispersion parameter has been chosen such that the resulting counts are only barely overdispersed with respect to a Poisson.) The purpose of the truncated, per-sample multiplier δ is to re-scale all abundances within a given sample by some factor of approximately 1 but by increasing the scale of hyperparameter g, increasing replicate noise can be added within a condition. This process is repeated for the second condition to give a set of absolute counts .

A new average observed total abundance (or library size) is randomly drawn as (7)

Finally, observed abundances z are generated through a multinomial resampling procedure similar to that of [10, 20, 23] which gives relative abundances as counts per million. Where i and k index different samples prior to resampling, i′ indexes the sample i after resampling, and total counts for sample i prior to resampling are given by n_i = ∑ y_i, we have (8) where superscripts have been suppressed as this procedure is identical across simulated “conditions.” The resulting P-length vector of counts for a given sample contains relative but not absolute abundance information. These vectors are collapsed into a P × 20 count matrix containing 10 replicate samples for each of two simulated conditions. In order to evaluate the discrepancy of differential abundance calling on observed versus absolute counts, we apply differential abundance methods to count matrices Z and Y respectively and score the differences.

Breadth of simulations

In order to generate simulations with a wide variety of characteristics, we swept in a grid over all our hyperparameters. Feature number P was stepped through values 100, 1000, and 5000. A maximum feature number of 5000 was chosen as these simulations were computationally intensive and major trends had become apparent at that scale. The degree of feature correlation was encoded in five realizations of scale matrices Q, encoding fully independent features at one extreme and 50% strongly positively correlated features at the other extreme. Log mean abundance (m) and the log variance (S) were independently incremented through low to high values. Likewise, the average log perturbation size (a) was swept from low to high in five steps, as a proportion of log mean abundance.

Replicate noise g varied from low to high in three steps. And finally, the proportion of differentially abundant features ranged across 20%, 30%, 50%, 70%, and 90%. Note that because many “perturbations” were very small, detectable differential abundance was generally only a fraction of the parameterized amount and most data sets contain a minority of differentially abundant features. Overall this 5625 simulated data sets were generated with almost continuous variation characteristics of interest.

We suggest that ranges of these parameter settings approximately represent different data modalities. We term the Microbial setting that with low to moderate feature number (P ≤ 1000) and largest average perturbation, in accordance with a belief that bacterial communities are often simple (in terms of sequence variants with more than negligible abundance) and that they are highly variable even at short time scales [44].

We designate the Bulk Transcriptomic setting as that with the largest feature number (P = 5000) and having a lower average perturbation, the rationale being that transcriptomes sampled in aggregate over many cells are complex but largely stable compositions. Similarly, we define the intermediate Cell Transcriptomic setting, approximately representative of single-cell RNA-seq data, to comprise simulations with moderate to large feature numbers (P ≥ 1000) and moderate perturbation sizes. These categories are intended as rough outlines and we note that within these settings the realized data varies in terms of 1) degree of feature correlation, 2) overall abundance, 3) (un)evenness of composition, and 4) within-condition variation in totals.

Calling differential abundance

Five differential abundance calling methods were used in this study. These were ALDEx2 v1.22.0 [23], ANCOM-BC v1.0.5 [24], DESeq2 v1.30.1 [25], edgeR v3.32.1 [26], and scran v1.18.5 [27]. Each of these methods relies upon the use of a reference quantity to renomalize sample total abundances. ALDEx2, DESeq2, and edgeR rely on various (trimmed) estimates of the per-sample mean or median abundance for this reference quantity. ANCOM-BC uses a model-derived estimate of the “sampling fraction” of total abundance represented in an observed sample. scran’s procedure learns and applies a condition- or cell type-specific normalization factor. For simplicity, we omit from consideration models which lack a rescaling. We also omit zero-inflation models. Although these are popular in single-cell mRNA sequencing data, the debate continues about whether these models are appropriate for these data [45, 46].

Finally, differential abundance calls made on observed counts must be evaluated against a reference in order to calculate discrepancy. For this reference, we used calls made by a negative binomial generalized linear model on absolute abundances as a pseudo-gold standard or “oracle” in all cases. Features with significantly different means according to the GLM were considered “true” instances of differential abundance. One caveat is that some of the discrepancy in calls between absolute and relative abundances will be due to differences in sensitivity between the models applied to the reference and “observed” data sets—i.e. between a stock NB GLM and DESeq2. In all settings, true positive, true negative, false positive, and false negative calls were manually spot checked to verify disagreements between methods were generally unambiguous. Accuracy (in terms of sensitivity and specificity) was calculated between differential abundance calls made on absolute abundances and observed abundances. Unadjusted p-values were collected from all methods and multiple test correction applied via p.adjust in R using Benjamini-Hochberg method.

Predictive modeling

In total, we trained 10 random forest models over 61 summary features, for each combination differential abundance calling method (ALDEx2, ANCOM-BC, DESeq2, edgeR, or scran) and accuracy measure (sensitivity or specificity). All such predictive models were fit with the randomForest package in R [47]. A random forest is an ensemble of decision trees and this tree-based approach was chosen because, while extensive, our simulations were not exhaustive. We anticipated that learning sets of decision rules might generalize well to unseen conditions, in particular feature numbers larger than those we explored in simulation. Feature importance was measured as “gain,” or the relative increase in predictive accuracy achieved by the inclusion in the model of a given feature, as computed by the caret package in R [48].

Predictive models built from these summary features attempted to estimate sensitivity and specificity values explicitly. Details on these models are given in the Methods. All models were trained on 80% of the simulated data and their predictive accuracy was assessed on the reserved 20%.

Real data processing

Publicly available data was downloaded from sources provided in the published materials for the studies cited. In general, relative abundances were converted to counts per million and sequences for external spike-ins extracted from these counts. Simple per-sample scaling factors were calculated from mean spike-in abundances and applied to the relative data to give “nominal” abundances.

In the inhibited and control cells of Klein et al., the expression of housekeeping gene Gapdh was used to normalize per-sample total abundance. In the 16S metagenomic data of Barlow et al. and Vieira-Silva et al., nominal abundances had already been estimated by the authors of those studies using quantitative PCR-based methods. In the single-cell expression data of Yu et al., log₂ observed total abundances correlated well with log₂ spike-in abundances and the observed data themselves were treated as true abundances. Relative abundances were derived by normalizing per-sample library sizes and scaling to give counts per million. Finally, in the coupled cell mass and expression measurements of Kimmerling et al., the estimated cell mass was used as scaling factor for observed expression to give nominal abundances.

Nominal and relative abundances were then filtered to exclude features (genes or bacterial sequence variants) present at an average abundance of less than a single count in either the nominal or relative count tables. This both reduced the size of the largest data sets—making them more computationally manageable—and reduced sparsity in the most extreme cases.