Primary objectives, research questions and hypotheses.

Aim 1: Assess the performance of 22 differential abundance tests on simulated data, corresponding to 38 experimental 16S microbiome sequencing data templates.

Research question: Which differential abundance testing methods among the selected 22 methods provide the most accurate results in terms of sensitivity and specificity for a prespecified 5% threshold for the false discovery rate?

Hypothesis: Certain differential abundance tests outperform others in terms of sensitivity and specificity.

Secondary objectives, research questions and hypotheses.

Aim 2: Assess how the performance of 22 differential abundance tests depend on sparsity of the dataset, effect size and sample size

Research question: How do changes in dataset sparsity, effect size, and sample size impact the sensitivity and specificity of differential abundance tests for a given 5% threshold for the false discovery rate?

Hypothesis: Performance advantages of differential abundance tests significantly depend on the sparsity of the data, effect size and sample size.

Exploratory objectives and research questions.

Aim 3: Identification of data characteristics that are predictive for performance advantages of specific differential abundance tests.

Research question: Which data characteristics are predictive for performance advantages and can be used to predict methods with beneficial performance?

Hypothesis: There are data characteristics which can be used to predict performance advantages.

Aim 1 (Primary).

Based on 38 experimental 16S microbiome sequencing data templates, simulated datasets are generated. The experimental data templates originate from diverse environmental contexts such as the human gut, soil, wastewater, freshwater, plastisphere, marine and built environments. These templates were selected because they were previously used as benchmark datasets in Nearing et al. for complementary benchmarking analyses. Moreover, we used these datasets for simulating data to validate the results of Nearing et al. [1] and for studying the capabilities of simulated data, to reflect characteristics of their data templates realistically. These 38 experimental data templates exhibit a broad spectrum of data characteristics, including varying sample sizes ranging from 24 to 2296 and feature counts from 327 to 59,736. A detailed overview of dataset characteristics for these templates is provided in Supplement 2.

Three simulation tools, metaSPARSim [3], sparseDOSSA2 [4] and MIDASim [5], are employed to generate for each data template 10 simulated dataset, closely mimicking the original characteristics. This is achieved by calibrating the simulation parameters for each data template individually. If a tool underestimates the proportion of zeros, an appropriate proportion of zeros will be added as suggested in [2]. Then, the two simulation tools that best reflect experimental data templates are chosen for subsequent analyses. The quantitative number to judge at this point is the total number of rejected equivalence tests over all datasets and data characteristics [2].

A known ground truth in the simulated data with regulated as well as not regulated taxa is integrated as described in the following. The simulation parameters for both tools are calibrated three times: 1) To obtain parameters that correspond to no difference between the two groups of samples, calibration is done using experimental data for all samples jointly, i.e., independent on the group information. 2)+3) To obtain parameters, that are different in the two groups, calibration is done in each group separately, i.e., the simulation tools are calibrated to the samples belonging to the first group, and calibrated to the samples belonging to the second group.

To obtain both, differentially abundant taxa and taxa with the same expectation in both groups, these calibrated simulation parameters 1) and 2)+3) are merged by estimating the proportion of differentially abundant taxa from the distribution of p-values, and then randomly draw differential abundant taxa. In more detail, the following procedure will be applied:

1. All DA methods are applied to the experimental data templates for calculating p-values
2. To obtain one p-value for each taxon and treat all DA methods equally, a p-value of a randomly selected DA method is assigned
3. The proportion pi0 of not differentially abundant features is estimated by the pi0est function in the qvalue R-package [6]. This function estimates the proportion of true null hypotheses, i.e., those following a uniform distribution [7].
4. Simulation parameters are calibrated (a) for all samples and (b) for each group separately
5. For the proportion pi0, simulation parameters are chosen from calibration 1), and for the proportion (1-pi0), simulation parameters are taken from calibration 2)+3). Differentially abundant features with simulation parameter from 2)+3) are drawn via ) to ensure that taxa with smaller p-values are more likely assigned to be differentially abundant.

Therefore, the population for aim1 is composed of simulated 16S microbiome sequencing data, for which the number of truly differentially abundant features is known.

Aim 2 (Secondary).

Here, additional variations for each of the simulations are generated by modifying sparsity, effect size, and sample size, such that each ranges from low over medium to high. Therefore, the population for aim 2 is composed of simulated 16S microbiome sequencing data settings, for which the number of truly differentially abundant features is known.

For a realistic range of sparsity, effect size and sample size in the simulated data, the minimum, median and maximum value for each of these properties are taken from the 38 experimental data templates and used as lower and upper bound in the simulated data. As an example, the minimal sample size is (Human - IBD), the median is and the maximal sample size is (Freshwater - Arctic). Thus, we simulate data for these three sample sizes.

The effect size is quantified using PERMANOVA R-square [8] as implemented in the adonis2 function of the vegan R-package [9]. First, the minimal and maximal R-squared values are calculated over all templates. In order to adjust the effect size to minimal, median and maximal values calculated from all templates, first the relationship between a fold-change multiplication factor F and PERMANOVA R-square [8] is evaluated. For this purpose, for each data template one simulated dataset is generated with F = 0, 0.1, 0.2, …, 1, followed by the calculation of the respective PERMANOVA R-square [8] value. The appropriate factor F for each template and each desired R-square value is chosen by linear interpolation of these 11 points. If this procedure yields multiple solutions because the relationship may not be monotone, the average of the resulting Fs is used.

For metaSPARSim [3], these fold-parameters are the ratios between the two estimated intensity parameters in both groups for each taxon. For sparseDOSSA2 [4], the fold-parameters are the ratios between the two parameters mu in both groups for each taxon.

Aim 3 (Exploratory).

For aim 3 the same datasets as for aim 2 are used

Visualization of the dataset selection process and its results in the form of a flowchart.

General benchmark setup/design.

Our study is an exploratory benchmark study based on simulated data. The overall workflow is summarized in Fig 4. Since the taxa that were simulated as differential abundant are known, we can calculate the sensitivity and specificity for each significance level α. We use the partial area under the receiver operator characteristic curve (pAUC) and calculate ranks(pAUC) over all 22 DA methods to compare their performance.

Studied methods and collected measures (incl. rationale).

The studied methods are the same 22 differential abundance (DA) tests that were used in Nearing et al. [1], but as implemented in the latest R version: ADAPT [10], ALDEx2 [11], ANCOM-BC2 [12], corncob [13], DESeq2 [14], distinctTest [15], edgeR [16], fastANCOM [17], glmmTMB [18], LEfSe [19], limma voom (TMM) [20], limma voom (TMMwsp) [20], linDA [21], MaAsLin2 [22], MaAsLin2 (rare) [22], MaAsLin3 [23], metagenomeSeq [24], t-test (rare), Wilcoxon test (CLR), Wilcoxon test (rare), ZicoSeq [25], and ZINQ [26]. Significant features will be determined using a 0.05 threshold for adjusted p-values (Benjamini-Hochberg correction).

The collected performance measures are sensitivity and specificity for each differential abundance test.

Employed validation procedure/technique.

N/A.

Parameters and/or configurations for each method.

We implemented the 22 DA methods and tested their applicability. The following code summarized the configurations that will be used in the study:

ADAPT (Analysis of Microbiome Differential Abundance by Pooling Tobit Models):

ANCOM-BC (Analysis of Compositions of Microbiomes with Bias Correction):

ALDEx (ANOVA-Like Differential Expression tool for high throughput sequencing data):

Corncob (Count Regression for Correlated Observations):

distinctTest (differential analyses via hierarchical permutation tests) REF:

EdgeR:

fastANCOM:

glmmTMB:

Limma voom TMM:

Limma voom TMM:

LEfSe (Linear discriminant analysis Effect Size):

linDA:

MaAsLin2 (Microbiome Multivariable Association with Linear Models):

MaAsLin2 rare:

MaAsLin3:

metagenomeSeq:

T test:

T test rare:

Wilcoxon rare:

Wilcoxon CLR:

ZicoSeq:

ZINQ:

To guarantee that FDR calculations coincide for all DA methods, we perform FDR adjustments manually for each DA method and each analyzed test via p.adjust(pvalues, method = ”BH”).

Altogether, we use the same configuration parameters (such as the threshold 100 for minimal library size of a sample) as chosen in Nearing et al. [1] to ensure comparability of the outcomes with the following main exceptions:

• Instead of conducting rarefication as a first step, we will apply the rarefy step directly before the DA method is called to be able to calibrate the simulation tools before
• Instead of using a custom script for ANCOM-II, we will use the ANCOM-BC2 R package that was published after the study by Nearing et al. [1].
• For LEfSe, we use p-values instead of the score. By default, LEfSe outputs only p-values for taxa, that are not filtered out according to three cutoff configuration parameters (kw_cutoff, lda_cutoff, wilcoxon_cutoff) used for filtering. Unfortunately, relaxing those cutoffs to values where all p-values are returned can lead to failure of LEfSe in rare cases. In [2] we chose a strategy where those cutoffs are iteratively relaxed, i.e., we iteratively changed them towards default values if LEfSe fails with relaxed cutoffs.. To better reflect real-world usage in this study, we decided to stick to single value for those cutoffs, if failure of LEfSe occurs in <=10% of the analyzed datasets. In case failures occurs in > 10%, we will apply the iterative changes as in [2]. In any case, we will calculate FDRs from the (possibly filtered) p-values, independently on violation of mathematical assumptions since this would presumably also be done in real-world applications. A more detailed justification of this procedure can be found online in our responses to reviewer comments. Taxa that do not fulfill all cutoffs will have NA as p-value.

Description of hyperparameter tuning, if applicable.

N/A.

Operationalization of hypothesis and evaluation metric.

We measure the performance of the DA methods by evaluating sensitivity and specificity. The relationship between sensitivity and specificity is evaluated by calculating the area under the receiver operator characteristic (ROC) curve. Since in practice differentially abundant taxa are only identified using a significance threshold <= 0.05, we only compute and compare the partial pAUC for 1-specificity = [0, 0.05] for each test and dataset.

Statistical techniques to evaluate hypothesis.

Aim 1 (Primary):

To estimate the average performance difference over all analyzed datasets, a mixed effects model is applied:

Now, the best performing method is used as intercept in order to estimate test performance loss of the other methods using the following mixed effects model:

As the above models assume normality of the residuals, this is tested by employing the Shapiro-Wilk test. In case of significance (p < 0.01), a Box-Cox transformation is applied to pAUC to resemble normality. Subsequently, p-values of the mixed effects model with transformed pAUC values will be calculated.

Additional descriptive analyses: A summary of performance results for each statistical test will be visualized using boxplots.

Aim 2 (Secondary):

For each simulation setting (in terms of sparsity, effect size and sample size), the 22 DA methods will be ranked according to their pAUC. The ranks are then analyzed using stepwise forward selection with as null model and as maximal model. Starting from the null model, this forward selection approach iteratively adds variables from the full model, evaluating their contribution to explaining pAUC until the optimal subset of predictors is determined.

Note, that the chosen rank transformation leads to ranks 1, 2, …, 22 for each combination of used data template, sparsity, effect size, and group size. Therefore, neither the used data template is required as predictor, nor the main effects for GroupSize, EffectSize and Sparsity. In case a test fails, NA will be assigned to the respective DA method and the remaining ranks will be scaled to the range [21,27] to not bias the rankings of the other DA methods.

Additionally, descriptive analyses will be employed to explore the defined research aims. For example:

PCA Plot of data characteristics: Principal Component Analysis (PCA) plots will be generated to visualize the overall variation in dataset characteristics using the prcomp function from the standard stats R package. If missing values occur, they are imputed for PCA by medians. Each data point (representing a dataset) can be colored based on its corresponding AUC value, providing insights into how dataset features relate to test performance.

Connected Dotplot: A connected dotplot can be constructed where each dot represents the performance (e.g., pAUC) of a statistical test on a specific dataset. Dots from the same test are connected, allowing for the visualization of trends or consistency in performance across datasets.

N-way ANOVA: N-way ANOVA will be applied for the selected multiple regression model to investigate whether performance benefits of specific tests significantly depend on multiple factor levels of the three simulation parameters. This statistical analysis will help identify significant differences in performance and potential interactions between tests and dataset characteristics.

Inference criteria.

We conduct the statistical analysis with linear mixed-effects models using the lme4 and lmerTest R packages. Specifically, coefficients and standard errors will be estimated via restricted maximum likelihood (REML), p-values using ANOVA and forward selection based on the Akaike Information Criterion (AIC). All p-values below a significance level a = 0.05 will be considered as significant.

Modification of differential abundance (DA) tests.

Inflated runtime

Datasets with a large number of features could lead to inflating runtimes for some statistical tests. We therefore define and apply a runtime threshold. This threshold will be set as the largest possible number, so that the predicted runtime of all DA tests does not exceed 3 months on a compute server with AMD EPYC 9454 48-Core Processor. Since we have to apply all DA methods to the experimental template once to estimate the number of not differentially abundant taxa (pi0), we use the observed runtimes for defining the runtime threshold.

If the runtime threshold for an individual test is exceeded for a specific dataset, we split the dataset into two subsets, each containing half of the taxa and try to complete the calculation within the maximal allowed runtime. To preserve the data structure as much as possible and avoid introducing additional randomness, we divide the data into even and odd rows. Specifically, the first subset contains taxa from rows 1, 3, 5,..., N_taxa−1, while the second subset includes taxa from rows 2, 4, 6,..., N_taxa. The DA method is applied separately to each subset, and the resulting p-values from both splits are subsequently merged.

If the runtime threshold is still exceeded for either subset, this split-and-merge procedure is repeated once more for each subset, effectively dividing the original dataset into a maximum of four subsets. For technical reasons, the runtime threshold is evaluated individually for each split and not to the total runtime summed across all subsets.

Here, we define the runtime threshold to be max. 1 hour per test. Then, in a worst-case scenario, the 22 tests for the 10 + 27 datasets for each of the 38 template (19684 combinations) and each simulation tool would need 820 days on a single core. Since we can conduct the tests on up to 96 cores, such a worst-case scenario would still be manageable.

Test failure

If a DA test throws an error, we apply the following procedure:

We first investigate whether failure of specific tests is related to performance, e.g., because it might happen that failures are more frequent for datasets which are more difficult to analyze, for example due to increased sparsity. Then, treating pAUC as NA would lead to bias. In order to investigate and prevent this, we apply logistic regression using the following model: failure ~ DA_method + DA_method:pAUC. Since pAUC is missing if a DA method failed, we have to impute those pAUCs. For this purpose, we use the median pAUC if the pAUC is available for respective DA_methods for datasets with the same simulation parameters. Otherwise, a mixed effects model pAUC ~ DA_method + (1 | dataTemplate) for all available pAUCs with the DA_method as fixed effect and the data template as a random effect is used for imputation.

If there are DA methods without significant interactions DA_method:pAUC, we omit the respective outcome and treat and reported them as NA (not available) as it would occur in practice without manual tuning the DA method. For significant interactions, we conduct a second analysis where NAs of those methods are replaced by the worst pAUC. This means that we treat failure of a DA like having the worst performance of all methods und we report this as outcome of or study. In addition, we compare these results with using NAs and report the respective results additionally.

List of software, central packages and dependencies (with version numbers) and their purpose.

R packages for data simulation: metaSPARSim_1.1.2, qvalue_2.32.0, SparseDOSSA2_0.99.2, vegan_2.6-4

R packages for DA tests: ADAPT_0.99.1, ALDEx2_1.32.0, ANCOMBC_2.2.2, corncob_0.4.1, DESeq2_1.40.2, distinct_ 1.16.0, edgeR_3.42.4, fastANCOM_0.0.4, glmmTMB_1.1.9, GUniFrac_1.8, limma_3.56.2, Maaslin2_1.14.1, maaslin3_ 0.99.3, metagenomeSeq_1.42.0, microbiomeMarker_1.6.0, phyloseq_1.46.0, rpart_4.1.23, vegan_2.6-8, ZINQ_2.0

R package for calculating data characteristics: amap_0.8-19, BimodalIndex_1.1.9,

R package for analyzing the performance of DA methods: lme4_1.1-35.1, lmerTest_3.1-3, pROC_1.18.5

Details on the implementation of the studied methods.

For a brief summary of the 22 evaluated DA tests, we refer to Nearing et al. [1]. Moreover, implementation details are provided in checklist item 18.

Extent to which code has already been written at the time of pre-registration.

Code to simulate data with metaSPARSim and sparseDOSSA2, calculate data characteristics and apply statistical tests are re-used from a former benchmark study entitled “Leveraging Synthetic Data to Validate a Benchmark Study for Differential Abundance Tests for 16S Microbiome Sequencing Data” which was partly conducted (80%) at the time when this protocol has been written [2]. In that study, we simulated data by calibrating the simulation parameter to each group separately. Since than all taxa have different parameters, there was not meaningful information about whether taxa are regulated and, thus, calculation of sensitivity and specificity were not feasible. All DA methods were tested on at least one experimental dataset, e.g., to provide details about how to call them and how to choose configuration parameters.

Description of the accessibility and availability of datasets, code, and material after the completion of the study.

Source code of the presented benchmark study and for reproducing the outcomes will be comprehensively published.

Other effort to ensure or improve reproducibility.

The code development will be performed using git as version control systems. Commits will be performed on a daily basis during the code development phase.

Known prior work based on the selected datasets, the analyzed measures in that work and its relation to the planned study.

The major findings from Nearing et al. [1] are:

Different differential abundance (DA) testing methods produce drastically different results when applied to the same microbiome datasets. The number and set of significant amplicon sequence variants (ASVs) identified varied widely across tools. The results also depend on data pre-processing, particularly whether rare taxa are filtered out before analysis. For many tools, the number of features identified correlates with dataset characteristics such as sample size, sequencing depth, and effect size of community differences. Some methods consistently identified more significant features than others, e.g., Limma voom, Wilcoxon (CLR), LEfSe, and edgeR tended to find the largest number of significant ASVs. Compositionally aware methods like ALDEx2 and ANCOM-II were generally found as more conservative, identifying fewer significant ASVs across datasets. ALDEx2 and ANCOM-II tended to identify significant features that were also found by other methods. Their findings highlight the importance of carefully considering methodological choices when conducting and interpreting microbiome differential abundance analyses and underscores the variability in results obtained from different DA testing methods.

Our ongoing benchmark study [2] indicates that metaSPARSim [3] and sparseDOSSA2 [4] can be used to generate datasets with similar data characteristics when all characteristics are considered on an aggregated level (e.g., by using PCA) (Figs 1A and 1B), whereas metaSPARSim [3] generates more similar results compared to sparseDOSSA2 [4]. Moreover, also individual data characteristics mostly coincide for metaSPARSim [3], except the following ones (preliminary results): sparsity, bimodality of the correlation of samples and effect size (Fig 1C). SparseDOSSA2 on the other hand tends to overestimate the library size (Fig 1D). We also see for both simulation tools that adding an appropriate proportion of zeros to the simulated data makes the synthetic data more realistic, i.e., reduces the number of non-equivalent characteristics. In our preliminary results, we also see that the ANCOM-II implementation in the R-package leads to different outcomes as the code used in Nearing et al. [1]. We also started to check, whether conclusions of Nearing et al. [1] can be validates with simulated data. Our preliminary results show that overall trends are reproducible, however only a few conclusions can be validated at a stringent level.

Prior knowledge about the datasets themselves.

N/A.

Neutrality statement regarding the investigated methods.

We confirm that all contributors to this study were not involved in the development of the evaluated DA methods, were not involved in developing the two simulation tools and not involved in the experimental studies where the data has been taken from. We also declare that there are no other competing interests that might lead to biased interpretations.

Steps taken to enhance and/or ensure the neutrality of the study (blinding).

N/A.

Plan for publishing the study and its results.

Public access of the generated data, analysis scripts, results and supplemental information is granted as indicated above. The results will be published in a peer-reviewed scientific journal, preferably in the same journal as this protocol.

Availability of results data after the study’s completion.

The 38 experimental datasets were downloaded from https://figshare.com/articles/dataset/16S_rRNA_Microbiome_Datasets/14531724 on February 9, 2024. There, Nearing et al. [1] made the datasets from their study available, therefore we incorporate the exact same datasets. We keep a local copy of this data in our Fredato research data management system https://nxc-fredato.imbi.uni-freiburg.de until 31.12.2030 and make it available if the original data is not available in the current version anymore and if this does not violate legal, data protection, or copyright regulations. Generated data, analysis scripts, results and supplemental information to this study will also be stored in the Fredato research data management. In case of unexpected technical limitations, we will make data, analysis scripts and supplemental information available via https://figshare.com.