Analysis of feature filtering for meta-omics prediction

To determine the degree to which sparse meta-omics features should be filtered out, we performed initial benchmarking of our experimental pipeline on three datasets containing paired metagenomics and metabolomics data (S1 Table): Franzosa et al. [12] (inflammatory bowel disease), Wang et al. [22] (end-stage renal disease) and Yachida et al. [23] (colorectal cancer). The latter two datasets were downloaded from The Curated Gut Microbiome Metabolome Data Resource, release v2.1.0 [24]. The IBD dataset was downloaded from the paper’s supplement. The IBD dataset contains metagenomics data in the form of gene families, while the other two contain taxonomic profiles at the species level. We note that MelonnPan was originally trained and tested on the same IBD dataset [16].

We evaluated two approaches for filtering out low-abundance features (species, genes, and metabolites):

• strict filtering: similarly to Mallick et al. [16], we retained only those features with at least 0.01% abundance in more than 10% of samples. We additionally filtered out features with less than 0.0001% abundance in more than 10% of samples. Features with more than 95% zeros were also filtered out across all feature types.
• lenient filtering: we retained only features with at least 0.005% abundance in more than 10% of samples. Features with more than 95% zeros were filtered out.

For each filtering method, we ran MelonnPan on all three datasets, with default settings, to predict metabolite abundances from metagenomics data. All benchmarking was performed on separate test sets. The data collected by Franzosa et al. [12] included an independently sampled validation cohort, which we used as a test set. We split the two remaining datasets into a training set and a test set, with a ratio of 80% to 20%.

The predicted data, along with the input metagenomics and experimental metabolomics data, were subsequently used to classify disease. To that end, we trained 10 random forest classifiers, initialized with different random seeds (the same ones shown in S2 Table) to predict phenotypes specific to each dataset. We used scikit-learn’s RandomForestClassifier (v1.4.1.post1), with default parameters.

Overview of the main experimental pipeline

Our main experimental pipeline uses data from the Inflammatory Bowel Disease Multi’omics Database (IBDMDB). First, we processed the data as follows: normalization (total-sum scaling), imputation of zeros, and feature filtering based on relative abundances. Datasets were then split at the patient level into training (80%) and test (20%) sets using fixed random seeds (S2 Table). Next, we applied data transformations specific to each prediction model (centered log-ratio, arcsin square root, quantile transformation). Any hyperparameter optimizations implemented by individual prediction models were performed using only the training data. Final model performance was evaluated on the held-out test set, within each split. All experiments were run on multi-core CPUs (see S1 Note).

Data processing on IBDMDB

Gut microbiome meta-omics data. We downloaded pre-processed metagenomics, metatranscriptomics, metaproteomics and metabolomics data from IBDMDB (Inflammatory Bowel Disease Multi’omics Database), which was assembled as part of the Integrative Human Microbiome Project [25]. The dataset contains longitudinal samples from 132 subjects, including a control group, as well as patients diagnosed with ulcerative colitis (UC) or Crohn’s disease (CD). Download links and dates are recorded in S3 Table. Before feature filtering, all meta-omics abundance profiles were normalized, such that feature values per sample sum up to 1. We used gene, transcript and protein abundance profiles annotated using Enzyme Commission numbers (ECs). As this data was originally stratified, we summed up ECs across taxonomic groupings to reduce dimensionality and sparsity. Additional experiments supporting all of our main results were performed using pathways and species abundances derived from mGx data, as shown in some of the supplementary results (S2 Table and S4 Table). For mBx data, we retained one LC-MS technology, namely C18 negative (C18-neg).

Data transformations.

Following feature filtering, each sample was normalized and the data was transformed to account for compositionality, sparsity, and feature scaling. To that end, we compared two standard transformations for compositional data, namely the centered log ratio (CLR) transformation, which requires the imputation of zeros, and the arcsin square root transformation, which also works on non-imputed data. The CLR transformation of a sample , with sum of elements , , and g(x) defined as the geometric mean of x, is given by:

(1)

For , with sum of elements and , the arcsin transformation is as follows:

(2)

As initial experiments showed that MelonnPan performed best among all benchmarked models, we also tested the quantile transformation implemented for this model, which maps normalized features to the quantiles of a normal distribution. This transformation was shown to improve the predictive power of standard regression models and neural networks [26,27]. Although Mallick et al. [16] only apply this transformation to the input features, we transformed the output features as well, to preserve consistency with other transformations that we benchmarked. This was implemented using scikit-learn’s QuantileTransformer (v1.4.1post1), with the output distribution set to “normal”.

Training and evaluation of meta-omics prediction models

Model evaluation.

We evaluated all models on each independent test set by comparing predicted features (transcripts, proteins, and metabolites) with the ground truth data. Consistent with methods reported in the literature, we used Spearman’s rank correlation coefficient to compare a predicted feature vector with the ground truth , transformed to ranks and R(y) [16,18,19,21]:

(3)

where is the covariance of the rank variables, and , are the standard deviations.

To compute scores across the 10 test partitions, we first computed the mean Spearman’s rank correlation coefficient per individual feature. We then reported the average for the top predicted features. The error was calculated as the mean standard deviation across features.

Training and evaluation of inflammatory bowel disease classifiers

To evaluate the applicability of meta-omics prediction models, we used the predicted features for the downstream task of inflammatory bowel disease (IBD) prediction. All classification tasks were performed using scikit-learn’s RandomForestClassifier (v1.4.1.post1), trained using random search cross-validation (S7 Table) with 50 iterations and a random state equal to the seed corresponding to each train/test partition (all random seeds are listed in S2 Table). We used 5 stratified cross-validation folds, divided based on study participants. We performed this 5-fold cross-validation exclusively within the training partition of each of the 10 main splits (defined previously for the regression models).

For each paired dataset, we reported the accuracy of IBD classifiers trained on the predicted data to that of classifiers trained on the input data used to generate the corresponding predictions. We additionally benchmarked these results against classifiers trained on ground-truth datasets of metatranscriptomics, metaproteomics, and metabolomics data. The training and test partitions were kept as the same ones used to train the cross-omics regression models. To provide a fair comparison, for each meta-omics input-output combination, we downsampled the classifier training sets to the size of the smallest dataset. In addition, each train set was downsampled to equal class proportions (IBD and healthy control). We recorded the number of samples per split and input-output combination in S8 Table.

1.1 A benchmarking pipeline for meta-omics prediction

To provide a standardized way to assess the performance of machine learning models to infer one meta-omics data type from another, we created a systematic evaluation protocol (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of our experimental set-up.

(A) We perform a pre-evaluation of MelonnPan [16] on three paired metagenomics and metabolomics datasets (S1 Table), comparing two filtering approaches for microbial features. In our main experimental pipeline, we use pre-processed gut microbiome data (B) to train regression models as meta-omics predictors (C), and subsequently evaluate these predictions for the downstream task of IBD prediction (D). Abbreviations: IBD (inflammatory bowel disease), ESRD (End-Stage Renal Disease), IBDMDB (The Inflammatory Bowel Disease Multi’omics Database), mGx (metagenomics), mTx (metatranscriptomics), mPx (metaproteomics), mBx (metabolomics), ECs (enzyme commission numbers), LC-MS (liquid chromatography-mass spectrometry), CLR (centered log-ratio).

https://doi.org/10.1371/journal.pone.0345919.g001

First, we evaluated the effect of feature filtering on model performance (Fig 1A). We used an existing prediction method, i.e., MelonnPan, proposed by Mallick et al. [16], to predict metabolite profiles from metagenomics data, using three datasets focused on different microbiome-associated conditions: inflammatory bowel disease, end-stage renal disease and colorectal cancer [12,22,23]. We compared two filtering approaches for sparse features: strict filtering, which resulted in a high number of microbial features being filtered out, and lenient filtering, in which fewer features were filtered out. For each filtering procedure, experimental and predicted data were also used to classify disease phenotypes.

Although the quality of metabolite predictions was invariant to the filtering approach (S1 FigA), we found that the filtering procedure did have an impact on downstream classification performance (S1 FigB and C). Across all three datasets, we noticed a significant drop in performance for classifiers trained on experimental metabolomics data, with the strict filtering approach. This suggests that features important for distinguishing phenotype were discarded as a result of strict filtering, so we reported results using the lenient filtering approach for the remainder of the manuscript.

Next, we designed an experimental pipeline to assess the utility of machine learning models for meta-omics prediction, integrating three main components: processing of paired microbiome data (Fig 1B), training and evaluation of meta-omics prediction models (Fig 1C), and, lastly, classification of inflammatory bowel disease with predicted data (Fig 1D). Using multi-omics data included in the Inflammatory Bowel Disease Multi’omics Database (IBDMDB), we selected paired samples from several combinations of meta-omics modalities, resulting in 11 paired datasets that enabled the prediction of transcript, protein and metabolite abundances from various input types. This data was then filtered for sparsity and transformed according to standard procedures for compositional data (see Methods). Afterwards, the following procedure was repeated ten times, to ensure that the reported performance was robust. Each processed dataset was divided into a training and test set, selected based on participant IDs, since the presence of samples from the same patient in both the training and the test set might have resulted in overfitting. Five multi-output regression models were then trained and evaluated for the task of meta-omics feature prediction: an elastic net [16], a feed-forward network [18], a sparse encoder-decoder [17], a deep neural network and a random forest regression baseline. The best model, i.e., the elastic net, was further used to generate meta-omics predictions in the last step in our pipeline, in which we compared the performance of IBD classifiers trained on three types of input data: predicted meta-omics data, the data from which it was predicted, and the ground-truth.

1.2 Machine learning models can reliably predict a subset of meta-omics features

To assess the generalization performance of machine learning pipelines designed for metabolite prediction, we first trained and evaluated several models from the literature on the task of predicting transcript and protein abundances, in addition to metabolite abundances. Average scores across 10 different train/test partitions were computed for 6 single-omics input-output combinations and 3 models from the literature MelonnPan, MiMeNet, and SparseNED [16–18]. To these we added two other classifiers: (i) a deep neural network (Deep NN, as described in S3 Note) and (ii) a random forest regressor (Fig 2) as an additional baseline model. Following a pattern established in the literature, we plotted the performance of cross-omics regression models for the 50 best predicted features (Fig 2A) [16,18,19,21]. For a more detailed statistical assessment, between-model differences were also evaluated with the Mann–Whitney U test (S9 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparison of meta-omics predictors.

(A) Mean test performance results of cross-omics regression models on several prediction tasks, calculated across 10 different dataset partitions. The average Spearman’s rank correlation coefficient was calculated for the 50 best predicted features for each output type. (B) For metatranscriptomics (mTx) and metaproteomics (mPx) predictions generated by MelonnPan [16], we also plot kernel density estimates comparing correlations between the input data and the ground-truth mTx/mPx data with those computed between the predicted data and the ground truth data. We perform this analysis on the 50 best predicted features for each output type, as well as all predicted features. Input types are represented through different colors, while cross-omics models are represented using different color intensities. Abbreviations: neural network (NN), metagenomics (mGx), metatranscriptomics (mTx), metaproteomics (mPx), metabolomics (mBx).

https://doi.org/10.1371/journal.pone.0345919.g002

Considering the top predictions, cross-omics regression models performed similarly in predicting metatranscriptomics and metabolomics (Fig 2A), with elastic nets achieving the highest average correlations, measuring 0.77 and 0.74, respectively. Protein abundances (mPx) were the most challenging to predict, with an average correlation coefficient of at most 0.4. In general, architectures like elastic nets and random forests were more robust across input-output combinations and performed best among the benchmarked models; we were also able to confirm these patterns on the basis of statistical testing (p-val < 0.05; S9 Table).

To investigate whether machine learning models provide more accurate estimations of transcript and protein abundances compared to the “gene-to-transcript-to-protein” assumption, we generated density plots comparing the distributions of correlations between different types of meta-omics data (Fig 2B). Top predicted transcript and protein abundances were significantly more highly correlated with the ground-truth data, when compared to the distribution of correlations between the input data and the ground-truth. This shows that a subset of features (transcripts or proteins) can be more reliably predicted using machine learning approaches, rather than relying on the assumption that genes encoded in the metagenome will be transcribed into mRNA and subsequently translated into protein. When plotting correlations for all features, this was still the case, but to a much lesser extent, ultimately indicating that only a subset of features can be reliably predicted.

1.3 Multi-omics integration does not lead to better predictions

To determine whether using a combination of different types of input features leads to better predictions of protein and metabolite abundances, we additionally trained MelonnPan, the overall best performing model identified in the previous section, on multi-omics input. Fig 3 shows a comparison between single-omics and multi-omics input in predicting metaproteomics and metabolomics. Results for other input types, such as taxonomic profiles and pathways derived from metagenomics, are recorded in S4 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Performance comparison of multi-omics and single-omics input data using MelonnPan [16].

Results of single-omics input types are shown as vertical colored lines. Model performance on multi-omics data is indicated relatively to the best single-omics input type in a combination. The combination is displayed as a two- and three-color diagonally spliced bar with colors indicating including respective data types. Improvements or downgrades in performance are indicated with arrows and the size of the bar. Abbreviations: metagenomics (mGx, blue), metatranscriptomics (mTx, pink), metaproteomics (mPx, green), metabolomics (mBx).

https://doi.org/10.1371/journal.pone.0345919.g003

While metaproteomics predictions marginally improved when combining metagenomics and metatranscriptomics, with 2% higher average correlation, combining single-omics modalities did not lead to more accurate predictions of metabolite abundances. Comparable performance was obtained when using metagenomics data processed in the form of pathways or species-level taxonomic profiles (S4 Table).

We also designed a more elaborate multi-omics integration scheme, using an auto-encoder trained with a joint reconstruction and regression loss (S3 Note and S2 Fig), but experiments did not show promising results. Therefore, we did not pursue this line of research further. S10 Table shows the performance of MelonnPan trained on concatenated multi-omics, compared to the embeddings learned by the auto-encoder architecture. Although we observed a decline in prediction accuracy, the models trained on latent features were more robust, as suggested by the very low variation in model performance across test partitions.

1.4 Core set of well-predicted features is robust to input perturbations

We evaluated the robustness of cross-omics models through an analysis of well-predicted features across dataset partitions and input types (Fig 4(A), 4(B) and 4(C)). We limited this analysis to results produced by MelonnPan, as we found this model performed best overall for the task of cross-omics prediction. However, additional experiments were performed with a deep neural network model (S3 Note), to study the effect of feature selection on model performance (Fig 4(D)).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Analysis of well-predicted features.

(A) Jaccard similarities between the sets of the 25% best predicted features by MelonnPan [16] for each output type (mTx, mPx, mBx), compared across 10 different train/test partitions. All predictions were generated from mGx data. (B) Jaccard similarities and Venn diagram of the sets of the 25% best predicted proteins, compared across input types. (C) Jaccard similarities and Venn diagram of the sets of the 25% best predicted metabolites, compared across input types. (D) Performance of a deep neural network model (S3 Note) trained on different feature subsets (all, 50%, 25% and 10%), based on a pre-training step for feature selection (S3 Note and S4 Fig). The best results for each input-output combination are highlighted. Abbreviations: metagenomics (mGx), metatranscriptomics (mTx), metaproteomics (mPx), metabolomics (mBx).

https://doi.org/10.1371/journal.pone.0345919.g004

A pairwise comparison of the top 25% well-predicted features across train/test partitions showed that these subsets share a selection of features, with some features being well-predicted across all test splits. While Fig 4(A) only shows predictions generated from mGx data, we found this to be the case for all single-omics input types (see S3 Fig). For each output type, a small set of features was found to be consistently well predicted across dataset partitions. For metaproteomics, this number was especially low (2.5% of the feature union). Glutamate dehydrogenase (1.4.1.3) and DNA-directed RNA polymerase (2.7.7.6) were the two enzymes included in this subset. Although low abundance of these enzymes has been linked to IBD, including associations between glutamate dehydrogenase and Clostridium difficile infections in IBD patients [28–30], their consistent predictability across dataset partitions likely reflects their role in basic cellular functions rather than disease-specific effects. Since they are involved in core metabolic and transcriptional processes, these proteins are common across many microbial taxa and may be detected more easily in metaproteomics datasets. Additionally, the consistent selection of these enzymes may also reflect dataset-specific characteristics, such as cohort composition and preprocessing, which can influence how well these proteins are detected.

We further examined functional enrichment of well-predicted features based on EC classes (S11 Table and S12 Table). For metatranscriptomics, no statistically significant enrichment was observed. For metaproteomics predictions, class 1 enzymes (oxidoreductases) were significantly (p-val < 0.05) overrepresented among the top well-predicted 10–25% of features. These enzymes perform fundamental energy-related reactions that occur in most cells, likely making them easier to detect and predict across samples and taxa.

Some well-predicted features were also shared across single-omics and multi-omics input types (Fig 4(B) and 4(C)). In total, 25 proteins were well-predicted from both metagenomics and metatranscriptomics data, while 401 metabolites were well-predicted from metagenomics, metatranscriptomics and metaproteomics data. We also did not find significant correlations between feature variance and prediction quality (S5 Fig).

However, regardless of what makes features easy to predict, these results imply that there is a core subset of features, for each output type, that can be reliably predicted. Consequently, we hypothesized that training a model on just a subset of features would lead to better predictions, as the trade-off between data dimensionality and the number of samples would be more balanced in that case. Note that a multi-output elastic net such as Melonnpan would not be a suitable model to perform such an experiment. This is because this kind of architecture combines outputs from multiple independent single-output regression models, and, as such, reducing the number of output features would not have any effect on how accurately an individual feature can be predicted [31]. On the other hand, a neural network architecture learns all output features simultaneously, so the number of output features matters during training.

To that end, we first ran a pre-training iteration which consisted of training 10 random forest models on different cross-validation splits, averaging feature correlations and retaining only a proportion of the top features (S3 Note and S4 Fig). We then trained a deep neural network (S3 Note), restricting the output to subsets containing 50%, 25% and 10% of features based on individual correlations obtained during pre-training (Fig 4(D)). In addition to learning dependencies between output features, deep architectures were shown to be better at bypassing the curse of dimensionality, particularly when modeling compositional functions [32]. Ultimately, network performance did not improve when training on a smaller set of output meta-omics features, but we also did not observe a decline in prediction accuracy (Fig 4D).

1.5 Predicted meta-omics data can classify phenotypes with performance comparable to experimental data

Lastly, to demonstrate an application of cross-omics regression models, we tested whether predictions generated by these models could be used for the downstream task of inflammatory bowel disease prediction. We compared the classification performance of random forest classifiers trained on input and predicted data, as well as experimental data of the same modality as the predictions (Fig 5). To ensure a fair comparison, datasets for each input-output combination were downsampled to the same size (S8 Table). Training sets were further downsampled to equal class proportions (IBD and healthy control).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Accuracy of random forest classifiers on the binary task of inflammatory bowel disease prediction, averaged across 10 test partitions.

Balanced accuracy of random forest classifiers on the binary task of inflammatory bowel disease prediction, averaged across 10 test partitions. From left to right within a group of bars, we show performance of classifiers using: input meta-omics data, predicted meta-omics data (striped; generated with MelonnPan [16]), and ground-truth meta-omics data for the predictions. Gray boxes indicate performance of random guessing, obtained from Dummy classifiers based on stratified sampling. Abbreviations: metagenomics (mGx), metatranscriptomics (mTx), metaproteomics (mPx), metabolomics (mBx).

https://doi.org/10.1371/journal.pone.0345919.g005

Random forests trained on predicted metatranscriptomics features could distinguish IBD from healthy controls with a balanced accuracy of approximately 68%. This performance is on par with that of the classifiers trained on metagenomics input data or on experimentally measured metatranscriptomics from the same samples (Fig 5). We did not find any statistically significant differences among these three data types, although all trained classifiers performed significantly better (p-val < 0.05) than the dummy baseline based on stratified random guessing (S13 Table). In contrast, analyses involving metaproteomics were constrained by small training and test sample sizes (S8 Table), and classifier performance did not differ significantly from random guessing.

For metabolomics, classifiers trained on ground-truth metabolomics achieved higher accuracy for IBD prediction than those based on metatranscriptomics or metaproteomics data. However, comparisons across modalities should be interpreted with caution, since the classifiers were trained and tested on different sample sets. In some cases, including models using metagenomics input data, classifiers trained on predicted metabolomics profiles had higher mean accuracy than those trained on the original input data, although these differences were not statistically significant (S13 Table). Overall, experimentally measured metabolomics provided the most reliable predictions, although predicted metabolomics data sometimes achieved comparable performance, particularly when derived from metatranscriptomics or multi-meta-omics inputs.

We also evaluated the classifiers based on ROC-AUC, precision, recall and F1 score (S14 Table). These additional metrics showed trends consistent with those seen for balanced accuracy.