Experimentation.

We applied SPARTA to six publicly available datasets, previously explored in articles such as MetAML [19] or DeepMicro [18]. These datasets contain taxonomic abundance tables issued from sequenced microbiota samples from cohorts of healthy controls and individuals diagnosed with Cirrhosis (Cirrhosis dataset), Colorectal Cancer (Colorectal dataset), Obesity (Obesity dataset), Type 2 Diabetes (T2D and WT2D datasets) or Inflammatory Bowel Disease (IBD dataset). For further details, see Materials and methods. SPARTA was launched with both RFs and Gini importance score [16] (default parameters), and with RFs and SHAP [27] importance score, for 10 runs, 5 selection iterations per run, and 20 trained models per iteration. We also evaluated the classification performances of SVM classifiers on the same datasets.

Machine Learning classification performances obtained from functional profiles are similar to those from taxonomic profiles.

Fig 3 illustrates the classification performances of the RFs [16] trained by SPARTA to distinguish between patients and healthy individuals, per profile and dataset. Classification on the taxonomic datasets prior to selection is analogous to the classification without representation learning method implemented in DeepMicro [18], with 20 RF (SPARTA) instead of 5 (DeepMicro) and dedicated test sets (SPARTA). For each RF trained by SPARTA, the area under the receiver operating characteristic curve (AUC) is calculated. Seeing as 20 RFs are trained within an iteration, the median of these 20 AUCs is retained to represent the performances of the iteration as a whole. The full iterative process is repeated 10 times, giving 10 median performance metrics per level of iterative selection (see Fig 2). The amount of selections that leads to the highest median among these metrics is deemed to be the optimal selection and is the one represented here for the taxonomic and functional profiles. The number of selective iterations corresponding to this selection are given in the ‘Optimal Selection’ column. For each dataset, a Mann-Whitney U-test was conducted comparing the performances based on the taxonomic and functional profiles at respective optimal selection levels. The details of all of the obtained performances are given in S1 File. We also performed the same process evaluation with SVMs and showed that RFs consistently outperform SVMs on the datasets presented in this paper (see S1 Fig), as well as RF with SHAP values as importance scores (see S1 Fig, no significant difference compared to RF and Gini importance scores). We also reported in S1 Table the classification performances on validation sets. Those are usually higher than on the test set. This is expected due to the iteration process where importance scores are averaged over all RF to perform variable selection.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Classification performances of RF models trained on taxonomic and functional profiles, and impact of the variable selection on performance.

Median classification performances (AUC) for all types of profiles and each dataset, on the original datasets as well as at the optimal level of selection over 10 full runs of the pipeline. Each of these runs involved a different randomly selected test set of individuals, which was used for both profiles. Performances and importance scores for each run were computed and averaged over 20 distinctly trained RF models. The amount of selection iterations required to obtain the best average among these median AUCs are represented beside each plot. Instances when the difference in performance between functional and taxonomic profiles using SPARTA is significant for a same dataset (based on a Mann-Whitney U-test) are signaled by a * symbol.

https://doi.org/10.1371/journal.pcbi.1012577.g003

For example, the Colorectal dataset’s functional (purple) and taxonomic (green) profiles have been tested over 10 runs by SPARTA. These tests have allowed us to detect the level of variable selection that yields the best median classification scores for each profile, which were then chosen for this representation. In this case, as shown in the ‘Optimal selection’ column, the functional dataset gives its best performance after 2 iterations of variable selection, whereas the taxonomic dataset gives its best performance after just one. The performances of RFs trained on taxonomic and functional profiles without selection are also represented, in red and blue respectively. Each of the 10 runs of SPARTA yields an average classification performance score, corresponding to the plotted dots. The boxplots represent the associated distribution and notably show that the functional profile has a median AUC of 0.85, against 0.86 for the taxonomic profile. The difference between both distributions was not found to be significant by a Mann-Whitney U-test, as shown by the absence of an asterisk symbol on this row.

Overall, we can see that taxonomic profiles yield better median classification performances than their functional counterparts, with the T2D dataset being the only exception. However, the difference in performance between both profiles is only significant in the case of the WT2D dataset, showing that though converting our data to the functional level comes at the cost of some performance, both profiles perform comparably as a basis for classification. These results are in line with the previous works of Douglas et al. [20] and Jones et al. [21]. However, the innovative potential of functional profiles resides more in their prospective contribution to a biological understanding of the diseases’ mechanisms than in their use for automatic classification.

Of note is also the asymmetrical benefit of variable selection. Functional profiles systematically benefit from a reduction of dimensionality, as their median performances after iterative selection (purple) are always superior to those obtained without variable selection (blue). For taxonomic profiles however, variable selection leads to a decrease in median results for three of the six datasets (Cirrhosis, WT2D, and IBD).

A comparative classification was made based on a functional profile built from the raw reads of the IBD dataset with HuMAnN3 [7], using the same parameters. The obtained results (S2 Fig) show that median classification based on functional profiles built directly from the reads are on par with those obtained using EsMeCaTa, as the differences in performance are not significant based on a Mann-Whitney U-test (p-value = 0.45). Both functional profiles’ performances are also non significantly different from the performance obtained on the IBD taxonomic dataset (p-value = 0.73 for HuMAnN and 0.36 for EsMeCaTa).

We will now focus on propositions to optimize the differential functional profiling of microbiotas in the context of a disease, as well as evaluate the added value of functional information in comparison to taxa for understanding the underlying biological processes.

Robustness of SPARTA’s feature selection: Comparative evaluation.

The datasets used in the previous section contained on average 484 taxa. Through EsMeCaTa’s [10] pipeline and its interrogation of UniProt [24], these taxa were linked to a total average of 10,510 FAs per dataset, resulting in a 22-fold mean increase in the amount of information, as shown in Table 1. For example: in total, the sequenced samples of the Cirrhosis dataset covered 542 taxa, which were associated by EsMeCaTa to a total of 10,434 FAs. Following SPARTA’s application, 72 of these taxa and 33 of these annotations were included in the candidate sublists. Among these, 32 taxa and 7 annotations were in the confident subset, and 23 taxa and 4 annotations were in the robust subset. The sizes of the selections obtained from selections based on SHAP importances are also available in S3 and S4 Figs. These results showcased that selections based on SHAP importance scores were less robust than those based on Gini importance, as the sizes of the Robust selections obtained through this method were consistently smaller than those obtained with Gini-based selection, for both functional and taxonomic profiles. Three of the functional datasets (WT2D, Obesity, and Colorectal) and one functional dataset (Colorectal) even gave empty Robust selections from the first iteration with SHAP, which does not happen on any dataset with Gini. As such, only results based on Gini selections were presented here. This is however illustrative of the impact of the ranking approach on the overall quality of SPARTA’s analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Application of the SPARTA selection process to identify signature taxa and functions on 6 reference datasets.

https://doi.org/10.1371/journal.pcbi.1012577.t001

To balance the increase in information when using FAs, SPARTA operates a selection of variables based on the features’ importance scores. These scores, when ordered from highest to lowest, display a kink-like shape. SPARTA automatically operates a cut-off at the inflection point of the kink and probes whether classification performances are improved (see Materials and methods). This selection aims to correct the redundancies and the dimensionality of the original dataset for better classification. It also generates one of the pipeline’s main outputs: a list of ranked features (either taxa or FAs) based on their average importance scores [16], and including an automatically computed cutoff that distinguishes significant and non-significant information. SPARTA provides the user with the list of important taxa and FAs for each iteration, the corresponding classification performance, and a focus on the best iteration after the first level of selection.

The amount of information retained per SPARTA run for all functional datasets is illustrated in Fig 4(A). The figure shows that the average amount of information to retain for optimal classification performance varies depending on the dataset. For example, retaining the top 500 annotations ranked by average Gini importance would give a selection similar to SPARTA on the IBD dataset, whereas the Obesity dataset would require the top 1,000 annotations to match the selection. This shows that an adaptive method like SPARTA, which makes a decision concerning the quantity of information to be retained by the selection, has an advantage over a selection based on a fixed threshold because it can adapt to the complexity of the problem at hand, which is shown here to be variable. SPARTA’s selection thresholds also do not match the more traditional thresholds, such as the top 30 features explored in Jones et al. [21], and can be used to get an estimate of the optimal amount of information to consider for discerning microbiota profiles.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison between SPARTA and limma functional selections.

A: Number of important selected FAs for each run at best iteration for the six datasets Amount of FAs selected by SPARTA and limma, for all datasets. Limma selections were effectuated with an adjusted p-value threshold of 0.05. Both selection methods were repeated 10 times, with a common test subset set aside each time. B: Comparison between robust and candidate FAs for T2D dataset The limma subsets were obtained using the classic threshold of 0.05. Values indicate the number of annotations in each intersection and do not represent the size of a category as a whole. The white circle includes all annotations from the full dataset.

https://doi.org/10.1371/journal.pcbi.1012577.g004

To explore whether SPARTA’s variable selection differs from classic linear approaches, we compared our approach with a standard method designed for continuous data [11] rather than for count data [28]. Specifically, selections obtained from direct pairwise comparison of the profiles using the limma tool [11] were compared. Variables were selected using a p-value threshold of 0.05, a classic threshold value exploited in several other studies that applied limma to metagenomic data [12–14]. Similarly to SPARTA, the selection process was iterated 10 times with variation induced from setting aside a subset of the samples, and variables were compiled into ‘robust’, ‘confident’, and ‘candidate’ categories depending on how often they were selected. Comparative results of this process are presented in Fig 4(A) and Table 2. For example, Fig 4(A) shows that, when applied 10 times to the Cirrhosis dataset, SPARTA selects a minimum of 6 annotations, and a maximum of 21, with a median of 11. In the same conditions, limma selects between 1,032 and 2,149 annotations, for a median of 1,642. These distributions are plotted, respectively, in purple and gray. Table 2 shows that with SPARTA’s selection, the Cirrhosis dataset outputs 4 robust annotations, 7 confidents, and 33 candidates, against a respective 878, 1,165 and 2,668 with limma. With these parameters, limma is a much more stringent selector than SPARTA on all datasets aside from Cirrhosis. For the Colorectal, WT2D and Obesity datasets in particular, all selections are empty, leading to an empty candidate subset as described in Table 2. The IBD dataset also proves to be unsuitable for this approach, yielding empty robust and candidate subsets, and an empty robust subset. Only the T2D and Cirrhosis datasets allow limma to yield a non-empty robust subset. SPARTA, on the other hand, consistently yields non-empty robust and confident selections, both of which are reasonably sized for interpretation when compared to the candidate subsets, being close to 50 times smaller in the case of the WT2D dataset’s confident and candidate subsets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Sizes of the SPARTA and limma selections.

Limma was applied with an adjusted p-value threshold of 0.05. From left to right, the columns present, for SPARTA and limma, the size of the robust, confident, and candidate subsets issued by the concerned selection method iterated 10 times with identical test subsets.

https://doi.org/10.1371/journal.pcbi.1012577.t002

Among these datasets, Cirrhosis stands out as an outlier. Indeed, it is by far the dataset on which limma selects the most information: in Fig 4(A), we can see that it selects 1550 annotations on average over 10 iterations, whereas the second highest amount, obtained with the T2D dataset, is only 26.1 on average. This also makes it the only case in which SPARTA proves to be the most stringent of the two selectors, with an average of 12 selections per run, for a Robust selection of size 4 against limma’s 878 (see Table 2). The four annotations in question are: GO:0016984 (ribulose-bisphosphate carboxylase activity), GO:0003779 (actin binding), GO:0004081 (bis(5’-nucleosyl)-tetraphosphatase (asymmetrical) activity) and GO:0018112 (proline racemase activity). Actin binding (GO:0003779) signals the participation of the gut in the maintenance of the intestinal epithelia, which plays a role in the prevention of liver diseases such as Cirrhosis [29]. The activity of proline racemase (GO:0018112) is also indicative of proline metabolism in the gut, which has also been shown to be upregulated in cases of Cirrhosis [30]. The activity of the bis(5’-nucleosyl)-tetraphosphatase enzyme (GO:0004081) is involved in the metabolism of both purine and pyrimidine according to KEGG [31], which are disturbed in mice gut during the development of Cirrhosis [32]. Finally, ribulose-bisphosphate carboxylase (GO:0016984), though it is mostly known for its role in photosynthesis, can also be involved in the salvage of methionine [33], itself key in the development of liver disease [34].

As such, in the case of Cirrhosis, SPARTA robustly highlights a small subsection of biologically relevant annotations, themselves consistently highlighted by limma as linear indicators of the prevalence of the disease. This could illustrate a case in which the dataset is “too easy” to predict, due to an abundance of features that linearly differentiate the profiles, and a small sample of which is sufficient to be efficient in classification. This could lead to an over-selection from SPARTA, as even when relevant features are removed by the iterated selection, the remaining variables still allow for good classification performance. In this case, it could be interesting to look at the selections from iterations before the optimum.

We then focused on the T2D dataset, which is the only other dataset on which limma i) extracts a non-empty robust selection with an adjusted p-value threshold of 0.05 (see Table 2) and ii) consistently provides non-empty FA selections. Fig 4(B) illustrates the overlap between limma’s and SPARTA’s robust and candidate annotations.

T2D’s limma selection is smaller than SPARTA’s, englobing a total of 103 annotations in its candidate subset against 1,575 for SPARTA, as shown in Table 2. As shown by Fig 4(B), all of these annotations aside from one are included in SPARTA’s candidate selection. Similarly, limma’s robust subset is entirely included in SPARTA’s robust selection.

To put these results in perspective, there is no guarantee that a 0.05 p-value threshold yields an ‘optimal’ selection for this dataset when applying limma. This choice of threshold is, however, a required external input for the method, that SPARTA does not need as it automates the choice of the selection’s size. As such, the chosen threshold could arguably be too restrictive for the T2D dataset. As an illustration, a p-value threshold of 0.255, obtained to generate a limma candidate selection as close as possible to the size of SPARTA’s, was applied, as illustrated by S5 Fig. This much less restrictive threshold yields a limma selection that still largely overlaps with SPARTA’s selection (74% of limma’s annotations being included in SPARTA’s).

Overall, limma does not yield exploitable selections with a classic p-value threshold on four of our six datasets. The examination of the remaining two datasets allows us to illustrate how SPARTA and limma behave comparatively in different situations. In T2D’s situation, the limma selection is smaller and largely overlaps SPARTA, with limma’s robust subset notably being entirely included in the SPARTA selection. For the Cirrhosis dataset, the SPARTA selection is the smallest of the two, however, it remains coherent with what limma selects, and yields information that is coherent with the biological question at hand. RF classification performances obtained on both selections and presented in S6 Fig, also show that limma’s selections perform under SPARTA’s as basis for classification, as neither of the recorded performances surpass their SPARTA counterparts.

Visualization of the robust shortlists.

An important output of SPARTA is the shortlist of robust variables that are selected by the method, allowing for downstream interpretability. This comes in the form of tables of robustly significant annotations and taxa, as previously described. The annotation shortlist for the IBD dataset is given in Table 3. It contains 59 FAs, alongside extra information that SPARTA helps associate with them. For example, annotation GO:0006520, corresponding to the amino acid metabolic process, is first in the table because it has the highest average Gini importance score over all 200 forests trained at this selection level, over 10 runs. It is on average 1.05 times as present in the diseased profiles as it is in the controls, the negative value of the ‘Ponderated average ratio’ meaning that the annotation is predominantly found in unhealthy samples. It is linked to a total of 358 taxa over all samples, of which 20 were found to be robust. The subsequent bibliographic analysis of this list graded its relevance to the disease as a 1, meaning that there is a known direct link between the annotation and IBD [35]. Detailed outputs are made available in S2 File.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Robust subset of annotations from the IBD dataset.

https://doi.org/10.1371/journal.pcbi.1012577.t003

A similar selection of robustly discriminant taxa is also available as an output of the pipeline, with the IBD output given as an example in Table 4. The same information as the previous table is available for each taxon, aside from the bibliographic categories. For instance, Alistipes finegoldii, identified in our process as Organism 73, similarly ranks first because it has the highest Gini importance score on average over all trained RFs. Its differential expression shows that it is expressed on average 16 times as much in control profiles as it is in the unhealthy samples. As previously, we can establish which annotations are attached to each taxon, with A.finegoldii expressing a total 1,220 FAs, 15 of which are robustly significant. The details of these associations are available in S3 File.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Robust subset of taxa from the IBD dataset.

https://doi.org/10.1371/journal.pcbi.1012577.t004

Bibliographic exploration of the functional robust shortlist.

Beyond the examples mentioned in this chapter, an in-depth bibliographic analysis of these outputs has been conducted for the IBD dataset and is available in S4 File.

The bibliographic examination was conducted on the integrality of the robust annotations from the IBD dataset, as well as samples of 20 annotations that were present in 50% of the significant sublists from SPARTA’s runs, and 20 non-candidate annotations. The methodology was to research the name of the annotation alongside the name of the disease on Google Scholar (https://scholar.google.com/). If none of the research results provided conclusive information linking this annotation to IBD, be it in a host model or the microbiota, the chemical products of the annotation and eventual alternative names of the annotation were similarly tested, followed by related (parent or child) annotations, and finally, the linked pathways listed in the BRENDA database [36]. From this exploration, the annotations were given a bibliographic relevance grade of 1 (most relevant to the disease) to 4 (least relevant to the disease) based on the following criteria:

1. Category 1: A direct link was established between the annotation, or a direct product metabolite, and IBD. This can come in the form of an explicit description of the metabolic mechanism’s involvement, or simply in the form of measured differential presence between unhealthy and control individuals. Note that conclusions derived from other ML-based approaches were not considered to be sufficient evidence, as they could suffer from biases similar to our approach.
2. Category 2: A direct link was established between a similar metabolic function and the disease. Were considered as similar: proteins or enzymes from the same family as the one involved in the annotation (i.e.: ATP-dependent and ATP-independent citrate lyases), and parent and child annotations, signaling notably that the annotation is indeed relevant, but at the wrong scale.
3. Category 3: An indirect correlation was established between the annotation and the disease. This can mean that the annotation was not directly linked to IBD, but that it is involved in a larger pathway or expressed by a taxon that has significance.
4. Category 4: No leads were found, or the annotation was proven to be irrelevant.

Among the robust annotations, several were found through bibliography to be relevant to the disease when expressed in the host organism as opposed to the microbiota. We considered both cases as a link found between the annotation and the disease, following the idea of permeability and interactions between the microbiota and its host [37].

When available, we also retrieved the group, namely unhealthy or control, most likely to express these annotations according to the bibliography. At the same time, SPARTA also retrieves the group that most expresses each of these robust FAs (see Materials and methods). We confirmed these associations between FA and group with limma as well, for better robustness. We found that bibliography predictions and prevalence in the IBD dataset patients were in agreement in 47% of cases. FAs where disagreement exists between bibliography and SPARTA/limma might point towards the rescue of important functions in the host by the microbiota [38].

A complementary comparative analysis was conducted through a Chi² contingency test with a 95% p-value threshold between the prevalences of each bibliographic category in the robust selection and those of randomly selected non-candidate annotations (see S2 Table for details). This test established that the robust group significantly diverged from the non-candidate group. This significant difference is notably driven, as seen in S2 Table, by a comparatively increased proportion of Category 1, and a decreased proportion of Category 4 annotations in the robust subset compared to the non-candidate selection. These results support the notion that SPARTA is a relevant selector of information.

Exploring the pairings between robust taxa and annotations highlights their non-redundancy.

The usage of the EsMeCaTa pipeline [10] in building functional scores allows us to make explicit and quantify the links between taxa and their FAs, as this tool retrieves the annotations associated with a given taxon in the UniProt database [24]. Applying the pipepline to the IBD dataset (443 taxa and 10,196 FAs), the results show that annotations can be associated with 47.8 taxa on average. One annotation is associated with the most taxa (437 taxa out of 443): GO:0016021, which is attached to the cellular membrane component and is therefore expected to be extremely widespread. Unique associations account for 37.5% of all annotations, thus a majority of annotations are associated with more than one taxon. Overall, no function is perfectly ubiquitous, and the majority of functions are linked to several different taxa.

To quantify functional redundancy among taxa, we used Jaccard proximity to measure the similarity of their functional associations. Taxa with a Jaccard proximity of 95% or more were considered functionally identical. Our analysis showed that 77.2% of the taxa do not have such close neighbors, indicating that they maintain distinct functional profiles from each other, despite sharing many annotations with other taxa. Detailed results are given in S5 File.

The observed disparities between taxonomic and functional profilings prompt the question of whether these profiles equally provide valid descriptions of a subject’s microbiota. A potential drawback of the taxonomic scale is the cumulation effect, wherein individual taxa may have little significance but contribute significantly to an essential metabolic process when grouped. As a result, this collective impact might go unnoticed when focusing solely on individual taxa. The dynamics in terms of specificity between annotations and taxa are illustrated in Fig 5, which plots the amount of robust taxa associated with each annotation as a function of the total amount of associated taxa. For illustration purposes, the represented annotations were assigned to four profiles based on their number of associated taxa. We labeled the top 10% as “Ubiquitous” (5 annotations, top right in Fig 5), the bottom 10% as ‘Specific’ (18 annotations, bottom left of Fig 5), and all others were labeled ‘In-Between’ (32 annotations). Finally, a fourth category was drawn up, independently of the previous criteria, containing 4 annotations that have no link to robust taxa, which we labeled as ‘Cumulative’. This representation shows that important annotations have differing relationships to their taxon counterparts and that an annotation’s importance can stem from the influence of several taxa, as is notably illustrated by the ‘Cumulative’ class.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Number of taxa associated to each robust annotation, as a function of the number of associated robust taxa for the IBD dataset.

Four groups of annotations are represented, three of which were determined based on the total amount of taxa attached to the annotation: those within the top 10% of these values’ scale were labeled ‘Ubiquitous’, those in the bottom 10% were labeled ‘Specific’, and the others were labeled ‘In-between’. The final category corresponds to the robust significant annotations with no relationship to the robust significant taxa (‘Cumulative’). The highlighted annotations are those used as illustrative examples in Fig 6.

https://doi.org/10.1371/journal.pcbi.1012577.g005

A detailed illustration of annotations’ pairings with their taxon counterparts, as well as the strength of these links determined as described in Materials and Methods, is proposed in Fig 6. The represented annotations were taken from each of the categories illustrated in Fig 5: GO:0006520 as representative of the ‘Ubiquitous’ class, 1.1.1.22 for the ‘In-between’ class, GO:0043130 for the ‘Specific’ class, and GO:0006144 as a ‘Cumulative’ example.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Associations between robust functions and the associated robust taxa predicted by SPARTA, for the best iteration on the IBD dataset.

Depicted annotations were selected to be representative examples of the different categories highlighted in Fig 5, and are presented with the same color scheme. Taxa are colored based on their normalized average differential expression between Control (red) and Unhealthy (blue) profiles. The width of the connections is proportional to the importance of the association. The arrow between a given function and the generic ‘Non-robust’ node represents the contribution of non-robust taxa to the considered function.

https://doi.org/10.1371/journal.pcbi.1012577.g006

From top to bottom in Fig 6, the first annotation (GO:0043130) is a case in which the feature’s significance appears to be due to a strong association to a single robust significant taxon, namely Akkermansia muciniphilia. This taxon has an established impact on IBD remission, and is researched as a potential probiotic treatment of the disease [39]. This is also in accordance with the annotation’s differential expression between profiles, as seen in Table 5, where the annotation is shown to be expressed in the control samples almost 3 times as frequently on average as it is in the sick samples. This kind of relationship could either indicate that this ‘Specific’ annotation derives its importance in our predictions from its strong and specific attachment to an important taxon, or that its impact on the disease is an important factor to explain this taxon’s beneficiary influence. GO:0043130 corresponds to ubiquitin binding, a mechanism that is known to regulate the inflammation process of intestines via different signaling pathways [40], and is categorized as a Category 1 annotation by our bibliographic research, showing that in the case of our example, the effects of the annotation and of its specifically associated robust taxon align. It should be noted that, as mentioned in our earlier discussion around our bibliographic work, the differential expression of a feature can be contradictory with its known effects, and should therefore be treated with caution. The second and third annotations (1.1.1.22 and GO:0006520), respectively from the ‘In-between and ‘Ubiquitous’ groups, are very widespread among robust taxa, without any particularly strong link to any of them. In cases such as these, meaning metabolic functionalities commonly expressed within taxa, the issue of significance is shown to not be a purely binary question of expression or absence, as both annotations are consistently present in both unhealthy and control profiles. Finally, the last annotation (GO:0006144) is exclusively linked to non-robust taxa. All such annotations, from the ‘Cumulative’ group, are associated with several taxa (13 minimum), meaning that their importance results from the cumulated influence of multiple, individually non-significant taxa, that have a significant role when grouped functionally. The reverse associations, plotted in S7 Fig, show that this form of cumulation is specific to FAs: the robustly significant taxon with the least associations to robust significant annotations, Coprococcus comes, is still shown to have a non-zero amount of correlations to robust annotations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Distribution of samples within the datasets of reference.

https://doi.org/10.1371/journal.pcbi.1012577.t005

Microbial and functional profiles.

A central discussion point of this study is the pros and cons of exploiting the microbiome’s FA data as opposed to the explored taxonomic profiles for classification and interpretation. It should first be noted that there is an inherent bias to the exploitation of metagenomic data [42], notably concerning the taxa of lower abundance which are susceptible of being false positives.

In terms of classification performance, as shown in Fig 3 and discussed in Results, the taxonomic profiles remain a better overall predictor of disease state, though the difference is not significant in most cases. These results are in line with the findings of previous studies [20, 21], and can be explained by the increase in the amount of features contained in the functional profiles. Indeed, for a set amount of data, augmenting the number of variables past a certain point is known to be detrimental to model performance [43], and the switch to functional profiles comes with 22 times as many variables on average, without any additional samples to balance this. This hypothesis is further supported by the fact that variable selection increases the functional profiles’ classification performances more consistently than the taxa.

The main benefit of the functional profiles is that they are more in line with the current demands of the medical community [4] when it comes to the required precision level for biological interpretation. A potential caveat however would be the optimal amount of features retained by SPARTA, which greatly varies between both profiles as seen in Table 1, with the amount of annotations retained for optimal classification being often greater than the equivalent for taxa. It seems intuitive that more metabolic functions would characterize unhealthy and control profiles when compared to taxa, however, the total amount of retained information in the case of FAs appears to be too extensive for biological interpretation to be practical in most of our examples. As such, we would recommend that interpretation of the FAs be limited on the first approach to the robust subset outputs, which are in more manageable numbers, though these lists are unlikely to extensively cover all of the features relevant to the characterization of the disease.

SPARTA feature selection.

SPARTA exploits the feature importance rankings that are inherent to the RF method to perform a selection of variables. This selection step impacts classification performance, as discussed in Results, but is also important for the interpretability of the outputs, by highlighting the important elements within an otherwise overwhelmingly large list of features.

Our results highlight the need to perform at least one iteration and several repeated runs to reduce the dimensionality of the functional datasets, while maintaining the classification performance, and derive a list of robust FAs. The number of required iterations depends on both the dataset and the user needs in terms of classification performance and interpretability. Concerning the number of iterations to perform, in this article, we presented results obtained over 10 runs, comprising 5 iterative selections each, and implemented these values as default for SPARTA. These values were chosen as a compromise between execution time and the robustness of the results. S4 and S8 Figs illustrate the impact that a variation of these parameters can have on the results. When it comes to iterative selection, S4 Fig showcases that the first selection is always by far the most important, and there is little variation in selection sizes past the second selection. Therefore, 2 selections could also be perceived as the upper limit by some users, though some of our datasets have shown better classification performance beyond this level of selection. S7 Fig illustrates, in the case of the IBD dataset, that the sizes of both the functional and taxonomic Robust selections stabilize and hit a plateau after only a few runs. In both cases, 10 runs is sufficient to attain stable content for the Robust selection. This conclusion could however only be attained a posteriori, once the results had been obtained. A user may want to reduce the amount of runs operated by SPARTA but should bear in mind that these results may vary depending on the dataset. Generally speaking, SPARTA’s criterion for optimal variable selection is to retain the subset that generates the best classification metric after one variable selection. An automatic test is implemented to ensure that classification performance after one iteration is not significantly lower than the one obtained with the initial dataset. Though this constitutes a strong basis for a first approach, previous works have also warned of it being potentially deceptive and encouraged to investigate the significance of the evolution in performance measurements [23]. As is, the exploration of the Confident subset or the exploitation of a lower level of selection than SPARTA’s proposal could be envisioned by the user if the content included in the recommended Robust output is deemed insufficient. Similarly, a higher level of selection can be exploited if the proposed amount of Robust variables is still overwhelmingly large. Users should also be mindful that the output list may not be as relevant if the classification performances are low.

We also compared SPARTA’s selection to other approaches, as reported notably in Fig 4(A). By relying on an automatically computed cutoff threshold, our approach has proven to be more adaptative and robust than selections based on common fixed thresholds. The relevancy of exploiting RFs to perform selection as opposed to a more direct statistical comparison of unhealthy and control profiles was also highlighted when SPARTA’s selections are compared to those obtained with limma, which measures differential expression. While it proved itself to be an efficient selector on datasets with clear distinguishing features, the latter tool did not detect any candidate features at realistic adjusted p-value thresholds when applied to half of our test datasets and did not have the internal coherence to generate a robust shortlist in two thirds of them. SPARTA on the other hand provided a robust subset for all datasets, showing it to be more consistent than limma when it comes to variable selection, especially in complex problems.

RFs are known to be capable of finding non-linear solutions to a problem [44], which explains the fact that a large amount of the information highlighted by SPARTA, including within the robust subset, remains undetected by limma even when the p-value threshold is unrealistically high, as shown by the results of S5 Fig. As such, the content of SPARTA’s selection includes new information when compared to what can be extracted from linear comparisons.

Classification methods.

Classification performances in the context of FAs have been reported to be on par or slightly inferior to classification performances based on taxa [20, 21]. This is also consistent with our observations. As a result, current FA-based approaches might not be best used for direct diagnostic prediction. The conditions in which a sample has been obtained, sequenced, and processed most likely impact classification performances, even for the same disease (see the differences in performance obtained on T2D and WT2D). The main advantage of current FA-based pipelines, SPARTA included, lies in the extraction of a robust list of important FAs, related to a dataset of interest, rather than the production of a ML model that is generic and directly reusable without need for retraining.

It should also be noted that in this article, only binary classification tasks were tackled. However, the key methods on which SPARTA relies are all compatible with multi-label classification tasks (SVM, RF, evaluation and importance metrics). As such, the pipeline could be compatible with such analyses.

Output interpretability.

SPARTA’s end output is a shortlist of interconnected features, illustrated notably by the examples in Tables 3 and 4. The method emphasizes the selection’s robustness, as it is derived from the consensus of several repetitions, and adaptability, as the threshold for selection is based on an automatic calculation rather than a fixed rank selection. Its content also underwent bibliographic validation, in the case of the IBD dataset’s output. Though the list is likely not exhaustive, SPARTA’s selection was shown to be significantly enriched in bibliographically significant features. This supports SPARTA’s efficacy when it comes to highlighting factors that discriminate health profiles, though this should also be confirmed on the outputs obtained on other diseases.

The previously reported mismatches between the differential score-based profile attributions of SPARTA, which match those of limma, and the conclusions of bibliographic research show that, in all probability, the underlying biological mechanisms involving these pathways are complex enough that a simple differential association is not sufficient to predict if an annotation is beneficial or detrimental to host health in the context of a given disease. A compensation mechanism could also be at play, as the gut microbiota is known to have the potential to compensate for metabolic functions that are lacking in the host [38]. A finer analysis of the RF’s trained decision trees could give more appropriate insight into this issue.

It should also be noted that several annotations couldn’t be directly linked to IBD through bibliography (categories 3 and 4). These features deserve special attention, as they could be the result of a weakness of the method, or novel perspectives for research surrounding the disease.

The combination of SPARTA’s outputs with a visualization method adapted for both of the employed nomenclatures, namely GO terms and EC numbers, would also be a complement to our outputs, allowing for a more intuitive exploration of their biological ramifications. A visualization such as this one could be the basis for an interpretation module for SPARTA.

Exploring links between taxa and functions.

Through Figs 5 and 6, we established the reality of a cumulation effect, with taxa that are less prevalent ending up having a detected influence on the microbiome’s metabolism through their combined contribution to a functional niche. This observation further supports the importance of exploiting microbiota information at the functional level rather than at the taxonomic level. Annotation GO:0006144, which corresponds to the purine metabolic process and is represented in orange in Fig 6, is a good illustration of this approach’s advantages. SPARTA’s outputs show that this annotation was not correlated to any robust taxon, and therefore would be difficult to derive from a taxon-based approach. Indeed, the bibliography shows that this annotation was linked to IBD through oriented research following a first mechanistic study [45], where our approach was capable of identifying it efficiently and without any pre-orientation.

Associating FAs to taxonomic affiliations: The EsMeCaTa pipeline.

The EsMeCaTa pipeline follows three steps. The first step, ‘proteomes’, takes as input a tabular that associates a given name for all the studied bacteria to their exact taxonomy. From this, EsMeCaTa interrogates the UniProt database for proteomes associated with the taxon in question. If none can be found, the step is re-iterated with the superior taxonomic rank, until at least one proteome can be associated with the unit. If more than 99 proteomes are associated with a taxon, a random selection of around 99 proteomes will be made, with respect to the taxonomic diversity of the initial proteomes set. The selected proteomes are then downloaded from UniProt.

The second step, ‘clustering’, selects protein clusters that are representative of the taxonomic unit within the downloaded proteomes. To do so, the MMseqs2 tool [71] is used to create clusters of similar proteins from the proteomes. If a protein cluster contains similar proteins from 95% of the proteomes attributed to the taxonomic unit, it will be retained as part of its meta-proteome.

The final step, ‘annotation’, fetches the FAs (GO terms and EC numbers) of the retained protein clusters by interrogating the UniProt databases. The final output is an ensemble of tabulars, one per taxonomic affiliation in the input, that contains all of the protein clusters kept in the taxon’s meta-proteome and their FAs.

Calculating a functional representation of the patient’s microbiota from taxon-annotation pairings.

To compute a representation of the gut microbiota on the scale of the FAs, mixing information concerning its specific composition with the associated metabolic mechanisms, we give each annotation (F) a score, labeled as a Score of Functional Annotation (SoFA), within a subject sample (i), similarly to [8], according to the following formula: (1) where n_t,i is the abundance value of taxon t within sample i, and x_F,t is the number of proteins within taxon t’s proteome that are linked to the function F.

As such, each annotation’s SoFA is equal to the sum of the abundances of all taxa that express it, weighted by the strength of said expressions, as measured by EsMeCaTa [10].

Normalizing and scaling data based on expected relevance with TF-IGM.

The TF-IGM method [41] is used to normalize the results presented in this article. It was originally exploited in Natural Language Processing, as a method to highlight terms in a corpus of texts that are significantly present within a text while penalizing those that are too widespread. The formula had to be re-adapted to fit our data and circumstances, and in our pipeline, it is calculated based on the following two components:

• TF (Term Frequency): equivalent to the frequency of an annotation within the totality of a sample i: (2) where SoFA_f,i is annotation f’s score within sample i, and J is the ensemble of the annotations recorded within sample i.
• IGM (Inverse Gravity Moment): for each annotation f, the calculated values for tf_f,i are ranked in decreasing order and noted as T(f)₁, …,T(f)_n, so that T(f)₁ > T(f)₂ > … > T(f)_n, n being the total number of samples. We then have: (3) where r is the rank of the T(f) score in the previously defined order.

The total TF-IGM score of an annotation f within a sample i will then be: (4) where λ is a value between 5 and 9. As per Chin et al.’s [41] recommendation, its value was set to 7 by default.

Training of RF models.

A RF [16] classifier is trained to sort individuals in two classes (here, patients or controls), based on the relative abundance profiles of their microbiota or their calculated mechanistic representation. Before any training, a subsample of 20% the size of the full dataset is set aside as a test set. During training, the remaining data is randomly split into a training set and a validation set, with a respective 80% / 20% distribution. To account for the disparity in representation between the unhealthy and control individuals within the datasets, both classes were given weights proportional to their frequency, as implemented by scikit-learn’s ‘balanced’ class weight parameter [62]. Therefore, SPARTA differs from DeepMicro [18] by iterating the variable selection process: it introduces a test set—to evaluate the final performance of the model—and validation sets—to compute the performance of the RFs and derive the variable ranking for selection.

When measuring the performance of our classification algorithms, the metrics used were the Area Under the Receiver Operating Characteristic Curve (AUC) [73] averaged over 20 training iterations. All of the described operations related to the selection of test and validation sets, and the training of RF classifiers in the context of a GridSearch algorithm, are seeded to ensure reproducibility. The initial seed can be changed at the user’s discretion.

Extracting significant information from trained classifiers.

Following the classifier’s training, the resulting feature importances are extracted. These importances can be based on one of two metrics, depending on the user’s input. The first option is the Gini Importance metric, calculating the mean accumulation of the impurity decrease within each tree, as implemented in the Scikit-learn Python library [62]. The other option is the SHAP importance [27], which calculates each variable’s contribution to a decision from the basis of a trained classifier. In our case, dealing with RFs, we relied on the SHAP package’s [27] implementation of the TreeExplainer [74], which is an algorithm for the calculation of SHAP values optimized for RF models. If multiple iterations of the classifier’s training are made, the feature importances are averaged over all iterations. Features are then ranked based on this metric in decreasing order. In SHAP’s case, this ranking is made based on the absolute value of the importance scores.

Once ordered, we aim to distinguish a separation between the features that were essential to the clasifier’s functionality, and those with a lesser impact. We place this threshold at the inflection point of the curve representing the decreasing importance scores, determined via an implementation of the Kneebow method [70], with all features above this point being labeled as “Significant”, and those below as “Non Significant”.

This process is iterated 5 times by default by SPARTA, and the optimal level of selection that is retained is the one that yields the highest median AUC during the classification process over 10 iterations of the pipeline.

Repetition of the SPARTA pipeline.

To obtain robust results, the process of selecting a test subset, training classifiers, and extracting significant features for a set amount of iterations, was repeated over 10 runs in our manipulations. Variations in the training conditions, with different test subsets selected for each run, result in 10 different shortlists of significant features per iteration. We label as ‘Robust’ the features that constitute the intersection of these shortlists, as ‘Confident’ those that are present in 75% or more of them, and as ‘Candidate’ those that are present in at least one of them. The amount of times a variable is labeled as significant by the optimal level of selection is an indicator of how reliable it is for the distinction of the differentiated profiles.