Simulated data

We used synthetic datasets for initial model training and testing. The data simulation methodology, depicted in Fig 1, builds on, and significantly expands on our previous work [18]. Reference bacterial and archaeal genomes were selected from Release 202 of the Genome Taxonomy Database (GTDB) [22]. Metagenomes were simulated from publicly available reference genomes via MGSIM (https://github.com/nick-youngblut/MGSIM). Simulation parameters varied in all combinations of i) community richness, ii) community abundance distribution, iii) reference genomes selected from the total pool, iv) read length, v) insert size distribution (i.e., the distance in between the forward and reverse read pair), vi) sequencer error profile, vii) sequencing depth, and viii) metagenome assembler (Table 1). The abundance distribution of each community was modeled as a log-normal distribution. We varied parameter σ to produce differing levels of evenness of relative abundances. Community richness was altered by random sub-sampling from the pool of reference genomes available in the training or test split. The ART [23] read simulator was used to generate paired-end Illumina reads of length 100 or 150 using either the default “Illumina HiSeq 2500” error profile or the “HiSeq2500L150R1/2” error profile used in CAMISIM [24]. Four paired-end read insert size distributions were simulated via the ART parameter settings (see “Insert size” in Table 1). We included multiple simulation replicates, in which the community and read simulation parameters were held constant, but each replicate differed via randomization of the genome sub-sampling within each simulation. The reads from each community were assembled independently with metaSPAdes [3] and MEGAHIT [4].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The ResMiCo simulation and training pipeline.

(A) Select reference genomes from the Genome Taxonomy Database (GTDB) at various abundances; (B) Simulate reads for the selected genomes using ART-Illumina; (C) Assemble reads into contigs using MEGAHIT and metaSPAdes; (D) Align reads back to the assembled contigs using Bowtie2, then extract features such as coverage, number of single-nucleotide variants (SNVs), mean alignment score, etc., for each contig using the given alignments; (E) Compute labels for each contig by aligning against the reference genomes using MetaQUAST; (F) For each contig, select a random section (that contains a breakpoint if the contig is misassembled), pad to the network’s input length if necessary, and feed the data into the ResMiCo model. Steps (D) and (E) are independent and can be parallelized.

https://doi.org/10.1371/journal.pcbi.1011001.g001

MetaQUAST [15] was used to identify truly misassembled contigs based on mapping all contigs to the reference genomes used for the simulations. The MetaQUAST-identified contig misassembly labels (“extensive misassembly”) were used as ground truth. The extensive misassembly label includes relocations, inversions, translocations and interspecies translocations. We also extracted MetaQUAST-identified breakpoint positions in the misassembled contigs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Parameter values used in the simulation pipeline.

The training dataset n9k-train was generated using all 1440 parameter combinations for the insert size distribution with mean = 270 & sd = 50 and mean = 190 & sd = 75. For the “mean = 350 & sd = 75” insert size distribution, 3 replicates for the “HiSeq 2500 L150R1/2” error profile and one replicate for the “HiSeq 2500 error” profile were generated (960 parameter combinations). For the “mean = 450 & sd = 120” insert size distribution, we used all parameters combinations, but only with the “HiSeq 2500 L150R1/2” error profile (720 parameter combinations). In total, we created 4560 samples. The parameter grid was not uniformly sampled due to the resource limitations. The test dataset n9k-novel contains one simulation replicate with the “HiSeq 2500” error profile, and using one (mean = 270 & sd = 50) insert size, which resulted in 240 parameter combinations.

https://doi.org/10.1371/journal.pcbi.1011001.t001

For initial model training and testing, we utilized a pool of 18,000 reference genomes selected from Release 202 of the Genome Taxonomy Database (GTDB). The pool was split at the family taxonomic level so that all genomes in the test dataset belonged to families not present in the training dataset. The resulting split was even, with 9000 genomes used for both training and testing. To reduce bias toward particular species, at most 50 genomes per species were included in the reference genome pool, with genomes selected at random. The pool was also filtered by CheckM-estimated completeness (≥ 90%) and contamination (≤ 5%). Other filtering criteria included: i) only “high” MIMAG quality, ii) no single-cell genome assemblies, iii) ≤ 500 contigs, iv) a genome size of ≤ 15 mbp, and v) a mean contig length of ≥ 10 kbp. We randomly subsampled reads mapped to each contig to a maximum mean contig coverage of 20. At this coverage level, assemblies of reasonable quality can be produced, and subsampling helps prevent out-of-distribution issues when applying ResMiCo to datasets with substantially higher sequencing depth than in our training dataset.

To create features for model training and testing, we mapped reads to the contigs from the corresponding synthetic metagenome via Bowtie2 [25], and the resulting alignment data was used to generate per contig position features, as listed in Table A in S1 Text.

We refer to the training dataset as n9k-train and to the test dataset, which consists of genomes novel at a family taxonomic level, as n9k-novel. All combinations of the metagenome simulation parameters produced a training dataset of 4560 metagenomes and 80M contigs (Table 1). We were limited to 1.3T of available space on our HPC cluster, so we had to randomly subset n9k-train to 3000 metagenomes, comprising 52M contigs (Table 2). The test set (n9k-novel) was generated using the subset of parameters and one simulation replicate to save computational time (Table 1). We also created a test dataset with higher intra-species diversity (33.3 ± 115) versus n9k-novel (2.4 ± 13.7), which we named n2k-novel-intra-species. We used only 2000 reference genomes in order to include enough families with a high number of intra-species genome diversity, but still include only families novel relative to the train dataset (as with n9k-novel). The simulation parameters for n2k-novel-intra-species were consistent with n9k-train, except i) only 50 and 1000 genomes for richness, ii) one simulation replicate, iii) one insert size distribution (mean = 270, sd = 50), and iv) sequencing depths of 2M, 8M, and 12M read pairs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. A summary of the five synthetic datasets used for training and evaluating ResMiCo.

The n9k-train and n9k-novel datasets were generated using our pipeline, all other datasets were created from CAMI reads. The n9k-train dataset was used for training and validation, while all other datasets were used for testing. Misassemblies are reported as a percent of the total number of contigs in the dataset. Misassemblies length is the sum of misassembled contig lengths divided by the total number of bases.

https://doi.org/10.1371/journal.pcbi.1011001.t002

CAMI simulated metagenomes

To benchmark ResMiCo’s performance in a new setting, we downloaded the paired-end reads of the Critical Assessment of Metagenome Interpretation (CAMI) human skin, human oral, human gut, plant-associated, and marine assembly challenges [26]. As with the n9k-train dataset, we assembled the reads via metaSPAdes and MEGAHIT, and identified true misassemblies via MetaQUAST based on the reference genomes in each of the 5 datasets. As shown in Table 2, the number of misassembled contigs in the CAMI datasets is ∼50% lower, while coverage is ∼50% higher relative to the n9k-train dataset. The breakpoint locations for misassembled contigs follow a nearly identical distribution for all datasets, with more breakpoints clustered towards the ends (Fig A in S1 Text).

Published, real-world metagenomes

We evaluated ResMiCo on 7 published metagenome datasets: UHGG [11], TwinsUK [27], Animal-gut [28], Pinnell2019 [29], Mantri2021 [30], MarineMetagenomeDB [31], and TerrestrialMetagenomeDB [32] (S1 Table). The UHGG consisted of randomly selected metagenomes associated with the UHGG MAG collection. TwinsUK consisted of human gut metagenomes from adults in the TwinsUK cohort [27], while the Animal-gut comprised gut metagenomes from a broad taxonomic diversity of vertebrates [28].Pinnell2019 and Mantri2021 consisted of benthic and soil microbial community metagenomes, respectively. MarineMetagenomeDB and TerrestrialMetagenomeDB consisted of a selection of metagenomes from each respective database. The following filtering criteria were applied prior to metagenome selection: i) ≥ 1e6 and ≤ 80e6 reads, ii) maximum read lengths of ≤ 150bp, and iii) ≥ 10 samples in the study, and for the UHGG: iv) ≥ 10 and ≤ 300 MAGs associated with the metagenome.

We also evaluated ResMiCo on 2 mock community datasets: BMock12 [33] and MBARC-26 [34]. We subsampled each metagenome to 2M read pairs in order to evaluate a sequencing depth on par with existing real-world metagenomes [35, 36]. Representative genomes were downloaded from JGI IMG and Genbank for BMock12 and MBARC-26, respectively. These representatives were used in conjunction with MetaQUAST to identify true misassemblies.

Metagenome read data processing was done as described in Youngblut and colleagues [28]. Briefly, reads were validated with fqtools [37]. Adapters were trimmed with Skewer [38]. The “bbduk” command from bbtools (https://sourceforge.net/projects/bbmap/) was used to trim and filter reads based on Phred scores. The “bbmap” command from bbtools was used to filter reads mapping to the hg19 human genome assembly. Read quality reports for each step of the pipeline were generated and visualised with FastQC and MultiQC, respectively (https://www.bioinformatics.babraham.ac.uk/, [39]). Metagenomes were assembled via metaSPAdes with default parameters, and contigs < 1000bp were removed. We did not also assemble the metagenomes with MEGAHIT, given the extra computational expense and methods complexity of evaluating both assemblers for each dataset, along the improved accuracy of metaSPAdes versus MEGAHIT [3]. Read mapping and ResMiCo feature generation were conducted as done for the simulation datasets.

Data preprocessing

Count features were normalized by coverage (the number of reads mapped to the position) such that they are in the 0–1 range. For numerical features, we pre-computed mean and standard deviations using all contigs in the n9k-train dataset and saved these values. For all datasetes, we standardized numerical features to set the mean to zero and the variance to one using values computed on the training set. Missing values were replaced by zero (the new mean). We summarize the preprocessing applied to each feature in Table A in S1 Text.

Since we observed that ResMiCo did not generalize well to insert size distributions substantially deviating from the training dataset (Table E in S1 Text), we excluded metagenomes for which the 0.05 and 0.95 quantiles of the mean insert size distribution lay outside the 0.02 and 0.98 quantiles of the mean insert size distribution of the n9k-train dataset, which are equal to 117 and 493, respectively.

Model and training

Architecture.

The ResMiCo neural network (NN) architecture is shown in Fig 2. It belongs to a class of deep convolutional residual neural networks. Residual connections enable deeper (more layers) neural networks and more efficient training compared to convolutional NNs [20]. Deeper models can capture more complex patterns spread over larger inputs. The main building unit, depicted at the bottom of the figure, is the residual block, consisting of two batch-normalized-convolutions [40] with a ReLu activation. The input of the residual block is connected with the output by simple element-wise addition. Since convolutions are not padded, the last 2*(K − 1) positions in the residual input, where K is the convolution kernel size, are cut off to match the output size. When the residual block is downsampling the data (by using a stride of S = 2 in the first convolution), the residual input is downsampled with a K = 1 convolution of an identical stride.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. ResMiCo architecture.

The input is first passed through multiple convolutional layers; then the convolved result is masked to eliminate the effect of padding and passed through an average pooling layer, followed by two fully connected layers of sizes 128 × 50 and 50 × 1. The convolutional part consists of a simple convolution, followed by four residual groups (RG) with 2, 5, 5, and 2 residual blocks, respectively. The bottom of the figure depicts the structure of a residual block with a given number of features (F), kernel size (K), and stride (S). The first convolution in RG2, RG3, and RG4 halves the input size (using a stride of S = 2) and doubles the number of filters, gradually from 16 to 128. B denotes the batch size, and M represents the maximum contig length. The “14” in “BxMx14” represents the number of the selected features in the input to the neural network. Overall, ResMiCo has 562,573 parameters, of which 559,441 are trainable.

https://doi.org/10.1371/journal.pcbi.1011001.g002

Residual blocks with the same number of filters and identical output shapes are grouped into residual groups. ResMiCo has 4 residual groups, with the center groups consisting of 5 residual blocks each, while the outside groups contain 2 residual blocks. Each of the last three residual groups starts with a convolution that doubles the number of filters and halves the input size using a stride S = 2. All layers use a ReLU activation, except for the last fully connected layer, which uses a sigmoid activation. The output of the convolutional layers is summarized along the spatial axis via global average pooling, resulting in an output shape that depends only on the number of filters in the last convolutional layer (128 in our case) rather than on the contig length, thus allowing ResMiCo to handle contigs of variable length. Contigs in a batch are padded to the longest length, and the effects of padding are neutralized by creating a mask that is fed to the global average pooling layer. The resulting 128 features of the global average pooling are fed into the final two layers: a fully connected layer with 50 neurons and a one-neuron output layer with a sigmoid activation function.

Feature selection.

Since ResMiCo uses a larger number of features than both DeepMAsED and metaMIC, it is important to understand the amount that each feature, particularly features unique to ResMiCo, contributes to model predictions. Borrowed from game theory, Shapley values provide a principled way of explaining the predictions of machine learning models. We approximated the Shapley values using the Deep Shap (SHAP) algorithm [42], a refined version of DeepLIFT [43].

In order to be able to compute SHAP coefficients, we had to make some adjustments to ResMiCo’s architecture: the input size was fixed, the padding was not masked, and the global average pooling layer was replaced by local pooling with a window covering the whole length. SHAP requires as input background samples as well as samples for which the predictions will be explained. We randomly sampled 200 contigs for the background and 200 contigs for explanations (100 correctly assembled and 100 misassembled) from the n9k-novel dataset. In Fig L in S1 Text, we show that SHAP led to the same conclusions when using subsets of the 200 contigs, suggesting 200 is sufficient for this analysis.

Fig 3 shows features ranked by their importance. For comparison, we also marked features present in the ResMiCo pipeline that were used by metaMIC and DeepMAsED. Feature names are explained in Table A in S1 Text. The top 14 features were selected as input to the ResMiCo model. We included at least one feature of each kind: mapping quality, alignment score, etc. Limiting the number of features resulted in significantly reduced training time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Feature ranked by their importance.

The lighter color marks features used by the ResMiCo model. We denote features used by DeepMAsED and metaMIC with a star outline and filled star, respectively. mapq and al_score are mapping quality and alignment score, as defined by Bowtie2. num_snp is the number of SNVs among aligned reads relative to the reference. num_query_[ATGC] is the base composition of aligned reads at the target position. num_orphan is the number of aligned reads in which only one of the pairs aligns properly. num_proper is the number of read pairs that align properly, as defined by Bowtie2. num_proper_snp is properly aligned reads with a SNV relative to the reference at the target position. ref_base is the reference base [ATGC] at the target position. Error bars correspond to the stdev computed over 5 runs.

https://doi.org/10.1371/journal.pcbi.1011001.g003

Comparison to the state of the art

We analyzed the performance of our proposed model in relation to the following existing methods:

• metaMIC [44]—applied with default parameters, except the minimum contig length was reduced from 5 kbp to 1 kbp, in order to test on the same data as the other methods;
• DeepMAsED [18]—we followed the feature generation scheme and the trained model provided by the authors;
• ALE [45]—we aggregated four positional sub-scores that ALE outputs (depth, place, insert, and k-mer log-likelihoods) with the same thresholds defined in [18]. The contig misassembly probability is computed as the number of positions with the sub-score below the threshold divided by contig length;
• Random—we assigned a random misassembly probability to each contig. This results in a horizontal line on a precision-recall curve plot with a precision equal to the prevalence of misassemblies in the dataset.

Since all datasets suffer from a class imbalance in the detriment of positive samples (misassembled contigs, Table 2), we selected the area under the precision-recall curve (AUPRC) as a metric to measure performance, rather than the area under the receiver operator curve (AUROC) [46]. However, AUPRC is not invariant to the prevalence of positive samples, so we used AUROC to compare the model’s performance across datasets with different percentages of positive samples.

Benchmarking ResMiCo resource requirements

We benchmarked ResMiCo with direct comparisons between utilizing one CPU versus one GPU. On the CAMI gut dataset, ResMiCo was > 2x faster with a GPU versus a CPU (108 ± 0.7 versus 38.7 ± 10.3 contigs per second). While a GPU is substantially faster than a CPU, one could still process nearly 140,000 contigs in 1 hour with a single CPU. Nonetheless, we note that multiple GPUs are recommended for training the model on large datasets, given that training on CPUs would not be feasible for large datasets, as in this work.

ResMiCo outperforms existing models and is robust to metagenome novelty

We first tested all models against the n9k-novel dataset, which consisted of family-level taxonomically novel genomes relative to any in the training dataset. ResMiCo outperformed DeepMAsED, ALE, and metaMIC by a large margin, with an AUPRC of 0.76 versus 0.25 for DeepMAsED, the second-best performing model (Fig 4A). Note that DeepMAsED’s AUPRC score dropped from 0.57 (reported in [18]) to 0.25 due to a higher variability within the n9k-novel test set. Importantly, ResMiCo’s AUPRC did not substantially differ between the training validation (0.73) and the n9k-novel dataset (0.76), thereby demonstrating that the model is robust to taxonomic novelty. ResMiCo’s AUPRC score typically varied from 0.6 to 0.8 across the various simulation parameter combinations (Fig H in S1 Text). The most challenging settings for ResMiCo were a low community richness and a low sequencing depth. We also found that the contig length distribution explained much of the variability in ResMiCo model performance. For the simulations with a median contig length longer than 2000bp, the AUPRC was between 0.4 and 0.6 (Fig E in S1 Text). In terms of misassembly type, ResMiCo performance was lowest for inversions (Fig G in S1 Text).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. ResMiCo performance evaluation.

Precision-recall curves and the corresponding AUPRC scores for ResMiCo and four baseline methods (metaMIC, DeepMAsED, ALE, Random) applied on the (A) nk9-novel, (B) CAMI gut, (C) CAMI oral, and (D) CAMI skin datasets. (E) Receiver operating characteristic curve and the corresponding AUROC scores for ResMiCo applied on five datasets: n9k-train (validation set only), nk9-novel, and three CAMI datasets.

https://doi.org/10.1371/journal.pcbi.1011001.g004

We simulated another test dataset (n2k-novel-intraspecies) with more genomes per species (33.3 ± 115) versus n9k-novel (2.4 ± 13.7). ResMiCo performance on n2k-novel-intraspecies declined relative to n9k-novel (AUPRC = 0.487, AUROC = 0.955), which was likely due to an increase in inter-genome translocations among closely related taxa. Still, ResMiCo greatly outperformed the closest competitor: metaMIC (AUPRC = 0.080, AUROC = 0.805).

We next evaluated ResMiCo on the CAMI gut, oral, and skin metagenome datasets, which are commonly used simulation datasets for the evaluation of metagenomics analysis tools. The CAMI datasets differed substantially from n9k-train and n9k-novel in regards to coverage (sequencing depth) and class imbalance (percent misassemblies) (Table 2). Moreover, reference genomes used for the n9k-train and n9k-novel datasets were selected from the entire GTDB, while the CAMI datasets consisted of biome-specific reference genomes [24]. Regardless of these differences, ResMiCo’s performance remained largely unaffected, and the model still clearly outperformed all competitors (Fig 4B, 4C and 4D). Given that the 5 synthetic datasets differ substantially in true positive rates (Table 2), we computed the AUROC score, which is unaffected by such differences. Fig 4E shows that the AUROC remains relatively constant across the n9k-train validation, n9k-novel, and the CAMI datasets.

We also evaluated ResMiCo on two CAMI datasets that simulate non-human biomes: CAMI-marine and CAMI-plant-associated. The performance of ResMiCo was comparable to the human-associated CAMI datasets (CAMI-marine: AUPRC = 0.831, AUROC = 0.990; CAMI-plant-associated: AUPRC = 0.611, AUROC = 0.965), suggesting that ResMiCo can generalize to metagenomes from highly varying biomes.

Lastly, we assessed 2 mock community datasets: BMock12 [33] and MBARC-26 [34]. While existing mock communities are orders of magnitude lower in richness than complex communities, such as soil or mammalian gut microbiomes, and usually completely lack intra-species genomic diversity, mock communities do provide the best available real-world, ground-truth datasets. ResMiCo performance was in line with our expectations based on our simulations of low community richness (Fig H and I in S1 Text), with an AUPRC of 0.475 and an AUROC of 0.951 when averaged across the 2 datasets (Table H in S1 Text). These findings demonstrate that ResMiCo can generalize well to real-world metagenomic data.

Improvement of assembly quality after filtering ResMiCo-identified misassemblies

A primary function of ResMiCo is to identify misassembled contigs so that they can be removed from the assembly. To illustrate the effects of such filtering, we discarded contigs in n9k-novel with a ResMiCo score of >0.8, which corresponds to a recall and precision of 0.72 and 0.65, respectively. Given that truly misassemblied contigs are known for the n9k-novel dataset, we could measure the true error rate before and after filtering. Filtering according to ResMiCo scores resulted in a reduction of the true error rate from 4% to 1% while keeping the contiguity metrics virtually unmodified (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Low-quality contig filtering.

Statistics before and after filtering low-quality contigs with the ResMiCo model applied on the n9k-novel test set. The ResMiCo score threshold was set to >0.8. Misassemblies length is the sum of misassembled contig lengths divided by the total number of bases.

https://doi.org/10.1371/journal.pcbi.1011001.t003

We also evaluated whether the MetaQUAST-defined “genome fraction” (the percentage of aligned bases in the reference genome) substantially declined as a result of filtering ResMiCo-identified misassemblies. For both the CAMI-gut and CAMI-marine datasets, the genome fraction did not substantially change after filtering (Wilcox, P ≥ 0.66; Fig K in S1 Text).

Assembly optimization based on ResMiCo-identified error rates

Since ResMiCo generates a score for each contig, we could use the number of contigs with a score above a certain threshold to estimate the misassembly rate of a given contig set (we used >0.8 in our experiments). The estimated misassembly rate could then be used to optimize metagenome assembler parameters (e.g., k-mer lengths) for real metagenomes, which lack ground-truth data. Assembler hyperparameters are generally optimized simply based on total contiguity (e.g., N50) or possibly via CheckM after binning contigs into MAGs. However, such methods do not directly assess contig assembly accuracy. In order to use ResMiCo for this application, model performance must be robust to assembler hyperparameter settings outside of the training distribution.

We tested ResMiCo’s performance as an oracle for assembler performance by simulating datasets in a similar fashion as n9k-novel, but with 6 different k-mer length hyperparameter settings for both MEGAHIT and metaSPAdes (see Methods). For each of the 6 k-mer length combinations, we generated akin to n9k-novel but utilized only 2 community richness (50 and 3000) and 2 sequencing depth (2M and 8M) settings. The rest of the simulation parameters were fixed: genome abundance distribution with σ = 1, read lengths of 150 bps, insert size distribution of mean = 270 & sd = 50, and the “HiSeq 2500 error” error profile. The percentage of actual misassembled contigs differed from <1% to 30% depending on the assembler and the chosen k-mer set applied on the same set reads (Fig 5). We then compared the percentage of misassembled contigs with the percentage estimated by ResMiCo for each of the four community richness/sequencing depth combinations (6 k-mer sets per combination).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Misassembly (error) rate produced by MEGAHIT and metaSPAdes assemblers with six different k-mer sets.

k-mer set names denote the k-mer lengths used for the assembly. (A) ResMiCo-identified error rate (y-axis) correlates with the true error rate. (B) N50 size and the true error rate are orthogonal measures of the metagenome quality.

https://doi.org/10.1371/journal.pcbi.1011001.g005

ResMiCo was able to accurately rank the assemblies by misassembly rate for all four simulation parameter combinations, achieving a Pearson correlation of 0.9. (Fig 5A). While ResMiCo has a tendency to overestimate the misassembly rate with a selected prediction threshold, and the ratio between predicted and true misassembly rate depends on sample richness and sequencing depth (Fig J in S1 Text), the ranking remained consistent in all considered scenarios. At the same time, for the most well-assembled metagenomes (low richness and high sequencing depth), we observed a correlation of 0.9 between N50 and true error rate (Fig 5B), which suggests that the high contiguity achieved together with high misassembly error rate. However this relationship does not hold for the samples simulated with other parameters, making possible to search for assembler parameters that produce good quality in terms of contiguity and error rate simultaneously. Consequently, we propose that ResMiCo can be used to rank assembler parameters for real-world metagenome data and identify parameters leading to the lowest misassembly rate.

Latent space visualization

To get an intuition on how ResMiCo internally represents data, we studied the output of the global average pooling layer. At that point, the input data is mapped into a 128-dimensional space. We used UMAP [49] with default parameters to project the embeddings into a two-dimensional space. UMAP was fitted on the n9k-train, n9k-novel, and CAMI-gut datasets. We used 10,000 randomly sampled contigs from each of the three datasets.

The latent space visualization indicates that n9k-train has more variability (due to the extensive set of parameters used in the simulations) than CAMI-gut, which is concentrated in a small subspace, while misassembled contigs from both datasets are generally clustered together (Fig 6A and 6B). Note that since ResMiCo has two fully connected layers following the visualized global average pooling, the two classes are not expected to be completely separable at this stage. Both community richness and average contig coverage strongly partition the latent space (Fig 6C and 6D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Contig embeddings learned by ResMiCo, projected using UMAP.

The first row shows (A) all contig embeddings and (B) misassembled contig embeddings for the n9k-train, the n9k-novel, and the CAMI. gut datasets. In the second row, contigs from the n9k-train dataset are colored (C) by the richness of the simulated community they originated from and (D) by their average coverage.

https://doi.org/10.1371/journal.pcbi.1011001.g006

ResMiCo detects a 3–12% misassembly rate in real-world metagenomes

We applied ResMiCo to published gut metagenome datasets from multiple studies in order to assess the prevalence of misassembled contigs in publicly available metagenomic data. We utilized three datasets of gut samples: UHGG, TwinsUK, and Animal-gut. UHGG consisted of a random subset of gut metagenomes associated with MAGs in the UHGG database, while TwinsUK and Animal-gut consisted of gut metagenomes from westernized adults and a broad taxonomic diversity of vertebrates, respectively (see Materials and methods). We also utilized 2 marine meatgenome datasets (Pinnell2019 and MarineMetagenomeDB) and 2 from soil (Mantri2021 and TerrestrialMetagenomeDB).

ResMiCo detected an average of 3.4%, 6%, and 8.8% misassembled contigs across all metagenome assemblies in the vertebrate-associated datasets: TwinsUK, Animal-gut, and UHGG, respectively. The marine and soil datasets contained more misassemblies (MarineMetagenomeDB: 8.3%, Mantri2021: 11.5%, Pinnell2019: 8.6%, and TerrestrialMetagenomeDB: 12.5%). Overall, we evaluated 8,235,502 contigs, of which 6% are misassembled according to ResMiCo’s predictions (≥ 0.8).

We provide an estimated the misassembly rate for each metagenome in S1 Table. Averaged across all metagenomes, the misasssembly rate was 7% ± 5.7 (sd). The high variability among samples and datasets suggests that sample-specific factors (e.g., community taxonomic complexity or variability among NGS library preparations) can substantially influence misassembly rates. Note that we did not apply ResMiCo on the minority of metagenomes that failed to meet our inclusion criteria (see the Materials and methods).