Variant callers.

Seven software packages capable of single nucleotide variant (SNV) calling were evaluated: Samtools [22,23], GATK [14], FreeBayes [11], Strelka2 [6], DeepVariant [13], Octopus [24], and Varscan2 [25]. The selection of these seven variant callers was guided by their use in published studies, open-source availability, and suitability for short-read whole genome and exome sequencing data. They represent a diverse range of variant detection strategies, including heuristic methods, Bayesian inference, haplotype-based modeling, and modern machine learning techniques. This diversity ensures that the comparative evaluation reflects both well-established approaches and state-of-the-art developments in SNV calling, offering insights into their relative performance under consistent benchmarking conditions. All versions used were the latest available at the time of the study. To ensure full reproducibility of analyses, all software versions, command lines, and configurations used for variant calling and benchmarking are publicly available in our GitHub repository (GitHub repository).

Each tool accepts BAM files of aligned reads or pileups generated from BAM files (for example, those created by SAMtools’ mpileup function). For this study, all tools were configured to call variants in tumor-only mode (for somatic variant callers) or with default settings for germline calling, meaning that mutations were identified relative to the reference genome. Each software outputs a variant call format (VCF) file.

Each variant caller comes with preset, yet adjustable, parameters such as minimum variant frequency, minimum coverage, minimum base quality score, and minimum mapping quality score. For analysis, the default recommended settings for each tool were used as showed in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Workflow for comparative performance evaluation of variant callers.

Schematic overview of the benchmarking methodology. Seven variant calling tools (FreeBayes, SAMTools, DeepVariant, GATK, Strelka2, VarScan2, and Octopus) were run on the same sequencing dataset using their default settings. Each tool generated a single Variant Call Format (VCF) file output. These VCF files were then evaluated against a high-confidence gold standard variant set using RTG Tools’ vcfeval function. This benchmarking process produced two primary categories of results: Performance Metrics (Precision, Recall, and F1-score) and detailed Variant Statistics (e.g., Ti/Tv ratio, Het/Hom ratio, indel counts).

https://doi.org/10.1371/journal.pone.0339891.g001

Analysis strategy.

The analysis was performed twice to ensure robustness in variant calling: first on a subset of the data comprising only chromosome 20, and then on the whole-genome sequence. This dual approach allowed a comparison of each tool’s performance and computational efficiency, both within a limited genomic region and across the entire genome. Analyzing chromosome 20 provided a quick assessment of each tool’s accuracy in a smaller region, while whole-genome analysis evaluated scalability and overall SNV detection precision [10,12].

All steps of the analysis were run in parallel, either using built-in multi-threading options (when supported by the tool) or by running multiple instances of each tool in parallel on smaller data chunks.

Due to computational constraints and available hardware, DeepVariant and Varscan2 were unable to complete the runs on the WGS dataset, and Strelka2 was unable to complete the run on the chromosome 20 subset due to difficulties in analyzing segmented genomes.

Performance assessment.

For performance evaluation, the following standard metrics were calculated. The precision is defined as the ratio of true positives (TP) to the sum of true positives and false positives (FP). That is,

(1)

where TP + FP are counts of positives detections.

Similarly, the recall (or sensitivity) is given as the ratio of true positives to the sum of true positives and false negatives (FN):

(2)

with TP + FN are the total number of positives in the data set.

Finally, the F1-score is computed as the harmonic mean of precision and recall:

(3)

which balances the trade-off between precision and recall.

For the NA12878 cell line, the gold-standard variant calls from [21] were used as the reference for true SNVs. The identified SNVs were compared against this reference using RTG Tools’ vcfeval function [26] and VariantEval from GATK. The vcfeval tool facilitates a detailed comparison of VCFs by evaluating the F1-score (see Eq 3) based on counts of true positives (TP), false negatives (FN), and false positives (FP). A variant that matches one variant from the gold standard is classified as a TP; if a variant within the expected genomic intervals is not detected, it is marked as an FN; and any variant detected within the regions but not present in the gold standard is considered an FP. Performance metrics such as precision (Eq 1), recall (Eq 2) and a F1-score (Eq 3), were computed to assess overall performance.

This approach provides a robust means of evaluating variant calling tools, as RTG Tools [26] ensures an accurate comparison of the variant caller outputs with a trusted reference dataset.

Correlation analysis between Chr20 and WGS.

To evaluate whether chromosome 20 benchmarking results predict genome-wide behavior, we computed Spearman’s rank correlation coefficients (ρ) and their two-tailed p-values between chr20 and WGS values for precision, recall, and F1-score across the four variant callers that completed both analyses (GATK, Octopus, FreeBayes, SAMtools). For each metric, chr20 (x) and WGS (y) values were ranked and Spearman’s ρ was obtained with scipy.stats.spearmanr (Python v3.10; SciPy v1.12.0).

All calculations were performed in Python (v3.10) using the scipy.stats.spearmanr function (SciPy v1.12.0). The resulting Spearman rank coefficients (ρ) and two-tailed p-values were reported for each metric. Because only four callers overlapped across scales (n = 4), p-values were interpreted with caution, emphasizing the effect sizes (ρ) and the consistency of direction across metrics.

Computational efficiency.

In addition to variant detection performance, computational efficiency was assessed based on memory usage and runtime required for each tool.

Filtering behavior and total variant calls.

DeepVariant stands out as the most stringent tool, with 59,970 variants failing the filters. This suggests that DeepVariant applies strong quality thresholds to prioritize high-confidence calls and reduce false positives. Octopus and Varscan2 also apply notable filtering (9,080 and 2,896 failed variants, respectively), albeit less stringently. In contrast, FreeBayes and Samtools report zero failed filters, indicating highly permissive behavior. FreeBayes, in particular, detects the highest number of variants (328,360), while GATK and Samtools present a balanced approach with 123,220 and 116,952 passed variants, respectively.

SNP detection.

GATK reports the highest number of SNPs (103,825), consistent with its known strength in identifying single nucleotide variants. Samtools and FreeBayes also detect high SNP counts (101,639 and 100,911, respectively), suggesting a more liberal calling approach. DeepVariant detects the fewest SNPs (85,840), likely due to its prioritization of precision over recall. Octopus and Varscan2 fall in between, with 89,832 and 93,615 SNPs, respectively.

Indel detection.

Regarding indels (insertions + deletions), Octopus detects the highest number (19,605), with a balanced distribution between insertions (10,323) and deletions (9,282). FreeBayes and Samtools follow, with 16,311 and 15,146 indels, respectively. Varscan2 reports lower indel counts (13,247), while GATK (275) and DeepVariant (346) detect very few indels, reflecting their stricter filtering behavior.

Transition/Transversion (Ti/Tv) ratio.

The Ti/Tv ratio is a key SNP quality indicator. DeepVariant achieves the expected human genome value of 2.00, suggesting high-quality SNP calls. Octopus and Varscan2 also present high Ti/Tv ratios (2.07 and 2.03), indicating well-calibrated SNP calling. GATK (1.92) and Samtools (1.93) follow closely, while FreeBayes shows the lowest ratio (1.88), which may suggest a slightly increased rate of false positive transversions.

Het/Hom ratios.

The total Het/Hom ratio indicates the balance between heterozygous and homozygous variant detection. GATK shows the highest ratio (2.50), suggesting high sensitivity to heterozygous variants. FreeBayes (2.39), Varscan2 (2.37), and Octopus (2.07) also display strong heterozygous detection. Samtools has a lower ratio (2.13), while DeepVariant (1.77) appears more conservative, potentially reducing false heterozygous calls.

Insertion/Deletion (In/Del) ratio.

A balanced In/Del ratio near 1.0 is desirable. Octopus (1.11), DeepVariant (0.97), GATK (0.96), and Varscan2 (0.98) maintain well-balanced insertions and deletions. FreeBayes is slightly deletion-biased (0.89), and Samtools shows a mild insertion bias (1.06). These ratios suggest that, except for FreeBayes, all tools handle both indel types with reasonable balance.

Filtering behavior and total variant calls.

FreeBayes reports the highest number of passed variants (12,560,020), substantially more than all other tools, suggesting a highly permissive approach that prioritizes recall by capturing a broader spectrum of potential variants, albeit with an increased risk of false positives. In contrast, GATK, Strelka2, Samtools, and Octopus each report approximately 4.5 to 4.9 million passed variants, indicating the use of stricter filtering thresholds that likely enhance precision.

SNP detection.

Samtools detected the highest number of SNPs (4,119,113), closely followed by GATK and FreeBayes. While Strelka2 and Octopus reported slightly fewer SNPs (3,796,849 and 3,856,679, respectively), this may reflect stricter filtering that improves call confidence. FreeBayes and GATK presented similar SNP counts, highlighting their sensitivity; however, FreeBayes’ liberal calling strategy may introduce more false positives, whereas GATK is designed to maintain a balance between recall and precision.

Indel detection.

FreeBayes reported the highest number of indels (103,179), far exceeding those detected by the other tools. GATK, Strelka2, and Samtools detected around 13,000 indels each, suggesting that these tools share similar stringency levels, potentially due to local realignment strategies that improve indel accuracy. Octopus reported the fewest indels (3,396), reflecting a more conservative and precision-focused approach.

Transition/Transversion (Ti/Tv) Ratio.

Ti/Tv ratios were highest for Strelka2 and Octopus (both 2.02), indicating high-quality SNP calls with fewer false positive transversions. GATK and FreeBayes both had Ti/Tv ratios of 1.95, while Samtools had a slightly lower ratio (1.93), suggesting minor variations in stringency among the tools.

Het/Hom Ratios.

The total Het/Hom ratios were highest for FreeBayes and GATK (both 1.71), indicating a strong ability to detect heterozygous variants. Octopus and Strelka2 showed slightly lower ratios (1.60 and 1.57, respectively), suggesting a more conservative profile. SNP-specific Het/Hom ratios followed a similar trend, with FreeBayes and GATK favoring heterozygous detection more than Octopus, Samtools, and Strelka2.

Insertion/Deletion (In/Del) Ratio.

Most tools showed balanced insertion-to-deletion ratios. Samtools had the most balanced profile (1.01), while GATK (0.95), Strelka2 (0.97), and Octopus (1.09) were close to 1.0, reflecting consistent handling of both variant types. FreeBayes was slightly biased toward deletions (0.88), which may affect downstream interpretation depending on the biological context.

GATK—HaplotypeCaller.

GATK’s HaplotypeCaller is one of the most widely used variant callers in genomics and employs a sophisticated Bayesian framework with graph-based local haplotype assembly to enhance variant detection [14]. The HaplotypeCaller algorithm begins by examining each locus in the genome and then locally assembles overlapping reads into haplotype graphs. This local assembly step enables GATK to detect variants by reconstructing small portions of the genome directly from the reads, which is particularly valuable in complex regions where alignment errors are common. The algorithm computes genotype likelihoods for each haplotype, comparing the observed reads to possible genotype combinations (homozygous reference, heterozygous, homozygous alternate) [14]. It applies a Bayesian model to calculate posterior probabilities, which combines prior information (such as known population allele frequencies) with the likelihood of the observed data given each possible genotype. By maximizing this posterior probability, GATK can select the most likely genotype for each position [14]. GATK’s haplotype-based approach significantly improves its ability to accurately call SNPs, small insertions, and deletions, especially in challenging genomic regions with repetitive sequences or complex variant structures. This approach contributes to its high recall rates (0.9847 in chr20 and 0.9837 in WGS) while maintaining a balanced precision (0.6617 in chr20 and 0.7766 in WGS). The use of local haplotype assembly enables GATK to balance sensitivity and precision effectively, minimizing the proportion of false positives among its predictions and consequently, increasing the number of true positives by recalibrating and realigning reads, thereby improving the confidence in variant calls.

FreeBayes.

FreeBayes uses a haplotype-aware Bayesian model that allows it to detect a wide variety of variant types, including SNPs, indels, and multi-allelic loci [11]. Unlike GATK, which performs local reassembly for each variant, FreeBayes operates directly on the alignment data, looking for mismatches between the reads and the reference genome. FreeBayes is particularly well-suited for multi-sample and polyploid data due to its haplotype-aware approach, which allows it to call variants across multiple samples simultaneously and estimate genotype probabilities based on observed allele frequencies within the population [11]. The Bayesian framework in FreeBayes incorporates prior probabilities based on known allele frequencies and uses a likelihood model that assesses the probability of observing the sequencing data given each possible genotype [11]. This model takes into account sequencing errors, mapping quality, and read depth. However, FreeBayes generally applies less restrictive priors than GATK, leading to a more permissive calling strategy. This permissiveness results in high recall values (0.9758 for chr20 and 0.9748 for WGS), indicating that FreeBayes is highly sensitive and capable of detecting a large number of true variants. However, FreeBayes’s less conservative Bayesian priors contribute to a lower precision (0.6286 for chr20 and 0.7394 for WGS). The algorithm’s tendency to retain variants with lower support makes it more susceptible to calling false positives, especially in single-sample or low-coverage settings where distinguishing real variants from sequencing artifacts becomes challenging. In these cases, FreeBayes may struggle to filter out sequencing errors, resulting in a higher rate of over-calling compared to more restrictive algorithms like GATK. FreeBayes applies less restrictive priors to maximize flexibility and adaptability across a variety of sequencing scenarios, including complex ploidy and heterogeneous populations. In contrast, GATK uses stricter priors to optimize for high-confidence variant calling, particularly in well-characterized and high-quality datasets like diploid human genomes. This difference reflects the contrasting goals of the two tools: FreeBayes prioritizes sensitivity and adaptability, while GATK emphasizes precision and recall.

Octopus.

Octopus is another advanced variant caller that combines Bayesian inference with a hidden Markov model (HMM) for haplotype-based variant calling [24]. Octopus begins by building candidate haplotypes from sequencing reads, then uses an HMM to compute the likelihood of observing each read given each candidate haplotype. The HMM structure allows Octopus to align reads to haplotypes dynamically, adjusting for potential alignment errors and improving its precision in detecting true variants. This haplotype-aware Bayesian framework in Octopus provides high precision and recall by focusing on the likelihood of specific haplotypes rather than individual bases. The algorithm calculates posterior probabilities for each possible genotype by combining the prior probability of the haplotype with the likelihood of the observed data. This Bayesian-HMM approach is particularly advantageous in regions with complex structural variations or high levels of polymorphism, where it helps to reduce false positives by assigning higher confidence to more probable haplotypes [24]. In WGS data, Octopus achieves a high recall (0.9838), demonstrating its sensitivity in detecting true variants even in challenging genomic contexts. This high recall, coupled with its precision (0.7412 in chr20 and 0.8005 in WGS), suggests that Octopus has a balanced approach to variant calling that reduces false positives more effectively than FreeBayes while maintaining high recall. However, Octopus may still exhibit a higher rate of over-calling compared to stricter tools like Strelka2, particularly in simpler regions where the complexity of its HMM model may not be as beneficial. For instance, in cases of low data quality or insufficient sequencing coverage, the full potential of haplotype modeling cannot be effectively realized, as the limited evidence may hinder the accurate reconstruction of haplotypes. Additionally, for straightforward variants, such as SNPs or small indels that are well-supported by sequencing reads, simpler single-variant models can achieve comparable precision without the need for the computational overhead associated with HMM-based approaches. Moreover, in contexts where computational efficiency is a priority, such as WGS studies or high-throughput workflows, the increased complexity of the HMM model may not justify the marginal improvements in precision it provides. This is particularly relevant in repetitive genomic regions or those with low complexity, where the ambiguity in mapping sequencing reads reduces the reliability of haplotype modeling.

In summary, GATK, FreeBayes, and Octopus all use Bayesian inference to integrate prior knowledge with observed data, enhancing their recall. GATK’s local haplotype assembly enables high precision in complex regions, balancing precision and recall by minimizing false positives through realignment and recalibration. FreeBayes, with its permissive priors, achieves high recall but is more prone to false positives, particularly in low-coverage or single-sample data. Octopus combines Bayesian inference with an HMM-based haplotype model, providing both high recall and improved precision in complex regions by dynamically aligning reads to haplotypes. These methodological distinctions contribute to each algorithm’s performance metrics, making them suitable for different genomic contexts and study designs.

Strelka2: Machine learning based filtering with random forest.

Strelka2 is designed for both germline and somatic variant detection and employs a likelihood-based model for variant calling, followed by a sophisticated machine learning-based filtering step. After identifying potential variants, Strelka2 uses a random forest classifier in its final filtration step. The random forest model, a type of ensemble machine learning method, consists of multiple decision trees that collectively evaluate a range of variant characteristics to classify each variant as high-confidence or low-confidence. In the random forest model, Strelka2 considers several features of each variant, such as the confidence that a read is correctly aligned to the reference genome (where higher mapping quality scores indicate more reliable alignments and, consequently, more reliable variant calls). Strelka2 also considers the genomic context around each variant, including whether the region is repetitive or has complex sequence structures that may increase the likelihood of alignment errors, and to handle indel error rates, it applies a binomial mixture model to estimate insertion and deletion error rates, which helps improve the precision of indel calling, especially in noisy regions. By combining these features in a random forest classifier, Strelka2 can assign each variant a probability score representing its confidence level. This machine learning model is particularly effective in reducing false positives by filtering out variants that have characteristics typically associated with sequencing artifacts or errors. The random forest model’s ability to adapt to the specific attributes of each dataset helps Strelka2 achieve high precision (0.8326 in WGS) while maintaining a strong recall (0.9813), resulting in an overall F1-score of 0.9009. This combination of high precision and recall makes Strelka2 highly effective at calling variants, particularly insertions and deletions, in complex regions where traditional filtering methods may struggle. The strength of Strelka2’s random forest classifier lies in its flexibility and adaptability. The model can effectively adjust to variations in sequence quality, strand bias, and genomic context, which allows Strelka2 to maintain high precision even in challenging genomic regions. Its ability to handle multiple variables simultaneously makes it well-suited to reduce false positives while retaining true positives, particularly in data with significant noise.

Varscan2: Statistical and heuristic filtering.

Varscan2 takes a more traditional approach to variant calling, employing heuristic filtering methods along with statistical tests to classify variants, particularly in tumor-normal pair analyses [25]. Varscan2 is well-suited for somatic mutation detection due to its capability to analyze matched tumor and normal samples simultaneously, which allows it to differentiate between germline and somatic mutations. One of the key components of Varscan2’s variant detection process is Fisher’s exact test, which is used to assess the significance of allele frequency differences between tumor and normal samples. Fisher’s exact test calculates a p-value based on the observed read counts for each allele in the tumor versus normal samples, helping Varscan2 determine whether a variant is likely somatic (specific to the tumor) or germline (present in both the tumor and normal samples). This test is particularly effective for identifying somatic mutations in cancer genomics, where somatic mutations appear at varying frequencies depending on tumor heterogeneity [25]. Varscan2 applies several additional filtering criteria to improve the quality of its variant calls, such as considering strand bias, discarding variants that are predominantly supported by reads from only one DNA strand, variant allele frequency, the position of each supporting read, and read depth. High strand bias often indicates sequencing artifacts rather than true variants. Varscan2 evaluates the position of each supporting read within the read itself; variants that are supported primarily by bases at the ends of reads are more likely to be false positives, as these positions are more prone to sequencing errors. For each variant, Varscan2 calculates the variant allele frequency, which is the proportion of reads supporting the variant allele relative to the total read count at that position. In somatic mutation detection, Varscan2 considers variants with significantly different allele frequencies between tumor and normal samples to be potential somatic mutations. Varscan2 requires a minimum read depth at each variant position, ensuring that low-coverage regions, where sequencing noise may obscure true variants, are not called. While these heuristic filters and statistical tests make Varscan2 effective at identifying somatic mutations, they are generally more rigid than Strelka2’s machine learning model, which dynamically adapts to the data. As a result, Varscan2 has a more limited ability to adjust to complex sequence contexts, resulting in lower precision (0.7220 in chr20) and recall (0.9322) compared to Strelka2. Varscan2’s reliance on fixed thresholds and binary filters can be beneficial for high-confidence variant calling in straightforward cases but may lead to reduced recall in more complex regions with high GC content or in regions with lower coverage, where a more flexible model might be advantageous. In terms of performance, Strelka2 achieves higher precision than Varscan2, particularly for indels and complex genomic regions, due to its random forest classifier, which fine-tunes variant calling based on multiple data features [25]. This approach is especially beneficial in scenarios where variant context, quality, and noise levels vary significantly, such as in whole-genome sequencing (WGS) or cancer genomics. Strelka2’s ability to adapt to these variations results in its high F1-score (0.9009 in WGS), making it a preferred choice for applications requiring a high degree of both precision and recall. Varscan2, in contrast, offers a robust but more rigid approach, suited to studies where high-confidence somatic mutations are the focus. Its use of Fisher’s exact test for comparing tumor and normal samples is particularly effective in detecting somatic mutations, providing a statistical basis for variant calling that can effectively filter out germline variants [25]. However, Varscan2’s reliance on heuristic filters limits its adaptability, which may lead to reduced performance in complex or high-GC regions and slightly lower recall. Its F1-score of 0.8137 indicates that while it is effective for somatic mutation detection, it may not capture the full range of variants in challenging regions or at lower frequencies.