3.1 Overview

Truvari is one of the most widely adopted SV comparison tools and serves as a reference standard in more than two hundred benchmarking studies [63–67]. It is designed to compare two VCF files and determine matching SV pairs based on configurable similarity criteria while preserving allelic diversity [68]. Truvari enables detailed comparison of the SV query set with respect to the truth set, calculating standard performance metrics precision, recall, and F1-score along with variant lists including true positives, false positives, false negatives. As with all comparison tools, Truvari’s performance depends on the quality of input sets and the appropriateness of matching parameters for the specific analysis context.

3.2 Matching strategy

Truvari (bench) performs pairwise matching by first identifying SVs that fall within a specified genomic distance and, incorporates haplotype information to refine candidate matches. When multiple candidates exist, the priority is given to the match with the best score. After identifying candidate pairs, multiple similarity constraints are applied, including breakpoint distance tolerance, size similarity, reciprocal overlap, and sequence similarity of alternative alleles.

This behavior is governed by key parameters such as --refdist (default: 500 bp), which defines the maximum allowed distance between SV breakpoints for candidate pairing; --pctsize (default: 0.7), which requires the relative size difference between variants to fall within a specified similarity ratio; --pctovl (default: 0.0), enforcing a minimum reciprocal overlap between SV intervals; and --pctseq (default: 0.7), which specifies the minimum sequence similarity between alternative alleles. An SV pair is considered a valid match only if all the constraints are satisfied. These parameters collectively define the permissiveness of the comparison. Relaxed thresholds increase recall but may inflate false positives, while strict thresholds favor precision at the cost of missed matches.

4.1 Overview

EvalSVcallers is designed to evaluate SV caller performance across standardized datasets [3]. EvalSVcallers enables comparison of SV query and truth sets based on reciprocal overlap, calculates standard performance metrics, including precision and recall, and reports true-positive and false-positive variant lists. A curated NA12878 reference SV dataset (including long-read and assembly-based datasets) is embedded within EvalSVcallers which also supports custom truth sets as input.

4.2 Matching strategy

Evaluation of SV callers in EvalSVcallers involves two phases. First, the output for each caller is converted to a standard form, then filtered by SV type and size restrictions. The determination of a match for the non-insertion SVs is based on the reciprocal overlap between called and reference variants, while insertions are evaluated strictly based upon their positions. The comparison is performed using a predefined truth set and evaluation metrics are computed for each SV type and size class.

This behavior is governed by key parameters such as --min_ovl (default: 0.5), which specifies the minimum required reciprocal overlap between called and reference non-INS SVs for a valid match; --sv_type (default: ALL), restricting the comparison to selected SV classes; --len and --xlen (default: 50 bp and 2000000 bp), defining the minimum and maximum sizes of called SVs; and --ref_len and --ref_xlen (default: 50 bp and 2000000 bp), applying equivalent size constraints to reference SVs. An SV is considered a match if it satisfies the overlap requirement (where applicable) and falls within the defined size and type filters, resulting in a binary matched–unmatched decision without additional breakpoint or sequence-level similarity constraints.

5.1 Overview

SVbenchmark is part of the SVanalyzer toolkit and provides an alternative strategy for SV comparison based on haplotype sequence comparison [69]. SVbenchmark enables comparison of the SV query set with respect to the truth set, calculates standard performance metrics, including precision, recall, and F1-score, and reports distance computations as well as false-positive and false-negative variant lists.

5.2 Matching strategy

SVbenchmark evaluates SV concordance by reconstructing and comparing alternate haplotype sequences implied by the query and truth sets. Candidate variant pairs are first screened based on maximum distance threshold, after which similarity is evaluated using normalized measures that account for positional shifts, size and sequence differences between alternate alleles.

This behavior is governed by key parameters such as --maxdist (default: 100000 bp), which restricts matches to variants whose genomic positions lie within a defined distance; --normshift (default: 1.0), which limits the allowable normalized alignment shift between alternate alleles; --normsizediff (default: 1.0), which constrains the normalized size difference of the reconstructed alleles; --normdist (default: 1.0), which sets the maximum permitted normalized edit distance between alternate haplotype sequences; and --minsize (default: 0 bp), which filters variants below a minimum length. An SV is considered a match if the reconstructed alternate haplotypes satisfy all normalized similarity constraints, yielding a binary matched–unmatched outcome based on sequence-level equivalence.

7.1 Samples and sequencing data

In this study, for HG002, the approximately 50 × pre-aligned BAM file was obtained directly from the NIST Genome in a Bottle Ashkenazim Trio repository. The reads were aligned to the Human UCSC hg19 reference genome using BWA v0.6.1-r104-tpx-isis within the BaseSpace BWA Whole Genome Sequencing workflow v1.0.0.1. PCR-duplicate flagging was enabled which also incorporated SAMtools v0.1.18 and GATK v1.6-22-g3ec78bd. For NA12878, paired-end whole-genome sequencing reads from SRA accession SRR17658585 were aligned to GRCh38 reference genome using BWA-MEM2 v2.2.1. The resulting alignments were converted to BAM format and coordinate-sorted using SAMtools v1.13. The final BAM contained no reads flagged as duplicate.

7.2 SV calling

Structural Variant call sets were generated independently for HG002 and NA12878 using the same caller versions and sample specific reference assemblies. Manta v1.6.0 was configured in diploid mode using the sample specific BAM and corresponding reference FASTA. Delly v1.2.6 was executed in call mode with the corresponding reference genome supplied through the -g option. Lumpy v0.2.13 was run through the lumpyexpress workflow; discordant paired-end and split-read alignments were extracted separately and supplied together with the full BAM, and the resulting calls were genotyped using SVTyper v0.7.1. GRIDSS v2.13.2 was executed using the gridss.CallVariants module with the sample specific BAM and reference genome. Wham v1.7.0-311-g4e8c-dirty was run with the sample specific BAM and reference genome supplied through the -f and -a options, respectively. The resulting caller VCFs were subsequently subjected to the common preprocessing and representation standardization workflow described below.

7.3 Data preparation and genomic context standardization

All five caller VCFs (Manta, Delly, Lumpy, GRIDSS, and Wham) were first passed through the OctopuSV v0.3.1 [79] representation-standardization workflow, including the conversion of BND records into standardized SV representations where applicable. Following caller-output standardization, common filtering criteria were applied to both caller and truth set VCFs. The standardized caller outputs and truth sets were subsequently restricted to autosomal chromosomes (chr1–chr22), PASS records, DEL and INS variants, and SVs with an absolute length ≥50 bp. This design ensures that all observed differences in the outputs of the comparisons are a result of the internal comparison logic of each framework rather than from variations in input data. Additionally, variants in HG002 were restricted to the high-confidence benchmark regions defined by the corresponding truth set BED file, thereby limiting the evaluation to the validated benchmark domain. No equivalent high-confidence BED file is currently available for NA12878. Because the genomic evaluation domains differed between the two samples, cross-sample differences are presented only descriptively and should not be interpreted as performance comparisons. Caller-specific record counts before and after OctopuSV standardization and subsequent preprocessing steps are provided in Supplementary Table D in S1 Appendix. The resulting standardized and filtered caller and truth set VCFs were then used as common inputs to the three benchmarking frameworks within each sample.

7.4 SV benchmarking

Benchmarking was performed independently for HG002 (GRCh37) and NA12878 (GRCh38), both using approximately 50 × whole-genome sequencing data. For each sample, the caller specific query sets described above were evaluated against the same corresponding truth set using Truvari v3.0.0, EvalSVcallers, and SVbenchmark through SVanalyzer v0.36.

7.5 Genomic block resampling

Genomic block resampling analysis was performed to assess how stable the default benchmarking results were across genomic regions. Autosomal chromosomes were divided into separate, non-overlapping 5 Mb genomic blocks for each reference genome/sample combination. Block files were created separately for each sample because NA12878 and HG002 were evaluated using different reference assemblies and chromosome lengths. 200 resampling iterations were generated for each sample. In each iteration, 80% of the autosomal genomic blocks were randomly selected without replacement. Sampling without replacement was used since the classic bootstrap approach could lead to duplicate VCF entries and affect the variant matching process. Therefore, this approach should be interpreted as a bootstrap-inspired genomic block resampling strategy instead of the classic bootstrap. In each resampling iteration, the same selected genomic blocks were applied to both the caller VCF file and the corresponding truth set. Thus, in each iteration, the truth and caller variants were evaluated on the same genomic regions. Each subset was then re-benchmarked using the default configurations of Truvari, SVbenchmark, and EvalSVcallers. This approach aimed to capture the sensitivity of benchmarking metrics to regional genomic composition and framework specific matching behaviors, as it re-runs the variant matching process within each genomic subset. For each sample, the mean, standard deviation, median, and empirical 2.5% to 97.5% percentiles of precision, recall, and F1-score values for the caller and benchmarking framework combination were calculated over 200 resampling iterations.

7.6 Baseline TP, FP, and FN characterization

Baseline benchmarking outcomes were additionally characterized at the variant level according to SV type and size. Variants were stratified as DEL or INS and into five size intervals: 50–99 bp, 100–499 bp, 500–999 bp, 1–9.9 kb, and ≥10 kb. FP and FN rates were summarized separately within each sample–caller–framework–SVTYPE–SVLEN stratum.

7.7 Parameter sensitivity analysis

Parameter sensitivity was performed using a systematic one-factor-at-a-time (OFAT) design in which one evaluation parameter was varied while all remaining parameters were held at the study baseline. Truvari and SVbenchmark were each evaluated using the baseline configuration and 16 alternative configurations, whereas EvalSVcallers was evaluated using the baseline and 8 alternative configurations. Across two samples and five SV callers, this resulted in a total of 430 sample-caller-framework-configuration evaluations. For Truvari, the baseline configuration was refdist = 500, pctsize = 0.7, pctseq = 0.7, and pctovl = 0.0. The OFAT analysis evaluated refdist values of 100, 250, 750, and 1000 bp; pctsize values of 0.5, 0.6, 0.8, and 0.9; pctseq values of 0.5, 0.6, 0.8, and 0.9; and pctovl values of 0.2, 0.4, 0.6, and 0.8. For SVbenchmark, the baseline was maxdist = 100000, normshift = 1.0, normsizediff = 1.0, and normdist = 1.0, with minsize = 0 held fixed. The alternative configurations evaluated maxdist values of 1000, 10000, 50000, and 200000 bp and independently varied each normalized criterion across 0.1, 0.3, 0.5, and 0.7. For EvalSVcallers, the baseline matching parameters were min_ovl = 0.5 and min_ins = 200. The xlen and ref_xlen parameters were held fixed at 2000000 bp because they define caller-side and truth-side SV-size limits for the evaluated domain rather than the pairwise variant-matching criterion. The OFAT analysis therefore focused on min_ovl and min_ins, evaluating min_ovl values of 0.3, 0.4, 0.6, and 0.7 and min_ins values of 50, 100, 300, and 500 bp. Sensitivity was quantified primarily using absolute changes in F1-score relative to the baseline (ΔF1) and parameter-specific F1-score ranges. This approach avoids disproportionately large percentage differences when the baseline F1-score is close to zero.

7.8 Variant level bootstrap analysis

Statistical assessment of parameter-associated F1-score changes was performed using paired variant level bootstrap resampling. For each sample-caller-framework combination, caller-side and truth-side variants were resampled independently with replacement in 5000 bootstrap iterations (using the variant-level TP/FP and TP/FN classifications obtained from the original benchmarking results). Caller-side TP/FP outcomes were used to recalculate precision, while truth-side TP/FN outcomes were used to recalculate recall. Within each bootstrap iteration, the same resampled caller indices and the same resampled truth indices were applied to both the baseline and the corresponding alternative configuration. This preserved a paired comparison between parameter settings while allowing the composition of the evaluated variants to vary across bootstrap iterations. A new F1-score was calculated for the baseline and alternative configuration in each iteration, and the difference was defined as ΔF1 = F1_alternative − F1_baseline. The 2.5th and 97.5th percentiles of the 5000 ΔF1 values were used as the empirical 95% bootstrap interval. A parameter-related change was considered bootstrap-supported when this interval did not include zero, indicating that the direction of the F1-score change remained consistent across the resampled variant sets.

7.9 Cross-framework variant level characterization

To enable variant level comparison across benchmarking frameworks, a unified query-variant universe was constructed for each sample-caller combination from the union of query-side variant records returned by the three frameworks. Framework-specific TP and FP outcomes were retained for each query variant, whereas variants for which a framework returned no outcome were assigned the label NONE. NONE was used only as a cross-framework bookkeeping category indicating the absence of a reported outcome and does not represent a classification generated by the benchmarking framework. To further characterize the variant properties associated with cross-framework agreement and disagreement, query variants in each unified variant universe were stratified according to the number of benchmarking frameworks that classified the same variant as a TP. Variants were categorized as TP in all three frameworks, TP in exactly two frameworks, or TP in only one framework. For each query variant, the nearest truth-set variant of the same SV type on the same chromosome was identified using absolute start-position distance. Breakpoint displacement was calculated as the absolute difference between query and truth start positions. Size agreement was summarized using the ratio of the smaller to the larger absolute SV length, with values approaching 1 indicating greater size concordance. For DEL variants, reciprocal overlap was additionally calculated as the smaller of the fractions of the query and truth intervals covered by their intersection. INS variants were characterized using breakpoint displacement, absolute size difference, and size ratio. Sequence similarity was not included as a common cross-framework characterization metric because an equivalent framework-selected sequence-similarity measure was not available consistently from all three benchmarking outputs. Using a sequence-similarity value derived from only one framework would make the common characterization dependent on that framework’s internal matching procedure. Similarly, one-to-one, clustered, and potentially one-to-many representation effects were not quantified as a common metric because these behaviors are handled differently internally by the three frameworks and cannot be reconstructed uniformly from their final outputs. Cross-framework TP agreement groups were additionally summarized across the same five query-variant size intervals used for the baseline FP/FN characterization (50–99 bp, 100–499 bp, 500–999 bp, 1–9.9 kb, and ≥10 kb). Categories containing fewer than 10 variants were retained for completeness but were not used for distribution-level interpretation since summary statistics from these small groups may be disproportionately influenced by individual variants. Representative loci were selected from predefined cross-framework agreement patterns to illustrate contrasting forms of concordance and discordance. Within each targeted pattern, the variant closest to the group median characteristics was selected using breakpoint displacement, size agreement, and reciprocal overlap where applicable, with feature distances scaled by their interquartile ranges to prevent differences in numerical scale from dominating the selection.

7.10 Reproducibility

All preprocessing, benchmarking, sensitivity, resampling, and post-processing steps were documented at command level to support reproducibility. The repository includes centralized configuration files, truth set and caller preprocessing workflows, framework specific baseline evaluations, systematic parameter sensitivity analysis, genomic block resampling, paired variant level bootstrap analysis, baseline TP/FP/FN characterization, cross-framework variant level characterization, figure and supplementary table generation scripts, environment definitions, and repository level validation utilities. The principal repository components and their roles are summarized in TableC in S1 Appendix (S1 Appendix).

8.1 Characteristics of the preprocessed benchmarking datasets

Following preprocessing, the HG002 truth set contained a total of 28023 variants (10662 DEL, 17361 INS), whereas the truth set of NA12878 consists of a total of 15296 variants (5698 DEL, 9598 INS). Figs 1 and 2 illustrate distributions of SV types in the query sets compared to HG002 and NA12878 truth sets respectively. Figs 3 and 4 depicts distribution of SV lengths of query sets compared to HG002 and NA12878 truth sets respectively. The box plot summarizes the distribution of absolute SV length (SVLEN) values for both caller-derived query callsets and high-confidence truth sets on a logarithmic scale, showing the median, interquartile range, and outliers. The SVLEN distributions indicate that, for deletions, most callers were located within a broadly similar range to the truth set (log10 ~ 2.0–2.5). Manta showed the closest median to the truth set in both samples, whereas Lumpy, GRIDSS, and Wham showed somewhat higher median deletion lengths. For insertions, the differences were more pronounced: Delly and Lumpy reported substantially larger insertion lengths, with median values extending above the range observed in the corresponding truth sets.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Distribution of SV types (SVTYPE) for HG002.

https://doi.org/10.1371/journal.pcbi.1014824.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Distribution of SV types (SVTYPE) for NA12878.

https://doi.org/10.1371/journal.pcbi.1014824.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Distribution of SV lengths (SVLEN) for HG002.

https://doi.org/10.1371/journal.pcbi.1014824.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Distribution of SV lengths (SVLEN) for NA12878.

https://doi.org/10.1371/journal.pcbi.1014824.g004

8.2 Baseline benchmarking performance

SV calls from Manta, Delly, Lumpy, GRIDSS, and Wham on the HG002 and NA12878 samples were evaluated using the Truvari, EvalSVcallers and SVbenchmark. Fig 5 presents the aggregate bar charts of precision, recall, and F1-score values for each sample-caller-framework combination under default framework configurations. The figure reveals both common performance trends and framework-dependent differences in the absolute metric values assigned to the same caller outputs. In the HG002, precision values generally remained higher than recall values in all frameworks. Specifically, for Manta, precision values were obtained as 0.9015 in Truvari, 0.9315 in EvalSVcallers and 0.9315 in SVbenchmark while recall values remained at 0.2319, 0.0907 and 0.2743 respectively. Similarly, for the other callers, precision was found to be relatively high, while recall was lower leading to medium or low level F1-scores. For example, the F1-score for Manta was calculated as 0.3689 in Truvari, 0.1653 in EvalSVcallers and 0.4238 in SVbenchmark. The same general trend was observed for Delly, Lumpy, GRIDSS, and Wham, but the absolute values of the metrics varied depending on the caller and framework. The same framework-dependent variation in precision, recall, and F1-score was also observed in NA12878. For Manta, precision values were calculated as 0.4344 in Truvari, 0.4718 in EvalSVcallers and 0.4595 in SVbenchmark while recall values were obtained as 0.1976, 0.0890 and 0.2135 respectively. In contrast, F1-scores remained at 0.2717, 0.1498 and 0.2915. Delly, Lumpy, GRIDSS, and Wham results similarly showed limited performance in terms of both precision and recall. In particular, recall values were quite low for Delly and Wham, which significantly reduced F1-scores. Overall, the three frameworks showed broadly similar precision–recall trends across callers, while the absolute precision, recall, and F1-score values assigned to individual caller outputs varied across frameworks.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Performance metrics under default configurations for HG002 and NA12878.

https://doi.org/10.1371/journal.pcbi.1014824.g005

8.3 SV type specific macro-F1 estimation

Fig 6 provides a type-specific evaluation of the baseline results. In this analysis, DEL and INS F1-scores were calculated separately for each caller and benchmarking framework. Macro-F1 was then calculated as the mean of DEL-F1 and INS-F1. DEL-F1 was generally higher than INS-F1 across most callers and benchmarking frameworks. Under Truvari and Svbenchmark, Manta showed the highest Macro-F1 values in both samples, whereas Delly, Lumpy, GRIDSS, and Wham generally showed lower Macro-F1 values, largely due to limited INS performance. Under EvalSVcallers, however, the highest Macro-F1 was observed for Delly in HG002 and GRIDSS in NA12878. The heatmap also shows that benchmarking framework choice affected the absolute scores assigned to the same caller, although the overall caller ranking was broadly consistent across frameworks. These findings support the interpretation that SV benchmarking results are also influenced by SVTYPE-specific detection behavior.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Benchmarking framework effects on type specific SV caller performance.

https://doi.org/10.1371/journal.pcbi.1014824.g006

8.4 Baseline TP, FP, and FN characterization

The analysis showed that relatively high precision could coexist with low recall because a comparatively low FP burden could occur simultaneously with a much larger FN burden (Figs A and B in S1 Appendix). This pattern was particularly clear for HG002-Manta. Under Truvari, FP rates for DEL variants were 0.11, 0.12, 0.10, 0.05, and 0.08 across the five size intervals, whereas the corresponding FN rates were 0.69, 0.62, 0.66, 0.24, and 0.47. A similar imbalance was observed for smaller Manta insertions: FP rates were 0.07, 0.07, and 0.18 for the 50–99 bp, 100–499 bp, and 500–999 bp intervals, respectively, while the corresponding FN rates increased from 0.75 to 0.86 and 0.96. SVbenchmark showed the same general pattern, with relatively low Manta FP rates but substantially larger FN rates across several DEL and INS size intervals. Thus, the high-precision/low-recall profile was not explained simply by an excess of false-positive calls; in several strata, the dominant imbalance arose because a substantially larger proportion of truth variants were classified as false negatives. The magnitude and distribution of FP and FN rates nevertheless remained strongly caller- and framework-dependent. For example, HG002 Wham INS variants showed FP rates approaching 1.00 across most size intervals under Truvari and SVbenchmark, whereas EvalSVcallers produced substantially lower FP rates in several of the same size categories (Fig A in S1 Appendix). FN rates for Wham INS remained high under all three frameworks, although their exact values also differed by size and framework (Fig B in S1 Appendix). Overall, the combined FP and FN profiles show that error burdens vary jointly with SV type, SV size, caller, and framework-specific variant-equivalence rules.

8.5 Genomic block resampling for default-configuration uncertainty analysis

Genomic block resampling analysis demonstrated that the default benchmarking results were generally stable across the resampled genomic regions. Empirical F1-score ranges were relatively narrow in most sample-caller-framework combinations. Across 200 resampling iterations, Manta consistently achieved the highest average F1-score under both Truvari and SVbenchmark for both HG002 and NA12878 (Fig 7). In contrast, GRIDSS ranked highest in both samples under EvalSVcallers. The narrow empirical F1-score intervals observed in the genomic block-resampling analysis indicate that default benchmarking results were not highly sensitive to the random selection of genomic regions. In other words, caller rankings were generally stable within each benchmarking framework across different genomic subsets. However, the highest-ranking caller changed between frameworks: Manta ranked highest under Truvari and SVbenchmark, whereas GRIDSS ranked highest under EvalSVcallers. This pattern suggests that the observed differences were not primarily driven by regional genomic composition, but by systematic differences in the matching and scoring logic of the benchmarking frameworks. Therefore, benchmarking framework choice itself represents an important source of variation in SV caller evaluation.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Default configuration F1-score uncertainty estimated using genomic block resampling.

https://doi.org/10.1371/journal.pcbi.1014824.g007

8.6 Parameter sensitivity uncertainty analysis

Precision, recall, and F1 scores across all evaluated configurations are shown in Fig 8. Across the evaluated sample-caller-parameter groups, SVbenchmark showed the largest mean parameter-specific F1 range (0.0136), followed by Truvari (0.0086) and EvalSVcallers (0.0053). However, the largest individual parameter-associated deviation was observed with Truvari for NA12878-Lumpy. Increasing pctseq from its baseline value of 0.7 to 0.9 reduced F1 from 0.1163 to 0.0455, corresponding to ΔF1 = −0.0708. Across the tested pctseq values, F1 decreased progressively from 0.1216 at 0.5 to 0.0455 at 0.9. Similar directional responses to increasing pctseq were observed for HG002-Manta and HG002-GRIDSS (Fig 9).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Performance metrics across parameter configurations for HG002 and NA12878.

https://doi.org/10.1371/journal.pcbi.1014824.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Truvari F1-score sensitivity across tested parameter configurations.

https://doi.org/10.1371/journal.pcbi.1014824.g009

SVbenchmark showed a particularly consistent response for Manta across both samples. In HG002, reducing normdist, normshift, and normsizediff from the baseline value of 1.0 to 0.1 decreased F1 by 0.0538, 0.0507, and 0.0503, respectively. The corresponding decreases in NA12878 were 0.0441, 0.0393, and 0.0388. For example, the HG002–Manta F1-score increased progressively from 0.3700 at normdist = 0.1 to 0.3946, 0.4079, and 0.4161 at values of 0.3, 0.5, and 0.7, respectively, approaching the baseline F1-score of 0.4238 at normdist = 1.0. A comparable directional pattern was observed for normshift and normsizediff and was reproduced in NA12878 (Fig 10).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. SVbenchmark F1-score sensitivity across tested parameter configurations.

https://doi.org/10.1371/journal.pcbi.1014824.g010

EvalSVcallers showed comparatively smaller overall parameter sensitivity. Its maximum absolute F1-score deviation was 0.0150, observed for HG002-Wham when min_ovl was reduced from the baseline value of 0.5 to 0.3. Under this change, F1-score increased from 0.0702 to 0.0852. Increasing min_ovl to 0.6 and 0.7 instead produced small decreases to 0.0688 and 0.0681, respectively. The min_ins response for the same caller was also modest above the baseline value of 200 bp: F1-score was 0.0702 at min_ins = 300 and 0.0713 at min_ins = 500, whereas reducing min_ins to 50 bp decreased F1-score to 0.0585 (Fig 11).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. EvalSVcallers F1-score sensitivity across tested parameter configurations.

https://doi.org/10.1371/journal.pcbi.1014824.g011

Overall, the expanded sensitivity analysis demonstrates that the effect of parameterization is not uniform across benchmarking frameworks. SVbenchmark showed the greatest average parameter-associated F1-score variation, whereas Truvari produced the largest individual deviation under the most stringent tested sequence-similarity setting. EvalSVcallers was comparatively stable across the evaluated matching parameters.

8.7 Variant level bootstrap analysis

Baseline-relative ΔF1 values and their bootstrap support across the evaluated parameter configurations are shown in Figs 9–11, where an asterisk (*) indicates that the empirical 95% bootstrap interval for ΔF1 obtained from 5,000 paired variant-level bootstrap iterations excluded zero. Among the 400 baseline-alternative comparisons, 323 (80.8%) were bootstrap-supported, including 122 of 160 Truvari comparisons (76.3%), 131 of 160 SVbenchmark comparisons (81.9%), and 70 of 80 EvalSVcallers comparisons (87.5%). The largest parameter effects remained clearly supported. For example, increasing Truvari pctseq from 0.7 to 0.9 for NA12878-Lumpy reduced F1 by 0.0708, with a 95% bootstrap interval of [−0.0748, −0.0667] (Fig 9). Similarly, SVbenchmark normdist from 1.0 to 0.1 for HG002-Manta reduced F1 by 0.0538 [−0.0566, −0.0511] (Fig 10). For EvalSVcallers, decreasing min_ovl from 0.5 to 0.3 for HG002-Wham increased F1 by 0.0154, with a 95% bootstrap interval of [0.0135, 0.0173] (Fig 11). In contrast, changes that were not bootstrap-supported were very small; the largest absolute ΔF1 among these comparisons was only 0.0004. Overall, the major parameter-related F1-score changes remained stable under repeated variant-level resampling, indicating that the observed parameter effects were not primarily driven by random variation in the evaluated variants.

8.8 Benchmarking framework concordance across callers and samples

Figs C and D in S1 Appendix present UpSet plots illustrating the overlap of true positive (TP) variant identifications across different benchmarking frameworks under default parameter configuration for each caller and sample (S1 Appendix). Among callers Manta shows the strongest consensus among frameworks in both samples. The three frameworks share 2942 TP variants in NA12878 and 6453 TP variants in HG002. In NA12878, this common intersection represents approximately 97.4% of the Truvari TP set, 90.7% of the SVbenchmark TP set, and 88.5% of the EvalSVcallers TP set. In HG002, it represents approximately 99.5%, 95.9%, and 95.9% of the corresponding TP sets, respectively. These high proportions indicate that most Manta variants classified as TP are interpreted consistently across the three frameworks. The relatively small framework-specific counts (e.g., SVbenchmark = 35 in HG002) indicate that only a limited subset of Manta TP assignments differed across frameworks.

GRIDSS is the caller where EvalSVcallers consistently reports notably more TPs than the other two frameworks in both samples. EvalSVcallers identifies 291 framework-specific TP variants in NA12878 and 426 in HG002. Its total TP set contains 1546 variants in NA12878, compared with 1229 for Truvari and 1259 for SVbenchmark, and 2913 variants in HG002, compared with 2470 and 2488, respectively. Thus, the EvalSVcallers TP set is approximately 1.2 times larger than those produced by the other frameworks in both samples.

Delly shows a complete separation between the TP set identified by EvalSVcallers and those identified by Truvari and SVbenchmark. In HG002, all 1249 EvalSVcallers TP variants are framework-specific, while 1082 variants are shared only between Truvari and SVbenchmark and 147 are unique to SVbenchmark. Similarly, in NA12878, all 1512 EvalSVcallers TP variants are unique to the framework, whereas 1366 TP variants are shared only between Truvari and SVbenchmark, and an additional 107 variants are unique to SVbenchmark. Consequently, no three-framework TP intersection is observed for Delly in either sample. Moreover, the complete Truvari TP set is contained within the SVbenchmark TP set in both samples. This pattern indicates a pronounced framework-specific separation involving EvalSVcallers rather than a difference arising solely from unequal TP set sizes.

In contrast, Lumpy shows substantial three-framework agreement in both samples. The common TP intersection contains 685 variants in HG002 and 1162 variants in NA12878. In HG002, the common intersection corresponds to the complete Truvari TP set and accounts for approximately 91.6% and 87.0% of the SVbenchmark and EvalSVcallers TP sets, respectively. In NA12878, the shared intersection accounts for approximately 99.4% of the Truvari TP set, 88.8% of the SVbenchmark TP set, and 84.5% of the EvalSVcallers TP set.

Wham shows a completely different profile in both figures, and it is most evident in HG002. In HG002, the total number of Wham variants classified as TP decreased from 1062 under EvalSVcallers to 18 under Truvari. This approximately 59-fold difference indicates a pronounced incompatibility between the representation of Wham calls and the matching criteria applied by Truvari in this sample, rather than a biological change in caller performance. Most TP variants identified by EvalSVcallers were not included in the much smaller Truvari TP set, while SVbenchmark produced an intermediate result. A related but less extreme asymmetry is observed in NA12878: EvalSVcallers identifies a total of 934 TP variants. Of these, 367 are unique to EvalSVcallers (39.3%), whereas 533 belong to the three-framework intersection. At the same time, the three-framework intersection accounts for approximately 98.2% of the Truvari TP set, indicating that the smaller Truvari set is almost entirely contained within the larger TP sets.

Figs E and F in S1 Appendix illustrate the pairwise similarity of TP variant sets among three benchmarking frameworks for each SV caller in HG002 and NA12878 (S1 Appendix). The left panels show the Jaccard indices, and the right panels show the overlap coefficient values. The Jaccard index evaluates the intersection of two TP sets relative to their union, while the overlap coefficient normalizes the intersection according to the size of the smaller TP set. Therefore, using both metrics together helps distinguish whether low Jaccard values are due to true TP set divergence or a difference in TP set size. Manta shows the most consistent TP set agreement among the frameworks in both samples. Jaccard values for Manta range from 0.95-0.98 in HG002 and 0.87-0.93 in NA12878. In contrast, the overlap coefficient values being in the range of 0.97-1.00 in both samples indicates that the variant sets classified as TP by the frameworks largely overlap. A similar trend is observed for Lumpy and GRIDSS. Although some Jaccard values are relatively lower in these callers, the overlap coefficient values being close to 1.00 suggests that the low Jaccard values are mostly influenced by the difference in TP set size rather than disagreement between frameworks. Dely shows a significant framework divergence in both samples. While the Jaccard value remains high between Truvari and SVbenchmark, the Jaccard and overlap coefficient values are 0.00 in pairwise comparisons of EvalSVcallers with Truvari and SVbenchmark. This indicates that the variant set classified as TP by EvalSVcallers for Delly does not overlap with the TP sets defined by the other two frameworks. Wham is the caller showing the most significant difference between the two samples. In HG002, the Jaccard values for Wham are very low, and the overlap coefficient values are also low in some comparisons; this indicates a real TP set divergence between frameworks. In contrast, in NA12878, although the Jaccard values for Wham are lower, especially in intersections containing EvalSVcallers, the overlap coefficient values are in the range of 0.98-0.99. This result shows that the low Jaccard values for Wham in NA12878 are largely due to the difference in TP set size, with the smaller TP set largely contained within the larger TP set. Overall, when the overlap coefficient and absolute TP set sizes are considered together, the agreement between frameworks is strong for Manta, Lumpy, and GRIDSS; Delly shows a framework-specific separation involving EvalSVcallers; and Wham shows both TP set divergence and size effects depending on the sample.

Figs G and H in S1 Appendix present Sankey diagrams illustrating how variant outcome labels (TP, FP, and NONE) assigned by one framework are redistributed across outcome categories in other frameworks (S1 Appendix). Each column represents a framework, nodes correspond to outcome categories (TP: green, FP: red, NONE: gray), and link thickness reflects the number of variants following a given transition path. Consistent with the Jaccard patterns in Figs E and F in S1 Appendix, Manta shows the most stable structure in both samples, with a dominant TP → TP → TP flow across all three frameworks. This is in line with its high pairwise Jaccard agreement (0.95–0.98 in HG002 and 0.87–0.93 in NA12878). In HG002, the green TP stream covers nearly the full width of the diagram, whereas in NA12878 the same pattern remains dominant but the FP band is slightly wider. Delly shows the strongest framework disagreement. In both samples, the EvalSVcallers NONE block is visually prominent, and the crossing links indicate extensive label reassignment between Truvari and EvalSVcallers, matching the zero Jaccard overlap previously observed between EvalSVcallers and the other two frameworks. Lumpy and GRIDSS display a more balanced profile, with a strong shared TP stream, a parallel FP → FP → FP band, and only a small NONE component. In both callers, the flows appear cleaner in HG002 than in NA12878. For GRIDSS, the EvalSVcallers TP block is visibly larger than the Truvari TP block in both samples, indicating that EvalSVcallers captures additional TP calls that are largely retained as TP in SVbenchmark as well. Wham shows the most divergent pattern and the clearest sample-specific contrast. In HG002, the Truvari column is dominated by the FP (red) block, with only a very small TP band, directly reflecting the near-zero Jaccard values and the very low F1-score observed for this caller–sample combination. A substantial portion of these Truvari FP calls then shifts into TP in EvalSVcallers and remains TP in SVbenchmark. In NA12878, the picture is more balanced: the Truvari TP band is clearly visible, and although crossing flows remain, the TP → TP → TP path is much more prominent than in HG002. Overall, the Sankey diagrams confirm that Manta showed the greatest cross-framework label consistency in these analyses, whereas Delly and Wham showed substantially greater framework-dependent label redistribution.

8.9 Variant level characteristics of cross-framework agreement and disagreement

To move beyond TP set overlap and label transition analyses, the properties of variants showing different levels of cross-framework TP agreement were examined directly. The SV size composition of the cross-framework agreement groups was additionally examined across five size intervals (Table E in S1 Appendix). Size distributions varied across agreement categories for several caller-sample combinations, but no uniform size-dependent pattern of cross-framework disagreement was observed across callers. Variant level characterization of cross-framework TP agreement revealed that disagreement was frequently accompanied by poorer positional and size concordance with the nearest same-type truth variant, although this relationship was not universal (Figs I and J in S1 Appendix). The clearest pattern was observed for DEL variants. For HG002-Manta, variants classified as TP by all three frameworks had a median breakpoint displacement of 0 bp, a median size ratio of 1.00, and a median reciprocal overlap of 1.00. The corresponding values changed to 24 bp, 0.71, and 0.66 for variants classified as TP by two frameworks, and to 163 bp, 0.49, and 0.11 for variants classified as TP by only one framework (Fig I in S1 Appendix). A similar pattern was observed for NA12878-Manta, for which median breakpoint displacement increased from 0 bp to 27 bp and 101 bp across the three agreement categories, while median reciprocal overlap decreased from 1.00 to 0.58 and 0.15, respectively. Lumpy DEL variants showed a comparable reduction in size and interval agreement as cross-framework TP agreement decreased.

However, framework disagreement could not be explained only by breakpoint or size discordance. Delly DEL variants provided a notable counterexample: in HG002, variants classified as TP by two frameworks had a median breakpoint displacement of 9 bp, a median size ratio of 1.00, and a median reciprocal overlap of 0.99, while variants classified as TP by only one framework retained corresponding values of 12 bp, 0.98, and 0.98. In NA12878, both groups showed essentially complete positional, size, and interval agreement with the nearest same-type truth variants. INS variants were also heterogeneous (Fig J in S1 Appendix). In particular, GRIDSS variants classified as TP by only one framework remained closely positioned to the nearest same-type truth variant: the median breakpoint displacement was 2 bp with a median absolute size difference of 1 bp and a median size ratio of 0.98 in HG002, compared with 3 bp, 4 bp, and 0.96, respectively, in NA12878. These results show that breakpoint displacement and size or interval agreement account for an important component of cross-framework disagreement, but a particular discordant classification can arise from representation handling, candidate selection, clustering, one-to-many matching, or another internal framework rule. Representative HG002 loci illustrating both geometrically discordant and geometrically similar framework-specific classifications are provided in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Example representative variants from HG002 unified variant universe illustrating cross-framework agreement and disagreement.

https://doi.org/10.1371/journal.pcbi.1014824.t001

Key Points

• SV benchmarking results depend on both the selected evaluation framework and its parameterization. Systematic variation of breakpoint distance, size similarity, sequence similarity, overlap, and normalized matching criteria produced caller- and sample-dependent changes in the reported metrics.
• Parameter sensitivity was not uniform across benchmarking frameworks. SVbenchmark showed the largest mean parameter-specific F1 variation, Truvari produced the largest individual F1 deviation under the most stringent tested pctseq setting, and EvalSVcallers showed comparatively smaller variation across the evaluated matching parameters. Paired variant level bootstrap resampling supported the robustness of most parameter associated F1-score changes: empirical 95% intervals excluded zero in 323 of 400 baseline-alternative comparisons (80.8%), while unsupported changes were limited to very small absolute F1-score differences.
• Genomic block resampling showed that default caller rankings were relatively stable within frameworks but could vary across frameworks. Manta ranked highest under Truvari and SVbenchmark, whereas GRIDSS ranked highest under EvalSVcallers in both evaluated samples. Whereas genomic block resampling assessed robustness to changes in regional genomic composition, variant level bootstrap resampling specifically evaluated the stability of parameter associated metric changes to variation in the evaluated variant sets.
• Variant level characterization showed that high precision could coexist with low recall because many truth variants remained unmatched even when a large proportion of evaluated caller records were successfully matched. The FN was generally greater for INS than DEL and varied across SV-size strata, while its magnitude remained framework dependent.
• Jaccard indices and overlap coefficients provide complementary information about TP-set agreement. A low Jaccard index can result from unequal TP-set sizes even when the smaller set is largely contained within the larger set, whereas simultaneously low Jaccard and overlap coefficients indicate stronger TP-set separation.
• Cross-framework variant level characterization showed that reduced TP agreement was often accompanied by greater breakpoint displacement and poorer size or interval agreement, particularly for DEL variants. However, Delly DEL and GRIDSS INS examples showed that framework specific TP assignments can persist despite close positional and size agreement with same-type truth variants.
• Precision, recall, and F1 scores, as well as the resulting TP, FP, and FN classifications, should be interpreted as conditional on the benchmarking framework and its matching assumptions rather than as absolute measures of SV caller accuracy. Transparent reporting of matching logic, parameter settings, truth set scope, preprocessing, and genomic evaluation domain is essential for reproducible and comparable SV benchmarking.