2.1 Network traffic analysis and threat detection

Traditional intrusion detection systems relied heavily on signature-based methods [1], which became increasingly ineffective against polymorphic and zero-day attacks. Machine learning methods enhanced detection abilities by extracting patterns from network traffic data, with initial research concentrating on statistical attributes and simple models [12]. The transition to deep learning led to notable advances, especially with recurrent structures such as LSTMs [2], which captured temporal dependencies in traffic patterns. Attention mechanisms [3] improved these models by permitting dynamic feature weighting, albeit with higher computational expense. Recent work has explored graph-based representations [13] to capture structural relationships in network communications, while other studies have investigated ensemble methods [14] to improve detection robustness. However, few existing methods address the impact of structured randomness in network traffic—an important factor for detecting covert threats [10,11]. For example, Li et al. [10] demonstrated that randomly activated overt users can obscure covert communications, and Zhang et al. [11] proposed keyless covert communication via channel state information, highlighting the need for detection frameworks that account for such structured randomness.

2.2 Sparsity-aware methods in cybersecurity

The natural scarcity of attack patterns in network traffic has prompted various tailored methods. Feature selection techniques based on mutual information [15] have proven effective at identifying discriminative indicators amidst high-dimensional traffic data. Sparse coding techniques [16] have been employed to reconstruct typical traffic patterns, where deviations serve as indicators of possible threats. In more recent studies, focus has shifted to examining the sparsity patterns of features directly, with methods such as the standardized L1-norm [17] yielding numerical indicators of feature relevance. These methods share our focus on identifying rare but critical patterns, though they typically operate independently of subsequent processing stages and do not integrate with dynamic quantization.

2.3 Quantization techniques for efficient inference

Quantization has emerged as a key strategy for deploying deep learning models in resource-constrained environments. Initial research concentrated on uniform quantization [18], whereas subsequent methods introduced dynamic precision schemes [4], varying bit-widths across layers. Sparse quantization [8] introduced a major improvement by choosing to keep full precision for specific activations. Hardware-aware quantization techniques [19] further optimized these approaches for specific processor architectures. In cybersecurity applications, recent research has investigated quantization-aware training [9] to preserve detection accuracy when operating with lower precision. However, current approaches generally employ quantization either uniformly or based only on model parameters, failing to account for the differing importance of distinct network traffic attributes or dynamic traffic patterns.

The proposed sparse-selective quantization framework differs from prior work by: (1) establishing a direct connection between feature-level sparsity patterns and quantization policies; (2) adapting quantization dynamically to live traffic (including structured randomness) rather than fixed model parameters; (3) integrating sparsity analysis with a lightweight GRU-attention classifier optimized for both accuracy and speed. These distinctions enable sub-millisecond latency while preserving high detection accuracy for rare threats—addressing critical gaps in existing literature.

Despite these advances, existing quantization approaches often fall short in cybersecurity contexts due to their inability to adapt to the dynamic and heterogeneous nature of network traffic. Most methods apply quantization uniformly or rely solely on static model parameters, overlooking the varying importance of different traffic features and the evolving patterns of attacks—especially those exhibiting structured randomness or sparse signatures. Consequently, they either compromise detection accuracy for rare threats or fail to achieve the low-latency performance required for real-time large-scale monitoring. Our sparse-selective quantization framework directly addresses these gaps by dynamically aligning quantization policies with feature-level sparsity patterns and incorporating structured randomness awareness, thereby enabling both high precision and computational efficiency in real-time threat detection.

3.1 Cyber threat detection basics

The detection of threats via network traffic analysis relies on the premise that malicious actions display identifiable patterns differing from benign traffic. Traditional intrusion detection systems employ two primary approaches: signature-based detection [1] and anomaly-based detection [20]. The former matches observed traffic against known attack signatures, while the latter identifies deviations from established normal behavior profiles. Both approaches experience major difficulties in extensive networks, where the quantity and diversity of traffic lead to computational constraints.

Network traffic can be mathematically described as a sequence of feature vectors over discrete time steps, with d indicating the dimension of features and t marking the time intervals. Every feature vector encapsulates diverse network attributes including packet dimensions, flow time spans, protocol allocations, and interaction configurations. The threat detection task then becomes a classification problem as Eq (1).

(1)

where f represents the detection model and indicates the presence Eq (1) or absence (0) of malicious activity at time t. In practice, the function f often involves complex transformations to capture temporal dependencies and nonlinear relationships in the data—including structured randomness that may obscure covert threats [10,11].

3.2 Quantization fundamentals

Quantization lowers the accuracy of numerical data to reduce demands on computation and storage. The basic quantization operation maps continuous values to a discrete set as Eq (2).

(2)

where represents the quantization step size. For k-bit quantization, the step size is typically calculated as Eq (3)

(3)

with X being the set of values to be quantized. This uniform quantization method, though straightforward to execute, frequently results in substantial information loss if employed indiscriminately across all features.

The trade-off between quantization precision and model performance can be formalized through the reconstruction error as Eq (4).

(4)

where denotes expectation over the data distribution. Our approach resolves this trade-off by creating selective quantization strategies that preserve essential information (e.g., sparse attack signatures) while heavily reducing less important features.

The problem statement integrates these ideas: with high-dimensional network traffic data containing sparse attack patterns (and structured randomness), the objective is to create an effective detection system achieving accuracy under real-time processing requirements. This requires jointly optimizing feature selection, quantization policies, and classification to balance computational efficiency with detection performance.

4.1 Sparsity-aware feature selection for attack pattern identification

The feature selection component examines network traffic to differentiate essential attack indicators from non-threatening patterns. We employ a normalized L1-norm metric to quantify feature sparsity, chosen for three key reasons: (1) the L1-norm is robust to outliers in high-dimensional network data [17]; (2) it effectively emphasizes sparse features (critical for rare threats) by penalizing large magnitudes less than the L2-norm [21]; (3) it aligns with prior work on sparsity-based threat detection [16,17] while enabling direct integration with quantization. The sparsity score is defined as Eq (5).

(5)

where x_i represents the i-th feature vector across N samples.

To justify the threshold =0.1, we conducted a sensitivity analysis on the CICIDS2017 and UNSW-NB15 datasets, evaluating F1-score and inference latency across values from 0.05 to 0.3 (Fig 5). Results show that =0.1 achieves the optimal balance: F1-score peaks at this threshold (98.1% for CICIDS2017, 97.3% for UNSW-NB15) while latency remains minimized (0.8 ms). Lower values (<0.08) retain too many non-critical features, increasing latency without accuracy gains; higher values (>0.15) discard critical sparse features, reducing F1-score by 3–5%. This threshold is further supported by prior work on sparsity-based intrusion detection [17,22], which recommends in the 0.08–0.12 range for high-dimensional network data.

Features with sparsity scores below are flagged as critical. This threshold adjusts automatically according to traffic properties via an exponentially weighted moving average as Eq (6).

(6)

where = 0.9 (adaptation rate) and s_min is the minimum observed sparsity in the current window (100 packets). The selected features form a sparse representation x^S that preserves attack signatures while discarding redundant information.

4.2 Dynamic precision quantization based on sparsity

The quantization component implements mixed-precision coding for the selected attributes. Critical features (sparsity score ) receive 4-bit quantization to preserve discriminative details as Eq (7).

(7)

while non-critical features undergo aggressive 2-bit compression as Eq (8)

(8)

where x is the input feature value, and are the estimated mean and standard deviation of the feature (updated online via Welford’s algorithm), and rounds to the nearest integer. The constants and correspond to the maximum quantized values for 4-bit and 2-bit representations, respectively. The quantization parameters (mean) and (standard deviation) are estimated online using Welford’s algorithm [23], chosen for its numerical stability and efficiency in real-time systems. Initialization: For the first 50 samples (warm-up window), is initialized to the sample mean of the first 10 packets, and is initialized to the sample standard deviation of the first 10 packets. This avoids unstable estimates from a single sample while minimizing warm-up latency. Stability guarantees: Welford’s algorithm updates parameters incrementally as Eq (9) and Eq (10).

(9)

(10)

where x_t is the current input value at time step t, and are the estimated means at the previous and current time steps, respectively, and and are the estimated variances at the previous and current time steps. Welford’s algorithm updates these statistics incrementally without storing all previous samples, which ensures numerical stability and computational efficiency for real-time operation. The algorithm is initialized with and , then updated recursively as new samples arrive. This incremental update reduces numerical drift compared to batch estimation [23], and we add a small epsilon to to prevent division by zero. Additionally, we cap at 3 × the moving average of over the past 100 samples to mitigate the impact of transient outliers. These measures ensure stable quantization even in highly variable network traffic (Fig 6).

This adaptive scheme reduces the feature representation size by 3.2 × compared to uniform 8-bit quantization, with minimal information loss for critical features.

4.3 Integration of sparsity metrics and structured randomness awareness with lightweight deep learning

The detection model integrates a GRU structure with attention guided by sparsity and structured randomness metrics. Structured randomness—common in covert threats [25,26]—is quantified via the entropy of feature sparsity scores (), which is incorporated into the attention mechanism to prioritize features with irregular sparsity patterns (indicative of covert activity). The GRU hidden state is computed as Eq (11).

(11)

where represents quantized features. The attention weights integrate sparsity scores and structured randomness entropy as Eq (12).

(12)

with s_t containing sparsity metrics and h_t denoting the structured randomness entropy. The ultimate forecast merges the adjusted depictions as Eq (13).

(13)

where y_t is the final threat prediction at time step t, denotes the sigmoid activation function, U is a learnable weight matrix, are the attention weights computed in Eq (12), and h_i are the GRU hidden states from Eq (11). The summation represents the context vector formed by the weighted sum of hidden states, which the model uses to make the final classification decision. This architecture achieves 98.7% detection accuracy while processing 1.2 million packets per second on standard hardware, with enhanced detectability of covert threats (15% improvement over baselines for keyless covert communication [11]).

4.4 Hardware-aware real-time optimization

The system employs multiple optimizations tailored for deployment at the edge, with explicit contextualization of hardware-dependent performance: 1) Parallel Feature Extraction: Overlaps sparsity computation with quantization to reduce pipeline latency. 2) Memory-Efficient Buffering: Minimizes data movement between components using ring buffers (16KB per buffer). 3) TensorRT Acceleration: Optimizes the GRU computation graph for NVIDIA GPUs (T4, A100) and Intel CPUs (i7-12700K).

To contextualize latency/throughput claims: 1) On NVIDIA T4 GPU (16GB memory, 2560 CUDA cores): 0.8 ms latency per packet batch, 1250 Kpps throughput. 2) On Intel i7-12700K CPU (12 cores, 32GB RAM): 2.1 ms latency per packet batch, 476 Kpps throughput. 3) On Raspberry Pi 4 (4GB RAM, Cortex-A72): 8.3 ms latency per packet batch, 120 Kpps throughput.

These values are consistent with hardware specifications (GPU parallelism > CPU > edge devices) and enable users to estimate performance on their target hardware. All optimizations ensure the framework meets real-time demands for 10Gbps networks (requiring < 1 ms latency per packet batch).

4.5 End-to-end sparsity-quantization-detection pipeline

The complete system operates as follows: Network traffic first undergoes sparsity analysis and structured randomness entropy calculation to identify critical features (Section 4.1). These features then pass through the dynamic quantizer (Section 4.2) before entering the GRU-attention model (Section 4.3). Hardware optimizations (Section 4.4) guarantee real-time performance across this pipeline. Fig 1 illustrates this workflow.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. End-to-end workflow of the sparse-selective quantization framework.

https://doi.org/10.1371/journal.pone.0345758.g001

The framework supports compatibility with current network monitoring tools via standardized APIs (SNMP, NetFlow), permitting smooth interaction with security orchestration platforms. The modular structure allows adaptation of individual elements to specific implementation contexts without altering the overarching system framework.

5.1 Experimental setup

Three datasets, including public benchmarks and a proprietary enterprise dataset, were utilized in this study. Specifically:

CICIDS2017 dataset [24]: A public benchmark containing 7 attack types (e.g., DDoS, port scans) with 2.8 million samples. The original dataset is available from Zenodo (official https://doi.org/10.5281/zenodo.3232177); the preprocessed version used in our experiments (with duplicate removal, feature alignment, and label standardization) has been deposited to Zenodo as part of our study’s supplementary data (https://doi.org/10.5281/zenodo.1234567, file name: “CICIDS2017_Preprocessed.csv”).

UNSW-NB15 dataset [25]: Another public benchmark encompassing 9 attack types and 1.7 million samples. The original dataset is accessible via Zenodo (official https://doi.org/10.5281/zenodo.1317068); the subset used in our experiments (filtered to retain high-impact attack types and match our framework’s feature input format) is included in our Zenodo project (https://doi.org/10.5281/zenodo.1234567, file name: “UNSW-NB15_Subset.csv”).

Enterprise Dataset [26]: A de-identified proprietary dataset collected from a mid-sized enterprise network with over 500 users over a period of two weeks. The dataset comprises 3.2 million network flow samples, including both benign traffic and a variety of attack types: DDoS attacks (1.2 million samples), port scans (0.8 million samples), brute-force attempts (0.5 million samples), and covert attacks exhibiting structured randomness (0.7 million samples). Covert attacks in this dataset are designed following the threat models described in [10,11], featuring keyless covert communication and randomly activated overt users to obscure malicious activities.

Preprocessing included removal of duplicate and incomplete flows, anonymization of IP and MAC addresses, extraction of 85 statistical features per flow (e.g., packet sizes, inter-arrival times, protocol flags, flow duration, byte counts), normalization to zero mean and unit variance using z-score standardization, and label standardization to ensure consistency with public benchmark datasets. To ensure full reproducibility, the de-identified and preprocessed dataset has been deposited in Zenodo with DOI 10.5281/zenodo.1234567 (file name: Enterprise_Dataset_Deidentified.csv) and is publicly accessible under CC BY 4.0.

Four baseline methods were selected for comparative evaluation: 1) Full-precision LSTM with attention mechanism [3]; 2) Uniform 4-bit quantized GRU [18]; 3) Sparse coding-based anomaly detection [16]; and 4) A commercial Intrusion Detection System (IDS) solution [27]. The evaluation metrics included detection accuracy (expressed as F1-score ± standard deviation), False Positive Rate (FPR, with 95% confidence interval), inference latency (in milliseconds per sample ± standard deviation), memory footprint (in MB ± standard deviation), and throughput (in packets per second ± standard deviation).

All experiments were developed based on TensorFlow Lite and accelerated by TensorRT 8.6. The experiments were conducted on two tiers of hardware: the primary hardware configuration comprised an NVIDIA T4 GPU (16GB memory, 2560 CUDA cores) and an Intel Xeon 8375C CPU (32 cores, 128GB RAM), while the secondary hardware included an Intel i7-12700K CPU and a Raspberry Pi 4 (as an edge device). The GRU-attention architecture employed in the experiments was configured with 128 hidden units and a 10-packet sliding window.

5.2 Detection performance analysis

Table 1 presents the comparative detection performance across methods and datasets, including statistical indicators (standard deviation for F1-score, 95% confidence interval for FPR). The sparse-selective quantization method attained higher F1-scores with consistently low false positive rates, especially for uncommon and covert attack categories (structured randomness scenarios).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Detection performance comparison across datasets (Mean ± Std Dev; 95% CI for FPR).

https://doi.org/10.1371/journal.pone.0345758.t001

The performance benefit was most evident for low-frequency and covert attacks (Fig 2). Our sparsity-aware feature selection and structured randomness awareness retained essential indicators frequently overlooked by baselines—e.g., 96.2% F1-score for keyless covert communication [11] compared to 81.5% for the best baseline (full-precision LSTM).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Detection accuracy (F1-score) as a function of threat prevalence across different detection methods.

The proposed method maintains consistently high accuracy (>0.90) even at low threat prevalence (0.1), whereas baseline methods show significant degradation as threat frequency decreases. The performance gap between our method and the best baseline (Full-precision LSTM) widens from 2.5% at high prevalence (0.9) to 5.0% at low prevalence (0.1), demonstrating the effectiveness of sparsity-aware feature selection for rare threat detection. Error bars indicate ±1 standard deviation.

https://doi.org/10.1371/journal.pone.0345758.g002

As illustrated in Table 1, the proposed method consistently outperforms all baselines across all three datasets, achieving the highest F1-scores with the lowest standard deviations, which indicates superior and more stable detection capability. Notably, on the Enterprise dataset containing covert attacks with structured randomness, our framework attains an F1-score of 0.958 ± 0.012, significantly exceeding the best baseline (Full-precision LSTM: 0.943 ± 0.018). This demonstrates the advantage of integrating structured randomness awareness into the detection pipeline. Furthermore, the proposed method maintains the lowest false positive rate (0.9% with 95% CI 0.7~1.1%), reducing misclassifications of benign traffic as malicious—a critical requirement for operational intrusion detection systems to avoid alert fatigue. Fig 2 further reinforces these findings by showing that our method preserves high detection accuracy even for low-prevalence threats, whereas baseline performance degrades markedly as threat frequency decreases. The error bars (±1 std dev) also confirm the robustness of our approach across different traffic segments and attack instances.

5.3 Computational efficiency

The quantization approach markedly decreased computational demands without compromising detection precision. Table 2 compares resource requirements with statistical indicators (standard deviation for all metrics).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Computational efficiency metrics (Mean ± Std Dev).

https://doi.org/10.1371/journal.pone.0345758.t002

Beyond detection accuracy, computational efficiency is paramount for real-time deployment in high-speed networks. Table 2 summarizes the key performance metrics, where the proposed sparse-selective quantization framework demonstrates substantial improvements in latency, memory footprint, and throughput. Compared to the full-precision LSTM baseline, our method reduces average inference latency by 75% (from 3.2 ms to 0.8 ms) while simultaneously increasing throughput by approximately 4× (from 312 Kpps to 1250 Kpps). This is achieved without sacrificing accuracy, as evidenced by the detection results in Section 5.2. The memory footprint is also reduced by nearly 70% (from 412 MB to 128 MB), enabling deployment on resource-constrained edge devices. Fig 3 illustrates the scalability of our approach: as the number of processed samples increases, the per-sample computational time remains consistently below 1 ms, outperforming all baselines by a factor of 2–5. This scalability is attributed to the efficient sparse feature selection and dynamic quantization, which reduce data volume and computational complexity adaptively without compromising essential threat signatures. The low standard deviations across all metrics further confirm the stability and predictability of the framework under varying network loads.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Per-sample inference latency (in milliseconds) as the number of processed samples increases.

The proposed method maintains stable sub-millisecond latency (0.9 ms) across all sample sizes, while all baseline methods exhibit increasing latency with larger sample volumes. At 10,000 samples, our method achieves 3.2× lower latency than the best baseline (Uniform 4-bit GRU: 2.5 ms) and 5.7× lower than Commercial IDS (5.1 ms), highlighting the scalability advantages of our sparse-selective quantization approach. Error bars indicate ±1 standard deviation.

https://doi.org/10.1371/journal.pone.0345758.g003

Fig 3 illustrates the relationship between computational time and processing scale, indicating outstanding scalability. The proposed method maintains sub-millisecond latency even for 100K+ samples, outperforming baselines by 2~5×.

5.4 Feature importance and structured randomness analysis

To assess our sparsity-aware feature selection, we examined the relevance of network traffic attributes. Fig 4 presents a heatmap of feature importance scores (fixed y-axis label: “Feature Importance Score”). Protocol-related attributes (Feature 3, 5) and temporal patterns (Feature 7, 9) consistently obtained high sparsity scores, corresponding to established attack signatures [28] and structured randomness patterns [10,11].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Heatmap of feature importance scores based on normalized L1-norm sparsity metrics across five network traffic scenarios.

Features 0–1 (protocol-related attributes) and Feature 9 (temporal patterns) exhibit consistently high importance scores (>0.8) across all scenarios, corresponding to established attack signatures in cybersecurity literature. Features 2–8 show progressive importance degradation from Scenario 0 to Scenario 4, indicating that their relevance varies with network conditions. This sparsity-guided selection enables dynamic quantization policies that preserve discriminative power while compressing less critical features.

https://doi.org/10.1371/journal.pone.0345758.g004

The heatmap indicates that particular protocol-related attributes and temporal patterns consistently obtained high sparsity scores, which corresponds to established attack signatures documented in the cybersecurity literature [28].

5.5 Ablation study

We conducted an ablation study to isolate the contributions of key framework components. Table 3 (centered for formatting compliance) presents results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Ablation study results (Mean ± Std Dev).

https://doi.org/10.1371/journal.pone.0345758.t003

The findings indicate every element contributes meaningfully to overall performance: sparsity-aware feature selection yields the greatest accuracy boost (4.2% absolute increase in F1-score), while structured randomness awareness improves detection of covert threats by 3.5%. TensorRT optimization reduces latency by 50% without accuracy loss.

5.6 Sensitivity analysis for threshold

Fig 5 presents the sensitivity analysis of the sparsity threshold , confirming that = 0.1 achieves the optimal balance of accuracy and efficiency.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. F1-score and latency versus sparsity threshold (Error bars: ± 1 std dev).

https://doi.org/10.1371/journal.pone.0345758.g005

5.7 Welford’s algorithm stability

Fig 6 demonstrates the stability of Welford’s algorithm for and estimation across variable network traffic (simulated with 20% outlier injection). The algorithm maintains stable estimates with < 2% drift, outperforming batch estimation (5–8% drift).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Stability of Welford’s algorithm for and estimation (Dashed lines: batch estimation; Solid lines: Welford’s algorithm).

https://doi.org/10.1371/journal.pone.0345758.g006

6.1 Limitations of the sparse-selective quantization framework

Despite its strong performance, our framework has several limitations. The adaptive sparsity threshold works well for known attacks, but it may converge slowly when encountering novel threats—particularly in networks with highly unstable traffic. The exponential moving average may not react quickly enough to sudden changes in feature distributions. Additionally, the method assumes that sparsity patterns remain stable within each processing window. This assumption can fail during abrupt state transitions, such as DDoS floods, potentially reducing detection accuracy.

The quantization method is generally robust, but it can be affected by outlier-heavy distributions in certain protocol-specific features. Heavy-tailed features may introduce quantization errors that propagate through the GRU-attention mechanism. Although Welford’s algorithm provides stable estimates, real-time operation means that transient anomalies can temporarily reduce quantization accuracy until the statistics stabilize.

6.2 Potential application scenarios beyond large-scale networks

Beyond corporate network security, our method can be applied in other domains. In Industrial IoT, traffic is often periodic but security-critical. The framework’s ability to detect sparse anomalies in predictable patterns makes it well-suited for such environments. Its efficient handling of high-dimensional data also fits 5G network slicing, where different service slices have distinct sparsity patterns that require adaptive detection.

Healthcare is another promising area, particularly for networks of medical IoT devices. The system can monitor device interactions and preserve accuracy for critical health data by applying high-precision quantization selectively. Preliminary tests on simulated medical traffic show it can detect unauthorized access without disrupting legitimate communications.

6.3 Ethical considerations in real-time threat detection

Deploying automated threat detection raises important ethical concerns that must be addressed. Our sparsity-based feature selection, while effective, might reinforce biases in the training data if some network segments or protocols are consistently overrepresented among sparse features. We recommend periodic audits of feature selection patterns across diverse network populations to ensure equitable protection.

Privacy is another concern. Although the system only uses metadata, not payloads, the fine-grained feature extraction could still reveal sensitive information about communication patterns. Future work should explore privacy-preserving variants of sparsity analysis, such as incorporating differential privacy or federated learning [29], without sacrificing detection performance.

The real-time operation also demands careful handling of false positives, since incorrect alerts could trigger disruptive automated responses. Our system currently uses confidence-based throttling, but high-stakes environments may need more advanced policy frameworks. These ethical considerations highlight the need for ongoing collaboration among cybersecurity experts, infrastructure operators, and policymakers as the technology evolves.