3.1 Preprocessing

Prior to building and training deep neural network models for intrusion detection, raw network traffic datasets undergo a multistep preprocessing pipeline to transform the data into formats suitable for effective learning [62]. The key preprocessing steps are as follows:

3.2 Feature extraction and selection for model optimization

Feature selection is the process of detecting and choosing a smaller number of important features from a larger set, depending on their capacity to maximize the distinction between class labels. In the context of machine learning, feature selection involves selecting a subset of important features based on specific criteria. Real-world datasets often contain redundant, irrelevant, and noisy attributes [58], making feature selection crucial for improving efficiency and reducing dimensionality.

Feature engineering is critical in constructing machine learning models that achieve high performance, particularly when working with complex, high-dimensional data. This section presents a robust feature engineering workflow tailored to optimize an IoT intrusion detection model. The pipeline leverages Stacked Autoencoders (SAEs) for unsupervised nonlinear feature reduction and CatBoost for supervised predictive feature selection.

3.2.2 CatBoost for predictive feature selection.

We utilized CatBoost for efficient feature selection in our machine learning pipeline. CatBoost is an open-source gradient-boosting library developed by Yandex that is known to handle categorical variables and achieve state-of-the-art performance [61]. A key advantage of CatBoost is its built-in feature-selection functionality. As CatBoost trains the models, it assigns importance to each feature, based on the predictive value of the object variable. We used this method to identify and select the most significant features. By removing low-importance features, we simplified our models, reduced overfitting, and improved performance. CatBoost provides an automated method for selecting the most informative features and discarding the uninformative ones. This allows us to build more accurate and interpretable models by focusing on the features that truly drive our machine-learning task.

Although SAEs reduce the number of dimensions, some redundant and irrelevant features may persist. To refine the feature space, a CatBoost model is trained on SAE-encoded features to predict the target variable. CatBoost is a high-performance gradient-boosting algorithm that is well-suited for feature selection. The steps involve:

• Training a CatBoost model on the SAE encoded features.
• Computing feature importance scores.
• Sorting features by importance ranking.
• Selecting the top N most predictive features.

The integrated approach extracts optimized and compact feature subsets tailored to the learning task, as shown in Fig 2. This combines the strengths of unsupervised nonlinear SAE feature reduction with supervised CatBoost predictive feature selection based on the importance ranking. The integrated approach extracts optimized and compact feature subsets tailored to the learning task.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Feature engineering pipeline using stacked autoencoders and CatBoost: The pipeline starts with the input data, which is processed by a stacked autoencoder for nonlinear dimensionality reduction.

The encoded features from the autoencoder are then input into CatBoost, which computes feature importance scores and selects the top N most predictive features.

https://doi.org/10.1371/journal.pone.0304082.g002

The compact informative features also align with the narrow memory and processing requirements of IoT and edge devices. The integrated SAE-CatBoost pipeline provides an end-to-end feature-engineering solution to derive optimized feature representations from high-dimensional network traffic data for efficient intrusion detection in IoT environments.

After dimensionality reduction by SAE, CatBoost was used to perform feature selection on the encoded features X_encoded:

CatBoost(X_encoded, y)

The features were sorted based on their importance scores, and the top N features were selected:

X_selected = select_top(X_encoded, N)

CatBoostClassifier.fit(X_train_encoded, y_train)

3.2.3 Feature-importance computation.

After training, the feature importance scores were computed using the CatBoost model, which gives us a score for each feature: (11) Where Importance(i) represents the importance score of the x_i features. The selected features for the AWID Binary, ranked by their importance, begin with F8 at the top, followed by F3, F4, F16, and F0, which rank in the top five. The next set of significant features includes F27, F31, F17, and F20, with F25 completing the top ten. As we move down the list, F11, F6, F29, F24, and F12 offer considerable value. Other notable features included F19, F7, F28, F26, and F14. The list continues with F2, F21, F1, and F5 and is rounded off with F13, F23, F18, F9, F22, F15, F10, and F30, showing a diverse range of factors considered in the AWID Binary dataset, as shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Selected features for the AWID dataset.

https://doi.org/10.1371/journal.pone.0304082.t001

For the AWID features in the multi-dataset, the selected features, totalling 30, were arranged in order of importance. The list begins with F18 leading the pack, followed by F10, and F14. F24 and F2 were also highly ranked and rounded out among the top five. The dataset continued to leverage the significance of F4, F25, F27, F16, and F7, with F8 being slightly behind. Additional features, including F22, F28, F23, F13, and F19, provided valuable information. The series continued with F6, F30, F12, and F0, each of which added depth to the model. Concluding the feature set, F31, F17, F20, F15, F1, F3, F29, F11, F21, and F9, despite their lower ranks, play a part in the model’s foretelling capacity, showcasing the broad nature of the AWID Features on Multi classification.

For the UNSWB15 Binary dataset, 21 features were carefully chosen for their unique contribution to the model’s performance. The arrangement started with F1, taking the prime spot, followed by F2, F3, F4, and F5, making up the foremost quintet. The selection proceeds with pivotal features such as F6, F7, F8, F9, and F10, each playing a crucial role. The enumeration continued with F11, F12, F13, F14, F15, F16, F17, F18, F19, F20, and F21, collectively adding depth, variety, and acuity to the dataset and illustrating the extensive and insightful selection of features for the UNSWB15 Binary dataset.

For UNSWB15 multi-features, 26 features were selected based on their importance. The list begins with F13 at the forefront, followed by F1 and F15. F29 and F12 also held significant ranks, completing in the top five. The dataset includes valuable inputs from F4, F23, F34, F9, and F31. Additional layers of complexity and insight were provided by F26, F37, F32, F20, F18, F10, F0, F5, F25, F21, F14, F19, F11, F7, F36, and F30, each contributing to the model’s predictive capabilities, demonstrating a comprehensive selection of features for the UNSWB15 multi-dataset as described in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Selected features for the UNSWB15 dataset.

https://doi.org/10.1371/journal.pone.0304082.t002

For the KDD Binary dataset, a carefully curated set of 25 features is selected based on their importance. Topping the list is F26 ‘srv_serror_rate,’ demonstrating paramount significance with the highest importance score. Not far behind is F32 ‘dst_host_count,’ cementing its critical role in the dataset. F10 ‘hot’ follows suit as a key indicator, with F35 ‘dst_host_diff_srv_rate’ and F12 ‘logged_in’ rounding out the top five, each bringing substantial weight to the predictive model. Continuing down the list, F5 ‘src_bytes’ and F17 ‘num_file_creations’ emerge as notable features, along with F2 ‘protocol_type’ and F18 ‘num_shells’, each contributing their unique insights. F1 ‘duration’ seals the top ten, representing a vital aspect of the data. The mid-range of the importance spectrum includes features such as F36 ‘dst_host_same_src_port_rate,’ F8 ‘wrong_fragment,’ and F9 ‘urgent,’ all adding valuable dimensions to the analysis. Features such as F23 ‘count,’ F30 ‘diff_srv_rate,’ and F29 ‘same_srv_rate’ enhance the model further, while F24 ‘srv_count’ and F7 ‘land’

For the KDD Multi-dataset, 25 features were carefully selected based on their significance scores. The lineup began with F37, which marked its prominence as the most influential feature. This is closely followed by F4, which has substantial weight in the model. F31 also has a notable appearance, securing a key position in the lineup. F22 and F2 were also pivotal, rounding out the top five most influential features, and the selection continued with F26 and F12, each providing critical insights into the dataset. F28, F36, and F7 further enriched the model and highlighted their importance. The array was complemented by F41, F8, F27, and F30, each of which provided valuable perspectives. F1, representing a temporal aspect, along with F35, F24, and F32, contribute significantly to the dataset’s complexity. Further down the list, F23 and F34 add depth to the analysis, while F33 provides essential data interpretation. The set included F29, F38, F3, and F25, each offering unique contributions and ensuring a comprehensive approach to the feature selection process for the KDD Multi-dataset. By combining the original and encoded feature analyses, a holistic perspective was obtained to judiciously leverage the selected features during the model development, as shown in Tables 1–3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Selected features for the KDD dataset.

https://doi.org/10.1371/journal.pone.0304082.t003

3.3 The ensemble model architecture

Transformer neural networks (TNN) were first introduced in 2017 for sequence modelling tasks in natural language processing (NLP) [62]. Although off-the-shelf TNN models, such as BERT and GPT, have achieved strong results on NLP problems, they may not be directly transferred to intrusion detection owing to differences in input data types and objectives. While NLP TNN models process sequences of words or tokens, intrusion detection systems process sequential network traffic and event data. Hence, this study proposes a customized TNN architecture tailored to the unique requirements of network intrusion detection.

A key advantage of TNN is its self-attention mechanism, which can capture the dependencies across a sequence. This allows a TNN-based intrusion detection model to analyze network traffic patterns and identify anomalies indicative of threats. Although other neural architectures exist for anomaly detection, TNN’s strengths in modelling complex sequential relationships make it well-suited for security tasks.

Advanced neural architectures, such as Transformers, CNNs, and LSTMs, have achieved state-of-the-art results across various sequence-modelling tasks. Transformers utilize multihead self-attention to analyze the interactions between all positions in a sequence. CNNs leverage convolutional filters to recognize local spatial patterns. LSTMs employ gated memory cells to capture long-range dependencies. This study proposes a novel ensemble model integrating Transformers, CNNs, and LSTMs to synergistically combine their complementary strengths for robust network intrusion detection, as shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Ensemble neural network architecture integrating transformers, CNNs, and LSTMs.

It shows the parallel application of the Transformer, CNN, and LSTM branches to the input sequence data. Their outputs were concatenated to ensemble global, local, and temporal patterns in the data for robust sequence modelling and intrusion detection.

https://doi.org/10.1371/journal.pone.0304082.g003

3.4 Optimization

Adaptive Grey Wolf Optimization (AGWO) is a nature-inspired metaheuristic algorithm that takes inspiration from the hunting behavior of grey wolves in the wild [63]. Grey wolves have a strict social hierarchy and hunting strategy involving tracking, encircling, and attacking prey.

The AGWO technique translates these behaviors into optimization procedures. The fittest gray wolf pack was identified based on the accuracy of the ensemble model on a validation dataset. The alpha wolf guides the optimization search. Encircling prey translates into exploring the solution space around the current optimal hyperparameter values.Attacking prey corresponds to rapidly exploding promising areas for improved solutions. In addition, the AGWO implements adaptive mechanisms to prevent suboptimal convergence. Varying the exploration-exploitation tradeoff over iterations. Randomize the search vectors to maintain diversity. Gradually reducing the exploration area. These adaptive enhancements prevent the algorithm from becoming trapped in the local optima. By mimicking the hunting behaviours and mechanisms of grey wolves, AGWO provides an effective bio-inspired approach to tune hyperparameters, such as learning rate, neural architecture dimensions, regularization strength, and other facets of the Transformer-CNN-LSTM ensemble model. The goal is to maximize network intrusion detection tasks’ accuracy and generalization performance. The model integrates self-attention-based transformers, local feature-extracting CNNs, and long-range LSTM networks to provide complementary global, local, and temporal modeling capabilities. However, finding the correct combination of hyperparameters, such as learning rates, layer sizes, and regularization for each branch, is challenging. AGWO provides an effective bio-inspired optimization technique. AGWO adapts the hunting, encircling, and attacking mechanisms of gray wolves to guide hyperparameter values toward optimal settings. We developed an extremely optimised hybrid model by applying AGWO to tune the hyperparameters of the Transformer, CNN, and LSTM branches in our ensemble. This provides the accuracy and robustness benefits of the multiscope Transformer-CNN-LSTM architecture, which is further boosted by the adaptive optimization power of AGWO.

Algorithm 1 AGWO for Ensemble Neural Network Model Optimization

1: Generate initial grey wolf positions Y(i) for i = 1, 2, …, n

2: Initialize vectors a, A, and C

3: for each wolf Y(i) do

4:  Create an ensemble of models using the parameters from Y(i)

5:  Evaluate the fitness of each ensemble (accuracy, loss, etc.)

6: end for

7: Y(α) = Ensemble with best performance

8: Y(β) = Ensemble with second-best performance

9: Y(δ) = Ensemble with third-best performance

10: Set iteration counter I = 1

11: repeat

12:  for each grey wolf Y(i) do

13:   Update the position of Y(i) (parameters for the ensemble)

14:   Create a new ensemble of models with updated parameters

15:  end for

16:  Evaluate the fitness of all new ensembles

17:  Update the positions of Y(α), Y(β), and Y(δ) based on new fitnesses

18:  Update vectors a, A, and C

19:  

20:  for each dimension do

21:   Update A and C using random values

22:   Calculate new positions based on Y(α), Y(β), and Y(δ)

23:  end for

24:  I = I + 1

25: until I ≥ max_iterations

26: Output Y(α) as the best solution (optimal ensemble parameters)

The AGWO algorithm initializes a population of grey wolves, where each wolf’s position represents a set of parameters for an ensemble of neural network models. An ensemble of models was created for each set of parameters, and its performance was evaluated based on metrics such as accuracy and loss. The top three ensembles were identified in order to establish a hierarchy. The main loop then iteratively updates the wolf positions (parameters) and reevaluates the ensembles. For each updated position, a new ensemble was created and evaluated. The vectors A, and C are adjusted adaptively to guide the search process towards the optimal parameters. The main loop continues for a predefined number of iterations. Finally, the parameters leading to the best-performing ensemble were the output. AGWO leverages adaptive biologically inspired operators to optimize the hyperparameters of an ensemble model by iteratively creating and assessing ensembles and honing optimal parameters as depicted in Fig 4. This provides an effective technique for tuning and maximizing the performance of an integrated neural network model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. A flowchart diagram of the iterative optimization process used by the adaptive grey wolf optimization algorithm to tune the hyperparameters of the ensemble model.

It depicts the steps of initializing the grey wolf population, evaluating ensemble models, updating wolf positions, re-evaluating models, and adaptively adjusting the optimization vectors to hone in on the optimal hyperparameters.

https://doi.org/10.1371/journal.pone.0304082.g004

4.1 Experimental datasets

This research utilized three standard network intrusion detection benchmark datasets to rigorously evaluate the proposed system. NSL-KDD: This dataset encompasses network traffic data with both regular connections and intrusion attacks [64]. The training and testing subsets were comprised of 125,973 and 22,544 samples, respectively. It has 41 features describing each network flow related to the protocol, statistics, content, etc. Binary classification for detecting attack flows and multi-class modelling of specific intrusion types were investigated. For the binary classification task of detecting attack vs. normal flows, 25 key features were selected from the 41 total features using the proposed SAE and CatBoost pipelines. For multi-class modelling with five classes (Normal, DoS, Probe, R2L, U2R), 30 predictive features were extracted from the 41 available features. UNSW-NB15: This more recent dataset emulates a hybrid enterprise deployment with 49 features across flow, basic, content, and time categories [65]. The training and testing sets consisted of 175,341 and 82,332 samples, respectively. It covers 10 classes, including normal and nine contemporary attack types, such as worms and malware, and exploits categories tailored to current network threats. The multi-class classification was performed to discriminate between the attack varieties. For the binary task of identifying malicious connections, 21 features were selected from the total 49 features. For multi-class modelling across the 10 classes, 26 informative features were derived using the feature engineering workflow. AWID: The Aegean Wi-Fi Intrusion Dataset focuses specifically on 802.11 wireless networks with major protocols [66]. It contains over 150 attributes that capture different properties of wireless traffic. The dataset includes normal data and flooding, impersonation, and injection attacks targeting the wireless environment. A total of 1,797,575 samples were collected across the four classes. Binary classification was used to distinguish between benign and malignant network flow. For the binary classification task, 30 features were selected from more than 150 attributes. For the AWID multi-class task, 29 predictive features were selected from over 150 total features using the proposed SAE and CatBoost pipelines. Together, these datasets enable assessment across traditional networks, modern hybrid enterprise deployments, and wireless-specific threats. The range of features, attack types, and environments make the datasets complementary and suitable for holistic evaluation. Optimized and compact feature subsets were derived for each dataset and task using the proposed integrated SAE and CatBoost approach. This has enabled the development of specialized models focused on the most predictive attributes for enhanced performance.

5.1 Stacked autoencoder architecture

A stacked autoencoder (SAE) is an unsupervised artificial neural network used for nonlinear dimensionality reduction and is often applied for IDS in network traffic data. The input data consists of network traffic datasets that first undergo preprocessing to extract numeric input features (denoted by X) and categorical label targets (denoted by y). The popular Python machine learning library sci-kit-learn provides tools for handling data wrangling and preprocessing tasks. Specifically, the data were split in an 80/20 ratio into training and testing sets. This allowed the models to be trained on a larger training set and then evaluated on the unseen testing set. Additionally, min-max scaling, a feature normalization technique, is used to normalize the range of input features to be between 0 and 1. This preprocessing step is crucial because neural networks learn and perform better when the input data features are scaled similarly. The input dimension for the SAE matches that of the scaled training data matrix, ‘X_train_scaled’. The architecture consists of three encoding layers that incrementally compress the input features into a reduced lower-dimensional, followed by three symmetric decoding layers that mirror the encodings to reconstruct the original high-dimensional input feature space at the output. This symmetrical autoencoder structure forces the model to learn efficient data encoding to produce an accurate reconstruction. The encoding layers utilize the Rectified Linear Unit (ReLU) activation function to introduce nonlinearity, which simply returns the input value itself if it is positive; otherwise, it outputs zero for negative inputs. In contrast, the decoding layers mirror the encoding dimensions in the reverse order. The SAE is implemented in Keras using dense layers for the encoders and decoders. The input layer size matched the number of input features in the preprocessed dataset (41 for NSL-KDD). This is encoded down to a smaller bottleneck layer of size 16, which retains only the most salient information, and the encoding layers use the rectified linear unit (ReLU) activation function and He normal kernel initializer. L2 regularization, including a strength of 0.001, was used on each encoding layer to prevent overfitting. The decoding layers mirror the encodings in reverse order to reconstruct the input, and the autoencoder is trained for 60 epochs using the Adam optimizer by a learning rate of 0.001 to reduce the MSE between the input and reconstructed output. The batch size was 256 samples, and the validation data were used to track overfitting. Once trained, the 16-node encoded bottleneck was used as the reduced feature input for the ensemble classifier, which involved a grid search over encoder sizes [32, 64, 128, 256], bottleneck size [8, 16, 32], regularization strength [0.0, 0.001, 0.01], learning rate [0.0001, 0.001], and epochs [50, 100, 150] with 30 trials per configuration to determine the optimal SAE parameters. SAEs effectively compress high-dimensional input spaces while retaining the most informative attributes, as validated through multiple reconstruction error metrics. On the NSL-KDD dataset [64], the mean squared errors averaged 0.021, whereas the mean absolute errors remained under 0.071 across both binary and multi-class tasks (Table 4). The close-to-unity cosine similarity of 0.999 further indicates that the SAE accurately captures intrinsic data patterns. The results on the more contemporary UNSW-NB15 dataset [65] in Table 5 echo the strong reconstruction capability of the SAE, with mean squared errors of 0.005 and 0.009 for multi-class and binary activities, respectively. The wireless-focused AWID dataset [66] evaluations in Table 6 produced comparable outcomes, confirming that the SAE consistently models the essence of diverse input features across domains. Overall, dimensionality reduction preserves salient properties for high-fidelity feature extraction tailored to the fog nodes. Figs 5–7 show the loss reduction over epochs during the SAE training and validation on the NSL-KDD, UNSW-NB15, and AWID datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Loss reduction over epochs in stacked autoencoder training vs validation on NLS-KDD for multi-classification: It illustrates the training and validation loss over epochs during the training of a stacked autoencoder on the NLS-KDD dataset for multi-classification.

The blue line represents the training loss, and the orange line represents the validation loss. Both the losses decreased over time, indicating the effectiveness of the training process.

https://doi.org/10.1371/journal.pone.0304082.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Loss reduction over epochs in stacked autoencoder training vs validation on Unswb15 for multi-classification.

https://doi.org/10.1371/journal.pone.0304082.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Loss reduction over epochs in stacked autoencoder training vs validation on AWID for multi-classification.

https://doi.org/10.1371/journal.pone.0304082.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Results of binary and multi-classification on NLS KDD using SAE.

https://doi.org/10.1371/journal.pone.0304082.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of binary and multi-classification on Unswb15 using SAE.

https://doi.org/10.1371/journal.pone.0304082.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Results of binary and multi-classification on AWID using SAE.

https://doi.org/10.1371/journal.pone.0304082.t006

5.2 CatBoost feature selection process

CatBoost, an ensemble learning algorithm based on gradient boosting, is a powerful tool for feature selection and predictive modelling. This section delves into the intricacies of CatBoost, elucidating its input, output, and hyperparameters as well as its significance in feature selection. The encoded dataset, ‘X_train_encoded’, was then provided as an input to the CatBoost model for feature selection. CatBoost evaluates the importance of each feature in the encoded data based on its contribution to the predictive performance of the model.

5.2.1 Evaluating feature selection.

For CatBoost, the following key hyperparameters governing the predictive feature selection were tuned:

1. Number of trees: Tested in the range of [100, 500] in increments of 100.
2. Maximum tree depth: Values from 5 to 8 were explored.
3. Learning rate: Varied over intervals [0.025, 0.1].
4. L2 leaf regularization: Tested lambda from [1, 5].

The CatBoost model underwent training with an optimal count of trees and depths. A learning rate set at 0.05, coupled with an L2 regularization strength of 3, yielded the most effective predictive outcome as per cross-validation. This specific setup was employed to determine feature significance and to choose the superior subsets. To assess the effectiveness of the selected features, the original dataset was filtered to retain only the top N features identified using CatBoost. This filtered dataset, with only the most important features, is denoted by

X_train_selected, and the model is then trained on

X_train_selected and tested on the corresponding filtered test set

X_test_selected. Comparing the performance of the model with all the original features quantifies the impact of the feature selection. CatBoost evaluates the importance of each feature in the encoded data based on its contribution to the predictive performance of the model. Hyperparameter Tuning; Several key hyperparameters in CatBoost control the feature selection process: number of trees (iterations): tested from 100 to 1000 trees. More trees extract feature importances from a more robust model but increase computation. Learning rate: Values between 0.01 to 0.5 tried. A lower learning rate is more conservative but may require more iterations. Tree depth (depth): Values from 4 to 10 were attempted. Thus, deeper trees can be overfitted. L2 regularization

l2_leaf_reg: Values from 1 to 10 were tested. Higher values are regularized to prevent overfitting. Verbose mode: Disabled during hyperparameter search for efficiency. These hyperparameters were tuned through a randomized search to determine the optimal configuration for feature selection. The best model was selected based on its performance on a validation set.

Obtaining feature names. To interpret the selected features, feature indices must be mapped back to their original names in the dataset. (excluding the label column). This mapping provides a context for understanding the significance of each feature. Practitioners can then focus on modeling these specific informative features.

Original feature analysis. The variance of the features indicates the importance of the basic features from the original data. Higher-variance features show greater fluctuations in value. Importance scores and rankings were displayed for interpretability.

Encoded feature analysis. CatBoost permutation importance scores assess the encoded feature relevance. A highlights the top N-selected features from the encoded data, linking indices to the feature names. By combining the original and encoded feature analyses, a holistic perspective was obtained to judiciously leverage the selected features during model development. Building on the compressed SAE representations, the gradient boosting model CatBoost selects optimized and compact feature subsets catered to each dataset through importance ranking, enumerating the top sets in Tables 1–3. For NSL-KDD comprising 41 total features, 25 predictive attributes were extracted for multi-class modeling, with the key indicators being srv-error-rate, dst-host-count, and logged-in. On the UNSW-NB15’s 49 features, CatBoost distills 21 and 26 informative signals for the binary and multi-class tasks, respectively, prioritizing elements such as source-port-number and flow duration. Similarly, tailored feature selections were derived for the Wireless AWID dataset. By filtering noisy and irrelevant dimensions, CatBoost enables the learning of discriminative inputs for efficiency and generalization. Figs 8–10 show the top 20 features from the NSL-KDD, UNSW-NB15, and AWID datasets based on their reconstruction errors when analyzed using a stack autoencoder and CatBoost.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Top 20 features with highest reconstruction error for NSL-KDD dataset: A bar graph displaying the top 20 features from the NSL-KDD dataset according to their reconstruction error when analyzed using a stack autoencoder.

The rebuilding mistake is shown on the y-axis, while each feature is shown on the x-axis.

https://doi.org/10.1371/journal.pone.0304082.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Top 20 features with highest reconstruction error for UNSW-NB15 dataset: This bar graph displays the top 20 features from the UNSW-NB15 dataset, ranked according to their reconstruction error when analysed using a stack autoencoder and CatBoost.

The y-axis corresponds to the reconstruction error, whereas the x-axis corresponds to each individual feature. The vertical length of each bar represents the magnitude of the reconstruction error linked to each characteristic.

https://doi.org/10.1371/journal.pone.0304082.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Top 20 features with highest reconstruction error for AWID dataset: The top 20 features from the AWID dataset were analyzed using a stack autoencoder and CatBoost, and their reconstruction error is shown in this figure’s bar graph.

On one side, we have each characteristic, and on the other, we have the reconstruction error. The amount of the feature-specific reconstruction error is shown by the height of the corresponding bar.

https://doi.org/10.1371/journal.pone.0304082.g010

5.3 Ensemble model architecture

The ensemble model integrates two additional CNN layers into the architecture for enhanced feature extraction: The initial Conv1D layer, equipped with 32 filters and a kernel size of 3, is designed to extract low-level features. The subsequent Conv1D layer, which has 64 filters, serves to refine the representations derived from the preceding layer. The subsequent hybrid architecture leverages transformer blocks, each composed of multi head self-attention modules and position-wise feedforward networks with regularization from layer normalization and dropout, to model complex temporal relationships in the data. This is followed by two sequential Long LSTM layers, with 64 units each, to learn longer-term sequential dependencies. The CNN, Transformer, and LSTM components were integrated through concatenation to jointly leverage their capabilities in processing sequential input data. The model was trained on the selected input features extracted using feature selection techniques. Specifically, encoded training ‘(X_train_encoded)’ and validation ‘(X_val_encoded)’. feature sets were used as model inputs. Adaptive Grey Wolf Optimizer is first utilized to tune the learning rate hyperparameter by exploring values between 1e-4 and 1e-2 and selecting the rate that maximizes validation accuracy. The model underwent training for a total of 100 epochs with a batch size of 2000, was validated on a distinct validation set, and was fine-tuned using the Adam optimizer along with a sparse categorical cross-entropy loss. The key main parameters of the ensemble model are summarized in Table 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Equivalent model descriptions.

https://doi.org/10.1371/journal.pone.0304082.t007

5.4 Evaluation metrics for the AWGO optimized ensemble model

The performance of the deep learning model was evaluated using several key metrics: Accuracy—Measures the ratio of correct predictions to total predictions. This gives an overall idea of how often the model is correct. Precision—The ratio of true positives to true positives plus false positives. Shows how useful the positive predictions are. Recall—The ratio of true positives to true positives plus false negatives. Indicates the model’s ability to find all positive cases. F1-score—Harmonic means of precision and recall, combining them into a single metric. Support—The number of samples for each class. Allows interpretation of other metrics per class. ROC-AUC encapsulates the balance between the true positive rate and the false positive rate as the threshold is varied. Values range from 0 to 1, with higher indicating better classification. These metrics were computed on both the encoded training set and the validation set. The validation set analysis shows how the model generalizes to new data. Hyper-parameters like learning rate were tuned via grey wolf optimization to maximize validation accuracy. Taken together, these metrics enable a thorough evaluation of the optimized ensemble model’s classification performance. Accuracy measures the ratio of correct to total predictions. (21)

Precision is the ratio of true positives to true positives to false positives. (22)

The recall is the ratio of true positives to true positives and false negatives. (23)

The F1-score combines precision and recall into the harmonic mean as follows: (24) Support indicates the number of samples per class. Feeding the reduced fog-optimized features into the cloud-hosted ensemble classifier yields strong and consistent anomaly detection performance spanning traditional networks, enterprise environments, and wireless deployments. On the NSL-KDD benchmark, multi-class discrimination between normal traffic and attack varieties, such as denial-of-service and remote access breaches, achieves 99.7% accuracy with 99% precision, recall, and F1 score per class, as documented in Table 8, showing reliable identification capabilities. The overall precision is 99.9%, while recall reaches 99.9%, as shown in Table 8, highlighting balanced performance generalization across the classes. Binary segregation between benign and malicious flows retains an accuracy of 99.5 with perfect precision, recall, and F1 for normal behaviour, as shown in Table 9. For attacks, the precision was 100% and recall was 99.9% (Table 9), demonstrating robust performance despite the prevalence differences. The NSL-KDD confusion matrices in Figs 11(a) and 12(a) (see Figs 11 and 12) validate these efficacies through significantly high correct labelling rates and low mispredictions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11.

Display the confusion matrices for binary classification across the NSL-KDD (a), UNSW-NB15 (b), and AWID (c) datasets. The diagonal elements, predominantly high, reflect the accurate classification of normal and attack traffic by the ensemble model.

https://doi.org/10.1371/journal.pone.0304082.g011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12.

Displays the confusion matrices for multi-class classification across the NSL-KDD (a), UNSW-NB15 (b), and AWID (c) datasets. The matrices exhibit strong diagonal elements, each exceeding 99%, indicating the precise classification of diverse attack types. This underscores the ensemble model’s accuracy in distinguishing intricate malicious behaviors such as denial-of-service, remote access, probing, injections, impersonation, flooding, and worm attacks.

https://doi.org/10.1371/journal.pone.0304082.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. NLS KDD multiclass classification performance.

https://doi.org/10.1371/journal.pone.0304082.t008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. NLS KDD binary classification results.

https://doi.org/10.1371/journal.pone.0304082.t009

Assessing the UNSW-NB15 contemporary hybrid enterprise benchmark containing 10 classes, the model distinguishes threats such as worms, shellcodes, and reconnaissance from normal traffic with 99.16% accuracy, as displayed in Table 10. The harmonious 0.991 F1 score in Table 10 indicates a balance between precision and recall across the classes. The accuracy for the crucial binary benign vs. malicious classification accuracy reached 99.6%, as shown in Table 11, enabled by the predictive features. The UNSW-NB15 confusion matrices in Figs 11(b) and 12(b). Figs 11 and 12 further substantiate the performance primarily via diagonal high values representing accurate classification and minimal misdirection. For the wireless AWID benchmark, multi-class discrimination between flooding, impersonation, and injection attacks exceeds 99% accuracy, with all precision, recall, and F1 metrics surpassing 0.99 as well in Table 12, corroborating reliable identification. Binary classification achieved 99.9% attack detection accuracy, as shown in Table 13. The AWID outcomes were reinforced through the confusion matrices in Figs 11(c) and 12(c) (see Figs 11 and 12), exhibiting comprehensive correct labelling via high diagonal values and limited errors. The consistently high performance is further validated through the ROC curves in Figs 13 and 14, showing robust true positive vs. false positive trade-offs with curve areas approaching, indicating stability across the operating thresholds. Comparisons using different autoencoder types of evidence-stacked autoencoders yielded 99% accuracy with the top F1-score and ROC-AUC, as shown in Table 14, confirming the integrated pipeline.The empirical results substantiate the proposed framework combining fog-stacked autoencoder feature extraction, CatBoost predictive feature selection, and a cloud-hosted adaptively grey wolf optimized Transformer-CNN-LSTM ensemble classifier that delivers efficient yet accurate network intrusion detection across traditional, modern enterprise, and wireless domains. The results validated the integration of stacked autoencoders for feature engineering in fog nodes, CatBoost for input space refinement through predictive selections, and an optimized transformer-CNN-LSTM ensemble model in the cloud combined end-to-end to deliver efficient yet accurate intrusion detection. The pipeline generalizes reliably across NLS-KDD, UNSW-NB15. Figs 15 and 16 show the training and testing accuracy plots over epochs for the optimized ensemble model on the binary and multi-class classification tasks across the NSL-KDD, UNSW-NB15, and AWID intrusion detection datasets. The temporal convergence patterns demonstrate effective learning dynamics, with the training accuracy closely tracking the testing accuracy in all cases. The minimal gaps between them highlight the fact that the model generalizes reliably to unseen data without substantial overfitting. The fast saturation to high-accuracy levels over epochs validates the modelling capability to distinguish between attack types and normal behaviours.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. ROC curves for binary classification of normal vs. attack traffic.

This figure presents the ROC curves for binary classification between normal and attack traffic across three datasets: (a) NSL-KDD, (b) UNSW-NB15, and (c) AWID. Each subplot demonstrates near-perfect classification performance with Area Under the Curve (AUC) values of 1.00. The ROC curves, represented by the orange lines, closely follow the top-left corner of the plots, indicating excellent discrimination ability between normal and attack instances. This performance is characterized by high true positive rates achieved while maintaining very low false positive rates across all discrimination thresholds. The consistency of these results across different datasets underscores the robustness and generalizability of the ensemble intrusion detection model, suggesting its potential effectiveness in real-world deployments for IoT and fog computing settings.

https://doi.org/10.1371/journal.pone.0304082.g013

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. ROC Curves for multi-classification illustrating multi-class intrusion detection performance across three datasets.

(a) NSL-KDD dataset: near-perfect classification for Normal, DoS, Probe, Privilege, and Access classes. (b) UNSW-NB15 dataset: varied performance across multiple attack types, with high AUC values for classes like Fuzzers (0.93), Generic (0.98), and Reconnaissance (0.90), and lower values for others such as Analysis and Backdoors (both 0.51). (c) AWID dataset: excellent discrimination capability for flood, impres, and inject attacks (AUC: 0.99, 0.98, 1.00 respectively) and Normal traffic (0.99). These results demonstrate the efficacy of the proposed ensemble learning approach with advanced feature selection for optimized intrusion detection in IoT and fog computing environments.

https://doi.org/10.1371/journal.pone.0304082.g014

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15.

Displays the training and testing accuracy over 100 epochs for binary classification across three datasets: (a) NSL-KDD, (b) UNSW-NB15, and (c) AWID. The graphs illustrate the performance of the ensemble model in distinguishing between normal and attack traffic, demonstrating consistent improvement in accuracy over the training period.

https://doi.org/10.1371/journal.pone.0304082.g015

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16.

Display the training and testing accuracy of the ensemble model for multi-class classification over 100 epochs across three datasets: (a) NSL-KDD, (b) UNSW-NB15, and (c) AWID. The plots illustrate the model’s ability to accurately classify multiple attack types, demonstrating its robust performance in handling diverse malicious behaviors.

https://doi.org/10.1371/journal.pone.0304082.g016

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. UNSWB15 multi-classification results.

https://doi.org/10.1371/journal.pone.0304082.t010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Unswb15 binary classification results.

https://doi.org/10.1371/journal.pone.0304082.t011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 12. AWID multi-classification results.

https://doi.org/10.1371/journal.pone.0304082.t012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 13. AWID binary classification results.

https://doi.org/10.1371/journal.pone.0304082.t013

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 14. Benchmarking against other classification methods.

https://doi.org/10.1371/journal.pone.0304082.t014

5.5 Comparative study

The proposed intrusion detection framework demonstrated consistent accuracy improvements over the related prior art, validating the benefits of our specialized innovations in feature engineering, ensemble deep learning, and bio-inspired optimization. Specifically, we achieved 7% higher detection accuracy on NSL-KDD compared with the 92.3% attained by Doaa and Osama [24] embedded autoencoder approach through our fog-cloud synergy. Furthermore, we attain 99.9% accuracy on the wireless AWID benchmark, surpassing Shafiq et al.’s [51] 97.0% IoT botnet detection rate using SVM and random forests via our deep ensemble tailoring. Additionally, we achieved 99.9% accuracy on the more contemporary NSL-KDD dataset compared with Khammassi and Krichen’s [29] 99.8% accuracy on the older KDDCup99 benchmark. Lastly, we significantly boost Al-Yaseen et al.’s [1] 90.6% NSL-KDD detection using shallow ensembles to 99.7% through feature-engineered deep learning suitable for fog nodes. Our techniques positively impact the accuracy across evaluation datasets spanning traditional, modern, and wireless IoT environments. Table 15 summarizes the results of the comparative evaluation. The outcomes highlight the benefits of the proposed innovations over prior shallow learning and deep learning baselines across the evaluation datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 15. Comparison with other methods.

https://doi.org/10.1371/journal.pone.0304082.t015

The evaluations demonstrate that our integrated framework, combining autoencoder-based feature extraction, CatBoost feature optimization, and an ensemble of Transformer, CNN, and LSTM networks, outperforms previous shallow and deep learning intrusion detection techniques. The results highlight the benefits of tailored feature engineering, model optimization, and specialized deep ensemble design for securing the IoT and fog infrastructure.

5.6 Ablation study on model components

To ascertain the significance of each constituent module, targeted ablation experiments were conducted by selectively excluding the components and quantifying the resultant performance impact: stacked autoencoders (SAE) for feature extraction, CatBoost for predictive feature selection, grey wolf optimizer for tuning, transformer network, 1D CNN layers, and LSTM network, which lowered the detection accuracy by 2.3% on the NSL-KDD dataset, highlighting the encoding efficacy for dimensionality reduction. Excluding CatBoost decreases the precision by 1.5%, confirming its role in distilling predictive features. The discarding optimizer decreases the ROC-AUC by 1.3%, validating the tuned generalization. Omitting LSTM and CNN layers reduces recall by 0.9% and 0.7%, respectively, confirming localized pattern identification capabilities, and the consistent performance degradation when individually removing each module affirms that they collectively unify synergistically. Specialized integration of complementary deep neural networks boosts intrusion-detection fidelity. Customizing architectures and hyperparameters are vital for maximizing suitability across diverse threats. Modular upgradability also lowers barriers to substituting more advanced models as research progresses to match dynamic attacks, and by quantifying accuracy, recall, and ROC-AUC differences from ablating SAE, CatBoost, 1D CNN, LSTM, and optimizer, the experiments validate the specialized tuning of the building blocks that push boundaries. The integrated framework distills multidimensional synergies across specialized components, demonstrating a significant headway. Full proposed model 99.7% Without SAE feature extraction 97.2%Without CatBoost feature selection 98.1% Without Transformer branch 98.3% Without CNN branch 98.5 Without LSTM branch 98.6% Without Grey Wolf optimization 98.9% The outcomes show that removing any component results in a drop in accuracy compared to the complete model. This affirms that each element provides unique value to the overall framework. The SAE and CatBoost feature optimization had the largest impact, with a >2% accuracy decrease when excluded. The neural network branches and grey wolf optimization contributed smaller but still significant improvements of 0.2-1%. Furthermore, ablation studies have quantified the contributions of the individual pipeline components. Excluding the SAE, CatBoost, optimizer, and LSTM modules independently incur accuracy losses of 2.5%, 1.6%, 1.5%, and 0.8%, respectively, on NSL-KDD. This demonstrates the synergistic additions that achieve the highest-performing integrated configuration. Modularity also simplifies the substitution of new algorithms to sustain state-of-the-art capabilities.

5.7 Implications

This study pioneered a specialized intrusion detection methodology spanning optimally engineered feature extraction techniques using stacked autoencoders, an integrated Transformer-CNN-LSTM ensemble classifier architecture tuned by leveraging adaptive grey wolf optimization, and a cohesive fog-to-cloud distributed deployment framework with noteworthy multi-disciplinary implications. First, for emerging fog computing infrastructure, the proposed solution delivers an intrusion detection framework explicitly customized for resource-limited edge nodes while harnessing complementary cloud capabilities. Next, concerning bolstering security for IoT ecosystems, the consistent near-perfect evaluation performance across datasets demonstrates generalizability to diverse operational environments using technologies ranging from traditional IT to cutting-edge wireless sensing. Additionally, to advance intelligent intrusion detection systems, the proposed model establishes competitive accuracy while pioneering innovations in neural architecture synergies. Finally, regarding machine-learning contributions, the framework unveils a novel methodology combining unsupervised representation learning, supervised predictive feature selection, and biologically inspired hyperparameter optimization. Overall, cross-disciplinary techniques pave the way for more secure and intelligent fog-computing and IoT platforms. As cyber-physical infrastructure proliferates globally, AI-driven intrusion detection capabilities will become increasingly indispensable. This research aims to make vital contributions towards that vision through an integrated framework that synergizes edge, cloud, and machine learning innovations for more trustworthy networks.

5.8 Feasibility analysis

While the results demonstrate strong classification accuracy and F1 scores, further analysis is imperative to quantify the processing and memory requirements for real-time deployment on resource-constrained fog nodes. Our customized SAE architecture requires only 9,447 parameters, resulting in a compact model size of 36.90KB based on 4 bytes per parameter. Evaluations on an Intel i7 desktop processor indicated that the SAE encoding throughput exceeded 60,000 samples per second. Encoding the 41 features in NSL-KDD required 418 FLOPs per sample, demonstrating a 5x reduction versus the original input size. This enables efficient feature extraction on fog hardware. The CatBoost model adds marginal overhead for feature selection, using 1,068KB for 500 boosting trees of depth 8 and achieving feature importance computation in 0.6 ms per sample. Prediction latency remains under 1 ms, facilitating real-time edge usage even on Raspberry Pi-level processors. The Transformer-CNN-LSTM ensemble encompasses 68,166 parameters (266.27KB model size) for the 64-node architectures. However, the cloud-hosted environment has relaxed computational constraints, allowing ensemble evaluation in 9.8ms per sample to detect threats based on edge-extracted features. By leveraging SAE compression and transmitting only 16 node codes, the communication overhead was lowered by more than 8x versus sending raw traffic data. The benchmarks showcase specialized designs that tailor computational load, memory usage, and communication flows to operate within fog resource limits while harnessing the cloud’s superior analytics capabilities.

5.9 Online adaptation

Although the proposed framework demonstrates reliable detection capability, real-world deployments require accommodating evolving threats. Continuous model evolution functionality could allow tailored transformers, CNN, and LSTM components to assimilate new attack patterns. One approach is to periodically retrain custom neural architectures on newly gathered data using transfer learning, allowing the models to retain prior knowledge while adapting to new samples. Additionally, the entire ensemble pipeline could be re-optimized using grey wolf techniques as new traffic is analyzed to choose optimal contemporary architectures and parameters. This would enable a dynamic response to shifts in data properties over time rather than using fixed blueprints. Architecting sustainable analytics engines adept at assimilating emerging threat indicators through updated model selection, transfer-tuned retraining or other techniques will prove vital for long-term robustness against sophisticated attacks. In essence, the key idea is to enable customized deep learning components such as the transformer as well as optimization strategies to continuously modernize based on new data patterns rather than remain static.