2.1 Structure of the paper

The remainder of this study is structured as follows. In Section 2, the proposed approach is presented along with the data processing, machine learning techniques, clustering, fuzzy logic, and threshold optimization. In Section 3, the results of the experiments are evaluated, and their performances are compared with those of the baseline models. In Section 4, model explainability is discussed with reference to explainable AI (XAI), and Section 5 concludes with a summary of the important findings and suggestions for future research.

3.1 Dataset description

We used a login dataset [19,20] that included 15 attributes and synthesized feature data from 33 million login attempts made by 3.3 million users. The purpose of this dataset is to further develop RBA system research and development. It was produced using actual login activity observed in a major Norwegian single sign-on (SSO) service. The properties of these datasets are listed in Table 1. of the 33 million records, we chose a random subsect of 2 million for the initial model selection step to ensure a representative sample of different login behaviors and attack patterns. This enabled us to perform an efficient comparison of the machine learning models in terms of precision, recall, and F1-score. The best-performing model was then selected, and we trained and evaluated it with the 30M dataset to verify the strength of its performance and suitability for practical use. Thus, we achieved a compromise between a computationally intensive process and a thorough examination of the model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Dataset attribute description details.

https://doi.org/10.1371/journal.pone.0349095.t001

The features chosen and their respective roles in detecting account takeovers are as follows: User agent string and device type changes indicate a device or browser switch; operating system and version indicate unexpected OS logins; login time and day of the week indicate out-of-pattern logins; round-trip time refers to network anomalies, such as proxies or VPNs; attack IP points to logins from malicious IPs; and Account Takeover, as the target variable, enables the model to learn from past compromises.

3.2 Exploratory data analysis

The first step in the ML model building process is dataset analysis, which aims to identify the main characteristics of the dataset, and is usually performed visual methods. This step is significant because it prepares the model by showing the data distribution. It is important to examine skewness, outliers, and the data distribution in general. For features that were highly skewed, a log transformation was performed, which resulted in data that were more suitable for modeling. This method is indeed very much effective in feature engineering and preprocessing as it helps in making the data properly organized and in the right condition for analysis. The (Figs 3a and 3b) portray this process.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. (a) shows box plots that help visually detect outliers, while (b) demonstrates addressing skewness by applying a log transformation.

https://doi.org/10.1371/journal.pone.0349095.g003

3.3 Feature engineering

The main login patterns were identified using time-based feature extraction. The feature login_hour indicates the time of login, which is helpful in identifying strange login times. The feature login_day_of_week tells which day of the week it is, so one can analyze user activity during the week and the weekend. Additionally, the feature login_is_weekend generates a binary signal for weekend logins, hence, it gives an idea of login patterns during non-working days. All these features have a positive influence on the model’s ability to detect irregularities in login activity.

3.4 Data preprocessing and splitting

At this point, we direct our attention to the process of feature extraction, which takes place in two steps: one for the numerical and the other for the categorical features. The numerical features of this model are ASN, login_hour, and login_day_of_week, while the categorical features are country, Region, City, user agent string, OS name and version, device type, and login_is_weekend. To make the numerical features suitable for mixing with the model, we used the standard scalar method for scaling, which means that the features were changed to have a mean of zero and a standard deviation of one. However, this scaling is vital mainly for building blocks that rely on the size of the feature, as it guarantees that no numeric feature is favored or treated differently than others. For the categorical attributes, we used one-hot encoding, and every category was turned into a binary matrix where all categories created a column of ones (for the existing category) and zeros (for others). The Column Transformer executed these conversions, and we simultaneously scaled the numeric features and encoded the categorical features. This combined strategy means that all preprocessing steps take place uniformly in the training and testing datasets, thus preserving the data integrity throughout the process [21]. First, the login dataset was divided using a stratified sampling method where the training and test sets were created by applying a train_test_split with an 80−20 ratio between them. Stratification was implemented to maintain the same class proportions in both sets.

3.5 ML models

The ensemble model with a Voting Classifier that combined Gradient Boosting, SVM, and XGBoost was used to classify account takeover attempts and normal activities with better accuracy and reliability after preprocessing. Among various machine learning models, XGBoost, and SVM were preferred over deep learning models due to their computational efficiency, good performance with small datasets, ease of interpretation, and faster training time. Moreover, these models are less likely to overfit which is a prominent feature for real-time applications as well as for processing large amounts of data. The SVM was selected for its capability to deal with nonlinear decision boundaries in high-dimensional areas with slight class margins. In addition to being able to work with unbalanced datasets, XGBoost also interacts with the features, deals with missing data, and minimizes overfitting through regularization. Gradient boosting was preferred because of its iterative learning which, in turn, improved accuracy in the case of large datasets with complex patterns. The hyperparameters for each model were as follows: gradient boosting learning rate = 0.1, n_estimators = 100, max_depth = 3; SVM with an RBF kernel, C = 1, and probability estimates; and XGBoost learning rate of 0.1, n_estimators = 100, max_depth = 6, and log loss as the evaluation metric.

3.5.1 Gradient boosting classifier.

In gradient boosting, the model is built sequentially, with each stage contributing a weak learner to solve the errors in the previous models. In each stage m, the technique incorporates a new function h_m(x) into the previous model to minimize the differentiable loss function .

(1)

In Equation 1, F_m(x) denotes the model’s prediction at stage m. The parameter represents the learning rate, which controls the impact of weak learners on the overall model. The term h_m(x) refers to a weak learner in stage m. The purpose of each new weak learner, denoted by h_m(x), is to reduce the residual error through training by conforming to the loss function negative gradient . Using this approach, the model can correct previous errors. Specifically, the model adds learner h_m(x), which approximates

(2)

After a specific number of iterations, this procedure produces the final model F_M(x), which is a weighted combination of all weak learners, as shown in Equation 3 where M is the total number of iterations.

(3)

By progressively incorporating these weak learners, the accuracy of the model is improved by rectifying the earlier errors in each step [22,23].

3.5.2 Support vector machine (SVM).

SVM is a key component of the voting classifier ensemble. SVC seeks to identify the optimum hyperplane for optimizing the margin between classes. When the data cannot be separated linearly, slack variables are introduced to accommodate misclassifications. The optimization function is defined as follows:

(4)

subject to:

(5)

where w is the weight vector for the hyperplane and C is the regularization value that optimizes the margin while minimizing the classification error. In this model, the SVM model was calibrated with the probability = true. By doing this, the SVM was able to provide class probabilities instead of just binary predictions. The SVC predictions suitable for soft voting are obtained by applying a logistic regression model to the decision values. The class_weight = ’balanced’ parameter is very advantageous for the model dealing with imbalanced datasets, as it automatically assigns weights to the classes relative to their frequency in the dataset.

The SVM classifier’s probability estimates in the ensemble were combined with those of the Gradient Boosting and XGBoost classifiers using a soft voting technique, which averaged the predicted probabilities. This method takes advantage of SVM’s strength in handling complicated decision boundaries and thus contributes to the total strength of the ensemble. The ensemble method increases the precision and versatility by merging the probability-based forecasts of all the classifiers, which is especially advantageous for imbalanced datasets [24,25].

3.5.3 XGBoost.

XGBoost was used in the ensemble model to make probability estimates, which were then averaged through the predictions of the Gradient Boosting and SVC classifier. XGBoost improves generalization through gradient boosting with combined regularization, as indicated by its objective function in Equation 6.

(6)

The log loss function for classification is represented as and regularization term

(7)(8)

Parameter T denotes the total number of leaf nodes, indicates the penalty for each leaf node, and corresponds to the L2 regularization applied to the weights w, thereby facilitating the control of the model’s complexity to avoid overfitting [26]. In the final stage of the ensemble model, the predictions of gradient boosting, SVC, and XGBoost are forwarded to the Voting classifier, which combines their outputs. This is through a soft voting strategy, whereby the overall probability for each class is calculated by considering the average of the predicted probabilities from all classifiers involved. A mathematical description of this process is given in Equation 9.

(9)

Here, P_ensemble(y) is the class probability predicted by the Voting Classifier. N = 3 indicates that three classifiers are involved. P_i(y) represents the probability of class y for the i-th classifier. By averaging these probabilities, the Voting Classifier exploits the strength of each model to obtain a more reliable and accurate prediction [27].

3.6 Model evaluation

Predictions were made by the classifier, and the model’s effectiveness was assessed using several key metrics, such as the classification report and ROC-AUC score [28]. The classification report encompasses accuracy, precision, recall, and F1-score metrics for both weighted and macro averages; further information is provided in Table 2. The ROC-AUC score reflects the area beneath the ROC curve, which is determined using Equation 10, thus indicating the model’s capability of distinguishing between positive and negative classes. An AUC score closer to 1 signifies better performance.

(10)

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Performance metrics and their mathematical definitions.

https://doi.org/10.1371/journal.pone.0349095.t002

To determine the initial medium threshold for classification, the optimal threshold (denoted as ) is selected by maximizing the difference between the true positive rate (TPR) and the false positive rate (FPR), known as Youden’s Index. This is mathematically expressed as:

(11)

3.7 Clustering

K-means clustering [29] was selected for its simplicity, low computational cost, and ability to handle large-scale datasets. Its centroid-based nature partitions the data into clusters by assigning each data point to the nearest cluster center, thereby minimizing the total intra-cluster variance (sum of squared distances between data points and their respective centroids). This results in clearer separation of data groups and improves classification performance by distinguishing low- and high-risk patterns more effectively. Compared to other clustering techniques, K-means is computationally efficient and well-suited for large datasets, which is essential for the proposed authentication framework. The objective function of K-means clustering is given by:

(12)

where S_i represents the set of points in the i-th cluster, denotes the centroid of th i-th cluster, and k is the total number of clusters.

In the proposed framework, K-means clustering is used to refine the low and high thresholds based on predicted probability scores. After determining the medium threshold using ROC analysis, the probability values are divided into two regions: values below the medium threshold and values above it. K-means clustering (with k = 2) is then applied separately to each region.

3.8 Fuzzy logic system

This study employs fuzzy logic [30] to categorize predicted probabilities into three risk levels: “low,” “medium,” and “high,” thereby enhancing the model’s decision-making abilities. The categories were established using triangular membership functions, where the parameters a, b, and c represent the start, peak, and end points, respectively. The variable x denotes the predicted probability.

(13)

The decision-making process of the system is guided by fuzzy inference rules such as IF; the probability is low; THEN the risk is low; IF the probability is medium, THEN the risk is medium; IF the probability is high; and the risk is high. The min-max approach is employed to merge these rules and assess the strength of each rule’s output. Finally, the centroid method is employed to represent the results for a particular risk value, as indicated in Equation 14, which is subsequently applied to the final decision making process in the classification task.

(14)

3.9 Threshold optimization

To improve the classification thresholds, the Limited-memory Broyden–Fletcher–Goldfarb–Shanno with box constraints (L-BFGS-B) [31,32] optimization algorithm is employed. The objective of the optimization is to improve the consistency of authentication decisions by minimizing the difference between initial and updated decisions.

(15)

where N is the total number of instances, a_i represents the initial decision (action), and denotes the updated decision after applying optimized thresholds. The objective function evaluates the consistency between initial and optimized decisions. This formulation ensures that the optimization process refines the thresholds while preserving stable decision behavior. Initial threshold values for low, medium, and high risk are set based on prior analysis. The optimization is performed under box constraints to ensure valid probability ranges:

(16)

The L-BFGS-B algorithm updates the thresholds iteratively using a gradient-based approach:

(17)

where x_k represents the current threshold values, is the gradient of the objective function, and is the step size. The optimization continues until convergence criteria are met, including minimal change in objective value, minimal change in threshold values, or reaching the maximum number of iterations. The optimized thresholds are then used within the fuzzy logic system to assign final risk levels.

The optimization process continues until one of the following convergence criteria is satisfied:

1. Convergence in Function Value: The change in the value of the objective function becomes sufficiently small between iterations.
2. Convergence in Parameter Values: The change in the threshold values between iterations becomes sufficiently small.
3. Maximum Iterations: Optimization stops after reaching the maximum number of iterations (maxiter = 20) if convergence is not achieved. The algorithm also terminates if the change in threshold values falls below the tolerance value ().

Once the algorithm converges, the optimized threshold values are obtained as . These thresholds represent the values that improve the consistency of authentication decisions. After optimization, the refined thresholds are integrated into the fuzzy logic system, which assigns risk levels based on the optimized threshold values.

4.1 ML and DL Models

This study evaluates the performance of different machine learning models, including Gradient Boosting (GB), Random Forest (RF), Support Vector Machine (SVM), and XGBoost (XGB), along with ensemble and deep learning approaches, using six metrics: accuracy, precision, recall, F1-score, MCC, and EER. The results shown in Table 3 highlight the differences between models in terms of classification performance and authentication effectiveness. For example, XGB (81.87% accuracy, 82.2% precision) and SVM (84.1% accuracy, 84.1% precision) show moderate performance, while GB (84.3% accuracy, 85.1% precision) produces relatively more false positives. To improve performance, ensemble methods are applied. Among them, the Soft Voting model (GB + SVM + XGB) performs the best, achieving 97.77% accuracy, 99.41% precision, 98.04% recall, and 98.72% F1-score. It also has the highest MCC (0.9035) and the lowest EER (0.0303), showing better reliability. The ROC curve in (Fig 4) further supports its strong performance.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Model performance comparison (results calculated for 2M dataset size).

https://doi.org/10.1371/journal.pone.0349095.t003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. ROC cure for ensemble model for 2M dataset size.

https://doi.org/10.1371/journal.pone.0349095.g004

In comparison, deep learning models such as MLP and LSTM also perform well, with accuracies of 94.8% and 93.9%, respectively. However, their performance is still lower than the proposed ensemble model, especially in terms of MCC and EER. Other ensemble methods also show slightly lower performance. Overall, the Soft Voting (GB + SVM + XGB) model gives the best results across all metrics and is the most suitable choice for risk-based authentication.

To further assess the stability of the proposed model and examine potential overfitting, repeated runs and cross-validation were performed. The model achieved an accuracy of 97.77% ± 0.21 across repeated runs. In addition, 5-fold cross-validation resulted in a mean accuracy of 97.65% with a standard deviation of 0.32, and a 95% confidence interval of [97.20%, 98.10%]. These results indicate consistent performance with low variance, suggesting that the model maintains stable performance and is not overly fitted to the dataset.

4.2 Threshold and action count analysis for ensemble model

The ensemble model showed a significant improvement when fuzzy logic was applied to the ROC + clustering thresholds. The low threshold increased significantly from 1.9601e⁻⁰⁶ (ROC) to 5.8246e⁻⁰⁶ (ROC + Clustering), the medium threshold remained the same at 0.3371, and the high threshold increased to 0.6928. Probability clustering visualizations are shown in (Fig 5).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. (a) Displays the thresholds from the ROC method before clustering, where (b) shows probability clustering using ROC  +  Cluster, which improves the classification boundaries for the 2M dataset.

https://doi.org/10.1371/journal.pone.0349095.g005

First, the model was able to process 89,798 allowed login actions, after which 208,335 actions were tagged for extra authentication and 101,867 actions were without login. The combination of ROC + clustering and fuzzy logic brought about improvement, which is reflected in Table 4, and the new figures were 349,432 low-login actions, 48,132 requests for additional authentication, and 2,436 denying login actions. The results demonstrate that the ensemble model is very effective at improving the decision process through threshold adjustment, thus making it a reliable option for this application.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Consolidated table for threshold values and action counts (2M dataset).

https://doi.org/10.1371/journal.pone.0349095.t004

4.3 Threshold optimization

In the case of the ensemble method, we utilized several optimization strategies, namely PSO, Bayesian Optimization, CMA-ES, Simulated Annealing, and Gradient-based optimization (L-BFGS-B). PSO made a significant difference in the results for the low and high thresholds, however, Bayesian optimization and CMA-ES were more conservative in their impact on the changes. Simulated annealing provided balanced adjustments, and L-BFGS-B produced stable thresholds, particularly at medium and high values. (Fig 6) illustrates the convergence of fitness values throughout the optimization iterations, as well as the decision boundaries before and after optimization using L-BFGS-B for the 2M dataset size. The L-BFGS-B optimizer proved to be the most effective for fine-tuning the threshold values and improving decision-making. Initially, the low-, medium-, and high-risk thresholds were 5.8246e⁻⁰⁶, 0.3371, and 0.6928, respectively. After the optimization process, the values for the decision boundaries were set to 0.3000, 0.5000, and 0.7000 respectively, to provide more distinct decision boundaries, as shown in Table 5. L-BFGS-B yielded the most distinct thresholds in contrast with other methods, where the thresholds resulteing from PSO (0.0000, 0.3277, 0.9000), Bayesian (0.0017, 0.1026, 0.2324), CMA-ES (0.0076, 0.1138, 0.5655), and simulated annealing (0.0264, 0.2284, 0.3617) overlappe or were inconsistent, thus leading to less reliable risk classification.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Threshold values and action counts before and after optimization for ensemble method (2M dataset).

https://doi.org/10.1371/journal.pone.0349095.t005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. (a) illustrates the convergence of the fitness values, whereas (b) depicts the decision boundaries before and after optimization for the 2M dataset size using L-BFGS-B.

https://doi.org/10.1371/journal.pone.0349095.g006

Furthermore, the L-BFGS-B method that was used for optimization brought about notable changes in the authentication actions, particularly the allowed logins which were increased to 349923, the extra login requests which were minimized to 48849, and the cases of Deny Login which were rightly calibrated to 1228, thus paving the way for a good trade-off between security and user-friendliness as depicted in Table 5. On the contrary, the other methods like PSO, Bayesian, and CMA-ES, merely maintained 349432 logins, 49106 additional authentication requests, and 1462 denials, indicating that they had less room for flexibility in the improvement of security actions. Simulated Annealing made some improvements, but it was still less effective than L-BFGS-B, which offered the best balance between security and usability. The distinct threshold separation, the optimized authentication actions, and the refined risk classification, which are shown in Table 5, all lead to the conclusion that L-BFGS-B is the most effective of the optimization methods used in this research.

The performance of the proposed ensemble model combined with fuzzy logic and L-BFGS-B optimization is presented in Table 6 across varying dataset sizes ranging from 2M to 30M records. The results indicate that, after optimization, significant improvements in accuracy and precision are achieved. The Soft Voting (GB + SVM + XGB) model achieves the highest accuracy of 97.77% on the 2M dataset, while the accuracy ranges from 79.9% on the 4M dataset to a peak of 98.14% on the 20M dataset. Precision remains consistently high, reaching a maximum of 99.41% for the 2M dataset. After optimization, threshold values for low, medium, and high-risk categories are adjusted, resulting in more reliable decision boundaries. Consequently, the counts of Allow Login and Deny Login actions improve, reflecting enhanced classification and decision-making performance. As a specific example, for the 30M dataset, the number of Allow Login actions increases from 5,504,412–5,705,847, while Deny Login actions decrease from 29,807–9,875 after optimization. (Figs 7–9) illustrate the overall results for the 30M dataset.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Data size, accuracy, precision, recall, F1-score and optimization results.

https://doi.org/10.1371/journal.pone.0349095.t006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. 30M dataset size performance: Confusion matrix & fuzzy output.

https://doi.org/10.1371/journal.pone.0349095.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. 30M dataset size performance: Fitness convergence.

https://doi.org/10.1371/journal.pone.0349095.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of decision boundaries before and after optimization for 30 million dataset Size.

https://doi.org/10.1371/journal.pone.0349095.g009

The observed variation in performance across different dataset sizes is primarily influenced by differences in data distribution, class balance, and sample representation. Smaller datasets may exhibit higher variability due to limited samples, while larger datasets provide more stable and diverse patterns, enabling better generalization. In addition, the proposed framework relies on probability-based thresholding and clustering mechanisms, which become more effective as the dataset size increases. This leads to improved separation of risk categories and more consistent authentication decisions for larger datasets.

4.4 Ablation study of the proposed framework

An ablation study is performed to understand the role of each component in the proposed framework. The ensemble model provides high classification performance, while the additional modules improve authentication decisions based on predicted probabilities. ROC-based thresholding provides an initial decision boundary, but it results in a high number of additional authentication requests and denied logins. Clustering improves this by generating adaptive thresholds from the data, leading to a better distribution of decisions. Fuzzy logic handles uncertainty by enabling gradual transitions between risk levels instead of rigid decisions. Finally, L-BFGS-B optimization adjusts the thresholds to achieve a better balance between security and usability.

The results show that each component contributes to improving the quality of authentication decisions, particularly by reducing unnecessary authentication requests and minimizing false denials (Table 7).

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Ablation study of the proposed framework.

https://doi.org/10.1371/journal.pone.0349095.t007

4.5 Adversarial robustness analysis

In real-world authentication systems, input data may be affected by noise, missing values, or potential manipulation. Therefore, it is important to examine how the proposed model behaves under such conditions. Robustness in this context refers to the ability of the model to maintain reasonably stable performance when the input data is slightly altered. To evaluate this, controlled perturbations were introduced into the dataset, including Gaussian noise, missing feature values, and simulated attack scenarios. Gaussian noise with zero mean ( = 0) was applied to ensure unbiased perturbation without shifting the original data distribution. Missing data was simulated by randomly masking a portion of the features, while simulated attack scenarios were created by scaling feature values to represent abnormal login behavior.

The results are presented in Table 8. The model achieves a baseline accuracy of 97.77% on normal data. When small perturbations are introduced ( = 0.05), only a slight decrease in performance is observed. With increased perturbation ( = 0.10), the performance decreases moderately but remains acceptable. In the case of incomplete data, the model continues to perform well, indicating its ability to handle missing information. Under simulated attack conditions, the model maintains reasonable performance, suggesting its capability to detect abnormal patterns. Overall, the results indicate that the proposed model is reasonably stable under different perturbation conditions, making it suitable for practical authentication scenarios.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Robustness analysis under perturbed and simulated attack conditions.

https://doi.org/10.1371/journal.pone.0349095.t008

5.1 Local explainability

Different methods have been applied to explain the predictions at the individual level. Here are waterfalls and decision plots to show the model’s behavior clearer for particular cases. The SHAP waterfall plot in (Fig 10a) illustrates the breakdown of every feature’s contribution to the model’s prediction for a single instance. The model’s base value is −12.494, which indicates the expected output before adding any feature contributions. The User Agent String feature accounts for −0.04, thus reducing the model’s prediction, where City Sossiprolo and City Potlogi each account for −0.01, which is only a minor decrease in the prediction. The login_day_of_week feature accounts for −0.01, which means that logins on this specific day have a slight negative impact on the event being classified as account takeover. In the same way, Petrachioia and ASN have a negative contribution of −0.01, thus making the model prediction even less likely. The final prediction after summing all feature contributions is −12.444, which indicate that this specific login attempt is categorized as not suspicious (normal). This breakdown shows that although the contribution of each feature may be small, it is still significant in determining the model output. The SHAP decision plot presented in (Fig 10b) has a starting point of −12.494, which indicates the average model output before any feature is considered. As different features contribute SHAP values, the blue line reflects the overall impact of these contributions. The user-agent string feature has a negative contribution, slightly lowering the prediction and suggesting a normal login. Similarly, the characteristics of the cities, Sossiprolo, Potlogi, and Petrachioia, also contribute negatively, further reducing the prediction. However, the Login Day of Week feature contributes positively, slightly increasing the prediction for a normal login. Finally, the ASN has a small negative contribution, that decreases the likelihood of an account takeover. The final prediction value of −12.44 indicates that the log-in is classified as non-suspicious.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Local explainability of the model is shown in (a) with the Waterfall Plot for the first instance and in (b) with the Decision Plot for the first 10 instances, displaying feature contributions to the predictions.

https://doi.org/10.1371/journal.pone.0349095.g010

5.2 Global explainability

As illustrated in (Fig 11a), the SHAP summary plot provides the most informative detailed picture of the importance of the features, along with their performance snippets on the model’s decision. The SHAP values displayed on the X-axis, representing how much each feature contributes to the model’s output, are the main points of the axis. Positive values on the Y-axis indicate a push of the prediction higher, and negative values indicate a lowering of the prediction. The Y-axis portrays the features according to their overall importance from bottom to top. The features at the top have a greater impact on the prediction than those at the bottom. The color gradient depicts the feature value, with blue signifying lower values and red denoting higher values. The feature string of the user-agent impacts the prediction of the model dually, that is, with both positive and negative contributions that vary according to the instance. The prediction is also affected by less impactful, but still important, other features such as city ossiprolo, city potlogi, ASN, and login_day_of_week. Thus, this plot provides a picture of the most influential features and how the respective values skylark the model’s decisions. (Fig 11b) shows an SHAP bar plot where the user-agent string is the feature with the most significant positive impact on the model with a mean SHAP value of approximately 0.035, which indicates that it greatly influences the prediction of normal logins. Sossiprolo, Potlogi, and Petrachioia were the three cities that contributed positively, each with a mean SHAP value of approximately 0.01. The login_day_of_week and ASN features also had small positive impacts, which were reflected in mean SHAP values of 0.005 and 0.005, respectively, implying that they slightly favored normal logins. To summarize, the user agent string was the most significant feature, whereas the city-related and time-based features showed a moderate influence overall.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Global explainability of the developed model is depicted in (a) through the Summary Plot, while (b) presents the Bar Plot showing the overall feature importance.

https://doi.org/10.1371/journal.pone.0349095.g011