2.1 Software defects

Bugs are also known as software defects which are characterized as mistakes or failures in a software program that lead to poor functionality or failure of the program to perform its intended task. These defects may stem from various factors such as errors that may have been coded, errors that may have been in the design of the application or may have been in the requirement specifications. Software defects can be classified into three major types: Nature based software defects, Priority based software defects, and Severity based software defects [16] as shown in Fig 1. Software defects can be classified into three types such as nature, priority, and severity. These include different features and effects of defects within each category, yet all offer valuable insights that help in acquiring essential knowledge on managing defects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Software defects types.

https://doi.org/10.1371/journal.pone.0307112.g001

Nature-based Software Defects: Functional Bugs: Defects causing software malfunctions. Metrics include “Defect Density” and “Requirement Coverage”, Unit-level bugs: Issues related to specific software units. Measured by Unit “Test Coverage” and “Defect Leakage”, Integration Level Bugs: Arise from combining multiple components. Metrics include “Integration Test Coverage” and “Defect Escape Rate”, Usability Defects: Affect user experience. Evaluated through “Usability Testing Scores” and “Task Success Rate”, Performance Defects: Impact efficiency like response time. Metrics include “Response Time” and “Resource Utilization”, Security Defects: Relate to vulnerabilities. Measured by “Vulnerability Count” and “Mean Time to Patch”, Compatibility Defects: Issues with device or software compatibility. Metrics include “Compatibility Test Coverage” and “Compatibility Issue Ratio”, Syntax Errors: Deviations from programming syntax. Assessed through “Static Code Analysis Metrics”, Logic Errors: Flaws in logic. Evaluated by “Code Coverage” and “Defect Discovery Rate”.

Priority-based Software Defects: Low-Priority Defects: Minor impact on operation. Metrics include “Defect Density” and “User Impact Rating”, Medium Priority Defects: Can be addressed later. Measured by “Defect Resolution Time” and “Severity Rating”, High-Priority Defects: Seriously affect operations. Metrics include “Defect Escalation Rate” and “Defect Resolution Efficiency”, Urgent Defects: Must be fixed within 24 hours. Measured by “Urgent Defect Resolution Time” and “Escalation Rate”, Missing Defects: Unmet requirements. Evaluated through “Requirement Coverage” and “Requirement Traceability Matrix”, Wrong Defects: Incorrect implementation of requirements. Metrics include “Requirement Correctness” and “User Feedback”, Regression Defects: Caused by code changes. Evaluated by “Regression Test Coverage” and “Defect Leakage Rate”.

Severity-based Software Defects: Critical Defects: Major impact on functionality. Metrics include “Defect Impact Analysis” and “Time to Resolution”, Major Defects: Significantly affect functionality. Measured by “Defect Density” and “Severity Distribution”, Minor Defects: Minor functional impact. Metrics include “Defect Aging” and “User Impact Rating”, Trivial Defects: No functional impact. Evaluated by “Defect Ratio” and “Defect Discovery Rate”.

2.2 Imbalanced class problem

The imbalanced class problem occurs when the dataset is non-uniformly distributed among the classes. When ML classification algorithms are applied to these datasets, their prediction performance decreases, and models lead to overfitting. Previous research efforts have attempted to address this challenge through various balancing approaches. However, these techniques have shown limited success in significantly improving prediction accuracy [17]. According to [18], the author has used the N-US algorithm to improve SDP accuracy and AUC significantly, outperforming other models on NASA datasets, but the potential information loss from under-sampling. For instance, while multi-objective SDP models coupled with innovative optimization algorithms have been proposed to handle under-sampled datasets, scalability remains a crucial concern [19]. In [20], the authors combined ensemble learning and various ML techniques to enhance the model’s performance, but they did not address the imbalance of data problem.

2.3 Previously developed SDP models using AI techniques

Researchers have proposed various ML-based SDP models to enhance the accuracy and efficiency of defect identification in software development. In [21], authors utilized PCA, DA, LR, and L-N Non-twenty-seven projects to evaluate performance based on predictive validity, quality, and verification cost. Similarly, in [22], the HMOCS-US-SVM model was introduced to address class imbalance and parameter selection challenges in SDP. However, limitations of this model include specific conditions for optimal performance and its efficacy across diverse datasets. In [23], a modified objective cluster analysis (OCA) was proposed for SDP, although with limited generalizability.

Furthermore, [24] introduced an optimization model using a cost-sensitive radial basis function with stacked generalization-based SDP. In a comparative study, [25] demonstrated that SVM outperformed ANN on various NASA datasets. Deep learning techniques were explored in [26], allowing for automatic learning of complex patterns without manual feature extraction. However, [27] noted the computational complexity associated with long short-term memory (LSTM) networks, posing resource challenges for large-scale projects.

Innovative approaches such as gated hierarchical LSTM networks were proposed in [28], though computational complexity remained a concern, particularly for extensive projects. Authors in [29] developed an effective model using AI-based techniques on the NASA MDP repository dataset, achieving promising results with adaptive neuron fuzzy inference system, SVM, and ANN. The FILTER technique proposed in [30] improved SVM-based SDP accuracy, although with sensitivity to hyperparameter tuning and overfitting.

Tree-based bagging ensembles were introduced in [31], but computational expenses were highlighted as a limitation. A framework eliminating the need for separate feature extraction tools was proposed in [32], although concerns about capturing crucial features arose. Convolutional neural networks (CNN) were employed [33] to enhance prediction performance, while SVM was utilized to avoid overfitting problems.

The challenge of predicting defects without labeled modules was addressed in [34] using a genetic algorithm-based LSTM-AST model. Additionally, a deep belief network (DBN)--based semantic features model was proposed in [35], which is affected by computational expenses. Despite advancements, [32] underscored the importance of reading software source code, suggesting a reliance on traditional techniques. In [36], limitations of the PCA-SVM hybridization technique and the slow convergence rate of the multi-verse optimizer algorithm (MOA) were acknowledged. By weighing these strengths and limitations, researchers can make informed decisions when selecting suitable methods for SDP and optimization tasks.

2.4 ML-based techniques

In [37], a NB classifier-based SDP model with normalization and noise reduction was proposed, yielding promising results. However, limitations were observed due to the limited independence assumption of the NB algorithm, which may not accurately capture interdependencies among attributes [38]. In [39], three defect prediction ML models based on the C4.5 algorithm were developed, enhancing prediction accuracy by reducing DT. Despite improvements, the model’s performance was hindered by its slow execution and lack of a method for quantifying case relevance.

Biological and immunological concepts inspired by the artificial immune system were employed in SDP [40], improving defect module detection results. Although the model exhibited better recall measures, it demonstrated low evaluation metrics. In [41], a hybrid technique combining ANN-ABC algorithms was proposed, achieving successful results in defect prediction. However, the model’s limitations included time-consuming training and the requirement for extensive data with multiple layers.

A semi-supervised ML-based Hybrid Self-Organizing Map (H-SOM) approach for automated SDP was introduced in [42], demonstrating good performance even without quality data. Nonetheless, challenges in obtaining accurate data and dealing with incomplete information were noted. In [43], SVM was employed as a RELIEF technique for SDP, enhancing performance metrics. However, the model’s suitability for large datasets and noisy data was questioned.

An ensemble approach using weighted majority voting techniques for SDP was proposed in [44], aiming to improve performance on imbalanced datasets. However, the constraint of identical contributions from each model raised concerns about adaptability to various circumstances. In [45], a hybrid approach combining adaptive dimensional biogeography-based optimization (ADBBO) and radial basis functional neural network (RBFNN) was employed, achieving good performance metrics. However, increasing complexity with the number of neurons was observed as a limitation.

Parameter adjustment in K-NN significantly influenced defect prediction performance in [46], with better results obtained using Dilca distance over Euclidean distance. Nevertheless, challenges persisted in handling large datasets and feature scaling.

2.5 Limitations of previously developed models

Using a high-quality dataset from a large software repository can improve model performance. However, there is still a strong need for a suitable defect prediction method.

4.1 Data description and data pre-processing

The datasets contain software measurements as attributes, along with indications of defects provided by the repository [54]. The Metrics information system is responsible for collecting and validating the data stored in the system. The authors have used the NASA MDP repository dataset to conduct an experimental investigation for SDP. The authors used CM1, PC1, and PC2 datasets that can be divided into test and train sets, and attributes are shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. PROMISE defects prediction attribute aspects.

https://doi.org/10.1371/journal.pone.0307112.t004

4.1.1 Original dataset.

The dataset taken from the NASA MDP repository is CM1, PC1, and PC2. The CM1 dataset is a NASA spacecraft instrument data written in C-programming language. PC1 and PC2 are also NASA metrics datasets, the data from earth-orbit spacecraft flight software written in C-programming language. Initially, this data was in code and converted into the numeric form using Halstead and McCabe feature extractor software metrics. These attributes were recognized in the 70s during features code characterization related to software quality, as shown in Table 5. Fig 2 shows the Defective (“D”) and Non-Defective (“ND”) in graphical form using CM1, PC1 and PC2 datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

(a). Dataset: CM1. (b). Dataset: PC1. (c). Dataset: PC2.

https://doi.org/10.1371/journal.pone.0307112.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Dataset detail division.

https://doi.org/10.1371/journal.pone.0307112.t005

The CM1 dataset contains 1988 observations. Among them, 97.6% are non-defective, and 2.4% are defective. The authors have split it into train and test sets for feeding the developed model (1391 training and 597 tests). In PC1, 97.8% of observations are non-defective, and 2.1% are defective. It contains 705 observations, which have also been divided into two parts (493 training & 212 tests) for the same. Similarly, the PC2 dataset contains 745 elements with 97.8 “ND” and 2.2% “D” It contains 745 observations, which have also been divided into two parts: 521 training & 224 tests.

4.2 SPAM-XAI model

The SPAM-XAI model reduces features, optimizes the model, and reduces its time and space complexity, enhancing its robustness. The working of the model can be illustrated in Fig 3, which shows the SMOTE implementation; the authors have datasets represented by the grey diagram and split data into train and test sets, as indicated by the blue and red charts. The training data is given to the SMOTE, i.e., data from the minority class set A (Rand-N)); the Euclidean distance is calculated using K-NN. The imbalanced data determines the testing rate N; the N models selected randomly from the closest K-NN can generate the set. The set generated by the K-NN is utilized to create a new model that will deal with the varying numbers under 0 and 1and. Finally, the desired result is sent to the next part of the proposed model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. SMOTE representation.

https://doi.org/10.1371/journal.pone.0307112.g003

Furthermore, this section represents the balance data generated by the SMOTE that is now fed to the PCA. Fig 3 shows that the PCA will take the training data generated by the SMOTE. This data is highly dimensional; Its dimensionality can be reduced using the mathematical covariance matrix. The Eigenvalues and Eigenvectors are also calculated. Moreover, Eigen Value Proportion (EVP) is computed for the ratio, and based on EVP, the necessary attributes are selected.

Additionally, this reduced dimensional data with the test data is provided to the MLP classifier, as shown in Fig 4. It will use multiple layers with weights; it consists of the input layer on which the data is given weights and some hidden layers. Every layer feeds to the next layer based on its computational output and continues with all the hidden layers. It uses forward propagation with an activation function (sigmoid) to optimize the values; with several iterations, the output value is compared with the original value. The following process, backpropagation, begins to adjust the weights during learning iteratively, and its main motive is to reduce the error or cost function. Finally, the optimized values are obtained. Lastly, the output layer will generate the “D” and “ND” classifications.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Internal architecture of the SPAM-XAI model.

https://doi.org/10.1371/journal.pone.0307112.g004

Fig 4 illustrates how the model functions and how data flows. The summary demonstrates how robustness is impacted by the unbalanced data (training data) provided to the SMOTE algorithm. PCA reduces the degree of dimensionality of training data (blue arrows indicate this). The input, reduced in dimension through PCA (indicated by the red arrow), is further processed by an MLP for categorization. Once the model’s parameters are fine-tuned, it can make classification predictions. The high-dimensional data was fed into PCA, while the low-dimensional data (PCA output) was used as input for the MLP model to facilitate classification.

4.2.1 Synthetic Minority Over-sampling Technique (SMOTE).

Previously, SDP models encountered challenges due to highly imbalanced datasets. Small classes were difficult for classification algorithms to detect accurately, necessitating balancing imbalanced datasets for precise SDP. Various balancing techniques exist in the SPAM-XAI model. SMOTE creates artificial data samples and adjusts the distribution of classes by oversampling the tiny class without substituting. This phase’s tiny class is oversampled, which involves attributes area activities. K-NN are chosen, and replicas are created based on differences between their NN and feature vectors of the target value. These synthetic samples are then created by inserting the feature vector of interest with a random value between 0 and 1 along a line segment connecting two distinct characteristics.

(1)

The notation “rand (0, 1)” denotes a random number selected uniformly between 0 and 1.

4.2.2 Principal Component Analysis (PCA).

In ML algorithms, the primary task is to find the desired features for the classification problem. Large and small datasets can extract essential features to mitigate overfitting issues by employing feature optimization and dimensionality reduction techniques. In our research, we utilized the PCA technique. Unsupervised ML algorithms, such as PCA and multivariate statistical approaches, aid in reducing the number of variables by selecting the most relevant ones. This technique affects the data by removing fundamental trends from the dataset. It produces a compact set of new multivariate data known as “Principal Components” (PC) by closely tying together strongly related variables. These elements capture a variety of information well. Cross-validation methods like bootstrap and jack-knife are used to assess the efficacy of the PCA model. The following procedures are involved in solving the PC problem mathematically.

• Examine the entire dataset that consists of the N + 1 dimension. Use the following formula (Eq 2) to get the average of each dimension for the whole dataset:

(2)

• After that, use the features approach below to generate the covariance matrix for the full dataset. Using Eq 3, x_A and x_B:

(3)

• Now, find the eigenvectors and corresponding eigenvalues of the covariance matrix. To do this, solve the characteristic equation (Eq 4):

(4)

Following eigenvector and eigenvalue acquisition, diminish dimensionality and craft a feature vector. The eigenvector linked to the highest eigenvalue signifies the dataset’s principal component. Construct a matrix, denoted as M, with dimensions N × I, arranging eigenvectors in a descending order based on eigenvalues. Opt for the I eigenvectors with utmost significance. Project the model into the fresh subspace by applying the N × I eigenvector matrix.

4.2.3 Multilayer Perceptron (MLP).

There are three types of learning: unsupervised, supervised, and semi-supervised. Classification categorizes them based on addressed problems. In SDP, our focus is classification to predict software defects. ML offers diverse classification methods like DT, LR, and RF. These techniques, foundational in AI, tackle classification challenges. Among supervised ML, ANN stands out. CM1, PC1 and PC2 datasets aid model training and validation. Dataset split precedes metadata initialization, encompassing weights, learning rates, hidden layers, minimum error, and epochs. The sigmoid function, depicted in Fig 5, is commonly employed for classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Illustration of MLP.

https://doi.org/10.1371/journal.pone.0307112.g005

It may be expressed mathematically: Vector defines an input layer that may be a distinct characteristic to identify software defects. (5) Where X represents the input features in the input layer, the output layer, Z = [Z₁, Z₂] will represent the projected class. (6) Where B represents the weights. The net input (net_j) to the j^th hidden neuron is (7) Where h_0j is the bias term for the j^th hidden neuron. x_i are the input features. b_ij are the weights from the i^th input features to the j^th hidden unit. The ∑x_i*b_ij represents the weighted sum of the inputs.

The output h_j of the j^th hidden neuron can be represented as: (8) f is the activation function applied to the net_j. (9) Where net input (net_k) to the k^th output neuron. It is the weighted sum of all the outputs from the hidden neurons (h_j) multiplied by their respective weights (v_jk) connecting them to the k^th output neuron, plus the bias term (h_0k) for that output neuron. (10) Where output (y_k) of the k^th output neuron, which is obtained by applying an output activation function (g) to the net input (net_k).

The terms “ND” and “D” can indicate error correction.

(11)

The term calculates the difference between the target output and the predicted output for a given neuron k outputlayerderivative g(net_k) represents the derivative of the activation function g with respect to its input net_k.

Updation of the weight can be done as follows: (12) where: δv_jk is the weight change for the connection between the j^th hidden neuron and the k^th output neuron. β is the learning rate parameter. δl is the error signal at the k^th output neuron, h_j is the output of the j^th hidden neuron.

For k^th neuron (Weight Change for Bias).

(13)

Here δv_0k is the weight change for the bias term associated with the k^th output neuron. The bias term is often considered to have a constant input of 1.

We have sent an error backward (Weight update). (14) Where, v_jk(new) Updated weight from the j^th hidden neuron to the k^th output neuron and v_jk(old) Previous weight from the j^th hidden neuron to the k^th output neuron (Bias Weight Update).

(15)

Here v_0k(new) Updated bias weight for the k^th output neuron. v_0k(old) is the previous bias weight for the k^th output neuron.

4.2.4 LIME (Local Interpretable Model-Agnostic Explanations).

Being an ANN, it provides a relatively recent model that explicitly provides the input’s eXplanation. It is a model-agnostic algorithm that can be utilized twofold with any predictor (i.e. classifier or regressor). Such information allowed territories to model the local environmental situation rather than the general plan of the whole empire and region. The simple meaning of LIME is to approximate a compounded model as a simple interpretable model near the input. The algorithm, in turn, peeks at the sample area of the given input and then serves a descriptive model that fits those samples. A complex framework, on the other hand, can be demonstrated by a simple line. Mentioning [55], LIME consists of a description of information networks. In [56], the authors have extended LIME for image classification, which is called LIME-Image get, integrated with LIME for image data, and in [57], the authors have also extended LIME for text classification, which is LIME-Text gets LIME for text data [58]. These researches have demonstrated that LIME can be a legitimate substitute for complex model explainers and help users understand the model structure. A LIME algorithm is based on estimating how the behaviour of the complex model generally performs using a simple linear model. Given a black-box model f, LIME compares the model’s behaviour by locally approximating it with a linear model g, which is described as: (16)

For x represents an entity, and w is a set of weights. LIME performs the approximation by taking instances from the ‘local neighbourhood’ around x, called the “local neighbourhood sampling”. These instances are then used to train the linear model g, which is learned by solving the following optimization problem: (17) Where f(x) represents the predictions from the complex model, g(x) is a linear function that takes x as input and produces an output. Supposing we define |(|f(x)-g(x)|)| as the difference between the predictions taken from the complex model. The regularization term Ω(g) typically represents a penalty on the complexity of the linear model g, aiming to prevent overfitting and ensure simplicity and interpretability. One common form of regularization is the L2 regularization, which penalizes the squared values of the coefficients of g. Mathematically, it can be expressed as: Where λ is the regularization parameter, controlling the strength of regularization. p is the number of features in the linear model g. w_j represents the coefficient associated with the j^th feature in g. ‖.‖₂ denotes the L2 norm, which computes the Euclidean norm of a vector. The term calculates the squared value of the coefficient w_j, and the summation over j ensures that the regularization term penalizes the overall complexity of the linear model.

The LIME algorithm also assigns different weights to the instances based on their proximity to x. These weights are computed using a kernel function, such as the exponential kernel or the radial basis kernel, represented as: (18)

In which x_i depicts a sample from the neighbourhood, x is a case to be discussed. indicates a distance between x_i and x, and σ stands for a parameter that will determine the width of the kernel. The last stage of LIME consists of assigning meaning to every feature of the linear model g by computing the absolute values of the model weights as the significance measure for each feature. The items fastest moving are.

Below is this flowchart of the SPAM-XAI model in black and white boxes, as shown in Fig 6. The black box uses the ML techniques to predict probable SDP conclusions. Moreover, the XAI is also included, which generates the reasons concerning a specific dataset and attribute influencing the SDP. Besides, the SPAM-XAI model can offer a detailed explanation of the software; thus, the developers, stakeholders, and other project parties can make clear decisions ahead of the development that will reduce the time, resources, and cost of software development.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Overview of SPAM-XAI model complete architecture.

https://doi.org/10.1371/journal.pone.0307112.g006

The black box here refers to our MLP component in Fig 6 above, which uses a neural network architecture for making predictions. The black box pipes in the preprocessed dataset that is obtained through SMOTE and PCA, and the output is that the predicted class label is either “ND” or “D”. Within the NN system is the difficult-to-understand portion that is not visible to the user which is why it has been called “black box”. The white box displays the LIME part of the model employed in the prediction explanation instead of the black box. LIME produces explanations of high interpretability for complex predictions by identifying and making the parameters that most contributed to the output of the black box explicit. According to that, the white box improves the transparency and interpretability of the black box. It provides users with insights on where the predictions were made and about the potential model errors or biases.

# Algorithm (SPAM-XAI)

Start

1. Import necessary libraries              // Import necessary libraries and files

2. Data = read (defect_dataset)                     // Read available data from a file

3. Data = Train Dataset, Testing Datasets. Divided (Ratio)       //dividing the data into two groups:

                                     a training model and a testing model, according to the appropriate ratio

4. SMOTE                     // Eq (1), Dataset balance

5. Set the learning rate to arbitrary, initialize the random weights W, and bias b with any number.

6. Optimized features                     // Decide which characteristics are relevant for the prediction.

7. Describe and input X                     // Eq (5)

8. Compute the deep layer’s net input (net_j)                     // as per Eq (7)

9. Determine the hidden layer’s entire output estimation f(h_input)                     //Eq (8)

                                              // Identification of the activation function is bipolar or binary sigmoidal

10. Calculate the overall input using the result level h_j                                          //Eq (8)

11. The total output of the final level g(net_k)                     // Eq (10)

12. Error calculation δl                            // Eq (11)

13. Error correction & weight Updation                            // Eqs (11–13)

14. Updates to weights and biases       //Eq (12) Minimal weight adjustments to determine the

                                                                                                                                           ideal gap

15. Apply LIME algorithm              // Eqs 16–18) to explain the predictions of the mode

           understand the features that have the most impact on the predictions made by the model l

16. Fine-tune the model              //by adjusting the parameters of SMOTE, PCA, MLP, and LIME

17. Repeat Steps 6-16:                     // Once epochs are finished, recalculate by predicting the outcome

                                                                                                                         using the updated W

18. End

19. Evaluate Performance                     // To assess the mode’s effectiveness.

20.

21. Output // Providing the model’s accuracy as an output

Stop

4.3 Metrics evaluation

This section shows the evaluation criteria for the SPAM-XAI model, which involves various methods, including a confusion matrix, classification accuracy, precision, recall, F-m, and AU-Roc curve.

4.3.2 Confusion matrix.

The confusion matrix is used to define the performance of the SPAM-XAI model as shown in Table 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Demonstration confusion matrix.

https://doi.org/10.1371/journal.pone.0307112.t006

4.3.3 Measures of classification.

It is essentially an expanded form of the confusion matrix. Other metrics than the confusion matrix also aid in achieving a more profound comprehension and studying our model’s functionality.

• Accuracy: There are two types of solutions in a confusion matrix: TP and TN. The authors must practically evaluate the SPAM-XAI model’s accuracy for industrial SDP applications. So, being more accurate can help in better decision-making and reduce the cost of testing and efforts.

• Precision: It measures the defects to the total predicted defects in the SPAM-XAI model. Precision in our model must be used to identify both “D” and “ND” items as defective, regardless of whether the classification was accurate or not.

• Sensitivity/Recall: It measures “D” values, which are all actual defects in the classification. It can detect all the “ND” values in the dataset. Without bothering about how “D” values are incorrectly or correctly differentiated. It can be represented as:

• F1-score: It employs precision and recall. It is an essential evaluation metric because it sums up the model’s predictive performance elegantly. It can vary from 0 to 1. Zero represents the worst possible outcome, and one represents the best. It can be described as:

• AU-Roc Curve: We have used the ROC curve because it gives an appropriate angle when the dataset is balanced. The coordinate of the ROC curve contains two variables; one is the True Positive Rate (TPR), which is the proportion of the truly predicted value that was correctly predicted for all true values. Another variable is the False Positive Rate (FPR), the false predicted value proportion to the total false values.

TPR=(TP/(TP+FN))

FPR=(FP/(TN+FP))

The curve is shown as a function of two variables: Actual Positive Rate (TPR) and Type I Error/Predicted Positive Rate (FPR)

5.1 Dataset CM1

The authors have worked on the CM1 dataset, driven from the NASA dataset repository. The confusion matrix of the SPAM-XAI model is shown in Table 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. SPAM-XAI confusion matrix.

https://doi.org/10.1371/journal.pone.0307112.t007

The SPAM-XAI model is done using the CM1 dataset, which is illustrated in Table 7. The testing dataset consisted of 597 sample observations. The model correctly classified “D” instances 571 times (True Positives) and included 10 as “ND” (False Negative). Also, it found that “ND” data was labeled as “D” 16 times more often (False Positive), whereas “ND” was labeled as “D” 0 times (True Negative). It illustrates the high true positive and true negative rates and good performance of the model.

Moreover, the ROC curve contains the model’s efficiency evaluation. The depiction is a curve plotted FPR versus TPR, which measures the model’s trade-off between sensitivity and specificity. An area under the curve (AUC) value of 0.91 was obtained. Such a high value suggests excellent classification accuracy. Altogether, we conclude that the model displayed the optimal AUC value of 0.91, which successfully confirms the ability to differentiate between positive and negative class labels, as shown in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Analysis of CM1 ROC curve.

https://doi.org/10.1371/journal.pone.0307112.g007

5.2 PC1 dataset

The authors have worked on the PC1 dataset, driven from the NASA dataset repository. The confusion matrix of the SPAM-XAI model is shown in Table 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. SPAM-XAI confusion matrix using PC1 dataset.

https://doi.org/10.1371/journal.pone.0307112.t008

The SPAM-XAI model is done using the PC1 dataset, which is illustrated in Table 8. The testing dataset consisted of 224 sample observations. The model correctly classified “D” instances 215 times (True Positives) and included 2 “D” instances as “ND” (False Negative). Also, it found that “ND” data was labeled as “D” 7 times more often (False Positive), whereas “ND” was labeled as “D” 0 times (True Negative). It illustrates the high true positive and true negative rates and good performance of the model.

Furthermore, the ROC curve contains the model’s efficiency evaluation. The depiction is a curve plotted FPR versus TPR, which measures the model’s trade-off between sensitivity and specificity. An AUC value of 0.79 was obtained. Such a high value suggests excellent classification accuracy. Altogether, we conclude that the model displayed the optimal AUC value of 0.79, which successfully testifies to the ability to differentiate between positive and negative class labels, as mentioned in Fig 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Analysis PC1 AU-ROC curve.

https://doi.org/10.1371/journal.pone.0307112.g008

The authors have evaluated the SPAM-XAI model on various matrices that show the performance of CM1, PC1, and PC2 datasets.

5.3 PC2 dataset

The authors researched using the PC2 dataset derived from the NASA dataset repository. The confusion matrix for the SPAM-XAI model is presented in Table 9 of their work.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. SPAM-XAI confusion matrix using PC2 dataset.

https://doi.org/10.1371/journal.pone.0307112.t009

The test dataset with 212 observations utilized comprising the PC2 dataset is shown in Table 9 below. The SPAM-XAI model was proven right in 186 cases (True Positives). While in 13 cases (False Negatives), it identified the “D” data as “ND”. Furthermore, the algorithm identified False Positives (“ND” data as “D”) 9 times and True Negatives (“D” declared “ND” data) 4 times. This distribution communicates a strong model performance as it maintains equally good True Positive and True Negative values. Besides that, the ROC curve is an additional evaluation measure of our model, showing our approach’s diagnostic ability. The graph concerns the relationship between FPR and TPR and goes along the curve from 0 to 1. Using the 0.5 reference line, we can assess performance. The ROC curve’s shape tells the model’s sensitivity and specificity levels, and the point with the highest specificity is attentive when the curve approaches the y-axis with 0.2 or lower. The AUC value of 0.59 suggests that the classification ability is satisfactory, greater than 0.5, and lower than 1. To sum up, the results of experiments demonstrated that the model significantly outperforms the PC2; the AUC is close to 0.60, as Fig 9 shows.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Analysis PC2 AU-ROC curve.

https://doi.org/10.1371/journal.pone.0307112.g009

Table 10 presents the performance metrics for defect prediction models evaluated on three datasets: CM1, PC1, and PC2, covering Precision, Recall, F-Measure, AU-ROC, cross-validation Accuracy, and validation Accuracy. For CM1, the model achieved a Precision of 95.00%, recall of 96.00%, F-Measure of 95.00%, and AU-ROC of 91.00%, with cross-validation and validation Accuracies of 98.20% and 97.91%, respectively, indicating high consistency and overall performance. The PC1 dataset showed a Precision of 97.01%, recall of 99.00%, F-Measure of 98.00%, and a lower AU-ROC of 79.00%, with cross-validation Accuracy at 97.51% and validation Accuracy at 92.10%, still maintaining high performance despite the drop in discrimination ability. For PC2, the model achieved a Precision of 94.00%, recall of 97.01%, F-Measure of 95.10%, and a notably lower AU-ROC of 59.00%. However, it excelled in cross-validation and validation, with an accuracy of 99.52% and 98.65%, respectively, highlighting challenges in distinguishing defect instances while maintaining high accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Statistical model evaluation for CM1, PC1 & PC2 datasets using SPAM-XAI model.

https://doi.org/10.1371/journal.pone.0307112.t010

6.1 SPAM-XAI model experimental comparison with baseline models

This section represents the rigorous experimental comparison, and the SPAM-XAI model is pitted against various baseline models, as shown in Table 11.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Experimental evaluation of various baseline models with the SPAM-XAI model.

https://doi.org/10.1371/journal.pone.0307112.t011

Fig 10A and 10B shows the graphical representation of the baseline model comparison using CM1 and PC1 datasets for experimental verification of the SPAM-XAI model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10.

(a). Baseline model Comparative Analysis with CM1 dataset. (b). Baseline model Comparative Analysis with PC1 dataset.

https://doi.org/10.1371/journal.pone.0307112.g010

6.2 SPAM-XAI model comparison with prior State-of-the-art

The estimation of the performance of the proposed model can be validated by comparing the results, and we have shown the comparative analysis of the proposed model with the previously developed one in Table 12.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 12. Analysis of previously developed models with SPAM-XAI model.

https://doi.org/10.1371/journal.pone.0307112.t012

Table 12 shows the discussion by comparing various models on CM1 and PC1 datasets. Table 12 shows the discussion by comparing various models on CM1 and PC1 datasets. The SPAM-XAI model is outperforms other models in SDP because of its holistic approach. Thus, the model reduces generalisation and rich prediction errors due to factors such as class imbalance and high-dimensional feature vectors. It also uses advanced XAI methods for model interpretability to avoid ambiguous decisions or lack of trust in the model’s predictions. Carefully fine-tuned hyperparameters further enhance its effectiveness and decrease its computational cost. However, there may be some previously developed models that may possess some evaluation metrics that outperform the SPAM-XAI model due to some domain-specific features, appropriate algorithm selection, data, and evaluation biases. These models may perform well in certain aspects depending on their comparative strengths in terms of alignment with certain characteristics of a given dataset or on the selection of some evaluation metrics as opposed to others and showcasing the importance of considering multiple factors in model evaluation and selection.

6.3 A combined comparison analysis using the CM1 and PC1 datasets with the previously created model

This section represents the graphical implementation of the combined comparative study of previously developed models. The illustrations of Fig 11 are shown below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11.

(a). Precision CM1 and PC1 dataset. (b). Recall CM1 and PC1 dataset. (c). F-Measure CM1 and PC1 dataset. (d). Accuracy CM1and PC1 dataset. (e). AU-ROC-area CM1 and PC1 dataset.

https://doi.org/10.1371/journal.pone.0307112.g011

The graph shown in Fig 11A illustrates that our model precision score (95.00% with CM1 and 97.01% with PC1) performs better than the others. The Recall value graph can be represented using Fig 11B. It includes the proposed model recall value (red and blue horizontal line) (CM1: 96.00 and PC1: 99.00) percentage, which is higher than the other models.

The graph in Fig 11C illustrates that the SPAM-XAI model F-measure score (95.00%with CM1 and 98.00%with PC1) performs better than the others. The accuracy value graph can be represented using Fig 11D, which shows that the proposed model’s accuracy value (CM1: 97.00% and PC1: 96.00%) percentage is higher than that of the other models.

Furthermore, the AU-ROC curve is a significant criterion for validating a model. The AU-ROC curve, as shown in Fig 11E (91.0% with CM1 and 79.00% with PC1), is higher than the previously developed one.

6.4 Time complexity and computational cost

In this section, we will discuss the time features of the SPAM-XAI model. We will focus on the execution time, which encompasses three main stages: the processing time for the SMOTE phase for data balancing, the MLP training stage, the LIME algorithm implementation, and any other possible stage of processing. This may include variety of operations such as manipulation of the data, transformations or computations. Furthermore, we will discuss the testing time, which directly relates to the time used for inferences or predictions. With this timing analysis, we will seek to bring insights into how the SPAM-XAI model is efficient in time, revealing useful information about how time is budgeted for each crucial stage of its use.

Where the SDA phase time involves Stacked Denoising Autoencoder (SDA), Ensemble Learning (EL) is the training stage.

SPAM-XAI model is created with the intention of being useful for making predictions while also being as computationally efficient as possible. We show the runtime complexity of the model, training time, comparison of time cost with other state-of-the-art models [2,61]. The SPAM-XAI model further highlight the focus on performance and computational efficiency. This comprises of the discussion regarding the temporal aspects of the model such as the time complexity of the model, the total duration required to train the model and lastly a comparison between the temporal costs in terms of other models are discussed in Table 13.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 13. Comparison of total training duration (in seconds) of previously developed models with SPAM-XAI.

https://doi.org/10.1371/journal.pone.0307112.t013

Table 13 below shows the performance of the SPAM-XAI model where it can be noted that during training SPAM-XAI consistently outperforms other state-of-the-art models in terms of training time. Overall the SPAM-XAI model is most efficient in computational performance compared to others. This lower time cost can be ascribed to the optimal feature selection using the MLP and the data balancing using SMOTE which were both model-efficient. This explains the significance of SPAM-XAI in time costs and it is therefore recommended as a novel approach for situations where computational effectiveness is high on priorities.

7.1 CM1 dataset

The SPAM-XAI model interprets the ML model’s outcomes as an application and a technique. Among those things is identifying the critical features that affect the probability of issues and handling how attribute values impact the prognosis. The SPAM-XAI model aims to provide a platform for software developers and engineers to give them guidelines on dealing with the source of the mess. This is achieved by providing transparency and unambiguous insight into the dataset. Fig 12 depicts the importance of sub-class attributes in SDP. For instance, the prediction production of SPAM-XAI on CM1 dataset output is shown in Fig 12. The result indicates prediction probabilities for the given sample: 0% for Irreparable and 100% for No Repairs at All. The model guarantees that the instance is defect-free with certainty at this high condensability. The subsequent section briefly details the feature set-up and corresponding values, which is critical in making the correct prediction. Among all the features, CALL_PAIRS has a power of -1.16, which has the highest correlation with the prediction, and the power weight is only 0.33. One of the most distinguishing features is LOC_BLANK, which has a value of -3.94 and is the importance of 0.7, indicating its effect. Varying other metrics, for example, the NO_OF_BRANCHES, DECISION_DEPTH, DECISION_COUNT, LOC_CODE_BRANCH, COUINIT_METRIC, CYCLOMATIC_DENSITY, INCIDENT_COUNT, and FACADE_LATERAL_SIZE play their role but to a much lesser extent the product provides the features’ values and their weights that were used to generate the prediction ultimately. In this case, the lack of any defect occurs because the model has analysed all the attributes involved and seen how each attribute contributes to the result.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. SPAM-XAI using the CM1 dataset.

https://doi.org/10.1371/journal.pone.0307112.g012

7.2 PC1 dataset

The SPAM-XAI model is shown to be valuable in the exploration of the ML model trained with the PC1 dataset. SPAM-XAI model predictions via locally fitting a linear multivariate regression counterpart. In addition, the SPAM-XAI model could enhance the examination of the relevance of particular features with the help of the feature importance scores. The model acquires better interpretability through this enhancement and cultivates the user’s confidence and faith, as shown in Fig 13.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. SPAM-XAI using the PC1 dataset.

https://doi.org/10.1371/journal.pone.0307112.g013

The predicted model of the sample (Fig 13) shows that the sample was an “ND” module with a probability of occurrence of 0.99, and a “D” module had an occurrence probability of only 0.01. The main component of the metric is “HALSTEAD_CONTENT” with 15.83, along with “CYCLOMATIC_DENSITY” and “PARAMETER_COUNT” having 0.20 and 0.00, respectively. These are the impacts of each feature on the prediction’s result, which can be seen by using the SPAM-XAI model as the explainability model of the algorithm. We would imitate the local model behaviour guiding us around the predicted instance and will provide a humanly interpretable explanation for the model predictions.