3.1 Research question

In this work, we will investigate three research questions (i.e., RQ1 to RQ3). These research questions aim to investigate the impact of SNA metrics on TDP models. In RQ1, we will examine whether the combined metric suit (CMS) can enhance the performance of the existing TDP model. Following an assessment of the effectiveness of SNA metrics in improving the TDP model, we will re-evaluate the performance of each classifier in the TDPSN models in RQ2 to identify the most suitable classifier for the TDPSN model. In the final stage of RQ3 and RQ4, we will utilize the best classifier obtained from RQ2 to investigate the impact of different combinations of metrics (TD-related and SNA-related metrics) on the performance of the TDPSN models.

3.2 Subject projects

We conduct our experiments on the 25 same Java projects as Tsoukalas et al. [19]. These Java projects cover a wide range of diverse fields and are representative. For example, Arduino represents the Internet of Things and embedded systems, while we have chosen Pdfbox and Libgdx for the PDF and game development fields respectively. The diversity of the projects can enhance the stability and generalizability of our research results. Table 2 details the 25 projects, including their names, descriptions, version information, and code line counts (Loc). The code line counts range from 7K to 482K, reflecting the varied project and code sizes we examined.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Selected projects.

https://doi.org/10.1371/journal.pone.0323672.t002

3.3 Metric suites

In this section, we briefly outlined the TD metrics and SNA metrics used in our study, as well as several combined metrics derived from them.

3.3.1 TD metrics.

We use the TD (TD-related) metrics by Tsoukalas et al. [19]. We refer to their classification of the TD index in subsequent work [34] and categorize the 18 TD metrics used in our study into 6 categories, as shown in Table 3. Different metric categories represent different information about software projects. Size metrics measure the size and complexity of a project, for example, Total Methods and Total Variables can represent the number of methods and variables in a codebase. Evolution metrics can provide information about the evolution of a codebase by analyzing its commit history and development activities. Duplication metrics assess the reusability and maintainability of a codebase by measuring the proportion of duplicate code lines in it. Complexity metrics provide information about the complexity of a codebase by measuring its structural and design complexity. Coupling metrics assess the coupling degree of a codebase by analyzing the dependency relationships between modules in it. Cohesion metrics measure the cohesion of a codebase by assessing the functional relatedness and consistency of modules within it. The SM (Simple Combined) metrics in the Combined Metrics category are a set of metrics that were simply combined from the Duplication Metrics, Complexity Metrics, Coupling Metrics and Cohesion Metrics categories to meet the research requirements of RQ3 in Section 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. TD metrics used in our study.

https://doi.org/10.1371/journal.pone.0323672.t003

3.3.2 SNA metrics.

Social network analysis (SNA) is a method used to investigate interpersonal connections and information dissemination. It has been widely applied in the field of software engineering as well. SNA simplifies the relationships between software modules into a network model, with nodes representing elements within the software and edges denoting their relationships, including data and call connections. By conducting social network analysis on this relationship network, various SNA metrics can be derived, which can be further categorized into EN (ego network) metrics and GN (Global network) metrics. The complete set of 64 SNA metrics along with their combined metric suit (CMS), including TD metrics, are presented in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. SNA metrics used in our study.

https://doi.org/10.1371/journal.pone.0323672.t004

Ego network (EN) metrics: EN is a network that focuses on examining a specific node and its connections and relationships with other nodes [17]. Within the self-network, “in," “out," and “un" represent the different types and directions of connections between a node and its neighboring nodes. For instance, “in" reflects the benefits and influence of the node itself within the network, such as contributions or support from other nodes to the focal node. On the other hand, “out" represents the contribution or influence of the focal node to its neighbor nodes, indicating its support or influence on other nodes. The term “un" refers to the bidirectional connection between self-nodes and neighboring nodes, without a clear direction. This indicates that there is mutual interaction and relationship between self-nodes and neighboring nodes, and the flow of information and resources between them is bidirectional. In our study, we utilized all three types of self-networks.

Global network (GN) metrics: GN refers to the structure and characteristics of the entire network system. Global network metrics are used to describe and analyze the properties and characteristics of the entire network, such as overall scale, average path length, network density, and clustering coefficient. These metrics can help researchers understand the overall structure of the entire network, the importance of central nodes, the efficiency of information diffusion, and the structures that exist in the network.

Finally, we combined the 18 TD metrics with 64 SNA metrics to obtain a combined metric suite (CMS) containing 82 metrics.

3.4 Data pre-processing

We utilized the dataset employed by Tsoukalas et al. [19] in their study as the foundation for our research. This dataset consists of a table with 18,857 rows (representing software modules) and 19 columns. The first 18 columns each correspond to a software quality metric related to TD, as outlined in Table 3. At the end of the table are the Max-Ruler values (A definition by Amanatidis et al. [36], i.e., whether the module is a high-TD module). "1" and "0" are used to distinguish whether a module is a high-TD module, "1" represents a high-TD module, and "0" represents a no high-TD module. In subsequent research, we will refer to this dataset as the TD dataset.

In this work, the 40 EN metrics and 24 GN metrics for each module are extracted from the Class Dependency Network (CDN) [11–15] of 25 Java programs using the UCINET tool [20]. The specific metrics are presented in Table 4. CDN is a directed network (CDN = (V, E)), where each node represents a class (or interface) c in the program, and the edge set E represents the class dependencies. Each edge , (, ) indicates the dependency relationship from class to class .

There are a total of 9 types of dependencies between classes [11–15]:

• Local variable (LV): Class i contains a variable of class j.
• Global variable (GV): Class i contains a field of class j.
• Inheritance (INH): Class i inherits class j, adding new functionality through the inheritance relationship indicated by the ‘extends’ keyword.
• Interface implementation (II): Class i implements the functionality of interface j.
• Parameter type (PT): Class i contains at least one method with a parameter of class j.
• Return type (RT): Class i contains a method with a return type of class j.
• Instantiates (INS): Class i instantiates objects of class j.
• Access (AC): Class i has at least one method accessing fields of class j.
• Method call (MC): Class i has at least one method calling a method on an object of class j.

Fig 1a presents a code snippet to illustrate the CDN more clearly, and Fig 1b depicts the CDN based on the modified code snippet. Similar to previous studies, the nodes in CDN represent classes, and the edges represent dependencies between classes. However, in our case, we further consider three additional types of dependencies: “instantiates,” “access” and “method call.” We also differentiate the “aggregation” relationship into “local variable” and “global variable.” As shown in Fig 1b, the code snippet in Fig 1a demonstrates various dependencies between classes: global variables (BD, EB), inheritance (DC), interface implementation (DI), parameter type (CB), instantiation (AB, AC, DC), and access (DE).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. An illustrative Java code snippet and its Class Dependency Network (CDN).

( a) A code snippet to illustrate the CDN more clearly. ( b) The CDN based on the modified code snippet.

https://doi.org/10.1371/journal.pone.0323672.g001

Then, we combined the SNA dataset with the TD dataset to obtain a final dataset containing 18,857 rows and 83 columns. Following the suggestion of Tsoukalas et al. [19], we conducted missing value treatment on the dataset and removed modules containing missing values. This operation resulted in a new dataset consisting of 17,646 modules. As the independent variables in the dataset exhibited non-normal distribution and extreme values, we utilized Local Outlier Factor (LOF) [37] to enhance model performance. Post outlier removal, the dataset contained 13,515 modules, with 4,131 modules identified as outliers and subsequently removed. However, experimental findings indicated that this process did not enhance model performance; instead, it resulted in a decrement. This outcome may be attributed to the excessive removal of modules (23% of the total) compared to the approach by Tsoukalas et al. [19], leading to reduced data dimensionality and information loss. Given the inherent disparities between the datasets and metrics, we opted to solely eliminate modules with outliers in the TD dataset. This procedure led to the removal of 739 modules, resulting in a final dataset of 16,907 rows and 83 columns. For future reference, we denote this dataset as the CMS (combined metric suit) dataset.

3.5 Model building

3.6 Performance evaluation metrics

To evaluate the performance of TDP models, we used four different performance metrics: precision, recall, -measure, and module detection (MI) ratio. In addition to precision and recall, which are common performance metrics in the prediction field, -measure and MI ratio are considered the two most important performance metrics for evaluating TD prediction models in other similar studies (e.g., Ref. [19]).

-measure: In practical TD prediction, accurate identification of minority classes (high-TD modules) is paramount, as missing any high-TD module may lead to substantial economic losses and systemic risks [19]. According to the recommendations of Tsoukalas et al. [19], the selection of models prioritized high-recall performance while maintaining reasonable precision. Traditional evaluation metrics often fail to balance this requirement effectively, whereas the measure addresses this limitation by harmonizing precision and recall. By setting the relative importance coefficient , more emphasis was placed on recall to ensure that the model detects all high-TD modules with maximum precision. For example, misclassifying low-TD modules as high-TD (false positives/ ) primarily causes resource waste (e.g., unnecessary refactoring) without direct system failure consequences. In contrast, misclassifying high-TD modules as low-TD (false negatives/ ) leads teams to overlook critical risks, potentially triggering exponential growth in maintenance costs. This demonstrates the asymmetric risk between and [19]. To evaluate the prediction capacity of minority classifiers, the measure was adopted as the primary performance metric, defined as:

(1)

(2)

(3)

Module inspection (MI) ratio: The MI ratio is defined as the ratio of modules predicted to be high TD to the total number of modules [41]. It refers to the percentage of modules that developers must check to find the number of high TD modules that the model can correctly identify, which is the recall efficiency. For example, for a model with a recall rate of 80% and an MI ratio of 10%, it means that we need to manually check 10% of the total number of modules (in other words, only check the modules that the model predicts to be high TD) to find 80% of the true high TD modules (). Such a model is more cost-effective than randomly checking modules. It is defined as follows:

(4)

Therefore, the combination of -measure and MI ratio paints a perfect picture for TD prediction. They cover the accuracy and practicality of the model, which means that the model can efficiently and accurately detect all high-TD modules to the greatest extent possible.‘

To evaluate the enhancement effect of Social Network Analysis (SNA) metrics on technical debt prediction, we apply the Wilcoxon signed-rank test [42]. We independently analyze 30 paired observations derived from 310-fold cross-validation. Moreover, we correct the obtained -values from the Wilcoxon-signed rank test with Bonferroni correction [43] to control for false positives. We do so to statistically quantify the number of datasets on which models trained on other metric families outperform models trained on TD metrics with a statistical significance (-values < 0.05).

4.1 Can SNA metrics improve the performance of TDP models?

Table 5 presents a collection of performance metrics for the classifiers under evaluation, which underwent validation and testing using the training-validation-testing approach outlined in Section 3.5. Specifically, the “validation" column displays the outcomes achieved by each classifier during repeated hierarchical cross-validation (the validation stage). These results reflect the overall metrics, averaged over 3 repetitions of performance indicators with k = 10, and include corresponding standard deviation values. In contrast, the “test” column illustrates the performance metrics achieved by each evaluated classifier on a separate set (i.e., data that was not utilized during training/validation and represents 20% of the total data). For model performance evaluation, our main focus is on the -measure and the MI ratio mentioned in Section 3.6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Evaluation results for all classifiers.

https://doi.org/10.1371/journal.pone.0323672.t005

The best performance for each classifier metric is shown in bold, and we also report the standard deviation.

The results in Table 5 demonstrate a significant improvement in the -measure and the MI ratio of the CMS dataset after introducing SNA metrics. Specifically, among the 7 classifiers in the CMS dataset, 5 classifiers outperformed those in the TD dataset during the testing phase in terms of score. These classifiers are RF, XGB, LR, SVM, and DT. Especially the XGB classifier, which saw an increase in score from 0.725 to 0.749 during the validation stage, surpassing other classifiers. Furthermore, it is worth noting that both KNN and NB have experienced some decline. However, it is important to highlight that in similar studies, KNN and NB are considered to be underperforming classifiers [19]. The results during the validation stage were obtained on a dataset that the model had never seen before, indicating that the TDP model has improved generalization ability after introducing the SNA metrics. Additionally, for the training stage, apart from RF and DT classifiers showing a slight decrease in score, all other classifiers demonstrated some performance improvement in both training and validation. This suggests that after introducing the SNA metrics, the model may have gained additional useful information to better fit the data and thus improve its performance.

On the other hand, in terms of the MI ratio, 6 out of 7 classifiers (except TD) achieved better performance on the test set (refer to the description of the MI ratio in Section 3.6, the smaller the MI ratio, the better the model performance). This indicates that SNA metrics can effectively improve the detection ratio of the model, which means that developers can increase cost-effectiveness and reduce economic losses.

For completeness, the results for precision and recall are also provided. As can be seen from Table 5, the recall on the test set for most classifiers in the CMS dataset, except for XGB, decreased compared to the TD dataset. In terms of precision, better performance on the test set was achieved by RF, XGB and DT classifiers. However, as mentioned in Section 3.6, what needs to be done is to consider both the recall and precision of the model. A decrease in recall or precision alone cannot be the standard for judging the good or bad of the model. The excellent performance of the classifier in the -measure indicates that after introducing SNA metrics, a better balance between recall and precision can be achieved by each classifier, which is what is being pursued.

Due to our primary focus on -measure and the MI ratio, the introduction of the new metric (SNA metrics) has significantly enhanced the overall model performance in both training and testing phases, thereby improving its generalization capability and predictive accuracy.

Of course, this is the result obtained by using the same classifier parameters as Tsoukalas et al. [19]. In order to more comprehensively evaluate the impact of introducing SNA metrics on the performance of each classifier, Grid-search [40] was used to fine-tune the hyperparameters of the CMS dataset, and the experiment was conducted again. Table 6 reports the fine-tuning results of the classifier parameters after hyperparameter tuning was performed. Table 7 presents the performance of each classifier on the CMS dataset after hyperparameter tuning. It is evident that the performance of LR, SVM, KNN, and NB remained unchanged after hyperparameter tuning due to their parameters remaining consistent before and after tuning. Upon closer examination of classifiers with adjustments in other parameters, it is apparent that the performance of two classifiers (XGB and DT) improved, while the RF classifier exhibited no significant change. Specifically, the performance of the XGB classifier on the test set increased from 0.749 prior to hyperparameter tuning to 0.772, indicating an enhancement in prediction performance as a result of hyperparameter tuning. Conversely, for the RF classifier, there was minimal change as its performance decreased from 0.766 before tuning to 0.764 post-tuning. However, upon further examination of other performance metrics, the recall of RF showed improvement (from 0.828 to 0.867), albeit at the expense of a decrease in precision and model detection ratio. Since is utilized as the objective function for hyperparameter tuning, it is expected that the tuned model will give priority to recall. The decision of whether to trade off a certain degree of precision and model detection efficiency for higher recall is contingent upon the company’s discretion. Some companies may be willing to incur substantial costs in order to detect as many high-TD modules as possible, thereby mitigating potential economic losses in the future. Similar trade-offs were observed with DT and XGB classifiers, which also sacrificed some precision and model detection ratio post-tuning in exchange for improved metrics and recall. Overall, both before and after hyperparameter tuning, the combined metric suit (CMS) demonstrated enhanced model performance across seven categories compared to the singular use of TD - related metrics in most classifiers. KNN and NB classifiers were exceptions, which highlights the efficacy of SNA metrics within the TDP model. Furthermore, our research results confirmed the underperformance of KNN and NB classifiers—a sentiment echoed by previous studies (e.g., Ref. [19]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Classification models of the experimental setup.

https://doi.org/10.1371/journal.pone.0323672.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Evaluation results for all classifiers after hyper-parameter tuning.

https://doi.org/10.1371/journal.pone.0323672.t007

The results for the other three classifiers indicate that, except for the RF classifier, the performance of the other two classifiers (XGB and DT) has improved in terms of score. In particular, the XGB classifier’s performance during testing has increased from 0.749 before parameter tuning to 0.772, indicating that parameter tuning has enhanced the predictive performance of the XGB model. As for the RF classifier, its performance decreased from 0.766 before tuning to 0.764, with almost no change; however, when considering other performance measures, RF’s recall rate has improved (from 0.828 to 0.867), at the expense of precision and MI ratio decreasing. Since was used as the objective function for parameter tuning, it can be foreseen that the tuned model will have a greater inclination towards improving the recall rate. Whether sacrificing a certain level of precision and model detection efficiency for higher recall rate depends on company decisions; some companies may prefer to invest significant costs in detecting as many high TD modules as possible to mitigate potential future economic losses. For both DT and XGB classifiers alike, they have sacrificed a certain level of precision and model MI ratio after tuning in exchange for improvements in measures and recall rates.

Among the five learning algorithms that demonstrated enhanced performance, Wilcoxon analysis revealed -values below 0.05 for all models except the Random Forest approach.This discrepancy may arise from RF’s built-in feature selection via out-of-bag error estimation, which could reduce the marginal contribution of SNA metrics relative to existing baseline features. The null hypothesis was consistently rejected across experimental evaluations (<0.05), confirming measurable performance disparities between different dataset partitions. This outcome corroborates our hypothesis about feature compatibility exerting measurable influence on predictive effectiveness, indicating that technical debt models achieve optimal performance when test data characteristics align with their training distributions.

However, comprehensive model evaluation requires interpretation through multiple metrics. We have consequently constructed precision-recall (PR) curves, which are more appropriate than other curve types for imbalanced datasets like ours (with a high-TD to non-high-TD module ratio of 1:15). Each classifier’s Area Under the Curve (AUC) value quantifies its threshold-agnostic performance, where higher AUC indicates better precision-recall balance, validating optimization effectiveness. Fig 2 and 3 display AUC values for models trained on the TD and CMS datasets respectively. The AUC results corroborate our experimental findings, showing improvements for all classifiers except KNN and NB after incorporating SNA metrics, particularly notable for DT’s AUC increase from 0.38 to 0.59.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Cross-validated precision-recall curves (TD).

The Precision-Recall curve for the TD dataset shows the tradeoff between precision (y-axis) and recall (x-axis) across probability thresholds, derived from 10-fold cross-validation.

https://doi.org/10.1371/journal.pone.0323672.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cross-validated precision-recall curves (CMS).

The Precision-Recall curve for the CMS dataset shows the tradeoff between precision (y-axis) and recall (x-axis) across probability thresholds, derived from 10-fold cross-validation.

https://doi.org/10.1371/journal.pone.0323672.g003

These results demonstrate that CMS metrics (combining SNA and TD indicators) generally outperform standalone TD metrics across classifiers. This enhancement likely stems from SNA metrics supplementing critical system information or synergistically capturing broader software characteristics with TD metrics, analogous to how SNA complements code metrics in software defect prediction (SDP). To verify our findings and assess feature set informativeness, we implemented the PCA method from Long et al. [22], measuring how many principal components (PCs) each metric set requires to capture 95% data variance – fewer PCs indicate higher information density.

Fig 4 illustrates cumulative variance curves, where the x-axis represents PC counts and y-axis shows explained variance. Key observations: TD metrics (green curve) need 13 PCs, indicating lower information density; SNA metrics (blue curve) require 33 PCs, demonstrating higher information content; CMS combined metrics (red curve) use 43 PCs – less than the additive total (46 PCs for 13 + 33), confirming metric complementarity. The CMS curve’s rightward shift reveals superior variance explanation per PC count.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. PCA cumulative variance plot.

The PCA cumulative variance plot is a graphical representation that illustrates the accumulated explained variance ratio (y-axis) versus the number of principal components (x-axis).

https://doi.org/10.1371/journal.pone.0323672.g004

This analysis concludes that TD-SNA integration achieves more efficient dimensionality reduction through complementary information synergy. Their combination creates richer feature representations, establishing merged TD/SNA metrics as a superior strategy for technical debt prediction models. Future research should prioritize such metric fusion approaches.

4.2 Which classifier performs best on TDPSN models?

As shown in Table 7, in terms of the validation phase, LR achieved the highest score of 0.767, followed by XGB and RF classifiers. On the other hand, the XGB classifier obtained the highest score of 0.772 in the testing phase. Following closely behind is RF with a score of 0.768 (the highest scores before and after tuning were selected). Then there are LR, SVM, and DT with scores ranging from 0.734 to 0.767. It is also noted that compared to the results in the validation phase, the performance of classifiers has even improved. To avoid the inflation problem caused by comparing multiple classifiers at the same time, we used the SK (Scott Knott) [44] algorithm to perform statistical hypothesis testing. Compared with other hypothesis testing methods, the SK algorithm generates clusters of classifiers with similar performance and sorts the clusters of classifiers according to performance. We input the scores of each classifier in the 3x10 cross-validation process, and finally the SK algorithm generated four clusters of classifiers with similar performance. The performance of the classifiers is ranked from best “A” to worst “D.” We report the results of the SK algorithm in the last column of Table 7. As we can see, RF, XGB, LR, and SVM are classified into the best classifier cluster, which means that they have similar prediction performance, which is also consistent with our previous experimental results. In addition, it further proves that KNN and NB can be considered the worst choices.

Although the measure is considered one of the primary performance metrics, it is important to consider other metrics in order to make the best choice. As shown in Table 7, DT, LR, and SVM exhibit the highest recall during the validation phase, with scores of 0.903, 0.901, and 0.892 respectively. Following closely are XGB and RF, while KNN and NB perform poorly in terms of recall and rank last. This trend generally holds true in the test phase as well; LR and SVM show improved recall compared to the validation phase, both reaching a score of 0.906. However, when it comes to precision, only RF and XGB classifiers have precision scores exceeding 0.5 during both validation and testing phases; all other classifiers fall below this threshold at between 0.3 to 0.5 range which may significantly increase the number of predicted false positives () since a precision score below 0.5 will result in more than true positives () [19]. Considering the results from Table 6 before and after parameter tuning reveals that only three classifiers (RF, XGB, and DT) have precisions exceeding 0.5. While we prioritize recall over precision, this does not mean that we can disregard precision altogether; rather we aim for a model with moderate precision along with high recall rate.

To supplement the comparison of the above classifier performance metrics,Precision-Recall (precision-recall) curves are plotted, precision-recall curves are a better choice for unbalanced datasets (the ratio of high-TD modules to non-high-TD modules in the dataset we use is about 1:15). We also provide the area under the curve (AUC) value for each classifier, summarizing their performance at different thresholds. The higher the AUC value, the higher the recall and precision. As we can see from Fig 3, the curves of RF, XGB, LR, and SVM classifiers are closer to the top right corner, almost reaching the point of coincidence. This means that compared to other classifiers, they can achieve similar recall scores by sacrificing lower precision. In addition, the AUC values of these four classifiers are all 0.77, tying for first place. This is followed by DT (0.59), KNN (0.43), and NB (0.41). This is also consistent with our previous experimental results. The four best clustering classifiers obtained by the SK algorithm are also in the same echelon in terms of AUC performance. The horizontal dotted line below represents a noskill classifier, which is a classifier that predicts that all instances belong to the positive class. Its y-value is 0.067, which is equal to the ratio of high-TD modules to non-high-TD modules in the dataset.

Regarding the second major performance metric, the MI ratio, by examining the test phase scores in Table 7 and 8, we can find that the XGB classifier has the best overall performance. It achieved the lowest MI ratio (0.064) before hyperparameter tuning and tied for the highest score (0.099) with the NB classifier after hyperparameter tuning. However, considering that the NB classifier’s precision is less than 0.5 and its recall is the lowest among all classifiers (0.631), this means that it cannot effectively identify high-TD modules. Therefore, the XGB classifier’s performance in terms of MI ratio can be said to surpass all other classifiers. The RF classifier also performed well in this regard, ranking second and third among all classifiers in terms of MI ratio before and after hyperparameter tuning, respectively, demonstrating stable performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Evolution results of classifiers for different metrics.

https://doi.org/10.1371/journal.pone.0323672.t008

Based on the above results, we can easily conclude that XGB is the best classifier in our study, consistently outperforming other classifiers at all stages. The RF classifier can be considered as the second choice, also demonstrating excellent predictive performance and widely regarded as one of the best classifiers in other studies (e.g., Ref. [19]). Such results are not surprising at all, as more complex algorithms (such as RF, XGB, and SVM) perform better than simpler ones (such as DT or NB). This may also be due to the presence of nonlinear underlying relationships in our dataset, leading to better performance of these classifiers.

4.3 Are SNA metrics more effective than TD metrics?

Table 8 presents classifier evaluation results across different metric sets, with detailed combinations described in Section 3.3. Given our focus on optimizing for -score during parameter tuning, the CMS dataset achieves the highest -values in both training and testing phases, outperforming all other metric sets. The TD and SM datasets follow closely. Notably, the SM dataset – derived from CMS via feature selection – secures the second-best -score with XGBoost, marginally trailing CMS. Contrastingly, RF classifier shows comparable performance between SM and TD sets: SM slightly outperforms TD during validation (0.758 vs. 0.756) but underperforms in testing (0.761 vs. 0.766). This counterintuitive outcome challenges theoretical expectations, as SM’s removal of highly correlated features should enhance predictive performance. Instead, it lags behind CMS and even TD in specific cases, aligning with findings by [22] that standalone SNA metrics show limited defect prediction capability but improve performance when combined with code metrics.

Further analysis from Table 8 reveals poor -scores for the full SNA set, underperforming even EN and GN metrics. This suggests potential masking effects from internal feature correlations or Simpson’s paradox [45], where beneficial individual metrics combine detrimentally. Remarkably, the CMS set (SNA+TD integration) achieves optimal results, demonstrating synergistic complementarity.

In order to better observe the performance differences of models trained based on different feature sets, the indicators in this paper are skewed and have unequal variances. Therefore, we used the non-parametric Mann-Whitney U test [46]. And we tested our hypothesis at a confidence level of 95% (= 0.05). This method mainly obtains a U statistic and the corresponding -value. The -value is used to determine whether there is a significant difference between the two groups of data. If the -value is less than the pre-set significance level (0.05), the null hypothesis is rejected, indicating that there is a significant difference in the distributions of the two groups of data; otherwise, the null hypothesis cannot be rejected.

Three sets of comparative experiments were conducted, with the indicator groups being “TD-related metrics + GN metrics,” “TD-related metrics + EN metrics,” and “TD-related metrics + SNA metrics” respectively, and compared them with the model using only TD-related metrics. We used the Mann-Whitney U test to compare the performance differences of the models. Table 9 shows the results of the Mann-Whitney U test. We found that except for the “TD + EN” indicator group, the -values of the other two groups of experiments were all less than 0.05. In particular, for the combination of “TD + SNA,” its -value was 0.0328, which indicates that there is a significant difference between the results of the two models. This further validates the experimental results of the previous chapter of this paper, that is, the SNA metrics can improve the existing TD-related indicator system, thereby enhancing the predictive performance of the model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Mann-Whitney U test results of TD metrics and SNA metrics.

https://doi.org/10.1371/journal.pone.0323672.t009

Based on the above results, we cannot clearly state that the SNA metrics are superior to the TD-related metrics, nor can we consider that the SNA metrics are better than the GN and EN metrics. However, we can draw a conclusion that when the SNA metrics are combined with the TD-related metrics, they can complement the information of the TD-related metrics, thereby improving the predictive ability of the model.To verify our conjecture, we used PCA analysis again to quantify the ability of different indicator sets to capture TD information. Table 10 shows the median number of components required for each indicator set to account for 95% of the data variance information. The larger the number of components, the greater the amount of information contained in the indicator set. We can find that CMS and SM have captured the most information. This also supports our hypothesis, indicating that the combined indicator sets (CMS and SM) have added additional information to the TD metrics. On the other hand, feature screening may have weakened some of the information, resulting in a decrease in the performance of the model (the amount of information in the SM dataset is less than that in the CMS dataset).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. The median number of components required by each metric family to account for 95% of data variance information.

https://doi.org/10.1371/journal.pone.0323672.t010

4.4 Can diverse metric combinations influence the performance of TDPSN models?

Table 11 presents the results of our experiments on the combinations of SNA-related metrics and TD-related metrics. We followed the baseline method in Section 3.5 for our experiments, using the best classifiers XGB and RF obtained in Section 4.2 as our research classifiers. The detailed classification of SNA metrics and TD metrics refers to the metric system in Section 3.3. Considering that the number of Duplication, Complexity, Coupling, and Cohesion metrics are much smaller than Evolution and Size metrics (3 at most, 1 at least), and we found in subsequent experiments that these four types of metrics have poor performance when used alone, so we combine these four types of metrics to form a TD metrics combination with 6 TD metrics, which we call the OM metrics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Evaluation results of classifier after metrics combination.

https://doi.org/10.1371/journal.pone.0323672.t011

From the results, it is evident that the Size metrics comprehensively outperforms the other two sets of metrics (Evolution metrics and OM metrics) when combined with the SNA metrics. Specifically, both RF and XGB achieve higher scores in when compared to their performance in other metrics combinations. Furthermore, the performance of the Size metrics in combination with the SNA metrics is comparable to that of the complete TD and SNA metrics combination. In fact, the score of the RF classifier even surpasses that of the complete TD and SNA metrics combination. Additionally, when combined with EN and GN metrics, the Size index also outperforms both sets of metrics in terms of scores, as well as demonstrating good performance in other performance metrics such as precision, recall, and MI ratio.

Based on the results, it is evident that Size-related metrics outperform other TD-related metrics. Specifically, when Size, metrics and Evolution metrics in TD metrics are combined with SNA-related metrics (SNA, EN and GN metrics), the performance of Size-related metrics is superior to other combinations of metrics across two classifiers in all nine experimental groups. Even when combined with SNA metrics, the RF classifier’s score exceeds that of “TD-related metrics + SNA metrics” combinations. Furthermore, upon observing other performance metrics (precision, recall, and MI ratio), Size-related metrics continue to demonstrate outstanding performance. This suggests that compared to other metrics, Size metrics has a significant impact on the performance of the TDP model as it can better capture key patterns or information related to TDP.

Results emphasize the superiority of Size-related metrics in TD identification. Combinations involving Size metrics and SNA metrics (including EN and GN) consistently dominate other TD metric combinations across classifiers and experimental groups, suggesting Size metrics effectively capture TD-related patterns. Mann-Whitney U tests (Table 12) reveal statistically significant differences (all -values = 0) between Size-based combinations versus Evolution and OM metrics in 6 comparison groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 12. Mann-Whitney U test results of GN metrics and EN metrics.

https://doi.org/10.1371/journal.pone.0323672.t012

Examining the performance of EN and GN metrics when combined with TD-related metrics: The RF and XGB classifiers each underwent four combination experiments. In three out of four combination experiments for RF classifier and two out of four for XGB classifier respectively, GN’s score exceeded that of EN. Particularly in experiments combining “OM-related metrics + EN/GN” and “TD-related metrics + EN/GN metrics” combinations where GN’s score surpassed EN’s on both XGB and RF classifiers. While we cannot definitively state that GN metrics outperforms EN metrics overall; however under certain circumstances within our project context at least there is evidence suggesting that GN metrics may exhibit better predictive performance than EN metrics within the TDP model framework.

Table 13 details comparison tests between GN and EN metrics in TD-related combinations. Mann-Whitney U tests show statistically significant superiority of GN in 3 out of 4 experimental groups (<0.05), except when combined with Evolution metrics. This confirms GN’s enhanced predictive capability in most combination scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 13. Mann-Whitney U test results of Size metrics.

https://doi.org/10.1371/journal.pone.0323672.t013

In conclusion, while CMS-trained models achieve the highest prediction performance, strategic feature selection and combination (particularly with Size-related metrics) can yield comparable or superior results. Future studies should prioritize GN over EN metrics and emphasize Size-related metrics for TDP, as global network metrics (GN) better characterize technical debt patterns than local network metrics (EN). For software maintenance practices, practitioners should prioritize high-size modules (e.g., complex modules) for TD remediation, as size-related factors significantly influence technical debt occurrence and system risk mitigation.