2.1. Ensemble effort estimation (EEE)

As discussed earlier, the phenomenon of conclusional instability has initiated a thorough discussion on overcoming inconsistent findings. Among other solutions, the utilization of ensemble approaches seems to attract the interest of SDEE community. Previous SDEE studies have also supported the concept of using ensemble. An SLR conducted by MacDonell and Shepperd [25] on SDEE reported that using a combined prediction coming from multiple models can improve prediction accuracy if no dominant technique is found. This point of view is also verified later by another SDEE research review done by Jorgensen and Shepperd [7] where it is explicitly concluded that aggregated results generated by single models are more accurate than results of individual techniques. These findings are further verified by ensemble effort estimation SLR conducted by Idri et al [6] The systematic review analyzed 24 studies published between 2000 to 2016 in the following aspect: solo learners utilized to construct ensembles, accuracy achieved, combination rule applied, performance achieved by ensembles in comparison with solo models. The principal findings derived from the studies are;

• 17 out of 24 studies used homogeneous ensemble, making them most frequently utilized ensemble technique.
• 16 solo learners are used to construct ensemble models out of which, ML techniques are frequently incorporated.
• For single base learners, 12 studies applied ANNs and DTs.
• Heterogenous ensemble is explore by 9 out of 24 studies, which yielded better performance compared to their base algorithms (with MMRE = 62.48%, Pred(25) = 43.35%, and MdMRE = 26.80%).
• 20 combination rules are utilized to generate final estimation, and more accurate results are achieved via linear combination rule, also referred as arithmetic mean combiner.
• Overall heterogenous ensembles used combination of 12 single techniques among which, DTs and kNN are frequent choices of heterogenous ensemble members.
• Overall, the results defined that ensemble models produced more accurate results, compared to their base leaners.

The SLR projected a thorough analysis on effectiveness of EEE, however no conclusion is found regarding the best EEE techniques. This work also analyzed recent EEE studies to identify commonly utilized ensemble techniques, base algorithm choices, combination rules and preprocessing technique applied.

A summarized view of EEE studies is presented in Table 1, containing base algorithms used in the study, ensemble technique applied, experimental setup along with model being compared to proposed approach and main conclusion derived from that study. The analysis of past EEE studies also brings conclusion on selecting solo ML leaners. The rationale behind choosing three base algorithms (i.e., RF, SVR and DeepNet) for creating ensemble are:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of EEE studies.

https://doi.org/10.1371/journal.pone.0300296.t001

1. Tree-based models (DTs) are mostly used as stated by SLR of [11] RF is also bagging form of DTs hence we considered this as a constituent of ensemble creation.
2. According to SLR on EEE conducted by Idri et al [6], most ensemble studies found, have selected ANNs as solo learners, hence DeepNet is decided to be part of ensemble in the study.
3. In Table 1, there is a trend of using models like RF, SVR and Neural network in most EEE studies as base algorithms.

All these conclusions bring us to use these three algorithms as solo leaners to carry out our proposed ensemble scheme.

2.2. Metaheuristic optimization (MO)

Parameter setting is a significant criterion for a single ML estimation technique to perform well in predicting effort. Training ML learner requires optimal set of hyperparameters, depending upon dataset and problem targeted. However, setting optimal hyperparameters is not trivial. Manual hyperparameter setting is done by trial and error or other search-based methods as grid search or random search [32]. Manual hyperparameter tuning requires a lot of overhead and become exhausting when parameter search space is large. For that reason, automatic hyperparameter tuning is being considered. Metaheuristic optimization is getting recognition in the field of ML hyperparameter tuning to increase algorithm’s efficiency. Metaheuristic algorithms are simple, highly parallelizable and work best on hyperparameter space [33, 34].

In past SDEE studies, metaheuristic algorithms contributed in providing optimized solution for hyperparameter of single ML technique. However, MO is not included for hyperparameters only. Few researchers included MO in EEE for optimizing weights of various individual models to get best estimation results. Most widely used MO in effort estimation domain include, particle swarm optimization (PSO), genetic algorithm (GA) [35] firefly algorithm (FFA) [36] and bat algorithm (BA), Artificial bee colony (ABC) [37, 38]. Table 2 shows previous EEE work particularly incorporated metaheuristic algorithms for hyperparameter optimization or optimized weight learning for ensemble.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Metaheuristic optimization work.

https://doi.org/10.1371/journal.pone.0300296.t002

2.3. Summary of literature work limitations

From the literature work discussed above, it is possible conclude:

• Utilization of both accuracy and diversity measure simultaneously are not well-formulated. A little to no attempt is made to assess the impact of two significant criteria i.e., accuracy and diversity in ensemble’s working.
• Although, MO is considered for achieving optimal hyperparameter for single ML, but utilizing MO in both perspectives, i.e., for hyperparameter optimization and weight optimization of ensemble is not considered under one framework. Although work of [42] applied the use of GA for optimizing parameters of analogy technique and then determining weights for ensemble creation, but ensemble constituents are homogeneous versions of analogy-based techniques. Ensemble weights optimization of diverse ML algorithms to create ensemble is still not incorporated.
• Hyperparameter optimization of ML algorithms entails search space to get best configuration of hyperparameters by search-based or metaheuristic algorithm. For a particular optimization problem, criteria for defining and selecting search space is not part of any SDEE work.
• Although ample amount of work is done in the field of EEE. Researches have fairly incorporated homogeneous and heterogenous ensemble in developing SDEE model but power of using both kinds of ensemble simultaneously is not explored Publicly available effort estimation repositories contain heterogenous datasets, differing in software projects size, field, organization and dissimilar effort drivers presented in each dataset. Homogeneous ensemble may give acceptable results on one configuration of dataset and fail to estimate effort for a different setting, while heterogenous is suitable in those conditions. For that reason, if both types of ensembles are present in one model, then it would bridge the gap of dataset and configuration uncertainties.

2.4. Problem formulation and research questions

Limitations presented in previous literature (Section 2.3: Summary of literature work limitations) are considered for problem formulation of this wok listed below:

1. Problem: Absence of accuracy and diversity considerations while creating ensemble.
   Solution: This work incorporates both measures by utilizing multi-dimensional bagging (diversity) and MO (accuracy).
2. Problem: Analyzing the impact of both optimization domains (hyperparameter optimization and optimal weights assignment) while creating ensemble is missing.
   Solution: Optimization of both perspectives I spart of this study. Hyperparameters of base algorithms are optimized (using MO), then combined in heterogenous ensemble with optimized weights (optimized with MO).
3. Problem: For ML hyperparameter optimization, no consideration is made on defining search-space selection criteria.
   Solution: This work proposed to include search-space division, for getting optimized hyperparameters from multiple sub-sections of same larger search-space.
4. Problem: Investigating the use of both ensemble mechanisms (Homogeneous and Heterogenous) simultaneously is overlooked.
   Solution: This work contributes in combining both kinds of ensemble, i.e. homogeneous (in form of multi-dimensional bagging) and heterogenous (in form of Metaheuristic- optimized weighted ensemble).

From this perspective, this study attempts to address following research questions:

1. RQ1: Does optimization included in both domains (hyperparameter optimization and optimal weights assignment) improve estimation performance?
2. RQ2: Does performing search space division endorse same results as utilizing entire search space?
3. RQ3: Does integration of Homogeneous and Heterogenous ensemble tend to improve the performance or Homogeneous/Heterogenous ensemble alone can give good performance?

3.1. Random Forest (RF)

RF incorporates bagging techniques, referred to as “bootstrap aggregation”. This involves generating a new dataset from an existing one through bootstrap sampling with replacement [43]. RF implementation involves two major strategies: (1) Forming a certain number of trees with data using different bootstrap samples (i.e., bootstrap with replacement). (2) For splitting each node, randomly chosen best feature among the “subset of predictors” is used, instead of taking best split among all variable for splitting each node, like standard trees [44–46]. This study implements RF working as follows:

1. Random bootstrap sampling: Random bootstrap sampling is performed with replacement for each dataset D, containing d observations, and F features. These bootstrap samples are then used for training ‘n’ number of base trees.
2. Random Feature Selection: For the n^th base tree, a subset of m features is randomly selected for model training. Constant tree structure Ͳⁿ_m is learned for nth base tree. For node splitting, instead of traversing every possible split in all F features, Ͳⁿ_m only consider splits in a randomly selected feature subset.
3. Pruning: Pruning operation is performed on constructed constant tree Ͳⁿ_m with M5 based method. After pruning, linear decision tree model ƒ_m (x) is formed by converting Ͳⁿ_m tree structure. For final pruned decision tree, having P leaf nodes, regression function for p^th leaf node is denoted by Һ_p (x). Prediction generated from entire decision tree model is presented by Eq 1, where ℊ(a) is a conditional function which returns 1 if a is true and returns 0 otherwise. (Eq 1)
4. Average aggregation: Randomized decision trees are constructed in parallel with above-mentioned process, resulting N base trees. For a sample x, prediction from N base trees is generated. Final estimation (ℱ (x)) is made by averaging the prediction of N base models (ƒ_m (x)) mentioned in Eq 2. (Eq 2)

Since regression trees has tendency to overfit for small datasets, which in-turn considerably effects performance of RF. To avoid this, there is a need of hyperparameters optimization for RF for better performance. Hyperparameters for RF include: the number of trees constituting the forest (ntree), the number of features randomly selected at level of each node (mtry), minimum number of data samples in a leaf node (nodesize) and the size of in-bag samples (sampsize).

3.2. Support vector Regression (SVR)

SVR is among most frequently utilized ML technique in SDEE domain, suitable for linear/nonlinear regression. [47–50]. Regression based on SVR consist of data where a_i is vector of independent variables, while b_i corresponding scalar real dependent variable. SVR regression equation in feature space can be represented as Eq 3: (Eq 3) where, w defines the weight vector, c is a constant, Φ(a) is the feature function while w.Φ(a) represents dot product of two terms. Lagrangian multiplier β and β^* are also incorporated and only non-zero coefficients, along with their input vectors, a_i, are termed the support vectors. The final form comes out as represented in Eq 4: (Eq 4)

By the help of kernel function K(xi,xj), the SVR function can be obtained by Eq 5 as given below: (Eq 5)

Kernal plays an important part in SVR implementation. Radial basis function (RBF) kernels are the most generalized form of kernelization and is one of the most widely used kernels due to its similarity to the Gaussian distribution. The RBF kernel function for two points xi,xj computes the similarity between them or how close they are to each other. RBF can be mathematically represented by Eq 6 as follows: (Eq 6)

||x_i − x_j|| is Euclidean (L2-norm) distance between two points x_i,x_j. In RBF, γ is parameter used to tune the equation, and computed as follows in Eq 7, where σ is the variance.

(Eq 7)

The key parameters need adjustment for SVR are; complexity or penalty parameter C (also referred as cost) which controls the trade-off between error minimization and margin maximization. Its optimization is necessary as to have an idea how much misclassifying must be avoided on each training sample. Another parameter of SVR, epsilon (ε), defines the extent to which deviations (i.e., errors) are tolerated. For model training, SVR works by finding a function with at most ε deviation from actual value of dependent variable in all data samples, while keeping function as flat as possible. epsilon (ε) optimization is required since, function to adjust tolerable error of the regression model is essential. Another optimizable parameter, gamma (γ) suitable for radial basis function (RBF) kernel [47]. gamma parameter defines how far the influence of a single training example reaches.

3.3. Deep neural network (DeepNet)

Deep learning is quite potential ML-SDEE technique, exploiting precision abilities of Deep Neural Network (DeepNet) [51]. The reason is its ability to represent complex relationships between dependent variable (effort) and independent variables (effort drivers). For building effort estimation model, DeepNet is construed using three basic layers. (a) An input layer containing input neurons (effort drivers); (b) hidden layers with neurons that calculate their output by means of an activation function. Let L denotes hidden layer, for modeling DeepNet, the number of hidden layers would be L > 2 (c) an output layer which takes output from hidden layer neurons and accumulates linear weighted sum, serves as prediction of network (i.e., estimated effort). Since effort estimation is regression problem, so in case of regression DeepNet has only one output neuron. The layer labeled X is the input layer which contains all the explanatory variables k in the data set. The layers labeled Z are the hidden layers. The number of hidden layers L can be arbitrarily set, however, in this study, three hidden layers are used. Each hidden layer can contain arbitrarily many neurons denoted by vl where l stands for the l^th hidden layer. Z is a vector of outputs from all neurons of the (l − 1)^th layer represented as follows in Eq 8 (Eq 8)

The output z_lv of the v^th neuron in the l^th hidden layer is represented in Eq 9: (Eq 9)

The function f(a) is referred as the activation function. Sigmoid is commonly used activation function and apart from Sigmoid, Rectified Linear Unit (ReLu) is also a popular choice and considered appropriate for the reduction of overfitting influence [52]. Sigmoid and ReLu are shown in Eqs 10 and 11.

(Eq 10)

(Eq 11)

The outputs of the first, hidden, and output layers are expressed as Eqs 12 and 13 and 14. (Eq 12) (Eq 13) (Eq 14) where W and b represent the weight matrix and bias vector of the l^th hidden layer, respectively. For the input layer, the vector of explanatory variables of input features (x) is used while for hidden layers, computed values of previous hidden layer (h_l-1) is used.

Performance of DeepNet is largely dependent on optimal hyperparameter configuration including: (1) number of hidden units (OR size of DeepNet) (2) decay (3) the number of epochs used in training (4) the learning rate and (5) the momentum. In this study, parameters selected for optimization are number of hidden neurons size and decay.size parameter can affect DeepNet’s performance since depth of hidden layers enhance data fitting capability of DeepNet model. Less hidden layer size causes underfitting, making model to inadequately detect the signals in a complicated data set. While increased hidden neurons may result in overfitting and unnecessary model complexity, i.e., model will have larger information processing capacity with little information to be processed on. Hence an optimized value of hidden layer neuron is required for better functioning DeepNet. Other parameter, decay, is regularization parameter preferably to work with L2 regularization to avoid over-fitting. It is used to keep the weights small to avoid oversizing the gradient. For a regression problem, L2 regularization is suggested to apply [53]. L2 norm of weights are added to the loss, which might result in loss getting quite large. Due to this, DeepNet model tend to set all model parameters to 0. Hence, optimized value of decay is required, to keep weights small and preventing weights to grow out of control. Also, optimization of decay is needed because, with too much weight decay model never quite fits and too little weight decay causes model to stop a little bit early.

3.4. Firefly Algorithm (FFA)

As discussed earlier, performance of ML technique largely correlates to accurate parameter setting. Moreover, hyperparameter tunning differs from dataset and the application context [23, 54]. Hence, for a given ML technique, using the same parameters for multiple datasets can influence its prediction ability. Therefore, this work put focus on getting optimal hyperparameters setting for each ML algorithm with respect to the dataset. Moreover, for heterogenous ensemble, weights assigned to each model’s prediction is also critical to be optimized, for getting high performing ensemble. A metaheuristic is a search-based heuristic, for getting best solution of optimization problems having partial scope with limited information present [31, 55]. MO provides optimal or even sub-optimal solutions for highly nonlinear and multimodal optimizations problems.

FFA falls under the category of bio-inspired algorithms, i.e., inspired from the behavior of the swarm such as bird folks, insects, fish schooling in nature. FFA was originally proposed by Xin-She Yang [56] and serves well in metaheuristic optimization. According to some recent studies, FFA evolves as promising algorithm as it outperforms other metaheuristics such as genetic algorithm [57].

FFA is based on population of fireflies and works by adapts their rhythmic flashing light behavior for attracting mating partners or potential preys. The light intensity of a firefly lessens when it moves away from other firefly and modifies according to inverse square law. The light intensity (brightness) of firefly (referred as L) decreases when distance h from its source increases, i.e., L ∝ 1 /h². Also, L decreases by light absorbed in the environment. This behavior contributes in moving from a locally optimal solution to a globally optimal solution. Rules of FFA suggest that all fireflies are unisex, so attractiveness criteria between two fireflies is “brightness” of each firefly. Brighter firefly attracts less bright firefly, hence less bright firefly moves towards brighter one. However, for a firefly, if there is no brighter firefly, it will perform random walk i.e., move randomly. Brightness L is directly related with the objective function. For a particular position p, the brightness of firefly will be is chosen as L(p) ∝ f(p), where f(p) is objective function in optimization problem which needs to be maximized or minimized. The combined effect of both, the inverse square law and absorption, can be approximated to Gaussian form as L(h) = L₀e ^−γh2. Since a firefly’s attractiveness is proportional to the light intensity, the attractiveness function of the firefly can be defined as; β(h) = β₀e ^−γh2; where β0 is the attractiveness at h = 0 and γ is the light absorption coefficient

Attractiveness of firefly depends on its brightness, which varies according to distance between two fireflies. So, for two fireflies, firefly i and firefly j, attractiveness β will be less if distance h_ij between them is greater. Cartesian distance is taken between any two fireflies i and j located at positions p_i and p_j respectively according to Eq 15 [58].

(Eq 15)

Where p_i,r is spatial coordinate p_i of ith firefly. If fitness of firefly i is greater than fitness of firefly j, then i will move from a current position p_i towards the position p_j (of firefly j), and movement is determined by Eq 16.

(Eq 16)

Where p_i is the current position of a firefly, the second term i.e., is the firefly attractiveness with respect to light intensity (L) observed by nearby fireflies. Third term α(rand– 1/2) is the random walk of firefly when there is no brighter firefly; where α is a randomization coefficient and rand is a random number, both can be drawn from normal distribution.

4.1. Tier-1: Multi-dimensional Bagging (Mdb) scheme

Tier-1 works by taking individual base algorithm (solo learners) at a time and creating multi-dimensional bagging scheme for each base algorithm. Base algorithms used in this study are RF, SVR, DeepNet.

For one particular base algorithm, initial search space (SP) of its hyperparameters is defined. Initial search space (SP) of hyperparameters is then divided using proposed multi-dimensional search sub-spaces (Sub_SPs) formation technique. Number of dimensions in initial search space (and in Multi-dimensional bagging) depends on number of hyperparameters to be optimized for single base algorithm i.e., for a base algorithms with m optimizable hyperparameters, m-dimensional SP and Sub-SPs will be formed for creating Mdb. Best hyperparameters from each Sub_SP are then obtained with the use of MO. In this study Firefly algorithm (FFA) [40] is used for optimization due to its promising results on tuning hyperparameter of three base algorithms (Appendix A: Table 13 in S1 File). Each Sub_SP gives one set of optimized hyperparameters. These N sets of optimized hyperparameters (coming out of N Sub_SPs) are then used to train N number of bags (bootstrap samples with replacement) from the dataset. These N-bags, trained from optimized hyperparameters are then combined in the form of bagging to get Multi-dimensional bagging (Mdb) Scheme for one base algorithm. Same process is repeated for remaining base algorithms to get Mdb scheme for each base algorithm. Fig 3 depicts complete working process of Tier-1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Tier 1-Multi-dimensional Bagging (Mdb) scheme.

https://doi.org/10.1371/journal.pone.0300296.g003

Steps for implementing Mdb scheme is as follows:

1. Take one base algorithm Z and select its hyperparameters to be optimized.
2. For m-hyperparameters, create m-dimensional grid. This m-dimensional grid will act as initial SP for algorithm Z.
3. Create N Sub_SPs by dividing the Initial SP (m-dimensional grid) according to Algorithm 1.
4. Feed each Sub_SP separately, into firefly algorithm (as range with lower bound and upper bound).
5. For N Sub_SPs, obtain N sets of optimized hyperparameters, according to Algorithm 2.
6. Create N bootstrap samples (with replacement) from the dataset. Use one sample_n to train algorithm Z with one set of optimized hyperparameter coming from steps 4–5. This will produce one bag for Algorithm Z.
7. Repeat the same process for remaining N-1 bootstrap samples and create total N-bags for algorithm Z.
8. Combine all N trained bags to have Multi-dimensional bagging (Mdb) scheme for algorithm Z.
9. Repeat step 1 to 8 for remaining base algorithms.

1. Algorithm 1: Sub-Search spaces (Sub_SPs) formation from initial SP
2. Input: Algorithm (Z); where [Z} ϵ [RF, SVM, DeepNet}
3. m = number of hyperparameters of Algorithm (Z) and highest dimension of grid
4. N = No. of Multi-dimensional bags to be formed;
5. error = error value point (MAE) in m-dimensional grid
6. Output: N search sub-spaces
7. Begin:
8. 1. Select Algorithm (Z)
9. 2. Select m-hyperparameters of Algorithm (Z)
10. 3.Creat m-dimensional grid for Algorithm (Z); where m = number of hyperparameters of Algorithm (Z)
11.           Algorithm(Z)_Grid [m]
12. 4. Define N error ranges
13.           split = Min_error + Max_error / N
14.     for i = 1 upto N
15.   do
16.           error_range(i) = split + error_range(i-1)
17.     end for
18. 5. Create N Sub_SPs [lower bound, upper bound]
19.         Sub_SP (n) = [(Lower_Bound (n)), (Upper_Bound (n)) ]
20.     for j = 2 upto N
21.   do
22.      Sub_SP (1) = [0, error_range(1)]
23.     Sub_SP (j) = [(Algorithm(Z)_Grid(error) > error_range(j-1)), (Algorithm(Z)_Grid(err) < error_range(j)) ]
24.     end for
25. Return N Sub_SPs
26. End

1. Algorithm 2: Firefly Algorithm for Optimized Hyperparameter of N Sub_SPs
2. Input: m = No. of hyperparameters to be optimized for base Algorithm
3. max_pop = Maximum population of fireflies
4. I_o = initial light intensity at h₀; γ = Light absorption coefficient; h = Distance between two fireflies;
5. t = iteration; α = randomization parameter; δ_i^t = vector of the random number drawn from Gaussian distribution at iteration t; β0 = Attractiveness at h = 0; α ∈ [0,1]; rand ∈ [0,1]; scale = (Lower_bound- Upper_bound)/Upper_bound
6. Output: Best firefly (optimal solution) for each N Sub_SP
7. Begin
8. 1. Assign j^th Sub_SP as range for FFA
9.         for each Sub_SP j = 1 upto N
10.   do
11.           Lower(Sub_SP (j)) = Lower_bound
12.           Upper(Sub_SP (j)) = Upper_bound
13. 2. Initialization random population of Fireflies; Xj; where j = {1,2, …, max_pop} and Xj = {x_j1, x_j2 … x_jm}
14.       (Eq (a))
15. 3. Calculate distance between firefly (Xj) and firefly (Xk) according to Eq(b)
16.    (Eq (b))
17. 4. Calculate the fitness (objective function) of each firefly
18.     fit_Algorithm(Xj) {
19.         return fitness = min MAE }
20. 5. Update position of firefly
21.     for t = 1 upto max_iteration
22.   do
23.     If t = max_iteration,
24.     then return X_j^t as Xj(best); return the best position as an optimal solution
25.     else if t ≠ max_iteration and
26.           fit_Algorithm(Xj) < fit_Algorithm(Xk); firefly(Xj) moving to more attractive firefly(Xk)
27.         then move position of firefly(Xj) toward firefly(Xk) according to Eq(c)
28.       (Eq (c))
29.           else if no brighter firefly than firefly (Xj) according to Eq(d)
30.           then perform random walk
31.      (Eq (d))
32.           end if
33. 6. Determine fitness of firefly on updated position
34.           Calculate fit_Algorithm (X_j ^t+1)
35.        If fit_Algorithm (X_j ^t+1) < fit_Algorithm (X_j^t)
36.        then Update
37.          firefly(X_j^t) with firefly(X_j ^t+1)
38.     end if
39.     return firefly (X_j(best))
40.     where X_j(best) has min(fit_Algorithm(X_j))
41.           end for each
42. Return Xj(best) for each Sub_SP (j); j {1, 2, …, N]
43. End

4.2. Tier-2: Metaheuristic-optimized Weighted Ensemble (MoWE)

Tier-2 works on combining the output coming from Tier-1 using Metaheuristic-optimized weighted ensemble (MoWE). At the end of Tier-1, three Mdb schemes (one Mdb for one base algorithm) are obtained. Weights for individual Mdb are learned from FFA and weighted ensemble is created. Effort coming from this MoWE serves as final effort prediction for a particular dataset. Steps for creating MoWE are as follows (Algorithm 3):

1. Divide entire dataset D into training and testing data using 70:30 ratio.
2. For training dataset, train three Mdb schemes separately (obtained from Tier-1).
3. Acquire predictions from each Mdb scheme on training dataset.
4. Obtain weights of each Mdb prediction using Firefly algorithm with minimized MAE
   (i.e. fitness = minimum MAE)
5. Repeat step 2 and 3 for 30% remaining test data and obtain test data predictions.
6. Make final prediction on testing data by creating MoWE, i.e., assign optimal weights (obtained from FFA in step 4) to predictions obtained in step 5.

1. Algorithm 3: Metaheuristic-optimized weighted ensemble (MoWE) creation
2. Input: Multi-dimensional bagging schemes for all three base algorithms = {Mdb_Scheme] ϵ {RF_mdb, SVM_mdb, DeepNet_mdb]; w_n = weight to be optimized
3. max_pop = Maximum population of fireflies; t = iteration
4. Output: Metaheuristic-optimized weighted ensemble (MoWE)
5. Begin:
6. 1. Divide dataset D^m into {D^m_train, D^m_test ] by {70:30] ratio
7. 2. Define fitness function for FFA
8.         fit_weights (w1, w2, w3) {
9.         return fitness = min MAE ]
10. 3. Define search space for weights
11.         weights search space = w1 → [0.1, 0.9], w2 → [0.1, 0.9], w3 → [0.1, 0.9]
12. 4. Start FFA
13.     for each t = 1 upto max_iteration
14.         for j = 1 upto max_pop
15. On training data, D^m_train
16. 5. Perform training on D^m_train using {mdb_Scheme]
17.         model1 = train (RF_mdb, D^m_train)
18.         model2 = train (SVM_mdb, D^m_train)
19.         model3 = train (DeepNet_mdb, D^m_train)
20. 6. Get prediction of above trained models on D^m_train
21.         pred1 = prediction (model1, D^m_train)
22.         pred2 = prediction (model2, D^m_train)
23.         pred3 = prediction (model3, D^m_train)
24.         ensemble_prediction = (pred1*w1) + (pred2*w2) + (pred3*w3)
25.    return fitness = min MAE
26.         end for
27.    end for each
28. 7. Obtain best values of ensemble weights from FFA
29.         return w1_best, w2_best, w3_best
30. On test data, D^m_test
31. 8. Repeat Step 5 and perform training on D^m_test
32.         return model1, model2, model3
33. 9. Repeat Step 6 and obtain predictions on D^m_test
34.         return pred1, pred2, pred3
35.         optimized_ensemble_prediction = (pred1*w1_best) + (pred2*w2_best) + (pred3*w3_best)
36.         return optimized_ensemble_prediction
37. End

5.1. Multi-dimensional bagging (Mdb): Implementation setup

This section describes the implementation details of Tier-1 of proposed framework. For this purpose, m-dimensional grid is formed for each base algorithm. From, it is cleared that, 3-dimensional, 4-dimensional and 2-dimensional grids are formed for RF, SVR and DeepNet respectively, which will serve as their initial SP.

Now for each base algorithm, Sub_SPs are created by dividing their initial SP as mentioned in Algorithm 1. Due to limitation of space, working of first 9 Sub_SPs of SVR, on Albrecht dataset is explained further in this section. Fig 4 represents one view of initial SP of SVR using epsilon and coef0 hyperparameters. From this initial SP, Sub-SPs1 to 9 are formed, represented in Fig 5. Red line shown in Fig 5 illustrates division of initial SP according to Algorithm 1. These Sub_SPs are used to create bags in proposed Mdb. Since performance of bagging algorithm is largely affected by appropriate number of bags. Hence for achieving sufficient number of bags, Sub_SPs needed to be increased. For this purpose, 2 hyperparameters are considered at a time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Initial SP of SVR (using epsilon and coef0).

https://doi.org/10.1371/journal.pone.0300296.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Sub_SP of SVR (using epsilon and coef).

https://doi.org/10.1371/journal.pone.0300296.g005

Hence, in Sub-SPs 1 to 9, ranges for epsilon and coef0 are taken from division mentioned in Fig 5 and ranges of remaining hyperparameters cost and gamma are kept same as initial SP (i.e., cost [1,100] and gamma [0.001,1]). For Sub_SP 1 to 9, Table 5 represents Sub_SP range for each hyperparameter. These Sub_SP ranges serve as lower and upper bound for implementing FFA. FFA gives optimized value for each hyperparameter coming from their Sub_SP range (shown in Table 5 under column “Optimized”).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Sub_SPs ranges and optimized values of SVR hyperparameters.

https://doi.org/10.1371/journal.pone.0300296.t005

Similarly, for Sub_SPs 10 to 18, combination of other two hyperparameters (gamma and coef0) is taken (Fig 6) and their Sub_SPs are formed (Fig 7). For Sub_SPs 10 to 18, remaining two hyperparameters (epsilon and cost) have ranges same as their initial SPs (Table 6). Same process is repeated for all combinations of SVR hyperparameters. Figs 8 and 9 represent Initial SPs and Sub_SP respectively for hyperparameters (gamma, cost). Similarly, Figs 10 and 11 show Initial SP and Sub_SP for hyperparameters (coef0, cost). Figs 12 and 13 for hyperparameters (epsilon,cost), and Figs 14 and 15 for hyperparameters (gamma,epsilon), respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Initial SP of SVR (using gamma and coef0).

https://doi.org/10.1371/journal.pone.0300296.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Sub_SP of SVR (using gamma and coef0).

https://doi.org/10.1371/journal.pone.0300296.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Initial SP of SVR (using gamma and cost).

https://doi.org/10.1371/journal.pone.0300296.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Sub_SP of SVR(using gamma and cost).

https://doi.org/10.1371/journal.pone.0300296.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Initial SP of SVR (using coef0 and cost).

https://doi.org/10.1371/journal.pone.0300296.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Sub_SPs of SVR (using coef0 and co.

https://doi.org/10.1371/journal.pone.0300296.g011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g013

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g014

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g015

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Initial SPs of three base algorithm.

https://doi.org/10.1371/journal.pone.0300296.t006

After applying above mentioned process for all possible combinations of SVR hyperparameters, 54 Sub_SPs are formed (in this paper, we mentioned only first 9 Sub_SPs due to limitation of space). These Sub_SPs are fed into FFA for getting best hyperparameters residing in each Sub_SP. Hence, total of 54 Sub_SPs are fed into FFA individually, to get 54 sets of optimized hyperparameters of SVR. These 54 sets of FFA-optimized hyperparameters are used to train 54 bags of SVR, which are then combined to get finalized Mdb scheme for SVR named as SVR_Mdb. Mdb schemes for RF and DeepNet are obtained using the same process and referred as RF_Mdb and DeepNet_Mdb respectively. Descriptive statistics of optimized hyperparameters for three solo learners (RF, SVR and DeepNet) are listed in Appendix A: Table 12 in S1 File.

5.2. Dataset

Implementation of proposed effort estimation scheme is performed on 8 datasets, available on SEACRAFT [63] repository. Software Engineering Artifacts Can Really Assist Future Tasks (SEACRAFT) is publicly available online data repository (formerly known as PROMISE [64]. Selected dataset for this study are: Albrecht [65] Deharnais [66] Miyazaki [67], China [68], Cocomo81 [69], Finnish [70], Kitchenham [71] and Maxwell [72].

These datasets were selected due to their frequent utilization in evaluating ensemble effort estimation techniques [6]. Table 7 summarizes descriptive statistics of all selected datasets including, number of projects each dataset, number of features, effort measurement unit along with median, mean, minimum, maximum, skewness and kurtosis measures of effort. As it is clear from Table 7, effort values of all datasets do not follow a normal distribution since skewness values ranged from 2.03 to 10.64 [73]. This is rendered using min-max normalization rule to keep degree of influence similar for all models. Reason for applying normalization first is that, although selected ML techniques work with any data distribution, but effort prediction is more accurate when values of output variable (effort) are symmetrically concentrated around the mean [74].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Descriptive and statistical details of effort estimation datasets used.

https://doi.org/10.1371/journal.pone.0300296.t007

5.3. Comparison models

We compared our proposed MoMdbWE (Mdb with MoWE) technique with multiple effort estimation methods. First, proposed Mdb schemes of all three models are compared with their respective solo base algorithms (RF, SVR, DeepNet), to verify improvement of proposed Multi-dimensional bagging over individual learners. Also, three Mdb schemes are compared with bagging ensemble [8] to confirm if Mdb provide better prediction than normal bagging.

After that, proposed MoWE model is compared with other heterogenous ensemble methods found in literature, including gradient boosting [75] stacking [76] majority voting and weighted ensemble (with non-optimized weights) [30]. Further, performance comparison of our proposed MoMdbWE techniques is made with previous EEE studies (Table 11).

5.4. Performance evaluation measures

This section includes performance criteria used in this study to evaluate proposed MoMdbWE technique in comparison with the models (mentioned in Section 5.3: Comparison models). Performance measures (error metrices) used in the study are: Mean absolute error (MAE), Root mean square error (RMSE), Magnitude of relative error (MRE), Mean magnitude of relative error (MMRE), Median of magnitude of relative error (MdMRE) and Pred(l), which are widely used evaluation metrices in SDEE literature [77]. SA and Δ are another important evaluation measures to analyze performance of effort estimation model. Many SDEE studies declared MMRE as unreliable measure for prediction [25, 78] due to its biasness towards underestimation and asymmetric nature. For this reason, SA and Δ are also incorporated in this study, proposed by Shepperd and McDonell [25]. Details of evaluation measures are as follows:

1. MAE is calculated by taking average absolute difference between estimated effort and the effort observed, represented by Eq 17
   (Eq 17)
2. RMSE is defined as the standard deviation of the differences between actual and calculated effort given by Eq 18
   (Eq 18)
3. The MRE for finding the relative error between actual effort values and observed effort values for each project of dataset is calculated as given in Eq 19.
   (Eq 19)
4. MMRE is calculated by taking average of MRE’s of all data samples, as given in Eq 20.
   (Eq 20)
5. MdMRE is the median of MREs of all projects of a dataset as shown in Eq 21
   (Eq 21)
   Pred(l) is the count of MRE less than or equal to a value (l), for all data points, represented as Eq 22 (Eq 22)
   Where N is total number of project instances, l is taken as 0.25, so PRED (0.25) will be calculated as total number of projects having MRE less than or equal to 0.25. An effort estimation model is considered satisfactory in terms of accuracy if MRE is less than 0.25 and PRED (25) is greater than 0.75 [3].
6. Standard accuracy (SA) reflects the percentage by which an effort estimation gives accurate prediction, in comparison with random guessing. SA is calculated based on MAE of a model. Effect size (Δ) is used to verify whether estimate of a models, occurred by chance and is there any improvement observed over random guessing. SA and Δ are shown in Eqs 23 and 24 respectively.
   (Eq 23)
   (Eq 24)
   Where is MAE of the estimation method ρ_t, and is mean of 1000 random guessing samples. is defined as “predicted effort of a target task by randomly choosing training instance with equal probability over all remaining T − 1 samples”. Assign Estimated Effort_t = Estimated Effort_r where r is randomly chosen from is standard deviation of random guessing samples. Further, Shepperd and McDonell [25] recommended using 5% quantile of random guessing while estimating a technique, to ensure likelihood of non-random prediction. SA near to zero reflects poor performance of model than random guessing, SA near to 1 ensures better performance while, negative SA values are worrisome [25]. Absolute value of Δ are interpreted in terms of small (≃ 0.2), medium (≃ 0.5) and large (≃ 0.8) categories as proposed by Cohen [79]. Values of Δ > 0.5 reflects better performance of a model.

5.5. Statistical evaluation

Statistical evaluation is very crucial part of any research, since it provides evidence that performance achieved by a model is significant enough to be compared. To ensure if proposed model’s results are statistically significant than other comparing models [80], Wilcoxon test is used [81]. Wilcoxon test is included to verify the significance difference between error measures of all comparing models. The p-value (significance level) of Wilcoxon test evaluates that the results of a model are not accumulated by chance. Wilcoxon test belongs to non-parametric statistical test category, i.e., it makes no assumptions about the probability distributions of the variables under test.

In this study, Wilcoxon test is implemented with Bonferroni correction [82] and significance level (α = 0.05) is applied to minimize Type I error [83]. For each dataset, statistical test is performed on absolute errors accumulated from testing data samples. For each pairwise comparison, samples of absolute error are taken from same test set.

The null and alternative hypothesis of significance test are shown in Eq 25: (Eq 25)

Null hypothesis (H_o): Difference between error means of two compared models is zero, i.e., a model_k (with error mean μ_k) have similar performance compared to model_m (with error mean μ_m)

Alternative hypothesis (H₁): Difference between error means of two compared models is not zero i.e., a model_k doesn’t hold equal performance model_m.

6.1. Results of proposed Multi-dimensional bagging (Mdb) and Metaheuristic-optimized weighted ensemble (MoWE)

Evaluation results of proposed Multi-dimensional bagging (Mdb) and Metaheuristic-optimized weighted ensemble (MoWE) are discussed in this section. listed SA and effect size (Δ) measures of all compared models.

Firstly, we will discuss the performance of all three Mdb schemes. As shown in the,. Mdb schemes achieved compelling estimation accuracy in all datasets. In terms of SA, RF_Mdb, SVR_Mdb and DeepNet_Mdb outperformed their solo learners (RF, SVR and DeepNet) as well as their normal bagging versions. If we look closely to SA values of, there are some case where solo learners surpassed their bagging versions. For instances, RF surpassed RF_Bagging for Albrecht, China and Finnish datasets, and both algorithms gave equal performance in Desharnais, Miyazaki, Kitchenham and Maxwell datasets. Similarly, SVR gave better performance than SVR_Bagging in Albrecht, China, COCOMO81 and Finnish datasets, while both are equal in Miyazaki dataset. DeepNet, however surpassed from DeepNet_Bagging in Albrecht, Miyazaki, Finnish and Kitchenham datasets while no equal performance in observed for both. Again, it is worth mentioning, no solo algorithm or bagging algorithm surpassed or gave equal performance in comparison with Mdb schemes i.e., Mdb schemes of all three solo learners (RF_Mdb, SVR_Mdb and DeepNet_Mdb) gave significantly improved performance compared to solo and bagging algorithms.

Results of also confirm that, performance of effort prediction further improvised when individual Mdb schemes are combined using FFA-optimized weights to form MoWE. SA of proposed MoWE is giving good performance evaluation in comparison with solo algorithms as well as other heterogenous ensembles. Moreover, MoWE in all datasets performed better then all three Mdb schemes. Among heterogenous ensembles (apart from MoWE), Majority voting has shown second best considerable performance in terms of SA for Albrecht, Desharnais, Miyazaki, Cocomo81, Kitchenham datasets, while Stacking is second best technique in China, Finnish and Maxwell datasets.

As discussed in Section 5.4: Performance evaluation measures, performance of effect size is interpreted as small (≃ 0.2), medium (≃ 0.5) and large (≃ 0.8) categories. Corresponding to the SA performance, similar results are observed in terms of Δ measure, where absolute values of Δ of all Mdb schemes are higher than solo learners and normal bagging.

For Albrecht, however, models shown effect size between small to medium category, except SVR and SVR_Bagging, which shows performance even less than small effect size category (i.e., Δ < 0.2). No models trained on Albrecht fall under large effect size. For Desharnais datasets, Δ lie between small to medium category only, showing quite moderate performance. For Miyazaki datasets, Δ values observed for all models are of large effect size except DeepNet and DeepNet bagging, which still lie under moderate Δ range. China dataset, showing 10 out of 14 models with moderate effect size category, while in Cocomo81, half models are observing medium Δ range. For Finnish dataset, all models found are in large to medium category (except Majority voting ensemble). Model trained on Kitchenham and Maxwell show small to very small effect size category. Overall, for Miyazaki and China dataset, models with good effect size range are observed. Proposed MoWE ensemble has shown medium to larger Δ category in all datasets (expect Kitchenham). MAE performance of Mdb schemes and MoWE is also expressed graphically in Figs 16–23 for all datasets. Clear, there is a decreasing trend in error (MAE) from base algorithms to Mdb schemes, to MoWE.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g016

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 17. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g017

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 18. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g018

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 19. Mdb and MoWE performance on Albrecht dataset.

https://doi.org/10.1371/journal.pone.0300296.g019

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 20. Mdb and MoWE performance on Cocomo81 dataset.

https://doi.org/10.1371/journal.pone.0300296.g020

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 21. Mdb and MoWE performance on Finnish dataset.

https://doi.org/10.1371/journal.pone.0300296.g021

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 22. Mdb and MoWE performance on Kitchenham dataset.

https://doi.org/10.1371/journal.pone.0300296.g022

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 23. Mdb and MoWE performance on Maxwell dataset.

https://doi.org/10.1371/journal.pone.0300296.g023

6.2. Result of statistical evaluation

For statistical evaluation, Wilcoxon test is performed based on absolute errors of all compared models. For reporting statistical test results, post-hoc analysis with Bonferroni correction [82] is performed by pairwise comparison. For tier 1, comparison is made between:

1. Three Mdb schemes (RF_Mdb, SVR_Mdb and DeepNet_Mdb) and their respective base algorithms.
2. Mdb schemes with normal bagging

For tier 2, Wilcoxon test with post-hoc analysis performed between MoWE and:

1. Three base algorithms
2. Normal bagging schemes of three solo models (RF_bagging, SVR bagging, DeepNet bagging)
3. Three Mdb schemes (Mdb_RF, Mdb_SVR, Mdb_DeepNet
4. Other heterogenous ensemble models.

The reason for this comparison is to justify, whether performance of Mdb schemes is significantly improved compared to their base algorithms and normal bagging versions of base algorithm. p-value of Wicoxon comparison is reported in Since no base algorithm and bagging algorithm outperformed in terms of SA and effect size on any dataset (Table 8), pairwise comparison of RF, SVR, DeepNet and bagging versions of these algorithms is not added.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. SA and effect size evaluation of Proposed Mdb and MoWE.

https://doi.org/10.1371/journal.pone.0300296.t008

Results of Wilcoxon comparison for Tier-2 is reported in. As discussed earlier, Wilcoxon test is performed on significant level α = 0.05 for statistically evaluating all model. However, models showing significance at α = 0.1 are denoted by (*). Analyzing the statistical results of Tier-1(Table 9) it is clear that RF_Mdb is showing significance at 0.05, compared to RF and RF_Bagging in all datasets, expect for Desharnais, where RF_Mdb and solo RF are significantly different but only at α = 0.1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Wilcoxon test results comparing Mdb schemes with solo learners and normal bagging algorithm.

https://doi.org/10.1371/journal.pone.0300296.t009

Similarly, in Albrecht dataset, SVR_Mdb and DeepNet_Mdb models are showing significance over normal bagging at α = 0.1. In all other models, Mdb schemes statistically outperformed (at α = 0.05) compared to their solo learners and normal bagging, hence rejecting the null hypothesis (Eq 25).

In statistical analysis of Tier-2 (Table 10), it is important to note that MoWE performed significantly well (at α = 0.05) compared to all solo leaners and normal bagging algorithms for all datasets. However, for Mdb schemes in some datasets, MoWE is showing lower significance. Particularly, comparing MoWE and SVR_Mdb, lower statistical significance is observed (i.e., at α = 0.1) for Cocomo81, Finnish, Kitchenham and Maxwell. It is worth mentioning that, MoWE showing difference at α = 0.05 compared to all models in Albrecht, Desharnais and Miyazaki datasets. In comparison with other heterogenous ensembles, significant improvement of MoWE is observed for all models in all datasets. For Cocomo81, Majority voting and Weighed ensemble are showing significance on only on α = 0.1. To conclude statistical significance exits in performance for proposed Mdb and MoWE technique either at α = 0.05 or α = 0.1. Hence performance superiority of proposed scheme is visible in all datasets, This rejects the null hypothesis stating that there is no difference in performance of proposed technique and other comparing models. Our experimental results clearly show that there exists statistical difference between proposed technique and other models either on significance level α = 0.05 or 0.1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Wilcoxon test results comparing MoWE with other models.

https://doi.org/10.1371/journal.pone.0300296.t010

A comparative performance is also performed in this study, in which proposed model is compared with previous SDEE studies employing state-of-the-art ensemble learning algorithms, shown in. Table contains ensemble techniques, performance metrices, working dataset and performance achieved by the study under-consideration. Also, performance improvement achieved by our proposed MoMdbWE is listed in Table 11. It is clear from the table that proposed scheme showed improved results compared to other ensemble techniques in compared performance metrices.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Comparative evaluation of past EEE work with proposed MoMdbWE scheme.

https://doi.org/10.1371/journal.pone.0300296.t011

RQ1: Does optimization included in both domains (hyperparameter optimization and optimal weights assignment) improve estimation performance?

Answer: As

Answer: As discussed earlier, FFA among all optimization algorithms gives better MAE on all datasets (Appendix A: Table 12 in S1 File). However, even performance of models in Appendix A: Table 12 in S1 File, is not higher than Mdb and MoWE (i.e., models involving FFA-optimized hyperparameters and weights simultaneously) represented in Appendix B in S1 File. As we can see, MAE of all models in Appendix B in S1 File (where both hyperparameter and weight optimization is involved simultaneously) is lower than MAE of all models in Appendix A: Table 12 in S1 File (where only hyperparameter optimization is involved). This reflects, optimization enabled in both aspects i.e., hyperparameters as well as ensemble weights contribute to better performance.

For further highlighting the importance of optimization in both domains, results of Mdb schemes and MoWE can also be compared. As it is evident from the results (Table 8 and Figs 16–23), proposed MoWE technique (including both optimization aspects simultaneously) performed better, compared to Mdb schemes (including hyperparameter optimization only).

It is important to note, all ensembles created without hyperparameter optimization (i.e., Bagging, Stacking, Majority voting, Weighted ensemble) failed to achieve good performance. This also reflects importance of hyperparameter optimization before constructing ensemble. To conclude, models enabled with optimization in both domains achieved better effort prediction in all contexts.

1. III. Problem: For ML hyperparameter optimization, no consideration is made on defining search-space selection criteria.

Solution: A well-defined, error-based search space division mechanism is proposed in the study. Creating Sub_SPs from large initial SP and finding best optimizing solution from each Sub_SP helped in improving performance in two regards:

1. Facilitates in getting multiple diverse models, eventually enabling diversity in ensemble.
2. Providing the chance to include models with slightly higher level of error, so that ensemble members remain uncorrelated and do not contrast the performance of each other (as discussed above).

From this perspective, answer to the RQ2 is given as follows:

RQ2: Does performing search space division endorse same results as utilizing entire search space?

Answer: Utilizing entire search space for creating estimation model is not giving results as good as the models aided with search space division. Appendix A: Table 13 in S1 File listed MAE performance of solo learners, with hyperparameter optimization on entire search space. As it is clear from the table, models are showing visibly larger MAE than all Mdb schemes (Appendix B in S1 File), in which search space division is applied.

1. IV. Problem: Investigating the use of both ensemble mechanisms (Homogeneous and Heterogenous) simultaneously is overlooked.

Solution: This study effectively established the effort estimation framework by integrating both types of ensembles. Mdb schemes (homogeneous ensembles), surpassed their base learner as well as normal bagging algorithm. All these accurate Mdb schemes led to further improved MoWE ensemble (Table 8). Wilcoxon statistical test performed on models signifies the same conclusion (Tables 9 and 10).

From this conclusion, RQ3 is address as follows:

RQ3: Does integration of Homogeneous and Heterogenous ensemble tend to improve the performance or Homogeneous/Heterogenous ensemble alone can give good performance?

Answer: Homogeneous ensemble (Mdbs) alone can give good performance, only when compared to their base learners and normal bagging. When Mdb schemes are combined in form of heterogenous ensemble (MoWE), prediction error reduces further. For MoWE, evaluation metrices gave improved results compared to all Mdb schemes, in all datasets (Appendix B in S1 File).

It is important to mention, on some datasets, performance attained by Mdb schemes is not significantly different (at α = 0.05) than their MoWE ensemble, (Section 6.2: Result of statistical evaluation). However, at significance level α = 0.1, MoWE models of all datasets performed significantly higher than their Mdb schemes. Hence, we can conclude, homogeneous ensembles integrated with heterogenous ensemble tend to improve performance of estimation, while in some cases, difference in performance of homogeneous and heterogenous ensemble is not significant on higher α.