Estimating the generalization error by leave-one-out cross validation.

Leave-one-out cross-validation is a variant of cross-validation in which the number of folds is equal to the number of instances in the dataset [26]. The candidate model’s prediction error is calculated for each value in the observed dataset using all other values as a training set and the chosen value as a single-item test set. The leave-one-out error R_loo, which is meant to be an “almost” unbiased (in the sense of [27]) estimate of the generalization error G_error [28], was introduced in various contexts in the late 1960s, including those discussed in [27, 29–31]. For the case of the empirical risk presented in Eq (6), the leave-one-out risk estimator (LOOE) can be defined as: (11) where fⁱ(x_i) denotes the model f calibrated on the training set obtained by removing the point (x_i, y_i) from D_l, an then evaluated at x_i.

The asymptotic capabilities of LOOE as a proxy for the generalization error have been studied previously [32], and it has been shown both theoretically and empirically that the leave-one-out error, whenever the learning algorithms are stable in the sense of [28, 33], is a proper proxy. The present paper considers the LOOE a true proxy for generalization error estimates. For a squared loss, the generalization error for a given model i can be expressed as: (12) where y_k is the true response at a given point x_k and is the value predicted at x_k by using the model approximated by the parameterized function g_n and calibrated from all the DoE points in except the data pair (x_k, y_k).

The previous literature covers the basic concepts of the learning process from a statistical point of view. In the next subsections, the idea of combining several candidate functions (surrogate models) to build the parameterized function g_n shall be discussed.

Polynomial Chaos Expansions—PCE.

Let an input-output model be represented by a function y = M(x), where x ∈ Rⁿ, y ∈ R^m, and n is the number of input quantities and m the number of outputs. For simplicity, the m = 1 case will be considered in the following description. Both x and y can be described as random variables X = (X₁, X₂, X₃, …, X_n) and Y, respectively, due to the uncertainties in the input variables and their propagation to the output [35–37]. For a specific value of x, a deterministic algorithm normally computes the corresponding response y. The joint pdf of the random vector X is denoted by f_X. Assuming that the input random variables X_i are independent, then f_X is a multiplication of the marginal probabilities, . A polynomial Chaos Expansion (PCE) approximates the response Y as a linear combination of orthonormal polynomials [2].

In a full PCE, the number of expansion factors NP depends on the polynomial order p and the number of random input parameters n, being given by . Also, in the context of multivariate basis of polynomials, they can be constructed as tensor products of univariate orthonormal polynomials which are closely related to the pdfs [36]. For example, in the case of uniform distributions, Legendre polynomials are the ideal basis function. For Normal random variables, on the other hand, Hermite polynomials are of interest [36].

After defining the univariate polynomial basis, regression analysis can compute the PCE coefficients in a non-intrusive and cost-effective manner. The regularized least squares optimization involved in the regression procedure can be solve by different methods, each of which will assume extra penalizations or constraints to the optimization problem. Naturally, different choices will provide different polynomial coefficients and, therefore, different surrogate models.

Nonparametric hypothesis tests.

A hypothesis test for equality of distribution can be used as a first step in determining how different random variables are when each sample is compared. Generally speaking, these tests will create a statistical framework to examine the degree to which two or more samples differ from two or more random variables.

Let X₁ and X₂ be the continuous random variables underlying two populations of interest, and F₁ and F₂ be their respective distribution functions. The general system of hypotheses when one compares two populations is: (17) where means that and means that with Pr(A) > 0.

Because they do not establish any assumptions about the distribution of each random variable being compared, nonparametric tests are especially appealing when the types of random variables studied are unknown. Two of theses tests are explored in detail.

• Kolmogorov-Smirnov Test

The Kolmogorov–Smirnov test may test whether two underlying one-dimensional probability distributions, with densities F₁(x) and F₂(x), respectively, differ. In this case, the Kolmogorov–Smirnov statistic KS is the supremum of the absolute difference between densities [50]. One drawback of this test is that it may be ineffective for the equality of distribution assessment. Some claim that the Cucconi test can be more effective for that [51].

• Cucconi test

For the Cucconi test: (18) where G(⋅) is the distribution function for a continuous variable with location 0 and scale 1, μ_i is the location of population i, and σ_i is its scale. Let observations be random samples from population i. For the location-scale problem, Cucconi [52] proposed a rank test based on: (19) where (20) and (21) n = n₁ + n₂, W_ji denotes the rank of X_ji in the pooled sample and ρ = 2(n² − 4)/((2n + 1)(8n + 11)) − 1. Under H₀, and Var(U) = VAR(V) = 1 [51]. The p-values associated with such tests can be calculated by bootstrap techniques [51].

Directed Bubble Hierarchical Tree clustering.

In a completely unsupervised and deterministic manner, hierarchical clustering algorithms enable discovery of relationships and structures within datasets. The Directed Bubble Hierarchical Tree (DBHT) uses the topological property of the PMFG (Planar Maximally Filtered Graph) to find the clustering [53].

The PMFG is a generalization of the Minimum Spanning Tree (MST) included in the PMFG as a subgraph. It is created using the same steps as the MST, except that the weaker planarity condition is used instead of the non-loop condition (i.e., each added link must not cut a pre-existent link). The PMFG can retain more links and information than the MST because of this less severe topological constraint. It can be demonstrated, in particular, that each PMFG contains precisely 3(N − 2) links.

The DBHT uses the topological structure of the PMFG to identify a clustering partition for each node in it [53]. The traditional agglomerative clustering process is then used to obtain an entire hierarchical structure (dendrogram) between and within clusters.

Normally linkage algorithms analyze the sorted list of distances D_i,j between nodes i and j and construct the dendrogram by compiling subsets of candidate models with the smallest distances; the clustering is then obtained from the dendrogram after pre-selecting the “number of clusters” desired. Instead, the DBHT reverses this process: first, the clusters are identified using topological analysis of the planar graph, and then the hierarchy is built between and within the clusters. Therefore, the distinction between the traditional agglomerative clustering process and DBHT involves the type of information used and the methodology.

In a recent study [54], researchers quantified the amount of information on stock return correlations filtered by various hierarchical clustering methods. Their findings demonstrate that the DBHT can perform better than other methods by retrieving more data with fewer clusters. Additionally, they demonstrate how, depending on the clustering method, the economic information is hidden at various levels of the hierarchical structures.

The DBHT algorithm considers a dissimilarity (distance) and a similarity matrix. In essence, both matrices are required because the PMFG is a weighted graph, and weights are typically similarity measures for edges (a larger weight of an edge corresponds to a stronger similarity between the connected nodes). The edges are also associated with a distance or, more generally, a non-negative dissimilarity measure [53]. Therefore, it is important to present some candidates for these matrices.

• Pearson correlation distance

Considering the works of [53, 54], an interesting distance metric can be defined in terms of Pearson’s correlation coefficients ρ_ij between pairs of assets. This way, a m × m distance matrix, whose elements are , can be defined.

• Kendall’s τ correlation distance

Similarly to the Pearson correlation distance, it is possible to consider Kendall’s τ rank correlation to build a distance metric τ_ij between pairs of assets. Literature reveals that this correlation metric better captures co-movements, while compared to Pearson correlation, especially in the realm of clustering financial time series [55].This way, a m × m distance matrix, whose elements are , can be defined.

• Spearman’s ρ correlation distance

Literature also indicates that Spearman’s ρ, ρ_S, has a good performance when used to cluster financial assets’ returns [55] and, therefore, can be used to build a distance metric ρ_s,ij between pairs of assets. This way, a m × m distance matrix, whose elements are , can be defined.

• Generalization error distance

It is possible to define the distance between two algorithms i and j by computing the absolute difference between the generalization error of each one. Thus, let Γ(i, j) denote this difference, then: (22)

• Relative Kullback–Leibler divergence as a distance

Consider the problem of comparing two approximate distributions, V and S, using a third reference pdf, P. Using the KL divergence to calculate the absolute value of the difference between the KL divergences of both V and S concerning the same function P is possible. Thus [56]: (23) where D(P||V) is the KL divergence between P and V, defined as [57]: (24) in which p(x) and v(x) are the densities of P and V, respectively.

Thus, from Eqs (23) and (24): (25) (26)

An interesting choice for the reference pdf is the Dirac delta pseudo-distribution. Such pseudo-distribution can be modeled as the limit of a Normal random variable with mean and variance tending to zero. Thus, in the case where the reference random variable is a Dirac delta centered at μ_ref, it can be seen that the generalized metric in Eq (25) becomes: (27)

Finally, for a zero-centered Dirac delta pseudo-distribution, it can be seen that the distance metric adopted could be: (28)

Eq (28) indicates that calculating the distance metric D_δ,0(V||S) reduces to the calculation of the density ratio between both distributions at the origin. The estimated density ratio function can be used in many applications, such as the inlier-based outlier detection [58] and covariate shift adaptation [59]. Other useful applications for density ratio estimation were summarized in [60].

Some python implementations of the RuLSIF (Relative unconstrained Least-Squares Importance Fitting) method can estimate the alpha-relative density ratio by minimizing the squared loss between the true and estimated alpha-relative ratios. This method is detailed in [58, 61].

Literature indicates that the density ratio problem can be approached by multidimensional densities via k-nearest-neighbor distances [62], by a probabilistic classification [63] or even an infinitesimal classification [64]. A general discussion can be found in [63].

• Generalization error similarity

For this case, it is possible to express the similarity of two datasets based on their Generalization error distance defined in Eq (22) as . Now that a brief review of the literature has been presented, the Material and Methods considered in the present paper shall be presented

Surrogate modeling.

Given that the main benefit of using surrogate models is to significantly reduce the time needed to perform in-depth analyses of a particular problem of interest, techniques that can produce predictions quickly after the surrogate has been trained are of special interest [34]. In the present paper, to show the applicability of the HPOSS, two classes of surrogates were considered: Machine Learning and Polynomial Chaos Expansion ones.

• Machine Learning Algorithms

We have picked a few of the most popular machine-learning techniques for our study. Using the Python scikit-learn library, similarly to [65], the authors gathered some of the available regressor-type estimators by importing the method “all_estimators” from “sklearn.utils” and then discarding some regressors which were not considered of interest. Such a pre-selection process was based on the authors’ previous experience calibrating the sklearn models, especially considering the time each algorithm takes to perform the learning process. This way, in order to enforce diversity, a total of forty algorithms gathered into ten model classes were considered, namely [66]: ensemble (which try to enhance the overall generalizability and robustness while compared to a single predictor by merging the forecasts of multiple predictors generated using a particular learning method—‘AdaBoostRegressor’, ‘BaggingRegressor’, ‘ExtraTreesRegressor’, ‘GradientBoostingRegressor’, ‘HistGradientBoostingRegressor’, ‘RandomForestRegressor’); linear_model (operate under the assumption that the desired output will be a result of combining the input features in a linear manner—‘BayesianRidge’, ‘ElasticNet’, ‘ElasticNetCV’, ‘GammaRegressor’, ‘HuberRegressor’, ‘Lars’, ‘LarsCV’, ‘Lasso’, ‘LassoCV’, ‘LassoLars’, ‘LassoLarsCV’, ‘LassoLarsIC’, ‘LinearRegression’, ‘LinearSVR’, ‘OrthogonalMatchingPursuit’, ‘OrthogonalMatchingPursuitCV’, ‘PassiveAggressiveRegressor’, ‘PoissonRegressor’, ‘QuantileRegressor’,‘RANSACRegressor’, ‘Ridge’, ‘RidgeCV’, ‘SGDRegressor’, ‘TweedieRegressor’); tree (acquire simple decision rules from the provided input data to construct a model capable of predicting the target variable’s value—‘DecisionTreeRegressor’, ‘ExtraTreeRegressor’); dummy (make predictions using uncomplicated rules, serving as basic benchmarks for comparison with other regression models—‘DummyRegressor’); gaussian_process (also known as kriging algorithms—‘GaussianProcessRegressor’); neighbors (assign a numerical value to a given point by calculating the mean(or other summarizing function) of the values from its closest neighbors—‘KNeighborsRegressor’); kernel_ridge (combines ridge with the kernel trick—‘KernelRidge’); neural_network (simple multi-layer perceptron regressor—‘MLPRegressor’); svm (support vector machine based models—‘NuSVR’, ‘SVR’) and compose (meta-estimator to regress on a transformed target—‘TransformedTargetRegressor’); For further details on each method, we refer the reader to the key references cited in [66], and also the work of [34].

When dealing with a vast list of machine learning models, it becomes evident that hyperparameter tuning may not be imperative [67]. This is due to the fact that when weak predictors are aggregated and combined, they possess the capability to produce robust estimates. By harnessing the collective knowledge of multiple models, even those with relatively modest individual predictive capabilities, the resultant ensemble is able to offset individual shortcomings and attain more dependable and precise predictions. As a result, the emphasis transitions from fine-tuning hyperparameters for individual models to the creation and combination of diverse models, thereby enhancing the ensemble’s overall performance and capacity to generalize. This is precisely the focus of the present paper.

• Polynomial Chaos Expansion

Many methods are available to generate the univariate polynomial basis for particular pdfs . One is the so-called three terms recurrence method, which is generated using Stieltjes [68] and Golub-Welsch method [69]. This is a considerably stable method, but do not work on dependent distributions. Other approaches include the one discussed in [36], known as the Askey scheme. In the present paper, PCE were fitted to the data using the three terms recurrence method implemented in the chaospy [70] algorithm.

• Making predictions with surrogate models

When a candidate surrogate model is calibrated, one wants to perform inferences for new input values. Point estimates are not the best alternative in this case since there are some errors due to sampling and the modeling procedure itself, which must be accounted for during the inferencing process.

If a given surrogate model is chosen, its analytical formulation is known and equal to r(.). Let’s consider that the parameters have been obtained from a given set of training samples. Then, it is possible to predict the expected value of the response variable given a new set of inputs x₀, as r(x₀).

Normally, it is assumed that the relation between the calibrated surrogate model and the output response is subjected to an additive zero-mean noise, which implies that the final estimation of the expected value of the output is exclusively dependent on the function r(.) and on the estimated parameters . This is a point estimate which may be misleading if solely analyzed. Interval estimations bring much more information, especially in risk analysis assessments, where the most extreme situation are sought.

It is possible, then, to consider the confidence interval of the expected value r(x₀). In this case, one is interested in predicting the mean response, the average response value for a given input x₀. To calculate such a confidence interval, one needs to account for the variance in , which may arise from the training sample sampling process.

On the other hand, if one is interested in predicting the specific output for a particular input x₀, then, to estimate the variance of such prediction, it is necessary to consider not only the uncertainty about , as in the case of the confidence of expectation interval but also the uncertainty related to our actual prediction. In this case, one is not interested in the mean response (expected value) but in a specific future value.

Overall, the major difference between the confidence of expectation and prediction intervals is that the former accounts for uncertainty in the sampling of the training samples, while the latter accounts for both this sampling issue as well as for the uncertainty of the model prediction itself (i.e., considers the variability of the possible outputs around the predicted mean). In general, prediction intervals are more suited for applications where one is not simply worried about the mean response. Calculating the prediction interval can be achieved by a bootstrap procedure.

A non-parametric estimation technique called Bootstrap was developed by Efron [71, 72] and enables one to calculate the confidence interval (in the statistical sense) for a given statistic of interest. The Bootstrap method is a statistical inference technique that relies only on currently available data (sample). One of its main features is the method’s lack of dependence on any consideration of the relevant random variables.

The Bootstrap technique can estimate the sampling distribution of a particular statistic (for example, the sample’s mean and variance) by considering that the sample is representative of the population from which the latter has been collected and that the observations are independent and identically distributed.

Thus, to calculate the 1 − ι confidence level prediction interval at a given value of the input value x₀, one can use the following bootstrap routine (simplified from the one presented in [73]):

• From the training dataset {W₁, W₂, …, W_k}, draw a new sample of size k with replacement. Each W_i (or , by consequence) is a pair with an input variable x_i (or ) and an output value y_i (or ).
• Train the surrogate model r(.) with the dataset to obtain r_j(.).
• Draw one element at random, e^j,*, from the errors sample such that .
• Calculate the estimated output value as the sum r_j(x₀) + e^j,*.
• Repeat B times the last four steps to obtain a bootstrap sample of outputs {r₁(x₀) + e¹,*, r₂(x₀) + e²,*, …, r_B(x₀) + e^B,*}.
• From the sample {r₁(x₀) + e¹,*, r₂(x₀) + e²,*, …, r_B(x₀) + e^B,*}, calculate the confidence interval for the mean value for a 1 − ι confidence level.

After the surrogates have been trained, it is interesting to discuss further how to cluster the candidate surrogate models, which is the topic of the next sub-subsection.

Clustering.

Clustering candidate models corresponds to clustering their performances during the prediction tasks. As indicated in the present paper, the performance of each model will be assessed based on the random variables ϵ_i, whose samples are obtained as the leave-one-out residues. Thus, studying how to group datasets according to a given clustering hypothesis is important.

Applying a clustering algorithm to gather similar surrogate models makes it possible to select m models worth stacking. The selected models are the ones that belong to the cluster that contains the method with minimum observed generalization error (i.e., minimize Eq (12)), which would be the chosen model in a winner-takes-all approach.

In the present paper, the DBHT is used to explore how candidate models can be clustered according to their performance (generalization errors) and analytical similarity (Pearson, Kendall’s τ and Spearman’s ρ correlations of leave-one-out residues). This allows one to filter, from a possibly large set of candidate models, the ones who could bring better quality to the stacking strategy. This choice of similarity and distance metrics is based on the fact that, for the DBHT algorithm, the similarity defines the bubbles (general structure of the graph), which are then hierarchically organized by the distance metric (inter and intra structure of the clusters). By using the generalization error similarity and choosing the cluster containing the minimum observed generalization error, one chooses the models with lower observed errors. On the other hand, using the Pearson correlation distance, the internal hierarchy of the bubbles is such that similar models (same analytical capabilities) are clustered together (because their residues are highly correlated).

The python package Riskfolio-Lib [74] presents a numerical implementation of the DBHT method, which shall be slightly modified (adapted to produce different plots) and used in the present paper.

The procedure proposed in [75], where the ratio of densities is calculated based on the ratios of their empirical cumulative distributions, is considered in the present paper to calculate the distance in Eq (28). In short, the empirical cumulative distribution is built by combining simple Heaviside step functions and piece-wise linear approximations. Then, such function is numerically differentiated using a finite difference approach, therefore obtaining an estimate of each density function involved [75].

An alternative expression, which considers that the random variables involved are approximately Normal, can be obtained as [56]: (29) where μ_s, σ_s and μ_v, σ_v are the means and standard deviations of the random variables S and V, respectively. Then, the second crucial step is considered: a stacking strategy is applied to obtain the weights w_i, i = 1, …, m of the selected models.

Stacking.

In the HPOSS context, the stacking strategy to be considered is the HPR financial approach. This approach is considered with the same distance metric used in the DBHT algorithm, i.e., Pearson correlation distance. Such choice ensures that similar models cannot dominate the weight distribution (avoid concentration), since the weights are calculated based on the global and local hierarchy instead of single model performance. For example, if two models are withing the same cluster, they will share a certain portion of total weight (ascribed to the whole cluster) instead of dominating the weight distribution. This privileges analytical diversity, as desired.

To assess the suitability of the HPOSS, in the study case, just two previously published stacking strategies are explored, both because they have shown to be superior to other simpler alternatives and because they explore the concept of generalization error minimization, which is discussed under different premises in the present paper. The heuristic formulation proposed by Goel et al. in [18] and the optimized weight factor of Acar and Rais-Rohani in [19] are the known stacking strategies selected.

• Heuristic proposed by Goel et al.

The strategy of selecting weights proposed by Goel et al. [18] is formulated as follows: (30) where m is the number of candidate models, y_k is the true response at a given point x_k, is the prediction from the i^th surrogate model calibrated from all the DoE except the data pair (x_k, y_k) and N is the size of the DoE. The parameters ω and β should be specified beforehand. For instance, ω = 0.05 and β = −1 is used in the work of [18]. A study of the effect of those parameters has also been performed [18].

• Optimization problem proposed by Acar & Rais-Rohani

The weight factors are here solutions for the minimization of a global error. The influence of the error metric choice is studied in [76]. In their original paper, Acar & Rais-Rohani [19] selected the generalized mean square error concerning the ensemble of models, . This metric is defined by: (31) where y_k is the true response at a given point x_k and is the value predicted by using a stacked model calibrated from all the DoE points except the data pair (x_k, y_k). The weight factors are the solutions to the following optimization problem: (32)

By changing the optimization problem in Eq (32) into a matrix form, Viana et al. [13] obtained an explicit solution. On the other hand, their solution is based on building a matrix based on the average cross-errors of each model obtained by leave-one-out cross-validation.

The problem in Eq (32) can be solved as it is, in an unbounded manner, and also can be changed to a bounded alternative, where the weights are considered to lie within the (0, 1) interval. Both approaches (bounded and unbounded) will be considered in the present paper.

The stacking strategies presented by Goel et al. and by Acar & Rais-Rohani will be used as benchmarks to test the capabilities of the proposed new stacking strategy. Other novel heuristic approaches will be also proposed in the present paper.

In summary, for the HPOSS, low generalization error is obtained by using the DBHT algorithm with a generalization error similarity. High model diversity is achieved by combining the DBHT algorithm and the HPR stacking strategy, both with the Pearson correlation distance metric. One could argue that it would be possible to directly calculate the weights using the HPR rationale on the global/local hierarchy obtained by the DBHT algorithm. This would be possible, but in the present paper, we applied such algorithms sequentially (instead of in a nested manner). To assess the suitability of the HPOSS, a few new general stacking alternatives that encompass statistical ideas will be proposed.

Direct optimization stacking approach.

In our interpretation, each of the m candidate models represents a risky asset whose returns are given as ϵ_i, for i = 1, …, m. Therefore, we aim to find the weights w_i, for i = 1, …, m, such that the generalization error of the stacked model is minimized. In the present sub-subsection novel analytical and numerical solutions are presented for some optimization stacking approaches. Besides, novel heuristic approaches are described. Overall, using all theses alternatives is of interest to show how the adequacy and robustness of the methodology hereby proposed.

• Unbounded weights

Theorem 1 indicates that this problem is reduced to minimizing the sum of the variance and the squared expected value of the random variable . Thus, let w be a m × 1 column vector whose components are the weights of each model. Also, let μ be a m × 1 column vector whose components are the expected values of each candidate model. By denoting Σ as the covariance matrix of the random variables ϵ_i, i = 1, …, m, then, the minimization problem can be represented as: (38) where e is a m × 1 column vector whose entries are all equal to one.

Let us consider that the weights w minimize the problem in Eq (38) are w_P, such that . Thus, this same vector of weights will also minimize the following problem: (39)

On the other hand, MPT brings an interesting insight into the optimization problem presented in Eq (39). This results in the determination of the values of weights of portfolios that belong to the so-called Efficient Frontier of assets selection.

The optimization problem in Eq (39) can be analytically solved by using Lagrange multipliers [79], leading to the variance of the portfolio being expressed in terms of μ_P as: (40) where it is assumed that Σ is non-singular such that its inverse, Σ⁻¹, exists and: (41)

Now, the next step is to combine Eqs (33) and (40) to solve the optimization problem in Eq (38). Thus: (42)

To obtain real-valued solutions to the quadratic Eq (42), it suffices to observe that: (43) which leads to: (44)

Thus, the minimum generalization error of the stacked model is obtained when Eq (44) is equality. Also, for this specific value, the generalization error circle only touches the Efficient Frontier once, precisely at the point where: (45)

From the values obtained in Eq (45), it is possible to explicitly obtain the m × 1 vector of optimum weights w_opt by defining a m × 2 matrix K = [μ, e], a 2 × 1 vector ω = [μ_P, 1]^T, a 2 × 2 matrix A = K^TΣ⁻¹K and: (46)

It is worth noticing that the optimization problem in Eq (38) is a portfolio management reinterpretation of the optimization problem presented in Eq (32) and explored in [19]. Therefore, the solution in Eq (46) is nothing but an explicit exact solution to Eq (32), given differently and more directly than the one presented in [13].

• Graphical interpretation

The graphical interpretation of the present optimization solution is presented in Fig 2. Such interpretation directly results from the mathematical rationale behind the formulas presented.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Markowitz frontier, minimum generalization error asset, and portfolio.

https://doi.org/10.1371/journal.pone.0290331.g002

First and foremost, Eqs (38) and (39) indicate that minimizing the generalization error of the stacked model implies that the selected portfolio belongs to the efficient frontier. Secondly, Theorem 1 states that the generalization error is nothing but the squared radius of a circle whose origin is at (0, 0) and touches the point (σ_P, μ_P), where μ_P and σ_P are the bias and squared root variance of the stacked model generalization error random variable.

This way, finding the portfolio with the least generalization error translates into finding the intersection of the smallest-radius circle centered at (0, 0) which touches the efficient frontier line. It is interesting to notice from Fig 2 that the winner-takes-all approach, where the model with smallest generalization error is chosen, can provide considerably higher generalization errors when compared to a stacked model.

Optimization with positive-defined weights.

An important observation of Breiman [39] is that, in general, imposing that the weights are non-negative, i.e., w_i ≥ 0, ∀i, provides predictors which, almost always, have lower prediction error than the single predictor having lowest cross-validation error.

Also, from a Bayesian perspective, Bayesian models can be weighted by their marginal likelihood. This is known as Bayesian Model Averaging [80–82]. This rationale could also be used for surrogate models, as described in [12]. This implies that the weights represent probabilities, so they should be non-negative. The marginal likelihood is extremely sensitive to the specification of the prior, whereas parameter estimation is not, and computing the marginal likelihood is typically a difficult process, so while this is theoretically appealing, it is troublesome in practice [83]. Therefore, this Bayesian technique shall not be explored in the present paper.

The new optimization problem would be: (47) (48)

Such a problem cannot be directly solved using Lagrange’s multipliers. Markowitz created the Critical Line Algorithm (CLA), a quadratic optimization technique for situations involving inequality-constrained portfolio optimization. This algorithm stands out because it cleverly gets around the Karush-Kuhn-Tucker requirements and ensures that the precise answer is obtained after a known amount of iterations [42]. This algorithm’s description and open-source implementation can be found in the literature [45].

The optimization procedure to solve Eq (47) consists of two steps: first, the minimum variance portfolio with bounded weights is obtained using the CLA algorithm. Then, such weights are used as initial values for the optimization procedure in Eq (47), which can be solved using the Sequential Least Squares Programming (SLSQP) algorithm of scipy [84]. This two-step approach is important to make the optimization algorithm look for weights around the tip of the inequality-constrained Markowitz frontier. If the bias of the minimum variance portfolio is small, there is a good chance that this will be the portfolio which also minimizes the problem in Eq (47).

A special case to Eq (47) happens when the optimization procedure considers the constraint that the mean value of is zero. This problem is hereby called ZeroMinStdWeights and will be explored in the applications section.

Novel heuristic approaches.

As alternative optimization approaches, some heuristic procedures are introduced in the present paper. The rationale behind these approaches is to enforce some sort of regularization during the optimization process, which would provide more robust solutions.

Overall, the basic principle hereby considered is that residues of errors of the portfolio of models, obtained as , will be approximately distributed as a Normal random variable with zero mean and standard deviation given by . This can also be understood as an approximation supported by Lindeberg’s Central Limit Theorem, which only requires that the random variables being linearly combined have finite variance, satisfy Lindeberg’s condition, and be independent. Lindeberg’s condition indicates that none of the random variables have a disproportional relevance to the calculation of the variance of the portfolio [85].

This way, the optimization problem to be solved is: (49) (50) where U(w, ϵ₁, …, ϵ_m) is a heuristic utility function on the weights and the residues. Several of those utility functions are discussed subsequently.

• MinADWeights

For this case, the utility function is: (51) which is nothing but the harmonic mean of the standard deviation of the portfolio (σ_stacked) and the modified Anderson-Darling statistic [86] calculated for the normalized portfolio residues (AD_stacked). This modified statistic considers that both mean and variance are unknown. Mathematically, both the mean of the portfolio and its variance are estimated from the sampled residues, , Φ(z) is the cdf of a standard normal random variable and: (52)

• NormWeights

In this case, the utility function is simply U(w, ϵ₁, …, ϵ_m) = σ_stacked, where σ_stacked has been obtained as maximum likelihood estimate for the variance of a Normal random variable with zero-mean fitted to .

• MaxLWeights

In this case, the utility function is the negative log-likelihood of the samples from being distributed as a Normal random variable with zero-mean and standard deviation σ_stacked, which is estimated from the samples of .

• MinKLWeights

In this case, the utility function is the KL divergence between two normal random variables P and Q, where P has mean zero and standard deviation σ_stacked and Q has mean μ_stacked and standard deviation σ_stacked. All these moments are estimated from the samples of .

Model calibration.

From all the possible regressors in scikit-learn [66], the ones cited in the Materials and Methods section were considered. As discussed, the algorithm ‘GaussianProcessRegressor’ is a kriging implementation, and no manual hyperparameter tuning was performed for any of the algorithms.

Polynomial Chaos Expansions were fitted to the data using the chaospy [70] algorithm. In this case, a point collocation fitting approach was considered. Seven types of multivariate linear regression models, available in scikit-learn, were used to obtain the polynomial coefficients to solve the least-squares problem (find the coefficients of the PCE), namely: “least squares”, “elastic net”, “lasso”, “lasso lars”, “lars”,“orthogonal matching pursuit” and “ridge”. The maximum order of the polynomials considered was such that the number of unknown coefficients did not exceed the number of training samples. In the case with four input variables and 80 training samples, at most, polynomials of order 4 were fitted. Besides, three base random variables were selected for the expansions: Uniforms in the range (−1, 1), standard Normals, and the Uniforms in the physical ranges mentioned (hereby referred to as Real expansion, in the sense of the real random variable). For all the three types of variables, the three terms recurrence method implemented in chaospy was used. It is worth noticing that classical Legendre and Hermite polynomials were retrieved for Uniforms in the range (−1, 1) and standard Normals.

In all the cases, the naming convention for the PCE expansions was: PCE − XXX YYY − Z, where XXX denotes which type of random variable was considered (Unif = Uniform (−1, 1), Norm = standard Normal and Real = shifted Uniforms in the physical ranges); YYY denotes which algorithm was used to perform the multi-linear least squares regression and Z is the polynomial expansion order.

For the machine learning models, it is important to highlight that scaling of the input values was performed by using the function MinMaxScaler of the scikit-learn package, which scales and translates each feature individually such that it is within a given range on the training set (in our case, between zero and one).

Overall, a total of 124 models were trained using the dataset described: 40 Machine Learning models and 84 PCE-based models (3 types of random variables considered fitted using 7 least square algorithms and up to fourth order expansions, i.e., 3 × 7 × 4 = 84). Among those, 99 models could be fitted without convergence and other numerical issues.

Model selection.

For the DBHT algorithm, the similarity measure chosen was the generalization error similarity. The clusters obtained from running the DBHT algorithm with different distance metrics (Pearson, Kendall’ τ and Spearman’s ρ correlation distances, approximate ratio correlation distance and approximate normal correlation distance) were also evaluated, and the clusters obtained for the Pearson, Kendall’ τ and Spearman’s ρ correlation distances are presented in Figs 5–7. The “PCE Real Lasso Lars-4” model is highlighted since this would be chosen based on the minimum observed generalization error. The selected cluster is the one that contains this model. It is important to highlight that the optimal number of clusters was established by using the two-order difference to gap statistic [89] implemented in the python package Riskfolio-Lib [74].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Radial dendrogram obtained from DBHT algorithm with Pearson correlation distance and generalization error similarity.

https://doi.org/10.1371/journal.pone.0290331.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Radial dendrogram obtained from DBHT algorithm with Kendall’ τ correlation distance and generalization error similarity.

https://doi.org/10.1371/journal.pone.0290331.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Radial dendrogram obtained from DBHT algorithm with Spearman’s ρ correlation distance and generalization error similarity.

https://doi.org/10.1371/journal.pone.0290331.g007

The analysis of Figs 5–7 indicate that the DBHT algorithm with both Kendall’ τ and Spearman’s ρ correlation distance metrics provided the same clustering result (same models clustered together with the “PCE Real Lasso Lars-4” model). Moreover, with the use of Pearson correlation distance metric, the DBHT algorithm provided a bigger cluster which contains not only the models clustered by DBHT using the other distance metrics, but also extra models. This indicates that if the Pearson correlation is used, a looser clustering process is carried out, which ends up including more models in the final cluster.

In the present paper, we seek to use the DBHT algorithm to filter among a large number of candidate models. Therefore, while we intend to eliminate a good share of uncorrelated models, we still want to have a good number of algorithms in the final cluster. Therefore, we chose the Pearson correlation distance metric to carry out the study case. Depending on the specific application envisioned, the reader may, on the other hand, choose a tighter clustering scheme, by selecting either Kendall’ τ or Spearman’s ρ correlation distance metrics together with the DBHT algorithm.

By analyzing Fig 5, the combination of the Pearson correlation distance with the generalization error similarity provides a good combination of low generalization errors and highly correlated residue random variables. This can be visualized by noticing that, on average, the bubbles formed by the DBHT algorithm gather low generalization error models (similarity defines the bubbles and we chose the cluster which contains the model with minimum generalization error) and the internal hierarchy is defined by models with similar analytical capabilities (highly correlated leave-one-out residues). Similar analytical capabilities can be confirmed by the fact that, in general, the clusters obtained gathered similar types of PCEs and linear regression methods.

Fig 8 presents the selected models in the bias-square root of variance space to better visualize how each combination of metrics leads to different clustering scenarios. The full black dot in Fig 8 represents the “PCE Real Lasso Lars-4” model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Model selection for each clustering algorithm considered.

For color blindness accessibility, the legend’s first(second) row describes the first(second) column of the grid.

https://doi.org/10.1371/journal.pone.0290331.g008

The legend in Fig 8 represents the hypothesis test approaches (Cucconi and KS) as well as the DBHT algorithm with the generalization error similarity (SimGenErro) and different distance metrics: generalization error distance (GenErro), Pearson correlation distance (Pearson), approximate ratio distance (RatioApp) and Normal approximation ratio distance (KL).

By considering the DBHT algorithm with Pearson correlation distance and generalization error similarity, from the 99 fitted models, a subgroup of 23 was selected to be stacked. This represents a reduction of about 77% in the total number of models, representing a good filtering scheme which still preserves diversity.

Weights calculation.

A total of eleven methodologies were considered to calculate the stacking weights. All the optimization procedures were solved using the Sequential Least Squares Programming (SLSQP) algorithm of scipy [84]. Fig 9 presents the bounded problems’ resulting weights. For unbounded procedures, the weights ranged from -30000 to 30000 and are not represented in Fig 9 due to scale issues.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Results for all bounded weights stacking strategies.

https://doi.org/10.1371/journal.pone.0290331.g009

The analysis of Fig 9 reveals that except from the “MinADWeights”, “MaxLWeights”, “HPRWeights”, “IVWeights” and “RobustWeights—1”, all the other methodologies tended to assign comparable weights to all the candidate models.

In special, Fig 9 shows that “MinADWeights” present an unbalanced scheme where too much importance was given to the “PCE Real lasso lars-2” model. This illustrates how a pure optimization approach may lead to less interpretable weights. Fig 10, on the other hand, presents a heatmap for the methods whose maximum weights were lower than 0.3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Heatmap of weights.

https://doi.org/10.1371/journal.pone.0290331.g010

Figs 9 and 10 also indicate that the novel heuristic methods “NormWeights”, “ZeroMinStdWeights”, and “AcarOptWeights” all resulted into the simplest weighting scheme: equal weights for all the models. This indicates that the convergence of the optimization problem was either problematic (since the initial weights used as first guesses were precisely equal ones) or that the equal-weight portfolio is the optimal one. Optimizing the “AcarOptWeights” by coupling it to the CLA algorithm of Markowitz did not provide convergent results when using the CLA algorithm presented in [45]. It is possible that the highly correlated and, sometimes, linearly dependent ϵ_i prevented convergence. Overall, optimization approaches struggled to converge to meaningful results. We chose to keep such results to illustrate how pure optimization techniques can be problematic and to indicate that direct approaches, such as the HPR strategy, are of great interest in this matter, as the “optimal” solution is deterministic and explicit. In other words, the HPR strategy can always provide weights, regardless of a possible ill-conditioning of the covariance matrix.

On the other hand, “GoelWeights”, “IVWeights” and “MaxLWeights” gave too much credit to the model with the least observed generalization error. Also, the “RobustWeights—1” provided an interesting result, where several models were almost discarded from the analysis as low weights components. The type of robust optimization hereby considered privileges lower bias at the cost of higher variance. Besides, the “HPRWeights” were such that two ordinary algorithms (“NuSVR” and “PCE Norm least squares—2”) were given considerably larger weights, while some other algorithms were almost disregarded with comparably lower weights.

Probability of failure calculations.

The plots in Figs 11–15 present the prediction intervals for the probabilities of failure calculated for each type of input random variable and the corresponding performance of the individual algorithms and stacking strategies. The prediction interval is defined by its boundaries [p_int,upper, p_int,lower] and was obtained as the 95% bias-corrected and accelerated bootstrap confidence interval (BCa) of the mean probability of failure after a bootstrapping prediction assessment for 300 times. This number of bootstrap realizations was considered adequate after numerical experiments. The bootstrap confidence intervals for the prediction values were obtained by applying the bootstrap method from scipy.stats Python package [84].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Prediction intervals of probabilities of failure considering that the input variables are linearly scaled Beta(2,5), i.e., the center of mass of the distribution closer to the left end of the interval, according to their respective physical ranges.

https://doi.org/10.1371/journal.pone.0290331.g011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Prediction intervals of probabilities of failure considering that the input variables are linearly scaled Beta(2,2), i.e., the center of mass of the distribution at the center of the interval, according to their respective physical ranges.

https://doi.org/10.1371/journal.pone.0290331.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Prediction intervals of probabilities of failure considering that the input variables are linearly scaled Beta(5,2), i.e., the center of mass of the distribution closer to the right end of the interval, according to their respective physical ranges.

https://doi.org/10.1371/journal.pone.0290331.g013

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Prediction intervals of probabilities of failure considering that the input variables are linearly scaled Beta(0.5,0.5), i.e., two centers of mass for the distribution located at both the right and left ends of the interval, according to their respective physical ranges.

https://doi.org/10.1371/journal.pone.0290331.g014

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Prediction intervals of probabilities of failure considering that the input variables are linearly scaled Beta(1,1), i.e., uniform distribution over the interval, according to their respective physical ranges.

https://doi.org/10.1371/journal.pone.0290331.g015

To properly compare the plots, it was considered that a failure happens whenever ζ_max > ζ_0.1 = 786260.04 Pa, where P(ι > ζ_0.1) = 0.1 and ι is the random realization of the exact model. Also, whenever a model or stacking strategy provided a prediction interval that contained the correct value (which was chosen as 0.1), it was colored green.

To assess how each algorithm and staking strategy performed, the following metric was defined to quantify how far the true probability of failure value was from the confidence interval. Thus, let υ be the performance metric defined as: (54) where υ_a = 0 if the prediction interval contains the correct probability of failure and υ_a = min(|p_int,upper − 0.1|, |p_int,lower − 0.1|) otherwise. Also, υ_b = p_int,upper − p_int,lower.

In other words, υ_a quantifies how far the actual probability of failure is from the predicted interval by either checking if this value is inside the gap (thus υ_a = 0) or which is the smallest distance from the correct value to the boundaries of the confidence interval. Also, υ_b quantifies how wide is the confidence interval. Overall, the best models would provide prediction intervals that contained the actual value, and the gap itself should be narrow such that lower values of υ indicate better model performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. Comparison of mean performance metric for probabilities of failure calculations considering all the input variables.

https://doi.org/10.1371/journal.pone.0290331.g016

Fig 16 shows that the HPR stacking strategy has the lowest mean performance metric among all the stacking strategies compared. However, this indicates that the HPR strategy performs well even when the random input variables have completely unseen probability distributions, indicating its performance can be considered superior to other methods.

In general, the lower the value of υ, the better the model. On the other hand, stacking is the best possible model choice, not necessarily the result with the lowest error metric. This comes from the fact that, usually, it is not possible to assess which is the individual model which presents the lowest metric (in our example, the PCE Norm ridge-3) because we do not know the function we are trying to approximate but only a few samples from its evaluation. Therefore, stacking becomes the best possible choice to provide an estimate in this context, as we don’t know how well each model will mimic the objective function. Besides, it can be seen that the best overall model (which has the lowest error metric) does not even have the lowest LOO generalization error. There is, likewise, no hint that allows us to choose it. In other words, it is impossible to pin down the best model beforehand, so stacking is necessary.

The study case carried out revealed some interesting aspects of the stacking problem, which follow:

• Intervals lengths:

Consistently, the prediction intervals are narrower for an ensemble of models than for single models. This is a known and expected result, as stacking models can be viewed as a variance reduction technique.

• PCE expansion quality:

Overall, custom PCE expansions for the same input random variables (even if they are slight modifications of traditional random variables with known closed-form PCEs) consistently perform worse than the conventional expansions (Uniforms, Normals).

Stacking strategies that ended up assigning higher weights to these custom (“Real”) expansion models got a severe performance hit. Also, except for the “MinADWeights” ensemble method, all individual custom (“Real”) expansions performed worse than the ensemble of models.

• Overall performance of surrogate models:

Very few algorithms provided prediction intervals that contained the actual probability of failure value. If a winner-takes-all approach had been considered, the algorithm with the lowest observed generalization error (PCE-Real lasso lars-4) would have been selected. On the other hand, such a model had a poor out-of-sample performance overall, which highlights why using ensembles is a good idea.

• Overall performance of stacking strategies:

Traditional optimization techniques, such as the unconstrained one by Acar & Rais-Rohani [19], did not perform well. The unbounded nature of the weights resulted in a volatile weighting scheme. Even the constrained version and the heuristic approach proposed by Goel et al. [18] did not outperform the proposed HPR alternative. Overall, stacking choices that involved optimization either did not converge or converged to a meaningless set of weights due to the correlation and, sometimes, linear dependence between the ϵ_i random variables samples.