https://orcid.org/0000-0002-7611-6558
Figures
Abstract
In this study, we propose and examine two procedures for constructing intervals that capture the uncertainty associated with determining the effective number of components in model selection problems or the shrinkage parameters in a regularization problem. The output of these methods is an interval (defined by two integer bounds) representing plausible values for the number of components and/or shrinkage parameters. Notably, these methods do not rely on the availability of a likelihood function, making them broadly applicable across various domains, such as regression, classification, feature and/or order selection, clustering, and dimensionality reduction. These techniques leverage the geometric properties of the error curve to construct intervals. Extensive experiments on both synthetic and real-world datasets demonstrated the effectiveness and practical utility of the proposed procedures. In addition, a MATLAB code is provided to facilitate adoption by practitioners and researchers.
Citation: Martino L, San Millán-Castillo R, Morgado E (2026) Automatic uncertainty evaluation for determining the number of components in nested models and the shrinkage regularization parameters. PLoS One 21(8): e0354966. https://doi.org/10.1371/journal.pone.0354966
Editor: Viacheslav Kovtun, Institute of Theoretical and Applied Informatics Polish Academy of Sciences: Instytut Informatyki Teoretycznej i Stosowanej Polskiej Akademii Nauk, UKRAINE
Received: January 9, 2026; Accepted: July 15, 2026; Published: August 14, 2026
Copyright: © 2026 Martino et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The underlying data are located at: https://zenodo.org/records/21639880.
Funding: This work was partly supported by (a) the grant PID2022-136887NB-I00 (POLIGRAPH) funded by MCIN/AEI/10.13039/501100011033, (b) by project Starting Grant for Rttb, BA-GRAPH “Efficient Bayesian inference for graph-supported data”, of the University of Catania, and (c) by the project LikeFree-BA-GRAPH funded by “PIAno di inCEntivi per la RIcerca di Ateneo 2024/2026” (PIACERI) of the University of Catania, Italy.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Model selection is a fundamental task in contemporary signal processing, machine learning, and statistical analysis. See the following works [1–3] and [4] as examples showing the wide range of applications in different fields. Moreover, in recent decades, uncertainty quantification and sensitivity analysis have emerged as highly relevant research topics in various scientific fields [5–9]. Particularly important is the scenario of nested models, that is, a family of models of different complexities, where the number of parameters can grow (i.e., the dimension of the vector of parameters can grow, building more complex models). This scenario appears frequently in different real-world applications: for instance, The order selection in polynomial regression or autoregressive schemes [10,11], feature selection [12], clustering [13], change point detection [2,14], and dimension reduction are just a few [15,16]. Other relevant examples in signal processing are the estimation of the number of signal sources [17] and the so-called structured parameter selection [18,19]. Throughout this study, the terms variables, components, features, and parameters are used interchangeably to refer to the elements of a model.
In the literature, three primary classes of methods are commonly employed to infer the complexity of the nested models. The first class is formed by cross-validation (CV) techniques [20,21] or similar strategies [10,22]. The second class is the so-called probabilistic statistical measures, formed by two main subfamilies: the information criteria (IC) [13,23,24] and the marginal likelihood approach (a.k.a., Bayesian evidence) used in Bayesian inference [4,25,26]. Related schemes can also be found [27,28]. The third class comprises methods grounded in geometric considerations, such as automatic ’elbow’ [29–32] or ’knee-point’ detectors [33,34].
In [29], the authors demonstrated that the automatic elbow detectors proposed in the literature can be reformulated as a specific instance of an information criterion. Various information criteria (IC) differ in their choice of the slope parameter , which governs the penalization of model complexity [35], such as the Bayesian-Schwarz information criterion (BIC) and Akaike information criterion (AIC) (Table 1 provides different special cases of IC). Another recent approach, known as the Spectral information criterion (SIC), considers the entire spectrum of possible values for the penalization parameter
, thereby encompassing other information criteria with linear complexity penalization as special cases. The SIC also returns a confidence measure of the proposed solution [36,37]. Similarly, other measures of reliability and confidence in the results in the context of elbow detection have been discussed [38]. These quantities attempt to show how ‘safe’ the solution is in terms of possible information lost by constructing a ‘too’ parsimonious model.
The main contributions of this study are twofold: we extend one of the derivations proposed in [29] and the SIC derivation to provide an interval of indices corresponding to different possible models. Namely, the output of the proposed methods is an interval of possible number of components (defined by two specified integer values). This interval embeds the uncertainty associated with the decision in a black-box manner, contingent on the specific analysis and observed data. In the first proposed method, the underlying idea employs geometrical considerations, and it is inspired by the concept of maximum area under the curve (AUC) in receiver operating characteristic (ROC) curves [39,40] and by the derivation of the well-known Gini index [41–43]. We also show the relationship between the proposed procedure and an alternating optimization considering conditioned information criteria. The second proposed method employs the confidence measure provided by SIC to obtain an interval of possible models as the final solution. Moreover, we further extended the proposed techniques to the case where the error curve V(z) is defined over a continuous parameter z, and is observed at non-uniformly sampled points. This extension enables their use in selecting (providing an interval solution) the shrinkage regularization parameters, as in LASSO or ridge regularization problems [44,45]. The proposed schemes can also be applied in more general contexts than the standard IC, even when a probabilistic model is not considered and the likelihood function is not defined. Unlike the measures proposed in [38], the convexity of the error curve V(k) is not required in this study. Since the proposed methods identify reliable ranges of models rather than potentially risky single-point solutions, their range of applications is broad, encompassing areas such as the Internet of Things (IoT), smart cities, and effective governance, among many others [46]. Numerical experiments with artificial and real data showed promising results. Related Matlab code is also provided https://zenodo.org/records/21639880.
2. Framework and background
2.1. The error curve V(k)
In numerous real-world applications, we desire to infer a vector of parameters of dimension k given a data vector
. An observation model that induces a likelihood function
is usually available [25]. The discrete variable k, which denotes the dimension of the parameter vector
, is often unknown and must be inferred from the observed data
. For example, k may represent the number of clusters in a clustering problem or the order of a polynomial in nonlinear regression task. In these application problems, a non-increasing error curve can be computed,
where K represents the maximum possible dimension of the parameter vector, i.e., a model with K parameters, . The function V(k) can be any metric that characterizes system performance. For simplicity and without loss of generality, we consider an integer variable k with an increasing step of one unit. This can be easily generalized. when a likelihood function is given, a usual choice of V(k) is
For example, as in [25,47,48]. Alternatively, V(k) can be directly defined as the mean square error (MSE), the mean absolute error (MAE), or transformations of them (e.g., as ). However, any other fitting measures can be considered.
Remark. For the sake of simplicity and without loss of generality, we assume that . Clearly, this condition can always be obtained with a simple subtraction,
. See Fig 1 for an example of V(k) (dashed line).
2.2. A general expression for several information criteria (IC)
In this section, we outline a general formulation for various information criteria (IC) and the underlying principles of this approach. Typically, the curve V(k) is constructed as a non-increasing function. Graphical examples are shown in Figs 1 and 2(a). A well-established approach in the literature involves incorporating a linear penalty to account for model complexity,
where the slope of the complexity penalization term is denoted as . Since V(k) is non-increasing and the penalty term
increases with respect to k, the cost function C(k) will exhibit at least one minimum. See Fig also 2(a), for a graphical example of V(k), the linear penalty
and the corresponding cost function
.
(a) Example of function V(k), the penalty (for a specific value of
), and the resulting cost function
(shown with dots). (b) Example of a piecewise constant function
yielded by SIC on a log-
scale.
The objective is to obtain as the index of the optimal model, serving as an estimator for the number of components in the nested model. Clearly, the optimal value
depends on the choice of
. Table 1 summarizes some relevant special cases of IC, considering the possible choice of error curve V(k) and of the parameter
. Some well-known examples are Bayesian-Schwarz information criterion (BIC), Akaike information criterion (AIC) and the Hannan-Quinn information criterion (HQIC), to name a few [47–49]. Each was derived in a distinct context and under different assumptions. In the literature, alternative IC with different analytical forms have been proposed (e.g., involving non-linear penalty terms); however, they are not as widely adopted as those with the form given in Eq. (1).
Assumptions on V(k). For clarity in the exposition, we have stated that V(k) is a non-increasing function. This assumption could even be relaxed, requiring only a decreasing trend, as in the (noisy) curves V(k) shown in Fig 5. Convexity is not a required condition. Regarding the geometric method in Section 3, the convexity of V(k) is a sufficient but not necessary condition for ensuring the uniqueness of the solution.
3. Interval solution based on geometric considerations
In this section, we present a novel technique for obtaining an interval of indices in the context of an elbow detection problem that encodes the uncertainty in model selection. More specifically, we extend one of the derivations of the universal automatic elbow detector, as presented in [29]. The underlying idea is inspired by the AUC approach in ROC curves for classification [39,40] and the derivation of the Gini index in the economic field [41,43].
3.1. Interval solution based on UAED approach
The algorithm is based on the construction of three straight lines: the first one passing through the points (0,V(0)) to ; the second passes through the points
to
and the last passes through the points
to (K,0), as shown in Fig 1 (clearly,
). The goal is to minimize the area under this piece-linear approximation of the curve V(k). The total area under this approximation is the sum of the two trapezoidal areas (A1 and A2) and a triangular area (A3), as shown in Fig 1. Namely, we have
where we have used the assumption V(K)=0. Given the previous considerations, the optimal interval which possibly includes the location of an “elbow” point is defined as
It is important to note that solving the optimization above is straightforward because k1 and k2 belong to a discrete and finite set. Note that by construction, the elbow provided by UAED in [29] is always contained in the obtained interval
, i.e.,
.
3.2. Relationship with the information criterion approach
If we keep fixed k2 as a constant, given Eq. (2), we have
We can rewrite it as a function of only k1 (given k2), i.e.,
where we have used that k2 and K are considered constant. The last expression
has the form of an information criterion where V(k1) plays the role of the error curve and the slope is here a function of k2, i.e.,
instead of being constant. If k2 = K and V(K)=0, we recover the slope associated with the automatic elbow detector in [29], as shown in Table 1. Fixing k1 as a constant, given Eq. (2), we have
and we can rewrite it as a function of only k2 (given k1), i.e.,
where we have used that k1 and K are constant, and set
that, again, has the form of an information criterion where the slope is a function of k1, i.e.,
instead of a constant. If we set k1 = 0, we again recover the slope associated with the automatic elbow detector in [29]. Then, considering an alternating optimization approach, minimizing
can be interpreted as minimizing iteratively different conditioned information criteria, considering
and
, which are connected to each other by the different slopes of the complexity penalty.
4. Interval solution based on SIC approach
4.1. Principles underlying the SIC technique
For simplicity, assume that V(k) is strictly decreasing, such that C(k) has a unique minimum. In [36], firstly the authors note that ,
Indeed, for we always obtain
. It is interesting to note the relationship between
in Eq. (7) and
in Eq. (4). Generally, the position of the minimum changes
, by varying
in
. Namely, the location of the minimum a function of
,
It is a non-increasing, piecewise constant function taking discrete values from 0 to K, where and
for
. It is a piecewise constant function since, even changing
, the value
can remain unvaried. See Fig 2(a) for an example of the function V(k) and the corresponding cost function
for a given value of
. The resulting function
is shown in Fig 2(b).
As shown in Fig 2(b), different values of and
can yield the same minimum denoted as j, i.e.,
and
, so that the function
remains constant in certain pieces, each one characterized by a specific “length”. Thus, varying
, to each possible solution denoted by an index {0,1,...,K}, we have associated a length value, i.e.,length of the constant piece; see Fig 2(b). We can convert these lengths into weights, associating to each index
a normalized weight
. This weight
is computed approximately in Eq. (10): it quantifies the robustness of the j-th minimum obtained in Eq. (8), with respect to variations in
. In other words, a large weight
indicates that the model with j components appears as a solution of the minimization in (8) repeatedly across a wide range of
values. Moreover, some solutions – say, for instance, with k components – may never appear for any value of
; in that case,
must be zero,
. Note that, by construction, we have
(by definition of
). Fig 3 provides a graphical representation of this probability mass.
The weights obtained by the SIC method correspond to the (black) error curve V(k) in Figure 2(a), and are used in the experiment in Section 7.4. Note that
, i.e., K = 122. It is important to emphasize that the number of nonzero weights is significantly smaller than 122.
A Monte Carlo procedure to compute approximately these normalized weights is given in Table 2, using
uniform samples from
. A more efficient alternative to the Monte Carlo approach is to use a fine grid dividing the interval
in
uniform sub-intervals, which is the strategy implemented in the provided MATLAB code. The two approaches are equivalent. The Monte Carlo procedure is perhaps more illustrative; therefore, it is described in Table 2. In contrast, the fine-grid procedure is more efficient, as it exhibits lower variance because it is equivalent to a stratified/systematic sampling approach.
Note that the normalized weights form a probability mass function (pmf). In order to design an interval including the possible elbow of the curve (and encoding the related uncertainty), we define a cumulative sum of the first k weights, denoted as
, i.e.,
with .
4.2. Probability interpretations and interval solution
In this section, we provide additional theoretical support and interpretation for the quantities obtained through the SIC methodology. In particular, we discuss their statistical interpretation and the rationale underlying their choice and use. Choosing uniformly , each weight represents the probability that the number of components of the model is
, i.e.,
Fig 3 shows an example. Then, The cumulative distribution has the following probabilistic meaning:
We note that sampling uniformly from the interval
provides an objective and robust strategy, as it avoids favoring the selection of any particular model.
At the same time, however, we aim to avoid selecting overly parsimonious models (i.e., excessively simple models), which are typically associated with large values of the penalization parameter ; see Fig 2(b). Note that
can be interpreted as the a detection probability, i.e., the probability of finding a model that correctly describes the phenomenon (the true model, denoted as
). Therefore, increasing the probability
– in order to reduce the risk of missing relevant components in the model— leads to a larger value of k. Several empirical studies reported in [36] indicate that threshold values around
provide a high probability of correct detection. Then, we set the interval solution
where
Note that play a similar role to a confidence level. After several empirical studies, we can assert that a very robust choice is given by
and
. Safer intervals can be designed by considering a more conservative value, such as
.
5. Interpreting the interval solutions
The methods described above return an interval as the final solution. In this section, we first explain how to interpret the interval. We then applied both techniques to ideal scenarios to validate and illustrate the intuition underlying these interpretations.
5.1. Interpretation
The two interval solutions obtained by the UAED-geometric and SIC approaches have two different interpretations, which are described below. In both cases, the true number of components should be contained in the interval
, that is,
. In both cases, Hence, shorter SIC intervals imply that we are more certain that
. However, the principles underlying the construction of the interval are different.
- the SIC-based interval is build to provide an interval where with high probability is contained the true model
, i.e., in some sense, maximizing the probability
. Since
represents a detection probability, the interval defines a range of solutions that reduces the risk of overlooking relevant components in the model.
- The UAED-geometric interval is constructed according to a more robust principle: ensuring that the elbow point is not contained in the intervals
and
. In other words, the UAED-geometric interval can be viewed as being designed to try to maximize the probability
.
Therefore, due to these two different construction philosophies, it is intuitively clear that the UAED-geometric intervals are generally wider than the corresponding SIC-based intervals.
5.2. Application to ideal scenarios
To illustrate the different principles employed by the proposed techniques, we consider several idealized scenarios based on artificial curves V(k) for which the location of the true solution (the elbow point) is exactly known. Moreover, these idealized scenarios were constructed so that the possible desired interval solutions could be readily inferred through simple visual inspection. Specifically, We assume that the curve V(k) is formed by three linear pieces (see Fig 4):
For any positive value a > 0, the elbow is clearly at . the degree of certainty associated with this solution depends on the value of a: the smaller the value of a > 0, the stronger the evidence in favor of the solution
. For a = 0, the elbow is at
. For a > 10, the curve V(k) is not convex.
The parameters ,
and b are chosen in order to have a continuous curve V(k) (at k = 10 and k = 30). To generate different scenarios, We used the value at k = 10 as the parameter of the experiment, namely V(10) = a. Choosing and fixing a, we have
See Fig 4 for an example. More precisely, as depicted in Fig 4(a), if a = 0 the true solution/elbow is at . Note that with a = 0, V(k) is formed by only two linear pieces. For any other positive value a > 0, the elbow is clearly at
. Moreover, if a > 10, the curve V(k) is not convex, as shown in Fig 4(f). Fig 4(e) provides the case with a = 10, and V(k) is again formed by only two linear pieces.
Note that the degree of certainty associated with this solution depends on the value of a: the smaller the value of a > 0, the stronger the evidence in favor of the solution . The intervals provided by the proposed schemes are listed in Table 3. First, note that all the intervals (obtained with both schemes) always contain the true value of the elbow
. The results of the UAD-based intervals can be summarized as follows:
- with
the returned interval is
,
,
- with
the returned interval is
,
,
- with
the returned interval is
,
.
The SIC-based intervals can be summarized as follows:
- with
the returned interval is
,
,
- with
the returned interval is
,
,
- with
the returned interval is
,
.
These results confirm that the UAD-based intervals are designed to maximize the probability , whereas the SIC-based intervals are constructed with the simpler objective of ensuring that the true elbow
lies within the interval.
6. Generalization of UAED and SIC for non-uniform sampled and continuous-valued parameter 
In the previous section, we have considered a uniform discrete index k = 0,1,2,...,K, is a discrete variable (with increasing step 1) that represents the number of components/parameters in the analyzed model. Thus, the error curve V(k) was (generally a non-increasing) function defined a discrete domain with a uniform sampling k = 0,1,2,...,K with step 1. In other frameworks (see an example below in Section 6.2), we can have an error curve that is a function of a continuous parameter
. However, we can observe only some points of this curve that are non-uniformly sampled at
. Thus we finally observe V(z) in a finite number of points,
and we assume, without loosing generality,
. Recall that generally the difference
varies with k, and generally
, but we always have
.
6.1. Extended algorithms
In the following, we assume V(z) is a non-increasing function so that .
The UAED solutions. Let define
Therefore, the two optimal points forming the interval solution are denoted as . Adapting the UAED solutions for this new framework is possible changing the definitions of the areas:
and the final solution will given by
where A1, A2 and A3 are defined in Eq. (17).
The SIC solutions. Considering the set of z values where V(z) is evaluated, we have a one-to-one (bijective) correspondence/mapping
, i.e., k = 0 corresponds to z0, k = 1 corresponds to z1, and so on. Furthermore, we can rewrite the penalized minimization problem as
where varying the position of the minimum changes. We can still refer to the optimal index
, i.e.,
where again . We need also to change the procedure to obtain
, i.e.,
Recall that we have and
, by assumption. Thus, for
the minimum is reached at z0 the
. Hence,
is still a non-increasing, piecewise constant function that takes discrete values from 0 to K, where
and
for
. Therefore, the rest of the algorithm remained unchanged.
6.2. Example of application: selection of the regularization parameter
For the sake of simplicity, in order to provide an example of application, We focus on a linear regularized regression problem. Identical considerations can be made for a nonlinear scenario or a classification setting. Consider the observed dataset where the
are input vectors,
is an
input matrix, and
is the corresponding output vector. We also define the vector of unknown parameters
, that can be estimated by the following minimization,
The second term is a regularizer, employed for (a) avoiding overfitting, (b) numerical issues, and (c) forcing some characteristics in the final solution (as sparsity in the vector
, when
). The scenario corresponding to
is known as ridge regression, whereas the scenario corresponding to
is the LASSO regression [44,45]. The positive continuous parameter
controls the strength of the regularization term.
The first term above is the well-known mean square error (MSE), that is, . Hence, for each possible
, we have an optimal
and a corresponding value of MSE, i.e.,
Clearly, with , we have the smallest MSE and if
we have
, i.e.,
. The optimal selection of
, which balances model fitting, generalization, and other desirable properties, remains an active area of research [44,45]. Note that if we set
and define
we have a non-increasing error function of z. Another possible definition (similar to other information criteria) is . Therefore, we can apply UAED and SIC to provide an optimal choice of
, and also an interval solution
with
, in order to quantify the uncertainty in the selection (as described in the previous sections).
7. Numerical experiments
In this section, we evaluate the proposed construction of the intervals across different settings, including experiments with artificial data (Sections 7.1 and 7.3), a synthetic curve V(k) defined analytically (Section 7.2), and two real-world datasets are analyzed in Sections 7.4–7.5. An application for selecting a LASSO-shrinkage parameter, applying the extensions presented in Section 6, is provided in Section 7.6. The MATLAB code used for the experiments is also made available in https://zenodo.org/records/21639880.
7.1. Order selection in an auto-regressive model with a Laplacian noise
We generate a dataset of T pairs , where t is an integer temporal index and the signal
is a scalar value for each t. We consider the following auto-regressive model,
where . We assume that
is a Laplacian noise, i.e.,
with zero mean, , and variance
. The goal is to infer the true order of the model
. This problem is very frequent in signal processing, statistics, and machine learning [10,11]. Note that it is possible to easily generate random samples from a Laplace density [50], Chapter 2]. Therefore, starting with null initial conditions, it is possible to generate data according to the model in Eq. (24). We test two levels of noise
, i.e., b = 0.35, with standard deviation
, i.e., b = 3.53. We also consider different numbers of data,
.
In this example, we test three possible values of the order of the model, , where we have employed the following formula for the coefficients:
, to ensure that the system in Eq. (24) is stable. More specifically, we have
Note that the last coefficients become smaller, making their detection/estimation more difficult, especially with . Hence, the scenario with
is more difficult in terms of estimating the order of the model, especially with a high noise power.
Given each combination of the values of , T, and
, we generated the data
according to the model (24). In all scenarios, we averaged the results with 103 independent runs, generating a new time series of T for each run. Moreover, in all the simulations, we consider
with
with
(setting K = 100), where
is induced by Eq. (24), in order to allow the comparison with other schemes in the literature, as shown in Tables 1 and 5. Note that, because we consider Laplacian noise,
is not provided in an analytic form. Indeed, it requires the knowledge of the maximum likelihood estimator
, then
. We consider the least-squares estimation of vector
as an approximation of
. Fig 5 shows 50 examples of the curves V(k) in different runs (for
,
and
), jointly with the median curve (black solid line) and a yellow area between two black dashed lines, corresponding to the 98% of the empirical distribution provided by the 103 runs.
In Table 4, we provide the results of different information criteria: BIC [47], AIC [48], HQIC [36], and UAED [29]. The best results are marked with yellow cells. We can observe that BIC and UAED provide the best results in terms of the correct decision rate , inferring the order of the model (i.e., the times that the method selects the correct order over the total number of simulations). Recall that the first proposed interval method is designed to extend the derivation of the UAED. Table 5 shows the median intervals (obtained using the two methods introduced in the previous section) over the 103 runs in the different scenarios. Moreover, The table gives the rate
of the intervals containing
(clearly, including the extreme values of the interval). Namely,
is the ratio of the number of times
is inside the constructed interval over the total number of runs. This value
plays the same role as the quantity
where
is the confidence level in classical interval estimation. We observed that both achieved excellent performance. The best results in terms of shorter intervals and
are marked with a green cell. Both methods appear to be virtually insensitive to the noise power. The geometric-based approach always obtains good results, even with a small amount of data. Whereas the SIC-based approach seems to work better with a larger number of data points, suffering more in the case of smaller data points (designing much longer intervals in these cases). This can be clearly seen by looking at the median lengths
of the intervals:
The SIC approach is able to return intervals with zero median lengths (maximum certainty) when T = 2000 (larger number of data points) and and
, but struggles when T = 200 where the median lengths are quite large, such as 85, 77, 90 and 90, compared with the median lengths obtained with the geometric approach, respectively 4, 4, 6 and 6. In this sense, the geometric approach appears to be more robust.
7.2. Artificial curve V(k) with known analytic form
In this section, we consider a synthetic error curve V(k) with a known analytic form. In this way, we can observe some convergence properties of the different schemes and the ability to encode the geometric invariant features of a curve V(k). More specifically, the goal of this experiment is to analyze the behaviors of the elbow detectors [29–32], the index of effective number of variables (ENV) recently introduced in [38], and the two constructions of intervals proposed in this work, the geometric-based and the SIC-based approaches. We consider the function
and study different values of K. For each possible value of , we define
, such that V(K)=0 in any case. We tested the elbow detectors [29–32], the ENV index, and the construction of the intervals, obtaining the results given in Table 6.
We can observe that the detected elbows are always contained in the geometric-based intervals, as expected and stated in Section 3. However, the geometric-based intervals present an undesirable dependence on K, particularly in the second extreme of the interval. On the other hand, the ENV index converges to the value as K increases, as expected [38]. The SIC-based intervals present the same stability property converging to the interval [30] as K increases.
7.3. Choice of the number of clusters
Let us consider a mixture of five bidimensional Gaussian distributions where:
,
,
,
,
,
,
,
,
- and
,
.
We generated 2500 simulated datasets from this mixture. Thus, the generated data form 5 disjoint clusters. In this experiment, we define , where
represents the inner variance of the j-th cluster. Each value of
was averaged over 200 runs, applying a k-means algorithm to define the memberships of each cluster at each run. The case k = 0 corresponds to a unique single cluster formed by all data. Hence, the total number of clusters is given by k + 1. We assume K = 50 as the maximum number of possible clusters. Note that With this choice of V(k), we can apply only UAED, whereas the rest of the IC in Table 1 cannot be applied. In this experiment, UAED suggested the right number of clusters, 5. The geometric-based interval in this example is [2,6], whereas the SIC-based interval is [5,6]. Both contain the correct number of clusters, and the SIC-based interval is shorter.
7.4. Feature selection in a soundscape emotion real dataset
In this section, we address the variable selection problem in a regression setting using real-world data, specifically focusing on a soundscape emotion dataset. More specifically, a dataset of N pairs is given, where each input vector
is formed by K variables, and the outputs
’s are scalar values. We assume
and a linear measurement model,
where is a Gaussian perturbation with zero mean and variance
, i.e.,
. In the soundscape emotion dataset analyzed for instance in [12], there are K = 122 features and N = 1214 number of data points. The output represents a variable defined as “arousal” in [12].
After ranking the 122 variables as suggested in [12], We set again where
with
, and where the likelihood function
is induced by Eq. (25). This choice of V(k) allows the computation of AIC, BIC, and HQIC (see Table 1). The results of the different IC and the intervals built by the proposed schemes are given below:
- BIC suggests a model with 17 variables.
- AIC chooses 44 variables.
- HQIC selects a model with 41 variables.
- UAED suggests a model with 11 variables.
- The interval built with the geometric procedure is [7,41].
- The interval based on the SIC procedure is [7,25].
Note that the geometric-based interval contains all the IC, except for the result of AIC (44 variables). The SIC-based interval also excludes the solution provided by HQIC (41 variables). In this experiment, the SIC-based interval was shorter than the geometric-based interval. Both intervals are in line with other previous studies regarding this dataset and with the experts’ recommendations in the literature, e.g., [12]. Fig 6(a) shows the corresponding curve V(k) and summarizes all the results provided by the IC and obtained intervals.
7.5. Feature selection in a classification problem with a nonalcoholic fatty liver disease real dataset
In this section, we consider an example of biomedical applications, which are nowadays extremely important in signal processing and machine learning [51,52]. In [53], the authors study the most important variables for predicting patients at risk of developing non-alcoholic fatty liver disease. The dataset contains 1525 patients. The authors of [53] employed a random forest (RF) algorithm as a classifier and ranked the input variables, selecting the most relevant ones. The resulting 4 most important features are according to this raking: (a) insulin resistance, (b) ferritin, (c) serum insulin levels, and (d) triglycerides. The authors in [53] employed cross-validation (CV) to find the optimal number of features (that is 4), and this result was supported by expert opinions.
In this section, we define as the error curve, as shown in Fig 6(b), using
obtained in [53] and after ranking the 35 variables as in [53]. Note that V(0)=0.5 representing a completely random binary classification. Note that, even with this choice of the curve
, we can still apply UAED and SIC, as shown in Table 1. Other information criteria cannot be applied in this context. However, we can obtain the intervals based on the geometric approach (which is related to the UAED derivation) and the SIC approach. As shown in Fig 6(b), the resulting geometric interval is [2,11] and the SIC-based interval is [3,11]. Both contain 4 variables, which is exactly the result provided in [53], obtained by applying a cross-validation approach and supported by the experts’ opinions.
7.6. Selection of the the LASSO-shrinkage parameter
Consider a dataset where the
,
are input vectors and
input matrix, whereas
is the output vector. We also define the vector of unknown parameters
, and assume the observation model
where is a noise perturbation that we consider Gaussian with zero mean and variance
. We set R = 30 and generate artificially N = 200 data pairs following model considering
(randomly drawn for each i, r at each run), and assuming the vector
Hence, the last 10 features do not affect the outputs . After data generation, we apply a LASSO techniques [44,45] to obtain and estimate the coefficient vector
. Namely, we follow the minimization of a square error with an L1 regularizer,
We use a thin grid of values: as
increases, the number of zeros in
, until a maximum value of
sun that
has only null entries.
We denote as the smallest value of
such that we have only 10 zeros (located at the last ten positions of
) and denote s
the biggest value of
such that we have only 10 zeros (located at the last ten positions of
). Hence, a reasonable choice of
for our problem is any
inside
. For each realization of the data, the groundtruth interval
is shown in Fig 7 with pink shaded areas. We apply the procedures in Sections 6 a 6.2 to find the interval solutions based on the UAED and SIC. We can observe in Fig 7, all the
values suggested by the intervals based UAED and SIC are completely contained in the groundtruth intervals
, with the exception of 3 runs (over a total number of 100 independent runs), where a small portions of the first parts of the intervals are slight out the groundtruth intervals (that, in these 3 scenarios, are highlighted with solid red lines). Hence, it appears clear that the interval solutions provided by UAED and SIC contain safe and useful
values to use in a LASSO regression problem.
In these intervals, the values provide exactly 10 zeros (well-located) in
. (a) The intervals provided by the UAED are shown in yellow shaded areas.The only three intervals – shown with red vertical segments – that are not completely contained in the yellow area are: [1.78,4.34], [1.70,5.82], and [1.80,4.19]. In all cases, the UAED-based intervals had a slightly smaller first element (1.57, 1.58, and 1.56, respectively). (b) The intervals provided by the SIC are shown in green shaded areas. The only three intervals – shown with red vertical segments – that are not completely contained in the green area are [1.81,4.26], [1.80,5.75], and [1.83,4.23]. In all cases, the SIC-based intervals have a slightly smaller first element (1.63, 1.65 and 1.60, respectively).
8. Conclusions
In this study, we proposed two alternative constructions for deriving intervals that capture the uncertainty associated with determining the number of components in nested models. We have also extended the proposed techniques to cases where the error curve , defined over a continuous parameter
, is observed only at non-uniformly sampled points. This extension enables their application for selecting shrinkage regularization parameters.
The proposed approaches do not require knowledge of a likelihood function, making them broadly applicable across various domains, including regression and classification, feature and/or order selection, clustering, change-point detection, and dimensionality reduction, among others. We have also extensively discussed the connection between our methods and widely used information criteria in the literature. Additionally, a MATLAB code is provided to facilitate the adoption of these methods by researchers and practitioners in applied settings.
Extensive experiments on both synthetic and real data demonstrated the strong performance of the proposed schemes. In particular, the results highlight the following:
- Both procedures can be applied for selecting the shrinkage parameters in classification or regression problems with regularizations.
- Both procedures design intervals that contain the true number of components (or the number of components suggested by the experts) in more than 90% of the runs/realizations.
- The geometry-based procedure appears to be more robust to variations in the number of data points; however, its performance is influenced by the total number of components K.
- The SIC-based procedure tends to yield the most accurate results as the number of data points increases. Conversely, when the data size is limited, this approach often produces relatively wide intervals. A notable property of the SIC-based method is the convergence and invariance of the resulting interval as
.
- Both approaches seems to exhibit a notable degree of robustness to the noise levels affecting the data.
The proposed schemes provide particularly suitable solutions when the error curve V(k) tends to be convex, as evidenced by the final two experiments involving real datasets and supported by our practical experience. It is also important to highlight that the proposed methods can be applied in the absence of a likelihood function. Consequently, we argue that both procedures serve as (a) universal, (b) automatic, and (c) computationally efficient tools for quantifying the uncertainty inherent in model selection problems without the need for resampling techniques or cross-validation schemes. As a future research direction, the problem of identifying the elbow point in a noisy error curve V(k), that is, considering uncertainty in the evaluations of V(k), appears particularly interesting and deserves further investigation. Distributed scenarios, such as federated learning frameworks designed to preserve privacy, can also be investigated. In such settings, each local node processes its own data and produces a corresponding error curve V(k), thereby introducing an additional source of uncertainty at the central aggregation node.
References
- 1.
Raschka S. Model evaluation, model selection, and algorithm selection in machine learning. 2018. https://doi.org/10.48550/arXiv.1811.12808
- 2. Ding J, Tarokh V, Yang Y. Model Selection Techniques: An Overview. IEEE Signal Process Mag. 2018;35(6):16–34.
- 3. Laubach ZM, Murray EJ, Hoke KL, Safran RJ, Perng W. A biologist’s guide to model selection and causal inference. Proc Biol Sci. 2021;288(1943):20202815. pmid:33499782
- 4. Yates LA, Aandahl Z, Richards SA, Brook BW. Cross validation for model selection: A review with examples from ecology. Ecological Monographs. 2023;93(1).
- 5. Abdar M, Pourpanah F, Hussain S, Rezazadegan D, Liu L, Ghavamzadeh M, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion. 2021;76:243–97.
- 6. Kwon Y, Won J-H, Kim BJ, Paik MC. Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation. Computational Statistics & Data Analysis. 2020;142:106816.
- 7. Yang G. Multilayer neuroadaptive constraint-handling control architecture for a family of nonlinear systems with uncertainty compensation. Information Sciences. 2025;690:121517.
- 8. Tan Q, Wen Y, Xu Y, Liu K, He S, Bo X. Multi-view uncertainty deep forest: An innovative deep forest equipped with uncertainty estimation for drug-induced liver injury prediction. Information Sciences. 2024;667:120342.
- 9. Zhang P. A novel feature selection method based on global sensitivity analysis with application in machine learning-based prediction model. Applied Soft Computing. 2019;85:105859.
- 10. Iba K, Shinozaki T, Maruo K, Noma H. Re-evaluation of the comparative effectiveness of bootstrap-based optimism correction methods in the development of multivariable clinical prediction models. BMC Med Res Methodol. 2021;21(1):9. pmid:33413132
- 11.
Khan DM, Yahya N, Kamel N. Optimum Order Selection Criterion for Autoregressive Models of Bandlimited EEG Signals. In: 2020 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), 2021. 389–94. https://doi.org/10.1109/iecbes48179.2021.9398836
- 12. Millan-Castillo RS, Martino L, Morgado E, Llorente F. An Exhaustive Variable Selection Study for Linear Models of Soundscape Emotions: Rankings and Gibbs Analysis. IEEE/ACM Trans Audio Speech Lang Process. 2022;30:2460–74.
- 13. Pandey AC, Kulhari A, Shukla DS. Enhancing sentiment analysis using Roulette wheel selection based cuckoo search clustering method. J Ambient Intell Human Comput. 2021;13(1):1–29.
- 14. De Ryck T, De Vos M, Bertrand A. Change Point Detection in Time Series Data Using Autoencoders With a Time-Invariant Representation. IEEE Trans Signal Process. 2021;69:3513–24.
- 15. Jia W, Sun M, Lian J, Hou S. Feature dimensionality reduction: a review. Complex Intell Syst. 2022;8(3):2663–93.
- 16. Ray P, Reddy SS, Banerjee T. Various dimension reduction techniques for high dimensional data analysis: a review. Artif Intell Rev. 2021;54(5):3473–515.
- 17. Ma B, Zhang T. Underdetermined Blind Source Separation Based on Source Number Estimation and Improved Sparse Component Analysis. Circuits Syst Signal Process. 2021;40(7):3417–36.
- 18. Jansen M. Information criteria for structured parameter selection in high-dimensional tree and graph models. Digital Signal Processing. 2024;148:104437.
- 19. Lee C, Luo ZT, Sang H. T-LoHo: A Bayesian regularization model for structured sparsity and smoothness on graphs. Advances in Neural Information Processing Systems. 2021;34:598–609.
- 20. Tougui I, Jilbab A, Mhamdi JE. Impact of the Choice of Cross-Validation Techniques on the Results of Machine Learning-Based Diagnostic Applications. Healthc Inform Res. 2021;27(3):189–99. pmid:34384201
- 21. Bates S, Hastie T, Tibshirani R. Cross-validation: what does it estimate and how well does it do it?. J Am Stat Assoc. 2024;119(546):1434–45. pmid:39308484
- 22. Vehtari A, Gelman A, Gabry J. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat Comput. 2016;27(5):1413–32.
- 23. Piironen J, Vehtari A. Comparison of Bayesian predictive methods for model selection. Stat Comput. 2016;27(3):711–35.
- 24. Reich BJ, Storlie CB, Bondell HD. Variable selection in Bayesian smoothing spline ANOVA models: Application to deterministic computer codes. Technometrics. 2009;51(2):110–20. pmid:19789732
- 25. Llorente F, Martino L, Delgado D, López-Santiago J. Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review. SIAM Rev. 2023;65(1):3–58.
- 26. Ward EJ. A review and comparison of four commonly used Bayesian and maximum likelihood model selection tools. Ecological Modelling. 2008;211(1–2):1–10.
- 27. Tibshirani R, Knight K. The Covariance Inflation Criterion for Adaptive Model Selection. Journal of the Royal Statistical Society Series B: Statistical Methodology. 1999;61(3):529–46.
- 28. Rissanen J. Modeling by shortest data description. Automatica. 1978;14(5):465–71.
- 29. Morgado E, Martino L, Millán-Castillo RS. Universal and automatic elbow detection for learning the effective number of components in model selection problems. Digital Signal Processing. 2023;140:104103.
- 30. Onumanyi AJ, Molokomme DN, Isaac SJ, Abu-Mahfouz AM. AutoElbow: An Automatic Elbow Detection Method for Estimating the Number of Clusters in a Dataset. Applied Sciences. 2022;12(15):7515.
- 31. Zhang J, Fu P, Meng F, Yang X, Xu J, Cui Y. Estimation algorithm for chlorophyll-a concentrations in water from hyperspectral images based on feature derivation and ensemble learning. Ecological Informatics. 2022;71:101783.
- 32.
Kaplan D. Knee Point. https://www.mathworks.com/matlabcentral/fileexchange/35094-knee-point 2024.
- 33.
Chen R, Li K. Knee Point Identification Based on the Geometric Characteristic. In: 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2021. 764–9. https://doi.org/10.1109/smc52423.2021.9658848
- 34. Diao W, Saxena S, Han B, Pecht M. Algorithm to Determine the Knee Point on Capacity Fade Curves of Lithium-Ion Cells. Energies. 2019;12(15):2910.
- 35. Zhang J, Yang Y, Ding J. Information criteria for model selection. WIREs Computational Stats. 2023;15(5).
- 36. Martino L, San Millán-Castillo R, Morgado E. Spectral information criterion for automatic elbow detection. Expert Systems with Applications. 2023;231:120705.
- 37. Millán-Castillo RS, Martino L, Morgado E. A Variable Selection Analysis for Soundscape Emotion Modeling Using Decision Tree Regression and Modern Information Criteria. IEEE Access. 2024;12:92622–34.
- 38. Martino L, Morgado E, Millán Castillo RS. An index of effective number of variables for uncertainty and reliability analysis in model selection problems. Signal Processing. 2025;227:109735.
- 39. Bishop CM. Pattern recognition. Machine Learning. 2006;128:1–58.
- 40. Fan J, Upadhye S, Worster A. Understanding receiver operating characteristic (ROC) curves. CJEM. 2006;8(1):19–20. pmid:17175625
- 41. Gastwirth JL. The Estimation of the Lorenz Curve and Gini Index. The Review of Economics and Statistics. 1972;54(3):306.
- 42. Lerman RI, Yitzhaki S. A note on the calculation and interpretation of the Gini index. Economics Letters. 1984;15(3–4):363–8.
- 43. Martin AJF, Conway TM. Using the Gini Index to quantify urban green inequality: A systematic review and recommended reporting standards. Landscape and Urban Planning. 2025;254:105231.
- 44. Freijeiro‐González L, Febrero‐Bande M, González‐Manteiga W. A Critical Review of LASSO and Its Derivatives for Variable Selection Under Dependence Among Covariates. Int Statistical Rev. 2021;90(1):118–45.
- 45. Tibshirani R. Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology. 1996;58(1):267–88.
- 46.
Pawar PP, Kumar D, Kumar Meesala M, Kumar Pareek P, Reddy Addula S, K S S. Securing Digital Governance: A Deep Learning and Blockchain Framework for Malware Detection in IoT Networks. In: 2024 International Conference on Integrated Intelligence and Communication Systems (ICIICS), 2024. 1–8. https://doi.org/10.1109/iciics63763.2024.10860155
- 47. Schwarz G. Estimating the Dimension of a Model. Ann Statist. 1978;6(2).
- 48. Spiegelhalter DJ, Best NG, Carlin BP, der Linde AV. Bayesian measures of model complexity and fit. J R Stat Soc B. 2002;64:583–616.
- 49. Hannan EJ, Quinn BG. The Determination of the Order of an Autoregression. Journal of the Royal Statistical Society Series B: Statistical Methodology. 1979;41(2):190–5.
- 50.
Martino L, Luengo D, Míguez J. Independent random sampling methods. Springer. 2018.
- 51. Binson VA, Thomas S, Subramoniam M, Arun J, Naveen S, Madhu S. A Review of Machine Learning Algorithms for Biomedical Applications. Ann Biomed Eng. 2024;52(5):1159–83. pmid:38383870
- 52. Park C, Took CC, Seong J-K. Machine learning in biomedical engineering. Biomed Eng Lett. 2018;8(1):1–3. pmid:30603186
- 53. García-Carretero R, Holgado-Cuadrado R, Barquero-Pérez Ó. Assessment of Classification Models and Relevant Features on Nonalcoholic Steatohepatitis Using Random Forest. Entropy (Basel). 2021;23(6):763. pmid:34204225