2.1 Original formulation (extended to general models)

Let X_i, i = 1, …, k be dimensional input parameters with units [X_i], taking values in [min_i, max_i] uniformly. If the parameter can only take integer values, it takes values in the set {min_i, min_i + 1, …, max_i}. The same holds for Boolean parameters, but then min_i = 0 and max_i = 1, where 0 encodes false and 1 stands for true. x_i denotes the dimensionless equivalent scaled to the unit interval, i.e. (2) henceforth referred to as scaled dimensionless parameters. The assumption of uniformly distributed inputs can be relaxed to include arbitrary distributions. Scaling the sampled parameter values from [0, 1] to the [min_i, max_i]-interval should then be done using the corresponding inverse cumulative density function (CDF). The dimensional outputs of interest are denoted by Y_j, j = 1, …, q with corresponding unit [Y_j].

The parameter points used for the analysis are sampled from a regular discrete subset of the complete scaled dimensionless parameter space (typically called Ω ⊂ [0, 1]^k) containing p_i regularly spaced points in the x_i-direction, and are then transformed to the actual parameter space. p_i is also called the number of levels for parameter x_i. The scaled dimensionless parameter x_i thus takes values in the set (3) see Fig 1, while (using Eq (2)) the actual parameter value is an element of (4)

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic representation of parameter sampling probabilities for a parameter x_i on [0, 1] with p_i = 4 levels.

The starting point of an arrow (circle) represents a sampled parameter value, the end point (triangle) represents the perturbed value. Together they lead to an Elementary Effect (e.g. Eq (7)). Equal sampling probability means that each discrete parameter value has an equal number of incoming arrows (indicated in red). Above: optimal choice for the step size |δ_i| = p_i/[2(p_i − 1)] = 2/3 leads to equal sampling probabilities. Below: non-optimal choice for |δ_i| leads to a higher probability of sampling interior points.

https://doi.org/10.1371/journal.pone.0293344.g001

This formulation restricts the choice of parameter bounds and number of levels for Boolean and integer parameters; for the former one must set p_i = 2, while for the latter, the following relation must be satisfied to ensure integer parameter values: (5)

If a parameter x_i takes values on the unit interval, an elementary effect of x_i on an output Y_j is given by the finite difference (6) where x = (x₁, …, x_k). Here, the superscript n is an index to distinguish different x, to emphasize that the elementary effect can be calculated at numerous points in the parameter space. δ_i is a predetermined value in the set {±1/(p_i − 1), ±2/(p_i − 1), …, ±1} such that x_i + δ_i still lies in [0, 1] (see S1 Appendix in S1 File). Morris [18] argues the optimal value for the step size |δ_i| is p_i/[2(p_i − 1)], where p_i is chosen to be even. This ensures equal sampling probabilities for all discrete parameter values, as shown in Fig 1. Note that this may necessitate the use of parameter-dependent values of δ and p; for Boolean parameters, one must choose p_i = 2, δ_i = 1, but for real and integer inputs a higher number of levels (hence δ_i ≠ 1) is typically preferred. In some cases, e.g. for an integer input parameter x_i with min_i = 1 and max_i = 3, Eq (5) shows it is not possible to use the optimal value for δ as p_i must be odd. To our knowledge, the current literature assumes a fixed (even) value of p and the optimal value for δ for all parameters.

For dimensional inputs and arbitrary input ranges we introduce the following generalized form of the elementary effect: (7) Here Δ_i = (max_i − min_i)δ_i. The effect given by Eq (7) is dimensional with units .

Note that the dimension of Δ_i is equal to the dimension of X_i. Hence, even if all p_i’s and all input parameter ranges are equal, thereby equalizing the magnitude of each Δ_i, one should still refrain from dropping the index, because the units of the Δ_i’s might be different.

The total number of elementary effects associated with input X_i (and output Y_j) is equal to the number of parameter points for which x_i ≤ 1 − |δ_i|. Those are the points for which an increase by |δ_i|, the other point needed for the calculation of an effect, still lies in the parameter space (see Fig 1). There are p_i − |δ_i|(p_i − 1) discrete values that x_i may take that fulfil x_i ≤ 1 − |δ_i| (all values except those larger than |δ_i|) and (8) combinations for the other parameter values, so the total number of elementary effects for input X_i is equal to (9) This result reduces to the one in [18] (p^k−1[p − |δ|(p − 1)]) if all the p_i’s and δ_i’s are equal. The goal in the original formulation is to estimate the distributions of these effects for each combination of input and output. Following Morris [18], these distributions are denoted by F_ij, where the first index depicts the input and the second the output. If there is only one output under consideration, we simply write F_i. It is generally not feasible (nor desirable) to calculate every possible effect; for k = 50 input parameters and p = 4 and δ = 2/3 for each input, this would amount to ∼10²⁹ simulations (Eq (9)). Instead, the goal is to generate a small set of Q = r(k + 1) simulation points, typically Q ∼ 1000, that still provide good coverage of the parameter space. Each F_ij is then characterized by its sample mean and sample standard deviation over r effects [18] (see Sec. 3 for more detail).

2.2 Trajectory generation using an optimized winding stairs design and fixed step sizes

The most naive way of sampling a set of r effects for each of the k inputs would be to randomly sample r base points in Ω. Since the calculation of each effect requires two output values, this would require a total number of 2rk simulations. However, by generating trajectories in parameter space (Figs 2 and 3) and using each point (except for the start and end) for the calculation of not one, but two effects, the number of required simulations decreases to r(k + 1). This approach is an example of a ‘winding stairs’ design (Fig 2). Alternatively, one can use a ‘radial design’ (Fig 4), leading to star-shaped trajectories in parameter space (described further in Section 2.3).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Realisation of trajectories in a winding stairs design with (red) and without (green) random column permutations.

Here, k = 3 and we set p = 4 and δ = 2/3 for all parameters. Effects are calculated using both endpoints of the arrows, as indicated in green. a) Trajectories in the discrete unit hyperspace Ω. Green is without column permutation, red is with column permutation. b) Trajectory (with column permutation) in the actual parameter space. X₁, X₂ and X₃ take values in [0, 4/3], [1/3, 1] and [0, 1], respectively.

https://doi.org/10.1371/journal.pone.0293344.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Example of an optimal set of trajectories for k = 2,r = 4, p = 4 and |δ| = 2/3 for both parameters.

https://doi.org/10.1371/journal.pone.0293344.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Radial design sample in the unit cube with k = 3 parameters.

https://doi.org/10.1371/journal.pone.0293344.g004

The following is an adaptation of the description by Morris [18]; notation differs slightly, and we account for allowing parameter-dependent step sizes δ_i. All calculations in this section are done on Ω, the discrete unit hypercube, using the scaled dimensionless quantities (x_i, δ_i, etc.). After a trajectory is generated on Ω, one simply transforms it to the actual parameter space using Eq (2). A winding stairs trajectory is a semi-random walk through Ω which has the following properties:

• there is exactly one value change in each dimension;
• the value in dimension i changes from x_i to x_i + δ_i or vice versa, with equal probability;
• the order in which dimension steps are taken is semi-random. Starting with the order sequence [x₁, x₂, …, x_k] (meaning the first step is in the x₁-direction, the second in the x₂-direction, and so forth), all elements are randomly permuted, but only with other elements that have the same corresponding δ_i and p_i. For example, if we have 3 input parameters, where δ₁ = δ₂ ≠ δ₃ and p₁ = p₂ ≠ p₃, there are two (2!) possible sequence orders: [x₁, x₂, x₃] and [x₂, x₁, x₃].

A winding stairs trajectory T_n can be fully characterized by the matrix , given by: (10)

Row j represents the j-th scaled dimensionless parameter point in the trajectory, while column i refers to the value of scaled dimensionless parameter x_i. J_n,m is the n × m matrix of ones, diag(|δ|) is the k × k diagonal matrix containing the (scaled dimensionless) step sizes |δ_i| and P is a k × k random column permutation matrix in which columns i and j may be permuted only if the corresponding parameters have the same number of levels, i.e. if p_i = p_j. D is a k × k diagonal matrix where each diagonal element is either +1 or -1 with equal probability. x_init is a 1×k row vector containing an initial scaled dimensionless parameter point, randomly sampled from the restricted subset of Ω denoted by {0, 1/(p_i − 1), …, 1 − |δ_i|}^k (to ensure that x_i + |δ_i| still lies in the unit interval), where the power denotes a Cartesian product. Finally, the (k + 1) × k matrix B₀ is given by (11)

Note that is a (k + 1) × k matrix where column i is given by either (12) with equal probability and independent of the other columns, i.e. if we ignore the random permutation matrix P, the value of scaled dimensionless parameter i would either start at x_init,i or x_init,i + |δ_i|, change to x_init,i + |δ_i| or x_init,i after the i-th step in the trajectory, respectively, and then remain the same in the rest of the steps. Thus, the permutation matrix randomly changes the sequence of parameter steps. Fig 2 depicts two trajectories, without and with random column permutations. Multiple trajectories can be put in matrix form by simply concatenating the r -matrices: (13)

The question now is how to create a good coverage of the parameter space with a relatively low number of trajectories. Campolongo et al. [19] introduced a simple but effective strategy called ‘optimized trajectories’, (OT) which is described here, as it is commonly used: M random trajectories are generated, typically M ≈ 1000, and the subset of size r ≪ M with the largest dispersion in the input space is selected. Typically, r ranges between 4–20 (likely due to historical computational limitations), although recent papers [27, 28] indicate it might be worthwhile to increase this number at the expense of additional runtime. The notion of spread of a set of trajectories is defined via the following measure of distance between two trajectories: (14) where denotes the i-th point of the m-th trajectory, i.e. d_ml is the sum of the geometric distances between all the couples of points of two trajectories [19]. The optimal set of trajectories is then found using a brute force approach, by considering the measure of spread given by (15) for all combinations of r trajectories out of M, denoted by the vector of trajectory indices (k₁, …, k_r), where k_i ∈ {1, …, M} and k₁ < … < k_r. Campolongo et al. [19] use the square root of this quantity, but that does not affect the location of the maximum. Finally the set of r trajectories with the highest value of is selected. We simply call this maximal value D in what follows, i.e., (16)

Fig 3 depicts a realisation of the optimized trajectories approach containing r = 4 trajectories for k = 2 scaled dimensionless parameters with p = 4 levels. This process is computationally taxing, since it involves calculating distance measures d_ml and the spread times. Especially the latter figure quickly becomes prohibitively large; for M = 1000 and r = 25, values of need to be calculated. Little computer memory is needed, since only the maximum value D thus far and the new value of need to be stored, and optimal sets of trajectories can be generated beforehand and independently of the actual model simulations. Nevertheless, the brute force approach is not feasible in practice. Khare et al. [21], although they use the tag ‘OT’, actually employ a different method, which we call Efficient Optimized Trajectories (EOT): for each of M initially generated trajectories, generate a set of r trajectories by successively adding those with the highest spread w.r.t. to those already in the set. This leads to M sets of r trajectories from which the set with the highest total spread is selected. In algorithmic terms:

EOT:

Step 1: Generate M trajectories {T¹, …, T^M}

Step 2: for i = 1 to M

     Set ;

     for j = 2 to r

       , where ;

     end

     Save spread .

    end

Step 3: Pick the set of trajectories with the highest spread D_i.

This produces a local spread maximum, which may be less than the global maximum, but greatly reduces computational cost; see Table 1. With this approach, we were able to replicate the computation times reported in Fig 3 of [21]. While none of the well-known papers (such as [19, 29]) explicitly mention this more efficient algorithm, it is likely that most papers have in fact employed EOT instead of the brute force OT approach (Ruano et al. [20] being the exception). We employ EOT in this study.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Runtime for generating the set of trajectories for different combinations of k (number of parameters) and r (number of trajectories).

†: M = 200 trajectories. *: initial pool of M = 500 trajectories. ‘n/a’: these take many weeks to complete (extrapolated from the k = 10, r = 4 case) and are thus not shown. ‘Standard’ Sobol Radial is representative for all QR-based approaches as listed in Section 5. All computations were done on a HP Zbook Studio G4 computer.

https://doi.org/10.1371/journal.pone.0293344.t001

In (E)OT, the trajectory starting points are sampled randomly. Selecting these points through Latin Hypercube Sampling (LHS) or a quasi-random (QR) sequence should increase the spread and coverage of the r selected trajectories. However, exploratory numerical experiments (not shown here) showed no significant change compared to a random sample (see also [19]). Apparently, the benefits of generating a large pool of M trajectories outweigh those of LHS or QR sampling.

2.3 Trajectory generation using a radial design and a QR-sequence (extended to general models)

A popular alternative to generating trajectories in a ‘winding stairs’ approach is to use a radial design [30]. The key difference is that in a radial design steps are taken from the same base point (Table 2). This approach is essentially r OAT-designs with different base points. Each point unequal to the base point differs in exactly one (unique) coordinate from the base point (Fig 4). An important difference between the radial design as described here and in e.g. [29, 30] and the standard winding approach for EE ((E)OT) is that the former no longer makes use of fixed step sizes |δ_i|. Instead, a step size may take any value in (0, x_i] (step to the left) or (0, 1 − x_i] (step to the right), and step sizes in the same direction may differ in magnitude for different base points. As such, the number of levels p_i are obsolete in this method. The steps δ_i (now a function of the specific trajectory) are not predefined, but calculated a posteriori.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Radial sampling design versus winding stairs sampling design without random column permutations.

k inputs are considered here, resulting in k + 1 points in parameter space. The base point is given by (a₁, a₂, …, a_k). In OT the a_i are elements from a discrete set and b_i = a_i ± |δ_i|, whereas in the radial design (as in [30]) a_i and b_i can take any value in [0, 1]. Table adapted from [30].

https://doi.org/10.1371/journal.pone.0293344.t002

To ensure a uniform distribution of the r base points in the parameter space, a quasi-random (QR) or low-discrepancy sequence is typically used; see e.g. [29, 30] for examples in radial EE, and [31, 32] and Section 5.1 for more on QR sequences in general. QR sequences are designed to produce point sets that cover a space both efficiently (i.e. with a low amount of points) and evenly (i.e. approximating a uniform distribution).

Sobol sequences are the most popular choice of QR sequence. These sequences use polynomials over the field to form successively finer uniform partitions of the unit interval and then reorder the coordinates in each dimension. To initialize the algorithm, a set of so-called direction numbers is required; we use those provided by Joe and Kuo [33]. The built-in function SobolSequenceGenerator in the Apache Commons Math 3.6.1 Java library is used to generate QR vectors. We do not give a detailed description here, but refer to [33–35], Chapter 5, for details.

We also consider the recently presented R_d sequences [36], which may have more favourable properties of rapid and uniform coverage [36, 37]. As far as we are aware, this sequence has not been used in GSA so far. The R_d sequence in k dimensions is simply given by (17) where α₀ is a fixed offset ( in this work) and in which ϕ_k is the unique positive root of the generalized golden ratio equation

For numerical purposes, ϕ_k can either be estimated through Newton iteration, or by truncating the identity

The QR sequence of choice is used to generate a set of r base points (as r × k matrix A) and a set of r perturbation vectors (r × k matrix B). In principle, there are two ways of filling the matrices A and B. One can generate a r × 2k-matrix, where each row is an element of a 2k-dimensional QR sequence, and subsequently set A to be the left half of the matrix and B the right half. This is the correct approach, and is used in [29, 30] and in this work. Alternatively, one can generate a 2r × k-matrix by concatenating 2r elements of a k-dimensional QR sequence, and use the top half for A and the bottom half for B. However, we found that this leads to erratic and non-converging behavior (errors in preliminary tests (not shown here) did not decrease with increasing r), so this approach should be avoided. It is not exactly clear why this happens, but it is likely to be related to the fact that subsequent points in a QR sequence (hence, subsequent rows in A and B) are dependent. In the case of Sobol QR, Campolongo et al. [29] note it might be worthwhile to use a shifted perturbation vector, i.e. to generate a (r + q) × 2k-matrix, and match base point A_i (i = 1, …, r) with perturbation vector B_i+q, i.e.,

In particular q = 4 is reported to give “good results” [29], which is adopted in this work. Nevertheless, in Sobol QR it may happen that one of the elements of the perturbation vector coincides with its base point counterpart, i.e. A_ij = B_i+q,j for some j = 1, …, k. The row vector [A_i+q, B_i+q] is discarded and regenerated when this happens.

For integer and Boolean inputs, arbitrary step sizes are not allowed, since they may lead to non-integer/Boolean sampling points (Eq (5)). To ensure allowable base points, we propose using the number of levels p_i and step size |δ_i| from the OT approach to pin the base point coordinates for integer/Boolean inputs to a discrete value as in Eq (3) and then use corresponding step size |δ_i|. That is, given a base coordinate x_i in [0, 1] generated by a QR sequence, we transform the coordinate to a discrete value by: (18)

While this quantity is not necessarily integer/Boolean, the coordinate in the actual parameter space is (using Eq (5)): for some , which is an integer/Boolean. Pinning the perturbed coordinate does not work, because there is a high probability (1/p_i) the pinned perturbed coordinate will coincide with the base point coordinate (leading to a step size of 0 and an undefined elementary effect in that direction). For example, if p = 4 the base coordinate x_i = 0.3 will be pinned to , but any perturbed coordinate in [1/4, 1/2] will be pinned to this same value. Therefore, we step with fixed step size |δ_i| (or |Δ_i|) (in the direction that keeps the perturbed point in the parameter space).

To distinguish the sampling strategies described here when testing them in Section 5, we refer to the radial design where all points are generated with a Sobol QR sequence by ‘standard’ Sobol radial and the equivalent using an R_d QR sequence by ‘standard’ R_d radial. Corresponding winding designs are indicated by the postfix winding instead of radial:‘standard’ Sobol winding and ‘standard’ R_d winding. Moreover, as a computationally efficient alternative to EOT (see Table 1), one could use QR sequences to generate the base points, and then transform them regardless of type (real, integer, Boolean) to a discrete value as in (3) (for a given chosen p_i) and step with fixed step size |δ_i| in a radial or winding design, as described above. These approaches are denoted by the prefix ‘pinned’ instead of ‘standard’.

2.4 Alternative approaches to trajectory generation

To further address the computational expense of the combinatorial optimization problem in Campolongo’s optimal trajectory strategy [19], to enhance uniform coverage of the parameter space and to further increase accuracy in sensitivity rankings, a number of alternative sampling designs have been proposed, discussed here for completeness.

Ruano et al.’s modified optimal trajectories scheme (MOT) [20] checks only a subset of all possible trajectory sets, leading to a considerable reduction of computation time (compared to OT and EOT) but at the cost of deteriorating parameter space coverage (compared to both OT and EOT) [21]. Khare et al. [21] introduce Sampling for Uniformity (SU), which aims to generate simulation points that are close to the asserted input parameter distributions, whilst also maximizing trajectory spread. SU outperforms EOT and MOT in some benchmark tests on computation time, uniformity and screening effectiveness, but scores lower on maximizing trajectory spread. Khare et al. [21] also list a number of older approaches. For a number of recent approaches (including cluster sampling), the reader is referred to [26, 38–41] and the references therein.

4.1 Scaling effects in X_i-direction

The necessity of scaling effects in the X_i-direction in dimensional models, or in models where parameters take values on non-unit intervals, becomes evident with the following two examples. Firstly, let be the output of interest. Assume both X₁ and X₂ are dimensional. It follows that their dimensions are [X₁] and [X₁]², respectively. By definition of an elementary effect we have (31) and similarly (32) where X₁ and X₂ are arbitrary values. The first effect has dimension [X₁], so the magnitude of the effect depends on the chosen units. On the other hand, the effect of the second parameter is a dimensionless constant. The same holds for the measures by which parameters are typically ranked (effect mean (Eq (19)) and standard deviation (Eq (20))): μ₂ = 1 and σ₂ = 0, while the measures for X₁ depend on the units of that parameter. In other words, if one does not scale the effects, one can choose units such that μ₁ ≪ μ₂ (e.g. km), making it appear like a parameter is relatively unimportant, but one can just as well select units such that the opposite is true (e.g. mm), indicating the parameter is in fact the most important one.

Secondly, let Y = X₁ + X₂, with X₁ ∈ [0, 20] and X₂ ∈ [9, 11], so that the inputs have equal mean but different standard deviation. Clearly X₁ contributes most significantly to the variability in the output. However, the unscaled effects for both parameters equal 1. Only by scaling can we obtain results consistent with our notion of sensitivity; see Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. The right scaling must be chosen so that the results agree with the notion of sensitivity.

EE applied to Y = X₁ + X₂, with X₁ ∈ [0, 20] and X₂ ∈ [9, 11], p₁ = p₂ = 4 and |δ₁| = |δ₂| = 2/3. Four trajectories are considered, as depicted in Fig 3. The effects are scaled by , or . In all cases σ_i = 0.

https://doi.org/10.1371/journal.pone.0293344.t003

4.2 What scaling to use in X_i-direction?

The following does not require dimensionality of the model, but only supposes that the inputs take values on non-unit intervals.

Again let Y = X₁ + X₂, with X₁ ∈ [0, 20] and X₂ ∈ [9, 11] uniformly. Take p₁ = p₂ = 4 and |δ₁| = |δ₂| = 2/3 and consider the 4 trajectories as depicted in Fig 3. As is shown in Table 3, scaling by the distributional mean of the input and the distributional standard deviation of the input or parameter range results in different rankings. Scaling by the distributional mean gives that both parameters contribute equally to the output mean, when our notion of sensitivity should require that Y is more sensitive to X₁. On the other hand, scaling by the distributional standard deviation or input parameter range does show that X₁ contributes most significantly to the variation in the output. We thus conclude, based on the examples presented here and existing literature (e.g. [11, 42]), that scaling by a function of the input range () gives the desired results. The simplest such scaling is , which is used here. There might be cases where another scaling might be preferred. For instance, if the input parameter is normally distributed (∼ (μ, σ)), it might make more sense to scale by the standard deviation. Alternatively one could think about how to systematically set max_i/min_i, e.g. as μ ± 2σ. This is not further explored in this work. Importantly, there is a significant drawback of scaling by a property of the input distribution, thereby making the effects directly dependent on this property, when there is uncertainty about that distribution.

4.3 Scaling effects in output direction

Scaling in the output direction does not affect relative results for a given output, since it just amounts to a multiplication of all effects by the same constant. Reasons for scaling nevertheless are i) to non-dimensionalize the effects and/or ii) to normalize the effects or measures to enable comparisons between outputs. The key difference between scaling in the input and output direction is that the inputs have a known (albeit assumed) distribution, while the outputs have an unknown distribution. This means that one can use sample-independent scalings for the inputs, such as the range, distributional mean or standard deviation, which is desirable when constructing a consistent sensitivity measure. For the outputs, one is limited to scalings that depend on sample-dependent values. One can think of mean or standard deviation of the model outputs at the sampled points, or the difference between smallest and largest output value across the sampled space. Alternatively, scalings based on empirical data may exist; for a crop model, biomass could be scaled by the mean biomass from field trials, but also this type of scaling is sample-dependent. As such, sensitivity measures involving scaling in the output-direction are best avoided. In the setting of Eq (30) this is equivalent to taking (33) which is done in this work.

4.4 A new sensitivity measure

Many of the alternative measures described in Section 3 (e.g. those proposed in [40, 43]), involve averaging or normalisation over elementary effects or effect measures. While this may be logical for dimensionless models, the summation of quantities with potentially different dimensions such as those in Eqs (22)–(25), cannot be interpreted. Moreover, most measures mentioned in Section 3 lack any scaling of the effects, potentially leading to erroneous ranking results. We therefore propose a synthesis of existing measures, resulting in a scaled, dimensionless and normalized measure agreeing with our notion of sensitivity, whilst preventing erroneous ranking results in dimensional models or models with inputs of arbitrary type and range. Taking either the mean of the absolute effects (; Eq (21)) or the median of the absolute effects (χ_ij, following [28]), we scale (following [11]) by (34) and normalize (following [40]), leading to the sensitivity measures (35) or (36) respectively. Note that . Hence, the measures (35) and (36) are dimensionless, independent of scaling in the output direction and are consistent with our notion of sensitivity. Furthermore, the measures take values in [0, 1] and sum to unity (for each output). This allows for the standardized way of identifying the (un)important parameters as described by Wu [40] (Eq (26)). Note that the measures (35) and (36) resemble a discretized version of the differential importance measure introduced by Borgonovo and Apostolakis [45]. Fig 5 shows a visualization of this approach for an example set of 50 sensitivity indices under different unimportance levels (i.e. h-values in Eq (26)).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Visualisation of (un)important parameters and ranking.

Considered here is an example set of sensitivity indices for 50 input parameters. Parameters are ordered based on S_χ (Eq (36)). Vertical lines show the unimportance threshold for h = 5 and 25% using Eq (26), i.e. all parameters to the left of said line are unimportant. Horizontal lines show the corresponding importance thresholds (Eqs (28) and (29)) for these h-values, i.e. parameters above this line are deemed important. Note that for h = 5%, all parameters are either important or unimportant; for h = 25%, there is one parameter that is neither.

https://doi.org/10.1371/journal.pone.0293344.g005

We do not use the standard deviation of the effects in this work, but instead focus on the median or mean of the absolute effects. Nevertheless, an interesting open question is how one should integrate the standard deviation into the analysis. One could for example consider a quantity analogous to the normalized sensitivity measures in Eqs (35) and (36): (37) where σ_i,j is as in Eq (20). The question is how one should reconcile the two rankings (S_μ*(i, j) or S_χ(i, j) and S_σ(i, j)), especially when parameters score high on one but low on the other. The work by Borgonovo and Rabitti [46] might be of interest, as they show is a biased estimator of the Sobol total sensitivity index (in the case of fixed step sizes). We leave this question for further research.

5.1 Spread and discrepancy of sampling strategies

The spread D (Eqs (14 and 15)) measures how far apart trajectories are, but does not necessarily indicate how well the points uniformly cover the parameter space. Discrepancy is a quantity that measures the uniformity of finite point sets [31]. It originated in the field of QR sequences (or low-discrepancy sequences), where the goal is to generate sequences with high uniformity. More information is provided in S2 Appendix in S1 File. In this work we use the L₂-based wrap-around discrepancy W₂ [47, 48], given in closed form by (38) where N denotes the number of points and k is the dimension of the parameter space. The lower the discrepancy, the better the set of points covers the space uniformly.

For sampling strategies w1, w2, r1, r2, r3 and r4, the spread (14) and (15) and discrepancy (38) are calculated for various combinations of k (the number of input parameters) and r (the number of trajectories). In particular, we vary r between 4 and 100 while fixing k to 50 (Fig 6), and vary k between 10 and 150 while fixing r to 6, 10, 20 or 35 (Fig 7). This covers the range of numbers of inputs and trajectories which are used in practice whilst being computationally feasible. Runtime restricts the number of initial trajectories in EOT (especially for large k); in Fig 6 the pool therefore contains M = 500 elements, while in Fig 7 it contains only M = 200 elements; preliminary experiments (not shown here) showed a negligible difference in spread and discrepancy upon enlarging this pool.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Discrepancy (left) and spread (right) for different sampling strategies.

The number of model inputs k = 50, while the number of trajectories r varies. ‘Standard’ Sobol winding has the overall lowest discrepancy, EOT has the overall highest spread.

https://doi.org/10.1371/journal.pone.0293344.g006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Discrepancy (left) and spread (right) for different sampling strategies.

In these figures, the number of model inputs k varies, while the number of trajectories r is fixed. ‘Standard’ Sobol winding has the overall lowest discrepancy, EOT and ‘pinned’ Sobol/R_d radial have the overall highest spread.

https://doi.org/10.1371/journal.pone.0293344.g007

The spread (Eq (16)) is scaled by the number of elements in its expression, i.e., (39)

The discrepancy (Eq (38)) is scaled by the number of points in a trajectory times the expected squared discrepancy of a random uniform sample of size r(k + 1), i.e., (40) These scalings ensure that all results are the same order of magnitude.

5.2 Test functions

In the subsequent two sections, four test functions are used. The K- and G*-functions are two commonly used dimensionless test functions, and are considered in Saltelli et al. [30], whose experiment we revisit. The six-dimensional test function f₆ has been previously presented in [3, 16]. The Penman-Monteith equation [49] is a dimensional equation describing evapotranspiration.

The K-function with k inputs is given by (41) where x ∈ [0, 1]^k uniformly. The G*-function is given by (42) where I[⋅] is the integer part, a_i, α_i > 0 and η_i ∈ [0, 1] for i = 1, …, k. The x_i are assumed to be uniformly distributed in [0, 1]. Table 4 lists the values for a and α for different k. The K-function contains less non-linearity than the G*-function. Likewise, the low-dimensional versions (i.e. with k = 10) are more ‘difficult’ than their high-dimensional counterparts (i.e. k = 75 for the K-function and k = 50 for the G*-function) because the additional parameters are all relatively unimportant; for the G*-function, this is in part due to the choice of constants in Table 4. As both test functions only contain multiplications of inputs, this has a dampening effect on the output. The six-dimensional test function f₆ [3, 16] is given by: (43) (44) (45) (46) (47) (48) (49) where x ∈ [0, 1]⁶ uniformly. Note that this model is purely additive, which causes QR radial and winding methods to produce identical results: f₆(x + δ_i) − f₆(x) = g_i(x_i + δ_i)−g_i(x_i), so effects are the same for a radial and winding design (given the underlying QR sequences are the same). As an example of an environmentally relevant dimensional test case with non-unit input ranges, we consider the Penman-Monteith equation for evapotranspiration [49], given in energy flux rate form by: (50) where Δ_ET is the rate of change of saturation specific humidity with air temperature, A_ET is the difference between net irradiance and ground heat flux (i.e. the available energy), ρ_a is the dry air density, c_p is the specific heat capacity of air, VPD denotes the vapor pressure deficit, g_a represents air conductivity, γ is the psychromatic constant and g_s represents stomatal conductivity. The units and ranges of the input parameters are listed in Table 5.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Values of a and α for the G*-function (Eq (42)) for different numbers of inputs k.

https://doi.org/10.1371/journal.pone.0293344.t004

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Units, input ranges and Sobol total indices of the input parameters of the Penman-Monteith equation for evapotranspiration (Eq (50)).

https://doi.org/10.1371/journal.pone.0293344.t005

5.3 Comparing trajectory generation methods for estimating Sobol total sensitivity indices

Here we compare the ability of different trajectory generation methods to estimate the Sobol total sensitivity index (see S3 Appendix in S1 File for details) for the K-, G*-, and Penman-Monteith functions presented in Section 5.2. While this paper focuses on EE, not Sobol/variance-based SA, this test is valuable because i) analytical values for the Sobol total indices are readily available (or can easily be approximated); and ii) we may expect results to carry over to ranking parameters in the EE framework (S3 Appendix in S1 File). Related work has recently been published by Hoyt and Owen [52], who compare radial and winding schemes in the context of computing the mean dimension (which can be expressed as a sum of Sobol indices).

Results.

For the K-function (k = 10; Fig 8(a)) differences between our ‘standard’ Sobol results and Saltelli et al.’s are caused by using improved direction vectors [33] in the Sobol QR sequence (the equivalent of a pseudo random seed). ‘Pinned’ Sobol radial is not visible in the plot, as the MAEs of this approach were much larger than the plot range (≈ 0.08). Our results do not show that a radial design unconditionally outperforms a (Sobol) winding design, which is reported in [30]; instead it depends on the chosen QR sequence. For this test function, sampling techniques with a small step size (QR radial/winding) are better than those with a large step size (‘pinned’ Sobol, EOT), with MAE values being 7–20 times smaller for small step size methods. Although there is significant variation present in the G*-function (k = 10, Fig 8(b)), indicating that 50 replicates might be low, it is clear that the ‘pinned’ method employing a large step size performs worse. Moreover, for this more non-linear function radial designs have lower errors (25 − 40% lower MAE) than their winding counterparts, which is consistent with [30]. For both the K- and G*-function (k = 10), the R_d sequence performs similar to or worse than Sobol sequences. In the higher-dimensional tests (i.e. K-function with k = 75, G*-function with k = 50 inputs; Fig 9), we discard the R_d-based sampling strategies as these performed equal to or worse than their respective equivalent using a Sobol sequence in the lower-dimensional tests. Surprisingly, ‘pinned’ Sobol radial seems to perform better (up to 70% lower MAE) than ‘standard’ Sobol winding for the G*-function with 50 inputs (as opposed to the k = 10 case), although the standard deviation bounds are large and overlap for low r. Nevertheless, ‘standard’ Sobol radial still results in the lowest MAE (up to 80% lower than ‘pinned’). In the 75-dimensional K-function differences between ‘standard’ Sobol radial and ‘standard’ Sobol winding are negligible, but ‘pinned’ Sobol radial again shows a significantly higher MAE (up to 10 times). Results for the K-function with k = 50 and k = 100 are not shown, but are similar to the k = 75 case. Results for the Penman-Monteith function (Fig 10) show a negligible difference between a radial and winding design and between a Sobol or R_d QR-sequence, but significantly larger errors for large step-size methods (‘Pinned’ Sobol radial and EOT; up to 20 times higher MAE). Finally, our results suggest spread and discrepancy are not useful proxies of trajectory generation strategy performance. Comparing the ordering of the different strategies based on spread/discrepancy (Sec. 5.1, Figs 6 and 7) with those based on the MAE (Figs 8–10), it is clear that the orderings are almost opposite in the case of spread, and there is only a partial agreement in the case of discrepancy (for low k, ‘pinned’ methods are correctly estimated to exhibit higher errors).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. MAE for Sobol total sensitivity index of K-function (a) and G*-function (b); k = 10 parameters.

(a): MAE (Eq (54)) of 50 replicates of the complete experiment with r = 186, 372, 745, 1489 trajectories applied to the K-function (Eq (41)). Green lines show results obtained by Saltelli et al. [30]. In (a), ‘pinned’ Sobol radial produced larger errors than the plot range shown here. (b): Mean MAE (Eq (54)) ± 1 std bounds of 250 (5 ⋅ 50) replicates of the complete experiment applied to the G*-function (Eq (42) and Table 4 with k = 10).

https://doi.org/10.1371/journal.pone.0293344.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. MAE for Sobol total sensitivity index of K-function (k = 75 parameters; (a)) and G*-function (k = 50; (b)).

(a): MAE (Eq (54)) of 50 replicates of the complete experiment with r = 3, 6, 10, 20, 50, 100 trajectories applied to the K-function (Eq (41)). (b): Mean MAE (Eq (54)) ± 1 std bounds of 450 (9 ⋅ 50) replicates of the complete experiment applied to the G*-function (Eq (42) and Table 4 with k = 50).

https://doi.org/10.1371/journal.pone.0293344.g009

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. MAE (Eq (54)) for Sobol total sensitivity indices of the Penman-Monteith function for evapotranspiration (k = 8 parameters; Eq (50)).

MAE is calculated over 50 replicates of the complete experiment with r = 3, 6, 10, 20, 50, 100 trajectories.

https://doi.org/10.1371/journal.pone.0293344.g010

5.4 Comparing parameter importance rankings from estimated Elementary Effects and rigorous Sobol sensitivity

While it may be expected that the ability to accurately estimate (of one approach relative to others) generally translates to an equal ability to rank parameters and/or identify (un)important parameters [56], the latter should be tested separately. Furthermore, estimating Sobol sensitivity indices does not give any information about the performance of EE aggregation methods (e.g. or χ_i). Therefore, we revisit the aforementioned test functions and sampling strategies, but now with a focus on the ability of the EE-based sensitivity measures S_μ* and S_χ to correctly rank the inputs when compared to Sobol sensitivity rankings.

Results.

As expected, almost all sampling strategies are capable of accurately ranking the input parameters of the K-function (both for k = 10 and k = 50) with a low number of trajectories (Figs 11(a), 11(b) and 12(a), 12(b))). The only exceptions are EOT and ‘pinned’ Sobol radial based on the mean of absolute effects for k = 10, which nevertheless still reach Pearson correlations over 0.9. The case of the G*-function is more interesting (Figs 11(c), 11(d) and 12(c), 12(d)). Approaches based on the median χ_i generally perform equal or better than their counterparts based on , especially in ranking important parameters (k = 10: ρ_pearson between 2% lower and 57% higher). For k = 10 (Fig 11(c) and 11(d)) EOT consistently yields among the lowest correlations, both based on and χ_i. Interestingly, ‘standard’ R_d radial gives among the highest correlations for both k = 10 and k = 50, while it was inferior in estimating total sensitivity indices. For the f₆-function the methods employing small step sizes (‘standard’ Sobol/R_d radial or winding) clearly outperform large step size methods (EOT and ‘pinned’ Sobol radial) (Fig 13). There are no clear differences between Sobol and R_d QR sequences. Median-based small step-size approaches result in higher Pearson correlations (up to 34%), but roughly equal Kendall correlations. Results for the Penman-Monteith equation paint a different picture (Fig 14). Approaches based on the mean clearly outperform their counterparts based on the median χ_i. Differences between sampling strategies are small (S_μ*: less than 0.1 difference in ρ_kendall for r > 10), although EOT generally results in the lowest correlations and ‘Pinned’ Sobol radial performs best. Fig 14 also highlights the need for scaling the effects; without scaling, no strategy is capable of accurately ranking the input parameters (all Kendall correlations less than 0.5).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Correlations ρ_kendall (Eq (55)) and ρ_pearson (Eq (57)) between estimated and actual parameter rankings for the K-function (Eq (41); (a)-(b)) and G*-function (Eq (42) and Table 4; (c)-(d)) with k = 10 parameters.

The Pearson correlation assigns more weight to important parameters. The means of the correlation coefficients are shown, based on 50 replicates of the full experiment.

https://doi.org/10.1371/journal.pone.0293344.g011

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. Correlations ρ_kendall (Eq (55)) and ρ_pearson (Eq (57)) between estimated and actual parameter rankings for the K-function (Eq (41); (a)-(b)) and G*-function (Eq (42) and Table 4; (c)-(d)) with k = 50 parameters.

The Pearson correlation assigns more weight to important parameters. The means of the correlation coefficients are shown, based on 50 replicates of the full experiment.

https://doi.org/10.1371/journal.pone.0293344.g012

Download:

• PNG
  larger image
• TIFF
  original image

Fig 13. Correlations ρ_kendall (Eq (55)) and ρ_pearson (Eq (57)) between estimated and actual parameter rankings for the f₆-function (Eq (49)).

The Pearson correlation assigns more weight to important parameters. The means of the correlation coefficients are shown, from 50 replicates of the full experiment. Note that ‘standard’ Sobol radial (yellow) and winding (green) produce identical results (given a sensitivity measure), as the test function is purely additive; only the yellow line is visible.

https://doi.org/10.1371/journal.pone.0293344.g013

Download:

• PNG
  larger image
• TIFF
  original image

Fig 14. Correlations ρ_kendall (Eq (55)) and ρ_pearson (Eq (57)) between estimated and actual parameter rankings for the Penman-Monteith function for evapotranspiration (Eq (50)).

The Pearson correlation assigns more weight to important parameters. The means of the correlation coefficients are shown, from 50 replicates of the full experiment. The middle figure shows ρ_kendall using unscaled effects, leading to incorrect rankings.

https://doi.org/10.1371/journal.pone.0293344.g014

To summarize, small step size methods (‘standard’) generally perform better than or equal to large step size methods (EOT/‘pinned’). There is no one sensitivity measure that always results in the highest correlations; our results indicate S_μ* (based on the mean of absolute effects) might in some cases be preferable, but in other cases S_χ (based on the median of absolute effects) yields higher correlations between analytical and calculated rankings. Notably, ‘standard’ R_d radial performs well across the range of r tested for the G*-function, making it an interesting trajectory generation method for further research. Finally, the results in this section are further proof that spread and discrepancy are poor proxies of trajectory generation method performance (see Sec. 5.1 and Figs 6 and 7).