2.1 Model formulation

When applied to a biochemical system, an ODE model describes the temporal evolution of abundances of n_x different molecular species x_i. The temporal evolution of x is determined by the vector field f and the initial condition x₀: (1)

Both f and x₀ depend on the unknown parameters such as catalytic rates or binding affinities. Restricting optimization to the parameter domain Θ can constrain the parameter search space to values that are realistic based on physicochemical theory and helps prevent numerical integration failures associated with extreme parameter values. For most problems, Θ is the tensor product of scalar search intervals (l_i, u_i) with lower and upper bounds l_i < u_i that satisfy for every parameter θ_i.

Experiments usually provide information about observables y which depend on abundances x and parameters θ. A direct measurement of x is usually not possible. Thus, the dependence of observables on abundances and parameters is described by (2)

Implementation in this study: All methods described here use CVODES from the SUNDIALS suite [32] for numerical integration of model equations. CVODES is a multi-step implicit solver for stiff- and non-stiff ODE initial value problems.

2.2 Optimization problem

To generate models useful in the study of actual biological systems, model parameters θ must be inferred from experimental data, which are typically incomplete and subject to measurement noise. A common assumption is that the measurement noise for n_t time-points t_j and n_y observables y_i is additive, independent and normally distributed for all time-points: (3)

Thus, model parameters can be inferred from experimental data by maximizing the likelihood, yielding a maximum likelihood estimate (MLE). However, the evaluation of the likelihood function involves the computation of several products of large terms, which can be numerically unstable. Thus, the negative log-likelihood (4) is typically used as objective function that is minimized. As the logarithm is a strictly monotonically increasing function, the minimization of J(θ) is equivalent to the maximization of the likelihood. Therefore, the corresponding minimization problem (5) will infer the MLE parameters. If the noise variance does not depend on the parameters θ, the objective function (4) has a weighted least-squares formulation. As we discuss later, properties of a least-squares formulation can be exploited in specialized optimization methods. Optimizers that do not require least-squares structure can also work with other noise models [33].

Implementation in this study: For the MATLAB optimizers fmincon and lsqnonlin, the objective function and its derivatives were evaluated using data2dynamics [34] (commit b1e6acd), which was also used in the study by Hass et al. [15]. For the Python optimizers ls_trf and fides, the objective function and its derivates were evaluated using AMICI [35] (version 0.11.23) and pyPESTO (version 0.2.10).

2.3 Trust-region optimization

Trust-region methods minimize the objective function J by iteratively updating parameter values θ_k+1 = θ_k + Δθ_k according to the local minimum (6) of an approximation m_k to the objective function. Δ_k is the trust-region radius that restricts the norm of parameter updates. The optimization problem (6) is known as the trust-region subproblem. In most applications, a local, quadratic approximation (7) is used, where f_k = J(θ_k) is the value, g_k = ∇J(θ_k) is the gradient and B_k = ∇²J(θ_k) is the Hessian of the objective function evaluated at θ_k.

The trust-region radius Δ_k is updated in every iteration depending on the ratio ρ_k between the predicted decrease and actual decrease in objective function value [19]. The step is accepted if ρ_k exceeds some threshold μ ≥ 0. When boundary constraints (on parameter values) are applied, the predicted decrease is augmented by an additional term that accounts for the parameter transformation (see Section 2.6) [31].

Implementation in this study: All optimizers evaluated in this study use μ = 0 as acceptance threshold. They all increase the trust-region radius Δ_k by a factor of 2 if the predicted change in objective function value is accurate (ρ_k > 0.75) and the local minimum is at the edge of the trust region ( for fmincon, lsqnonlin and fides, for ls_trf). All optimizers decrease the trust-region radius if the predicted change in objective function value is inaccurate (ρ < 0.25), but fmincon, lsqnonlin and fides set the trust-region radius to , while ls_trf sets it to .

When the predicted objective function decrease is negative (, i.e., an increase in value is predicted) fides and ls_trf set ρ_k to 0.0. Positive values for may arise from the augmentation accounting for boundary constraints. Setting ρ_k to 0.0 prevents inadvertent increases to Δ_k or step acceptance when and ΔJ are both negative. However, in contrast to fides, ls_trf does not automatically reject respective step proposals and only does so if ΔJ < 0. fmincon and lsqnonlin only reject step proposals with ΔJ < 0, but do not take the sign of into account when updating Δ_k.

When the objective function cannot be evaluated—for ODE models this is typically the result of an integration failure—all optimizers decrease the trust-region radius by setting it to (fmincon and lsqnonlin), (fides) or (ls_trf). These subtly nuanced differences in the implementation are likely the result of incomplete specification of the algorithm in the original publication [31]. In particular, handling of , which does not occur for standard trust-region methods, was not described and developers needed to independently work out custom solutions.

2.4 Hessian approximation

Constructing the local approximation (7) that defines the trust-region subproblem (6) requires the evaluation of the gradient g_k and Hessian B_k of the objective function at the current parameter values θ_k. While the gradient g_k can be efficiently and accurately computed using first order forward or adjoint sensitivity analysis [36], it is computationally more demanding to compute the Hessian B_k [37]. Therefore several approximation schemes have been proposed that approximate B_k using first order sensitivity analysis. In the following we will provide a brief description of approximation schemes considered in this study, an overview of schemes and their characteristics is provided in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Overview of properties of different Hessian approximation schemes.

BFGS is the Broyden-Fletcher-Goldfarb-Shannon algorithm. SR1 is the Symmetric Rank-one update. GN is the Gauss-Newton approximation. SSM is the Structured Secant Method. TSSM is the Totally Structured Secant Method. FX is the hybrid method proposed by Fletcher and Xu [45]. GNSBFGS is the Gauss-Newton Structured BFGS method. The construction column indicates whether pointwise evaluation is possible or whether iterative construction is necessary. The positive semi-definite column indicates whether the approximation preserves positive semi-definiteness given a positive semi-definite initialization.

https://doi.org/10.1371/journal.pcbi.1010322.t002

Gauss-newton approximation: The Gauss-Newton (GN) approximation is based on a linearization of residuals r_ij (8) which yields a symmetric and positive semi-definite approximation to B_k and does not account for negative curvature. At the maximum likelihood estimate, B^(GN) is equal to the negative empirical Fisher Information Matrix assuming σ_ij does not depend on parameter values θ. For parameters dependent σ, the log(σ) term in (4) cannot be assumed to be constant, which results in a non least-squares optimization problem. For non least-squares problems, the adequateness and formulation of the GN approximation is not well established. Thus, Raue [38] proposed to transform the problem into a least-squares form by introducing additional error residuals and adding a corresponding correction to the Gauss-Newton approximation B^(GN) from (8), yielding B^(GNe): (9) where C is some arbitrary, but sufficiently large constant so that 2 log(σ_ij(θ)) + C > 0. This condition ensures that residuals are real-valued and the approximation B^(GNe) is positive semi-definite, but inherently makes optimization a non-zero residual problem. Adding the constant C to residuals adds a constant to the objective function value and, thus, neither influences its gradient and Hessian nor the location of its minima. However, C does enter the GNe approximation, with unclear implications. Instead, Stapor et al. [37] suggested that one ignores the second order derivative of the log(σ) term in (4), which corresponds to the limit lim_C→∞ B^(GNe) = B^(GN).

Iterative approximations: In contrast to the GN approximation, Broyden-Fletcher-Goldfarb-Shanno (BFGS) or Symmetric Rank-one (SR1) are iterative approximation schemes, in which the approximation in the next step is constructed based on the approximation in the current step B_k and some update Δ(x, s, B^k), where s = Δθ_k and z = g_k+1 − g_k.

The BFGS update scheme guarantees a positive semi-definite approximation as long as a curvature condition z^Ts > 0 is satisfied and the initial approximation is positive semi-definite [19]. Thus, the update scheme is usually only applied with line search methods that guarantee satisfaction of the curvature condition by selecting the step length according to (strong) Wolfe conditions [19]. However, BFGS can also be used in trust-region methods by rejecting updates when the curvature condition is not satisfied, although this invalidates some theoretical convergence guarantees [19].

The SR1 update scheme can also yield indefinite approximations, incorporating negative curvature information, and has no step requirements beyond ensuring that the denominator of the update is non-zero.

Structured secant approximations: The accuracy of the GN approximation depends on the magnitude of the residuals, since the approximation error is the sum of products of the residuals and the residual Hessians (10)

For non-zero residual problems, in which there are indices i, j such that r_ij(θ*) ≫ 0, or in the presence of strong non-linearities, the second order term in (10) does not vanish and the GN approach is known to perform poorly; it can even diverge [39, 40]. This issue is addressed in structured secant methods [39, 41, 42], which combine the pointwise GN approximation with an iterative BFGS approximation A_k of the second order term. In the Structured Secant Method (SSM) [42], the matrix A_k is update using a BFGS scheme:

Similarly, the Totally Structured Secant Method (TSSM) [43] scales A_k with the residual norm, to mimic the product structure in the second order term in (10), and, accordingly, scales the update to A_k with the inverse of the residual norm:

Despite the use of a BFGS updating scheme, the SSM and TSSM approximations do not preserve positive semi-definiteness, as the matrix is updated at every iteration without any additional safeguards. Structured secant approximations have been popularized by the NL2SOL toolbox [44], but are not featured in the standard optimization libraries in MATLAB or Python.

Hybrid schemes: Other hybrid schemes can dynamically switch between GN approximations and iterative updates when some metric indicates that the considered problem has non-zero residual structure. For example, Fletcher and Xu [45] proposed an approach to detect non-zero residuals by computing the normalized change in the residual norm and applying BFGS updates when the change is smaller than some tolerance ϵ_FX:

As is positive semi-definite and the BFGS updates preserve this property, the FX approximations are always positive semi-definite.

To address the issue of possibly indefinite approximations in the SSM and TSSM approaches, Zhou and Chen proposed a Gauss-Newton structured BFGS method (GNSBFGS) [40]. that combines the TSSM approach with the dynamic updating of the FX approach. As sum of two positive semi-definite matrices, is always positive semi-definite. The authors demonstrate that the term plays a similar role as the term in the FX algorithm. However, they only prove convergence if the tolerance 2ϵ_GNSBFGS is smaller than the smallest eigenvalue λ_min(∇²J(θ*)) of the Hessian at the optimal parameters and if ∇²J(θ*) is positive definite. This condition is not met for problems with singular Hessians, which are often observed for non-identifiable problems.

Implementation in this study: fmincon and lsqnonlin were only evaluated using the GNe approximation, as implemented in data2dynamics. ls_trf can only be applied using the GNe approximation. Fides was evaluated using BFGS and SR1 using respective native implementations in addition to GN and GNe, as implemented in AMICI. We used the default value of C = 50 for the computation of GNe in both data2dynamics and AMICI. We provide implementations for BFGS, SR1, SSM, TSSM, FX and GNSBFGS schemes in fides. FX, SSM, TSSM and GNSBFGS were applied using GNe, as they require a least-squares problem structure. Hyperparameters ϵ_FX = 0.2 and ϵ_GNSBFGS = 10⁻⁶ were picked based on recommended values in respective original publications.

2.5 Solving the trust-region subproblem

In principle, the trust-region subproblem (6) can be solved exactly [19]. Moré proposed an approach using eigenvalue decomposition of B_k [46]. Yet, Byrd et al. [47] noted the high computational cost of this approach and suggested an approximate solution by solving the trust-region problem over a two dimensional subspace , spanned by gradient g_k and Newton search directions, instead of . Yet, for objective functions requiring numerical integration of ODE models, the cost of eigenvalue decomposition is generally negligible for problems involving fewer than 10³ parameters.

A crucial issue for the two-dimensional subspace approach are problems with indefinite (approximate) Hessians. For an indefinite B_k, the Newton search direction may not represent a direction of descent. This can be addressed by dampening B_k [19], but for boundary-constrained problems additional considerations arise and require the identification of a direction of strong negative curvature [31].

Implementation in this study: fmincon and lsqnonlin implement optimization only over , where the Newton search direction is computed using preconditioned direct factorization. For preconditioning and direct factorization, fmincon and lsqnonlin employ Cholesky and QR decomposition respectively, which both implement dampening for numerically singular B_k. fides and ls_trf implement optimization over (denoted by 2D in text and figures) and (denoted by ND in text and figures). For ls_trf, we specified tr_solver = “lsmr” for optimization over . To compute the Newton search direction, ls_trf and fides both use least-squares solvers, which is equivalent to using the Moore-Penrose pseudoinverse. fides uses the direct solver scipy.linalg.lstsq, with gelsd as LAPACK driver, while ls_trf uses the iterative, regularized scipy.sparse.linalg.lsmr solver. In fides, the negative curvature direction of indefinite Hessians is computed using the eigenvector to the largest negative eigenvalue (computed using scipy.linalg.eig).

2.6 Handling of boundary constraints

The trust-region region subproblem (6) does not account for boundary constraints, which means that θ_k + Δθ_k may not satisfy these constraints. For this reason, Coleman and Li [31] introduced a rescaling of the optimization variables depending on how close the current values are to the parameter boundary ∂Θ. This rescaling is realized through a vector which yields transformed optimization variables and a transformed Hessian (11)

As the second term in (11) is positive semi-definite, this handling of boundary constraints can also regularize the trust-region sub-problem, although this is not its primary intent.

Coleman and Li [31] also propose a stepback strategy in which solutions to (6) are reflected at the parameter boundary ∂Θ. Since this reflection defines a one-dimensional search space, the local minimum can be computed analytically at negligible computational cost. To ensure convergence, Δθ_k is then selected based on the lowest m_k(p) value among (i) the reflection of p* at the parameter boundary (ii) the constrained Cauchy step, which is the minimizer of (7) along the gradient that is truncated at the paramtere boundary and (iii) p* truncated at the parameter boundary.

Implementation in this study: fmincon, lsqnonlin and ls_trf all implement rescaling, but only allow for a single reflection at the boundary [48]. In contrast, fides implements rescaling and also allows for a single or arbitrarily many reflections until the first local minimum along the reflected path is encountered.

2.7 Optimizer performance evaluation

To evaluate optimization performance, Hass et al. [15] computed the success count γ, which represents the number of “successful” optimization runs that reached a final objective function value sufficiently close (difference smaller than some threshold τ) to the lowest objective function value found by all methods, and divided that by the time to complete all optimization runs t_total, a performance metric that was originally introduced by Villaverde et al. [49]. In this study, we replaced t_total with the total number of gradient evaluations across all optimization runs n_grad for any specific optimization setting. The resulting performance metric ϕ = γ/n_grad ignores differences in computation time for gradients having different parameter values and prevents computer or simulator performance, node load and parallelization from influencing results. This provides a fairer evaluation of the algorithm or method itself and is particularly relevant when optimization is performed on computing clusters with heterogeneous nodes or when different number of threads are used to parallelize objective function evaluation, as it was the case in this study. Since ϕ ignores potentially higher computation times for step-size computation, we confirmed that step-size computation times were negligible as compared to numerical integration of model and sensitivity equations. For all trust-region optimizers we studied, the number of gradient evaluations was equal to the number of iterations, with the exception of ls_trf which only uses objective function evaluations when a proposed step is rejected. Thus, 1/n_grad is equal to the average number of iterations for the optimization to converge (divided by the number of optimization runs, which is 10³ in all settings). We therefore refer to ν = 1/n_grad as the convergence rate. Performance ϕ is equal to the product of γ and ν.

To calculate γ, we used a threshold of τ = 2, which corresponds to the upper limit of the objective function value in cases in which a model cannot be rejected according to the AIC [50]. Similar to Hass et al. [15], we found that changing this convergence threshold did not have a significant impact on performance comparison, but provide analysis for values τ = 0.05 (threshold used by Hass et al., divided by two to account for difference in objective function scaling, Fig A in S1 Text) and τ = 5 (the threshold for rejection according to the AIC and BIC [21], Fig B in S1 Text) in the Supplementary Material.

2.8 Extension of boundary constraints

For some performance evaluations, we extended parameter boundaries. Even though initial points are usually uniformly sampled in Θ, we did not modify the locations of initial points when extending bounds. The Schwen (Table 3), problem required a different approach in which the bounds for the parameter fragments were not modified, as values outside the standard bounds were implausible.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of problem characteristics for benchmark examples, as characterized by Hass et al. [15].

https://doi.org/10.1371/journal.pcbi.1010322.t003

As previously reported [15], extending boundaries can expose additional minima having globally lower objective function values. Thus, success count γ for optimization settings with normal boundaries were computed using the lowest objective function J_min found among all settings excluding those with extended boundaries. γ for optimization settings using extended boundaries were computed using the minimum of J_min and the lowest objective function value found for that particular setting.

2.9 Statistical analysis of optimizer traces

During the statistical analysis of optimizer traces, we quantified several numerical values derived from numerical approximation of matrix eigenvalues with limited accuracy (due, for example, to limitations in floating point precision). It was therefore necessary to account for this limitation in numerical accuracy:

Singular Hessians: To numerically assess matrix singularity of Hessian approximations, we checked whether the condition number, computed using the numpy function numpy.linalg.cond, was larger than the inverse of the floating point precision ϵ = 1/numpy.spacing(1).

Negative eigenvalues: To numerically assess whether a matrix has negative eigenvalues we computed the smallest (λ_min) and largest (λ_max) eigenvalues of the untransformed Hessian approximation B_k using numpy.linalg.eigvals and checked whether the smallest eigenvalue had a negative value that exceeded numerical noise λ_min(B_k) < −ϵ ⋅ |λ_max(B_k)|.

2.10 Implementation

fides is implemented as modular, object-oriented Python code. The subproblem, subproblem solvers, stepback strategies and Hessian approximations all have class-based implementations, making it easy to extend the code. Internally, fides uses SciPy [51] and NumPy [52] libraries to store vectors and matrices and perform linear algebra operations. To ensure access to state-of-the-art simulation and sensitivity analysis methods, we implemented an interface to fides in the parameter estimation toolbox pyPESTO, which uses AMICI [35] to perform simulation and sensitivity analysis via CVODES [32]. This approach also enabled import of biological parameter estimation problems specified in the PEtab [53] format.

2.11 Benchmark problems

To evaluate the performance of different optimizers, we use the benchmark problems (Table 3) introduced by Hass et al. [15]. As discussed in the introduction, these problems are excellent representatives of ODE-based biochemical models and include realistic experimental data for model calibration. We selected a subset of 13 out of the 20 models based on whether they can be encoded in the PEtab [53] format and imported in AMICI and pyPESTO. The exclusion of some models in Hass et al. [15] does not reflect a limitation of fides itself, as it supports optimization for any objective function that provides routines to compute its gradient. In summary, the Hass problem was excluded because it includes negative initial simulation time, which is not supported by PEtab; the Raia model because it involves state-dependent value for σ, which is unsupported by AMICI; Merkle and Sobotta because they are missing SBML implementations; Swameye because it includes spline functions that are not supported by SBML; Becker because it involves multiple models, which is not supported by pyPESTO; and Chen because forward sensitivity analysis is prohibitively computationally expensive for this model. There is no evidence that the excluded models are different in any systematic way from the models we do consider.

All benchmarks problems were previously published and included experimental data for model calibration as described by Table 3, which also provides a brief summary of numerical features of the various benchmarks. These problems cover a wide array of common model features such as preequilibration, log-transformation of observables as well as parameter dependent initial conditions, observable function and noise models. A more detailed description of the biochemical systems described by these models is available in the supplemental material of the study by Hass et al. [15].

2.12 Simulation and optimization settings

We encountered difficulties reproducing some of the results described by Hass et al. [15] and therefore repeated evaluations using the latest version of data2dynamics. We deactivated Bessel correction [15] and increased the function evaluation limit to match the iteration limit. Relative and absolute integration tolerances were set to 10⁻⁸. The maximum number of iterations for optimization was set to 10⁵. Convergence criteria were limited to step sizes with a tolerance of 10⁻⁶, and ls_trf code was modified such that the convergence criteria matched the implementation in other optimizers. For all problems, we performed 10³ optimizer runs. To initialize optimization, we used the initial parameter values provided by Hass et al. [15].

2.13 Parallelization and cluster infrastructure

Optimization was performed on the O2 Linux High Performance Compute Cluster at Harvard Medical School, a typical academic High Performance Compute resource running SLURM. O2 includes 390+ compute nodes and 12,000+ compute cores, with the majority of the compute nodes built on Intel architecture; all nodes run CentOS 7.7.1908 Linux with MATLAB R2017a and python 3.7.4. Optimization for each model and each optimizer setting was run as a separate job. For MATLAB optimizers, optimization was performed using a single core per job. For Python optimizers, execution was parallelized on up to 12 cores. Parallelization was always carried out on an individual node, avoiding inter-node communication.

We observed severe load balancing issues due to skewed computational cost across optimization runs for Bachmann, Isensee, Lucarelli and Beer models. To mitigate these issues, optimization for these models was parallelized over 3 threads using pyPESTOs MultiThreadEngine and simulation was parallelized over 4 threads using openMP multithreading in AMICI, resulting in a total parallelization over 12 threads. For the remaining models, optimization was parallelized using 10 threads without parallelization of simulations. Wall-time for each job was capped at 30 days (about 1 CPU year), which was only exceeded by the GNSBFGS and FX Hessian approximation schemes for the Lucarelli problem after 487 and 458 optimization runs respectively and also by the SSM Hessian approximation scheme for the Isensee problem after 973 optimization runs. Subsequent analysis was performed using partial results for those settings.

Compute times could not be reliably compared across methods and problems because it was necessary to use different degrees of parallelization for different problems and optimizations were run on different nodes with distinct processor models. We therefore evaluated performance based on the number of optimizer iterations. The number of iterations is independent of the degree of parallelization and processor models, but for any given implementation, the compute time will be proportional to the number of optimizer iterations.

3.1 Validation and optimizer comparison

The implementation of trust-region optimization involves complex mathematical operations that can result in error-prone implementations. To validate the trust-region methods implemented in fides, we compared the performance of optimization using GN and GNe schemes against implementations of the same algorithm in MATLAB (fmincon, lsqnonlin) and Python (ls_trf). The subspace solvers and Hessian approximation used for each analysis are denoted by the notation implementation subspace/hessian.

We found that fides 2D/GN (blue) and fides 2D/GNe (orange) were the only methods that had non-zero performance (ϕ > 0) for all 13 benchmark problems (Fig 2A), with small performance differences between the two methods (0.72 to 1.12 fold difference, average 0.96). This established fides 2D/GN as good reference implementation. In what follows we therefore report the performance of other methods relative to fides 2D/GN (Fig 2B). The ls_trf method outperformed fides 2D/GN on three problems (1.54 to 22.4-fold change; Boehm, Crauste, Zheng; purple arrows), had similar performance on one problem (1.15-fold change; Fiedler), exhibited worse performance on four problems (0.02 to 0.55-fold change; Brannmark, Bruno, Lucarelli, Weber) and did not result in successful runs (zero performance ϕ = 0) for the remaining five problems (Bachmann, Beer, Fujita, Isensee, Schwen). Decomposing performance improvements ϕ into increases in convergence rate ν (Fig 2C) and success count γ (Fig 2D) revealed that increase in ϕ was primarily due to higher ν, which was observed for all but four problems (Bachmann, Bruno, Isensee, Schwen). However, in most cases, improvements in ν were canceled out by larger decreases in γ.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparison of MATLAB and Python optimizers.

Colors indicate optimizer setting and are the same in all panels. A: Performance comparison (ϕ), absolute values. B: Performance comparison (ϕ), values relative to fides 2D/GN. C: Increase in convergence count γ, values relative to fides 2D/GN. D: Increase in convergence rate ν, values relative to fides 2D/GN.

https://doi.org/10.1371/journal.pcbi.1010322.g002

For fmincon (green) and lsqnonlin (red), ϕ was higher for one problem (2.34 to 2.94 -fold change; Fiedler; red/green arrow), similar for two problems (0.92 to 1.00 fold change, Boehm, Bruno) and worse for the remaining 10 problems (0.00 to 0.80 fold change, Bachmann, Brannmark, Fujita, Isensee, Lucarelli Schwen, Weber, Zheng), with zero performance on two problems (Beer, Crauste) (Fig 2B). Since we observed similar ϕ for fides 2D/GN (blue) and fides 2D/GNe (orange) on all problems, the differences in ϕ between fides 2D/GN and the other implementations are unlikely to reflect use of GNe as opposed to a GN scheme. Instead we surmised that the differences were due to discrepancies in implementation of the Newton direction (this would explain the similarity for the two identifiable problems (Boehm, Bruno, Table 3)).

Overall, these findings demonstrated that trust-region optimization implemented in fides was more than competitive with the MATLAB optimizers fmincon and lsqnonlin and the Python optimizer ls_trf, outperforming them on a majority of problems. Simultaneously, our results demonstrate a surprisingly high variability in optimizer performance among methods that implement fundamentally similar mathematical operations (i.e., the same algorithm). This variability may explain some of the conflicting findings in previous studies that assumed that differences in optimizer performance arose from the “mathematics” rather than the computational implementation [54, 55].

3.2 Parameter boundaries and stepback strategies

One of the few changes in implementation that we deliberately introduced into the fides code was to allow multiple reflections during stepback from parameter boundary conditions [31]. In contrast, ls_trf, lsqnonlin and fmincon only allow a single reflection [48]. The modular design and advanced logging capabilities of fides make it straightforward to evaluate the impact of such modifications on optimizer performance ϕ and arrive at possible explanations for observed differences. For example, when we evaluated fides 2D/GN with single (orange) and multi-reflection (dark-green, Fig 3A–3C) implementations and correlated changes to ϕ with statistics of optimization trajectories (Fig 3D and 3E), we found that the single reflection performance ϕ was reduced on four problems (0.59 to 0.65-fold change; Beer, Lucarelli, Schwen, Zheng; orange arrows Fig 3A). Lower performance was primarily due to a decrease in convergence rate ν (Fig 3B and 3C). We attributed this behavior to the fact that a restriction on the number of reflections lowered the predicted decrease in objective function values for reflected steps. This, in turn, increased the fraction of iterations in which stepback yielded constrained Cauchy steps (Pearson’s correlation coefficient r = −0.85, p-value p = 2.3 ⋅ 10⁻⁴, Fig 3D) as well as the average fraction of boundary-constrained iterations (r = −0.82, p = 6 ⋅ 10⁻⁴, Fig 3E), both slowing convergence.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Evaluation of stepback strategies.

Colors indicate optimizer setting and are the same in all panels. All increases/decreases are relative to multi-reflection fides 2D/GN with normal bounds. A: Performance comparison (ϕ). B: Increase in convergence count γ. C: Increase in convergence rate ν. D: Association between increase in average fraction of constrained Cauchy steps with respect to total number of boundary constrained iterations and increase in ν for the single-reflection method. E: Association between increase in average fraction of boundary constrained iterations and increase in ν for the single-reflection method. F: Association between increase in average fraction of iterations with a numerically singular transformed Hessian and decrease in γ for the multi-reflection method for fides 2D/GN with bounds extended by two orders of magnitude. G: Association between increase in average fraction of iterations with integration failures and increase in γ for multi-reflection fides 2D/GN without bounds.

https://doi.org/10.1371/journal.pcbi.1010322.g003

A naive approach to addressing issues with parameter boundaries is to extend or remove them. We therefore repeated optimization with fides 2D/GN (multi-reflection) with parameter boundaries extended by one (blue) or two (pink) orders of magnitude or completely removed (light green). We found that extending boundaries by one order of magnitude reduced ϕ for 6 problems (0.12 to 0.74 fold change; Bachmann, Beer, Fiedler, Isensee, Lucarelli, Weber; blue arrows Fig 3A) and extending boundaries by two orders of magnitude reduced ϕ for an additional 4 problems (0.13 to 0.85 fold change; Bruno, Crauste, Schwen, Zheng; pink arrows Fig 3A). We found that decreased ϕ was primarily the result of lower ν (Fig 3B), which we attributed to a larger fraction of iterations in which the transformed Hessian was singular (r = −0.78, p = 1.6 ⋅ 10⁻³, Fig 3F). Removing boundaries decreased ϕ for all problems, a result of lower values of γ, which we attributed to higher fraction of iterations with integration failures (r = −0.82, p = 5.7 ⋅ 10⁻⁴, Fig 3G).

These findings demonstrate the importance and difficulty of choosing appropriate optimization boundaries, since excessively wide boundaries may lead to frequent integration failures and/or the creation of an ill-conditioned trust-region subproblem. In contrast, using boundaries that are too narrow has the potential to exclude the global optimum. When managing boundary constraints the use of multi-reflection as compared to single-reflection yields a small performance increase, albeit significantly smaller than the variation we observed (in the previous section) between different implementations of the same optimization algorithm.

3.3 Iterative schemes and negative curvature

To further study the positive effect of improving the conditioning of trust-region subproblems on the optimizer performance ϕ, we carried out optimization using BFGS and SR1 Hessian approximations. BFGS and SR1 can yield full-rank Hessian approximations, resulting in well-conditioned trust-region subproblems, even for non-identifiable problems. Moreover, in contrast to GN, these approximations converge to the true Hessian under mild assumptions [19]. The SR1 approximations can also account for directions of negative curvature and might therefore be expected to perform better when saddle-points are present.

We compared ϕ for fides 2D/GN (blue), fides 2D/BFGS (orange) and fides 2D/SR1 (green) (Fig 4A) and found that fides 2D/BFGS failed to reach the best objective function value for one problem (Beer) and Fides 2D/SR1 for two problems (Beer, Fujita). Compared to fides 2D/GN, ϕ for fides 2D/BFGS was higher on three problems (2.02 to 9.88 fold change; Boehm, Fiedler, Schwen; orange arrows) and four problems for fides 2D/SR1 (1.32 to 7.12 fold change; Boehm, Crauste, Fiedler; green arrows), it was lower for a majority of the remaining problems (BFGS 7 of 13 problems, 0.05 to 0.49 fold change; SR1 8 of 13 problems, 0.07 to 0.78 fold change). Decomposing ϕ into improvements in convergence rate ν (Fig 4B) and improvements in success counts γ (Fig 4C) revealed that SR1 improved ν for 7 problems (2.23 to 5.97 fold change; Beer, Boehm, Crauste, Fiedler, Fujita, Weber, Zheng; green arrows Fig 4B). Out of these 7 problems, BFGS improved ν for only four problems (2.33 to 6.63 fold change; Beer, Boehm, Fiedler, Zheng; orange arrows Fig 4B). We found that for both approximations, the increase in ν was correlated with the change in average fraction of iterations without trust-region radius (Δ_k) updates (BFGS: r = −0.8, p = 1.1 ⋅ 10⁻³; SR1: r = −0.61, p = 2.8 ⋅ 10⁻², Fig 4D). Δ_k is not updated when the predicted objective function decrease is in moderate agreement with the actual objective function decrease (0.25 < ρ_k < 0.75, see Section 2.3), likely a result of inaccurate approximations to the Hessian. Thus, higher convergence rate ν of SR1 and BFGS schemes was likely due to more precise approximation of the objective function Hessian.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Evaluation of iterative Hessian approximation schemes.

Color scheme is the same in all panels. All increases/decreases are relative to fides 2D/GN unless otherwise noted. A: Performance comparison (ϕ). B: Increase in convergence count γ. C: Increase in convergence rate ν. D: Association between increase in average fraction of iterations without updates to the trust-region radius Δ_k and increase in ν for fides 2D/SR1 (green) and fides 2D/BFGS (orange). E: Association between average fraction of iterations where smallest eigenvalue λ_min of the transformed Hessian is negative and ν for fides 2D/SR1. F: Association between fraction of iterations without updates to B_k and increase in ν relative to fides 2D/SR1 for fides 2D/BFGS. G: Association between fraction of iterations with a numerically singular transformed Hessian and increase in γ for fides 2D/SR1 (green) and fides 2D/BFGS (orange). Change in γ was was censored at a threshold of 10⁻² to visualize models with γ. H: Association between fraction of iterations without updates to Δ_k and increase in ν for fides ND/GN. I: Association between change in average fraction of iterations where smallest eigenvalue λ_min of the transformed Hessian is negative and γ relative to fides 2D/SR1 for fides ND/SR1.

https://doi.org/10.1371/journal.pcbi.1010322.g004

To better understand the origins of performance differences between BFGS and SR1, we analysed the eigenvalue spectra of SR1 approximations and found that SR1 convergence rate ν was correlated with the average fraction of iterations where the transformed Hessian approximations had negative eigenvalues (r = −0.71, p = 6.7 ⋅ 10⁻³, Fig 4E). This correlation suggests that directions of negative curvature approximated by the SR1 scheme tended not to yield good search directions; handling of negative curvature is therefore unlikely to explain observed improvements in convergence rates. In contrast to negative eigenvalues, we found that difference in ν between BFGS and SR1 was correlated with the average fraction of iterations in which the BFGS approximation did not produce an update (r = −0.88, p = 8.3 ⋅ 10⁻⁵, Fig 4F). The BFGS approximation is not updated when the curvature condition is violated (see Section 2.4). Such a high fraction of iterations not prompting updates is surprising, since the violation of the curvature condition is generally considered to be rare [19]. It is nonetheless a plausible explanation for lower convergence rates, since the BFGS approximation is not expected to always converge to the true Hessian under such conditions [19].

For three problems, the BFGS (orange arrows, Fig 4C) and/or SR1 (green arrows, Fig 4C) approximations increased γ (1.29 to 3.12 fold change; Boehm, Crauste, Schwen), but for most problems γ was reduced by more than two-fold (−∞ to 0.46 fold change; SR1+BFGS: Beer, Fiedler Fujita, Isensee, Lucarelli Weber, Zheng; BFGS: Bachmann; SR1: Brannmark), canceling the benefit of faster convergence rate for many of these problems. Paradoxically, we found that convergence count changes were correlated with changes in the average fraction of iterations having ill-conditioned trust-region subproblems (BFGS: r = 0.86, p = 1.6 ⋅ 10⁻⁴; SR1: r = 0.73, p = 4.8 ⋅ 10⁻³, Fig 4G). Therefore, improved conditioning of the trust-region subproblem, unexpectedly, came at the cost of smaller regions of attraction for minima having low objective function values.

We complemented the analysis of 2D methods by evaluating their respective ND methods, which almost exclusively performed worse than 2D methods. We found that Fides ND/GN outperformed Fides 2D/GN on two problems (1.43 to 3.25-fold change; Crauste, Zheng) and performed similarly on one problem (0.99-fold change; Fiedler). Fides ND/BFGS outperformed Fides 2D/BFGS on three problems (1.16 to 1.37-fold change; Boehm, Fujita, Schwen) and performed similarly on three examples (1.01 to 1.06-fold change; Bachmann, Bruno, Fiedler). Fides ND/SR1 outperformed Fides 2D/SR1 on two problems (1.51 to 1.70-fold change; Crauste, Schwen). These results were surprising, since the use of 2D methods is generally motivated by lower computational costs, not better performance; the ND approach gives, in contrast to the 2D approach, an exact solution to the trust-region subproblem. For GN, the change in convergence rate ν was correlated with the change in average fraction of iterations in which the trust-region radius Δ_k was not updated (r = −0.8, p = 1.1 ⋅ 10⁻³, Fig 4H). For fides SR1/ND, the change in γ with respect to fides SR1/2D was correlated with the change in average fraction of iterations in which had negative eigenvalues (r = −0.72, p = 1.2 ⋅ 10⁻², Fig 4I). This suggests that inaccuracies in Hessian approximations may have stronger impact on ND methods as compared to 2D methods, thereby mitigating advantages that are theoretically possible.

Overall these results suggest that BFGS and SR1 approximations can improve optimization performance through faster convergence, but often suffer from poorer global convergence properties. Thus, they rarely outperform the GN approximation. BFGS and SR1 perform similarly on most problems, with the exception of a few problems for which BFGS cannot be updated due to violation of curvature conditions. We conclude that, while saddle points may be present in some problems, they do not seem to pose a major issues that can be resolved using the SR1 approximation.

3.4 Hybrid switching approximation scheme

We hypothesized that the high success count γ of the GN approximation primarily arose in the initial phase of optimization, which determines the basin of attraction on which optimization will converge. In contrast, we the high convergence rate ν of the BFGS approximation seemed more likely to arise from more accurate Hessian approximation in later phases of optimization, when convergence to the true Hessian is achieved. To test this idea, we designed a hybrid switching approximation that initially uses a GN approximation, but simultaneously constructs an BFGS approximation. As soon as the quality of the GN approximation becomes limiting, as determined by a failure to update the trust-region radius for n_hybrid consecutive iterations, the hybrid approximation switches to the BFGS approximation for the remainder of the optimization run.

We compared the hybrid switching approach using different values of n_hybrid (25, 50, 75, 100) to fides 2D/GN (equivalent to n_hybrid = ∞) and fides 2D/BFGS (equivalent to n_hybrid = 0). Evaluating optimizer performance ϕ, we found that the hybrid approach was successful for all problems, with n_hybrid = 50 performing best, improving ϕ by an average of 1.51 fold across all models (range: 0.56 to 6.34-fold change). The hybrid approach performed better than fides 2D/GN and fides 2D/BFGS on 5 problems (1.71 to 6.34-fold change; Crauste, Fiedler, Fujita, Lucarelli, Zheng; + signs Fig 5A). It performed better than fides 2D/GN, but worse than fides 2D/BFGS only on one problem (Boehm; (+) sign Fig 5A). The hybrid approach performed similar to fides 2D/GN on 5 out of the 7 remaining problems (0.89 to 1.03-fold change; Beer, Brannmark, Bruno, Isensee, Schwen; = signs Fig 5A). Decomposing ϕ into γ and ν, we found that hybrid switching resulted in higher ν for the four problems in which fides 2D/BFGS had higher ν than fides 2D/GN (Beer, Boehm, Fiedler, Zheng), as well as three additional problems (Crauste, Fujita, Lucarelli). These were the same three problems for which SR1 had higher ν (Fig 4B), but BFGS did not, as a consequence of a high number of iterations without B_k updates (Fig 4F). Consistent with this interpretation, we confirmed that the hybrid approach had very few iterations without B_k updates. In contrast to BFGS, the hybrid switching approach maintained a similar γ as GN, meaning that higher ν generally translated into higher ϕ. Evaluating the overlap between start-points that yielded successful runs for GN showed a higher overlap for the hybrid switching approach as compared to BFGS (Fig 5D), suggesting that higher γ for the hybrid switching was indeed the result of higher similarity in regions of attraction. These findings further corroborate that local convergence of fides 2D/GN is slowed by the limited approximation quality of GN.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Evaluation of hybrid switching approximation.

Color scheme is the same in all panels. All increases/decreases are relative to fides 2D/GN. A: Performance comparison (ϕ). B: Increase in convergence count γ. C: Increase in convergence rate ν. D: Overlap score for start-points that yield successful optimization runs with respect to fides 2D/GN.

https://doi.org/10.1371/journal.pcbi.1010322.g005