^{1}

^{*}

^{2}

^{3}

^{4}

^{5}

^{1}

All authors conceived and designed the experiments. RNG performed the experiments, analyzed the data, and wrote the paper. RNG, JJW, FPC, and CRM contributed reagents/materials/analysis tools.

The authors have declared that no competing interests exist.

Quantitative computational models play an increasingly important role in modern biology. Such models typically involve many free parameters, and assigning their values is often a substantial obstacle to model development. Directly measuring in vivo biochemical parameters is difficult, and collectively fitting them to other experimental data often yields large parameter uncertainties. Nevertheless, in earlier work we showed in a growth-factor-signaling model that collective fitting could yield well-constrained predictions, even when it left individual parameters very poorly constrained. We also showed that the model had a “sloppy” spectrum of parameter sensitivities, with eigenvalues roughly evenly distributed over many decades. Here we use a collection of models from the literature to test whether such sloppy spectra are common in systems biology. Strikingly, we find that every model we examine has a sloppy spectrum of sensitivities. We also test several consequences of this sloppiness for building predictive models. In particular, sloppiness suggests that collective fits to even large amounts of ideal time-series data will often leave many parameters poorly constrained. Tests over our model collection are consistent with this suggestion. This difficulty with collective fits may seem to argue for direct parameter measurements, but sloppiness also implies that such measurements must be formidably precise and complete to usefully constrain many model predictions. We confirm this implication in our growth-factor-signaling model. Our results suggest that sloppy sensitivity spectra are universal in systems biology models. The prevalence of sloppiness highlights the power of collective fits and suggests that modelers should focus on predictions rather than on parameters.

Dynamic computational models are powerful tools for developing and testing hypotheses about complex biological systems [

One way to cope with the dearth of reliable parameter values is to focus on predictions that are manifestly parameter-independent [

In calculating their uncertainties, Brown et al. found that the quantitative behavior of their model was much more sensitive to changes in certain combinations of parameters than others. Moreover, the sensitivity eigenvalues were approximately equally spaced in their logarithm, a pattern deemed “sloppy.” Such sloppy sensitivities were subsequently seen in other multiparameter fitting problems, from interatomic potentials [

In this work, we begin by empirically testing 17 systems biology models from the literature, examining the sensitivity of their behavior to parameter changes. Strikingly, we find that Brown et al.'s model is not unique in its sloppiness; every model we examine exhibits a sloppy parameter sensitivity spectrum. (Thus, in the models we've examined, sloppiness is also universal in the common English sense of ubiquity.) We then study the implications of sloppiness for constraining parameters and predictions. We argue that obtaining precise parameter values from collective fits will remain difficult even with extensive time-series data, because the behavior of a sloppy model is very insensitive to many parameter combinations. We also argue that, to usefully constrain model predictions, direct parameter measurements must be both very precise and complete, because sloppy models are also conversely very sensitive to some parameter combinations. Tests over our collection of models support the first prediction, and detailed analysis of the model of Brown et al. supports the second contention.

Sloppiness, while not unique to biology, is particularly relevant to biology, because the collective behavior of most biological systems is much easier to measure in vivo than the values of individual parameters. Much work has focused on optimizing experimental design to best constrain model parameters with collective fits [

Our collection of 17 systems biology models [

We quantified the change in model behavior as parameters ^{*} by the average squared change in molecular species time courses:
^{*}. Here _{s}_{,c}(_{c}_{s}

For each model, the sum in _{c}_{s}

To analyze each model's sensitivity to parameter variation, we considered the Hessian matrix corresponding to ^{2}:
^{χ2} corresponds to approximating the surfaces of constant model behavior deviation (as quantified by ^{2}) to be _{p}_{p}^{χ2} and related quantities are found in Methods. ^{χ2} for each model.)

The quantities we calculate from ^{χ2} are illustrated in (A), while (B) and (C) show that all the models we examined have sloppy sensitivity spectra.

(A) Analyzing ^{χ2} corresponds to approximating the surfaces of constant model behavior change (constant ^{2}) as ellipsoids. The width of each principal axis is proportional to one over the square root of the corresponding eigenvalue. The inner ellipsoid's projection onto and intersection with the _{1} axis are denoted by _{1} and _{1}

(B) Plotted are the eigenvalue spectra of ^{χ2} for our collection of systems biology models. The many decades generally spanned indicate that the ellipses have a very large aspect ratio. (The spectra have each been normalized by their largest eigenvalue. Not all values are visible for all models.)

(C) Plotted is the spectrum of

The models are plotted in order of increasing number of free parameters and are: (a) eukaryotic cell cycle [

The principal axes of the ellipsoids are the eigenvectors of ^{χ2}, and the width of the ellipsoids along each principal axis is proportional to one over the square root of the corresponding eigenvalue. The narrowest axes are called “stiff,” and the broadest axes “sloppy” [^{6}, indicating that the sloppiest axes of the ellipsoids illustrated in

The Hessian matrix is a local quadratic approximation to the generally nonlinear ^{2} function. Principal component analysis of extensive Monte Carlo runs in the Brown et al. model, however, indicates that the sloppiness revealed by ^{χ2} is indicative of full nonlinear ^{2} function [

Along with their relative widths, the degree to which the principal axes of the ellipsoids are aligned to the bare parameter axes is also important. We estimated this by comparing the ellipsoids' intersections _{i}_{i}_{i}_{i}

Naively, one might expect the stiff eigenvectors to embody the most important parameters and the sloppy directions to embody parameter correlations that might suggest removable degrees of freedom, simplifying the model. Empirically, we have found that the eigenvectors often tend to involve significant components of many different parameters; plots of the four stiffest eigenvectors for each model are in

These characteristic parameter sensitivities that evenly span many decades and are skewed from bare parameter axes define a “sloppy” model [

The difficulty of extracting precise parameter values from collective fits in systems biology modeling is well-known [_{1} in

As in

(A) Collective fitting of model parameters naturally constrains the parameter set ensemble along stiff directions and allows it to expand along sloppy directions. The resulting ensemble may be very large, yet encompass little variation in model behavior, yielding small prediction uncertainties despite large parameter uncertainties. (Σ_{1} denotes the 95% confidence for the value of _{1}.)

(B) If all parameters are directly measured to the same precision, the parameter set ensemble is spherical. The measurement precision required for well-constrained predictions is set by the stiffest direction.

(C) If one parameter (here _{2}) is known less precisely than the rest, the cloud is ellipsoidal. If not aligned with a sloppy direction, the cloud will admit many model behaviors and yield large prediction uncertainties. (Note that the aspect ratio of the real contours can be greater than 1,000.)

Unless one has direct interest in the kinetic constants for the underlying reactions, uncertainties in model predictions are generally more important than uncertainties in model parameters. The parameter set ensemble illustrated in

Direct parameter measurements, on the other hand, will have uncertainties that are uncorrelated with the model's underlying stiff and sloppy directions. For example, if all parameter measurements are of the same precision, the parameter set ensemble is spherical, as illustrated in

Does the sloppiness of these models really prevent one from extracting parameters from collective fits? Here we discuss a test of this prediction using an idealized fitting procedure.

Our ^{2} measure of model behavior change (^{*}; ^{χ2} is the corresponding Fisher information matrix (

We defined the relative 95% confidence interval size Σ_{i}

Shown are histograms of the relative confidence interval size Σ for each parameter in each model of our collection, after fitting 100 times as many time-series data points (each with 10% uncertainty) as parameters. In most cases, a large number of parameters are left with greater than 100% uncertainty. (A parameter constrained with 95% probability to lie between 1 and 100 would have Σ ≈ 100.) Labels are as in

The fact that nonlinear multiparameter models often allow a wide range of correlated parameters to fit data has long been appreciated. As one example, a 1987 paper by Brodersen et al. on ligand binding to hemoglobin and albumin empirically found many sets of parameters that acceptably fit experimental data, with individual parameter values spanning huge ranges [

We worked with the 48-parameter growth-factor-signaling model of Brown et al., shown schematically in

(A) Our example prediction is for ERK activity upon EGF stimulation given PI3K inhibition in this 48-parameter model of growth-factor signaling in PC12 cells [

(B) Shaded regions are 95% confidence intervals calculated via exhaustive Monte Carlo for our example prediction given various scenarios for constraining parameter values.

(C) Plotted is the relative size Σ of the 95% confidence interval for each parameter.

The scenarios represented are: (red, squares) all model parameters individually measured to high precision, (blue, triangles) all parameters precisely measured, except one estimated to low precision, (yellow, circles) all parameters collectively fit to 14 real cell-biology experiments. Precisely measured individual parameter values enable a tight prediction, (B) middle red band; but even one poorly known parameter can destroy predictive power, (B) wide blue band. In contrast, the collective fit yields a tight prediction, (B) tightest yellow band; but only very loose parameter constraints, (C) circles. The large parameter uncertainties from the collective fit, (C) circles, calculated here by Monte Carlo are qualitatively similar to those seen in the linearized fit to idealized data (

One important test of the model was a prediction of the time-course of ERK activity upon EGF stimulation, given inhibition of the PI3K branch of the pathway. The yellow shaded region in

How precise would direct parameter measurements have to be to yield as tight a prediction as the collective fit? For this prediction, the PI3K branch (inhibited) and C3G branch (NGF-dependent) of the pathway are irrelevant in the model; the remaining reactions involve 24 parameters. To achieve the red prediction in

What if a single parameter is left unmeasured? For example, consider high-precision measurements of 23 of the 24 involved parameters, all but the rate constant for the activation of Mek by Raf1. For this unmeasured parameter, we assumed that an informed estimate could bound it at 95% confidence to within a total range of 1,000 (e.g., between 1 ^{−1} and 1,000 ^{−1}). The resulting prediction (blue in

To usefully constrain Brown et al.'s model, direct parameter measurements would need to be both precise and complete. By contrast, collective parameter fitting yielded tight predictions with only a modest number of experiments. These results are expected given the model's sloppiness.

By examining 17 models from the systems biology literature [

What causes sloppiness? (1) Fundamentally, sloppiness involves an extraordinarily singular coordinate transformation in parameter space between the bare parameters natural in biology (e.g., binding affinities and rate constants) and the eigenparameters controlling system behavior, as discussed in [^{χ2} for a version of the Brown et al. model in which all concentrations and rate constants were scaled to 1. The resulting model remains sloppy, with eigenvalues roughly uniformly spanning five decades (

Parameter estimation has been a serious obstacle in systems biology modeling. With tens of unknown parameters, a typical modeling effort might draw some values from the literature (possibly from in vitro measurements or different cell lines) [

Computational modeling is a potentially invaluable tool for extrapolating from current experiments and distinguishing between models. But we cannot trust the predictions of these models without testing how much they depend on uncertainties in these estimated parameters. Conversely, if we insist upon a careful uncertainty analysis, it would seem unnecessary to insist upon tight prior estimates of the parameters, since they do not significantly enhance model predictivity. Because the behavior of a sloppy model is dominated by a few stiff directions that nonetheless involve almost all the parameters, direct parameter measurements constrain predictions much less efficiently than comparably difficult experiments probing collective system behavior.

Our suggestion of making predictions from models with very poorly known parameters may appear dangerous. A model with tens or hundreds of unmeasured parameters might seem completely untrustworthy; we certainly believe that any prediction derived solely from a best-fit set of parameters is of little value. Uncertainty bounds derived from rigorous sensitivity analysis, however, distinguish those predictions that can be trusted from those that cannot. Of course, successful fits and predictions may arise from models that are incorrect in significant ways; for example, one model pathway with adjusted parameters may account for two parallel pathways in the real system. A model that is wrong in some details may nevertheless be useful in guiding and interpreting experiments. For computational modeling to be useful in incompletely understood systems, we must focus not on building the final, perfect, model with all parameters precisely determined, but on building incomplete, tentative, and falsifiable models in the most expressive and predictive fashion feasible.

Given that direct parameters measurements do not efficiently constrain model behavior, how do we suggest that experimentalists decide what experiment to do next? If the goal is to test the assumptions underlying a model, one should look for predictions with tight uncertainty estimates that can be readily tested experimentally. If the goal is to reduce uncertainty in crucial model predictions, one must invoke the statistical methods of optimal experimental design, which we have studied elsewhere [

In our approach, the model and its parameters cannot be treated in isolation from the data that informed model development and parameter fitting. This complicates the task of exchanging knowledge in the modeling community. To support our approach, standards such as SBML [

Every one of the 17 systems biology models we studied exhibits a sloppy spectrum of parameter sensitivity eigenvalues; they all span many decades roughly evenly and tend not be aligned with single parameters. This striking and apparently universal feature has important consequences for the modeling process. It suggests that modelers would be wise to try collective parameter fits and to focus not on the quality of their parameter values but on the quality of their predictions.

^{χ2} can be calculated as

Second derivative terms (^{2}_{s}_{,c}(^{*},_{i} d_{j}^{χ2} at ^{*}. _{s}_{,c}(^{*},_{j}

The projections _{i}_{i}

To rescale ^{χ2} so that it corresponds to fitting _{d}^{χ2} by _{d}^{2}. In the quadratic approximation, the one-standard-deviation uncertainty in the logarithm of parameter _{i}

The red and blue prediction uncertainties shown in

All computations were performed in the open-source modeling environment SloppyCell, version 0.81 [

(1.1 MB ZIP)

(112 KB PDF)

(94 KB PDF)

(48 KB PDF)

(46 KB PDF)

(42 KB PDF)

Models discussed that appear in the BioModels database [

We thank Eric Siggia for suggesting study of the rescaled model of Brown et al. We also thank Rick Cerione and Jon Erickson for sharing their biological insights, and John Guckenheimer, Eric Siggia, and Kelvin Lee for helpful discussions about dynamical systems. Computing resources were kindly provided by the USDA-ARS plant pathogen systems biology group in Ithaca, New York, United States. Finally, we thank several anonymous reviewers whose comments strengthened the manuscript.

Systems Biology Markup Language