Notions of error

The term “error” is used in many contexts and with numerous meanings. We discuss two broad forms of error, namely, measurement error and model error. Here, “model” refers to mathematical objects that serve as primary tools for interpretation, inference, simulation, and prediction. Models may be deterministic or probabilistic (statistical), or some combination of these. There is a spectrum of complexity and a variety of uncertainties.

Measurement error is pervasive in the production of data. The common informal definition is “measurement error = observation—true value”. Similarly, in numerical modeling we have “model error = numerical result—true value”. This definition applies to numerical models that are computable in closed form as well as those that only provide computer model output. Other names for model error include discrepancy, bias, and offset.

The informal definitions of error above must be generalized in problems for which simple additive errors are insufficient. Further, the usual goal in probabilistic modeling is the formulation of a probability distribution for quantities of interest that matches the true probability distribution of those quantities. We offer a clarification in use of the term “error”. Typically, statisticians use the term to refer to misspecifications of probability distributions used in an analysis. This may lead to phrases like “model error in the specification of the distribution of model error”. We merely point this issue out and trust that interpretations in specific examples will be clear.

The casual definition of measurement error above is often an oversimplification. Specifically, quantities viewed as observational data may be results of some numerical model, itself subject to model error. Very important examples arise in remote sensing. Typically, remote sensors record electromagnetic radiation (or photon counts). Models are applied to that raw information to produce “observations” of quantities of interest (e.g., [2–4]). In many cases, use of statistical summaries of observations in the production of observational data may introduce a variety of errors or biases. Though this is an important topic, we focus on computer model error here.

There are two primary sources of computer model error. First, errors arise due to numerical model inadequacies, such as those resulting from approximations (e.g., “ignoring higher order terms”), as well as algorithmic or computer mistakes. The second category comprises apparent model errors that arise due to misspecification of model inputs. Most numerical models require specification of parameters or processes, such as boundary or initial conditions. Errors in such inputs may produce model output that is in error, even for virtually error-free models. (We discuss this issue further in Section 4.) These points are related to intrinsic versus extrinsic error classifications. Also, the issues are related to research known as uncertainty quantification (UQ). In our framework inputs are modeled as random variables with distributions that reflect uncertainties and errors. Hence, the output of even deterministic models would be treated as random. Also, the output of very high-dimensional and complex, including chaotic, models may often be usefully treated as random.

There are several statistical and inference problems associated with model error. Seminal contributions regarding the design and analysis of computer experiments include [5, 6]. Other examples are model assessment such as verification and validation [7–9]. Calibration is important for the eventual use of the model [10]; [11]. Models which display satisfactory assessments and are well-calibrated may be used to obtain large ensembles to learn about responses to variations of parameters, etc. This notion is highly related to model emulation [12–14]. In this article we focus on prediction.

Bayesian modeling overview

We use Y to denote observational data, X to denote the processes that are the main targets of our analyses, and θ to denote unknown parameters. Bayesian analysis is based on probability modeling for all such quantities. For random objects, say U and V, we use the notation [U, V] to denote the joint distribution of U and V, [U|V] to denote the conditional distribution of U given V, and [U] to denote the marginal distribution of U. The following skeleton of a Bayesian model is useful in formulating models [15]. The model consists of three primary probability distributions:

1. Data Model: [Y | X, θ_y]. The data model is the conditional probability distribution of Y given X and perhaps a collection of unknown parameters θ_y.
2. Process Model Prior: [X | θ_x]. The process model prior is a probability distribution for the processes of interest, perhaps depending on some parameters θ_x.
3. Parameter Model Prior: [θ_y, θ_x]. Bayesian modeling is completed by specifying a prior distribution of all unknown parameters.

After observing Y, the data model is combined with the process and parameter models via Bayes’ Theorem to produce the posterior distribution (1)

The specification of all three of the model components are subject to modeling error. In this article we focus on the treatment of error in specifying and using process model priors. In our development below, physical models are components of process model priors. There is substantial research on modeling error regarding data models and parameter priors; [16, 17] provide substantial discussion, also see [18]. We note that there are challenges regarding the impacts and interactions of model errors among all three of the primary components of a Bayesian model. Though very important, this issue is beyond the scope of this article. Discussion and references regarding robust Bayesian analysis are given in [19, 20]. Realistic models are often complicated and involve many more unknown quantities than indicated in the descriptions of the distributions involved in Eq (1). For example, X may include spatial-temporal fields of several variables. Bayesian hierarchical modeling (BHM) provides tools for developing complex models, clarifying assumptions, and quantitative uncertainty management. A hierarchical probability model for a collection of random variables is a sequence of conditional probability models that specifies a bona fide joint probability distribution. The product of the three distributions on the right hand side of Eq (1) is an example of a BHM, but is also deceptively simple compared to the sort of models that arise in practice.

Physical-statistical models

Historically, many analysts separated modeling into two categories: statistical modeling and physically-based mechanistic modeling. In this article we combine the two notions by developing physical- (or mechanistic-) statistical models (e.g., [21]). To clarify, suppose X represents a spatial-temporal field. Our starting point for the mechanistic-model contribution is consideration of a deterministic dynamic model for X in continuous time t, (2) where Z is a collection of relevant processes and θ_z are parameters that arise in the modeling. Important examples of Z, particularly in prediction, are known as explanatory or independent or predictor variables or covariates. Further, Z may include quantities such as initial and/or boundary conditions, forcings, etc. Despite the importance of Z, we assume it and θ_z are known so that we can focus on model error. We return to consideration of Z in the Conclusions. Till then, we suppress dependence on Z and θ_z. Also, many models are discrete time and/or space models. Important examples are computationally implemented approximations of (2). For brevity, we do not rewrite continuous models as discrete ones in the general discussion.

We suppose that nature uses a “true model” to produce a “true” space-time process where denotes the space (s)—time (t) domain being considered. However, the true model is unknown to us. Further, we cannot observe exactly but only subject to measurement error. Our use of the notation X rather than is a response to such limitations. Relying in part on the model in (2), we formulate a physical-statistical model (3) where η is a stochastic process used to represent error in the specification of g and θ_x are additional unknown, random parameters that arise in the modeling. Our goal is to formulate useful statistical models or otherwise manage the impacts of η.

Note that is our modeled model error process. We define the true model error process as (4)

We have emphasized the differences between and X and between δ and η. This is rarely done in the literature and hereafter we use X, not , to denote the quantity of interest and in specifying the data model and the process model prior. We reserve the use of in the specification of the “true process” in our numerical examples in the “Illustrations” Section. Similarly, our probability model for η may be viewed as our prior distribution for δ, though only η is used in the rest of this article.

Class I

We suppose the models as in Eq (3) are sufficiently manageable to allow incorporation of a model error process into the dynamics or an approximate model based on those dynamics (e.g., discretization of a continuous model). The key points are that we can formulate probabilistic models for the model errors η and use observational data to compute updates of all features of the Bayesian model.

In [23] the authors considered prediction of tropical surface winds given high-resolution satellite (i.e., scatterometer) observations and low-resolution assimilated model output obtained from the National Centers for Environmental Prediction. The authors applied physical-statistical modeling at multiple scales using both a solution to an approximating system of differential equations (“shallow fluid equations”) and multi-resolution wavelets. The resulting space-time model for the north-south components v_t of surface winds is (6) where μ is a mean, Φ is a matrix containing basis functions of the differential equations and Ψ contains wavelet basis functions (the model for the east-west components of the winds is analogous). They formed a linear dynamic model for the coefficients a_t: (7) the η process represents model error in the spectral domain (a similar model was used to model the b_t’s). In the spirit of Class I, model error represented by η is incorporated directly into the (transformed) physical-statistical model.

An example in the context of glacial dynamics is provided in [24]. They include model error in a physical model that improves model behavior. Also, they are able to offer scientific interpretations of the posterior distribution of that error suggesting spatial regions where the physical model requires adjustment.

In [25] the authors use a BHM for surface wind forcing within a variational data assimilation scheme (weak constraint) to identify regions in (s, t) of suspected model error for a high-resolution model of the California Current System upwelling/downwelling processes. Model error identification leads to improved model forecasts.

Class II

We suppose that direct incorporation of dynamical, stochastic error processes is not practical. Examples arise when the quantities modeled are of very high dimension. Specifically, suppose that X is a vector of length in the 10’s to 100’s of millions. Though we can “write down” Eq (3), we believe that usefully modeling and subsequently learning about the error process η from data in such large dimensions is implausible. Similar concerns may arise in low to moderately sized problems involving complicated and computationally expensive models. (The meanings of “large” dimensions and “implausible” and/or “computationally expensive” evolve as both data collection and computational assets improve.) However, our challenge remains the use of computer model output in a fashion that accounts for model error. The strategy pursued in this article treats computer model output as biased observations of the truth. That is, we will formally model output in much that same fashion as we treat observational data.

Let be a dimension-reduced summary of X. For example, the operator may be a matrix whose elements indicate the selection of specific subsets of variables, spatial and/or temporal averaging of X, etc. Even in moderate dimensions we often must settle for comparing estimated expectations (means, probabilities, information measures, etc.) and other summary statistics such as medians, coverings, etc. Let Y^c represent the corresponding summary derived from model output. The superscript “c” emphasizes the fact that these “data” are computer generated as opposed to observational data Y. We incorporate the computer output based on a statistical model of the form (8) where β represents bias that arises as an unknown function of the large-scale model error η, is the appropriate vectorized form of , and γ is a vector of zero-mean errors with covariance matrix Σ^c. For clarity, in numerical simulated examples presented later, we actually produce as (9)

We note that it is possible to formulate and as functions of unknown parameters. While we envision an underlying model (3) and the existence of a function that maps η to β, we assume that estimating this function is implausible. Hence, our fundamental modeling challenge is to formulate useful statistical models for β directly. Note that this means the form and behavior of β vary with the choice of . We note that the dimension-reduction strategy may induce a further error. We don’t account for this separately, but rather assume that this is incorporated in β.

By treating model output as if they are observations, we mean that we assume Y^c has a conditional distribution [Y^c | X, β, θ_y^c]. Note that the covariance matrix Σ^c of the errors γ in Eq (8) is an example of θ_y^c. Assuming that the observational data, now denoted by Y^o for clarity, and the model output are conditionally independent, the BHM that combines observations and dimension-reduced model output is summarized as (10) (11) (12)

Formulation and interpretation of the priors in this approach merit further discussion. First, though we have written the process model prior as a conditional probability distribution for X, it may be the case that we only form a prior probability model for the low dimensional summary variable . Next, an important by-product of this strategy is that it offers opportunities to combine simplified, low-dimensional physical-statistical models or purely statistical models for with output from large-scale computer models. Downscaling is one potential application. An example is reviewed at the end of Section 2.3. The dimensions of Y^o, Y^c may also be too large to be usable “as is”. This potentially requires introducing a further operator, acting on these variables, generally performing another level of dimension reduction. Though very important, we do not pursue these points further here. Rather, we assume that dimension reduction is performed on X only. Finally, forming priors on parameters is a common, though not typically easy, task in Bayesian analysis. Here, we also face the problem of formulating a prior probability distribution for β. Though challenging, this problem may also be worth the effort. Data-informed posterior distributions for biases may help us to understand model error and suggest possible improvements to models. Also, note that we have included β in the prior process model and allowed for additional flexibility by using a distribution that may be parameterized by parameters θ_β. This is particularly useful when incorporating more than one numerical model (see the next section).

In practice additive-errors data models such as (5) and (8) may be inappropriate. Hence, the general probabilistic notation above is important in formulating models and performing computations.

Multiple computer models

This approach offers a method for combining model output from multiple models (e.g., [26]). Suppose we are given model output from K computer models. Our model data consists of K vectors of length J₁, …, J_K, respectively. Following (8) we assume that (13)

Note that we have allowed each model to suffer from its own biases. Further, the vary with the computer model, may be of differing dimensions, and even different variables in X. If maintaining the conditional independence assumption is reasonable, the BHM extends to (14) (15) (16)

The conditional independence assumptions among the various models does not mean the model outputs are marginally independent. Indeed, we expect some similarities among the models. Rather, it means that model output departures {γ_k} from the means when X and the biases are known are independent between the models. Further, suppose the all represent the same summary of the large-scale process. We then have the opportunity of building similarities of the models by appropriate formulation of the priors for β_k.

Berliner and Kim [27] (hereafter BK) provides a Bayesian analysis for process variables X defined as hemispheric- and monthly averaged surface temperatures for the period 1882–2097. They used observational data over the period 1882–2001 to form a statistical time series model for the temperatures. Climate model projections obtained from two climate system models were incorporated as biased observations to drive the model for the period 2002–2097. A key challenge in their case is that the prior for the model biases β_*(s, t) should accommodate their temporal evolution. In some cases, analysts may have sufficient prior information to construct meaningful time series models for the biases. BK assumed very little prior information. In their comparatively uninformed analysis they assumed that the biases were constants locally in time. A single bias assumed over the entire time period seems implausible and could lead to overconfidence in the final conclusions. At the other extreme biases modeled to change at each time point, but are otherwise unstructured, would lead to very weak inference about the biases. Based on some preliminary data analysis, BK assumed that the biases are constant over intervals 2002–31, 2032–51, 2052–71, 2072–91, and 2092–97. Note that the observational data is used to form a process model prior [X | θ_x] for prediction of X after 2001. In a “real” analysis, we would seek observations and model output that overlap in time. This could improve inferences for model biases.

Class I approach

Results.

To assess the quality of estimates (i.e., t ≤ 12) and predictions (t > 12) of X, we consider spatially averaged mean squared errors (25) where E(X(x, y, t) | Y) is the posterior mean (or expectation) of the true process conditional on the observations Y. For all four error processes, we compute MSE in two cases: (i) average over the observation sites and (ii) average over all sites. The most obvious comment based on inspection of Fig 2 is that use of any of the three nontrivial models for η lead to better results than those obtained using assuming that η = 0 everywhere. This is suggestive of the Bayesian approach’s ability to use observational data effectively when using flexible, though even incorrect, models. The influence of the observations is evident when comparing achieved MSE at all sites with the MSE at observation sites only.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Predictive ability and percent reduction in MSE.

Left: predictive ability: no model error (blue line), model (22)—black line, model (23)—green line, model (24)—red line. Right: percent reduction in MSE for the three models—versus “no model error”.

https://doi.org/10.1371/journal.pone.0286624.g002

Recall that the true model involves spatially varying diffusion coefficients while the assumed model uses constant diffusion coefficients. This may explain why using error models that involve spatial partial derivatives of X(⋅, ⋅, t) is useful. This indicates that state-dependent error models are suggested in general. Model error prior models based on spatial gradients in X strikes us as mandated in the context of nonlinear physical models. Further, when model errors may be due to misspecifications of parameters, selection of error models based on the type of misspecifications deemed most likely may prove effective. In practice, we would not observe so calculation of the MSE as done here is not possible. However, some information is available by approximating MSE by replacing by Y at observation sites. Statistical model building ideas such as fitting the model using subsets of the observations and investigating how the models perform in predicting at the remaining sites may be valuable. We note that selection of error models may require substantial numerical computations of posterior distributions and comparisons of results. When feasible such model building approaches are recommended and potentially valuable in improving physical models.

Class II approach

We again demonstrate approaches for the advection-diffusion Eq (17), but, for brevity, we only consider a spatial version. We assume the same 335 grid points as the observational network.

Example 1.

Fig 3 displays the posterior means and standard deviations of X and β for Example 1. The pattern of observational data sites stands out in the plot of the posterior standard deviations of X. As expected, these assessments of posterior uncertainty are dramatically smaller at observed sites than those at sites with no observational data. This feature is also indicated by the relatively small values at observation sites in the plot of the true estimation of errors. We note that the posterior standard deviations of β are much less variable spatially than those of X. This is reasonable since we “observed” the computer model at all grid points. The magnitudes of the posterior standard deviations of β are roughly concentrated in the middle of the range of those of X. The pattern and values of both the posterior means of the biases and the true biases in the western half of the region suggest that the computer model systematically underestimates the large values of the true and overestimates its small values. We also remark that this pattern of biases appears to be translated to the east by roughly 10°. Further, the posterior means of the biases capture the spatial pattern of the true biases very well. Histograms of residuals (i.e., ) are shown in the top row of Fig 4. For Example 1, we define while for Example 2, . These graphics confirm and quantify the comparatively excellent estimation behavior at sites with physical observations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Example 1.

Top Row: Posterior means of X (left panel) and β (right panel). Middle Row: True estimation errors: (left panel) and model biases: (right panel). Bottom Row: Posterior standard deviations of X (left panel) and β (right panel).

https://doi.org/10.1371/journal.pone.0286624.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Histograms of residuals for Examples 1–4.

Histograms of computer model residuals (left side) and observation residuals (right side) in rows 1–4, respectively. We note that the scale of the x-axis varies from plot to plot.

https://doi.org/10.1371/journal.pone.0286624.g004

Example 2.

Fig 5 displays the posterior means and standard deviations of X and β for Example 2. Recall that in this case we use local averages of computer model output in defining the means of model observations. The most obvious and expected impact of this procedure as opposed to that in Example 1 is a general degradation in the accuracy of estimates of the true process. Histograms of estimated residuals are again shown in Fig 4, second row. These graphics confirm and quantify the comparatively excellent estimation behavior at sites with physical observations. Further, when compared to their counterparts in Fig 4, they show the expected tendency of larger (in magnitude) residuals.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Example 2.

Top Row: Posterior means of X (left panel) and β (right panel). Middle Row: True estimation errors: (left panel) and model biases: (right panel). Bottom Row: Posterior standard deviations of X (left panel) and β (right panel).

https://doi.org/10.1371/journal.pone.0286624.g005

Impacts of selected changes in the prior models

Example 3.

We assume that the biases are equal at all grid points. Specifically, we assume that (35) where is a vector of length 703 containing only 1’s. Again, we select an IG(4, 0.1) prior distribution for . Asummary of results is presented in Fig 6. As anticipated based on our earlier review of BK, this prior leads to relatively poor results, especially regarding the bias. As indicated in Fig 7, the posterior mean value 0.50 of β₀ is very small compared to the range, roughly −5 to 5, of biases suggested in Fig 3. In combination with the very small posterior uncertainty regarding β₀ as reflected by the narrowness of the 90% credible interval, the results suggest with high confidence that the model bias is small. This would likely lead to overconfidence in the computer model. Heuristically, the reason for this behavior is that there is a very large number of observations being used to form inference for a scalar.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Examples 3–4. Constant-bias prior.

The left panels correspond to a bias prior model which assumes that β is constant everywhere (Example 3). The right panels correspond to a bias prior model which assumes that β is row-wise constant (Example 4). Top: the posterior mean E(X | Y^o, Y^c). Middle: True estimation errors: . Bottom: posterior standard deviations of X.

https://doi.org/10.1371/journal.pone.0286624.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Examples 3–4.

Posterior means and 90% credible intervals for biases in both the everywhere constant bias (red) and row-wise constant bias (black) examples.

https://doi.org/10.1371/journal.pone.0286624.g007

Example 4.

Next, we used a prior that assumes the biases are equal within each of the R = 19 rows of grid points. Such a prior assertion might arise in anticipation of highly zonal (east-west) behavior. Specifically, we set where each is a vector of all 1′s and of length equal to the number of columns (37) of grid points. The prior for the R horizontal biases used is a one-dimensional GMRF with un-normalized density given by (36) where we set δ₃ = 0.5 and .

Fig 7 summarizes posterior inferences for the row-wise-constant biases and the everywhere-constant bias. The posterior standard deviations of X shown in Fig 7 coupled with the histograms of the residuals shown in Fig 4 again indicate poor behavior relative to that in Example 1 though somewhat better than the behavior in Example 3.

Varying GRMF priors on β: Recall the GMRF prior for β given in Eq (31). The parameter δ₁ controls the smoothness of realizations of the model. As δ₁ grows, the degree of local spatial correlation grows, leading to the appearance of increasing smoothness. To assess the degree of smoothness, Fig 8 shows realizations of β generated from a GMRF with δ₁ set to δ₁ = 0.01, 0.1, 0.5, 1.0. These draws do not necessarily look “like” the posterior mean of β we estimate, since this posterior mean is ultimately determined by the model and the data, as desired. The role of these draws is to show how much smoothness is imposed by the prior versus by data/statistical model. As we see, the data play the main role in determining the bias. Throughout the examples above, for the prior distribution on β we set δ₁ = 0.5. We performed additional simulations using the setup of Example 1, but with δ₁ = 0.1 and 1.0. For comparison with Fig 3, in Fig 9 we display posterior means for the process X and biases β where we used δ₁ = 0.1 (left panels) and δ₁ = 1.0 (right panels). Naturally, the choice of the prior for β has an effect on the ensuing inference, as observed in the posterior means for the two cases. When δ₁ = 0.5 and 1.0, the posterior means E(β | Y^o, Y^c) appear much smoother than typical draws from the corresponding priors (see Fig 8), indicating that the data play a crucial role. In the left panel of Fig 10, we overlay histograms of residuals for the three cases δ₁ = 0.1 (blue), δ₁ = 0.5 (red) and δ₁ = 1.0 (green). Visually, it appears that the two additional priors (δ₁ = 0.1, δ₁ = 1.0) yield slightly larger residuals for observations. The message is different when analyzing the computer model residuals. When δ₁ = 0.1, the prior model for β “allows” for high gradients as can be seen in the bottom left panel of Fig 9. As such, the dangers of using uninformative (non-regularized) priors are evidenced again in the right panel of Fig 10. When setting δ₁ = 0.1, the computer model residuals are very small (narrow, blue histogram in the right panel of Fig 10). In this case, by exhibiting high gradinents, the estimated bias E(β | Y^o, Y^c) compensates for nearly all of the discrepancy between Y^c and X.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Draws from different prior distributions on β.

Simulated random fields from the prior distribution on β with parameters δ₁ = 0.01 (top-left), 0.1 (top-right), 0.5 (bottom-left) and 1.0 (bottom-right).

https://doi.org/10.1371/journal.pone.0286624.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Results with different priors on β.

Posterior means of X and β where δ₁ = 0.1 (roughest, left panels) and δ₁ = 1.0 (smoothest, right panels).

https://doi.org/10.1371/journal.pone.0286624.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Histograms of residuals.

The observation residuals (left) and computer model residuals (right) for the three choices of prior models: δ₁ = 0.1 (blue), δ₁ = 0.5 (red), δ₁ = 1.0 (green).

https://doi.org/10.1371/journal.pone.0286624.g010

Related models

In Sections 1.1 and 1.3, we contrasted numerical model error with apparent model errors such as those resulting from errors in predictor variables. Specifically, on the right hand side of Eq (2), ∂X/∂t = h(X, Z, θ_x, θ_z), we included Z to represent relevant processes and θ_z are additional parameters that arise in the modeling.

Approaches to predictive problems depend on whether or not the values of the predictor variables are known or unknown. If the predictors are known, say Z = z, we can define the process model prior as [X∣z, θ_x, θ_z]. This view is very common. In some cases, observations or other estimates of Z are used to produce z. For example, [10] analyzed use of a hydrological model to predict ground-water flow (X). One of several inputs (z) they used was “a time series of measured rainfall data”.

If Z are modeled as unknown, our analysis depends on the sources of information regarding Z. As in the case of X, one source are observational data, say Y^z, with data model . Another source is a physical model, which we suppose is of the form assumed in Eq (2) and leads to a physical-statistical model (analogous to Eq (3)) (37) where ζ is a stochastic process used to represent error in the specification of q, and θ_z is an additional unknown multivariate parameter. These issues are beyond our scope here.

Prediction of a dynamical system based on a time series of observations is another relevant setting. The example in Section 3.1 is such a case. Suppose we can break up a discrete space-time field X to form a time series of spatial vectors x₁, …, x_T. Model error may be modeled as a time series η₁, …, η_T. We typically incorporate observational data vectors y₁, …, y_T sequentially. The details are much like those associated with the familiar Kalman filtering procedure, so we do not present them here.

Discussion

Our priority here is the development of statistical prediction models that combine data and physically-based, mechanistic models and account for model error. We considered two classes of mechanistic models. In the first, we incorporated a model error process into a physical model. The resulting posterior probability distribution for model error

• improves predictions and enables quantification of associated uncertainties;
• may provide opportunities to improve physical models and perhaps lead to new science.

We also discussed the issue of selection of priors for model error and provided some illustrations.

The second class arises when models are deemed to be too large and complex for the Class I approach to be practical. Our suggestion revolves around dimension reduction. We consider low-dimensional summaries or functions of the primary processes of interest. The corresponding summaries of computer model output are modeled as biased observations of the true dimension-reduced processes. Combining such computer model “observations” and observational data is conceptually simple. We also indicated how this approach is very useful for combining output from multiple models.

An important issue is how to perform statistical inference for such complex models. The illustrations presented in this manuscript are fitted via a traditional Metropolis-Hastings algorithm. When necessary, classical Bayesian inference methods can be combined with more modern samplers, such as Hamiltonian Monte Carlo (see [30] for a recent overview), or Approximate Bayesian Computation [31, 32].

We believe that the use of low-dimensional summaries of large-scale computer model output is necessary and hence pervasive. For example, climate system models produce state vectors of dimensions in 100’s of millions. Such massive model output becomes useful when dimension reduced variables are studied. Further, summaries such as a time series of model estimates of global average temperatures are useful when explaining climate change to policy makers and the public.

We view the biases described here as unknown, inaccessible functions of model errors present in large-scale model output. We discussed some of the difficulties in formulating prior distributions for these biases. Adding to the difficulties, the functions leading to biases may be state- and/or parameter-dependent. Gaussian processes and hierarchical (deep) Gaussian processes are a very flexible class of models, capable of capturing a wide range of behaviors, and thus they are the go-to choice as statistical models for these unknowns. Nevertheless, The importance and prevalence of dimension-reduced variables derived from model output indicates that our suggestions merit further research.