Introduction

Across the sciences, model-estimation and -prediction errors are often summarized and analyzed using dimensioned [1] and dimensionless [2] measures. While dimensionless error measures have received considerable attention [3–5], dimensioned measures are better suited to summarizing the magnitude of model error in meaningful units. When given a set of model predictions (P_i, i = 1, 2, …, n), where each P_i corresponds to a reliable observation (O_i), the mean squared error (MSE) and the root mean squared error (RMSE): (1) (2) are routinely reported [6]. The mean absolute error (MAE): (3) is reported less often, even though it has a clearer interpretation than RMSE because MAE is the average error [7].

We and others have shown elsewhere that error statistics based on sums-of-squares have a number of issues that make them less interpretable than those based on absolute values [1, 4, 7–12]. This is especially the case when they are used as measures of average model error. An additional drawback to using MSE is that its squared dimensional units are difficult to interpret. As a result, MAE is the preferred measure of average model error. Even so, sums-of-squares measures continue to be assessed and reported, partially due to inertia, but also to their amenability to mathematical decomposition into additive variance-based measures. In the context of evaluating model error, this property was used by Willmott in 1981 [13] to decompose MSE into systematic (MSE_s) and unsystematic (MSE_u) components: (4) (5) such that (6)

For both (MSE_s) and (MSE_u), ordinary least-squares (OLS) regression of the model predictions on the observations typically is used to obtain (i.e., a linear fit of P on O).

As computed above, (MSE_s) is typically interpreted as consistent over- and/or under-prediction of the observations by the model (i.e., the model has non-zero mean bias and/or the regression slope is not one). The unsystematic component provides an estimate of the model’s random error or scatter about the regression line. Forming the ratios of MSE_s/MSE and MSE_u/MSE gives estimates of the fraction of total error (as estimated by MSE) that is identified as systematic or unsystematic. This decomposition has served as a relatively insightful summary of model error (e.g., [14]) and has been used as a guide to model improvement because a model that has a large amount of systematic error usually can be respecified to reduce the consistent over- or under-prediction.

While decomposing MSE into its constituent components has been a useful approach, MSE is a flawed measure of average model error. Using MSE to identify systematic and unsystmatic components of error, therefore, can produce misleading summaries of the types of errors that various models contain. Even more importantly, models may be inappropriately adjusted to reduce systematic error that has been misidentified by the MSE-based approach, e.g., when the impacts of outliers are overemphasized. As a result, our goal here is to develop and present a more rational approach for error decomposition that uses MAE as the baseline for average model error.

Decomposition of MAE into three components

Although our goal is to partition MAE into components that represent systematic and unsystematic error patterns, we also want to move beyond the traditional two-part decomposition to further divide systematic errors into two separate components. One can be used to indicate the amount of bias in a model and the other to represent the extent to which the model predictions systematically under- or over-estimate observations falling below and above the observed mean (the regression slope is not one). The latter is referred to as proportionality error and is distinct from model bias represented in the under- or overestimation of the observed mean.

For each of these three types of error—bias, proportionality, and unsystematic—we develop a weighting function that can be used to partition MAE into its three components. We offer a diagram (Fig 1) using a small, synthetic dataset (Table 1) to illustrate the estimation of the three components.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Representations of model-prediction errors showing aspects of the decomposition into systematic and unsystematic components (using data from Table 1).

Predictions (in the upper left panel) can be decomposed in the traditional way using MSE, as shown in the upper-right panel where the lengths of the red and blue dotted vertical lines determine the partitioning of the errors. The bottom-left panel shows the predictions after bias is removed (i.e., ), while the bottom-right panel shows the magnitudes of the bias, proportionality, and unsystematic components of our three-part, weight-based decomposition of MAE.

https://doi.org/10.1371/journal.pone.0279774.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Data and error components for the example in Fig 1.

For the components whose sum (Σ) is given in the far right column, dividing by 6 gives the mean value (i.e., , , MSE, MSE_s, (MSE_u), and MAE). For the other three rows that have absolute values and are used to form the b, p_i, and u_i weights, which then determine , and (see Eqs 14–16), the sums are not relevant and, therefore, are not given.

https://doi.org/10.1371/journal.pone.0279774.t001

An example of model-estimation errors

To illustrate the properties of our newly derived measures of model error, we use a tree-ring based reconstruction model developed by Meko et al. [15]. This reconstruction provides over 1200 years of annually resolved flow predictions for the Upper Colorado River at Lee’s Ferry. The large majority of the annual flow in the Colorado River comes from upstream of Lee’s Ferry [16], so the reconstruction is an essential indicator of historical water availability for the Colorado River. The observed and reconstructed flow are water-year totals for a large river and, therefore, are reported in billions of cubic meters per year. Both the observed data and reconstructed values are based on estimates of “naturalized” streamflow, which corrects for the anthropogenic alterations of flow (i.e., reservoirs, irrigation, etc.). In a recent article [14], the model was bias-corrected so that its empirical probability distribution better matched that of the observations (e.g., compare Fig 2a and 2b). Here, we employ our new decomposition of model errors to compare the three sources of error in the original and bias-corrected reconstruction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Model-estimation errors before and after bias correction.

Model-estimation errors for (a) the reconstruction of annual Upper Colorado River flow (in billions of cubic meters) from [15] and (b) the same reconstruction after applying the bias-correction procedure of [14].

https://doi.org/10.1371/journal.pone.0279774.g002

Prior to bias correction (Fig 2a), the Upper Colorado River reconstruction has low overall error, with a MAE of 2.12 billion m³ (i.e., when compared to the of 18.53 billion m³). The small value of MAE_b (0.08 billion m³) also shows that the reconstruction model faithfully reproduces the observed mean. From the scatterplot and the substantial amount (34%) of error in MAE_p, however, it is clear that high flow years are underestimated (and, to a lesser extent, low flows are overestimated). Even with these substantial proportionality errors, the majority (62%) of the mean absolute error is in MAE_u), which is desirable (i.e., the majority of error is unsystematic). At the same time, the traditional decomposition into MSE_s and MSE_u masks the distinction between bias and proportionality error while also providing an underestimate of these combined systematic errors because it is inflating its representation of the unsystematic error (MSE_u) by squaring the model-predicted deviations from the regression line. As a result, the MSE-based measures suggest that there is little room for improvement when there is.

Bias correction (Fig 2b) produces a reconstruction model that has similar MAE (2.05 billion m³) to the original model. But, bias correction has produced much lower error in the two systematic terms of MAE, reducing MAE_b to 3% and MAE_p to 19% of MAE. From the slope of the regression line, however, it is clear that there still is some proportionality error that the bias-correction procedure has not entirely removed. The MSE-based measures present a rosier picture of the reduction of systematic error, again due to the inflation of the unsystematic error produced by the squaring of the deviations around the regression line. Overall, the MAE-based approach shows that there is still room for additional improvement in the original reconstruction (Fig 2a) and in the bias correction procedure (Fig 2b) than is evident in the MSE-based measures. In particular, the additional systematic component introduced here, MAE_p, suggests that high flows still need to be adjusted upward.

Conclusions

Traditional decomposition of sums-of-squared errors into systematic (MSE_s) and unsystematic (MSE_u) components has been a popular approach for characterizing the different components of model error. These sums-of-squares-based measures, however, have been shown to be imprecise and, at times, deceptive indicators of average error and its constituents. As a result, evaluations of model estimates and predictions should increasingly use absolute-value-based error measures such as MAE. To fill the need for a decomposition of MAE into its constituent components, we present new measures that are formed as weighted averages of the absolute error. As a result, MAE can now be decomposed into three components that represent bias (MAE_b), proportionality (MAE_p), and unsystematic (MSE_u) errors. These measures provide a more intrepretable standard for evaluating model errors while also pointing to more specific types of error that may be reduced.

Supporting information

Acknowledgments

The authors appreciate the informative and constructive comments of the two reviewers.