1.1 Quality of data

The question of how much relevant information a dataset contains is fundamental to applied statistics as a field. Fisher [1] describes the three problems that statisticians face as problems of specification, estimation, and distribution. However, Mallows [2] describes the “zeroth problem” of applied statistics, which must be addressed before any analysis can be done, as “considering the relevance of the observed data, and other data that might be observed, to the substantive problem. (p. 2)” Without the proper data, no modeling approach will yield acceptable results. However, there is no commonly agreed upon method for defining appropriate and relevant data. Pipino et al. [3] give sixteen dimensions on which data can be judged, including attributes like accessibility, accuracy, and amount of data, all scored on a subjective 0–1 scale. In contrast, the concept of “fitness for use” is a becoming a standard for judging the quality of a dataset [4]. “Fitness for use” is the ability of the data to answer the questions in the specific case that the user requires. Thus, the quality of a dataset is relative to the specific use for which it is intended. Shmueli and Kenett [5] formalize the ideas of data quality and fitness for use into a concept they call Information Quality (InfoQ). Information quality seeks to answer Mallows’ “zeroth problem:” determining how much information a dataset has about a specific outcome of interest. “InfoQ is determined by the quality of its components ‘quality of goal definition’, ‘data quality’, ‘analysis quality’, and ‘quality of utility measure’ and by the relationships between them.” The bounds generated in this paper are a quantitative measure of the “data quality” portion of InfoQ.

1.2 Predictability

This methodology further contributes to the research on the predictability of outcomes. Across the social and physical sciences, researchers are interested in what outcomes are relatively more or less “predicatble”. Wright and Goodwin [6] suggest that a result is predictable if well-calibrated estimates for outcome probabilities are able to be generated, and the true underlying probability distribution of the outcome is not too broad. Ehrenberg and Bound [7] claim that in order for a result to meet these conditions, it must have “recurred consistently under a known range of different conditions. (p. 167)” Taleb [8] argues that some outcomes, especially those in the distant future, are essentially unpredictable and unable to be accurately forecasted. Chaotic systems, those with extreme sensitivity to initial conditions, topological mixing, and periodic orbits, are nearly impossible to predict in the long-term [9]. This phenomenon makes it virtually impossible to predict weather more than two weeks into the future. On the other hand, processes that abide by relatively straightforward rules and have direct causes and effects are more likely to be predictable.

There has been research into which outcomes are predictable and which are not in a variety of fields, including health forecasting [10], information systems [11], information security [12], macroeconomics [13], risk management [14], water management [15], and energy demand planning [16–18], among others. These papers discuss individual outcomes or fields, but do not generate a systemic framework for assessing the predictability of a new outcome, or the use of a specific dataset in that prediction.

A stream of prior research has focused on generating bounds on classification performance for specific outcomes or specific algorithms. Seeger [19] derives bounds on minimum classification error for a family of Bayesian Gaussian Process classifiers, under specific assumptions about the data generating process. Rigollet [20] derives error bounds on semi-supervised classification problems, but assumes homogenous clusters in the data. Cover and Hart [21] show that a nearest-neighbor classification model will have less than double the probability of error compared to any other feasible classification model, parametric or not. These previous papers all have focused datasets with specific features, or on specific algorithm performance. In contrast, the bounding method described in this paper is generally applicable both in terms of data types and predictive algorithm used.

In addition to research on bounds of binary classification accuracy, there is also a stream of research using information theory and the idea of entropy to estimate maximum prediction performance on time series data. These papers generate a measure of the entropy in the outcome of some time series (e.g. location, demand, etc.), to estimate how much randomness is in the process. The rest of the time series is then theoretically able to be predicted. Song et. al [22] measured the entropy of human mobility using cell phones, and estimated that only 7% of location changes were completely random or unpredictable. Using this method, Zhao et al. [23] find that up to 83% of New York City taxi demand can be predicted using time series data. Lu et al. [24] show prediction performance of 87% on human mobility predictions using cell phone location data, compared to a theoretical maximum of 88%. In general, these papers find that the entropy-maximum prediction is approachable by machine learning prediction algorithms.

Another stream of work focuses on the prediction errors of individual observations. Bousquet and Elisseeff [25] develop upper bounds on leave-one-out prediction and classification error for any given algorithm, based on its overall performance on a dataset. They base their proofs on the idea of stability; small changes in the inputs of a dataset should not lead to large changes in the predictions of that algorithm. Other authors (e.g., [26–29]), use algorithm stability as a method to generate upper bounds on prediction accuracy for a given algorithm, clustering technique, or as a criteria for model selection. This stability assumption, that small changes in input correspond to at most a small change in output, is the same assumption that undergirds the upper bounding model presented in this paper. Table 1 shows the contribution of this paper in relation to the other most related papers.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Summary of related approaches.

https://doi.org/10.1371/journal.pone.0296904.t001

This paper offers a new approach to measuring the quality of information in a dataset. Discussions of the bias/variance tradeoff assume there is some reducible error that can be minimized through model complexity and metaparameter tuning [30], and some “irreducible error” which is inherent to the problem and data, and can never be eliminated [31–33]. This contribution of this paper is to give a model-agnostic method to quantify the amount of irreducible error [33] for a given prediction problem. While the idea of irreducible error, or best-possible performance has been widely accepted in the machine learning community, few efforts have been made to directly quantify the amount of this error in a general manner. Our paper is a first attempt to directly quantify this for any data set and for any modelling approach. While the model performs well in a range of situations. In practice, this paper presents a method of generating tighter bounds on a wider range on datasets than the other papers in the literature, at the cost of being a purely data-driven, heuristic approach that offers no performance guarantees or proven theoretical bounds. The limitations section discusses some cases where the method may struggle, due to this lack of theoretical assurance of performance.

2.1 L.P. formulation

Mathematically, the performance bound can be expressed as the following linear program (LP): (1) (2) with the following notation:

• P_i: The prediction for observation i.
• Y_i: The true outcome for observation i.
• and : positive and negative difference between the prediction and the true outcome of observation i.
• D(i, j): The distance in the input space between observations i and j.
• δ: A distance threshold, closer than which two observations are considered to be similar.
• f(D(i, j)): A function linking distance in the input space and the maximum allowable deviation in predictions.

Constraint (2) ensures that the absolute error for each observation gets added to the objective function. The optimal solution to the LP is the set of predictions that will minimize the sum of these errors. Constraint (1) ensures that if two observations are closer than some threshold, δ, then they will have similar predictions. The amount by which two observations are allowed to differ while still considered similar is determined by how far apart they are in the input space, as measured by the distance function D. Common distance functions, like the L1 or L2 norm, typically work well in numerical experiments. This paper reports results using the L2 norm. Once a distance function is defined, observations that are closer than the threshold, δ, are allowed to differ by no more than some function of that distance, denoted by the linking function f.

If δ is zero, the LP only has constraints restricting predictions of identical observations. At the opposite extreme, if δ is very large, greater than the maximum distance between any two observations, there will be a constraint on the maximum difference in predictions between every pair of observations. Somewhere between these two extremes is a value of δ that generates constraints for similar observations, but allows distant observations to vary freely from each other. In Section 3, simulated data are used to generate rules of thumb of how to choose a good value of δ. In Section 4,those rules are used to generate bounds on performance for real world datasets.

Without loss of generality, let us assume that the input features are normalized to have a standard deviation of 1. The linking function, f(D(i, j)), should be informed by the input data. For example, observations that have a distance of k between them (using the distance function chosen above) might be allowed to differ by β_max × k, where β_max is the largest absolute coefficient in a standardized linear regression model. If a one-standard deviation increase in any input variable is expected to change the outcome variable by no more than β_max, then a total one-standard deviation of distance between two points across all the inputs cannot predictably result in predictions farther apart than β_max. In this paper, β_max × D(i, j) is used as the linking function, and is demonstrated to generally work well in practice. Section 5 discusses edge cases where it may not work well, and proposes alternative linking functions in those cases.

2.2 Measures of predictability and performance

Based on the bounds methodology introduced in the previous section, a statistic, “predictive score” for evaluating a predictive model is defined. Adapting the basic idea of r² to the predictive case, and using the estimated upper bound, the predictive score represents the percent of the overall predictable variation that is correctly predicted by a model. The formula is given by: where MAE_Null is the mean absolute error from the null model, i.e. predicting the median for each observation, MAE_Model is the MAE of the predictive model being evaluated, and MAE_Bound is the lower bound on MAE generated by solving the LP formulation above. The null model is simply predicting the median for each observation, as this minimizes the MAE. Section 5 discusses minimizing mean squared error rather than MAE. In that case, the null model would be to predict the mean. The numerator measures the difference between the error of the model and the error of the bound and the denominator measures the total amount of predictable variation. This measure compares how well a predictive model performs relative to the difficulty of a problem or amount of information in the dataset, and is comparable across problems and domains. It can be interpreted as how much room for improvement in the model is theoretically possible, given the data available.

Let us also define a measure, ϕ, that quantifies how much of the variability of an outcome variable is possibly predictable by a particular dataset, as: Here, the numerator measures how much of the outcome is unpredictable, while the denominator measures the total amount of distance from the median in the outcome. This measure, ϕ, is a measure of how much irreducible error is in a prediction problem. A higher ϕ value implies that a higher fraction of the total variability in an outcome is irreducible. The ratio of these is the amount of the variation in the outcome that is not possible to predict due to the incompleteness of the data or the inherent randomness of the outcome. Subtracting this ratio from 1 gives us the percentage of the overall variability that is possible to predict; (1 − ϕ) of the outcome is unpredictable. If ϕ is 1, that means that MAE_Bound is 0, meaning that perfect predictive performance may be possible. If ϕ is 0 (and MAE_Bound = MAE_Null) it implies the outcome is completely random, or the data available have no information about the outcome.

The next sections provide examples of the methodology applied to simulated and real world datasets, and show how the Predictive Score and ϕ can be used to characterize datasets and prediction problems.

3.1 Linear data

This section showcases the performance of the methodology on a range of simulated datasets, emphasizing the accuracy of the performance bounds generated. The benefit of simulated data is the knowledge of the data generating process and therefore the bound can be compared to the true underlying model. The simulations vary the number of observations (N), number of explanatory variables (M), and amount of noise in the data. For each set of parameters, the mean absolute error of the true model and the minimal mean absolute error suggested by the bounds are measured, which are generated for a range of similarity thresholds, δ.

In the example below, each dataset consists of N observations of M i.i.d. Normal (0, 1) variables, and a Normal (0, σ²) noise term. Datasets were generated with 100 to 10,000 observations of 3 to 12 covariates, with σ ranging from 1 to 4. The outcome variable in each case is the sum of the input variables plus the noise variable. (i.e. Y = X₁ + ⋯ + X_M + σ). The simulations use δ ranging from 0 to 3.5, with the L₂ norm as the distance function. The linking function is f(D(i, j)) = 1 × D(i, j). This distance function is appropriate here as a one unit change in any of the input variables leads to a one unit change in the outcome variable. As a result, if the total distance between two observations is k units, then their predictions are allowed to vary by no more than k.

To evaluate the tightness of each bound, the difference between the fraction of overall variation explained by the true model, and ϕ, the fraction predictable indicated by the bound is calculated. Because the data generation process for the simulated data is known, the performance of the true model can be measured, and can be compared to the estimated bound by computing the difference. As an example, if the true model, as defined in the data generating process, predicts 73% of the total deviation in the outcome, whereas the generated bound indicates that up to 80% of the deviation is predictable, the looseness would be 7%, or 0.07. The closer the looseness is to zero, the more accurate the bound. Ideally, for the appropriate selection of δ, D and f, this difference would be zero. A negative looseness would imply that the generated bound on the error is smaller than the error of the true model.

The results of the simulations are summarized in Figs 1 and 2 below. Fig 1(a) shows the performance bound when N equals 750, and M equals 4, as the “closeness” threshold (δ) and amount of noise in the dataset (σ) are varied. In general as noise is increased in a prediction scenario the predictive performance deteriorates. This fact is reflected in the bound. For a given value of δ, the bound on MAE increases as a function of σ, that is, as the level of noise increases so does the bound on achievable MAE. Further, for a given level of σ, as δ increases so does the bound on achievable MAE, that is as the predictions of more and more pairs of observations are restricted to be similar, the bound increases. There are some boundary conditions. When σ is small, the observable input features do a good job of explaining the variation in the outcome, and bound on the error is very low, although increasing in δ. Similarly, if δ is small, very few constraints are added to the model, and thus the computed performance bound is again low, and remains very low for even high values of σ. When both σ and δ are high, the bound on the prediction performance, MAE, is high. If there is substantial noise in the dataset (σ is large), and δ is large enough, there are enough constraints in the LP model to enforce similar observations to have similar predictions, reflected by the high bound on the prediction error.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. The performance bound (a) and the Looseness (b) as a function of δ and σ when N = 750 and M = 4.

https://doi.org/10.1371/journal.pone.0296904.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Looseness as a function of (a) δ and M (N = 750, and σ = 3) and (b) N and M (σ = 3, and δ = 3).

https://doi.org/10.1371/journal.pone.0296904.g002

Fig 1(b) shows how the bound performs relative to the true model, as a function of δ and σ. As expected when δ = 0, and only identical observations are restricted to have identical predictions, the looseness of the bound is 1. When δ is large enough, (typically above 3 standard deviations in this case), the generated bound is close to the performance of the true model. In this case (M = 4, N = 750), the average looseness of the bound is approximately 10% or less when δ is equal to 2, and is under 7% when δ is equal to 3.

The bound on prediction error gets tighter (closer to the performance of the true model) as more constraints are added to the LP, which happens when N and δ increase. When δ increases, more constraints are added because more observations are considered similar. Increasing N makes the feature space more densely packed, which means more data points will have close neighbors, and thus more constraints will be added to the LP. The lower bound on prediction error gets looser as more explanatory variables are added. This happens because as the dimension of the input space grows, fewer observations are close to each other, and thus there are fewer constraints in the model; increasing N can help counteract this effect. Fig 2 demonstrates this behavior.

The behavior of the bound is studied in 5140 simulation runs, varying N, M, σ, and δ as previously described. On average, the bound on MAE ranges from 0% to 40% below the true model’s MAE, with it typically staying within 5% when both N and δ are sufficiently large.

The choice of δ plays a critical role in determining an appropriate bound. An overly large δ becomes restrictive, while an excessively small δ produces loose bounds. Based on the simulations, the following rule of thumb emerges: selecting the smallest δ that yields 5% of the total possible constraints (and at least an average of 5 constraints per data point, especially in smaller datasets) generally results in a tight bound. When δ is arbitrarily large, there will be N * (N-1)/2 constraints in the model, i.e. every pair of points is considered to be close to each other. Following this rule, the average MAE of the bound typically falls within 10% of the true model’s performance. This rule often results in bounds closer to the true model when there is lower noise in the dataset and when N is smaller. Nevertheless, it performs consistently well across a wide range of parameter configurations. The least effective application of this rule observed still maintained a looseness below 20%. Fig 3 shows the looseness of the bound as a function of N and σ when choosing the smallest δ that meets the 5 constraints or 5% rule.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. The looseness of the bound compared to the true model, as N and σ change, using the 5% rule of thumb to choose δ.

https://doi.org/10.1371/journal.pone.0296904.g003

3.2 Nonlinear data

Section 3.1 demonstrates that the bound works well on a wide range of datasets generated via linear data generating processes. In this section, more complicated datasets are simulated, including quadradic terms, interaction effects, and discontinuities. The bound still performs well, generating tight bounds that are close to the true model’s performance. Again, using the 5% rule for choosing δ and using the maximum absolute standardized regression coefficient for the linking function yields bounds that are tight and accurate, when certain broad conditions are met. To investigate the effect of discontinuities in the outcome on the performance of the bound, datasets with discrete jumps in the outcome variable are generated. In this case, four independent predictor variables are generated, and an outcome that is equal to the sum of the four inputs, plus Normally distributed noise, the same as the previous section. However, now a fixed value is added to the outcome for each input variable that is more than 2 standard deviations above the mean. The amount of noise in the dataset and the size of this discontinuity are varied, and bounds on the resulting datasets are generated. Fig 4 shows that the bound performs best, giving the tightest bounds when the discontinuities are small and when there is at least a moderate amount of noise in the data. When there are very large discontinuities in the data, the assumption of internal consistency that the bounding methodology relies upon is violated, and the bound returns a negative looseness. The true model can correctly predict these large discrete jumps, but the locally smooth predictions generated by the bounding algorithm cannot, leading to better predictions than the model claims are theoretically possible.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Looseness of the bound as a function of the size of discontinuities in the data and noise.

https://doi.org/10.1371/journal.pone.0296904.g004

Next, continuous nonlinear datasets are generated. Again, the datasets have consider four independent input features, but in this scenario, the outcome variable is constructed as the sum of the inputs added to the sum of α times the square of the inputs and further added to the sum of β times X_i multiplied by X_i−1, with an additional Normally generated noise term included.

The weights assigned to the interaction terms and on the squared terms are varied to assess the performance of the bound. Fig 5 presents the results of this analysis.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Looseness of the bound as a function of the size of interaction and squared terms in the data generating process.

https://doi.org/10.1371/journal.pone.0296904.g005

Moreover, the investigations revealed that when β_max is considerably large, the resulting constraints appear to be rather lenient. When the model allows a minor alteration in inputs to correspond with a substantial fluctuation in the anticipated outcomes, the solution space for the Linear Program becomes expansive. This extensive solution space often leads to the discovery of solutions with a minimal MAE, which consequently results in more relaxed bounds.

Conversely, in scenarios where β_max is either moderate or small, and the data doesn’t exhibit significant discontinuities, the formulated bounds exhibit commendable precision. Specifically, these bounds are tight, typically deviating by no more than 10% and notably, they do not tend to exceed their limits.

4.1 Technology adoption

In their paper, Agarwal and Karahanna (2000) build and test a theoretical model explaining intention to use the internet as a function of cognitive absorption. Specifically, they hypothesized that perceived ease of use and perceived usefulness are the two main explainers of behavioral intention of internet use. They collected survey data on 288 undergraduate students to measure these constructs and test their relationship. In all, they find significant relationships, with perceived ease of use and perceived usefulness explaining 48% of the variation in behavioral intent.

Using the data from this study, upper bounds on the ability of perceived ease of use and perceived usefulness to predict behavioral intent are generated. The two independent variables and the dependent variable are standardized to a mean of zero and a standard deviation of one. The distance between points is calculated using the L2 norm (using the L1 norm results in almost identical results) and used f(D(i, j)) = 0.478 × D(i, j) as the linkage function. This function is chosen because a linear regression of behavioral intent on perceived usefulness and perceived ease of use has the equation: Intent = 0 + 0.306 × Perceived Ease Of Use + 0.478 × Perceived Usefulness. A small change, ε, in input features could, therefore, be expected to lead to no more than a 0.478 × ε in the outcome.

The null model, predicting the median for each observation, has a MAE of 0.602, compared to 0.528 for the linear regression model reported in the paper. Depending on choice of δ, a minimum achievable MAE of 0.138 with δ = 0 to 0.388 with δ = 0.25 is found. That is, requiring only identical observations to have identical predictions, the minimum achievable MAE is 0.138. Increasing the closeness parameter from 0 to 0.25 (meaning that observations less than 0.25 standard deviations away from each other are considered similar), increases the minimum achievable MAE to 0.388. A value of 0.22 for δ is used because it is the smallest value for which the “larger of 5% or 5 constraints per observation” rule of thumb is met.

Using these numbers, , or that at most 35.5% of the total absolute deviation in the outcome is explainable using the data collected. The other 64.5% of the outcome is either due to inherent randomness, or to unmeasured factors of each subject. Of the 35.5% of the outcome that is predictable, the model reported is able to explain of the total variability that is explainable. As there is a large gap between the regression model performance and the theorized optimal performance generated by the upper bound, multiple machine learning algorithms to the data were fit. Both a neural network as well as a single regression tree have better performance than the simple linear regression. The best performance was achieved by a neural network with two hidden layers, of 6 and 3 nodes, respectively, with a MAE of 0.402, corresponding to 93.5% of the total explainable variability. Fig 6 summarizes the performance bound as a function of δ, with a vertical line at 0.22, the chosen δ value. As δ increases further past 0.25, the bound flattens out, because the allowable difference between observations increases as the distance between points grows larger. After a point, the additional constraints added to the LP are so loose that they do not change the optimal solution at all. The neural network model has a 24% lower MAE than the regression model. However, further refining of the model and tuning of the parameters should lead to at most marginal improvements in the performance of this model, as its MAE is now very close to the generated lower bound on MAE. Indeed, extensive parameter tuning efforts were unable to improve the performance of the model.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Minimum MAE as a function of δ for the technology adoption study.

https://doi.org/10.1371/journal.pone.0296904.g006

4.2 Automobile MPG data

Next, bounds were generated for the prediction performance of machine learning models on the automobile-MPG dataset from the UCI machine learning library [34]. This is a commonly used dataset for machine learning papers, with typically good performance reported on out-of sample predictions, with MAEs as low as 2.02 MPG [35]. It contains information on the weight, acceleration, horsepower, model year, continent of origin, and fuel economy of 392 cars. The null model’s error, using the sample median as the prediction for each car, is 6.524 MPG, and a simple linear regression gives an MAE of 2.5 MPG. The best machine learning model found was a neural network with an MAE of 0.90. Generating the lower bound as a function of δ (summarized in Fig 5), found a lower bound of 0.85 on the MAE, using the L2 norm as the distance function, and the 5.70 × D(i, j) as the linking function derived from a simple linear regression model on the normalized data (each standard deviation of weight decrease fuel economy by 5.7 MPG). In this case, δ = 1.25 was chosen as the similarity threshold, as this is the smallest value that satisfies the 5% rule of thumb. This implies that there is little room to improve the neural network model with a reported MAE of 0.90, compared to the estimated lower bound of 0.85. Given these car characteristics, 87% of the car’s MPG is predictable, while the remaining 13% is explained either by other factors, or by idiosyncrasies of each car. The results of this study are shown in Fig 7.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Minimum MAE as a function of δ for the automobile MPG study.

https://doi.org/10.1371/journal.pone.0296904.g007

4.3 Stock returns

Last, the use of the performance bound methodology on a large dataset of historical stock prices is demonstrated. A bound is generated using ten thousand randomly selected stock quarters from the NYSE between 2000 and 2010. The outcome of interest is the fractional change in value of the stock based on financial fundamentals. Using normalized input vectors of the stocks’ income, liabilities, earnings per share, and stock price, the model predicts how much a given amount of money invested in a stock at the start of a quarter will be worth at the close of the quarter. Because the returns are skewed, the logged value is used as the outcome variable.

The linking function is f(D(i, j)) = 0.013 × D(i, j), because when running a linear regression on a normalized dataset, the maximum absolute coefficient is 0.013. This means that a marginal change, ϵ, in any input variable corresponds linearly, on average, to no more than a 0.013 × ϵ change in the outcome (i.e. a 1 standard deviation change in the inputs can correlate with no more than a 1.3% increase or decrease in value).

Using the L2 distance measure and a δ of .04, which is the smallest value that satisfies the rule of thumb derived above, that the lower bound on MAE corresponds to 24% of the total variation in the sample. The lower bound for the mean absolute error is 0.2128 log-dollars, when the initial investment in each stock is $10,000. The null model, predicting the median for each observation, has an MAE of 0.2815. This leaves the remaining 76% of the deviation in stock returns as random or only predictable using data outside these fundamentals. The fact that so little of the outcome is predictable means that there is no model that can consistently give accurate predictions. Compared to the MPG problem, where the mileage is mostly predictable, the bound methodology shows that predicting stock returns from fundamentals is a “hard” problem, with little usable information in the dataset. Indeed,the bound shows this to be true: linear regression and neural network models have mean absolute errors of 0.282 and 0.280 respectively, hardly any different than random guessing.