Model selection

Linear regression is the simplest and most popular modeling technique that finds the line of best fit by minimizing the sum of squared residual errors. The strength of linear regression lies in the easy interpretability of resulting coefficients. Among the many widely known limitations of linear regressions, an important one is its focus on minimizing in-sample error, which can lead to problems of “over-fitting” and poor performance while predicting out-of-sample data. For improving the predictive capability of a model, it is important to reduce the out-of-sample error, even at the expense of trading off in-sample error. Machine learning techniques offer an empirical way to manage this trade-off and improve out-of-sample predictive performance [54]. Specifically, in this paper we use 10-fold cross validation technique to obtain the out-of-sample prediction errors (also called Test Mean Standard Error or Test MSE) for comparing the performance of different models.

In this paper, we use linear regression, linear regression with forward and backward subset selection, lasso regression, random forest, and XG Boost models to investigate the socio-economic determinants of new firm starts in Texas counties. We have selected these techniques for their ability to model a continuous response variable, as well as their ability to provide interpretable outputs, albeit in different forms. Other models such as K-nearest neighbor (KNN) regression and Principal Component Regression can also be applied but offer less interpretable results. A number of textbooks such as [55] provide a detailed introduction to a range of machine learning techniques that may be applied over the interpretability vs. complexity tradeoff spectrum.

Forward and backward subset selection models work by iteratively adding (or removing) variables from a series of models, one-at-a-time, at each step considering which variable offers the greatest additional improvement (or reduction) in model fit. From this series of models, at each step the one with the least Bayes Information Criterion (BIC) statistic is selected as the best performing model, thus yielding a “greedy” (i.e., suboptimal) but computationally efficient approach. Lasso regression belongs to a class of “shrinkage” or “regularization” models that apply a tunable penalty term while minimizing the residual sum of squares, resulting in a sparse model wherein less important coefficients could be set to zero. Random forest is a class of tree-based models that aggregates repeated, decorrelated decision trees build from random subsets of predictors at each step. Finally, extreme gradient boosting, or XG Boost, is a technique that uses the processes of gradient boosting and regularization to improve the prediction rate by adjusting the weights and penalties for errors made by previous weaker models.

The random forest, lasso regression, and XG boost models were tuned to optimize the hyperparameters to minimize the out-of-sample errors, as is standard in the literature [56–58]. The random forest model was tuned to select the optimal mtry value (where mtry is the number of variables randomly sampled at each split). In the lasso regression model, cross validation was used to select the lambda value which minimizes the mean cross-validated error. In the XG boost model, and in line with prior research, a grid search with 5-fold cross validation was performed to tune five hyperparameters—the maximum depth of a tree (max_depth), subsample ratio of the training instance (subsample), subsample ratio of columns when constructing each tree (colsample_bytree), lambda, and alpha values [59–62].

In terms of outputs, linear regressions provide the magnitude and direction of the coefficients associated with each predictor, along with p-values indicating their level of significance. Subset selection and lasso regression models result in a selected subset of predictors along with their magnitude and direction of impact. Random forest models generate variable importance plots, which is a summary of the extent to which each predictor contributes to the reduction in mean squared error and gini index, which are both indicators of their relative level of importance.

We chose these techniques for their ability to model a continuous response variable, as well as their ability to provide interpretable outputs, albeit in different forms. Other models such as K-nearest neighbor (KNN) regression might be used but offer less interpretable results. Many other techniques such as Principal Component Regression, Support Vector machines, Linear and Quadratic Discriminant Analyses and other tree-based techniques can be applied based on the nature of the research questions and the underlying data. Several textbooks such as (James et al., 2013) provide a detailed introduction to these machine learning techniques.

A multi-model interpretation framework

Each machine learning method provides a different type of output to assist in interpreting the importance of predictors. The strength and relative importance of predictors might vary across different methods, and models may themselves have relatively different predictive capabilities. To assess the overall contribution of each predictor from across all the models considered, we develop a novel combined interpretation framework that allows for systematic aggregation of outputs from multiple models. The rationale for this combined interpretation framework is to achieve three objectives: (a) prioritize variables with higher magnitude of association within each model, (b) prioritize variables that appear as significant across multiple models, and (c) prioritize variables from models that have relatively better predictive performance.

The combined interpretation framework consists of three steps (Fig 3). First, we take the list of variables from each model output and rank them according to their importance. The ranking scheme depends on the nature of output from each model. In the random forest model, we rank predictors according to their contribution to reducing the percent mean-squared error. In all other models, we rank the predictors according to the magnitude of their (standardized) coefficients. This ensures that the predictors that are more important in each model get higher ranks. Further, the predictors missing in each model and predictors that are not statistically significant in the linear regression model are penalized by assigning their rank as one greater than the total variables in the largest model. This ensures that the predictors that are not significant in each model get lower ranks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. A three-step framework for interpretating of results of multiple ML models.

https://doi.org/10.1371/journal.pone.0287217.g003

Second, once the rank for each predictor is assigned for all models, these ranks are weighted by the respective model’s predictive accuracy. In our case for example, we weight the ranks from each model by that model’s 10-fold cross validation error. This ensures that the predictors in better performing models get higher weightage.

Finally, we combine the weighted ranks from all models. To do this, we take the average rank of each predictor across all models. Then, we rank those averages. This ensures that the predictors appearing across multiple models get higher ranks. This is the final measure of each predictor’s overall importance in our models, as described below.

County characteristics and demographics

Prior research has suggested that high rates of firm creation are associated with a number of demographic and cultural characteristics of a local population, including its age distribution [11], education levels [14, 64], ethnic diversity [16], culture [12], social capital [65], population growth [66], rates of immigration and age of immigrants [67], population density [16], and (oddly) the ratio of Protestant to Catholic religious adherents [11]. In the consolidated results in Table 6, some of these stand out, including age distribution, ethnic diversity, social capital, and migration, in descending order of predictive variable importance.

One other factor merits discussion here: the number of new firms started in the previous year. There are at least two ways to interpret that the number of firms started in the prior year is predictive of number of firms started in this year. First, we might understand it structurally: Those factors that are associated with higher entrepreneurial rates carry over from year to year. Second, we might understand it culturally, as many of the authors in Table 1 theorize: Once an entrepreneurial culture is established in a place, it becomes more culturally appropriate to take entrepreneurial risk [27].

Economic variables

Prior research has also identified strong associations between firm creation and economic variables, including access to capital [9], industry diversity [24], interest rates, changes in technology, and the rate of business failures [11]. From this perspective, demographics—education levels, unemployment rates, income equality—become measure of labor market readiness, and the availability of local workers has been shown a predictor of entrepreneurial outcomes [8]. We included a wide range of economic variables in our analysis; The most important across all models include the unemployment rate. Not salient were hypothesized economic factors such as total employment, county GDP, patents, interest rates, and business density.

Understood through the lens of labor market economics, the middling ranking total employment in our models stands in contrast to prior research, which suggests that low levels of unemployment make it less attractive for would-be entrepreneurs to start businesses and more difficult for them to hire employees [26]. More in line with previous research is the association between the number of local banks and firm creation. Internationally-focused research on firm creation across countries often includes variables regarding access to finance [10, 14], though US-focused research has not. This may be in part because banking in the United States has become so consolidated that the importance of local banking institutions with a physical presence has been overlooked.

Two surprising results are the relatively low predictive power of patents and business density on firm creation. Literature on patents suggests a strong set of links among patents, innovation and entrepreneurship [11]. But in two of our models, that relationship is significant and negative. This lines up with some recent scholarship suggesting that the United States is unusual in how much patents are concentrated in its large cities [68] and that patents are a less helpful proxy for innovation in rural areas than urban ones [69, 70]. Future work might examine this more closely by decomposing patents into invention, design, and plants patents. Business density, too, has been shown by prior research to be associated with economic resilience [3]. But our results suggest that it is not greatly predictive of firm creation. Interpreting these two outcomes together, it may be that the agglomeration effects present in large cities may not be as powerful in urban areas, where more dedicated work is needed to induce spillover effects of innovation [71].

One variable receiving considerable attention across the country and in Texas specifically is access to broadband [21, 72]. While perhaps of importance in areas like equitable access to health care and education, broadband access does not emerge in this data as predictive of firm creation. We should note here that broadband access data was only available for two years; this is absence of evidence, not clear evidence of absence.

Energy and other industry variables

Recent scholarship has explored the relationship between the oil and gas industry, using oil production both as an instrumental variable [73] and an explanatory one [26], finding that oil and gas expansion is negatively associated with firm creation. In our analysis, we re-examined this relationship, along with two other energy industries (solar and wind) that have grown rapidly in Texas in the last twenty years, along with other sectors (manufacturing, farming, extraction, education, healthcare, and elderly care) that are of particular importance in many rural areas.

Of our industry variables, four emerged as most important: wind, oil, healthcare, and elder/childcare. The negative association between oil production and firm creation in two of our parametric models is not surprising, given prior research suggesting that large extractive industries depress firm creation [26]. It may be that firms that spring up in support of that industry are outnumbered by the firms crowded out by higher wages in the oil industry. That oil, but not gas, is negatively associated requires some explanation. One possibility is that extraction of oil and gas in Texas are more properly considered two industries than one, with different value chains—something that makes practical sense given the global integration of oil markets as compared to the primarily domestic use of natural gas in the United States.

The relationship between wind energy and firm creation is clear, novel, and worthy of further study, especially in comparison with the absence of a relationship between solar energy and firm creation. Prior research has examined the positive economic impacts of specific wind energy projects [74], but to our knowledge this is the first test of the association between wind energy and entrepreneurship.

Comparing predictors of urban and rural entrepreneurship

In addition to the models above, which predict firm creation in rural counties, we ran a second set of models for non-rural counties. Table 7 compares these results. It includes the top fifteen predictive features in both rural and non-rural counties, according to our multiple-model variable importance rankings (holding aside the control variables for specific years). The list of factors that predict non-rural firm creation is materially different from those predicting non-rural entrepreneurship. Patents, local bank deposits, education levels, employment levels, and natural gas production emerged as influential predictors of urban firm creation. The first three of these, at least, are well-studied in existing (and mostly urban-focused) scholarship on firm creation but did not emerge as important predictors of firm creation in rural areas.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Comparing urban and rural predictors of firm creation.

https://doi.org/10.1371/journal.pone.0287217.t007

Predictive in rural areas, but less so in urban areas are the size of the healthcare and elder/childcare industries, oil production, age distribution, the unemployment rate, social capital, the number of local banks, and in-migration. Common to both are population size, the number of firms created in the previous year, wind capacity, and ethnic diversity.

Implications for rural entrepreneurship research

Existing research on rural entrepreneurship is epistemologically and empirically disjointed. That is not a surprise on an issue as complex as the creation of new firms. One logically sound approach to such a complex issue is to break it into smaller, model-able parts. This approach has led researchers to clarify on the causal relationship between some variables and firm creation. But it has also led to an econometric version of the parable of multiple blindfolded investigators attempting to describe the proverbial elephant. A full, clear description of a single feature of an elephant may get us a little closer to describing the whole beast, but it is hard to know that we have isolated the most useful piece of information.

For research on entrepreneurial determinants, a machine learning-driven approach offers an avenue for us to take a step back from a complex problem and observe its multiple theoretical parts empirically, identifying which facets merit greater attention. In the case of rural entrepreneurship, those factors include the age distribution, ethnic diversity, migration patterns and social capital of a rural population; specific industries, such as oil, wind, healthcare, and local banking; and some economic variables, such as the unemployment rate. It may also reveal which features might have gotten too much attention, either because of political palatability (such as access to broadband) or because existing research, focused on urban areas, focuses on factors (such as patents) that fail to translate into rural contexts.

Implications for rural entrepreneurship policy

These results can help policymakers in two ways. First, they can help policymakers base decisions on better predictions of what kind of firm creation to expect in their rural counties and states. Second, they can help policymakers identify what factors should get more or less time on the research and policy agenda. In many of these cases, a direct policy may not be possible—a county cannot grow its population by fiat, for example. But it can work together, as some small towns across the U.S. have, to create enough linkages with other nearby areas to operate as a bloc, perhaps mimicking some of the advantages of larger populations.

In other areas, this research suggests other potential levers that rural policymakers might want to examine more closely, including the number of local banks; attracting younger and more racially diverse workforces; investing more in healthcare infrastructure; and (where geographically feasible) exploring wind energy production, for example. It should be reiterated here that this research establishes predictive relationships, not causal ones: It can help us prioritize which policies to investigate more deeply and which to deprioritize. However, estimating the potential effects of a particular policy option requires a shift from examining prediction to establishing causation.

In other cases, these results may suggest levers that policymakers cannot pull. Counties in Southeast Texas, for example, might not be able to develop wind capacity. But there may be other major infrastructure projects whose contracting could be made to look more like those used in the wind industry, potentially creating conditions for firm creation.

The results also suggest that the importance of some pro-entrepreneurship policies proposed by lobbying groups and think tanks, such as greater access to broadband, is not clear when ranked against other predictive variables. And these results suggest that some well-studied variables shown to be important in urban areas, such as the number of patents, are less predictive of firm growth in rural areas. If the presence of variables provides a clue as to what policymakers should prioritize, the repeated absence of variables suggests variables they should consider deprioritizing.

Some of the variables that emerged in these models as important are just as complex as those of firm creation. How would a rural county increase the number of local banks, for example? This question, too, is surrounded by a fog of often-disconnected policy briefs and consultant recommendations; getting a clear answer of where to focus lends itself equally well to the type of multi-method machine learning we have conducted in this paper.

Implications for public policy scholarship

The process laid out here offers a model for how policy scholars might approach other kinds of questions, under certain conditions. First, these methods are most helpful when scholars are concerned with comparing policy options rather than identifying the impact of a particular intervention. If an independent variable of interest has been identified, traditional regression methods may be more appropriate. But when dealing with policy problems as complex yet nascent as rural entrepreneurship, scholars often apply multiple theoretical approaches to a particular problem and seem to talk past each other. In such an example, the initial research goal may be to establish the relative importance of the independent variable to others.

Second, these methods are most useful under certain data conditions: Specifically, with large data sets, and those where the number of predictors (p) is much larger than the number of observations (n). Over the last couple of decades, many machine learning methods have been designed precisely to overcome this curse of dimensionality, which is expressed in shorthand as p>>n. Conversely, where there are a great many more observations than predictors than observations (n>>p), classical econometric models may be more appropriate depending on the nature of the data and may provide the necessary leverage for pointing to causal mechanisms. For p>n and even n>p (as in this dataset), there is less clear-cut guidance on the types of models to use. The same is true for complex datasets with many features (independent variables) in applied domains, such as rural entrepreneurship, that are still evolving and so do not have a firmly established set of features that are standard across all models. Under these circumstances, the analytical framework we present in this paper could serve well to advance understanding of the underlying data and to offer fruitful pointers for deeper study.

Lessons for policymakers interested in complex problems

For decades, policymakers looking to implement evidence-based policy have turned to research for answers. That research has turned most often to a set of linear regression models designed to identify as tightly as possible the causal relationship between two variables, often transformed into just the right shape for analysis. Even when those results are meaningful, they can only answer one type of question that policymakers might ask. Machine learning techniques offer a new and growing set of tools to help policymakers answer additional questions about what policy inputs they should prioritize given a particular policy outcome of interest. Answering such questions is especially important in the early phases during which the policy agenda is still taking shape before fully crystallizing. In instances where policymakers’ task is to identify where to put scarce public resources to develop deeper and more robust understanding of potential solutions, or in scenario planning situations where prediction is the central task, predictive machine learning methods can prove useful additional tools to traditional causality-seeking approaches.

These models are not silver bullets, for at least two reasons. First, the added complexity of the models means that they are open to misinterpretation or intentional manipulation (or what might be called disinterpretation). [28, 40] provide two recent discussions of these issues with the application of machine learning in public policy research. Second, machine learning models are just a tool, and when wielded without care have the potential to be discriminatory. In the human context, data is one manifestation of reality, created in a particular socio-political context, and as such is often shrouded with complex, idiosyncratic drivers. Who gets to interpret the data and models is inextricable from questions of power in that context. [75, 76] offer discussions of these dangers and how they might be addressed.

This is not a new problem, however, nor one unique to the kind of computational methods we use in this paper. In fact, the potential for policy actors to abuse emerging methods was one inspiration for the multi-method approach we lay out here. Data is always collected by and filtered through the hands of humans embedded in institutions with their own biases. This multi-method approach has the potential to mitigate that in two ways: first, it provides a mechanism through which more data can be incorporated into analysis, lowering the possibility of omitted variable bias. Second, it makes it harder for a policy actor to choose a single model that supports their agenda. Third, it has the potential to spur cross-agency coordination on data collection, and especially for the variables that show up as important in limited variable analyses and our larger variable pool study. With these dangers and possibilities in mind, we interpret our and others’ models humbly—and we encourage our readers to do the same, especially when our results line up with our prior beliefs and biases.