3.1 Stacking

The IHDS micro-data are the point of departure for building our stratification frame. We generate cell counts for cells in the IHDS that are defined by the joint distribution of {State—Zone—Gender—Age—Educational Attainment—Family Role—Marital Status—Rurality—Religion—Literacy—Caste—Yearly Household Income}. Note that these cells are deeper than necessary for the vote choice model but we select a surplus of variables to help with multiple-imputation.

The IHDS dataset has limitations. First, the marginal distribution of each variable by state is unrepresentative of the true state population. A second related limitation is that the underlying interactive structure across the variables, i.e. the way in which educational attainment is correlated with caste, will be spurious at the state-level. A third concern is that the IHDS does not include political variables such as past vote-choice and turnout behaviour. We address all three of these issues by implementing a large data-integration technique: we merge the IHDS with data from the decennial census; the Indian National Election Study (INES) [36] and the Lok Dhaba (https://tcpd.ashoka.edu.in/projects/) [19] archive of past election results.

First, we update the underlying interactive structure of the IHDS variables. We want the underlying joint distribution at the state-level to match that of the population. Our state population targets for the joint distribution are derived from the deepest decennial census tables we can find, namely tables containing counts for cells defined as {State—Zone—Gender—Age—Educational Attainment}. Tables containing caste, arguably the most predictive variable for vote-choice a-priori, were not as deep, failing to contain a breakdown by age and gender; moreover, the caste classification used in the census was different from that used in the IHDS, and hence was not immediately reconcileable. We could in principle directly rake the IHDS to these census targets, though a problem emerges—namely some of the voter-categories that exist in the census do not exist in the IHDS. This is due to some of these cells being relatively rare in the population, and hence not featuring in the relatively small random sample collected by the IHDS. To avoid instability in raking weights [37], we choose to populate these missing cells using a stacking procedure.

The stacking procedure begins by creating a “census” micro-data set from the joint-distribution tables provided by the Indian census agency. We know each cell’s joint-distribution (defined by state, zone, gender, age and education) in the population. This determines this cell’s probability of selection into a random sample of the population. We extract a 0.5% random-sample of this population which gives us a micro-data sample of 3, 800, 213 individuals. This new sample size is defined in order to be large-enough to include the largest cells from every state.

A second step in this procedure stacks this “census” micro-data sample below the full IHDS data, treating variables that are present in the IHDS but not in the census sample as ‘missing’ (this will be the case, for example, for the religion or caste variables). This leads to a new augmented dataset of size 3, 936, 197, with a swath of missing data for records coming from the census. The resulting joint-distribution for the overlapping variables is now dominated by the census and reflects their actual distribution in the population. The price we pay for correcting the joint distribution is that we now have a large amount of missing values; we deal with this problem at the end of this Section.

Our final step in the stacking procedure augments the covariate space with political variables, and, in particular, turnout and vote-choice from 2014. As Table 1 indicates, we use the post-election National Election Study (NES) from 2014 for this augmentation exercize. It contains cells defined by the joint distribution of {State—Zone—Gender—Age—Religion—Caste—Educational Attainment—Yearly Household Income—2014 Turnout—2014 Alliance Vote}. We perform the same stacking procedure, treating non-overlapping variables as entirely missing. Once we append the NES to the augmented dataset from the previous two steps we have a complete augmented data set with 3, 958, 492 records; although with a significant number of completely missing values. S2 Table in S1 Appendix shows a subset of the stacked dataset.

3.2 Multiple imputation

We propose an imputation strategy for estimating quantities for the large number of missing values in this augmented dataset. To do so, we rely on the Multiple Imputation via Chained Equations (MICE) [38] paradigm. [6], identify two pre-requisites for the MICE procedure to produce meaningful imputed values for the stacked object. First, the values should be ‘missing at random’ [39]. This is a plausible characterization of the three datasets we used for constructing the augmented dataset. Second, these datasets should share a data generating process. We assume this is true a-priori given that all three datasets are designed to be representative of the true population. The INES stands out as a possible exception to this common DGP assumption. While the dataset is largely representative of demographics, it is almost exclusively a sample of likely voters, with 91% of respondents reporting to have cast a ballot in 2014, highlighting important selection-effects. To overcome this problem, we rake the INES data to the known 2014 election results, at the state level, hence recovering the correct vote-choice and turnout marginals at the state-level. Raking works by re-weighting the observations in the INES to match the marginal distribution of each of our target variables (turnout and vote-choice over the three alliances) in each state. The target distributions are derived by the official 2014 election results in each state; we then proceed to take a bootstrap sample of the dataset, using the standardized raking weights as the sampling probabilities. The resulting dataset can then reasonably be thought of sharing a DGP with the IHDS and the census, though deeper selection issues relating to the INES are to be expected. Deeper selection issues here refer to non-observable selection effects, which cannot be mitigated by simple re-weighting. We have reason to believe that, even conditional on all observable attributes, respondents of the INES will be different from comparable profiles in the population. We suspect this is the case based on the recent experiences with opinion-polling in the UK and US, where we find selection into non-representative samples (of which the INES clearly is one) is correlated with typically non-measured attributes such as social trust [40]. We consider large biases in turnout to be an important indicator of potential selection on non-observables.

MICE can be extraordinarily computationally expensive, as it seeks to model a series of full-conditional distributions over multiple iterations. missForest [41] is a MICE-based algorithm designed to deal with high-throughput data, and hence more ameanable to processing our large-dataset. missForest uses Random Forests [8] to predict missing values at each MICE step. It is preferable to parametric MICE as it performs automatic variable selection and regularization; can deal with mixed-data types; and has been shown to out-perform traditional imputation models under multiple settings [42]. To further improve the speed of our imputation model, we rely on the ranger [9] package for the R [43] statistical programming language to train the random forests at each MICE step; the ranger algorithm leverages a C++ backbone to optimize memory-handling and processing of Random Forests, enabling a fast implementation in high-dimensional settings. The missRanger [44] package enables us to implement missForest with the superior ranger algorithm. We can control a number of hyper-parameters in missRanger; we limit ourselves to increasing the number of trees to 1, 000 (num.trees = 1000), a number we consider large-enough to preclude Monte-Carlo effects in the imputation estimates, while keeping other relevant parameters as the ranger default; each tree is fit to a complete bootstrap-sample of the original data (sample.fraction = 1, replace = TRUE); no maximal depth is imposed on each tree (max.depth = NULL); and all available (complete and incomplete) covariates are used as split-candidates.

To make the imputation more robust, we introduce a number of state-level predictors, including: past election results at the state-level from the Lok Dhaba archive; demographic characteristics derived from the census; and a spatial matrix indicating each state’s distance from any other state—in order to account for a degree of local-smoothing [45]. The total number of area-level predictor variables is 175: 35 are distances from other states; 32 are past-vote variables, including, for example, past vote by alliance at the state and zone level, as well as past turnout, for the last three elections; and the rest are zone- and state-level aggregation of individual-level demographics. The algorithm takes approximately 12 hours to converge after 6 iterations, on a r4.4xlarge AWS EC2 instance—details on Instance types are available on the AWS website: https://aws.amazon.com/ec2/instance-types/.

3.3 Iterative proportional fitting

Having performed multiple imputation, we now have a complete dataset with 3, 958, 492 individual records. But this constructed sample of the Indian population remains unrepresentative at the state-level. We relied on information from the IHDS and INES to impute much of the state-level joint-distributions. And as we pointed out the INES estimates of vote choice and turnout have limitations; both the IHDS and the INES lacked statistical power for many of the joint-distributions estimated; and of course the MICE estimations are likely to be quite noisy given the volume of missing values in the data. S1 and S2 Figs in S1 Appendix show the distribution of 2014 voting behaviour at the zone and state levels described by the post-imputation stratification frame. The Indian states are organized into six zones: Northern, North Eastern, Central, Eastern, Western and Southern. The state-level imputation is poor. At the zone-level the imputations perform better; errors ranging between 7 and 9 points on average for the two main parties. At the zone-level the imputed turnout is reasonably accurate, though clearly the imputed turnout is larger than the true 2014 turnout across the board, revealing perhaps further evidence of INES selection effects and over-reporting. It is worth noting that we do not need these imputations to be perfect at the area-level because we will aggregate over these area-level estimates to produce a national-level forecast. Nevertheless, we want our frame to be as representative of the true population as possible at the deep-cell level in order to allow for robust disaggregated forecasts.

To improve the representation of the dataset at the deep-cell level we employ IPF, as suggested by Lauderdale et al. [5]. The IPF iteratively re-weights each individual in our large post-imputation micro-frame, until these conform to a target set of distributions. The population targets are for our most precisely measured variables: a) the joint distribution of state, age, gender and education as reported in the census tables; and b) the joint distribution of state and 2014 voting-behaviour. IPF is implemented with the R package anesrake [46]. We rake areas independently, in order to speed-up convergence and avoid having the algorithm reconcile distributions at different levels. As a result, raking weights could change the relative sizes of the states. To ensure the states remain consistently sized, we re-scale the weights to match the size of each state implied by the census.

S3 and S4 Figs in S1 Appendix show the resulting match between the observed 2014 voting behaviour and our post-imputation and post-raking frame, at the zone and state-level respectively. Typically, state-level 2014 voting behaviour in the frame now exactly matches actual voting outcomes. We do less well matching state-level turnout in the frame compared to actual turnout statistics; though vastly better than the pre-raking estimates. The poor state-level turnout performance is largely a result of the limited turnout training data we have. Recall, the INES had little reported non-voting. Our efforts to correct for this by taking a re-weighted bootstrap sample were not entirely successful. As a result, non-voters are under-represented in the post-imputation frame. This limits the re-weighting options of the IPF algorithm, effectively precluding a perfect match. Nevertheless, the match is perfect at the zone-level, and close-to-perfect at the state-level for the major parties.

As a final step in the construction of a representative stratification frame for the 2019 Lok Sabha election, we aggregate the weights to the cell-level. Each row now represents a unique joint distribution, with its appropriate weight. This reduces the size of the object to keep in memory by reducing the number of rows. Table 2 displays a subset of the completed frame. The frame is representative of 2011 demographics and 2014 voting behaviour. Given the fast-evolving demographics of India this is not optimal but reflects the most current data available.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Subset of the complete stratification frame, aggregated at the cell-level.

https://doi.org/10.1371/journal.pone.0260092.t002

4.1 Turnout training data

The accurate measurement of turnout patterns at the deep-cell level is a challenge in traditional opinion polling under optimal conditions, and it becomes even more challenging in India, where traditional polling practices (i.e. simple random samples) are not viable. An accurate estimate of cell-level turnout weights is primarily of importance here because they translate into more robust estimates of national vote-shares that are conditional on turnout. Our ultimate forecasting goal is at the national level although deep-cell turnout estimates are an important contributor. We argue for two important features of these forecasts: a) the leveraging the INES turnout responses instead of those from our online pre-election survey; and b) we use the past-election turnout distribution to predict the present.

4.2 Severity of non-representation

We get an indication of the extent to which the two convenience samples and the India NES, are not representative by comparing their distributions to those of the stratification frame. S6–S14 Figs in S1 Appendix plot the distributions of a select number of variables in the stratification frame (our target population), against their convenience sample counter-parts. S6 Fig in S1 Appendix presents the population v. sample comparison for the distribution of 2014 voting behaviour; S7 and S8 present comparisons for the distributions of states and zones; S9 compares population and sample in their gender composition; S10 and S11 look at religion and caste respectively; S12 and S14 provide insights on the distributions of income and education, whilst S13 presents the comparison between population and sample in the distribution of age categories. Note that we employed sampling constraints based on zone, age and gender in the later stages of sampling for the Facebook pool which resulted in a more representative pool compared to the Mechanical Turks, though these constraints seem to have been more effective on geography than gender or age. Fig 1 showcases the degree of ‘representation’ for the two convenience samples and the INES across a number of marginal distributions (these are disaggregated in the S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Summary plot assessing the degree of bias, when compared to the estimated stratification frame, across a number of marginal distributions, in each of the three samples used in the analysis.

The most severe discrepancies are highlighted. Each dot represents the % of an attribute, such as education level, caste or income, in the population as a whole and in the sample at hand. If the sample at hand is perfectly representative of the population of interest, the dots should lie on the y = x line.

https://doi.org/10.1371/journal.pone.0260092.g001

With respect to geographic distribution across the samples (S7 and S8 Figs in S1 Appendix), our work confirms previous findings [55, 56] regarding the extreme over-representation of Tamil Nadu and Kerala; together they make up only 9.8% of the voting age population in our stratification frame and yet represent over 80% of Mechanical Turks. Geographic over-representation in the Facebook sample is not severe, partly as it is controlled by sampling quotas, and partly because the geographical distribution of Facebook users is more representative of the population [55]. The distribution of states in the 2014 INES mirrors that of the population.

Gender (S9 Fig in S1 Appendix) is properly balanced in the INES. Both convenience samples over-represent males—with Mechanical Turks being further skewed toward males than Facebook users. Religious affiliation in our convenience samples is similar to that in the stratification frame, though unsurprisingly due to the geographical skew in favour of Kerala and Tamil-Nadu, Christians are over-represented amongst Mechanical Turks (S10 Fig in S1 Appendix). With respect to age, the INES is representative of the population at large while the convenience samples are biased: Mechanical Turks are overwhelmingly between 25 and 35 years old, while Facebook users are even younger, exhibiting over-representation of all under 35s (S13 Fig in S1 Appendix).

Perhaps unsurprisingly, Brahmins make up a plurality in both our convenience samples (S11 Fig in S1 Appendix). We heavily over-represent individuals in the ‘Other’ caste category which includes people who do not subscribe to caste system, refused to be labeled or explicitly selected an ‘Other’ option (a response significantly more common amongst Facebook users). Individuals from ‘Scheduled Tribes’ are under-represented. The distribution of caste in the INES, on the other hand, seems to over-represent Backwards Castes.

The INES best approximates the population in terms of yearly household income (S12 Fig in S1 Appendix) while the Mechanical Turk and, in particular, the Facebook samples are less representative on income. Mechanical Turks are richer than the voting-age population: 16% of the sample is in the top two income brackets. Facebook users are overwhelmingly high-income, with 59% of the sample clustered in the top-two income categories. In the reference population 1.6% are concentrated in these top-two categories. Education levels exhibit the most severe bias (S14 Fig in S1 Appendix). The INES over-samples individuals with no formal education, but otherwise displays a reasonable distribution of education levels. The two convenience samples are both made up largely of graduate-educated individuals: 92% of our Mechanical Turks have some graduate education, in contrast with a striking 95% of our Facebook sample and a mere 8.9% of the population at large.

Regarding voting behaviour in the last election, the 2014 INES best approximates the vote, though again significantly over-reporting turnout levels (S6 Fig in S1 Appendix). Individuals from the Facebook group are significantly more likely to have voted for the NDA in 2014, and significantly less likely to have voted for the UPA. Mechanical Turks are somewhat more likely to have voted NDA in the last election, though they are also more likely to have voted for the UPA—and significantly less likely to have voter for a third alliance. Turnout behaviour is exceedingly unrepresentative for both convenience samples, with 97% of all respondents saying they turned out to vote, compared to 66% of eligible voters in 2014, reflecting both a higher degree of political engagement due to their demographics, but almost certainly a high degree of over-reporting.

In summary, our two, non-representative, convenience samples exhibit similar biases: these samples are more male, younger, higher-status, highly educated, and highly politically engaged. This is consistent with most online convenience samples of this nature that require online literacy and English language proficiency. There are mode-specific selection effects. The work-environment provided by the Mechanical Turk platform results in participants who are more diverse in their income; less diverse in their age-cohort; less diverse geographically. This reflects to some extent an uneven penetration of desktop internet access. The leisure and connectivity provided by Facebook results in more geographically diverse participants from multiple age-cohorts and more balanced gender representation. These Indian Facebook sample characteristics are similar to those documented in other Facebook recruited samples from developing countries [47]. And overall, our findings are consistent with previous Indian studies employing online sampling [55–58].

5.1 Multivariate outcomes

The parametric density Pr_i(V∣T, X) is multivariate in nature—that is, we are interested in predicting the vote-choice probabilities for individual i with respect to each option j = {NDA, UPA, Other}. To estimate these quantities we rely on the one v. all [59, 60] strategy for predicting choice-probabilities for each category. This involves normalization of the resulting scores into probabilities via either standard normalization, or a more complex multivariate link function such as softmax [61]. In our Lok Sabha study, given we are dealing with a single layer of predictions for three alliances with similar levels of popularity, we have no reason to think employing softmax normalization will provide any advantage over standard normalization. Softmax is especially relevant if we encounter corner-probabilities, as it will tend to add weight to probabilities close to 1, and down-weight probabilities close to 0, relative to the other components one is normalizing over. The context where it is most useful is in multi-layer neural networks for classification, where it works by increasing the probability of extracting strong and definitive signals over noise. Hence we resolve to normalize the predicted mean of the vote-choice densities as shown in the Eq (9).

5.2 Choosing a learner

We employ Random Forests [8, 62] to estimate Pr_i(V | T, X) and Pr_i(T | X). RFs are a bagging algorithm (also known as bootstrap aggregation) combined with an ensemble method [12]. The bagging algorithm iteratively takes a large number of bootstrap samples of the training records and features in order to fit, ideally, uncorrelated CARTs. The ensemble feature refers to its predictions being a simple average of the bootstrapped CARTs. RFs are optimized for speed, especially in the ranger implementation [9], as they can rely on parallel processing and other computational features to generate trees of optimal depth. RFs are amongst the most consistent performers in terms of their ability to produce high-quality predictions in various competitions. Some researchers claim ‘the random forest is clearly the best family of classifier’ [63], although this is contested [64].

Eq (10) generates our RF predictions. The RF predictor is the mean behavior of a given cell i; μ_i, is the the average of B regression trees, each fit to a random sample of training data sets T(y_o, x_o), ∀o = 1, …, O, where o indexes the respective training observations, according to a set of random parameters Θ = {θ₁, …, θ_B}: (10) where ϕ_i,b stands for the prediction of component tree b for cell i.

As we pointed out earlier, we can also estimate Pr_i(V | T, X) and Pr_i(T | X) using multilevel regression [65]. Conventional MrP has performed well for many election forecasts. In particular its strengths can be condensed to the following: its ability to regularize predictions (“borrow strength” from aggregate-level data) [66] and to incorporate level-specific heterogeneity in the modeling; its parametric nature, which makes multilevel regression highly interpretable, when compared to non-parametric black-box models; its ability—especially if estimated under a Bayesian framework—to incorporate uncertainty in the predicted quantities and estimated parameters. As with any estimation strategy, it has its limitations: MrP predictions can be hugely sensitive to model specification, potentially hindering its performance in cases where not much is known about an underlying functional form; attenuation bias [45] due to over-regularization can reduce the accuracy of small-area estimates by shrinking these toward the global mean; multilevel regression models do not scale—under current computational constraints—beyond moderate sample sizes. Monte-Carlo estimation of Bayesian Multinomial models does not support large N—current state-of-the-art applications for the UK take 12 hours, for around 100, 000 responses. The MrP literature has began to address these challenges by moving towards more semi-parametric modeling: Sparse regression [67] is a proposed solution that enforces automated variable-selection. Variational inference [68, 69] improves scalability but at the cost of typically biased uncertainty estimates. Regression and post-stratification practitioners have also turned to machine learning: Bayesian Additive Regression Trees (BART) [70] have been shown to out-perform regular MrP in some applications. Random Forests [7] have been used to deal with high-dimensional settings. Stacking [71] of several non-linear learners shows promise, though it too suffers from computational speed issues.

We also recognize that the advantages of the RF estimation we employ here are sensitive to the data and modeling goals. Our Random Forest estimator deploys automatic variable selection, adaptive shrinkage and non-linear modeling. These can enhance modeling performance when sample sizes are relatively large, which is the case here since we have approximately 7, 000 observations in our vote-choice model, and over 14, 000 in our turnout model. With smaller public opinion samples, non-linear effects are limited and additive functional forms are well-understood, limiting any gains from machine learning. The principal objective of this modeling exercise is correctly predicting national election outcomes. Calibrating the effect size of covariates is not a major concern. Machine learning methods are not ideal for modeling exercises in which the interpretation of effect sizes is paramount. Some learners—such as Classification and Regression Trees (CART) [72]—retain intuitive interpretation. The literature has developed measures of importance and significance of the input features [73, 74] for more complex algorithms but they typically are not intuitive.

The prediction uncertainty associated with RF modeling has only recently received serious attention. Cross-validation [75] is the industry-standard for obtaining uncertainty estimates around machine-learning predictions. This assumes that the training sample provides a deep and detailed picture of the population of interest. This might be a plausible assumption under big-data conditions but it quickly breaks down for relatively small convenience samples. Recent advances in the statistical understanding of bagging estimators are a promising direction for improving assessments of prediction uncertainty [76–78].

5.3 Hyper-parameter tuning

RFs are ensembles of CARTs, and as such are governed by numerous hyper-parameters which describe the tree-growing mechanism. Unless otherwise specified, we use the same set of hyper-parameters for both our turnout and vote-choice predictions. We set num.trees = 1500, hence growing a large-enough number of trees to ensure no Monte-Carlo effects are present in our estimates. We set the number of candidate features to split, mtry = p, where p is the total number of available features, hence requiring that all features be considered for splits in each tree. This effectively transforms our RF to an ensemble of bagged CARTs. We do this in order minimize the shrinkage effect of the forest. Building on [11] we prioritize a model that allows for cross-level interactions to emerge in each tree. If these interactions explain residual variance and if trees are precluded from considering these interactions because a subset of the relevant variables is randomly out-of-bag, each tree will be a weaker predictor, and the ensemble will be weak as a result. Furthermore, the recent literature on growing probability-forests [79] explicitly suggests a large mtry—and in particular mtry = p outperforms other tuning choices in some key UCI machine-learning repository datasets [80]. In principle, we could have set mtry to any arbitrarily large number; however we found that bootstrapped trees, which are the conceptual equivalent, to be more attractive, as they represent a distinct, well-defined modeling strategy in the literature [12, 62]. On the other hand, having each tree consider the same covariate space may increase the correlation across trees, and decrease the ensemble predictive gains [12]; as a result, we reduce the size of each bootstrapped sample, and impose sampling without replacement, replace = FALSE, such that each record in each bootstrap sample is unique, and the samples are small-enough that they are extremely likely to be highly heterogeneous amongst themselves. Note that this will also help reduce fitting time. No limit is explicitly imposed on the depth of each tree, max.depth = NULL. The tuning of min.node.size is dependent on the complexity of the dataset at hand: for a relatively well balanced dataset, the default value for a probability machine is assumed to be min.node.size = 0.1×N, where N is the size of the training data [81]. However, our own experimentation revealed this value is inadequate for imbalanced datasets, as predicted values are excessively regularized by the dominance of the most likely outcome in the terminal nodes. Hence, to obtain meaningful differences in cell-level predictions, we set min.node.size = 30 for the turnout model, and respect Malley et al.’s advice for the vote-choice model. The choice of the value 30 is arbitrary, but driven by the need to have large-enough terminal nodes such that they don’t collapse entirely onto 1 or 0, but rather provide meaningful probability values, as well as the need to relax node-size-driven regularization to increase heterogeneity in predictions. S20 Fig in S1 Appendix shows a reasonable degree of heterogeneity for mean predicted turnout across cells in the stratification frame, though a clear upward bias persists.

5.4 Prediction error

Wager and Athey [76, 77] have shown that Random Forest predictions are asymptotically Gaussian in typical regression settings, and provide a consistent estimator of the asymptotic variance via the infinitesimal jackknife procedure [82]. Lu and Hardin [78] have instead developed a consistent estimator of the conditional prediction error for RFs, leveraging the OOB observations in each tree. The simple version of their procedure is highly intuitive: for every cell in the test-set, they go through each tree in the forest and record its out-of-bag neighbours—observations which were randomly not chosen to train the given tree, but were predicted to be in the same terminal-node as the cell-observation. They then calculate the error for each of these neighbours; since their true observed outcome is available it can be matched against the prediction for each tree. They then take a weighted average of this error based on the number of times each of these observations is an out-of-bag co-habitant of the test-cell. The resulting score is the conditional MSPE.

We should note here that, to our knowledge, no estimator for the prediction error of probability machines—i.e. models which output predictions in the range [0, 1]—has been thus far presented in the literature. The Lu & Hardin estimator does not impose a functional form over the distribution of the error term, hence it can be directly employed to parametrize non-gaussian distributions with [0, 1] support, which are also indexed by some measure of MSPE. The Beta distribution is such a density, as it can be fully parametrized by its mean and variance, as suggested in Eqs 7 and 8 earlier in this section.

It follows from the definition of the Beta distribution that: i) and ; ii) and . In our application, the first set of conditions regarding the mean holds for the entirety of our RF predictions; the second set—the conditions relating to the variance parameter—are more problematic. S16 Fig in S1 Appendix illustrates the effect the truncation has on the distribution of mean turnout and mean vote-choice across cells. The red dots in the plot indicate predictions whose MSPE exceed the beta-imposed limit. We see that these out of limit predictions are a small proportions of the total cells, for each predicted quantity. The limit imposed by the distribution is typically close to the estimated MSPE and is only really a problem for means closer to the corners of the distribution; hence we do not expect the correction to the limit to have significant consequences on our estimates. Having assumed this functional form, we can simulate D = 500 draws from our probabilistic model and obtain credible uncertainty estimates around our cell-level behavioural predictions. These can be aggregated by each simulation draw to preserve uncertainty at the national-level.

The MSPE estimator is implemented with the forestError package for R which can be computationally expensive. To limit computational costs, we randomly sample 1% of the cells in our stratification frame—an arbitrary amount large-enough to contain a reasonable amount of heterogeneity, but small enough to limit the computational burden. We then run the forestError algorithm on this smaller data set in order to derive their conditional MSPE. These predicted errors are used as training data to estimate the error of the remaining cells via RF regression implemented by ranger using the default hyper-parameters. This provides an approximate measure of uncertainty for each cell in the stratification frame. Note that our practical problem is to obtain reasonable confidence bounds around national-level estimates of vote-share, and as a result we can afford to be imprecise at the cell-level, as we expect errors to cancel-out by aggregating simulations at the national level.

6.1 National behaviour

Fig 2 shows the proportion of individuals who would turn out to vote over the course of the voting period. These predictions are based on the NES 2014 training data, after re-weighting to control for observable selection, and assuming stable turnout dynamics over demographics. Our point estimate is 78.9% against an observed turnout of 67.5%—a difference of 11.4%. This is in line with upward turnout bias observed in other countries in high-quality surveys [51]. The three standard deviation prediction uncertainty over the mean is just over ±1, which is rather narrow.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Predicted distribution of turnout at the national level.

https://doi.org/10.1371/journal.pone.0260092.g002

Fig 3 shows the national vote share predictions over the course of the campaign. Once voting began, we stopped collecting survey data and our predictions are simply projected forward according to a random walk. As the five-week voting period began, our predicted point estimate for the NDA is 41.8%, against an observed value of 45.0%—contributing to a prediction error of 3.2% points. The point estimate prediction for the UPA is just under 30.8% against an observed vote share of just under 31.3%, contributing to a prediction error of around 0.5%. Finally, the remaining error on the ‘Other’ alliances is just under a 3.7% over-estimation. On average, we have a prediction error of 2.5%, which is well within the range of standard polling error using traditional polling methods [84]. With respect to uncertainty, again we see a three standard deviation interval of a very narrow ±1. Projecting our estimates to the last day of the election, according to a random walk, the 3 s.d. prediction intervals for NDA, UPA, and ‘Other’ alliances expand respectively to ±5.1, ±5.3 and ±7.4 points. The uncertainty over the results on the last day of the election, obtained by projecting the last estimate of vote-choice forward according to a random walk, ends up being quite large, which reflects the fact that voting behavior over the course of the monitored weeks was rather variable from week to week.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. National vote share for the three major alliances over the course of the campaign; monitoring stops before the beginning of voting, and vote-share after this point.

https://doi.org/10.1371/journal.pone.0260092.g003

It is important to understand whether the positive performance outlined above is attributable to the quality of the online pre-election polling, or whether it is the result of our modeling strategy. S4 Table in S1 Appendix presents the national-level predictions at 6-weeks to the final election day, derived from the un-modeled, raw online sample. It is clear that modeling adds value: the national NDA vote share predicted from the raw sample amounts to 50.8% (error = −5.8%), a more severe bias, though in the opposite direction, compared to the modeled prediction; the UPA raw vote share was rather reasonable, at 32.4% (error = −1.1%), though the ‘Other’ alliances are severely under-valued, at 16.8%(error = + 6.8) points. This analysis shows that modeling and post-stratification is worth around 2 percentage points in error reduction per alliance on average. The value of modeling will become even clearer in the seats predictions phase, as the large over-estimation of the raw NDA share would lead to a collapse of the UPA according to a uniform swing prediction.

It is further worth highlighting the role that turnout modeling plays in error reduction—put plainly, to what extent is our prediction more accurate as a result of modeling conditional on turnout, and to what extent is the turnout modeling architecture justified? These questions are answered in S21 Fig in S1 Appendix, where the national projections from a model which completely ignores turnout are presented. A model which ignores turnout weights and relies only on vote-choice of online respondents to monitor public opinion would have an average error per alliance of 3%, about 0.5% higher per alliance on average. The turnout modeling seems to affect the alliance levels of support, and works specifically by reducing the support for ‘Other’ alliances, and increasing the support for the UPA; not much of a change can be reported relative to the NDA.

Another element of our modeling effort which is worth highlighting is the ability to capture real-time shifts in public support due to salient campaign events. We focus on the spike and subsequent stabilization of support in favour of the NDA from Fig 3, between the beginning of measurement and 9 weeks to the final day of the election. This spike corresponds to a military skirmish that occurred in February and March between India and Pakistan, which saw the NDA and its leader Narendra Modi styling themselves as ‘Watchmen’, highlighting the NDA’s focus on national security [85]. The ramifications of the border skirmishes with Pakistan on Indian public opinion are well captured in our public-opinion estimates, and amounted to a pro NDA shift worth just under 2.3% at the height of the tensions. Repeated low-cost surveying of online convenience samples, and subsequent modeling adjustments, enable the estimation of such real-time preference-shifts.

6.2 Feature importance

There are hypothesis tests that assess the contribution that different independent variables make to RF predictions [73]. We implement a permutation variable importance metric (VIM) [86, 87] that indicates whether our priors (such as the correlation between age and turnout) are being learned and leveraged in some way by the algorithm. We also use VIM to rank variables according to their predictive contribution.

The VIM score is calculated by measuring the change in average OOB predictive accuracy when each feature is rendered more stochastic via random permutation. Larger changes in predictive accuracy provide an indication of the relative importance of a predictor variable. Non-parametric hypothesis testing can then be performed by first simulating a distribution of the predictors’ VIM under the null hypothesis that the predictor is ‘unimportant’. The null distribution is obtained by randomly permuting the response variable P times, and recording the associated ‘null importance’ of the features for each permutation. We can then simply calculate the percentage of null-scores that are greater than the observed VIM. When strong predictors are correlated, such that there can be clusters of highly predictive correlated features, the Altmann permutation VIM [86] distributes importance across correlated features, and assigns significant p-values to all of the predictive features in the group. We should not interpret the effects independently but rather report the total effect with their group-cohabitants. S17 Fig in S1 Appendix presents the most significant feature importance scores for each of our behavioural models.

Beginning with our turnout model, we can see the random forest picks up a large number of significant important variables—though a large number of them have seemingly little to no impact on the prediction if taken independently. This is in part a direct result of the low min.node.size we specified, which encouraged the forest to explore a large number of statistically meaningful—but small in magnitude—deep interactions. A few variables stand-out as being responsible for a meaningful magnitude of error reduction: individual-level alliance of choice is the most impactful variable for making our turnout predictions, followed by religious affiliation, area-level age distribution, caste, income and education level. The model therefore points to three broad classes of determinants of turnout: political preferences; age distribution and class.

With respect to 2019 vote choice, 2014 vote is the most predictive variable across all alliances. Caste is also a significant determinant of vote-choice. Religious affiliation plays a key role in the UPA vote, perhaps indicating the relatively strong Muslim affiliation with the alliance. State-level caste-composition and age profiles are also significant. Many distance variables, that measure geographic proximity, are significant across alliances. This indicates there are cross-state correlations and geographical patterns in the vote resulting in a degree of local smoothing. Weeks to the end of the election play a significant role for all alliances which suggests significant campaign effects. There are more significant predictors of the vote for the NDA than for other alliances. The ‘Other’ group is the least predictable (low VIM magnitude per variable and little significance across these), perhaps a sign of the heterogeneity of the ‘Other’ category which contains many diverse regional and national parties.

6.3 Sub-national behaviour

Typical MrP applications have as primary objective the estimation of behavioural outcomes at the sub-national level [65]. Our application is different, in that we use regression and post-stratification simply as a way to produce more robust national-level forecasts from non-representative surveys. Nevertheless, we explore the degree to which relatively small and unrepresentative surveys can be smoothed to predict sub-national behaviour in this highly heterogeneous context. We include two geographical levels of interest: zones and states; there are 6 zones and 35 states in our application. Though Telangana formally seceded from Andhra Pradesh in 2014, we ignore this split and still consider these two states as one. This allows us to avoid having to adjust census and IHDS population estimates for a state which at the time was not recorded. Fig 4 and S18 Fig in S1 Appendix compare our predictions at the sub-national level with the observed election results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Zone-level predictions v. observed zone-level 2019 behaviour for Turnout share (top-left), and Alliance vote-share (NDA top-right, UPA bottom-left, Other bottom-right).

https://doi.org/10.1371/journal.pone.0260092.g004

Overall, sub-national accuracy is poor. We expected this to be the case, especially given the highly skewed and unrepresentative geographic distributions of our survey data. Zone-level predictions outperform state-level ones significantly, showing reasonably high cross-zone correlations for the two major alliances. In particular, state-level forecasts for the NDA vote are correlated with the observed outcomes at 0.94, suggesting a reasonably strong sub-national signal was present in the pro-NDA survey response. The poor performance of our predictions at the sub-national levels suggests a lack of area-level statistical power. In circumstances where there is high cross-area correlations, as is often the case in US or UK, sub-national estimations, area-specific statistical power is less of an issue [1] than it is in India, where cross-state heterogeneity in political behaviour can be dramatic.

6.4 Seat projections

Our forecasting method allows us to ‘call’ the election in terms of seats. We assign a predicted number of seats to each alliance and measure the probability of obtaining a majority in the Lok-Sabha. Our national-level vote-share estimates are converted into seats by applying a “uniform swing” [83, 88]. In this model every constituency ‘swings’ with the national vote, such that a national change from the last election in the vote-share of each party will be reflected in an identical change at the constituency level. More formally, we can estimate the 2019 constituency-level vote share of a given party as follows: (11) (12) where δ_j,t indicates the change in vote share for a given party, for an election held at time t; λ_j,t,c is the vote share for party j at time t in constituency c, for c = 1, …, C.

Under this model, a specific constituency may be poorly predicted, but the headline share of seats tends to be accurate. This results because the true constituency swings tend to be realizations of a Normal distribution with the national-level swing as its mean. At the national level, the uniform swing model tends to work quite well in the US and the UK due to the relatively uniformity of the political climate (meaning political discourse is common across constituencies); the relative homogeneity of constituencies in terms of important socio-economic variables such as class and income; and the relative stability of the voting choices. While India is a far more heterogeneous country, there is evidence that a uniform swing may apply [83, 89]. To further support our use of uniform swing in the Indian context, we provide an analysis of historical swings from 1989 (the first year election where the BJP/NDA was a serious contender) in S19 Fig in S1 Appendix. Note this analysis assumes the 2019 alliance-party composition, and is possible that under different formulations of these three alliances we may observe patterns of swing which are less compatible with the uniform hypothesis. Nevertheless, from our analysis it is clear that constituency swings appear to be symmetrically distributed around the national swing, suggesting that on average the uniform swing hypothesis holds in India.

Table 3 compares our estimates of the election result to state-of-the-art Indian election polling. In India, the most common opinion polls published on a relatively regular basis, before the beginning of the election period, are seats-projection polls. We obtain publicly available opinion polls for the 2019 election from Wikipedia (https://en.wikipedia.org/wiki/Opinion_polling_for_the_2019_Indian_general_election). For each poll reported on the Wikipedia page, we try to distinguish between the polling company and the news organization that commissioned the poll. Where no details on the polling company are available, we assume the news organization did the polling in-house. Pre-election polling stopped at around 40 days before voting begun; hence we compare our estimates at 5 weeks until the end of the election, with the last published poll of each of 10 polling houses that completed their last published poll at least a year before the last day of voting in 2019. Amongst the polling houses, we can identify three broad groups: 6 of the houses in question, namely ‘CNX’, ‘Jan Ki Baat’, ‘VMR’, ‘News Nation’, ‘CVoter’ and ‘Zee 24 Taas’ polled the campaign continuously, and published their last poll in the final week before voting begun—within 50 days to the end of the election. 3 pollsters, namely ‘VDP Associates’, ‘Karvy’ and ‘India Tody’, polled less regularly and published their last projections between 140 and 80 days to the end of the election. Finally, ‘CSDS’ and ‘ABP News’ released their last projection between a full year and 200 days to the end of the election. Fig 5 presents our seats projections and Table 3 describes how our error rates compare to other pollsters.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Seats projections at 5 weeks till the end of voting.

Pr(majority) indicates the probability that a given party obtains an outright majority—272 seats or more—in the Lok Sabha.

https://doi.org/10.1371/journal.pone.0260092.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Seat projections and errors across pollsters.

Performance of two baseline predictions (the past-election results and the uniform-swing projection from the raw, non-modeled online data). Underlined are the best performing pollsters for each column. In some case we present two bold quantities per column because the best performing was so outdated—we highlight a second-best performing poll that was more recent and comparable.

https://doi.org/10.1371/journal.pone.0260092.t003

Firstly, it is immediately clear that our uniform-swing projections outperforms other pollsters: our poll has the lowest absolute error, 89 seats—together with the Jan Ki Baat poll; a gulf of 54 stands between the top two performing polls and the next best. Importantly, our methodology vastly outperforms other pollsters on the NDA-UPA lead, missing the mark by a mere 8 seats, when compared to the next-best-performing Jan Ki Baat poll, which misses the mark by 67 seats. This is the most important predicted outcome because it typically directly determines government formation. At the alliance level, our method is the best performing on both NDA and UPA vote shares, with Jan Ki Baat coming in second for the NDA vote, and CNX for the UPA vote. Jan Ki Baat misses the NDA share by 44 seats, compared to our 27 seats error. The CNX forecast of the UPA seat share is off by 27 seats, compared to ours which is off by 18. Our seat forecast for the ‘Other’ category performs less well. As a final point, we also benchmark our uniform-swing predictions from the modeled data (regressed and post-stratified) against uniform swing projections based on the raw, un-modeled proportions form the online convenience samples. In every comparable prediction, our modeled forecasts outperform the raw un-modeled forecast. This further justifies the use of our complex modeling architecture to clearly add value to the non-representative sample at our disposal.

Fig 5 presents our seats projections incorporating their uncertainty intervals. The vertical red bar identifies the actual seat outcomes for the three categories. Finally, the uncertainty around the seats predictions is very narrow: the NDA forecast ranges 21 seats, between 321 and 342; the UPA forecast ranges 13 seats, between 73 and 86; and the ‘Other’ forecast ranges 15 seats, between 125 and 140. An indication of the over-confidence of the forecast is that even if our poll is the best performing, none of our intervals include the observed results.