Survey

We conducted a survey through the RAND American Life Panel, which is an online panel of adults ages 18 and older living in the United States. The ALP uses a probability-based sampling method of the US population that includes individuals regardless of whether they are online or not. ALP provides panelists who are not already online by providing home internet access to individuals without regular internet access who agree to participate in the panel. We invited 716 members of the panel with children under the age of 21 years to take the survey, and we received 615 responses (overall completion rate 85%). We asked about knowledge and beliefs on 35 items, which included: 8 political conspiracies, health conspiracies, 9 putative vaccine side effects, 2 gestalt vaccine endorsements, and 6 items about vaccine biology/epidemiology.

All belief survey responses were on a Likert scale from 0 to 6 where “0” indicated “Strongly disagree is true”, 3 indicated “Undecided or not sure”, and 6 indicated “Strongly agree is true. We include the full list of survey items in S1 Table.

Twitter data

We pulled tweets that were posted in 2017 about topics corresponding with the 35 belief items from our prior survey, because this was the most recent complete year as of summer 2018, when we pulled the data. We conducted our data pulls using Gnip, a major vendor of Twitter data that enables historical data pulls with keyword and other criteria for inclusion. The first step in acquiring the Twitter data for our analysis was to develop a set of 35 Twitter search queries (hereafter “tags”) corresponding to the 35 belief items from our prior survey. After developing an initial set of 35 queries, we pulled a preview of tweets posted on a single day that matched our queries. There was wide variation in the number of tweets we received on each query term; some queries resulted in no matching tweets while a “deep state” query resulted in over 18,000 tweets in a one-day preview. We used an iterative refinement process in which we reviewed up to 20 tweets returned in the preview for each tag. If we observed false positives in the samples we reviewed, we refined our query terms for that tag to be more specific. False positives were tweets clearly about a different topic than the belief we intended, for example, tweets about household cleaning products when we were searching for chemtrail conspiracy beliefs. For tags with no or very low single-day volume (<10/day), we broadened our query terms. We additionally included relevant hashtags identified in our reviews of tweets. For example, we included #vaccineswork as a term in our queries about vaccine endorsement. At this phase, our queries were intended to identify only tweets about a topic, but not identify the tweet’s stance (agreement or disagreement with the belief). For example, tweets about the false belief that Barack Obama was not born in the United States are at times sent both by those endorsing this belief and by those critiquing it, but all such tweets would be about “birtherism” and would be picked up by the birtherism search query tag.

There were 5 belief items for which we could not identify a set of query terms that were specific enough to give us a low false-positive rate, but general enough to result in any hits. These were “CIA Created HIV”, “Vaccine Spread HIV Africa”, “CIA Bin Laden Vaccination”, “CDC Hides Side Effects”, and “Vaccine Profit Motive”. The full list of belief items from the survey are shown in S1 Table. In addition, we did not have a query for the belief that the HPV vaccine can cause sterility (“HPV Vacc Causes Sterility”); but, we had a query for the more general belief that vaccines can cause sterility (“Vaccines Created to Sterilize”). These were separate beliefs in our survey, but were covered by the same terms in our Twitter query. As a result, we had 29, rather than 35, belief items in our Twitter query. The final query terms used to pull the data are listed in S1 Table. Tweets that matched a specific query were given a “belief tag” describing the corresponding belief. In some cases, tweets were tagged with multiple beliefs.

The Twitter query returned a total of 8,829,987 tweets generated by 1,350,451 unique users. We removed accounts that tweeted once during the sampling time frame as a low threshold to restrict our analysis to engaged accounts. This resulted in 8,064,173 tweets from 584,636 users in the sample.

Next, we used the Bot-o-meter API [12, 13] to identify accounts that were likely to be bots. The Bot-o-meter is a machine learning algorithm trained to predict the likelihood that a user is a bot based on the user’s profile, friends, social network structure, temporal activity patterns, language, and sentiment. The models also calculate a Complete Automated Probability (CAP)—the probability of an account being completely automated, by taking into account the estimated overall prevalence of bots [12, 13]. We used a CAP threshold of 0.5 to remove tweets from our data set that the tool determined to have at least a 50 percent likelihood of being a bot. The final de-botted file had 7,578,547 tweets from 551,738 unique accounts. We note that because we pulled historical data from Gnip, our data did not contain tweets from suspended accounts [14], including Russian bots and trolls that were found to amplify the vaccine debate [15]. Recognizing that bot detection is imperfect, we conducted a sensitivity analysis in which we used the stricter criteria of only including accounts with a less than 20 percent likelihood of being a bot (CAP<0.2).

File pairs

We created four pairs of data files; one member of each data file pair was derived from our survey data, and one was derived from the Twitter data. These file pairs enabled us to examine how different data processing steps affected whether similar conclusions about belief covariation could be drawn from Twitter and survey data. We first will give an overview of each data file pair including its analytic purpose and file contents. Following this overview, we present a more detailed description of how each file in the pair was created from the original survey and Twitter data files. S1 Fig in the supplementary material presents a flow diagram depicting the relationships among the different analytic files that we created.

File pair overview

1. Topic data file pair. The purpose of this data file pair was to examine the agreement between Twitter and the survey data with only minimal processing of the Twitter data. For this pair, the Twitter data were processed to be a 1 if the account tweeted about a topic and otherwise 0. To make a conceptually comparable transform of the survey data, for this pair, the survey data were processed to be a 1 if respondents selected the most extreme boxes on either the agree or disagree end of the scale. All other selections were set to 0. We also conducted a sensitivity analysis in which any non-neutral survey response was coded to be 1 and only the neutral response (3 on the 0 to 6 Likert scale) was coded as 0.
2. Stance data file pair. The purpose of this data file pair was to examine whether inferring tweet stance (i.e. agreement or disagreement with the topic tweeted about) improved the concordance between Twitter and survey data. We inferred stance on 10 belief tags by manually coding websites commonly linked to by the five most commonly tweeted about vaccine beliefs and the five most commonly tweeted about political or conspiracy beliefs. Data elements in these files could take on values 0 (neutral stance), -1 (negative), or +1 (positive). For the Stance data file pair, the survey data were processed such that individuals who selected the most extreme disagree box were -1, the most extreme agree box were 1, and all others were 0. We also conducted a sensitivity analysis in which any agree response to the survey was coded as +1 and any disagree response was coded as -1.
3. Limited topic file pair. The purpose of this data file pair was to serve as a robustness check on any conclusions that we could draw from a comparison of the analysis of the Topic and Stance data file pairs. The Topic and Stance data file pairs differ in three ways. First, the Stance data file has information about whether respondents/tweeters agreed or disagreed with the topic. The Topic file pair does not have agree/disagree information. Second, the Stance data file pair contains fewer belief items than the Topic data file pair because we inferred stance for only 10 of 29 belief items. Third, the Stance data file pair contains fewer individuals since we dropped survey respondents and tweeters for whom the inferred stance on all 10 belief items was zero. The limited Topic file pair contains only the belief items and observations that are included in the stance file so that the only difference between the Limited Topic and Stance file pairs is the inference of stance.
4. Resampled stance file pair. We created a file pair in which Twitter data with stance inferred, was re-sampled so that the distribution of the number of topics about which individuals expressed strong beliefs was similar to that found in the survey data. The survey data for this file pair are identical to those described for the Stance file pair.

Table 1 shows an overview of the four data file sets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Description of data file Pairs.

https://doi.org/10.1371/journal.pone.0239826.t001

Resampled stance Twitter file.

We re-sampled the Twitter Stance data file so that the distribution of the number of topics individuals tweeted about was the same as the distribution of the number of topics an individual expressed a strong belief about (i.e., belief value was +1 or -1) from the Stance survey file. Table 2 shows the distribution of the number of non-zero belief tags in the Twitter and Survey stance files before resampling. Note the large number of single-stance Twitter accounts relative to single-stance survey respondents. Resampling was designed to correct for this. When we re-sampled, each row (account) in the Twitter stance file was categorized as tweeting about 1, 2, 3, 4 or 5+ topics. To create the re-sampled Twitter stance file, we randomly selected 68 rows (the number of single-topic rows from the survey stance file) from the twitter stance file that tweeted about 1 topic to include in the re-sampled file, 67 that tweeted about 2 topics, and so on.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Distribution of non-zero belief tags in Twitter and survey stance files (before resampling).

https://doi.org/10.1371/journal.pone.0239826.t002

Sensitivity analyses

We conducted three sensitivity analyses whose results appear in S5 and S6 Tables. The first was an alternate survey coding. In the main results, we code only the “Strongly agree is true” (6 on a Likert scale) and “Strongly disagree is true” (0 on a Likert scale) survey responses as non-neutral. In the alternate survey coding, we code all agree statements (4, 5, or 6 on the Likert scale) and all disagree statements (0, 1, or 2 on the Likert scale) as non-neutral. The second sensitivity analysis was a strict bot removal in which we kept only accounts with a Complete Automated Probability (CAP) <0.2 instead of CAP <0.5, which corresponds to keeping accounts with less than a 20% probability of being a bot. In the third analysis, we re-sampled the Twitter data 100 times for the resampled stance data file pair.

Principal Component Analysis (PCA)

We ran principal component analysis (PCA) on our data in R using the prcomp function from the stats package [18]. For all of our analyses, we ran the PCA on centered, scaled data. We tested the similarity between the first principal components in two ways. First, we compared the PC1 loadings. We examined the qualitative agreement in the Twitter and survey PC1 loadings by comparing which items loaded in the same direction (positive or negative) in the two PC1s. We also calculated the correlation between the PC1 loadings estimated using the survey (D_S) and Twitter data (D_T), which is a method that has been used to compare factor loadings [19]. Comparing the loadings gives insight into how comparable two PCAs are in terms of covariation among the belief items in the data set. We also used a method described in Jolliffe 2003 [20], which compares PC scores estimated from two different PCAs. This method gives insight into how comparable the two PCAs are in describing variation among the observations in the data.

To compare the PC1 scores derived from the Twitter and survey data, we estimated the PCA transformation using both the survey data—we call this projection P_S—and the Twitter data. The Twitter projection is P_T. These PCA transformations are characterized by the loadings of each of the original variables on the first principal component from either the survey or Twitter data. We then projected both the Twitter data and the survey data into the new PCA coordinate basis using both P_S and P_T. In other words, we used the loadings from PC1 as derived from the survey data to calculate predicted PC1 scores for the Twitter data, and likewise we calculated PC1 scores for survey data from the PC1 loadings estimated from the Twitter data. Mathematically, we calculated (P_SD_S)_*,1; (P_TD_S)_*,1; (P_TD_T)_*,1; and (P_SD_T)_*,1. The subscript “*,1” indicates the first column of a resulting matrix. We then used linear regression to estimate the models and where β is a coefficient estimated in the model and c is a constant estimated in the model. The R squared in the estimated models provides a measure for the similarities between the first principal components of the projections P_S and P_T [20]. It is the proportion of variance from one set of PC1 scores that explains the variance in the other.

Inter-rater reliability and stance validation

We inferred tweet stance by manually coding the stance of common website domains linked to in tweets. The stance of a tweet containing a weblink to a coded domain was assigned the stance of domain. To validate this approach, two authors of this paper, SAN and LJM, independently coded the stances of 20 tweets from each topic included in the stance data files. The average interrater reliability for tweet coding across the ten tags was 86% agreement, Cohen’s kappa was 0.55. Tag-level results are shown in S3 Table. For two tags, all tweets were coded with the same stance and the Cohen’s kappa was therefore undefined. These tags were included in the calculation of the average percent agreement, but excluded from the calculation of the mean Cohen’s kappa. For each coder, we compared the stances assigned directly to the tweets to the stances inferred using weblinks. The average agreement between coded and inferred tweet stance was 67% across all tags, and average Cohen’s kappa was 0.45. Tag-level results are shown in S4 Table. When we inferred stance in the Twitter data, we found that of the 7,578,547 tweets in the de-botted file, 3,539,561 (47%) contained a url, and 432,011 out of 551,738 (78%) of accounts had a least one tweet with a url. When we restricted the data set to tweets tagged with the 10 belief items for which we inferred stance and tweets for which we inferred a non-neutral stance, the resulting file had 909,479 tweets from 183,169 accounts.

Belief frequencies in the data file pairs

Fig 1 shows the percentages of individuals or accounts with different values in the Twitter and survey files in each of our file pairs. In the survey Topic data file (Fig 1A), the percentage of individuals expressing strong beliefs (black bars) fell between 25% and 75%; there were no items for which a very high or very low percentage of individuals expressed strong beliefs. In contrast, in the Twitter data (Fig 1B), we found that in our sample, which included individuals who tweeted at least twice about any of our belief items, most items were discussed by fewer than 25% of our sample. The “Deep State” belief tag was by far the most common in the Twitter data set and was tweeted about by approximately 75% of individuals. The low frequencies of most belief tags in our data set is largely a reflection of the fact that many accounts that tweet about “Deep State” do not tweet about any other belief tag.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Proportion of individuals or accounts with different belief tag values in the survey and Twitter data files for each of the four data file pairs.

In panels a, b, e, and f, black indicates presence (1) and yellow absence (0) of a belief. Within panels c, d, g, and h, blue indicates agreement (+1), yellow absence (0), and orange disagreement (-1) with a belief.

https://doi.org/10.1371/journal.pone.0239826.g001

When we inferred stance in the Twitter data (Fig 1D), we found that “Deep State” and “Vaccines Benefit Public” remained the most common beliefs. Over half of individuals in this data set expressed agreement with the “Deep State” belief tag (blue bar), while around 20% expressed disagreement (orange bar). In addition, around 25% of accounts in this data set express agreement with “Vaccines Benefit Public” and a very small percentage expressed disagreement. When we re-sampled the Twitter data so that the distribution of the numbers of items that individuals expressed beliefs about was similar to the distribution we observed in the survey (Fig 1H), we found that in addition to the “Deep State” belief tag, over half the sample expressed agreement with “MMR Autism” (a link between vaccines and autism) and “Chemtrails” (a belief that airplane contrails contain chemical or biological agents that are released into the atmosphere as part of secret government programs).

We now turn to the results of our PCA. We focus on the results for the first principal component because scree plots from the survey PCAs suggested that there was one dominant principal component. While the scree plot for the Twitter topic data file suggested that up to four principal components were important, after stance inference and resampling, the Twitter scree plot suggested a single dominant principal component. Scree plots for all analytic files used in the analysis are shown in S2–S5 Figs.

Comparison of first principle component loadings

Fig 2 shows the loadings of the first principal component from the PCA. All belief items loaded in the same direction on the first principal component in the Topic survey data (Fig 2A). This suggests that some people tend to use the extreme ends of the scale when responding to the survey, which would result in those individuals having most or all items coded as “1”, while others tend to answer towards the middle of the scale. When we conducted the principal component analysis with the comparable Twitter data set, we found that all items with the exception of the “Deep State” belief tag loaded in the same direction, suggesting that the first principal component captured the tendency of people to either tweet about the “Deep state,” or to tweet about other topics. The first principal component in the Twitter and survey Topic data sets appeared to describe qualitatively different phenomena. When we incorporated stance information (Fig 2B), we found that for both the survey and Twitter data, all items loaded in the same direction with the exception of the “Vaccines Benefit Public” belief. All other beliefs examined were conspiracy theories and unproven or disproven negative beliefs about vaccines. In this case, the results of the principal component analysis using both the survey and Twitter information had similar qualitative interpretations. The results of the Limited Topic data analysis (Fig 2C) were qualitatively similar to the results of the Topic data analysis; and, the results of the Resampled Stance analysis (Fig 2D) were qualitatively similar to the results of the Stance data set analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Loadings on PC1, the first principal component from the PCA analysis for each data file.

Values of ρ are the Pearson’s correlation coefficient for each pair of PC1 loadings.

https://doi.org/10.1371/journal.pone.0239826.g002

Comparison of first principle component scores

Fig 3 shows the results of the linear regressions that describe the association between the actual PC1 scores for each data set (survey or Twitter) and predicted PC1 scores calculated using PC1 loadings from the other data set (Twitter or survey). We found that the agreement between PC1 scores was highest when stance information was included in the analysis (Fig 3C, 3D, 3G and 3H) compared to when it was not (Fig 3A, 3B, 3E and 3F). In addition, we found that for the Topic, Stance, and Limited Topic data pairs (Fig 3A–3F) predictions for survey PC1 values using the Twitter PCA analysis were much better than predictions of Twitter PC1 values using the survey PCA analysis. However, when we re-sampled the Twitter stance data set so that the distribution of the numbers of items on which individuals expressed strong beliefs was more similar to the distribution from the survey data (Fig 3G and 3H), we found improvement–that Twitter PC1 values were now predicted very well by the survey PCA. This resampling procedure reduced the number of individuals who tweeted about only a small number of topics; recall from Table 2 that 90.7% of accounts in the Twitter stance file tweeted about only a single topic. We resampled the data so that 12.4% of observations in the resampled Twitter stance file tweeted about only one topic. The fact that predictions of PC1 scores for the Twitter data improved markedly after the resampling procedure suggests that it is more difficult to predict PC1 scores for individuals who tweet about a limited number of topics than those who tweet about several. This makes sense given that the PC1 value for an individual who tweets about a single topic is the PC1 loading for just that topic. Overall, we find very good qualitative and quantitative agreement for the first principal component of the PCA using the survey and Twitter data in the data sets that contain stance information and where the Twitter data is re-sampled to better match the “number of topics” distribution we observe in the survey data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Scatter plot and linear fit of predicted PC1 score and actual PC1 score.

The “actual PC1 score” was calculated using the loadings from the data set for which we were estimating the scores. I.e., actual scores for Twitter data were calculated using Twitter PC1 loadings and actual scores for survey data were calculated using survey PC1 loadings. The “predicted PC1 score” for Twitter data sets was calculated using survey PC1 loadings and the “predicted PC1 score” for survey data sets was calculated using Twitter PC1 loadings. For each plot, the predicted and actual PC1 scores were re-scaled to lie between -1 and +1, which did not alter the correlation.

https://doi.org/10.1371/journal.pone.0239826.g003