Community detection and labeling

To ensure that our data contain viable communities, these time window graphs were used to calculate the global clustering coefficient, a metric that quantifies the density of connections among triads in the graph as a value [0,1]. Social network graphs, or in our case, graphs that model genuine social interaction, tend to have higher clustering coefficients than randomly chosen graphs of the same size [36]. For each time window, the clustering coefficient of its nodes was calculated according to:

(1)

where T(u) is the number of directed triangles through node u, deg(u) is the combined number of in- and out-edges, and deg^rec is the number of reciprocal edge pairs containing u. The global clustering coefficient is then taken by averaging these values over the entire graph. We find that the average clustering coefficient is 2.9298E-2 for our Twitter data and 5.2375E-2 for our Parler data. Each time window is then compared to a random graph of similar size. We generated a random Erdős-Rényi graph with an identical number of nodes and edges, then repeated the clustering coefficient calculation [37,38]. An Erdős-Rényi graph is generated by specifying a number of nodes and of edges, then a graph is randomly chosen from the set of unique graphs with that amount of nodes and edges. We find that the average clustering coefficient is 1.1099E-5 for the random Twitter-sized graphs and 2.7381E-4 for the random Parler-sized graphs. This multiple order of magnitude improvement in clustering coefficient over the random graphs suggests that running community detection will lead to viable communities. We additionally compare our networks, which are comprised primarily of political discussion posts, to the non-political social networks from the Higgs-Twitter dataset [39]. The Higgs-Twitter retweet network has a clustering coefficient of 1.5600E-2, which is on the same order of magnitude as our Twitter and Parler data. This is an additional confirmation that our data retains the characteristics of a typical online social network.

At each time window, the Louvain algorithm was run to segment the data into clusters, a procedure which is both deterministic and efficient on large graphs. This is a greedy algorithm that assigns nodes to a cluster with high intra-cluster density by optimizing the modularity of each cluster, or the ratio of within-cluster edges compared to between-cluster edges, according to the following equation:

(2)

where m is the number of edges in the graph, A is the adjacency matrix, k_i is the degree of node i, γ is a parameter set to 1, and is a Kronecker delta function that returns 1 if nodes i and j are in the same community and 0 otherwise. We find that the average modularity is 0.7057 for our Twitter data and 0.7281 for our Parler data. These results demonstrate that our networks contain well-defined communities roughly in line with the modularity findings of social networks from the literature [40].

For each of the resulting communities, we calculate several cluster-level values that are used in the analyses of clusters. It is at this level of the Louvain cluster that we define the ideological homogeneity of groups in the network. In addition to this value, we calculate the average bias of the cluster’s articles, the number of intra-cluster and inter-cluster retweets or reposts generated by the cluster’s articles, and the size of the cluster. We also calculate the values of genre perplexity from the cluster’s articles, described below.

Ideological homogeneity metric

Previous research has shown that an individual’s media diet is typically diversified [41], but follow-up work in psychology argues that people give higher regard to media that they have actively interacted with [42]. Therefore, a retweet network captures these more highly regarded pieces of media, and our metric captures the diversity of these more salient sets of information. We are careful to refer to these retweeted articles using the ideologically neutral term highly regarded media as the use of retweets as a method to measure endorsement is still subject to debate in the literature. Previous works show that users often retweet articles for the purpose of debunking or arguing against its contents [43,44]. Whatever their intentions, a user repeatedly retweeting biased sources serves to amplify those sources, and the network effects are functionally similar [45,46].

To measure the extent to which users in our graph have balanced media interactions, we propose a novel metric for measuring the ideological homogeneity of a subset of users in a social network. The metric develops on the graph theoretic concept of the clustering coefficient and the information theoretic concept of entropy. We quantify ideological homogeneity as a function of the diversity of a user’s set of news sources, measured by calculating a variation of entropy on that source set. The value is then weighted based on the number of users in the cluster. Each of our clusters consist of edges that represent information flowing from a producer (the source node) to a consumer (the target node), and each edge has a bias label corresponding to the specific web domain it links to. Our metric measures the extent to which the total set of consumers in a graph have balanced media interactions. Our metric is also unique from other models, such as latent ideology detection, because our metric does not rely on subjective classification of text [47]. Current state of the art methods for latent ideology detection in text add significant complexity to data annotation. Such procedures are typically only accurate on domains where there is richly labeled training data, and those domains are often too specific to be useful generally [48,49].

For a group G(V,E), the consumer set is defined as the set of all target nodes for the edges of G. For a given consumer node, we define the label set L as the set of bias values for each article that consumer node has retweeted. Thus, for a group G with consumer set C, and label set L, the ideological homogeneity of G is calculated as:

(3)

(4)

where is the entropy of the set of bias labels for all edges into node . Additionally, the probability term corresponds to the fraction of the edges of node which have bias label l. As defined in Eq 3, this metric quantifies the extent to which G has ideologically homogeneous interactions as a value in [0,1] [50]. When each potential bias label is present in equal proportions in the label set, the entropy of the label set is maximized at 1. In this case, the term will resolve to 0, since a cluster that shares articles of all bias labels in equal proportions is not ideologically homogeneous. Conversely, if a cluster shares articles from only one bias label, the entropy of the label set is minimized at 0; therefore, the term resolves to 1 since this is an ideologically homogeneous cluster. The equation then weights that value based on the number of users in the cluster. I(C) will gradually decrease along the [0,1] range as more ideological diversity is added to users’ articles. For each time window, each Louvain cluster was assigned a ideological homogeneity value according to Eq 3. Previous work has identified that upwards of 20% of the US population is polarized, and follow-up work uses this upper quintile to examine membership in polarized bubbles online [51–53]. Therefore, for both Twitter and Parler, we define ideologically homogeneous clusters (IHCs) as the set of clusters in the top 20% with respect to this value. To further test the robustness of our results, we repeated the graph topology experiments with two additional cutoff values (15% and 25%). These results do not differ significantly from the 20% cutoff, so we continue using the 20% as our baseline IHC cutoff value. The full distribution of ideological homogeneity values and the IHC cutoff for both platforms are visualized in S3 Fig.

Article spread using retweets

After dividing our data into clusters and labeling the most homogeneous clusters as IHCs, we measure the extent to which clusters are able to spread their message throughout the network. To do so in a way that uses our available data, we measure the retweets garnered by articles posted by members of a cluster. Measuring impact this way requires careful interpretation because the retweets on an article do not necessarily correlate with overall acceptance of its contents. Regardless of the level of acceptance, a cluster’s ability to gather retweets from the wider network demonstrates its ability to set the tone of online discourse. We compare IHCs on both platforms to less homogeneous clusters in their ability to gather retweets.

To ensure that comparisons are made between similarly sized clusters, we group all clusters in the full dataset into 40 bins as a function of their size. 40 bins were chosen because our most frequent cluster size (3) makes it so that 40 is the largest number of bins that fits all clusters of a certain size into the same bin. The bins generated by this procedure are described in detail in S2 Table. After controlling for cluster size, we calculate the average number of retweets per user in IHCs and the retweets per user in all other clusters. The average retweet value R for a set of clusters is defined as

(5)

where i is an individual cluster, RTs(i) is the total retweets for posts in that cluster, and size(i) is the number of users in that cluster.

Genre perplexity

In addition to the graph metrics, each cluster is assigned a score to quantify the extent to which its articles use language indicative of certain literary genres. A trigram language model was trained using the SRI Language Modeling toolkit for each of 14 genres available in the BookCorpus dataset [54,55]. The list of genres is included as S2 Table. Each language model was trained on the BookCorpus data corresponding to the specific genre of writing, and the trained model is a probability distribution of trigrams contained in the genre writings [56]. We use an n-gram based language model rather than a neural one since we need the model to focus on lexical cues and local context, which we cannot guarantee in a neural model. Additionally, since we can train the n-gram model on solely on the BookCorpus, we avoid data leakage.

For each pair of article and genre, the genre language model is used to predict the text of the article. The predictions are reported as perplexity scores and averaged at the article level. Perplexity scores measure the amount of surprise toward seeing a given sequence of words, and they are often used as a similarity metric to determine similarity between texts; lower perplexity means the given text is more similar to the training corpus [57]. In our experiment, this implies that the article’s text is similar to the writing style indicative of the genre. We intentionally use pre-neural language models in order to avoid data leakage and to focus on local, morph-syntactic similarities. With other language model architectures, it is unclear on what linguistic level the similarity is determined.

We then compare each cluster’s average perplexity to the dataset average by measuring the Z-score as in Eq 6. For a cluster c, let x_c,g be the average perplexity value for c’s articles with respect to genre g. This cluster’s perplexity is then compared to the dataset average for all articles shared with respect to that genre, specified by the mean and standard deviation and .

(6)

Characterizing the ideologically homogeneous clusters

Based on these clusters, we calculate the ideological homogeneity of the set of clusters in each time window. The most homogeneous quintile are then labeled as ideologically homogeneous clusters, or IHCs. To uncover where in our data the most polarized groups are overrepresented, we first examined ideological homogeneity with respect to the cluster size (Fig 2B). We find that on Twitter the smaller sized clusters are enriched with IHCs, containing much higher proportions than are attributable to random chance (indicated by a horizontal dashed line at 0.2). This is expected behavior, as smaller groups are more likely to be homogeneous than larger groups. Smaller groups minimize the denominator of our ideological homogeneity metric and have fewer opportunities to diversify their information sources [61,62]. Clusters of size 3 through 7 have statistically significant enrichment of IHCs, which was verified using permutation tests (p < 0.01).

On Parler, IHCs are distributed differently than on Twitter as the larger clusters with size 100 through 1,000 users are significantly enriched with IHCs (inset, Fig 2B). We used permutation tests to find that clusters between size 99 and 965 are significantly more likely to be IHCs than random chance would indicate (horizontal dashed line at 0.2; p < 0.01). One potential explanation for this divergence is based on Parler’s largest growth periods being driven by conservative commentators publicly promoting the platform, and subsequent bursts of users joining Parler are dominated by followers of these commentators [24]. Such a trend potentially led to Parler being more monolithic in its cluster structure, as well as its bias.

Ideologically homogeneous clusters spread articles more widely

Having described the ideologically homogeneous subpopulation on each platform as a function of cluster size, we subsequently determine the effect they have on the wider social networks they participate in. If the most homogeneous clusters post content that is incendiary but fails to resonate with users, their real impact is negligible. On Twitter and Parler, users can retweet or repost another user’s content as a means to broadcast to their followers. This functionality allows information to potentially cascade through a network rapidly. Thus, the amount of retweets a group can gather on its posts is a useful proxy for understanding which groups are resonating best with the user base. For this reason, we examined the retweet and repost distribution of ideologically homogeneous clusters relative to the less homogeneous clusters with respect to three retweet metrics.

Our first, and simplest, measure of a group’s message spreading through retweets is the absolute number of retweets gathered by IHCs. Fig 3 compares the average retweets per user in IHCs to non-IHC groups for 40 cluster size bins. In Fig 3A, we observe that in every cluster bin IHCs receive a higher number of retweets or reposts, a finding that was confirmed using a Welch’s t-test (p < 0.01). Interestingly, we find that the largest differences occur for cluster sizes with fewer IHCs. In addition to the analysis of raw retweets, which may include factors like retweets from out-of-sample users or a high number of retweets coming from within a post’s cluster, we directly examine the extent to which a cluster’s posts entangled users from other clusters. Notably, such calculation offers insight into how effective IHCs are at spreading messages outside of their immediate contacts. Similarly to the previous analysis, we control for cluster size by segmenting the data into 40 bins and count for each cluster the number of edges to other clusters. These values are averaged on a per user basis. In Fig 3B we observe a divergence between the two platforms, where both IHCs and less-homogeneous clusters have similar inter-cluster retweet values that tend to be low on Parler. Such observations suggest that users of the alternative social media platform tend to confine themselves to their immediate contacts regardless of their ideological homogeneity value. On Twitter, inter-cluster retweet values correlate with cluster size, and larger clusters reach more external clusters. Additionally, IHCs larger than 500 users reach significantly more external clusters (Welch’s t-test: p < 0.01).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Retweet characteristics of IHCs.

(A) Retweet spread ratio as a function of cluster size on Parler and Twitter (inset). Retweet spread ratio is defined as the ratio of retweets per user in IHCs and retweets per user in all other clusters. For all cluster sizes, the IHCs have significantly higher retweet spread than similarly sized but less homogeneous clusters. On Twitter, the IHCs are much more effective at gathering reposts when compared to Parler. (B) Inter-cluster spread ratio as a function of cluster size for Parler groups and Twitter groups (inset). Inter-cluster spread ratio was calculated as the ratio of the number of edges from a cluster to another unique cluster and the number of users in that cluster. The results show that users on Parler tend to refrain from interacting outside of their primary clusters, regardless of the ideological homogeneity value. On Twitter, IHC groups are more effective at gathering out of cluster retweets, a differential that grows for larger clusters.

https://doi.org/10.1371/journal.pone.0338318.g003

Finally, we examine the participation of IHCs in the edge boundary on Twitter. For this purpose, we consider the edge boundary of a retweet graph as the set of edges that connect a right-leaning cluster to a left-leaning cluster. We consider a cluster to be left-leaning if the average leaning of its users is less than –0.5 and right-leaning if the average leaning of its users is greater than 0.5. For each time window, we calculate the fraction of edges along the boundary that include at least one user that belongs to an ideologically homogeneous cluster. On average, across each time step in the Twitter data, only 2.87% of boundary edges contain a user from an IHC. If edges in the graph were assigned randomly, we would expect 15.05% of boundary edges to contain users from an IHC, suggesting that ideologically homogeneous clusters very rarely interact with differently biased groups (S3 Fig).

Article texts reveal a divergence in language use

Previous works have argued that a central theme in conspiracy theory texts is the frequent use of worldbuilding language [63–65]. Conspiracy theory texts, as a general template, do not attempt to logically persuade the reader to reject the mainstream view and adopt their own worldview. Instead, they offer a characterization of the world where their views are portrayed as a natural consequence [66,67]. Therefore, we also measure the linguistic and stylistic characteristics of the articles that are shared in clusters with high ideological homogeneity and higher likelihood to spread their messages through the network. From these insights, we determine whether shared right-biased articles supportive of the voter fraud conspiracy theory have similarity to worldbuilding language as suggested in previous work.

Detecting a high-level concept such as worldbuilding is not a simple task. To accomplish this task, we first had subject matter experts assign genre labels to each corpus based on its texts’ narrative and formal features. The writing genres were divided into three categorically similar groups, Worldbuilding and Speculative Fiction genres, Hybrid genres, and Non-Speculative genres. These classifications can be seen in Table 1. The Worldbuilding and Speculative Fiction genres represent fiction writing that dedicates significant time to characterizing its world, while the Non-Speculative genres contain minimal amounts of text describing the world and other contextual factors. The Hybrid genres contain writing along the spectrum of worldbuilding that contains elements of speculative writing, but not enough for texts to be considered full Speculative Fiction. Detailed descriptions of the 14 genres are included as S2 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Genres categorized by theme.

https://doi.org/10.1371/journal.pone.0338318.t001

We trained a trigram language model on each of these 14 genres to learn the probability distributions of words in each genre. We then passed each article shared within our Twitter and Parler datasets through these language models, assigning a perplexity score to quantify its similarity to each of the 14 writing genres. Such a perplexity score measures how well the language model is able to predict the next word of a text using the three previous words and its genre-specific training data. Lower perplexity scores indicate that a text is easy to predict using the training data, which we infer implies the text is stylistically closer to the training genre. Notably, each cluster is assigned a score to roughly quantify the extent to which its articles use language indicative of certain literary genres.

To show that our genre experiment methodology produces legitimate results, we run a baseline experiment using Facebook data from pro-vaccine and anti-vaccine groups [16]. The results show that the anti-vaccine groups post content that is most similar to the speculative fiction genres and several hybrid generes (S4 Fig). These genres, specifically the Epiphanic and Eschatology genres, point to a narrative describing an awakening to the reality of corrupt activity hidden from view, a result consistent with descriptions of conspiracy narratives. Comparatively, the pro-vaccine texts are most similar to the Political Treatises hybrid genre and the Activism non-speculative genre, both indicative of lower amounts of worldbuilding.

For our subsequent experiment on the Twitter and Parler data, we calculate the average perplexity score across each dataset for the 14 genres we investigated. The perplexity scores measure the word prediction entropy of a language model trained on writings from a certain genre; lower values of perplexity indicate that a text is closer to that genre in terms of style and word choice.

Our experiment seeks to determine whether IHCs in our data are likely to show higher similarity to the Worldbuilding genres than expected by chance. We consider an IHC to be highly similar to a genre if it has a Z-score below –1, indicating that it is more than one standard deviation below the genre mean. Each IHC was then given a binary label to signify whether its texts are highly similar to each genre. These binary labels are used as the input to a permutation test, the result of which identifies the genres that are enriched with IHCs more than would be predicted by random chance. Fig 4 quantifies this random chance cutoff as a vertical dashed line and demonstrates for both platforms which genres’ highly similar clusters are enriched with IHCs, visualized as bars crossing this cutoff, more than random chance would predict. To further increase the precision of our experiment, we split the set of IHCs on Twitter and Parler into subsets of left-leaning IHCs and right-leaning IHCs. We consider a cluster to be left-leaning if the average leaning of its users is less than –0.5 and right-leaning if the average leaning of its users is greater than 0.5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Fraction of all IHCs that are at least one standard deviation below the average value with respect to writing genre.

(A) On Twitter, left-leaning groups post content that have high similarity to the Alternative History genre and the Activism Non-Speculative genre, while the right leaning groups posts are most similar to both Speculative Fiction and Hybrid genres. Conversely, on Parler (B), the right-leaning groups have significant similarity to the Hybrid genres as well as the Alternative History and Activism genres indicative of the election context. The dotted line indicates the expected, random distribution of genres.

https://doi.org/10.1371/journal.pone.0338318.g004

On Twitter (Fig 4A), we find that content that the right-leaning IHCs post have above average similarity to most of the speculative fiction genres and hybrid genres, as well as the Activism genre. When these ideologically homogeneous election skeptic groups are broadcasting to a wider mainstream audience, they rely on worldbuilding to frame their worldview as a compelling call-to-action. In particular, there exists a mixture of worldbuilding narratives and factual reporting among the articles shared in these right-leaning IHCs. In turn, left-leaning IHCs are most similar to the Alternative History speculative genre and the Activism and Political Treatise genres. We suppose that Alternative History and Political Treatises are genres common to all IHCs because these genres are indicative of the political subject matter being discussed. Otherwise, we find that on Twitter, the ideologically homogeneous right-leaning groups rely more on using worldbuilding language to convert users on the more popular platform to their worldview.

The results for right-leaning clusters on Parler (Fig 4B), paint a different picture. The Alternative History and Dystopia speculative fiction genres are significantly represented as they were on Twitter, but these were the only worldbuilding genres with significant similarity to articles shared among Parler IHCs. All other significantly similar genres may contain conspiracy tropes but do not contain distinct worldbuilding language. Due to the majority of Parler IHCs being right-leaning, the left-leaning IHCs which do exist on the platform showed no noteworthy genre similarities. These results indicate that, when among users with similar views, or more specifically among users that do not need to be convinced to question the election, the ideologically homogeneous election skeptic groups do not follow worldbuilding forms as strictly.