Target Stance Classification

Target Stance Classification focuses on classifying the stance of a user or document for a predefined topic or target. Task 6 of Semeval 2016 is a common benchmark for target stance classification. Authors have achieved SOTA results on this benchmark using architectures ranging from end-to-end neural ensemble models [13] to hand-crafted feature-based classifiers [14]. The latter relies on domain knowledge to determine the most informative features for a given problem. Due to the limited amount of data available, algorithms that rely on hand-crafted features are still prominent and achieve competitive performances with Deep Learning algorithms. Additionally, the vast majority of stance-detection literature focuses on classifying documents. Considering users adds additional complexity to the task as users can post many unrelated documents or express contradictory or nuanced stances towards targets that may not be captured in a single post. Even though there exists work that has explored user-level stance detection on Twitter data [15], the proposed approach requires pairwise similarity calculations for all users of interest, which limits the scalability of the approach to large social media networks like those we use in this work. To make the approach tractable, the authors dropped all but the most active 5k users—the pairwise similarity bottleneck limits the ability of the approach to scale to millions of users.

GNNs for Stance Detection

Graph Neural Networks (GNNs) have become increasingly popular in recent years, but to our knowledge, have only recently been used for stance detection at the user level. Graph Neural Networks have been used in tasks related to stance in conversations at a document level, including several papers that explore GNNs for fake news classification and rumor detection. For example, a gated graph neural network model, PCGNN [16], was proposed for rumor detection on Tweet threads achieving SoTA results on the PHEME dataset [17]. Reference [18] propose a modified label propagation algorithm that they benchmark against graph neural network architectures for user-level stance prediction. Reference [19] proposed an unsupervised BERT training approach that leverages user retweet interactions and fine-tunes this for user political stance detection. Most works focus on text-level classification. While this is effective for fine-grained stance detection, no strong inference can be made about the alignment or intentions of the user who posted the false or misleading content. To combat misinformation at scale in an environment with state-backed propaganda campaigns, troll farms, and bot networks, stance detection at the user level is essential for moderation at scale.

Generalizing Stance classification

Several recent efforts have been made to study the generalization capabilities of stance classifiers to different domains, contexts, or targets. One vein of this research, known as cross-target stance detection, focuses on applying a target stance classifier trained on one domain to a different, but related, domain. Work in this area has explored how to leverage labeled instances from related political topics to increase generalizability across targets [20,21], or how effectively stance predictions transfer across languages [22]. Similarly, other work has explored how synthetic data, generated from related targets, can improve cross-target stance detection on Tweets [23]. However, in contrast to our work, research in this field improves the generalizability of classifiers across different targets by leveraging their semantic relationship at a document level. For this reason, each of these works explores stance at the tweet/ document level, rather than at a user level. More recent work [24], proposes a methodology for multi-modal cross-target stance detection on 3,871 Twitter users, however, their model uses followership and friendship networks for each user, which are often not feasible to extract on large networks given API limitations. Importantly, our work differs from these approaches as, for a fixed target, we explore how varying the amount of contextual information for a given user (an individual post, all a user’s posts, or their local ego-network) improves within-sample and out-of-sample predictions.

A similar vein of research, known as context-sensitive classification, models context in the form of spatial and temporal locality and is crucial to many tasks, including search engines, and context-aware Web applications [25]. This is highly relevant for stance detection in online social media where users engage with one another in a variety of ways, and thus focusing on an isolated sentence will likely result in an incomplete assessment of a user’s stance, as they can employ humor, satire, ask a question, or talk about something else entirely [5]. Although some works have implicitly demonstrated the benefits of this approach by incorporating different levels of context in their architecture [26,27], there is a need to systematically quantify the resulting gains in classifier performance as different levels of contextual information become available to the model. This line of questioning has been identified as an outstanding issue in the task of stance classification [6], and is something this work attempts to explore.

3.1 The 2020 Chilean Plebiscite

Throughout the 2019 Chilean protests, different social movements made calls in favor of drafting a new constitution. This social pressure reached a boiling point in November of that same year, which led the Chilean National Congress to agree to hold a National Plebiscite. The Plebiscite was scheduled for the start of 2020 but was delayed because of the Coronavirus pandemic. In October of 2020, it was overwhelmingly approved with 78% of the vote. Using Twitter’s v2 full-archive search endpoint feature which was available on their Academic Research Track, we collected tweets from September 25 to November 10 of 2020 (a month prior and two weeks after the plebiscite took place). This research program allowed full historical access to publicly available tweets matching complex queries. The queries matched 124 hashtags and terms relevant to the event (e.g.: #Plebiscito2020, #PlebiscitoChile, #NuevaConstitucion, etc.). We then labeled these hashtags to identify useful “Stance Tags”.

The labeling procedure was as follows: an expert in the South American region who is fluent in Spanish reviewed a sample of tweets at the end of the collection period. The annotator then established if the tweets were used consistently in favor or against the approval of the new Constitution (or of the referendum process in general), or if this was not possible the hashtag was labeled “Undetermined”. This was done in separate meetings with the authors of this paper, where the reasoning for each label was openly discussed. This process resulted in 27 “Undetermined”, 64 “Against”, and 32 “Approve” stance tags. Instead of using hand-labeled Political Figures for the second stance signal, as was done for the construction of the protest data [7], we opted to identify the hashtags used in the user description that were explicitly rejecting or approving the referendum in first person. This resulted in 16 “Approve” (#Apruebo, #AprueboCC, etc.) and 21 that “Against” (#YoRechazo, #Rechazo, etc.). This set is used to label users based on their user descriptions. We found that the usage of stance tags in the descriptions as a replacement signal, despite requiring less annotation effort than labeling political figures, also resulted in meaningful polarized stance partitions based on the same language metrics used to validate the protest weak-labels.

Following the previously described methodology for the construction of the South American protest data [7], tweets or descriptions were assigned a stance if they used stance tags with consistent stances. We only proceeded with users that had at least 8 tweets with a consistent stance or if at least one description was consistent. A stance was assigned to a user if at least 90% of their tweets had the same stance. For description-based stances, a user was assigned a stance if all different user descriptions observed during the period were assigned the same stance. The final user stance was determined by combining the labels obtained from both sources and validating the stance assigned to a subset of the users. The final number of user counts and their distributions are presented in Table 2. Moreover, as shown in Table 3, 43.89% of users labeled were part of the 2019 Chilean protest training set while, by construction, no tweets were part of the protest data. We also note that only 0.015% of retweets that occurred in the referendum collection reference a tweet that occurred during the protests (this is consistent with the diffusion dynamics of retweets [30]). Moreover, to improve the resolution of the networks available for the labeled users, we collected their timelines during the event. Timelines were not always collected in the original protest data, but we hypothesize that the additional timeline context for each user will improve the quality of user embeddings for the referendum data.

Following X’s January 2023 User Protection Policy update, tweet IDs related to sensitive political events cannot be publicly shared. We respect this policy, and instead share only anonymized network edges, the type of tweet the edge represents (Original, Reply, or Quote), and user weights for replication. These resources can be found at: https://doi.org/10.5281/zenodo.14207926. The code for this paper is publicly released at https://github.com/rvillaco/Protest_Stance_Detection and the weights for the trained models are available at https://doi.org/10.5281/zenodo.15571840.

4.1 Tweet encoder

The first component of the model creates embeddings for the different tweets produced by the users. For this purpose we pre-trained a BERT [8] language model, which we call TwBETO v0, following the robust approach introduced in RoBERTa [31]. We opted for the smaller architecture dimensions introduced in DistilBERT [32], namely, 6 hidden layers with 12 attention heads and a hidden dimension of 768. We also reduce the model’s maximum sequence length to 128 tokens, following another BERT instantiation trained on English Twitter data [33]. We utilize the RoBERTa implementation in the Hugging Face library [34] and optimize the model using Adam with weight decay, a linear schedule with warmup and a maximum learning rate of 2e-4. We use a global batch size (via gradient accumulation) of 5k across 4 Titan XP GPUs (12 GB RAM each) and trained the model for 650 hours.

The model was trained with a corpus comprised of 155M Spanish tweets (4.5B words tokens), as determined by Twitter’s API, and includes only original tweets (retweets are filtered out) with more than 6 tokens, while long tweets were truncated to 64-word tokens. The data was compiled from the following sources:

• 110M Tweets (3B word tokens) from the South American protests collected from September 20 to December 31 of 2019.
• 25M (0.7B word tokens) Tweets collected around the Coronavirus pandemic from April 01 to December 31 of 2020.
• 3M (0.3B word tokens) Tweets collected around the Chilean referendum from September 25 to November 10 of 2020.
• 17M (0.5B word tokens) rehydrated targets across all the collections listed.

Tweets are pretokenized using the “TweetTokenizer” from the NLTK toolkit [35] and use the emoji package to translate emotion icons into word tokens (in Spanish). We also preprocess the Tweets by replacing user mentions with “USER_AT” and, using the tweet JSON, we replace media urls with “HTTPMEDIA” and web urls with “HTTPURL”. We found that this new model produced significantly better quality embeddings than the only other currently available Spanish BERT variant for Twitter [9]. We hypothesize that this is a result of the latter being trained mainly in European Spanish with fewer data and it not applying the RoBERTa pretraining framework. We make the pretrained TwBETO v0 language model available through the Hugging Face model hub (https://huggingface.co/Ramavill/twBETO_v0).

The output of our TwBETO model, after being fed a batch of tweets, is pooled and passed through an activation layer. We use the standard pooling method used for BERT models, namely using the first element of the output of the final layer (corresponding to the “” token), as input for a fully connected layer and passing through an activation. This output serves as our final Tweet Embeddings of size h_hid = 768 per tweet. For the Tweet-level Context prediction, the embeddings are used as input for a classification head comprising a fully-connected layer and a softmax to predict the stance of a batch of tweets. Given that our main focus is to evaluate the effect of increasing the contextual user information available to the model, we do not fine-tune the TwBETO parameters in any training settings.

4.2 User encoder

As shown in the center part of Fig 1, the second component of our architecture is comprised of a stack of Transformer Encoder blocks [36] which operate on the Tweet Embeddings for a given user. As this requires a fixed input size, we introduced a parameter n_max that determines the maximum number of tweets to consider for each user. In this work, we use n_max = 15. When users exceeded this limit, we sampled n_max tweets before assigning them to a batch. This way, we avoided wasting information as different tweets can be included each time a user is sampled. We hypothesize that this, combined with the resampling approach taken to account for class imbalance, improved training stability as repeated users had a high probability of including different tweets (at least 83% of the users in each training set had 15 or more tweets in the dataset). Given the power law distribution of user tweet counts and to reduce sampling times when constructing the batches, we chose only to keep the first 150 tweets available for each user. For the Tweet Embeddings to serve as input for this component we reshape them and append a [CLS] parameter vector at the start of each user’s tweets, obtaining a tensor with dimensions: . Where b_size is the number of users included in the batch and is the aforementioned maximum number of tweets allowed per user in the batch. We used trainable positional embeddings to maintain the temporality of the tweets and introduced a second type of trained embeddings to encode the type of each tweet as Original, Reply or Quote. Twitter users can interact in various ways and we hypothesized that different interaction types would have different impacts on stance. This was confirmed in our ablation study (done for Bolivia), as the inclusion of each of these components improved the validation macro-F1 score at this context level. These 3 embedding tensors are then added and normalized (by layer normalization) and serve as input for our encoder stack. Based on validation results, we use an encoder stack of size L_x = 3, an intermediate embedding size of 2048, 6 attention heads and a GELU activation for the transformer stacks.

The output of the last encoder layer is pooled following the same strategy used for the TwBETO model as this allows the [CLS] parameter to attend to all tweets in the sequence and be optimized for stance classification. The output of this pooling layer serves as our final User Embeddings which maintain a dimension size of h_hid = 768. As before we feed the embeddings to a classification head to perform user-level classification, or use this as input for the next component or our architecture.

4.3 Network-based prediction

We take the embeddings produced by the user encoder and predict users’ stances using their interactions in the social network. We explored several different graph neural network architectures, including GraphSage layers [37], Graph Convolutional layers [38], and W-L Graph Convolutional layers [39]. Ultimately, we achieved the highest macro F1 score using Graph Attention Network (GAT) layers [40]. In the social setting imposed by Twitter’s platform, there are also different user information context levels that can be leveraged by our Network-based classifier to improve stance detection. To test this, we trained separate homogeneous models for the network retweets, the network of responses (the union of replies and quotes) and the combined network (obtained by the union of both). Finally, we trained a heterogeneous model where we distinguish response and retweet edges.

For homogeneous graph experiments, after testing different alternatives, we use two GAT Layers with dropout, each with four attention heads, followed by 4 batch-normed linear layers with dropout and relu activations. The corresponding output is fed to a classification head for our final network-based user stance classification. We observed our heterogeneous model quickly overfit the data with our homogeneous architecture, so for the heterogeneous model, we removed 2 linear layers and use only one attention head (for each edge type). Given the size of the underlying graphs, we sample neighborhoods for each node using the neighbor loader proposed in [37].

In our GAT models, for a given user i, the input to the attention layer is the User Embeddings of a random sample of k one-hop neighbors of i denoted as . In our homogeneous GAT networks, the attention mechanism calculates the importance of to the label of i. In our heterogeneous GAT networks, the attention mechanism calculates the 𝒩_k(i) with relation (retweet or response) to the label of i.

4.4 Baselines

To demonstrate the overall effectiveness of our approach, we sought to re-implement as baselines four recent user-level stance detection papers on our data that both leverage user-interaction networks. However, two of these approaches were not possible to re-implement as we discuss next. The first proposed an unsupervised clustering methodology for user-level stance detection [15], that did not scale to the dataset explored in this work as it requires pairwise similarity calculations for all users of interest. The second approach proposed a multi-modal cross-target stance detection architecture [24] that uses followership and friendship networks for each user which were not possible to recover due to API constraints.

The first approach that we successfully adapted to our dataset was TSPA [18], which uses label propagation first over the retweet network, and then over the reply network. We implement the TSPA version without weighted propagation as it achieves comparable performance to weighted propagation, and it provides us with a baseline that uses only network information. This helps us understand the signals associated with the natural community structure that exists within the network. As the original authors, we were unable to run the algorithm on the Chilean dataset given its size. We followed the same pre-labeling approach used by the authors, but note that not all labeled users could be assigned predictions as some components of the interaction for each country did not receive labels. For fairness of comparison, we evaluate the accuracy and F1 only on users for which the algorithm assigned labels (a total of only 21 users across the Bolivia, Colombia and Ecuador test sets were unlabeled). As TSPA is an extension of the label-propagation algorithm, cross-country baselines are not possible with the algorithm. TSPA uses only network signals, and apart from initial hashtag label seeds, does not incorporate any external user text or features.

The final baseline implemented was Retweet-BERT [19], which performs unsupervised training on Sentence-BERT to contrastively embed user descriptions based on user retweet networks. They then fine-tune this trained model on user political stance task. As not all users in our data have descriptions, and many user descriptions changed, we implement the unsupervised fine-tuning step only on the first description for each labeled user using the retweet network of all labeled users. We then fine-tune this model on training users with descriptions and evaluate the model on validation and test users with descriptions. As users were randomly split into train, validation, and test sets, this approach impacted each split roughly equally. In each country between 70% and 80% of labeled users had descriptions. As data were weakly labeled using hashtags, hashtags could provide too easy a signal. Consequently, all hashtags and urls were removed from tweets and user descriptions baselines and for our proposed architectures.

5.1 Main results

In this section, we present the results of the different components of our architecture to evaluate the effects of increasing the level of context available to a classifier. We use the Tweet-level classification as a baseline for our ablation studies. It is important to note that this task is evaluated at a tweet level (not at a user level) by assigning each its corresponding user stance. We define four different baseline experiments to assess the performance of classifiers trained at this context level:

• Weak-Labeled Tweets: This model was trained using tweets that contained “Stance Tags” as identified in [7]. All trailing hashtags were removed during pre-processing to avoid overfitting to these tokens.
• All Original Tweets: This model was trained using all original, replies and quote tweets produced by the different weakly-labeled users.
• User Average of Original Tweets: This model was trained by averaging the predicted labels assigned to a given user. A user is assigned the majority of their predicted labels. When no majority exists, a stance is assigned randomly.
• User Average of All Tweets: This User-Level prediction exercise is similar to the one described above, but also includes the retweets of a user. Retweets are assigned the label predicted for their target.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance of in-country Stance Classifiers at different context levels.

Model performance metrics correspond to the median of 1000 bootstrapped samples of the test set. For each metric, the classifiers whose performance is not statistically different, at a 95% confidence level, from the best-performing model are highlighted in bold. We also highlight in bold the name of the best-performing model if it obtains the best results (or is statistically tied) in all presented metrics. The corresponding confidence bands, computed from the bootstrapped empirical distribution of each metric, are included in Table 1 of S1.

https://doi.org/10.1371/journal.pone.0324697.t004

We include the last two user-level predictions to evaluate how effective our proposed User Transformer is at aggregating tweet information. We also include the TSPA model [18] (see Sect 4.4) as a network baseline that uses only a user’s ego-network information. This helps us understand the signals associated with the natural community structure that exists within the network. However, as mentioned before, TSPA could not be run on Chile due to its data size. We evaluate the performance of each model using accuracy, macro F1 (due to the observed asymmetries in the label distribution), and the Area Under the Receiver Operating Characteristic (ROC AUC) on the held-out test set. The latter metric accounts for the possible differences in the optimal decision threshold that might exist across the different datasets. To assess whether the differences seen are statistically different, we recomputed the model performance metrics based on 1000 bootstrapped samples of the corresponding test set. We report the median result of the bootstrapped empirical distribution for each metric and compute its corresponding confidence band to assess whether the differences are statistically significant (these are presented in the Supporting Information).

In Table 4, we present the results of the ablation studies. As shown, there is a clear performance improvement when more contextual information is available to the classifier. As expected, focusing only on the weakly-labeled tweets presents an easier task than focusing on all original tweets, but still falls short of User-Level averages. When retweets are included, the performance improves but is still significantly below the User Transformer in all countries and across all metrics (see Table 1 of S1 for the 95% confidence bands of each metric). Interestingly, we observed that, for Colombia, adding retweets hurts the user-average performance. The User Transformer does not exhibit these disparate behaviors, highlighting its ability to identify tweets that are more relevant to the stance of the user. While the user-level transformer significantly outperformed Retweet-BERT [19] in all settings, it yielded statistically significant lower accuracy and M-F1 scores than TSPA in Bolivia and Ecuador. In the case of Colombia, the differences are not significantly different, which is consistent with what was noted before for the Retweet Average baseline. TSPA’s relatively strong performance demonstrates the strong stance signals present in each network, even without user text features. Nonetheless, the User Transformer significantly outperforms it, across all countries, when considering their ROC AUC score, which shows that no model uniformly outcompetes their counterpart. Importantly, and as hypothesized at the start of this work, leveraging the ego-network of the user, in conjunction with the semantic features aggregated by the User-level Transformer, can consistently improve upon its already high bar. In all four countries, the heterogeneous model, which leverages all network information, yielded macro F1 scores greater than 94.1 and accuracy scores greater than 95.7. The difference in model performance is significantly different in Chile while for the remaining countries, other network models or the User Transformer can match its performance in some metric. However, as we will explore next, the differences in performance are exacerbated when exploring each model’s generalization capabilities.

5.2 Robustness analysis

Cross-country robustness.

Even though the addition of social context can lead to an in-sample increase in classification performance, we also seek to evaluate how context can affect the generalizability of our proposed classifiers. Given that our target countries share a common language, we hypothesized that the models should be able to extrapolate the stance learned from one country to another. Protests in Bolivia had opposing ideological motivations in comparison with the motivations observed in the other countries, which we hypothesized would degrade cross-country performance significantly. The results for the cross-country ablation are shown in Table 5. For these experiments, we excluded from the test country (column) any user that was part of the training set of the other country’s classifier (row). Refer to Table 3 for more details on the overlap between the different collections. As before, we recomputed the model performance metrics based on 1000 bootstrapped samples of the corresponding cross-country test set. For the economy of space, we only include the macro F1 score and the ROC AUC metric for the best-performing classifiers at each context level. As mentioned in Sect 4.4, we do not include the TSPA baseline as it is an extension of a label propagation algorithm that relies on hashtag seeds that are not reliably used in different protest collections.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Macro F-1 (%) score and Area Under de ROC curve for out-of-sample cross-country predictions for classifiers at different context levels. Model performance metrics correspond to the median of 1000 bootstrapped samples from column-country. This excluded users seen during training of the country classifier (row). For each metric, the classifiers whose performance is not statistically different, at a 99% confidence level, from the best-performing model are highlighted in bold. We also highlight in bold the name of the best-performing model if it obtains the best results (or is statistically tied) in all presented metrics. For the Bolivian case study, the worst-performing classifier is highlighted. The corresponding confidence bands, computed from the bootstrapped empirical distribution of each metric, are included in Table 2 of S1.

https://doi.org/10.1371/journal.pone.0324697.t005

As shown, the Bolivian case serves as an adversarial setting for classifiers trained in other countries, which suggests that, when applied to this country, the semantic features leveraged by the classifiers are operating on an ideological dimension. This is consistent with results presented in [7], where the authors found that language polarization remained along ideological lines when comparing protests of any of the three countries with Bolivia. This was not the case when comparing protests of the other three countries. As expected, Chile, Colombia, and Ecuador exhibit strong pairwise performance consistent with their ideological alignment. We can also note that the Tweet-Level performance varies significantly, falling in some cases 30 points below the user Transformer’s macro F1, which is far more stable (73–85% in these countries). However, when contrasting their ROC AUC performance, both models remain competitive, which suggests that there are country-specific thresholds that can significantly improve the average tweet-level classification performance. Nonetheless, given that we seek to evaluate the one-shot performance of the different models (including its optimal decision threshold for the respective country), we will focus mainly on differences in macro F1 performance for the robustness analysis.

We found that on average, excluding Bolivian predictions, the heterogeneous network model increased cross-country prediction macro F1 score by 7.62 points and significantly outperforms all other models in all metrics and all country combinations except for one discussed below. In the best case, the Ecuadorian instance yielded a zero-shot macro F1 of 95.5 on Colombian data, improving on its User Transformer counterpart by 15 points. Interestingly enough, the only case where the heterogeneous model underperforms its User Transformer counterpart (by 4 points) is when applying the Colombian models to Ecuadorian data. It is worth noting that although the heterogeneous model achieves this in countries with similarly motivated protests, it also significantly underperforms other alternatives in the Bolivian case (except for the Colombian classifiers). This is consistent with the observed tendency of deep learning models to be overconfident in their predictions and highlights the importance of domain knowledge when applying these models to different domains. If we leverage the knowledge of the opposed motivations for the Bolivian protests by inverting the heterogeneous model’s predictions, its performance would be in line with what was observed for the other countries. We take the approach of inverting the predictions when we report the results of the 2020 Chilean referendum.

Robustness over time.

To test the effect of context over time, we applied the Chilean protests classifiers to predict the stance of users, in a zero-shot setting, towards the 2020 Plebiscite vote for drafting a new constitution. In contrast to previous experiments, we report the results for all users, regardless of whether or not they were seen when training the Chilean models. However, we segment the results based on this condition, as it allows us to contrast the performance of the different models on new users. We take this approach given that (1) it is often realistic on applied settings and (2) there is no overlap between the tweets seen in both collections (see Table 3). Additionally, only 0.015% of retweets in the referendum collection reference a tweet that occurred during the protests.

We hypothesized that opposition to the government during the protests should signal endorsement for the new constitution (the models should be good inverse classifiers). Table 6 presents the results of this exercise. We also include the predicted Referendum vote obtained when applying each classifier, based on the two-hop neighborhood of labeled users. The benefits of the heterogeneous model are more starkly observed in this setting, as it significantly outperforms all other variants across all metrics. However, surprisingly, the homogeneous network models underperform the User-level Transformer in all settings and these differences are statistically significant at a 99% confidence level. We hypothesize that this might be due in part to differences in network construction: the Referendum data included the timeline of all users collected, whereas this was not done for the country-level protest data. This implies that the networks constructed for the former dataset are denser and include more reply and quote interactions between users. Hence, it more likely that we capture both positive and negative responses from one user to others of the same or different stances. While the User-level transformer can leverage the semantic context of each tweet as it pertains to the stance of the user, the homogenous models are forced to collapse these disparate signals into one interaction link. As such, we see that the Retweet-based homogenous model shows the least degradation (as retweets provide more consistent endorsements), while the combined network model drops significantly below other simple average user-level models. However, when the model is allowed to leverage the heterogeneous signals provided by retweets and other responses, its performance improves dramatically. Moreover, except for the heterogeneous network model, the performance of all classifiers is significantly reduced when dealing with completely new users at a 99% confidence level. This again highlights how the disparate signals present in a user’s ego-network can improve the robustness of the classifier over time. The User-level Transformer shows the second least degradation, while other models show a 10-point drop in performance. With regards to the predicted referendum vote (shown as “Pr-Ref” in Table 6), we note that semantic-based classifiers tend to undershoot the observed Referendum tallies (the final vote was 78% in favor of the new constitution), while the single-network-based classifiers do the opposite. Interestingly, the heterogeneous model predicts the voting ratio almost perfectly which is encouraging. Albeit, we stress that given that there is no evidence that the sample constructed is representative of the Chilean population, the closeness of this prediction is merely suggestive of good performance and more research is necessary to assess if this behavior is truly generalizable. The network models’ strong performance, when allowed to leverage the heterogeneous interactions between users, can be in part explained by the fact that language and conversation topics change faster than social ties, and the fact that social ties themselves can alter language [41]. As a result, features extracted from interaction networks may carry strong signals that are more generalizable to different cultural contexts.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Out of sample Predictions for the Chilean Referendum at different context levels. These correspond to the classifier trained on the 2019 Chilean Protest Data, but with inverted labels (“Pro” government are considered “Against” the referendum and vice-versa). Model performance metrics correspond to the median of 1000 bootstrapped samples of the Referendum collection, disaggregated by whether a user was seen during training of the Chilean Protest classifier. For each metric, the classifiers whose performance is not statistically different, at a 99% confidence level, from the best-performing model are highlighted in bold. We also highlight in bold the name of the best-performing model if it obtains the best results (or is statistically tied) in all presented metrics. The corresponding confidence bands, computed from the bootstrapped empirical distribution of each metric, are included in Table 3 of S1.

https://doi.org/10.1371/journal.pone.0324697.t006

To summarize, in all but one of the robustness experiments conducted, our heterogeneous graph neural network yielded the highest accuracy and F1 scores. The strong performance of the heterogeneous model aligns with intuition, as different edges on Twitter have different social functions. Retweeting with no commentary is more likely an endorsement than an argument, whereas replying could imply a fight or it could imply an agreement. To further test this explanation, for each node i in labeled Bolivian retweet and response networks, we calculated the percent of neighbors of node i that shared the same label as node i and then took the average of those percentages. In the retweet network, the average neighborhood-agreement percentage was 93%. In the Response (reply, quote, mention) networks, neighborhood agreement was only 69%. This suggests that ability to differentiate between relation types should help the model’s performance, which was validated by the results of the referendum experiment. Our results confirm that including interaction types in models can yield modest improvements in predictive power in in-country stance detection tasks. However, we find that the inclusion of a user’s heterogeneous ego-network information yields much larger improvements in related country-context assessments or its robustness over time.