Related tasks

Topic modeling has been widely applied as an unsupervised method to discover the underlying organization of a corpus into topical groups [2, 6]. For example, Gao et al. leverage topic modeling on social media and news data for event summarization [16], while Nerghes and Lee compare topics discovered from news and from social media to perform narrative comparison analysis [38]. Such topical groupings are typically not sensitive to the fine-grained viewpoints of the author toward the subject.

Stance detection focuses on the extraction of an author’s stance towards a given target from a fixed set of options (e.g. Favor or Against) [27]. For example, Dey et al. leverage a two-stage LSTM-based model to first classify the overall subjectivity of tweets, followed by a stance detection step [12]. While these methods typically require pre-defined target and stance categories, recent work has tackled the unsupervised stance detection task. Darwish et al. leverage unsupervised clustering of user embeddings to identify core users to represent each stance [11].

Narrative detection approaches seek to represent online messages in a narrative framework. Jachim et al. develop a method called TrollHunter which leverages correspondence analysis to identify nouns and verbs which are commonly used together in a corpus of tweets [21]. Kwak et al. develop FrameAxis which detects semantic dimensions in the word embedding space of the documents which correspond to different framings of a given issue [28]. Tangherlini et al. construct a graph-based representation of the actants and their interactions as described in a set of documents and performs community detection on this graph to extract sub-graphs corresponding to narrative frameworks [47].

Modeling methods

Weak supervision is a model training framework which leverages large datasets of automatically generated noisy labels in lieu of high quality annotations which are time and effort intensive to generate [42]. Weakly supervised approaches have been previously applied within NLP problems ranging from text and document classification [33, 34] to information extraction such as named entity recognition [29, 30]. In this work, we leverage the social proximity of users observed through interaction networks as our weak signal to infer the similarity of tweets and their viewpoints towards a particular topic. Other recent work has leveraged a combination of textual signals and social network information to infer user stance or viewpoint information [23, 46, 52, 53]. However, while these methods require both tweet content and social interaction information to infer the stance of a new tweet, our weak supervision approach is able to infer tweet viewpoint using tweet content alone.

Contrastive representation learning is often applied to different views of the same data instance. For example, in computer vision, the goal of the contrastive learning framework is to map similar images (e.g., original and augmented) close together while pushing dissimilar ones further away in the embedding space. The adoption of this framework has led to state-of-the-art performance on unsupervised image classification tasks [8, 25]. The same idea has also been applied to natural language text, especially in tasks involving text retrieval [48], text generation [1], and sentence embedding representations [15], often resulting in significant performance improvements over multiple baselines. Recent work has leveraged a contrastive training approach for stance detection by incorporating counterfactual data augmentation of matched and unmatched stances [26].

COVID-19 narrative analysis

Many studies have applied topic modeling to understand the topics on online discussion related to COVID-19 [4, 7, 13, 22, 31, 49, 51]. Jing and Ahn leverage FrameAxis to characterize differences in the framing of COVID-19 narratives between different political parties [24]. Shahsavari et al. leverage the narrative framework discovery pipeline developed in Tangherlini et al. [47] to study COVID discussion [45]. In several works Ng et al. leverage sentiment analysis combined with topic modeling to identify prominent topics and themes associated with negative sentiment towards both influenza and COVID-19 vaccination [40, 41]. These prior frameworks often leverage contextual embeddings to group texts with similar semantics, creating candidate clusters for narrative extraction. Our proposed approach aims to enhance the embeddings of tweets by incorporating viewpoint information. The resulting embeddings can help identify more meaningful and fine-grained tweet clusters, and thus improve the characterization of narratives.

Tweets processing and topic annotation

We preprocessed the tweets as follows: (1) we removed duplicate entries, such as multiple collections of the same tweet ID, (2) we ignored retweets and only considered actions with textual content written by the user (i.e., tweets, replies, and quotes); for simplicity, we refer to all these actions as tweets, (3) we removed mentions, URLs, and special characters from the tweets to focus on the natural language content, we keep only tweets that are assigned a topic by Top2Vec [2], due to its use in our data sampling pipeline, (5) we kept only tweets in English, and (6) we performed a fuzzy text deduplication technique to remove repeated tweets with almost the exact same content by stripping the text of all non-alphanumeric characters and dropping duplicates. We also removed tweets containing less than 20 characters as they constitute less than 1% of the total tweets in the dataset and often lack textual signals necessary for identifying viewpoints. For example, many of these tweets contained minimal natural language text, often including only a few words, emojis, or punctuation marks, which is not useful for our analysis. After preprocessing the tweets, we end up with 362,146 unique tweets from 191,823 users. The preprocessing steps, along with corresponding data amounts, are illustrated in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Diagram illustrating the data processing approach to extract corresponding user and tweet pairs for training the contrastive model.

The number of tweets (speech bubble icons) and users (network icons) remaining after each processing step are specified.

https://doi.org/10.1371/journal.pcsy.0000089.g001

We employ Top2Vec [2] to assign a topic to each tweet in the dataset using pre-trained embeddings from the universal-sentence-encoder model. For clustering parameters, we use and . This results in 67 topics with sizes ranging from 861 to 26,585 tweets. We further explore these broad Top2Vec topics and identify ten higher-level categories: Conspiracy, Policy, Science, General Anti-Vaccination, Personal, Politics, Rights, General, Memes, and Negative Vaccine Effects.

Since the Avax dataset does not include ground truth for viewpoint detection, we manually annotate binary user viewpoints for a selected subset of topics to create a reference dataset for model evaluation. The resulting set of manually annotated tweets facilitates the evaluation of various embedding approaches, specifically in their ability to capture both the semantic content and viewpoint of tweets. We focus on topics that are likely associated with controversial issues, identified through manual inspection of Top2Vec-generated keywords.

We found six controversial topics that were prevalent in our collection of tweets: Face masks, Re-opening schools, Vaccine passports, Depopulation agenda, Miscarriage agenda, and Vaccines. The prevalence of these topics was influenced both by the keywords used for collection of the Avax data and by the time period of the data collection when these were salient discussion areas. For each of these six topics, one of the authors randomly selected a tweet and was instructed to annotate it with a binary viewpoint towards that topic (either support/oppose or believe/refute). If the tweet did not express a clear viewpoint, it was not annotated. The annotation process concluded once we had approximately 30 tweets per viewpoint for each of the six topics. Another author of the paper reviewed the annotations to ensure that each tweet was properly labeled.

Table 1 presents the selected topics, their corresponding viewpoints, and the relevant statistics from the annotated set. We use these 365 annotated tweets as a test set to validate how well the trained models encode shared user viewpoints.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Identified set of controversial topics and viewpoints for evaluation.

The number of tweets for each topic is in parentheses. VP = Viewpoint, S = Support, O = Oppose, B = Believe, R = Refute.

https://doi.org/10.1371/journal.pcsy.0000089.t001

Constructing social networks from Avax tweets

We retrieve tweet information using the twarc Python library to rehydrate the tweet IDs. Each record in the dataset includes the following fields: the unique identifier of the retweet, the user identifier of the retweet’s author, the timestamp of the retweet, and the original tweet ID along with its author’s ID. The X (Twitter) API does not provide complete data on retweet cascades, making it challenging to reconstruct full retweet chains, including intermediate connections between retweets. In this work, we assume that each retweet directly references the original tweet.

We construct the social network by leveraging user retweet interactions. We focus on the retweet network because these interactions are most likely to indicate endorsement of a shared viewpoint between users, as opposed to other actions such as replies or mentions, where disagreement may be expressed [14, 35, 44]. We expect that users who exhibit close proximity in the network would likely share similar viewpoints, while users who are distant in the retweet network are more likely to hold differing views. By leveraging social network information as weak signals, we can improve the representation of tweets, as the implicit connections between users can offer valuable insights into user perspectives and viewpoints.

To effectively capture the diverse nature of social connections and understand the impact of noise and outliers, we construct three distinct retweet networks. Each network is designed to measure the strength of social connections in different ways, allowing for a more nuanced analysis of user interactions and their influence on viewpoint detection. We represent the retweet network as a directed graph, , where nodes represent user accounts, and an edge refers to the number of times a social media user retweeted another user. The different types of retweet networks constructed are as follows: (1) Original network, which includes all retweet interactions without filtering; (2) Active network, which includes users who have made at least two retweets over the entire observation period; and (3) Strong network, which includes only edges with an edge weight of at least 2 (i.e., each user must have retweeted another user at least twice). The Active network is a subset of the Original network, and the Strong network is a subset of the Active network. This network creation workflow is illustrated in Fig 1.

Table 2 shows the summary statistics for the giant connected component of each network. Our analysis focuses on the interactions within the giant connected component in each retweet network, as these comprise the majority of users (94%, 98.6%, and 81.3% in the original, active, and strong networks, respectively). We chose to disregard disconnected components because of their limited occurrence and the possibility that they could introduce noisy examples not directly relevant to discussions about COVID-19. We observed that all networks exhibit a high level of modularity, which indicates a strong division of the network into communities. Additionally, the filtered versions of the networks are significantly smaller, specifically by 69% for the active network and 95% for the strong network when compared to the original. Similarly, the number of engaged users (i.e., those users who retweet and also post original tweets) decreases consistently. While we expect both the active and strong networks to be less noisy, there is a trade-off in the number of examples we can generate from the smaller pool of candidate users.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Social network properties for the giant connected component in each of the three retweet networks in this study.

The Engaged Users property denotes the number of users who participated in both retweeting and generating original posts in Avax.

https://doi.org/10.1371/journal.pcsy.0000089.t002

Self-supervised social signals

To demonstrate the utility of social signals for viewpoint encoding, we first generate Node2Vec [19] embeddings for each user in the Avax Original retweet network. We concatenate the baseline DistilRoBERTa embeddings for tweets with the corresponding Node2Vec embeddings of their authors, incorporating network position information to augment the initial content embeddings. The 2D UMAP projections of this concatenated embedding space are shown in Fig 2c. We observe that this approach is able to partially separate the annotated test set by both topic and user viewpoint, in contrast with the sentence embeddings alone, showing the utility of social information. However, this approach relies on the availability of a complete view of the retweet network. In the Avax dataset, only 53% of tweets were authored by users who have interacted with other users via retweet interactions. Hence, we cannot utilize the Node2Vec representations to infer the stance or viewpoint of the users who authored the remaining 47% of the tweets. This problem is further exacerbated in less comprehensive data collections that have more limited interaction information.

Our weak supervision approach is designed to transfer the weak signals from tweets where social network information is available to those where they are lacking. For example, we can identify the textual signals that make a pair of tweets appear as if their authors were closely related or distant in the social network. This allows us to cluster all tweets using their text alone with a focus on the textual features that are indicative of the likely social relationships of the user. To this end, we leverage the social proximity between two users, which is represented by their shortest path distance in the retweet network.

To demonstrate the effectiveness of this proxy signal, we compute the correlation between two metrics: the shortest path length between users in the network and the cosine similarity of sentence embeddings for their respective tweets. We randomly sample 1000 user pairs from the Original retweet network and compute their pairwise shortest path lengths and pairwise tweet cosine similarity. We found a Pearson correlation of approximately –0.25, a Spearman correlation of –0.23, and a Kendall’s Tau correlation of –0.17 between proximity and tweet similarity across networks. The negative correlation suggests that the smaller the distance in the network, the larger the cosine similarity of the tweets. This indicates that there is a presence of signals that can help to discriminate social proximity based only on the content of tweets. The correlation scores for all three network types and three correlation measures can be found in Table 3, showing similar correlations for all three of the networks.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Correlation scores between shortest path length (social proximity) and tweet similarity (content locality) across the retweet networks extracted in Avax.

All correlation scores yield p-values below 0.05, indicating statistical significance. The tweet embeddings were extracted using DistilRoBERTa.

https://doi.org/10.1371/journal.pcsy.0000089.t003

Contrastive model

We develop a contrastive model to infuse the text embeddings with social network information. The goal of our contrastive fine-tuning task is to encode social proximity likelihood by making the embeddings for pairs of tweets similar if their authors are close (positive samples) and dissimilar if their authors are distant (negative samples). To achieve this, we leverage a contrastive loss function based on cosine similarity to measure proximity in the latent space.

where x₁ and x₂ are vectors, y is the label (1 for positive samples and –1 for negative samples), cos is the cosine similarity function, and m is a margin value. Through experimentation on the full training data, we tested margin values between 0 and 0.5, selecting m equal to 0. This means that the cosine embedding loss penalizes dissimilar pairs that have positive cosine similarity.

We tested two different pre-trained embedding models implemented in HuggingFace: DistilRoBERTa, which uses the all-distilroberta-v1 model (https://huggingface.co/sentence-transformers/all-distilroberta-v1) [43], and CT-BERT, which uses the covid-twitter-bert-v2 model (https://huggingface.co/digitalepidemiologylab/covid-twitter-bert-v2) [36]. We expect that DistilRoBERTa will better preserve the topical similarity of tweets, as the model was specifically trained on semantic similarity-related tasks, but it may potentially struggle in separating user viewpoints. On the other hand, we expect that the embeddings produced by CT-BERT will better match the data domain. We emphasize that there is no risk of train-test data contamination for CT-BERT as its data collection period is different from that of the Avax dataset. Specifically, the CT-BERT model was trained on tweets from January 12th to July 5th, 2020, while the Avax data starts on October 18th, 2020. Given that the time span of the CT-BERT training set was in the early phase of the pandemic, its in-domain discussion was likely less vaccine-focused than the Avax dataset.

Topic-aware contrastive model

While the pre-trained embedding representations successfully encode the topic similarity of tweets, the embeddings resulting from fine-tuning on the social proximity task might not be able to preserve this information. In fact, due to the nature of the task, the fine-tuned embeddings will most likely encode information from underlying social processes present in the network (e.g., polarization) and might disturb the overall latent topic space.

To address this issue, we experiment with an approach that consists of concatenating the initial pre-trained embedding representations (topic vector) with an extra n-dimensional vector. This new n-dimensional vector is learned by fine-tuning the embeddings of the pre-trained model and then passing them through a linear layer which reduces the vector to a size of n. The concatenation of the frozen pre-trained embedding and the learned extra dimensions is then used in the contrastive loss function. This approach aims to preserve the topic similarity of tweets in the latent space by using the frozen embedding while also encoding additional information related to shared viewpoints through the additional dimensions.

Evaluation metrics

We evaluate the performance of the fine-tuned contrastive models on three different metrics: The first metric involves measuring the correlation between the content similarity of tweets and the social proximity of users. The other two metrics, ViewpointNN and TopicNN, are used to assess the models’ performance on the task of clustering tweets with similar viewpoints and topics.

Correlation We compute the Pearson, Spearman, and Kendall Tau correlation scores between the content similarity of tweet pairs, given by the cosine similarity of their text embeddings, and the social proximity of the authors of the tweets, given by their shortest path length in the retweet networks. The correlation metric does not measure the model’s ability to encode viewpoints, but indicates the performance on the proxy task corresponding to weak social signals. A higher correlation score indicates that the latent space has encoded more social proximity information into its semantic representation.

ViewpointNN To probe the ability of the embeddings to encode viewpoint towards a particular topic, we used the idea of k nearest neighbors (k-NN). Specifically, for each tweet in the manually annotated evaluation set, we find its k nearest neighbors in the embedding space of all tweets from the same topic (i.e., tweets belonging to a different topic are not considered). We then predict the viewpoint class (e.g. Support or Oppose) by majority voting. The ViewpointNN metric is the proportion of correctly classified tweets among all the tweets in the annotated set. This metric will be higher if the learned latent space does a better job of separating the labeled data by viewpoint.

TopicNN To evaluate the ability of the embeddings to capture a shared topic, we also leverage k-NN using the manually annotated data. In this case, for each tweet, we find the k nearest neighbors in the embedding space of all other tweets in the annotated set (i.e., we consider all topics), and predict the topic class by majority voting. The TopicNN metric is the proportion of correctly classified tweets among all tweets in the annotated set. This metric will be higher if the learned latent space does a better job of separating the labeled data by topic.

Data splitting for model training

To create the train, validation, and test data splits for the contrastive models, we sample 70% of users for training and 15% each for validation and testing using a stratified approach to maintain the distribution of topics across splits. For each split, we sample user pairs and compute their shortest path length in the corresponding retweet network. We transform the shortest path length values into a two-class representation: (1) user pairs in close proximity and (2) user pairs in distant proximity. We rely on a two-stage sampling approach to generate candidate pairs. First, we use a snowball sampling strategy to sample close connections by exploring the 1-hop and 2-hop neighborhoods of a random set of users. Second, we incrementally add more examples by randomly selecting two users, computing their shortest path length, and retaining them if their distance is larger than 2. Our sampling approach also has the benefit of mitigating the bias in the dataset towards prolific users. Our dataset, as in most social media sets, exhibits a significantly skewed distribution of tweets per user, with the most active 10% of users making 42.8% of the tweets. However, because we sample pairs of users for training without regard to their number of tweets, we will not be overly biased towards the more active users, mitigating concerns that the results will be only reflective of a vocal minority.

Furthermore, we augment the number of close pairs in each split by adding pairs of tweets authored by the same user, which are likely to reflect similar viewpoints. We select pairs of tweets from each pair of users considering only tweets from the same topic to encourage the model to learn differing viewpoints towards the same topic. We use a shortest path length cutoff value of 4, which means that path lengths in the range of 1-3 are considered close while values of 4 or greater are distant. We found that models trained using a cutoff of 4 yielded better performance on average (as shown in Fig 3). To address the potential data imbalance arising from the chosen cutoff, we also down-sample the majority class to achieve more balanced class distributions in the resulting data splits.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. ViewpointNN vs. TopicNN performance with K = 5 of contrastive models and the two baselines.

We include performance for the following: three models trained on the active retweet network with different cut-off values of shortest path length (4, 3 and 2), four models trained with different pre-trained embeddings on the original network, and the best-performing model trained on the strong network. FT refers to fully fine-tuned models, and TA-FT refers to topic-aware models.

https://doi.org/10.1371/journal.pcsy.0000089.g003

Table 4 shows the summary statistics for each data split across the various types of retweet networks considered.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. The number of sample pairs, tweets and users for the training, validation, and test data splits per dataset.

https://doi.org/10.1371/journal.pcsy.0000089.t004

Model hyperparameters

For each contrastive model, we use the Adam optimizer with a learning rate of and weight decay of . Each model was trained for 200 epochs with a mini-batch size of 32. We used early stopping to prevent over-fitting by tracking the loss on the validation set. A dropout rate of 0.2 was used for each network. We set the margin of the cosine loss embedding function to zero. For the topic-aware contrastive models, we set the extra embedding dimensions of the shared viewpoint vector to 100.

Hardware and software resources

The experiments were conducted on a computer with a Dual Intel Broadwell E5-2620 v4 @ 2.10GHz CPU. Each node has 16 cores with 64GB of memory. The operating system is CentOS Linux version 7. For deep learning training, we use an NVIDIA A100 GPU with 40GB of memory. All code was implemented in Python 3.8, making use of various software libraries, including Pandas and NumPy for data exploration and preprocessing; PyTorch for model building and training; scikit-learn for model evaluation; and BERTopic, Top2Vec, UMAP, and HDBSCAN for clustering and topic extraction. Visualizations were generated using Seaborn and matplotlib.

Model performance

Table 5 shows the full set of metric results for all three social networks (Original, Active, and Strong), each starting embedding (DistilRoBERTa and CT-BERT), and each weak supervision method (fine-tuned and topic-aware fine-tuned). The ViewpointNN and TopicNN metrics of select models are also shown in Fig 3 to better highlight the performance trade-off between these two metrics.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Model performance over the three types of retweet networks.

We report performance on ViewpointNN and TopicNN metrics with k = 5 as well as their average (NN Mean), and the Pearson correlation metric. The Baseline performance for ViewpointNN and TopicNN metrics is the same across different retweet networks as the baselines do not leverage the social network information. The correlation metric is computed on the corresponding test split for each network type. FT refers to fine-tuned contrastive models, and TA-FT refers to topic-aware contrastive models. The best results for each metric is shown in bold.

https://doi.org/10.1371/journal.pcsy.0000089.t005

First, we evaluate our models on the proxy task of encoding the social proximity of users within the tweet embeddings. The contrastive training leads to consistent improvements relative to the starting embeddings in all correlation scores, which measure the similarity between text embeddings and the shortest path distance of their respective authors. Our fine-tuned embeddings (FT) have an average improvement of 0.05 in correlation and the topic-aware embeddings (TA-FT) have an average improvement of 0.10 in correlation relative to the starting embeddings. This suggests that our fine-tuned embeddings capture more informative signals for discerning social proximity based solely on the content of the tweets.

To assess the ability of our trained models in encoding text representations with viewpoint information, we evaluate the fine-tuned embeddings and the two baseline embeddings on the annotated set of tweets described in the data section. We observe an improvement in ViewpointNN performance with both fully fine-tuned models (FT) and topic-aware models (TA-FT) compared to their corresponding baselines. Specifically, when considering models trained on the original retweet network, the FT models have a performance improvement of 12.3% with DistilRoBERTa embeddings and 12.2% with CT-BERT embeddings. Similarly, the TA-FT models also present a performance improvement of 10.7% with DistilRoBERTa embeddings and 12.2% with CT-BERT embeddings.

The observed improvement in viewpoint performance is associated with a degradation in topic performance as measured by TopicNN. Specifically, the FT models show a decrease in performance of 19.4% compared to the DistilRoBERTa baseline and 39.8% compared to the CT-BERT baseline. The best performance on the TopicNN metric is achieved by the DistilRoBERTa baseline, which is expected given that its embeddings are trained on sentence-similarity tasks, allowing them to encode more robust semantic similarities. However, these baseline embeddings have the worst ViewpointNN performance.

The observed trade-off between viewpoint and topic performance is driven by the nature of our proxy task. For example, tweets from distinct topics but likely similar viewpoints may be projected into similar regions in the embedding space, which inevitably leads to a reduction in topic performance. TA-FT models can mitigate this problem as they achieve a considerably smaller degradation in TopicNN compared to FT models. Specifically, the TA-FT models show a decrease in TopicNN performance of only 8.1% compared to the DistilRoBERTa baseline and 17.0% compared to the CT-BERT baseline while achieving an improvement on ViewpointNN performance.

The domain-specific pre-training of CT-BERT improves the encoding of viewpoint embeddings by 26% relative to starting from the DistilRoBERTa embeddings. This highlights the importance of leveraging domain-specific embeddings over out-of-domain embeddings, as the former contains more relevant contextual information and vocabulary tailored to the specific domain of interest. In comparing the performance across the three different networks, we found that the model trained with the retweet network of strong relationships demonstrated the best performance with an improvement of 12.2% in ViewpointNN and a reduction of 8.0% in TopicNN compared to the CT-BERT baseline. This can be attributed to its consideration of less noisy interactions, thus avoiding spurious interactions by chance.

We also investigate the impact of different values of k, which controls the number of neighbors to be considered for the ViewpointNN and TopicNN metrics. Fig 4 shows the performance for TA-FT CT-BERT models in ViewpointNN and TopicNN at different values of k. We present results for five models: two baselines, each using a different pre-trained embedding model, and three models fine-tuned using our topic-aware approach on data derived from three distinct types of retweet networks. The latter three models use CT-BERT embeddings as their initialization. We observed that increasing k has little effect on ViewpointNN, which indicates that our embeddings are robust in capturing different viewpoints even when varying the number of nearest neighbors. However, for TopicNN, larger k values result in tweets of different topics being included in the neighboring examples, which leads to a decrease in topic performance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Model performance in ViewpointNN and TopicNN at different values of k for topic-aware contrastive models trained with CT-BERT embeddings.

TA-FT refers to models leveraging both fine-tuned embeddings and static topic-aware embeddings. Models shown here are trained using the original, active, and strong retweet networks. (a) ViewpointNN. (b) TopicNN.

https://doi.org/10.1371/journal.pcsy.0000089.g004

So far, we have demonstrated the improved performance on viewpoint encodings using our method on an aggregate basis across our annotated data. Next, we explore the performance within individual topics to understand whether the improvements are attributable to specific subsets or appear to generalize across topics. Fig 5 shows the ViewpointNN performance of our best-performing model (TA-FT CT-BERT Strong) and the two baselines across each topic in the annotated set of tweets. We find that our model outperforms the two baselines across all topics, showing that the approach learns improved representations across multiple topic areas within this case study.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. ViewpointNN performance with k=15 of the best contrastive model (TA-FT Strong) and baselines across the different topics in the annotated set of tweets.

https://doi.org/10.1371/journal.pcsy.0000089.g005

Latent space analysis

We visualize the latent space of the annotated set of tweets by using the UMAP algorithm to project the embeddings for each tweet to a 2D space. Fig 6a shows the latent space embedding of the model that achieves the best trade-off between ViewpointNN and TopicNN performance, TA-FT CT-BERT Strong, which can be compared with the two baseline embeddings shown in Fig 2. DistilRoBERTa baseline embeddings are successful at encoding tweets with similar topics in similar regions of the latent space but fail to distinguish between the binary viewpoints on each topic. CT-BERT baseline embeddings can also identify semantically similar groups of tweets, though the resulting clusters are not as cleanly separated. Interestingly, the CT-BERT baseline is able to capture some partial viewpoint information within some of the topics. The embeddings generated by our topic-aware fine-tuned CT-BERT model demonstrate that tweets sharing similar viewpoints tend to group together, while those with opposite viewpoints are pushed to distant regions of the latent space, showing a binary pattern that likely reflects the polarization of the social network.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. (Left) UMAP visualizations of the embeddings from the best performing topic-aware fine-tuned model trained with the strong retweet network.

(Right) UMAP visualization of annotated set of tweets with their corresponding node2vec embeddings. (a) TA-FT CT-BERT Strong. (b) Node2Vec.

https://doi.org/10.1371/journal.pcsy.0000089.g006

To further demonstrate the ability of the fine-tuned embeddings in capturing narrative viewpoints, we visualize the latent space of three out of the six topics as identified in Table 1: depopulation agenda (Conspiracy), miscarriage agenda (Conspiracy), and vaccines (General). Fig 7 shows the UMAP visualizations per topic for two baselines and the best-performing fine-tuned embeddings. We observed that DistilRoBERTa embeddings struggle the most in distinguishing opposing viewpoints towards a particular topic as there is no clear separation between tweets. The CT-BERT baseline seems to partially capture viewpoint information as shown by visualizations on the depopulation and vaccines topics, but still struggles in other topics such as the miscarriage agenda topic. Overall, the CT-BERT TA-FT embeddings provide the best performance in capturing viewpoint information across multiple topics by visually demonstrating the ability to separate most tweets by their stance in each topic. These improved separations occur even though the model has not been trained directly on the viewpoint tasks but instead has inferred the new clustering based on indirect training on social proximity information.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. UMAP visualizations of baseline embeddings and the embeddings from the best performing topic-aware fine-tuned model across different annotated topics.

The TA-FT CT-BERT model shown uses the strong network for training. (a) Base DistilRoBERTa (Depopulation Agenda). (b) Base CT-BERT (Depopulation Agenda). (c) TA-FT CT-BERT (Depopulation Agenda). (d) Base DistilRoBERTa (Miscarriage Agenda). (e) Base CT-BERT (Miscarriage Agenda). (f) TA-FT CT-BERT (Miscarriage Agenda). (g) Base DistilRoBERTa (Vaccines). (h) Base CT-BERT (Vaccines). (i) TA-FT CT-BERT (Vaccines).

https://doi.org/10.1371/journal.pcsy.0000089.g007

Topic modeling analysis

Next, we qualitatively explore the clustering behavior of the embedding spaces and aim to determine whether the viewpoint-infused embeddings support improved performance on downstream tasks such as topic modeling. We employed BERTopic [18], a topic modeling technique, on the text representations generated by our best-performing model and the two baselines. BERTopic uses UMAP for dimensionality reduction and HDBSCAN for document clustering. It also offers the flexibility to incorporate custom embeddings. The algorithm assigns documents to a specific topic, and if a document is not representative enough, it is classified as noise. In particular, BERTopic identified 7 topics using DistilRoBERTa baseline embeddings with 20 tweets as noise, 9 topics using CT-BERT baseline embeddings with 40 tweets as noise, and 11 topics using the TA-FT CT-BERT Strong embeddings with 96 tweets as noise.

The extraction of meaningful narratives relies on the ability of each cluster to encompass tweets that share both an underlying topic area and corresponding viewpoints. Fig 8 shows the distribution of annotated topic categories across each topic identified by the BERTopic algorithm. The annotated topic categories are color-coded based on two groups of viewpoints that are more likely to be associated based on the polarization observed in the social network. This association can be seen in Fig 6b, where we projected the annotated tweets with their corresponding Node2Vec embeddings into a latent space. DistilRoBERTa embeddings generate topics that are roughly evenly split between opposing viewpoints on specific themes, highlighting its strong topic clustering performance and weak viewpoint clustering performance. CT-BERT embeddings exhibit similar results to DistilRoBERTa, with exceptions in topics 3 and 6, where the majority of tweets express vaccine opposition and face masks support, respectively. On the other hand, while our fine-tuned embeddings do not entirely preserve the similarity of tweets sharing the same topic, they effectively group tweets into informative regions of the latent space with shared or associated viewpoints. Specifically, topics 4, 5, 6, 7, and 9 are composed primarily of a single topic and viewpoint, while topics 1, 2, and 8 comprise viewpoints toward multiple topics that tend to be associated. Overall, we found that the majority of the detected topics clearly focus on a similar viewpoint towards one or multiple subject areas.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Stacked bar plot showing the distribution of topic categories across BERTopic-identified topics.

The y-axis displays the BERTopic topics, while the x-axis represents the number of tweets belonging to the 12 annotated categories, which are listed in the legend. Viewpoints are shaded blue or red based based on two groups of viewpoints more likely to be associated based on the social network polarization. (a) Baseline DistilRoBERTa. (b) Baseline CT-BERT. (c) TA-FT CT-BERT Strong.

https://doi.org/10.1371/journal.pcsy.0000089.g008