2.1 Network embeddings

In their seminal paper, Mikolov et al. [16] introduced Word2vec, a neural approach for learning low-dimensional representations of word meaning based on the neighboring words observed in very large amounts of text. Unlike traditional approaches of distributional semantics, Word2Vec is highly scalable, and has been shown to effectively project word semantics onto a compact space of several hundreds of dimensions [20].

The success of Word2Vec has inspired many related works, including in the networks domain. The model of DeepWalk [21] learns latent representation of vertices in a network, using a similar architecture to Word2Vec. Rather than model word co-occurrences within local text sequences, the DeepWalk algorithm samples node sequences from the network via a random walk process, where it is assumed that proximate nodes in the sampled sequences are closely related. The Node2Vec method [22] further generalizes this approach, employing a biased random walk procedure to introduce flexibility in the way a node’s neighborhood is defined. Both methods explore the whole graph to learn node embeddings. Accordingly, these methods were applied to relatively small graphs of up to 1M vertices [21, 22]. In this work, we wish to describe popular entities in terms of other entities with which they co-occur on social media. The entities of interest correspond to a small fraction of a very large social network that contains many millions of nodes [23]. Applying transductive algorithms such as DeepWalk would be prohibitively inefficient and practically infeasible for our purposes. Instead, we rely on a sample of users and the entity accounts that they follow. This setting lends itself to a different formulation of node neighborhoods that is efficient and scalable, where we only model relevant co-occurrence statistics among the entities of interest.

Previously, a similar approach was proposed for learning item embeddings from user-item rating history for recommendation purposes [24]. In that work, the embeddings of music items were computed, considering other music items liked by the same users as relevant contexts. Recommendation was then performed based on item similarity in the embedding space. In their experiments, the learned item embeddings were shown to outperform SVD, especially when the rating matrix was sparse.

2.2 Entity embeddings

Researchers have previously induced low-dimensional entity representations from factual information sources. Below, we describe in detail two popular and high-performing entity embedding schemes. In our experiments, we will compare our learned social entity embeddings with these entity representations.

2.3 Social inference tasks

In the lack of a common resource of social world knowledge, socially-oriented tasks are addressed ad-hoc, using data and methods that are specific to the problems in question. In this work, we demonstrate how two different tasks can be formulated and processed using our social entity embeddings, showing competitive or preferable results to alternative task-tailored solutions. Below, we introduce these tasks, and the main approaches using which they were addressed in related research.

3.1 Identifying entities of interest from the social network

Users on social networks typically associate themselves with–aka, follow–other accounts of interest, to regularly consume the content distributed by them. These associations correspond to a directed graph structure in which vertices denote user accounts and edges represent follower-to-followee relationships. Naturally, a small number of accounts are broadly followed, whereas the vast majority of the users form a long-tail of accounts that are followed by small social circles. We identify the most popular accounts in the social network, as indicated by their number of followers, considering them to be entities of general interest. To obtain the required statistics, we perform the following steps:

1. We sample a large number of Twitter users U uniformly at random.
2. For each user u_i ∈ U, we obtain the set of accounts that they follow, {a_i ∣ u_i → a_i}.

Let A denote the union of all the user accounts that are followed by some user in our sampled sub-network g, A = ∪_i{a_i}. We assess the popularity of each account a_t ∈ A according to the number of users who follow them, f(a_t) = ∣{i ∣ a_t ∈ {a_i}}∣. Finally, we define entities as the subset of the most popular accounts, E ⊂ A, practically setting a threshold K over the number of their followers in g, i.e., a_t ∈ E if f(a_t) > K.

3.2 Social context modeling

The neural algorithm of Word2Vec builds on the theory of distributional hypothesis in linguistics, by which words that are used in the same contexts tend to have similar meaning [54]. Accordingly, it learns word embeddings that are predictive of neighboring words as observed in large amounts of text. Here, our goal is to learn entity embeddings of social meaning. Similar to local word co-occurrences that demonstrate linguistic and topical regularities, we seek to capture social entity semantics from entity co-occurrences that denote meaningful social contexts.

A key observation that underlies our approach is that the set of entities that are co-followed by an individual user form a coherent unit of social context. Evidently, the links established by a user reflect their personal preferences, interests, and their socio-demographic characteristics. It is well established, for example, that individuals tend to follow news sources with similar political orientation to their own [8, 40]. As we demonstrate later in this paper, the entities that one follows are also correlated with their gender, age group, education level, ethnicity, and more (Sec. 7.3). Thus, similar to word sequences which form linguistically and topically related contexts, the sets of entities followed by individual users demonstrate social and topical inter-relatedness. Accordingly, our objective in learning the entity embeddings would be to predict, for each user and entity pair, the additional entities that are co-followed by the same user. Provided with a large sample of users and the entities that they follow, we expect the neural embedding model to learn meaningful dimensions of social knowledge.

Fig 1 summarizes our approach of data sampling and modeling. As illustrated in the figure, once information about the entities followed by the sampled users is obtained, we discard information about the identities of those users. We further remove information about accounts followed by the sampled users, which are not defined as entities. Notably, entity accounts correspond to a very small portion of the accounts followed, i.e., E << A. By that, we diverge from graph embedding approaches like DeepWalk that learn embeddings for all of the nodes in the source graph. Our framework is therefore highly efficient in that it focuses on inter-entity contextual evidence. In Section 4, we detail relevant data statistics observed in a large subgraph sampled from Twitter in practice.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

A summary of our social context modeling approach: (a) Given a graph of sampled users and the accounts that they follow, we identify accounts of high in-degree, assuming that they represent entities of general interest; the figure illustrates a popular user as a blue figure, and unpopular user–in grey. (b–c) We consider the sets of accounts followed by each sampled user (in red) to be socially related. (d) We focus on entity co-occurrences within these sets, where we discard the sampled users identity, and avoid the modeling of unpopular accounts.

https://doi.org/10.1371/journal.pone.0283700.g001

3.3 Learning of social entity embeddings

Next, we describe the framework for learning the entity embeddings from the sampled context data. Our approach follows closely on the Word2Vec algorithm. A main difference is that we model large unordered sets of co-occurring entities as context, as opposed to local word neighborhoods that are modeled by Word2Vec.

For clarity, we hereby briefly outline the Word2Vec approach, focusing on the popular skip-gram with negative sampling (SGNS) model [16]. Given unlabeled text corpora, comprised of a sequence of words , SGNS learns to predict, for each word w_i in turn, the neighboring words that surround it within a window of a fixed size c. The loss function of the model is defined as: (1) where the conditional probability P(w_c∣w_i) is defined using the softmax function: (2) where u_i ∈ R^d and v_i ∈ R^d are latent vectors that denote the target and context representations of a given word in the vocabulary, w_i ∈ W. respectively. It is overly costly to compute Eq 2 with respect to the whole vocabulary W however. Negative sampling alleviates this computational burden, replacing the softmax function with: (3) where σ = (1 + exp(−x))⁻¹, and N is a parameter. Thus the task becomes to distinguish the target word w_i from a noise distribution, considering N negative samples for each data sample. The negative examples are typically drawn from the unigram word distribution, raised to the 3/4rd power, so as to improve the representation of infrequent words [55]. Finally, training is performed by minimizing the loss function using stochastic gradient descent.

Following our definition of inter-entity relatedness, for each user-entity pair, we with to predict the other entity accounts that are followed by the same user. Accordingly, we define our loss function as follows: (4) where e_i and e_j are entity pairs that belong to E_i, denoting the set of entities that are followed by user u_i. Notably, we attribute equal importance to all of the entities that co-occur within the set E_i, modeling the correspondences between all of the respective entity pairs. By that, we take full advantage of our reference data. Alternatively, one may opt for a stochastic approach, modeling relatedness between sampled entity pairs [24].

In addition to the skip-gram neural model, we consider the CBOW model variant of Word2Vec [16]. Using CBOW, the goal is to predict each focus word w_i in turn given the aggregate representation (sum of embeddings) of its neighboring words. Similarly, we aim to each focus entity e_i from the aggregate representation of all of the other entities that are followed by the same user. Due to the aggregation operation, CBOW is substantially faster to train compared with SGNS. Performance-wise, SGNS has been found to give comparable results on syntactic tasks, and preferable results to CBOW on tasks that model semantic similarity between words [16, 56]. We considered both variants and compare between them in our experiments.

3.4 Ethics statement

The proposed framework relies on public social network information. As described in Sec. 4, we obtained relevant network information for sampled users via a public Twitter API under the terms of Twitter developer account. Our automated data processing framework discards user identities (Sec. 3.2), and projects the large-scale anonymized user data onto a low-dimensional space (Sec. 3.3). For these reasons, personal information cannot be recovered from the generated entity embeddings, which rather reflect global social patterns. The data which we obtained for the purpose of this research may be recovered by other researchers using the same procedure and under the same terms. Details about our user sample, as well as our code used for extracting relevant network information and learning the entity embeddings, are available at github.com/nirlotan/SocialVecTraining.

4.1 Account popularity statistics

First, let us examine the distribution of accounts by their popularity, and discuss the procedure of defining popular accounts as entities. Fig 2 shows the distribution of account popularity as measured by the number of users in our sample who follow each account. As shown, a small number of accounts (1.4K) are followed by more than 25,000 users each, i.e., by ∼2% of the users in our dataset. Roughly ∼5.5K accounts are followed by 10K users or more, i.e., by more than 0.8% of the users. As expected, the vast majority of accounts form a long tail of the popularity distribution, being followed by less than 1K users. Learning the embeddings of all the accounts would be challenging computationally as well as quality-wise, considering that sufficient contextual information is needed for learning meaningful semantic representations. Furthermore, we wish to focus on widely-known accounts for our purpose of social knowledge modeling. We therefore set a threshold over account popularity, considering the accounts that are followed by at least k = 350 users (∼0.03% of the users in our sample) as entities. There are roughly 200K accounts (201,247) which meet this condition and comprise our vocabulary of entities, E.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Account popularity based on our sample of 1.3 million Twitter users and the accounts that they follow: Number of accounts vs. number of their followers (log-log scale, where the number of accounts is illustrated by point size).

There are 90 million unique accounts with at least one follower in the sampled data. A long tail of accounts are followed by up to 100 users (top-left part of the plot). We learn the embeddings of the most popular ∼200K accounts, which have 350 followers or more.

https://doi.org/10.1371/journal.pone.0283700.g002

A question of interest is, to what extent does our Twitter-based vocabulary of entities E represent entities that are indeed of general interest? To answer this question, we assess the extent to which the identified entities E are included in existing sources of world knowledge. Admittedly, knowledge bases are incomplete in their coverage. Nevertheless, the inclusion of an entity by other sources of world knowledge may be considered as an indication of public interest in that entity.

Exploiting the fact that the collaboratively-managed Wikidata links entities with their respective page on Wikipedia as well as with their account handle in Twitter, we aligned the entities E whenever applicable with Wikidata and Wikipedia. (See Wikidata property: P2002, and twitter user numeric id property P6552. We used Wikidata’s SPARQL query service available at https://pypi.org/project/qwikidata/ to retrieve this information.) Overall, we found that 31.5K out of the 200K accounts that comprise E (15.8%) have a Wikipedia page or Wikidata entry associated with them. Naturally, there is a correlation between account popularity on Twitter and knowledge base coverage. Among the 10K most popular accounts in E, more than 50% are included in Wikidata, where this ratio goes down to 5% for the 10K entities that are least popular. Considering that the coverage of the reference knowledge bases and the alignment to these sources are imperfect, these figures form a conservative and encouraging assessment of entity relevance. Notably, there are differences between formal knowledge bases and social networks like Twitter with respect to representation scope. In manual examination, we observed that some popular accounts in Twitter concern professional topics or hobbies; these accounts carry social information but are of low relevance to factual knowledge bases. On the other hand, knowledge bases like Wikipedia include many abstract, scientific, and historical concepts, which are not present in social networks.

4.2 Context statistics

Table 1 includes statistics about the number of accounts followed by individual users in our dataset. As shown, users follow almost 1,000 other accounts on average, where the median number of accounts that a user follows is 526. As shown in the table, out of those, users follow 228 popular accounts that included in our entity vocabulary E on average, where the median number of entities that a user follows is 108. As further shown, more than half of the entities followed by the users have a respective entry in Wikipedia. This means that our approach models as context entities that are well-known alongside entities that are mainly popular on social media.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Statistics regarding the number of accounts followed by individual users in our sampled network, considering: All accounts, popular accounts which we consider as entities, and entity accounts for which a respective entry in Wikipedia has been found.

https://doi.org/10.1371/journal.pone.0283700.t001

4.3 Learning setup

To learn the entity embeddings, we adapted the Word2vec algorithm as implemented in gensim [58]. Importantly, we set the context size parameter to a large value, c = 1000. Since most users follow a substantially smaller number of entities (228 on average; see Table 1), this means that the model is practically trained to predict all of the pairwise entity co-occurrences within the context of every user (Eq 4). We applied common parameter choices in training the model, setting the initial learning rate to 0.03, and having it decrease gradually to a minimum value of 0.0007. Considering the large context size on one hand, and computational requirements on the other hand, we increased the number of negative examples to N = 20, which we selected randomly from the probability distribution of the entities followed in our data, applying downsampling by a factor of 1e-5. Similar to Wikipedia2Vec entity embeddings, we set the size of the social entity embeddings to 100 dimensions. In our experiments, we created and evaluated embeddings of different sizes based on cross-validation performance on the task of personal trait prediction (Section 7), and found that 100-dimensions provide comparable performance to higher-dimension representations.

The training of the models was conducted using an Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz computer with 24 CPUs, with 128GB RAM and an NVIDIA Corporation GV100 [TITAN V] (rev a1) GPU. Generating the entity embeddings using the SGNS model required about five days in running time. In addition, we trained social embeddings using the CBOW configuration, which was about five times slower.

5.1 Evaluation methodology

5.2 Exploratory evaluation of entity similarity

To illustrate the learned social semantics, we consider several example entities, exploring other entities in their vicinity. Table 2 lists the top (5) entities that were found to be closest in terms of cosine similarity to the entities of Pfizer, Princeton, X-Men, Star Trek and Hillary Clinton. In addition to SocialVec, the table also details the most similar entities to each query entity using Wikipedia2Vec and Wikidata RBG embeddings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The most similar entities to exemplary query entities, as ranked using cosine similarity by the different entity embedding methods.

https://doi.org/10.1371/journal.pone.0283700.t002

We observe that SocialVec models semantic similarity. For example, the closest entities to Pfizer include other pharmaceutical and Biotechnology companies, including Amgen, Novartis, Sanofi, and AstraZeneca. The account of ‘Pharmaguy’ is topically-related–it is a newsletter on pharmaceutical marketing. Likewise, the entities that are most similar to Princeton University include other prestigious universities, most of which are also members of the Ivy league. And, the most similar entities to X-Men are other movies that depict fictional superheroes by Marvel. The most related entities to Star Trek, which is defined by Wikipedia as a pop-culture franchise, include actors in the Star Trek movie series. (Dan Aykord is rather known for playing a role in a satire sketch of Star Trek.) These results suggest that modeling user preferences on social media is useful for eliciting social and cultural, as well as semantic inter-entity similarity. Lastly, considering Hillary Clinton as the focus entity well-demonstrates the encoding of political social knowledge within SocialVec. The most related entities in this case include the Democratic former first lady Michelle Obama, as well as the Democratic Senators Elizabeth Warren and Cory Brooker. In addition, Clinton is found similar in the embedding space to the Planned Parenthood organization and Jim Acosta, a CNN journalist. These accounts, which span over politicians, organizations, and the media, all belong to the Democratic political camp.

Considering the top entities retrieved using Wikipedia2Vec, we observe somewhat different semantics. Similarly to SocialVec, Wikipedia2Vec places Pfizer next to pharmaceutical and Biotechnology companies, and X-Men–next to other superheroes movies by Marvel. But, the most related entities to ‘Star Trek’ include movies and some fictional characters (e.g., ‘James Kirk’) that are not well-represented on Twitter. Further, the closest entities to Princeton University according to Wikipedia2Vec include the city of Princeton, as well as the Institute for Advanced Studies and Nassau Hall building. Indeed, these entities are affiliated with Princeton University, and are directly linked with the university page on Wikipedia. Yet, we find that these responses fail to capture the social perception of Princeton as a prestigious university. For Hillary Clinton, Wikipedia2Vec assigns top similarity to other Presidential candidates, as well as the page of ‘2016 United States presidential election’. This notion of similarity fails to capture political affinity, notably placing the Republican Trump in close vicinity to Clinton. In the case of Wikidata RBG graph embeddings, we find some bias towards functional similarity. For example, Cisco and Bank of America are placed close to Pfizer, despite being companies in different sectors. Or, President Adams is highly similar to Hillary Clinton who ran for Presidency, despite being a historical figure. Overall, we find that unlike these existing entity representation schemes, SocialVec reflects popular social knowledge and conception of entities.

6.1 Method

Let us recall that our entity embeddings were learned from context information comprised of popular accounts that are co-followed by users on social media. Presumably, individual users follow the accounts of media sources, politicians, and other entities with similar political orientation to their own. It is therefore expected that entities that are associated with the same political camp demonstrate high similarity in the embedding space, as opposed to entities of opposing camps. Accordingly, we compute the bias of news accounts based on their similarity to popular anchor accounts that represent the two political poles. Specifically, we consider the accounts of Barack Obama, the former Democratic U.S. president and the incumbent president at the time that our data was collected, the Republican Donald Trump. As of 2020, these accounts were ranked as first and fourth most popular Twitter accounts based on the number of their followers. (Trump’s account was suspended in January 2021).

Let us denote the embedding of a specified news account as e_n, and the embeddings of the Democratic and Republican anchor accounts, which we set to the accounts of Obama and Trump, as e_D and e_R, respectively. We first measure the similarity of the news source in the embedding space with respect to these Republican and Democratic anchors, and then assess the political orientation (PO) of account e_n as the difference between the two similarity scores: (5) A positive score indicates on overall conservative (Republican) orientation, and a negative score–on a liberal (Democratic) bias. A greater gap between the similarity scores suggests there is a greater political bias of entity e_n.

In our experiments, we rank specified news accounts according to their computed political orientation scores, assessing our results against formal polls. We experiment with SocialVec entity embeddings, as well as with the alternative embeddings schemes. The success of each entity representation method on this task reflects the extent to which it captures the social phenomena of political leaning.

6.2 Ground-truth data

Two polls were conducted by Pew Research in 2014 and 2019 with the goal of gauging the political polarization in the American public [19]. The participants in the polls were recruited using random sampling of residential addresses, and the data was weighted to match the U.S. adult population by gender, race, ethnicity, education and other categories. In both polls, Pew researchers classified the audience of selected popular news media outlets based on a ten question survey covering a range of issues like homosexuality, immigration, economic policy, and the role of government. The media sources were then ranked based on the party identification (Republican or Democrat) and ideology (conservative, moderate or liberal) of the survey participants.

Overall, the polls conducted in 2014 and 2019 include 36 and 30 news media outlets, respectively. The union of these two sets comprises 43 unique media outlets. We were able to identify the Twitter handles of most of the news sources included in the earlier poll (31/36), and practically all of the news sources included in the more recent poll (30/30). As detailed in Table 3, all of these media accounts are encoded as entities in SocialVec, where most albeit not all of the accounts are represented by Wikipedia2Vec and Wikidata RBG embeddings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results of assessing the political bias of news sources.

The table reports Spearman’s correlation of conservative-to-liberal ranking of selected news accounts generated based on different entity embeddings, compared with poll-based rankings reported by Pew Research in 2014 and 2020. The number of relevant account embeddings is given in parenthesis for each method.

https://doi.org/10.1371/journal.pone.0283700.t003

6.3 Results

6.3.1 Ranking-based evaluation.

The surveys by Pew Research assign a numerical score to each of the news sources that reflects their political polarity. In assessing our method, we therefore consider the relative ranking of the various news sources–ranging from conservative/Republican to liberal/Democrat. Table 3 reports the alignment between the poll-based rankings and the rankings generated according to Eq 5 using different entity embedding methods, measured in terms of Spearman’s correlation [59]. A perfect Spearman correlation of +1 indicates that the rankings are identical, whereas a correlation of -1 means that they are perfectly inverse. As shown in the table, the rankings produced using SocialVec closely match the ground-truth rankings, yielding high Spearman’s correlation scores of 0.82 and 0.85 per the two polls. In contrast, the rankings produced by Wikipedia2Vec yield low scores of 0.36 and 0.28. The rankings generated using graph-based Wikidata embeddings are not meaningful altogether, yielding negative correlation scores.

Fig 3 illustrates the distribution of political orientation scores of news sources included in the poll of 2020 as computed using the SocialVec embeddings. The accounts are spaced along the range of Democratic (left) to Republican (right) according to their scores. Similar to the reference poll results, we observe that some news sources lie very close to each other on this scale of political bias. This means that using the measure of Pearson’s correlation may be overly sensitive, as it penalizes any difference in the ordering of accounts regardless of the difference between their scores.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Ranking of political polarity based on our embeddings.

https://doi.org/10.1371/journal.pone.0283700.g003

6.3.2 Binary polarity prediction.

We report the results of a more lenient evaluation In Table 4, measuring the ratio of news accounts for which the polarity is correctly estimated. In this mode, the computed political orientation score is expected to be positive if the news source is considered to be conservative/Republican according to the reference poll, and vice versa (Eq 5). As shown in the table, the political orientation predicted using the social entity embeddings is accurate in almost all cases: prediction accuracy is 94% and 97% per the polls of 2014 and 2020, respectively. In each case, there was a single mistake in predicting the binary political bias. In error analysis, we found that the two faulty predictions apply to news accounts for which the number of contexts (i.e., followers) in our sample of Twitter was the lowest among the accounts included in each poll, reinforcing the fact that sufficient context information is required for learning reliable low-dimension representations. In comparison, the Wikipedia2Vec embeddings yield low accuracy scores of 55% and 60%, and the Wikidata graph embeddings yield poor accuracy of 32% and 27% per the two polls. As mentioned above, some of the news outlets were not included in Wikipedia of Wikidata. To account for differences in coverage, Table 4 also reports prediction accuracy for the subset of accounts that are represented by all methods (‘common’). As shown, this evaluation mode shows the same trends.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Results of assessing the political leaning of news sources as binary polarity.

Prediction accuracy is reported for all the accounts available per method (‘all’), and for the news accounts that have embedding in all methods (‘common’).

https://doi.org/10.1371/journal.pone.0283700.t004

Based on these results, we conclude that information about the political orientation of Twitter entities is well encoded in the social embedding space. In contrast, entity embeddings derived from factual sources fail to reflect political affinity. As reviewed in Sec. 2.3, related works devised specialized content analysis methods or collected and analyzed relevant network data ad-hoc for this purpose. Unlike those works, our approach is unsupervised and general in that we consider political bias as one aspect of social knowledge. Importantly, we believe that this approach may be employed for assessing additional types of social biases.

7.1 Dataset

We refer to a dataset of Twitter users labeled with respect to the following personal attributes: age, gender, ethnicity, family status, education, income level, and political orientation [48]. The user labels were obtained by means of crowd sourcing: human Workers were asked to make judgments about each user with regards to the specified properties based on public information in Twitter, including the user’s self-authored description, their meta-data and the historical tweets posted by the user. In order to simply labeling, real-valued attributes were discretised into binary ranges. Originally, the dataset included 5,000 users, having subsets of the dataset labeled per category according to the availability of trait-related user information. We tracked 3,558 active user profiles overall in Twitter out of the original user set as of October 2020. This means that our experimental dataset is substantially smaller in size compared with the original dataset. Furthermore, since we retrieved user information some time after the annotation process was conducted, the category assignments of some attributes (for example, age) might have changed. Yet, since the labels are coarse-grained and were obtained based on subjective human judgement, we believe that labeling accuracy is not severely compromised. We note that beyond the labels being subjective, the dataset may present a selection bias, e.g., due to varying levels of self-exposure on social media by different sub-populations. Yet, we evaluate multiple alternative methods using the same data, and in the same conditions, where this forms a viable evaluation setup. Table 5 details the label annotation scheme, and the number of labeled users that comprise our dataset per category.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Personal trait prediction: Dataset statistics.

https://doi.org/10.1371/journal.pone.0283700.t005

7.2 Methods

We perform supervised classification experiments, treating the prediction of the various attribute values as independent binary classification tasks. For each target attribute, we randomly split the set of labeled examples into distinct train (80%) and test (20%) sets, maintaining similar class proportions between these sets. Once the models are trained, prediction performance is evaluated against the gold labels of the test examples. Tuning was performed based on cross-validation performance using the training example sets.

7.2.1 Entity embeddings as features.

We obtained information about the accounts followed by the users in the dataset using Twitter API, associating each user with the SocialVec embeddings of the entities which they follow. In learning, we follow the common practice of averaging the values of the bag-of-entity-embeddings that describe each user into a unified summary vector of the same dimension [60]. We then feed the averaged vector representation to a logistic regression classification network, in which the output layer consists of a single sigmoid unit. While we experimented also with multi-layer network architectures, we found this single-layer classifier to work best.

Table 6 includes detailed statistics about the number of entity embeddings that are available per user in the dataset using multiple entity embedding schemes. As shown, SocialVec provides a substantially larger coverage of the entities followed compared with the other methods. Specifically, the median number of encoded entity embeddings that are associated with each user is 104 using SociaVec versus 27 and 19 using Wikidata2Vec and Wikidata RBG, respectively. Consequently, the ratio of users that are poorly represented, being associated with less than 10 entity embeddings, is 4% using our method, vs. 23% and 33% using the alternative methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Personal trait prediction: Statistics of the number of popular account embeddings that are associated with each user in the dataset using the different entity embedding methods, and the proportions of users in the dataset that have a limited number of embeddings (less than 5 or 10) associated with them using each method.

https://doi.org/10.1371/journal.pone.0283700.t006

7.3 Results

Table 7 details classification performance per each of the target attributes in terms of the ROC AUC measure [49]. The table includes the results using SocialVec entity embedding features, as well as the results using Wikipedia2Vec and Wikidata RBG entity embeddings. As shown, SocialVec embeddings outperformed the other entity representation schemes by a large margin across all of the target attributes. For example, classification performance on age prediction is 0.74 in terms of ROC AUC using SocialVec versus 0.69 and 0.61 using Wikidata’s and Wikipedia2Vec embeddings, respectively. On the target of ethnicity, ROC AUC using SocialVec embeddings is as high as 0.95 versus 0.86 and 0.70, and so forth.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Personal trait prediction results [ROC AUC].

*The results by Volkva et al. were obtained using a earlier version of the dataset which was substantially larger, and different tweets, and are therefore not directly comparable.

https://doi.org/10.1371/journal.pone.0283700.t007

In order to account for the gaps in coverage between the different embedding schemes (Table 6), we experimented with a restricted variant of SocialVec, where we discard the embeddings of entities that are not represented by any of the other methods from the user representation. As detailed in Table 6, this variant (‘SocialVec∩Wiki’) has similar coverage as the Wikipedia-based methods, with a median of 25 entity embeddings associated with individual users. As expected, classification performance using this limited feature set is lower, e.g., ROC AUC drops from 0.74 to 0.71 on the age category. Nevertheless, classification performance using the SocialVec embeddings remains superior on all of the target categories. Thus, we conclude that SocialVec entity embeddings are more informative compared with embeddings learned from factual knowledge bases on the social task of personal trait prediction.

7.3.1 Data analysis.

To inspect the social information that is encapsulated in user-entity interactions, we examine the entity accounts which users tend to follow with distinctively different probability across class labels in the dataset. Table 8 shows the most distinctive accounts per label as computed using the pointwise mutual information (PMI) measure [64]. Concretely, for each entity account e_i and target attribute C, the PMI between e_i and attribute value c_j is computed as , where Pr(c_j) is the proportion of users in the dataset that are labeled with attribute value c_j, Pr(e_i) is the proportion of users who follow entity e_i regardless of the attribute label, and Pr(e_i, c_j) is the joint probability of these events. In words, high PMI values indicate on distinctive as opposed to random entity-class co-occurrences.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. The top Twitter accounts that are characteristic to different subpopulations as measured using our datasets labeled with personal attributes and the Pointwise Mutual Information (PMI) measure.

https://doi.org/10.1371/journal.pone.0283700.t008

Table 8 illustrates various social phenomena that are encoded in platforms like Twitter. We observe, for example, that the entity accounts which characterize male users specialize in sports, and that female users most distinctively follow accounts that belong to women. Likewise, the top accounts that characterise Afro-American users all belong to Afro-Americans (and vice versa). Similarly, young people (below 25 years of age) are associated with music bands and singers born in the 90s, while older users follow older TV hosts and celebrities. (Due to high class imbalance, the PMI values were lower for the age category, and this information was omitted from the table.) We further find that users with an academic degree distinctively tend to follow media accounts such as the New-Yorker and the Economist magazines, whereas non-academic users tend to follow rappers. Finally, the entity accounts that are most distinctive of Republican users in the dataset are non-political entities that are yet known as Republican oriented, including an account of country music, and the accounts of Tim Tebow, a former professional football player, and the fast-food brand of Chick-fil-A, both of which are known for their conservative views; see “Tebowing” at https://en.wikipedia.org/wiki/Tim_Tebow, and https://en.wikipedia.org/wiki/Chick-fil-A: Same-sex marriage controversy. The accounts of the satirical Basrstool Sports are also identified as Republican oriented, echoing the findings of a recent poll; https://morningconsult.com/2020/07/24/barstool-sports-trump-interview-polling/. SocialVec entity embeddings encode these and other social patterns in an unsupervised fashion as observed from a large corpus.

8.1 Implications of this research

We believe that this research can enhance applications that concern world knowledge in general and social knowledge in particular. Hereby, we discuss some potential research directions, placing emphasis on the inter-disciplinary field of Computational Social Science, and the prospects of personalized and socially contextualized text processing.

8.2 Limitations

There are naturally some limitations to our approach. SocialVec embeddings are learned from the social network of Twitter, which has biases; for example, Twitter users are younger and more Democrat than the general public; (https://www.pewresearch.org/internet/2019/04/24/sizing-up-twitter-users). In addition, while public figures like politicians and music artists typically maintain Twitter accounts, some entity types, e.g., locations, are not well-represented in this platform. Furthermore, accounts may be banned from social networks like Twitter. (A well-known example is the suspension of the private account of former president Donald Trump in January 2021). Another inherent limitation of embedding methods like SocialVec, regardless of the underlying information source, is that high-quality embeddings requires sufficient context statistics. Finally, the learned SocialVec entity embeddings capture social knowledge at some fixed point in time, whereas social networks are dynamic, and new entities emerge over time. We believe that the core of social knowledge changes slowly; consider, for example the political biases of new sources. Nevertheless, our approach of learning entity embeddings from a sample of the social network is efficient and can be readily applied to learn entity embeddings at different points in time. Repeated sampling of social knowledge may allow the study of temporal changes of social entity representations.

8.3 Ethical considerations

In our experiments, we use SocialVec entity embeddings as features in predicting the personal traits of individual users. In general, the task of learning user profiles from their digital footprints is well studied. In order to protect user privacy in applying such techniques in practice, users should be informed and approve the use of automatic personalization methods. We note that our approach, like other data-driven prediction method, may result in stereotypical user profiling. There are ongoing efforts within the research community that aim to mitigate and raise awareness to potential biases of this sort in using and interpreting machine predictions. A more detailed discussion of the ethical aspects involved in characterising users for the purpose of improving natural language processing is included in several recent position papers [10, 67].

In this research, we processed entity embeddings from public network information of sampled Twitter users. We discarded the users’ identities, and processed the large-scale network information into a low-dimensional space. Thus, there are no traces of individual user information in the learned entity embeddings. The learned embeddings reflect general social contexts that characterise popular users, and do not pertain to any information that is associated with the entity accounts directly. We make SocialVec embeddings publicly available for research purposes, hoping to promote social knowledge modeling and exploration.