2.1 Fundamentals

The modeling of interactions among different users using network/graph representations is widespread in the literature. Yet, as previously shown [24, 28, 50–53], the variety of patterns emerging from such interactions and influencing the phenomenon under study can be quite large, including sequentiality, periodicity, and sporadicity. Moreover, to study a given phenomenon, it is often the case that one needs to look at large sets of interactions over a relatively long period of time. However, not all interactions are equally important for the study. In fact, it is often the case that many such interactions occur only sporadically or purely by chance and therefore have only a weak relation, or no relation at all, to the phenomenon under study.

As previously argued [24, 50, 51, 54, 55] it is unclear the extent to which the aforementioned diversity of interaction patterns affects the study of the phenomenon under consideration. For example, the presence of a large number of random, sporadic and thus weak edges in a network representation may indeed obfuscate those edges more closely related to the phenomenon [10, 12, 20, 56–58], here referred to as salient edges. As such, these weak edges represent noise to the study. This situation gives rise to the following question: What makes a particular edge relevant to the study of a given phenomenon?

As argued by Grady et al. [32], the definition of edge salience is based on an ensemble of node-specific perspectives in the network and quantifies the extent to which there is consensus among nodes regarding the importance (or representativeness) of a link. Thus, there are a variety of factors related to the phenomenon under study that define whether an edge is salient or not, making the identification of a salient edge quite challenging. Given a proper definition of edge salience, another challenge is how to implement and evaluate the identification and extraction of these edges from the original network. The set of salient edges is called the network backbone. Next, we discuss prior studies that deal with the use and design of backbone extraction algorithms.

2.2 Prior studies on network backbone extraction

Starting with applications, multiple works in various fields have shown the importance of backbone extraction methods to deal with random, sporadic, and weak edges that may obfuscate the phenomenon under study. For example, several studies applied early proposed methods to study phenomena in biological networks [59, 60], economic networks [61–63], co-authoring networks [64, 65], human mobility networks [66–68] as well as congressional voting networks [25, 69, 70].

More recently, some studies have highlighted the importance of this task in social media applications. For example, Pacheco et al. have proposed a network-based methodology for identifying coordinated actions in social media [20]. This methodology is based on (i) exploring different types of user interactions (e.g., users who used the same hashtag sequence or who retweeted the same tweet sequence) and (ii) applying a threshold-based approach to remove edges whose weights fall below a certain threshold. This simple threshold-based backbone extraction approach has been widely used in the literature [69–74], including studies on online hate communities [12] and communities with online news exposure [10]. However, some studies point to possible misinterpretation of results when using such approach, as they may introduce bias into the analysis [72]. To name a concrete case, the appropriate threshold setting is context-dependent and can therefore be quite complex to be determined. Sometimes it is not even clear whether the same threshold should be applied to all edges [75].

Another example of study is the analysis of information dissemination in publicly accessible groups on WhatsApp, a currently very popular communication platform. The authors of [21, 76] investigated a latent network structure built by connecting users who shared the same content. By applying a probabilistic backbone extraction method called Disparity Filter [77], they revealed the formation of communities of users that cross the boundaries of existing groups, favoring dissemination of information at large [76], including misinformation [21]. Similarly, different backbone extraction methods have been applied to uncover patterns describing the dynamics of online discussions on Instagram, Twitter, Facebook and Reddit [19, 22, 23, 56–58, 78, 79].

Moving to the body of works that focuses on the proposal of new backbone extraction methods, some authors compared methods to alternatives in light of specific phenomena of interest in various domains, such as transportation, finance, and ecology [31, 32, 35, 77]. Some works propose specific rules to the selection of salient edges, e.g., as in [80, 81] where salient edges are identified based on the idea of overlapping communities. A large number of previous studies has proposed and analyzed methods for extracting backbones in bipartite networks [37, 38, 82]. Considering a bipartite structure consisting of artifacts and agents, a family of methods, such as the Fixed Sequence Degree Method (FSDM), the Fixed Fill Method (FFM), the Fixed Row Method (FRM), the Fixed Column Method (FCM), and the Stochastic Degree Sequence Model (SDSM), identify salient edges by constraining the degree sequence of either artifacts (FCM), agents (FRM), both (FSDM and SDSM), or neither (FFM).

Most of these studies rely on structural/topological properties to evaluate different backbones extracted from the same network, including node and edge coverage, clustering coefficient, centrality measures, and community quality measures. As such, they offer only a partial view of the quality of the backbones. Contextual (i.e., phenomenon-specific) criteria, capturing the extent to which the extracted backbone represents the phenomenon under study, are not considered. More recently, some studies have proposed and compared backbone extraction methods based on regression-models as a means to capture contextual attributes specific to the phenomenon, relating them to topological properties of the backbone [33, 36, 47].

In this paper, we do not intend to introduce another backbone extraction method, but rather to propose a principled methodology to evaluate existing solutions for a target study. In a similar direction, Dai et al. [43] have evaluated six methods for extracting the salient edges from an air transportation network. As most prior studies, the authors considered only topological properties in such evaluation, which seems adequate given the interest in network connectivity (i.e., paths). Similarly, Mukerjee et al. [45] investigated the impact of method parameters on network connectivity. They proposed to choose the best method and its parameters based on topological properties, by maximizing the number of edges while maintaining the connectivity of the network. Other studies also evaluated existing methods for specific cases of bipartite networks, but once again considering only topological properties [24, 37, 82].

In contrast, our focus is on collective human behavior in networks emerging in social media applications, which are expected to have more nuanced aspects. Specifically, unlike other networks (such as transportation networks) that are driven by existing and slowly changing rules, our focus is on user behavior, notably on social networks, which is driven by a multitude of both endogenous (e.g., mechanisms employed by the platform) and exogenous factors (interactions on other platforms and in the real world). These aspects deserve to be investigated from both topological and contextual perspectives. We propose a principled methodology to select and evaluate the best method among alternatives for a given target phenomenon, taking into account whether the assumptions and requirements of each method fit adequately the properties of the phenomenon. In the following, we present a brief description of ten backbone extraction methods selected for our study.

2.3 Selected backbone extraction methods

In selecting the backbone methods, we consider those that have been proposed for weighted networks and applied to the analysis of collective behavior, often in projected networks [21–23, 31, 33, 35, 56, 64, 76, 77, 79, 83, 84]. We consider methods that have been explored by prior studies, restricting our focus to those applied in social media applications. We include also two alternatives [32, 36] proposed in other contexts to extend the set of to methods.

The selected methods are: naive threshold-based backbone extraction, High Salient Skeleton [32], Random rElationship ClAssifier STrategy (RECAST) [34], Disparity Filter [77], Polya Urn Filter [36], Marginal Likelihood Filter [35], Noise Corrected (NC) [33], Global Statistical Significance (GloSS) Filter [31], Tripartite Backbone Extraction (TriBE) [22], and Stochastic Degree Sequence Model (SDSM) [37].

Here we briefly describe and emphasize the differences among the methods. We provide a short summary of each algorithm in the Appendix A and direct the reader to the original papers where the methods have been proposed for a detailed description.

The aforementioned methods have been analyzed in the context of various phenomena. Yet, no prior study compared all ten methods under the same analysis framework. For illustration purposes, Fig 1 shows which methods have been compared to each other in previous studies mentioned in Section 2.2. Some of them (e.g., RECAST and TriBE) have not been compared to any alternatives. Clearly not all methods are adequate to all studies, which justifies the lack of some comparisons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Selected backbone extraction methods: Edges connect methods already compared to each other in prior work.

Details of each methods are provided in Table 1.

https://doi.org/10.1371/journal.pone.0274218.g001

3.1 Problem statement

Given (i) a particular phenomenon of interest driven by collective behavior, and (ii) a dataset capturing real interactions that represent manifestations of such phenomenon, how can we evaluate alternative network backbone extraction methods and select the one that, when applied to a network model of the input interactions, is able to accurately reveal key properties associated with the phenomenon of interest?

One major assumption that guides our effort is that not every network backbone is adequate to study the given phenomenon. Rather, key characteristics of such phenomenon must be matched to the assumptions and requirements of each method. Thus, a characterization of these properties is of utmost importance to drive the analysis. A mismatch between those characteristics, assumptions and requirements may lead to biases and misinterpretations.

Specifically, we are interested in identifying the methods that provide the best agreement between topological properties associated with the connectivity of vertices in the network and the contextual properties associated with factors driving the phenomenon that emerges from those patterns. Since our interest is in collective behavior patterns, the topological properties of interest are mostly associated with communities representing tightly connected groups of users who exhibit common behavior. One key challenge we must face is that each backbone extraction method removes edges (and nodes) from the network based on its own definition of edge saliency. Note that nodes that end up isolated after edge removal are also removed from the backbones. Thus, backbones extracted by different methods would reveal different topological structures, with properties that, though possibly strong and clear, may not be relevant (or related) to the phenomenon being studied.

Let us start presenting a simple case to exemplify the complexity of the problem. Consider the network in Fig 2a) built by connecting different users (nodes) who shared the same content on WhatsApp. Edges are weighted by the number of times the users shared the same content. This network, consisting of 190 nodes and 6 760 edges, is a subgraph of the network we analyze in Section 6. Suppose we build this network to investigate evidence of users’ coordination to speed up content spreading on the platform. Fig 2b–2d show three different backbones extracted from the same original network by three different methods, namely the threshold-based method, Gloss Filter and Disparity Filter. As evident, each backbone contains a different subset of the original edges and nodes. The question that arises is: Which backbone is the best one to study our phenomenon of interest, i.e., coordinated behavior?

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example network and the backbones extracted from it by three different methods (modularity values presented within parentheses).

Edge thickness represents edge weight and nodes’ color possible coordinated users’ communities. Node position is the same as in the original network.

https://doi.org/10.1371/journal.pone.0274218.g002

All methods remove a large fraction of the original edges and reveal clear topological structures in terms of communities, which reflected by the modularity [85] which improves w.r.t. the original network (see values in captions of the figures). The Gloss Filter backbone (Fig 2b) is quite different and misses the strong groups of users found by the Threshold and Disparity Filter that form a very tightly connected community. These users shared the same content many times (indicated by the reddest edges), which is a strong evidence of coordination. The spatial layout of the nodes is fixed for all networks. This makes nodes that are members of the same community spread out, which makes it hard to see how interconnected they really are. The threshold-based and Disparity Filter backbones (Fig 2c and 2d) look somehow similar. Both reveal four tightly connected groups of users, but the threshold-based model misses some strong edges among users, resulting in smaller communities. At the end, in this simple case, the Disparity Filter provides the best results.

Note that such conclusions cannot be based solely on topological/structural metrics. Indeed the modularity of the Gloss Filter backbone is the highest, but the extracted communities are not linked with the phenomenon under study. We must thus also consider contextual aspects related to the specific case study.

To generalize and tackle our target problem, in the following we propose a methodology to target the studies of collective behavior emerging from networks. Despite the focus on social media, our proposed methodology is generic enough to be applied to phenomena in other online and offline domains that are also modeled by noisy networks (e.g., co-voting [69, 70] and co-authorship [65] networks).

4.1 Step 1—Building a network model

We assume the availability of a dataset containing a temporal sequence of user interactions gathered from the target system taking place over a period of interest. In essence, these interactions may occur among multiple users simultaneously, being thus referred to as many-to-many interactions. They are observable actions (e.g., comments posted in a social media application) reflecting different user behavior patterns. We are interested in revealing those patterns that are fundamentally related to (and drive) the phenomenon that will be studied.

Such many-to-many interactions can be modeled as a network by building bipartite, a hypergraph or, more broadly, higher order models [53]. Yet, a large number of backbone extraction models work directly on projected networks where edges are added to represent interactions among pairs of users [10, 12, 19, 20, 56–58]. This is the case of Disparity Filter, Noise Corrected, among others [33, 36, 77]. Other methods adopt alternative network models by explicitly representing users as well as phenomenon-specific artifacts through which users interact with each others. As such these methods adopt an n-partite network model. Examples are TriBE that works on a tripartite model representing commenting users, influencers and their posts, and SDSM, which in turn works on a bipartite network representing, for instance, members of congress and voting sessions [22, 26, 70]. However, both methods (as others [26, 37]) end up building a backbone composed of salient user-user edges, thus effectively building a projected backbone. Therefore, we chose to present here the common projected network model. Yet, we note that both Tribe and SDSM operate on the original n-partite network to extract the projected backbone. We thus adopt an undirected and weighted projected network G = (V, E) as the base model for our methodology, such that:

• V is the set of users who interacted at least once during the period of interest;
• E is the set of undirected and weighted edges connecting pairs of users, such that the weight of edge connecting users i₁, i₂ ∈ V is γ(i₁, i₂) = f(i₁, i₂), where f(i₁, i₂) is any aggregation function (e.g., count) defined over the set of interactions between i₁ and i₂ and/or any contextual information available associated to them. Examples include the sharing of similar content (e.g., same URLs, same hashtags, or same messages) and/or temporarily synchronized activities [8, 10, 12, 18–20].

Our methodology must give us the means to extract the backbone of the original network where noisy edges are filtered out. Before proceeding, we note that one might be interested on the dynamics of such backbone over different periods of time covered by the input dataset. In that case, one strategy is to break the original data into subsets covering non-overlapping and consecutive time windows (e.g., weeks or months) and build one network model for each window. Given that the phenomenon under study remains the same, it is reasonable to assume that the contextual criteria impacting the selection of the best backbone extraction method would be maintained across network models. Thus, the methodology could be applied to one of such network models to identify the most adequate backbone extraction method. Such method could then be used to extract different backbones (one for each window) allowing an assessment of the temporal evolution of their properties (as done in [21, 22, 25, 55, 69, 70, 76, 86]).

4.2 Step 2—Selecting candidate backbone extraction methods

In principle, any backbone extraction method could be applied to a given network model, and the backbones extracted by different methods may be quite different (as illustrated in Fig 2). Some backbones may miss a few important edges while still offering important insights, whereas others may be composed mostly of edges of little relevance to the study. Detecting the latter is not always easy, especially for large-size networks. Thus, we argue that a careful and principled selection of candidate methods must be performed before evaluating the extracted backbones to avoid misinterpretations and facilitate evaluation. To that end, our goal in this step is to shortlist backbone extraction methods that are adequate to study the given phenomenon. By adequate we mean that their assumptions and requirements are in alignment with key characteristics of the phenomenon, at the cost of generating completely unrelated backbones otherwise.

In the following we offer a characterization of ten alternative methods (cfr. Section 2.3) and discuss issues one must consider to study a target phenomenon. The discussion below reflects our analyses of the methods’ applicability to different scenarios. To guide this discussion, in Table 1 we present a summary with some key properties of each method. We categorize the methods along with four aspects that are important to assist one in determining the suitable methods for a given case study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Our characterization of selected backbone extraction methods.

https://doi.org/10.1371/journal.pone.0274218.t001

Before we look at the edge salience criteria, the first aspect that should be considered when deciding which backbone method to apply is the nature of the edge weights (2^nd column of Table 1). This preliminary aspect is a fundamental step in our proposed methodology to discard methods that are not applicable to the problem. Most of the methods considered here are limited to discrete weight values (Noise Corrected, Disparity Filter, TriBE, RECAST and SDSM), while others work with continuous values too (HSS, Threshold and GloSS). Similarly, another factor to be considered is whether the method expects only positive weights—e.g., only the Threshold-based approach can work with negative weights among the ten evaluated methods.

4.3 Steps 3 and 4—Backbone extraction and evaluation

Having identified a set of backbone extraction methods that could be employed in a particular study, step 3 consists of applying the selected methods to the original network to extract the corresponding backbone. Specifically, each candidate method c in a set of methods C identified in step 2 is applied to extract a backbone , such that consists of only edges considered salient by c and is the set of nodes with at least one edge in the backbone extracted by method c. In other words, after backbone extraction, all isolated nodes are disregarded. Step 4 consists of evaluating the quality of the produced backbones. In case multiple methods were selected in step 2, the best alternative should be chosen according to a trade-off between the metrics discussed next. The backbone produced by the best method would then be used to carry out the study.

Building on prior work [33, 36, 47, 64, 85, 93], we consider metrics of backbone quality in two categories: topological, which are closely related to network and community structure, and contextual, which refers to phenomenon-specific attributes.

5.1 Characterization of the phenomenon

Our first case study focuses on online discussions on Instagram. In this social media application, top public profiles (called “influencers”) create posts that attract the interest of users, who may like or leave comments associated with them. Specifically, Instagram users engage in online discussions by commenting on posts [22, 23, 103–107]. The study of such discussions can offer important insights into how information is disseminated on the platform and how the online debate impacts our society [108–113].

We rely on a sample of the dataset gathered and analyzed in our earlier work [22, 23]. It is composed of public content published by political influencers in Brazil in the week surrounding the first round of Brazil’s 2018 general elections (e.g., from September 30th to October 6th). We gather posts by the eight candidate runners: @jairmessiasbolsonaro, @fernandohaddadoficial, @lulaoficial, @cirogomes, @_marinasilva_, @guilhermeboulos.oficial, @cabodaciolo, @ad.alvarodias. We use all posts created by these profiles during the election week, as well as the comments they received from other Instagram users. We choose not to include users who commented on a single post, as these clearly reflect sporadic behavior. In total, we analyze 41099 users who made 376779 comments on 540 posts.

5.2 Step 1—Building the network model

We model the interactions among users commenting on the Instagram posts using the network model proposed in [22, 23]. This network model is defined as a weighted undirected graph G_Instagram = (V, E), where:

• V is the set nodes representing the users who commented on posts; and
• E is the set of edges, where each edge (i₁, i₂) ∈ E connects nodes i₁ and i₂, representing two users who commented on the same post. The edge weight γ(i₁, i₂) is the number of posts that received comments by the same two users.

Many interactions represented by edges in this network model may not reflect actual discussions. For example, a very popular post naturally attracts many users, who comment on them often in an independent manner, without actually engaging in a discussion about the topic. Moreover, some users are more active than others. Such cases lead to the emergence of a number of edges simply by chance. As these edges do not reflect discussions among the users/commenters, they represent noise to the study of the target phenomenon.

5.3 Step 2—Selection of candidate backbone extraction methods

The understanding on how the inherent properties of the phenomenon impact the network model helps us to focus on backbone extraction methods that do take such properties into consideration to identify salient edges. In this context, salient edges are those with greater evidence of reflecting online discussions. Hence, we are looking for methods that consider the effects of user activity level and post popularity on the emergence of irrelevant edges. As such, methods that are based on the assumption that edge salience is necessarily related to edge weight (e.g., methods that assume that edges with larger weights are more likely to be salient), either from a local or a global perspective, are not adequate. These methods tend to retain in the backbone only edges representing repetitive patterns of the most active users, disregarding interactions reflecting discussions carried out by less active (though still important) users.

Given these considerations, we select the following set of candidate methods for further evaluation: C = {MLF, NC, Gloss Filter, TriBE, SDSM}. These methods are fundamentally local and statistical, and factor both user activity level and content popularity. TriBE and SDSM explicitly build a reference model based on both characteristics. MLF, NC and GloSS Filter evaluate the salience of an edge taking these factors into account indirectly, by exploring structural aspects of the network, notably node strength and degree, which are affected by user activity level and content popularity (cfr. Section 4.2.4).

Notice that although we argue that the SDSM method is adequate to the study of online discussions [37], it has some theoretical advantages and disadvantages with respect to the other methods. First, while SDSM explicitly controls the degree sequence of both artifacts and agents, it does not consider that some artifacts can only have links to certain subsets of agents. In the context under study, this implies the assumption that all 41 k users (i.e., agents) can comment on all 540 posts (i.e., artifacts). Even though this is possible in theory, it does not happen in reality, as users select specific posts and influencers to comment on [22, 23]. Thus, the expected value of an edge between two users is underestimated, making it more likely that edges are considered salient. In contrast, the other considered methods, which work on a projected network (i.e., [23, 31, 33, 35]), allow for disjoint subsets of commenters per post as only the actual edges (which capture user preferences) are represented in their reference models. In particular, TriBE [22, 23] was designed to capture such inherent heterogeneity of user behavior and preferences. It does so by considering user engagement towards posts of specific influencers, i.e., a disjoint relationship between subsets of artifacts (posts) and subsets of agents (users). In other words, a tripartite structure is considered, which is able to compute different distributions for a subset of users and posts and thus more accurately determine the expected value of an edge.

5.4 Step 3—Backbone extraction

We apply each candidate backbone extraction method in C to the network model built in step 1. Recall that all the selected methods require alpha as parameter, and we test a range of possible values. Table 2 summarizes the topological properties of the original network and the backbones extracted using the considered methods. The reported backbone results were obtained with the following alpha parameters: TriBE = 0.05, SDSM = 0.001, GloSS = 0.10, MLF = 0.001, and Noise Corrected = 0.00001. Results for other values of alpha are reasonably consistent—see Table 6 in Appendix B. Recall that we chose alpha that offered the best tradeoff between modularity and regression results while still producing a large subset of common edges across methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Online discussions on Instagram: Topological metrics of the network and backbones extracted by each candidate method.

Columns 2-3 contain total numbers for the original network and also the corresponding percentages for backbones.

https://doi.org/10.1371/journal.pone.0274218.t002

Columns 2-7 in the table show the results for topological metrics: nodes and edges (total numbers for the original network and also corresponding percentages remaining in the extracted backbones), average degree (Avg. Deg.), density, average clustering coefficient (Avg. Clust.) and the number of connected components (# C.C.). Due to the high execution time, we use 20% of the nodes in each network to estimate the clustering coefficient. Overall the backbones are sparser than the original graph, since a large fraction of the nodes and edges have been removed. Moreover, the average clustering coefficient shows a moderate number of connected triangles in the network for all methods. Interestingly, four out of the five resulting backbones have only one connected component, as is also the case for the original graph, suggesting the presence of key users promoting online discussions across different Instagram profiles and connecting salient edges into a single component.

The three rightmost columns of Table 2 show results of community-related metrics. We consider only communities with more than 100 users. The number of communities (# Comm.) is larger than in the original network. This is expected since backbones are sparser than the original graph. Finally, the rightmost column of Table 2 shows the modularity results as a measure of community quality. All backbones clearly have more strongly connected communities (i.e., higher modularity) than the original network, although we still need to assess the representativeness of these backbones using contextual information.

Since the methods extract backbones with different topological structures, we further analyze how each method deals with edges with respect to their weights. Fig 4 shows the distributions of weights for edges retained in the backbone by each method. Each plot also shows the distribution of edge weight in the original network for comparison. TriBE and SDSM (Fig 4a and 4b) remove many edges with small weights as well as some edges with large weights. GloSS Filter (Fig 4c) is more aggressive towards heavy edges and removes some of the edges over the whole range of values. Note that this method filters out all heaviest edges. NC (Fig 4d) is the most conservative method with respect to heavy edges and removes a smaller fraction of the edges through the whole range of weight values. MLF (Fig 4e) follows a similar pattern as NC, preserving heaviest edges. In summary, the results show that while all five methods share similarities (e.g., they all capture the effects of user activity levels and post popularity), they have their peculiarities when it comes to identifying an edge as salient, and they produce quite different backbones.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Online discussions on Instagram: Weight distribution for edges retained in the backbone by each method (distribution for original/complete network shown for comparison).

https://doi.org/10.1371/journal.pone.0274218.g004

5.5 Step 4—Backbone evaluation

5.5.2 Contextual evaluation.

We shift to the quality of the backbones from a contextual point of view. We build regression models considering the following key assumption: If two users i₁ and i₂ engage in the same online discussions, then the individual activities (comments) performed by each user are strongly correlated with the joint activities performed by both users. If, however, comments posted by one user (or both) are mostly reactions to popular content or to some automatic tool (e.g., advertising or personalization mechanisms), or simply sporadic behavior, the activities of the user individually are only weakly correlated with the joint behavior of both users.

Based on this assumption, we build a regression model for each backbone and the original network. Given the edge weight γ(i₁, i₂) as dependent variable, the explanatory covariates are: (i) number of posts that user i₁ commented on, (ii) number of posts that user i₂ commented on, (iii) number of influencers that user i₁ commented on, and (iv) number of influencers that user i₂ commented on. We capture user activities by considering both the number of influencers and the number of posts each user commented on because it is often the case that the same influencer has multiple posts on different topics, each one attracting a different group of users (community) [22, 23]. Thus, we expect that only edges representing user interactions driven by joint engagement to be reasonably well explained by these covariates. Thus, the better the fitting of the model to the edge weights in a backbone, the better the quality of this backbone from a contextual perspective.

We check whether the covariates are linearly related to the dependent variable, a key assumption to use a linear regression model. We found that such linear relationship exists if a log transformation is applied to all covariates and to the dependent variable. Such transformation is often employed in variables with very skewed distributions, which is the case of edge weights (see Fig 4) and measures of user activity in social media applications [114, 115].

Table 3 shows the results of the model fitting for the five backbones and the original network. We assess the quality of model fitting using the coefficient of determination R² and the NMRSE for the training edges and the NMRSE for the test edges.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Online discussions on Instagram: Contextual evaluation of backbones by regression analysis.

https://doi.org/10.1371/journal.pone.0274218.t003

Focusing first on the results for the training edges, we see that both SDSM and TriBE achieve significant improvements over the original network, both in terms of R² and NMRSE. Indeed, these two methods are able to filter out many noisy edges and retain those more closely related to the phenomenon under study, which is reflected in the covariates used to build the regression models. SDSM and TriBE, by explicitly taking into account both the users’ activity level and the posts’ popularity when building the reference model, lead to high R² values of 0.91 and 0.87, and an NRMSE of only 0.13 and 0.18. In turn, the other three methods, NC, MLF and GloSS, despite filtering out many edges (as shown in Table 2), lead to only moderate improvements over the original network.

The same conclusion holds for the test edges. Compared to the original network, the fits for both SDSM and TriBE show a notable reduction in NRMSE. This suggests that the edges considered salient by NC, MLF, and GloSS deviate the most from the common set of edges considered salient by all methods. This observation, in turn, suggests that these three methods retain a large fraction of possibly non-salient edges, which ultimately affects the fitting of the regression model.

SDSM performs slightly better than TriBE. Since we are only studying 8 popular influencers talking about politics, the effect of the third part (influencers) explicitly captured by TriBE but ignored by SDSM is mostly marginal. We expect a greater difference between the two methods, favoring TriBE for scenarios with a larger diversity of influencers in terms of both popularity and topics of posts.

In conclusion, our results indicate that evaluating backbone quality based solely on a single perspective may be misleading. For example, Table 2 shows that the backbones extracted from NC and MLF have the highest modularity scores. Yet, the regression analysis shows that the edges identified as salient by both, although well structured into strongly connected communities, do not offer a clearer understanding of the user behavior patterns driving the online discussions than the (poorly structured) original network. SDSM and TriBE, in turn, stand out as the best approach when considering a tradeoff between the quality of their communities and the ability of the selected edges to capture the user interactions that are driving the online discussions.

6.1 Characterization of the phenomenon

Our second case study concerns coordinated actions to disseminate information in WhatsApp groups. The platform connects users in end-to-end as well as group conversations. Despite being limited to only 256 simultaneous members, WhatsApp groups have been shown to be effective channels for the large dissemination of information [21, 76, 116, 117], notably misinformation [21, 118]. We here adopt the following widely used definition of coordination of users [20, 119, 120]: coordinated users typically exhibit a repetitive and synchronized pattern of activity.

Our present investigation relies on a dataset of anonymized messages shared in publicly accessible political-oriented WhatsApp groups in Brazil [21, 76], originally collected by the WhatsApp Monitor [116]. We focus our analysis on the month of the general presidential election in Brazil (October 2018), a time of great political mobilization and strong evidence of message coordination and orchestration in WhatsApp [21, 76, 116, 121, 122]. In summary, we analyze 4341 users who participated in 155 groups and shared 91417 unique pieces of information, in the form of text messages, images, audios and videos.

6.2 Step 1—Building the network model

We use the same network model adopted in [21, 76], referred to as media-centric network, which is defined as an undirected and weighted graph G_WhatsApp = (V, E) such that:

• V is the set of nodes representing users who shared at least one message in one of the monitored groups during the period of analysis;
• E represents the set of edges, where each edge connects two users if they share similar content in the same or different groups. The similarity between message content was estimated using a set of heuristics for filtering and identifying (near-)duplicate content. We refer the reader to [21] for more details on these heuristics. Each edge is weighted by the number of times the two users shared similar content.

In light of the adopted definition of coordination, salient edges are those whose weights are unusually high. The network may contain several noisy edges due to sporadic or weak interactions. For example, endogenous factors (e.g., temporary common interest or even large popularity of some particular content) may cause different users to share similar content, which may overshadow the actions of coordinated users who regularly and repeatedly share the same content. Therefore, the interest is to separate users who persistently engage in such common sharing from users who only sporadically exhibit such behavior. This separation implies favoring as salient those edges with heavier weights, either taking a local perspective (e.g., other edges incident to the same two nodes) or a global perspective (i.e., all edges in the network). This principle is used as a guideline in selecting candidate backbone extraction methods, as discussed next.

6.3 Step 2—Selection of candidate backbone extraction methods

We aim at selecting methods that explore the heterogeneity of the network by identifying as salient the edges with unusually heavier weights, based on individual (local) or network (global) patterns, as representative of persistent and repetitive interactions. As argued in Section 4.2.1, Threshold, HSS and RECAST are global methods that explore the heterogeneity of the edge weight distribution, favoring as salient the edges with heavier weights in the whole network. From a local perspective, Polya Urn Filter and Disparity Filter (DF) select as salient those edges whose weights are heavier than the weights associated with a subset of the edges (e.g., edges incident to the same pair of nodes). Thus, we define the set C of candidate methods as C = {Threshold, HSS, RECAST, Poly Urn and DF}. Recall that both Threshold and HSS explore structural properties, whereas the other three methods rely on statistical reference models to identify the salient edges.

6.4 Step 3—Backbone extraction

Table 4 summarizes the topological characteristics of the original network and the backbones extracted by each candidate method. The candidate methods have different parameters. Disparity Filter (DF), Polya Urn Filter (Polya) and RECAST require a a, which we set to 0.05, as shown in Table 7 in Appendix B. The Polya Urn method also requires a second parameter a related to the heterogeneity of the network. This parameter was set to 0.25, following a fine-tuning process, as briefly mentioned in Appendix A. The High Salient Skeleton (HSS) and Threshold approaches, on the other hand, take an arbitrary threshold value τ as input parameter. In both cases, we select τ to retain the top-k% most salient edges. Table 4 shows results for values of τ corresponding to the top 5% edges but we also tested for other values of k (thus of τ), as reported in Table 7. According to Table 4, the fractions of edges retained by both HSS and Threshold are slightly below 5%. This discrepancy is due to the removal of very small components (up to 3 nodes) of the backbone and the original network.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Coordinated behavior on WhatsApp: Topological metrics of the network and backbones extracted by each candidate method (Columns 2-3 contain total numbers for the original network and also corresponding percentages for backbones).

https://doi.org/10.1371/journal.pone.0274218.t004

In each graph, nodes belonging to the same community are represented by the same color, and edge weights are represented by both edge thickness and color (heavier/lighter edges are colored in red/blue).

As shown in the table and the graphs, DF, Polya and Threshold retain somewhat similar fractions of nodes (11-18%) and edges (4%) in the extracted backbones. Despite such large removal of nodes and edges, all three backbones have a number of connected components that approximate the original network. Interestingly, we also find that these backbones have larger density (especially the backbone extracted by Threshold) and comparable (if not higher) average clustering coefficient to the original network. RECAST, in turn, is the most aggressive method, retaining only around 7% of the original nodes and fewer than 1% of the original edges. This leads to a smaller number of components and average clustering coefficient, though the backbone’s density is still comparable to that of the DF’s and Polya’s backbones. At the other extreme, HSS extracts a rather large backbone while preserving all the original nodes. This backbone has a very different topological structure than the others, being much sparser and exhibiting lower density, degree, and average clustering.

Turning to the analysis of the communities (3 rightmost columns of Table 4), we observe that the number of communities varies considerably, being smaller for the backbones extracted by Threshold and RECAST. The former may be a consequence of the much denser backbone, while the latter may be related to the smaller number of nodes in the backbone. Recall that we still need contextual information to investigate the extent to which such communities are representative of the phenomenon.

To further understand how the selected methods work, we analyze the distribution of edge weights, reported in Fig 5. All methods except HSS remove mostly edges with small weights. In particular, all edges with weights between 1 and 3 are removed by all four methods. HSS, in contrast, removes large fractions of edges across the whole range of weight values. As we will see in the next section, the HSS backbone is inferior to the others from a contextual perspective and, therefore, in terms of how well it captures edges related to the phenomenon under study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Coordinated behavior on WhatsApp: Weight distribution for edges retained in the backbone by each method (distribution for original/complete network shown for comparison purposes).

https://doi.org/10.1371/journal.pone.0274218.g005

6.5 Step 4—Backbone evaluation

6.5.2 Contextual evaluation.

The following key assumption guides our contextual evaluation of the backbone extraction methods: If two users i₁ and i₂ are acting in coordination to share the same pieces of content repeatedly, such coordination should be reflected in their sharing patterns and user activity. Guided by this assumption, we build a regression model where the dependent variable γ(i₁, i₂) is related with the following 11 explanatory variables: (i) total number of messages shared by i₁ (i₂); (ii) number of distinct messages shared by i₁ (i₂); (iii) number of messages with new content introduced (i.e., shared first) by i₁ (i₂); (iv) number of groups i₁ (i₂) participates in (inferred by the groups he/she shared content at least once); (v) Gini index of the number of messages shared by i₁ (i₂) across different groups; and (vi) number of common groups both i₁ and i₂ participate in. All variables, but the last one, are computed separately for i₁ and i₂, thus contributing as two covariates to the model. These variables capture different facets of user activity. The only variable related to the joint behavior of both users is the number of common groups, which indirectly captures whether or not the two users act in the same subset of observed groups. For example, users may act in a coordinated manner by frequently sharing the same content (i.e., heavy edge), even though the number of groups in which both participate is small. This could indicate, for example, that each of them forwards the same content to a particular subset of groups. As in our first case study, we also tested for the assumption of linearity, finding that it holds reasonably well after a square root transformation is applied to all covariates and the dependent variable.

Table 5 summarizes the results of the model fitting for each backbone and the original network for both training and test edges. Compared to our first case study, the fittings are generally poorer (note the lower R² values in the training edges). We emphasize that it is much more challenging to perform a contextual evaluation of the network and backbone structures in this case study because we are focusing on messages shared in only 155 groups. All monitored groups belong to the same context (political domain) and are strongly interconnected as many users belong to multiple groups. However, these groups offer only a partial view of WhatsApp. The same users might participate in other groups, where they share and forward content, contributing to the information spread at large (see [116]). Thus, our analysis is limited by the lack of an unknown number of edges that most probably exist in the real underlying network connecting these users.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Coordinated behavior on WhatsApp: Contextual evaluation of backbones by regression analysis.

https://doi.org/10.1371/journal.pone.0274218.t005

Under this constraint, the contextual evaluation leads to results consistent with those of topological evaluation. HSS has the highest R², but also the highest errors (NMRSE), which are almost similar to the original network. The backbones extracted by DF, Polya Urn and Threshold, have the lowest errors in both the training and test sets. Concerning R² the results are generally poor, with DF and Polya Urn performing better. Threshold performs worse than DF and Polya, suggesting that a global approach may leave out some important edges for the investigation. Indeed, as shown in Fig 5, both DF and Polya retain a more diversified set of edges in terms of edge weights.

In sum, we find that both DF and Polya Urn are the best methods to uncover evidence of user coordination when sharing similar content on WhatsApp. If one has to choose a method, DF is possibly the best choice as it contextually reveals communities closer to the phenomenon.

A Backbone Extraction Methods Summary

Threshold-based backbone extraction. one of the simplest, most intuitive and most used methods [69–71, 74]. It consists on removing edges whose weights are smaller (or higher) than a pre-defined threshold τ, that is, edge saliency refers simply to edge weight. This method is adequate to studies where the salient edges are those with higher (or lower) weights. Otherwise, as previously argued [72], thresholds may bias the analysis and lead to misinterpretation of the results.

High Salient Skeleton (HSS) [32]. the backbone is extracted by first normalizing the edge weights and then computing shortest-path trees from each node to all other nodes in the network. Edge saliency is defined based on the frequency of its occurrence in the shortest path trees: edges with frequency below a pre-defined threshold τ are disregarded. In doing so, this method attempts to capture edges that simultaneously have heavy weights and are fundamental for keeping nodes connected. As such, the notion of edge saliency is inherently connected to network topology. Moreover, like for the threshold-based method, the use of a global threshold may lead to biases and misinterpretation [32].

Disparity Filter (DF) [77]. it assumes that an edge connecting a given pair of nodes is salient if it has a disproportionate weight compared to the other edges leading from the nodes to their respective neighbors. In other words, salient edges are those whose weights deviate significantly from the null hypothesis that the weights of all edges incident to a given node are uniformly distributed.

Polya Urn Filter [36]. similarly to DF, this method assumes that edge weights emerge from the aggregate process of individual nodes’ preferences to interact with each other over time. It also assumes that interactions between nodes are maintained and reinforced, such that the larger the number of interactions between the same two nodes, the higher the probability of they interacting again. A reference model is built for each edge, using the Polya Urn model [123] which captures the reinforcement of existing interactions by examining the degree and strength (the sum of the weights of all edges incident to the node) of each node incident to this edge. This reinforcement mechanism can be regulated and estimated by the system through a fine-tuning process. Salient edges are those that deviate significantly from such reference model (according to a given alpha).

Marginal Likelihood Filter (MLF) [35]. assumes that edge saliency should be analyzed in light of the strengths of the two nodes the edge connects. The higher the strengths the larger the edge weight must be to be considered salient. Specifically, the method builds a reference edge weight distribution model for each edge: the probability that edge between nodes i and j ends up with weight w_ij is based on a Binomial distribution with parameters n defined by the total strengths of all nodes in the network and p computed based on the strengths of nodes i and j. An edge is considered salient if the observed weight deviates significantly from the one predicted by the reference model.

Noise Corrected (NC) [33]. similarly to DF and the Polya Urn Filter methods, NC also assumes that edge saliency arises from the cooperation between nodes. However, unlike those methods, NC preserves peripheral-peripheral connections, which is crucial for capturing edges that, despite having small weights, may still be considered relevant for the phenomenon under study. These connections may be preserved by estimating the expectation and variance of edge weights using a hypergeometric distribution, taking into account the propensity of both nodes to send and receive edges. It also provides a direct approximation through a per-edge reference Binomial distribution (similarly to the MLF method). The main advantage of NC, though, is the ability to estimate an error for the expectation of the weights. As in the other methods, an edge is considered salient if its observed weight significantly exceeds the expected weight (given the strengths of both nodes).

Global Statistical Significance (GloSS) Filter [31]. it assumes that salient edges cannot be identified independently of the overall network topology, once nodes have different degrees. As such, it builds a single (null) reference model that preserves the edges between nodes as well as the overall edge weight distribution. Yet, when selecting salient edges, i.e., edges whose observed weights significantly deviate from the reference model, the method estimates the probability of observing an edge weight between two given nodes considering the nodes’ observed degrees and strengths as constraints.

Tripartite Backbone Extraction (TriBE) [22, 23]. this method was proposed to study phenomena driven by user interactions in social media applications. It exploits the tripartite structure commonly found in such platforms, that is, a piece of content, the content creator, and the other users (e.g., the followers) who interact with each other in reaction to that content (e.g., by commenting on a post, retweeting the same tweet, etc). As such, the method addresses the heterogeneity in user activity level and content popularity typically observed in social media applications. Specifically, it builds a reference weight distribution model for each edge, based on a Poisson binomial distribution, whose parameters are computed based on the distributions of content popularity and user engagement towards content from the same creator (as estimated by prior interactions). Once again, salient edges are those whose observed weights significantly deviate from their corresponding reference models.

Stochastic Degree Sequence Model (SDSM) [37]. This method considers a bipartite network to evaluate the salience of an edge in the projection. The SDSM specifies the degree sequence of agents and artifacts of a bipartite network on the average constraint such that their expected row sums and expected column sums are equal to the observed ones. In this way, a reference model is built based on the Poisson binomial distribution that selects as salient those edges whose observed weights differ significantly from the expected value given the degree sequence constraints.

B Parameter sensitivity analysis

We report in this appendix additional results from a sensitivity analysis we performed to parameterize the backbone extraction methods. Table 6 shows results of the impact of the alpha (input parameter) on the methods selected as candidates for the study of online discussions on Instagram (case study 1 reported in Section 5). Table 7 shows corresponding results for the methods selected as candidates for the study of coordinated behavior on WhatsApp (case study 2, discussed in Section 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Online discussions on Instagram: Impact of method parameters on topological and contextual metrics.

https://doi.org/10.1371/journal.pone.0274218.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Coordinated behavior on WhatsApp: Impact of method parameters on topological and contextual metrics.

https://doi.org/10.1371/journal.pone.0274218.t007

As pointed out in Section 4, defining such parametrization leads to a multicriteria problem. To guarantee the existence of a common set of edges under the notion of saliency consensus between the methods, we adopted the following strategy. In both scenarios, we based the most aggressive method in terms of the number of removed edges in the backbone and adjusted the parameters so that each method maintains an approximate percentage of edges in the backbone. For the first case study in Table 6, this bound was based on the GloSS Filter retaining 0.18% of the edges in the backbone. However, keeping this approximate percentage of edges for all methods does not allow us to create the minimum common set. To account for this, we parameterized the GloSS Filter with alpha = 0.1. Consequently, we look for parameters within the tested values for all methods that hold approximately 2.58% edges in the backbone. We found that such parameterization is detrimental to the GloSS Filter, but on the other hand, it allows us to create such a set that is representative of a consensus among methods with respect to a set of edges that are considered salient.

For the second case study, presented in Table 7, we started with RECAST, which is quite aggressive and consequently alpha = 0.05. Note that RECAST produces a global discrete distribution and thus some alpha values do not change the backbone extraction. Thus, we assumed that each method has about 0.85% of the edges in the backbone. However, with such a restrictive value, we could not create a common set. After incrementally increasing each method and assuming that this was the limit for RECAST, we found that a percentage of 4% to 5% of edges would satisfy this requirement.