Number of signals

The relationship between the number of signals an individual receives in its network, social influence, and the likelihood that said individual will adopt the behavior indicated by the signals is closely related to the linear threshold model in which an actor adopts a behavior after the signal count reaches an optimal threshold [23]. What, though, is the impact of repeated signals on the decision-making process, and more specifically how many signals are required to reliably influence an individual’s decision-making? There are mixed findings regarding the benefit of multiple exposures on the diffusion of information necessary to reach this threshold [24]. People may prefer multiple confirmations from their peers to reassure themselves before making a decision [15, 25]. Experimental human studies using games from behavioral economics like the Prisoner’s Dilemma tend to find that the impact of zero to three signalers increases behavioral and decision-adoption in a linear fashion, and that a key threshold for maximum social influence exists between four and five signalers. This means that two signalers exert more influence than one, three exert more than two, and four to five exert more influence than three. There is not much difference between the impact of five and six or more. However, debate still exists—some have found the threshold to be two signalers [26] while others report it at three [17, 27]. Still, others have found the reverse trend. In one study, repeated exposure to online signals in a social networking site slowed the subsequent spread of information. This might affect decision-making and behavioral adoption by inhibiting social influence [24, 28].

Network structure

A network structure’s describes characteristics of the network as a whole. This can include properties like clustering/decentralization (to what degree actors form ties), modularity (how densely connected nodes are within clusters), homophily/heterophily (similarity or dissimilarity of actors predicts tie-forming), and centralization (how connected are actors). For instance, the density of a decision-maker’s social network can influence the choices they make because signalers’ intentions are more ambiguous in densely connected networks [29]. Similarly, the choices that individuals make and the contribution of social influence have also been attributed to the network communities they belong to, while studies related to patterns of subgraphs that result out of such individual interactions have been popular [30]. The dynamism, or how easily entities can move in and out of networks, also influences decision-making. Social decision-making modelers have found that the more dynamic and mobile a network, the greater concurrence of their decisions in behavioral economics games [31, 32]. Because the present studies are primarily interested in the effects of a) the number of signals and b) the pattern by which they are received, we attempt to control for the effects of network structure by holding the structure constant throughout all experimental conditions in both studies. Please see the Methods sections for further details.

Information parsing

The manner in which individuals search for information affects their decision-making. Individuals employ search strategies to reduce the number of choices [5]. This includes revising their initial opinions by processing and averaging the different influences acting on them [33] and social information provides one mechanism through which this is achieved. These manners of decision making have also been linked to the concept of dual process theory, the notion that two different systems of thought co-exist; a quick, automatic, associative, and affective-based form of reasoning and a slow, thoughtful, deliberative process [34]. Fast thinking involves conditions of “cognitive ease” and so social influence factors into this process of slowing down the decision making system by presenting alternating evidences for reconsideration. When social signals point towards a specific outcome or opinion, individuals will often adopt the opinions and behaviors of signalers [22, 35], however while adopting behaviors based off social influence can be cost effective, it does not always lead to the most effective or efficient decision. Individuals must trade-off between trusting their own knowledge and trusting other’s opinions [13]. Biased social signals can influence individuals to choose wrong or less effective answers in a variety of domains, particularly if multiple peers back the behavior or decision [15, 22, 35].

Product utility

When making product decisions, outcomes related to the quality and need for a product change its utility or perceived value and therefore the risk of the selection. For instance, when decision-makers were asked to purchase songs in an online market where they could see the decisions of other purchasers, the perceived quality of the songs predicted their choices even when they witnessed peers purchasing songs of poor quality. Songs of medium quality were most subject to the effects of social influence [36]. The need for a quality product also influences selection decisions. Kraut and colleagues found that when deciding between different video-teleconferencing technologies, people with the most communication intensive work—those who relied on video-teleconferencing the most—placed greater value on the product’s ability than those with less need [4]. The perceived value of a product predicts how much an individual will search for information to inform their selection [21], and consumers and decision-makers are more likely to engage in information search, including social information—when purchasing products is risky—such as when the costs of making a sub-optimal choice are high, and when they lack prior knowledge about the technology [21]. When we feel that we know enough about a product, we believe that we already have enough information stored in our memory to make the best decision, and therefore additional social information is unnecessary.

4.1 Design

Participants were randomly assigned to five groups with each group having unique members not involved in decision making as part of other groups. The entire game was partitioned into two phases. For the first phase comprising 12 time steps, no other information except a short excerpt about six potential providers was given. After the participants made their selection for a given time step, they saw the number of attacks their provider had prevented in that step. As mentioned earlier, in the absence of the knowledge of the utilities of the technology providers (or the number of attacks it prevents), the first 12 time steps allow for individual decision making and exploration.

In the second phase of the experiment which started at Time Step 13, we introduced interventions by allowing participants access to extra information from six other individuals who are supposedly their peers (but are really bots controlled by us). After participants make their choice at a time step, they can view the selections made by their peers in the previous time step. Each participant in all groups bar one are subjected to six peers—We call the decisions of these peers which the participant views in the second phase as peer signals in this work. Fig 2 shows the screen of a participant from time steps 13 to 18 when they were able to view the decision of their peers. The platform was hosted by the Controlled Large Online Social Experimentation (CLOSE) platform and developed at Sandia National Laboratories [38]. While the participant is able to view the technology selections of all their peers at each time step, we control social influence by administering a randomly selected sub-optimal technology C_u (among the five providers) through the peers of the participant u—so using our nomenclature, C_u is the influence decision for u. For each u, this technology C_u was selected as the choice that is disproportionately signaled by its peers over time (this pattern of influence or PoI was manipulated by us). The motivation behind this deliberate selection of sub-optimal C_u (controlled by us) as the influence decision was to investigate whether participants would be tempted to select this technology C_u in the presence of its peer feedback. Note that we attempted to avoid network effects by randomizing this technology or the influence decision C_u specific to the user u—this allows us to avoid any deliberate collisions among peer choices of different users that could be representative of network effects in the real world.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example screen in the cyber-defense provider selection task.

Participants in the Uniform Messages (UM) condition of the study have access to a screen that resembled this one. The Feedback section displays the number of attacks the participant prevented after each time step. The Decisions section displays the six provider choices that it has. Finally, the Messages section is displayed after Time Step 12, where the participants can view what their peers selected in the previous time step.

https://doi.org/10.1371/journal.pone.0234875.g002

We denote by A_u(t) the number of peers of the participant u, who we administer the sub-optimal technology/influence decision C_u at time step t (so A_u(t) ≤ 6). We now describe the pattern of influence A dropping the subscripts to generalize for all users or PoI as we denote it, for the five groups (the first two being the control groups for comparison with the next three treatment groups) (Fig 3 shows the signal patterns for the groups):

1. No Message (NM): Participants receive no message from the peers, so the last six time steps are exactly the same for the participants as the first 12.
2. Uniform Message (UM): Here we send C using one peer of a participant at each time step. So A = {1, 1, 1, 1, 1, 1} denotes uniform influence.
3. Linear Cascade (LC): Here we incrementally activate one peer with the technology C at each time step. So A = {1, 2, 3, 4, 5, 6} denotes uniformly increasingly influence as shown in Fig 4.
4. Delayed Cascade (DC): Here we send only one signal for the first three time steps and send four, five and six signals at the last three time steps in order. So A = {1, 1, 1, 4, 5, 6}. The objective is to see whether the sudden change in the number of signals acts as a catalyst for successful influence at the later stages of the experiment.
5. Early Cascade (EC): In this setup, we send a higher magnitude of signals from the beginning setting A = {4, 5, 6, 6, 6, 6}. This pattern allows us to ask if an early trigger is able to sustain the levels of influence, or whether participants will return to the optimal choice at the later stages.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The signal vs time step plots for the 4 patterns—Note that for the NM group (not shown here), no peer signal in the form of pre-selected sub-optimal technologies were sent to the participants at any time step.

https://doi.org/10.1371/journal.pone.0234875.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Illustration of the linear cascade diffusion.

The technology C_u chosen by us as the sub-optimal technology (influence decision for user u (in dots) cascades through the peers of u over the six time steps. Colored nodes denote the activated peers with respect to C_u (manually preprogrammed by us) at each time step. Note that although at time steps starting at 13 and ending at 18, there are subjects (uncolored) among peers who have not adopted C_u, their selections (which may not be C_u) are visible to u. However, which users among the peers have been preprogrammed manually is by default unknown to the target subject u.

https://doi.org/10.1371/journal.pone.0234875.g004

Accordingly, A_u(t) would be different for users in each group, for e.g. for a u in LC group, A_u(t = 1) = 1, A_u(t = 2) = 2 while for EC group, A_u(t = 1) = 4, A_u(t = 2) = 5 and so on. An example of the linear cascade setting is shown in Fig 4, where a participant receives social signals from its six neighbors—our influence decision C uniformly cascades through the peers of the participant.

At Time Step 13 (start of the second phase), a signaler u selects a sub-optimal provider C_u, and over the next five time steps, the remaining peers adopt the same influence decision one after another. Note that although we program only selected peers (bots) of a subject to administer C_u over time, the subjects are able to view all of their peers’ decisions in their dashboard for the last six time steps. The rest of the non-controlled peers at a time step show random technologies to the participant. Also, note that in all conditions, users can switch back to any choice in the next time step after having selected an option in the current time step. We considered the NM and UM groups as our baseline groups and LC, EC, DC groups as our treatments groups of interest.

For both the hypotheses H1 and H2, the outcomes of interest are the decisions made by participants in the last six time steps, in the presence of social signals from peers. We explore whether decision-makers will be more likely to choose cyber-defense providers which are not the optimal choice when they have knowledge about the utilities and when they observe peers opting for choices which are not optimal. We note that people get feedback about their choice on the very next screen, and so choosing a technology during an intermediate time step is more of a data-gathering exploration rather than their final choice. In order to allow for this initial bandwidth for exploration, we keep the first 12 time steps (time steps) the same for all subjects devoid of any interference This helps in overcoming bias related to an individual’s own knowledge about the utilities in the second phase of the experiment when they are treated to social signals.

4.2 Participants

We recruited a total of 357 participants for this study to play the same cyber-defense provider game. Based on the responses provided by the participants regarding their demographics in the form of a survey response 1 Some participants declined to provide a response regarding their demographics, we have 151 females and 190 males in the study. Most of our participants are between the age group of 26-35 years and full details regarding their age groups have been provided in Tables 1, 2 and 3 in Appendix A in S1 Appendix. Additionally, majority of the participants in our experiment had a Bachelor’s degree (See Appendix for more details). Participants were paid $2 with the opportunity to earn up to $4.52 since as mentioned before, they received a bonus of $0.02 for every attack they prevented. Thus, the participants might have motivation to prevent more attacks in order to earn more money. Some of the participants did not complete the full length of the game and so we observed slight discrepancies among the group sample sizes for the statistical tests.

5.1 Distribution of attacks prevented

Table 2 shows the distribution of attacks prevented by subjects in each group. We observe that, on average subjects in the EC and LC groups prevent less attacks compared to others. However based on two sampled t-tests, we did not find any statistically significant differences between the groups based on the means of the distributions. Based on a survey analysis, we found that none of the traits like computer anxiety, computer confidence, computer liking, intuition or neuroticism were correlated to the number of attacks prevented in all groups. The details of the survey analysis are presented in Appendix A in S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Average number of attacks prevented by subjects in each group.

The lower attack numbers suggest participants deviated more from the optimal decision responding to social influence.

https://doi.org/10.1371/journal.pone.0234875.t002

5.2 Distributions of decisions by individuals

As a first step towards investigating hypothesis H1, we analyze the kinds of social signals or the cyber security technologies (which were not the optimal technology) chosen by the peers of each participant and whether they are uniform across all the groups. To simplify nomenclature hereon, we denote the six available technology choices as decision d_i, i ∈ [1, 6]. In our experimental setup and for this work throughout, we would refer to d₁ as the influence decision—It is the optimal choice preventing seven attacks, while the rest of the technologies prevented five attacks and are termed as sub-optimal choices. As mentioned in Section 4.1, the influence decision C_u ∈ [d₂, d₆] is randomized for each u, so we first observe the distribution of the sub-optimal decisions as the choices for C_u. For each group, we define as the proportion of users in the group who were administered technology or decision d_i, i ∈ [2, 6] as the influence decision in the second phase. From Fig 5, we observe that the random selection of C_u introduces some disproportionate values of P(d_i) among the groups. For the UM group, around 25% of users were administered d₄ as the influence decision C_u (this proportion P(d₄) for UM is the highest among all other decisions) while 29% of users in the EC group were sent d₅ (P(d₅) being the highest for EC) as their infuence decisions C_u and 28% of users in the DC were sent d₅ (P(d₅) being the highest for DC). However, we see that for the LC group, P(d_i) was similar for all decisions d_i that could be selected as C_u.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. P(d_i)—Proportion of users in each group who were administered technology or decision d_i as the influence decision.

Note that only the decisions that are not optimal are sent as prospective influence decisions/social signals in the second phase of the experiment.

https://doi.org/10.1371/journal.pone.0234875.g005

Having observed that there was not one pre-programmed peer choice C_u as the strategical obvious sub-optimal choice across all groups, we proceeded with investigating H1. In order to detect any implicit occurrence of a selection bias over the participants, we analyzed whether there is any significant difference in the groups with respect to choices made in the first 12 time steps. To accomplish that, we plotted the probability that an individual makes each decision when aggregated over the first 12 time steps. We found that there is clearly no evidence of differences in the mean statistics of the distributions of all choices between the treatments groups (LC, DC, EC) and the control groups (UM, NM) (Refer to Fig 1 in Appendix B in S1 Appendix). Before we conducted pairwise t-tests to check for differences between the control and treatment groups for the decision distributions made by the participants in the second phase, we conducted three one-way ANOVA tests considering the sample means of the two control groups and each treatment group, one at a time. We conducted these three tests for each decision from d₁ to d₆. The null hypothesis for each test constituted the situation where the means of the number of times a decision was chosen by the participants belonging to the two control groups and one of the treatment groups, are the same for the technology or decision in consideration. We find the following significant results: for the LC group, we find that for d₄, the null hypothesis is rejected (F(2, 203) = 3.7, p = .03). Similarly for the DC group, we reject the null hypothesis for d₄ as well (F(2, 205) = 3.01, p = .04) and for the EC group, we find differences approaching statistical significance for d₅ (F(2, 195) = 2.99, p = .05). We will come back to this case of EC group shortly while discussing the differences. These tests shed light on the differences in adoptions between the control and treatment groups that occurred in the second phase of the experiment in the presence of social signals.

Following this, we conducted two sampled t-tests to measure the differences in the distributions of decisions adopted by users between pairs of control and treatment groups. We conducted two sets of tests for the two phases of the experiments. The null hypothesis constituted the situation where the means of the decision selection distributions of the two groups being tested for, are not different. The results (p-values for each treatment group with respect to the control groups) in Tables 4, 5 and 6 in Appendix C in S1 Appendix suggest no significant difference in the distributions among the groups. This rules out any bias among the participants themselves in the absence of externalities. However, in the second phase of the experiment (time steps 13 to 18 aggregated), we find differences in the selection patterns among the decision-makers in their respective groups. We find the following observations from Fig 6(a) and 6(b) for our treatment groups (see tables in Appendix C in S1 Appendix):

1. LC: With respect to the NM group, there are no statistically significant differences in the decisions taken by the participants in LC. We carried out a similar statistical test comparing the group pairwise means as done for the first 12 steps. On the other hand, we find that there is a statistically significant difference for the LC group participants (M = 3, SD = 2.4) in the means of the distributions compared to the UM group participants (M = 3.8, SD = 2.53) for the optimal decision or d₁ (t(149) = 1.9, p = .04) at α = 0.05. The difference shows that a significantly reduced number of participants are tempted to choose the optimal technology provider in the presence of linear cascading signal pattern than when a single signal is sent across all time steps.
2. DC: When considering the number of times participants choose d₄, we find that the participants in the DC group (M = 0.58, SD = 1.08), differ from the UM group (M = 0.30, SD = 0.61) and this difference is statistically significant (t(155) = −2.02, p = .04). However, with respect to NM group or UM group participants, we observe that the users do not differ in their selections when it comes to choosing the optimal provider. Also it does not differ with respect to the most common choice among the peers for DC group—the Decision 6 shown in Fig 5.
3. EC: When we considered the choice of d₅, the technology that was administered to the majority of the EC group participants, the users in this group (M = .79, S = 1.08) differ from the NM group (M = 0.38, SD = 0.66) in their selection and the difference is significant (t(128) = −2.7, p = .007). We found similar significant results from our ANOVA tests prior to this analysis and this is a successful case of social influence when considering the macro-adoption process for the group as the users not only steered away from the optimal choice, but they also steered towards the decision of their peers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Probability of decisions made in time steps 13 to 18.

(a) Probability of making the optimal decision. (b) Probability of making the sub-optimal (other 5) decision. The error bars denote standard error over the distributions.

https://doi.org/10.1371/journal.pone.0234875.g006

This suggests that while the social signal to some extent influences an individual in the LC group to deviate from the optimal choice, it does not always translate to the influence decision that was chosen for the corresponding individual. It rather gears the user towards more exploration. However, an early burst of signals in the EC successfully translates towards social influence wherein on aggregate majority of the users sway towards the influence decision more. As a side experiment to measure the degree of drift away from optimal decision, we also analyze the fraction of users who shift away from d₁ (the optimal) at each time step similar to what has been shown in Fig 7. However, we do not find any clear distinctions among the groups in terms of the fraction of subjects who move away from the optimal decision d₁ when aggregated over all the six time steps in the second phase. Just the fact that all users eventually move away from the optimal decision when exposed to social signals does not contribute much in distinguishing the PoI.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Fraction of users in each group adopting the influence decision chosen by their peers.

https://doi.org/10.1371/journal.pone.0234875.g007

5.3 Degree of influence

This first analysis of H1 does not shed any light on the temporal variations in the decision making process exhibited by users in different groups. It shows some statistically significant differences in the choices made for specific decisions (or technology providers) over the entire second phase. While it did show that not all cascade patterns successfully influenced users towards deviating from “decision1” or the optimal provider, it brings up the question that is posed for hypothesis H2: what constitutes successful influence and does the manner in which the signals are sent determine successful influence?

To this end, we analyze how the proportion of users who switch to the influence decision ignoring the optimal choice, evolves over the times steps. We note that for each user u, the influence decision C_u is randomly selected before the second phase starts. Fig 7 shows the fraction of users in each group adopting the influence decision from time steps 13 to 18, when the experimental participants observe their peers’ decisions. From the observations regarding H1, we find that the probability of successful influence for participants in EC has the strongest effects on decision-making in the early stages of the second phase. Participants in EC are most likely to deviate from selecting the optimal provider as shown in Fig 6—for the first three time steps in the second phase, participants in the EC group exhibit the maximum adoption compared to other groups denoted by a higher fraction of adopted participants. Additionally, for the EC group, Time Step 15 had the maximum retention where 35% of users adopted their peer behavior, and when participants were exposed to all six of their peers selecting the same technology C_u. This may be due to new users adopting the influence decision or due to cumulative build-up from previous time steps who do not switch back. We will explore this in the following sections.

The participants in the DC group exhibit successful response towards the sudden increase in signals at Time Step 16 which is shown by a 65% increase in user adoption of the influence decision compared to the previous step. These results become close to the 25% of users making influence decision selection at Time Step 16 and which is also the maximum among all groups surpassing the early cascade adoption ratio. However, there is no substantial increase in the adoption fraction for the users in the linear cascade group—we do note that the adoption peaks at Time Step 15 for the linear cascade users before it drops again. These observations from Figs 6 and 7 suggest that while an early burst of social signals successfully persuades users in EC to adopt the influence decision, causing an aggregated overall maximum selection of the influence decision, a sudden impulse in the quantity of social influence also successfully steers users towards the influence decision in the later stages.

5.4 Measuring the effect of quantity of signals

In this section, we investigate whether the quantity of signals alone stand out as the sole factor of influence. Before going into the analysis, we define a few notations: we denote a subject in this study as u where u can belong to any group. We denote the decision taken by a subject u at time t, t ∈ [1, 18] by D_u(t). We define T_treat as the sequence of time points during which the subjects receive signals from their peers i.e. T_treat = [13, 18]. Following this, we denote the time step t, t ∈ T_treat when an individual u first switches to C_u as . For each signal quantity s, we measure the proportion of individuals (in each group) who made their first switch to their influence decision only after they were exposed to s signals. Formally, for any t ∈ T_treat, (R(1) in the LC group denotes the proportion of individuals in LC group who made their first switch to their peer influence decision after being exposed to just one signal. Similarly, R(6) in the EC group denotes the proportion of individuals in EC group who made their first switch to their peer influence decision only after being exposed to six signals, so this can happen in any of the last four time steps in EC). The denominator in the formula here denotes the number of individuals in the group.

We bin the values from R(s) based on s and take the mean for each group, since for some groups, there can be multiple time steps with the same number of exposures or signals. From Fig 8(a), we observe that for the EC group, ∼30% of users under the influence of four signalers, for DC, ∼18% of users at four signalers and for LC, ∼15% of users at three signalers (all these being the maximum ratio) made their first switch to the influence decisions in the second phase of the experiment. However, on close observation, we find that the number of exposures alone does not explain the adoption behavior. When we compare the EC participants with those in the DC group, we find that with four exposures in T_treat (at Time Step 13 for EC and at Time Step 16 for DC), the proportion of adopters in EC making their first influence decision switch (30% of users) is higher than the proportion in DC (18% of users), although the same four quantity of signals are delivered at different time steps for the two groups. However, while there is a constant decrease in the number of adopters making first switches in the EC group going from four to six exposures denoted by the corresponding mean R(s), we see that the same quantity for the DC group does not decrease the same way for four to five to six exposures. This suggests that the sudden stimulus from the delayed exposure somehow succeeds in influencing more users to make their first switch to the influence decision compared to the EC group (note that all the users under the quantity R(s) for each s are unique since we measure their first switch). On the other hand, for the linear cascade group we do not find one quantity that is most effective in the influence. In the LC setup, there is no one exposure that impacts the adoption behavior the most in terms of successful influence. In addition, we measure the cumulative adoption ratio for each group defined as: for any t ∈ T_treat, . In simple terms, it measures the number of individuals in a group who adopt the influence decision under s exposures irrespective of whether it is the first switch. This is demonstrated in Fig 8(b). When we combine the results obtained in Fig 8(a) with Fig 8(b), we find an interesting observation for the LC group. The linear cascade pattern is able to retain most of the users even after first switch at Time Step 13 as the cumulative ratio increases up to three exposures (which occurs at Time Step 15). This suggests that the LC pattern is effective in terms of retention in the early stages of the cascade.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Plots of adoption under the influence of s exposures.

(a) For each signal quantity s, the proportion of individuals who made their first switch to their influence decision after being exposed to s signals, (b) The proportion of individuals who adopt their influence decision after being exposed to signals from s influence signals.

https://doi.org/10.1371/journal.pone.0234875.g008

5.5 Quantitative analysis using growth modeling

To understand the patterns of change over time by incorporating the heterogeneity among individuals in each group and among the groups, we resorted to the widely used statistical tool of growth modeling through random coefficient models [39]. Briefly, the technique of growth modeling allowed us to test the longitudinal effects of the peer signals on the users and test any source of heterogeneity in decision making among individuals that lead to the observed outcomes. One of the advantages of growth modeling comes from treating time as a predictor of influence outcome in the absence and presence of the peer signals. The error-covariance matrix from these models informed us about the variations among the individuals in the presence and absence of peer signals over the phases of the experiment.

We utilized four regression models where the outcome of interest denotes whether an individual selected the influence decision C_u(t) at time t in the second phase of the experiment. Let the linear predictor, η be the combination of the fixed and random effects excluding the residuals. Considering a linear mixed effects model LMEM y = X β + Uχ + ϵ, where , (and k = n * [t]) denotes the outcome response of individuals over all time points t ∈ [1, [t]] − k thus denotes the number of instances in the model. In this work, we have [t] = [T_treat] = 6. denotes the design matrix of endogenous variables, f being the number of predictors, denotes the matrix with random effects (the random complement to the fixed X), w being the number of predictors with random effects, ϵ denotes the residuals, and β and χ are the parameters of interest corresponding to the fixed effects and the random effects in the model. Since the outcome in our work is binary, we used Generalized Linear Mixed Effects models (GLMEM) [40] in this work. In a GLMEM model, we used the same equation as LMEM but with a linear predictor η such that η = X β + Uχ + ϵ where g(E[y]) = η, g(.) being the link function. To accommodate for the binary outcomes, we used the logit link function. To quantify the effects using growth modeling, for each individual, we used the following regression models for modeling the growth functions: (1) (2) (3) (4) where the indices i denote the observation instance number (or the row number in a table of data) and j denotes the individual in the group of all users. So, M0-FE represents the fixed effects model where the only independent variable is the time. Intuitively, this model tests how individual responses to peer signals evolve over time, when time itself is the only factor in consideration. From the results in Table 3, we find that among the three treatment groups, the probability of outcome is most positively correlated with time for the Delayed Cascade group. This simple regression model ignores the fact that the observations are nested within individuals and accordingly, the next step in growth modeling is to add a component of random intercept to the model—this is denoted by M1-RE. Note that in this model, the random intercept χ_0j is specific to each individual j. On comparing the parameter estimates, we find that the time coefficient β₁₀ remains similar for both models even after incorporating between-person differences for all the groups. This shows that time is significantly correlated to the outcome in the absence of the knowledge of peer signal treatment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results of Fixed Effects (FE) and Random Effects (RE) modeling.

“Inter”. denotes intercept in the regression models. Values in brackets denote standard errors.

https://doi.org/10.1371/journal.pone.0234875.t003

Next, we added the “Time” variable to the random components, so that time could randomly vary among the users. This is denoted by the model M2-RE given by Eq 3. The correlation between the slope and intercept for this model for the LC, DC and EC groups are respectively 0.7, 0.7 and 0.4 respectively. The positive correlation indicates that individuals who have a high propensity to move towards the influence decision in the beginning tend to have strong slopes—this weakly suggests that individuals with some degree of uncertainty towards the optimal decision at the beginning tend to be more susceptible to influence over time. Next, we explored the idea that “Signals” have a role to play on the outcome of the individuals. We added to M2-RE, the fixed effects from Signals shown in M3-FE. From the results in Table 3, we found that for all the three groups, the number of signals is positively correlated with the outcome while the time factor is negatively correlated for the groups. Among all the groups we found that for the DC group, the Signals factor has the highest coefficient suggesting the strong correlation between the treatment with signals and the outcome. These results suggest a positive correlation of the treatment of signals on the outcome when time has an important role to play. The interaction between time and signals are important here given that the correlation of time changes once the effect of signals is considered.

5.6 Interplay between influence and susceptibility

We ended this study with a retrospective analysis to understand the dynamics of adoption under a slightly relaxed setting. In real-world networks, not all individuals are susceptible to social change emanating from their neighborhood—some people have stronger beliefs than others [41]. In an attempt to quantify the effect of the influence on subjects in a more constrained setting, we considered only those users u who have been influenced to adopt the influence decision C_u at least once between time steps 13 to 18 in the second phase. We measure at every time step, the ratio of individuals who adopted the influence decision at that time step to the number of individuals in the group they belong to, i.e., who switched to their influence decision at least once within their lifecycle (T_treat). Note that this is different from previous measures in two ways: first we retrospectively filter out users who never adopted their influence decision (in the real world these are users who would not be susceptible to influence or are immune as such). Second, we analyze this ratio at the end of their exploration phase, in Time Step 18, when everybody has supposedly settled down. We define a symbol N_u as the number of time steps for which a user u adopts C_u in T_treat (this is measured retrospectively aggregating all time steps beforehand). Formally it is defined as: . The denominator denotes the number of individuals who have adopted the influence decision at least once from Time Steps 13 to 18. The comparison shown in Fig 9 among the four groups (the No message group does not have any influence decision) demonstrates that while the EC group adopts the influence decision more quickly than other groups, the stimulus in signals quantity at Time Step 16 in the DC group affected the participants. This is confirmed when the effects of DC strongly outstrip those observed from EC in the last time step where both groups receive six signals. At the end of the game, at Time Step 18, we find that the highest number of such susceptible individuals come from the DC group—these results reinforce some of the conclusions we had from Section 5.3 and Fig 7 regarding the late retention capabilities of the DC group strategy—however, this measure makes the differences clearer when we consider individuals who are more prone to social influence.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Success ratio.

https://doi.org/10.1371/journal.pone.0234875.g009

6.1 Simulation setup for ABM

There are two ways in which we can model agent behavior to simulate behavior diffusion—the single agent behavioral model which predicts individual behavior when individuals are observed in isolation and the multi agent behavioral model that extends the single agent behavioral model to a population level by executing the simulation in a multi-agent environment. In such scenarios, since the agents influence each other through their own actions over a period of time, the behavior of individuals can no longer be measured without taking the environmental factors into account. Fig 10(a) shows the agent-based model for a single agent where an agent makes a decision based on its utilities at every time step. Once the utilities have been determined, the agent picks a decision corresponding to the maximum utility. Once the agent has acted, the environment is updated and external factors that might also impact the agent’s decision in the next step is accounted for. Prior to the next time step, the utilities of the choices are updated based on the previous decision. However in this lifecycle of the decision making process, the agent is observed in isolation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Agent based models for measuring social influence with multiple choices.

(a) Single agent behavior model, (b) Multi-agent behavior simulation.

https://doi.org/10.1371/journal.pone.0234875.g010

In our work, we used this single-agent behavior model as a generative element to simulate agent behavior but we observed the behavior at a population level especially when the agents influenced each other by making actions and by virtue of being in a networked environment. In order to simulate and evaluate the effect of social influence through multiple exposures in the real world, we chose a specific real world case study to implement the environment. As shown in Fig 10(b) which represents the multi-agent behavioral model lifecycle, agent behavior is simulated using the single agent behavior model in Fig 10(a) with specific input parameters for the agent which would be learned from the empirical data. We ran the ABM and evaluated the results from the ABM using different training and test splits but from the same distribution of the empirical data. The agent specific parameters were learned prior to the start of the simulation iterations since we mapped the agents to real world users. Although we did not use feedback from the evaluation obtained from one run of the simulation from the real world to optimize the simulation environment for future runs, this step can be additionally performed to obtain monotonically increasing evaluation performance. In our environment, each agent has a choice to reshare a piece of information in the situation where resharing is not the optimal choice. The following sections describe the agents and their choices in details.

6.2 Agents

In our multi-agent simulation environment, the agents and the users in the social network of the real world are one-one mappings and so by default the agents are embodied in a networked environment where they can share and be exposed to others’ messages. We modeled a social network as a directed graph N = (V, E), where a node v ∈ V represents and individual and the edge (u, v) ∈ E exists if u follows v, in our case v influences u’s decision. So in the context of this social network, our agents are the nodes in these networks. Each agent can play both the role of a neighbor influencing a user or a susceptible user who is being influenced. For each agent u, as mentioned before, we considered the set of agent’s peers to be homogeneous with respect to the influence they exerted on u. In the context of the ABM, we separately modeled the probability that each agent was being influenced based on factors that are going to be discussed in the next section.

6.3 Modeling agent behavior towards sub-optimal decision making

Instead of considering multiple choices for the agent decision stages, we considered a binary choice model where at each time step, each agent had to make a selection among two choices which are reciprocal to each other, and the selection is based on the choice that comes with the higher probability of activation. In the real world case study used for the simulation, the action of resharing a piece of information is a sub-optimal choice and so the peers of an agent who reshared the same piece of information exerted influence towards sub-optimal decision making.

Also, note that this is a simpler version of the online controlled experimental setup where the user had five choices. One of them was optimal and our setup could be extended to include multiple choices for a relevant real world case study. Following this, the agent could only be in two states based on the choice they made, that is they have either reshared a piece of information identifying the cascade or they have not. Additionally, as in most real world adoption scenarios with a binary choice model, the user cannot transition back to the state it arrived from. Before delving into the technical details of the components for the agent utilities, agent states and the exposure effect based ABM, described in details below is how we quantified the individual decision factor and the peer influence through exposures.

6.4 Learning model parameters

With the models of activation stated as above, we set the parameters of the model in the simulation procedure. We specifically worked with information cascades representative of real world diffusion [47] while discussing the dataset. Since we considered rumor diffusion as the real world study and we calibrated the parameters of the models to this dataset by splitting it into training and evaluation sets as is prevalent in machine learning setups. Specifically, we started the behavior diffusion simulation of the agents after observing part of the diffusion cascades till T_thresh. That is, we first observed the cascades from their beginning to a specific time span T_thresh for learning and left the rest of each cascade after T_thresh to be used for evaluation of the ABM. This helps us in two ways: first, it allowed us to perform simulation with the assumption that the agents had the time to form some opinion of their own using the exploration strategy as setup in the controlled experiments. Second, it allowed us to learn the parameters specific to each agent in a data-driven way prior to start of simulation and allows us to perform evaluation of the ABM based diffusion process after T_thresh.

We treated the latent utility factor x_u(t) as a parameter of interest and considered that the agent’s individual decision utility was fixed. We considered x_u(t) = x_u for all time steps t ∈ [1, T]. Specifically, the parameters of interest specific to agents are θ_u = {x_u, η_u, β_u} and the parameter σ which we set to a constant during our evaluation. Since it is not easy to map the controlled experimental environment to situations in observational studies, we did not consider agent histories, so instead of learning individual agent parameters {x_u, η_u, β_u}, for each u, we divided all the agents into L latent groups. This also captured the notion that agents in a connected network belong to a latent block structure or specifically stochastic block models [48]. In a stochastic block model, each agent is assigned to a block and the pattern of influence between different agents depends only on their block assignment. Following this, the overall probability that u belongs to class l ∈[1, L] is given by p_l = P[l_u = l] with . So all the individual agent specific parameters {x_u, η_u, β_u} are now replaced by {x_l, η_l, β_l}. Let θ_l = {x_l, η_l, β_l} be the parameters specific to the latent class l. Denoting z_u(t) = 1 if the agent reshares the message (sub-optimal decision) and 0 otherwise, Z_u = {z_u(t)}, ∀t ∈ [1, T], the likelihood contribution of u belonging to latent class l is then given by: (12)

Denoting the set of parameters Θ = [θ₁, …, θ_L], P = [p₁, …, p_L] and A_u = {A_u(t)}, ∀t ∈ [1, T], the likelihood of the model is then given by: (13)

So our log-likelihood is: (14)

We attempted to compute the posterior distribution of the parameters given the observations: (15)

Denoting W_u = {A_u, Z_u} for all agents u, we first attempted to compute the posterior distribution of p_l,u = P(l_u = l|W_u), given the observations. And formally it is given by: (16)

The lower bound log-likelihood following Eq 14 takes the form (17)

Taking the derivative of ll with respect to x_u and keeping other parameters fixed, we get (18)

The derivative of p_u(t) considering the base model BM with respect to x_u keeping other parameters fixed is (19)

Similarly, the gradients with respect to other parameters η_l, β_l and p_l can also be calculated and the parameter updates in the M step can be performed via gradient descent using the following procedure. This is a standard finite mixture model where the parameters are estimated by the Expectation Maximization (EM) framework [49]. The brief steps to obtain the parameter estimates are as follows:

6.5 Dataset

We used the Twitter dataset released publicly by authors in [50] which analyzed how people orient to and spread rumors in social media. As discussed in that study, adapting the existing definition to the context of breaking news stories, a rumor is defined as a “circulating story of questionable veracity, which is apparently credible but hard to verify, and produces sufficient skepticism and/or anxiety so as to motivate finding out the actual truth”. The tweets from that study were collected from the streaming API relating to newsworthy events that could potentially prompt the initiation and propagation of rumours. Selected rumours were then captured in the form of conversation threads. The authors used Twitter’s streaming API to collect tweets in two different situations: (1) breaking news that is likely to spark multiple rumours and (2) specific rumours that are identified a-priori. They collected a total of nine events pertaining to these situations but we used the threads from three events in this paper. The twitter threads were related to the following three events:

• Charlie Hebdo Shooting: Two brothers forced their way into the offices of the French satirical weekly newspaper Charlie Hebdo in Paris, France killing 11 people and wounding 11 more, in January 2015.
• Ferguson unrest: The citizens of Ferguson in Missouri protested after the fatal shooting of an 18-year-old African American, Michael Brown, by a white police officer on August 9, 2014.
• Putin missing: Numerous rumors emerged in March 2015 when Russian president Vladimir Putin did not appear in public for 10 days. He spoke on the 11^th day, denying all rumors that he had been ill or was dead.

Since the publicly released dataset only contained a sample of the threads as compared to the those used in [50], we picked these three events which had relatively larger proportion of rumor threads among all the events. The authors in the study curated the annotations for the threads as to whether they were rumors or not with the help of several journalists. We considered all the threads for the events which were tagged as rumors such that the stories related to these threads were later verified as false. Fact checking for these threads were performed by a group of annotators after these events and while the entire event might have been later described as true, there were threads related to those events that spread misinformation. The statistics of the data pertaining to these three events is provided in Table 4. For each of the events, the dataset provides us with the following segregated information modules:

1. Initialize the parameters Θ, P and evaluate the log likelihood of the model using Eq 14.
2. E-Step: Evaluate the posterior probabilities using the current values of Θ, P, with Eq 16.
3. M-Step: Update the parameters Θ, P with the current values of the posterior using the gradients obtained through maximization of the log likelihood with respect to parameters.
4. Evaluate the log-likelihood with the new parameter estimates. If the log-likelihood has changed by less than some small ϵ, stop, else reiterate Steps 2 and 3.

Algorithm 1: Simulating the diffusion of cascade q based on social influence and individual decisions.

Input: T_thresh, Activated Set A (till time T_thresh), Time limit T_sim, Θ, P, σ, users V_q, G, Model Type MT

Output: Diffusion Node Set DF_q[t], ∀t ∈ [1, T_sim]

activated ← A

DF_q[0] ← {}

for t = 1 to T_sim do

 curr_activated ← {}

 DF_q[t] ← DF_q[t − 1]

 for each agent u∈ V_q\activated do

  l_v ←

  /* Calculate individual factor μ_u(t)       */

  ϵ ∼

  compute μ_u(t) with Eq 6 using ϵ and

  /* Calculate utility from influence ζ_u(t)     */

  if MT == BM then

   compute ζ_u(t) with Eq 8

  else

   compute A_u(t) using G with Eq 9

   compute ζ_u(t) with Eq 8

  /* Calculate probability of activation p_u(t)     */

  p_u(t) ←

  if p_u(t) > 0.5 then

   DF_q[t] ← DF_q[t] ∪ {v}

   curr_activated ← curr_activated ∪ {v}

 activated ← activated ∪ curr_activated

return DF_q

1. Who-follows-who network: This social network is sampled to cater to specific users who participated through either replying to tweets or retweeting that specific event. This allowed us to focus our simulation for each event using this network N instead of using one large social network common to all events. It helped in part by enabling us in the evaluation to compare our set of activated individuals at each time step to the actual users who were activated.
2. Retweet cascades: We considered retweet cascades and did not include users who simply replied to a particular tweet since it is challenging to deduce whether a user agreed to the agenda of the cascade while replying—the notion that induces a cascade of like minded individuals. So we restricted users in the cascade to those who only retweeted the source tweets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Statistics of the data used for simulation relating the 3 events.

https://doi.org/10.1371/journal.pone.0234875.t004

We operationalized our simulation of behavior diffusion described in details in the next section, for each event separately. We used the follower networks for each event to simulate the diffusion and used the retweet cascades to learn the agent parameters specific to each event.

6.6 Simulation algorithm

We now describe the algorithm for operationalizing the simulation of behavior diffusion based on the influence setup described in Section 6.3. As mentioned before, agents refer to users in the social network and a one-to-one mapping to the actual network of the events from the dataset. We used the follower networks relevant to each event, which are directed in its edges and we refered to such networks as G. The algorithm for the agent-based model for social influence based diffusion process is described in Algorithm 1.

The diffusion simulation unfolds in discrete time steps and at each step, multiple agents can change their states (initial state being the state where the user/node has not reshared the message)—however, once they transition from non-shared to shared state, they cannot switch back. We observed the users who participated in the first T_thresh steps of the cascades and used that to learn the parameters, specifically the latent class l specific parameters θ_l, p_l for all classes l ∈ [1, L]. We then use the activated nodes (which have already reshared the rumor prior to T_thresh) along with these learned parameters as input to the simulation algorithm. From thereon, we ran the algorithm for a span of T_sim discrete time steps. For each t ∈ [1, T_sim], the algorithm outputs the number of agents who reshared the rumor message at time t or were activated at time t. In each step t, we looped through all the agents in the network that had yet to reshare the message. Since we projected each agent into a mixture of latent classes L, prior to the simulation step, we needed to categorize each agent into one of the latent classes in order to compute p_u(t) with the respective parameters of the latent class they belong to. We observe D_u(t), A_u(t) of each user u till T_thresh in the real world data and then the latent class can be decided by the maximum of posterior (Eq 16) from the data. Then the probability of activation is computed based on Eqs 10 and 11 depending on whether the base model or the augmented exposures model is used (the algorithm mentions the general form of the equation based on Eq 5). If this calculated probability is higher than 0.5, the agent is activated and the simulation continues for other agents for that time step.

6.7 Evaluation of ABM

One of the specific goals we had while setting up the ABM was to compare the two models we proposed—the base model which relies on the magnitude of peer signals as the factor for social influence and the augmented model which allows for accommodating the sequential changes in peer signals. To this end, we compare the diffusion trajectory of the cascades from simulations based on each model and the real world data. Note that all the parameters of the models represented in Eqs 10 and 11 are learned from the data, except for the noise random variable η that adds uncertainty to the individual utilities. We sampled ϵ and following this, we executed 100 runs of the simulation algorithm for each cascade (thread) for each event in the data to account for this uncertainty. For learning the parameters as mentioned before, we observed the cascades for each event till time T_thresh. Since the time span of resharing actions for each cascade is different and in the absence of any normalization procedure that could be applied to decide on a single T_thresh for all cascades, we instead observed the first 40% of each cascade (in terms of total number of reshares in the cascade) in the chronological order of reshares. This allowed us to keep the T_thresh dynamic for each cascade while allowing the rest of the cascade to be used for ABM evaluation.

To evaluate the simulation results, for each cascade q, we consider the set of users for each network relevant to the event, such that all users in V_q reshared q in the time span of the cascade. We ran each simulation for T_sim = 20 steps. For each time step t in our simulation, we computed the following metric for each cascade: , the number of actual activated users which are also part of the activated users from the simulation algorithm at time t. We call this measure the True Diffusion Rate of our simulation algorithm. The metric does not measure prediction results here since there is no way in which we can precisely map a time step in our simulation to a numeric time interval in the real world dataset. The metric allows us to measure recall over the users who reshared the rumors while allowing us to simulate the trajectory of the diffusion over time. Fig 12 show the results of the ABM simulation for the two simulations. For each event, we ran the 2 models as described in Section 6.3. So we have a total of six models and we learned six sets of parameters and run 100 simulations for each model that learn the parameters of the ABM.

Fig 12(a), 12(b) and 12(c) show the plots corresponding to the true diffusion rate over time and it compares the two models: the BM model where the peer influence at time t is characterized by the number of neighbors of a user who have reshared the rumor message till t and the AEM model where the peer influence at time t is characterized by the augmented exposures till t given in Eq 9. For all the results, we plotted the mean of the True Diffusion Rate at each time step taking all cascades for the respective events into account. Below are the results from the Charlie Hebdo and Putin missing events followed by the Ferguson arrest events:

Charlie Hebdo and Putin missing: For the cascades related to both of these events, the AEM model which takes into account our notion of augmented exposures, exhibits a faster diffusion rate compared to the BM model. The results are not surprising in this scenario since the additive factor of influence coming from the spike in neighbor information in the AEM model augments the utility from social influence resulting in faster diffusion. We also observed that for the cascades in the Putin Missing events, the diffusion rate for the AEM model is an order of magnitude higher in the initial stages until seven time steps and the AEM model reaches the saturation point of the curve faster than the BM model.

On closer analysis, we found that the network structure in this case has an important role to play. The degree distribution for the network used for the Putin missing event displayed in Fig 11 shows that there are only a few nodes with high in-degree or the number of potential peer signals. So the spikes in the adoption curve for the Putin missing event happens when these few nodes with high in-degree (or higher potential of exposure to peer siganls) are exposed to the message from multiple peers and they reshare the message. This happens earlier for the AEM model than the BM model. Consequently, this result shows that faster diffusion in real world networks can often be attributed to the presence of a few nodes that are more susceptible to exposure, resulting in faster adoption at a population level. It shows how peer signals in the initial stages can drive diffusion faster for the rest of the trajectory.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Degree distribution of the follower networks.

https://doi.org/10.1371/journal.pone.0234875.g011

We also observed that in both of these events, the dynamics of adoption do not vary much in that at no point does the adoption rate induced by the BM model surpass the AEM model. This also suggests that when population-level adoption is faster shown by the fact that in both cases, almost 70% of the actual users who reshared the message were activated in our AEM model by Time Step 7, user uncertainty does not account for much and the peer signals drive the dynamics.

Ferguson arrest: For the cascades belonging to the Ferguson arrest event, we fund that in contrast to the results discussed above, the AEM model resulted in slower diffusion than the base model after time step 4 as shown in Fig 12(c). On closer analysis, the reasons behind this can be attributed to two key observations: (1) Slow initial diffusion prior to the start of simulation for cascades belonging to the Ferguson arrest event. It took an average of T_thresh = 25 and 31 hours for the cascades to reach 40% of their final affected population for the Charlie Hebdo and the Putin missing events respectively, while it took roughly T_thresh = 134 hours to reach the same 40% of the final size for the cascades in the Ferguson unrest events. This also led to the models setting high values of μ_u(t) from the learning procedure prior to start of simulation i.e. the user uncertainties were learned to be higher at the start of the simulation resulting in slower diffusion through the simulation given that it took 10 time steps to reach 70% of the population in contrast to seven time steps for the other two events. (2) The slower diffusion led to many nodes not experiencing any increase in peer signals, resulting in lower probability of activation over time, since the factor e^{A_u(t)−A_u(t−1)−1} could be less than 1 when A_u(t) = A_u(t − 1) i.e. there was no increase in peer exposures which happens to be the case in this situation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. ABM results for the simulation using Algorithm 1 on three Twitter who-follows-whom network for the events (a) Charlie Hebdo, (b) Putin missing and (c) Ferguson unrest.

https://doi.org/10.1371/journal.pone.0234875.g012

Consequently, the social influence factor dropped in AEM model after Time Step 4 compared to the BM model which explains the plot Fig 12(c). This suggests that in networked situations unlike our experimental environment, slower initial diffusion can result in initial higher uncertainty which can eventually result in the decay of the influence factor later on. In such cases delayed stimulus through interventions would be the only way for peer influence to play a bigger role which of course would be undesirable in such sub-optimal choice diffusion scenarios. As mentioned before, one of the limitations of this model lies in the setup that a user cannot transition back to a state it has explored. This limits us in measuring the retention effect concurrent to what we tested for the Early Cascade phenomenon. However we believe that our binary choice model can be extended to multiple choice data given relevant case studies. Tthis would then allow for understanding of the retention effect in the real world and whether early exposures are not an effective tool for retention over the long run as we concluded from our controlled experiment setup.