The four information theoretic deception models

The information-theoretic models of deception are derived from two important ideas in information theory, specifically Shannon’s idea of channel capacity and the notion of information-theoretic similarity between two messages. A brief outline of these two concepts is included in Appendix 1.

In the Borden-Kopp model of deception [48], four information-theoretic models are defined, Degradation, Corruption, Denial and Subversion, each of which is a specific form of altering the victim’s perception.

Two of these models involve manipulation of terms in Shannon’s channel capacity equation, one model involves manipulation of similarity, and one model involves the manipulation of internal information processing methods, effectively by altering some internal algorithm or process in the victim system.

The different labels employed for the models in early publications reflect the different paths Borden and Kopp followed in identifying the model initially, and should be properly considered as descriptive mnemonics for identifying the respective models. We do not use the labels employed by Bell and Whaley [3, 5]. Abbreviated labels based on Borden’s nomenclature for the first three models are employed in this paper, as Borden’s model conflates the Denial and Subversion models under Denial [42].

A player can apply any number of the four models, concurrently, or separately, to change the opponent’s perceptions to gain an advantage [49].

The Degradation deception model conceals or hides information in noise, or other background messages, to introduce uncertainty or a false perception in a competing player’s belief. This model exists in overt (active) and covert (passive) forms. In the overt form, the deceiver produces the noise signal with sufficient magnitude that it prevents the victim from reliably recognising arriving information, but alerting the victim to the fact that it is being attacked [35, 39].

In the covert form, the deceiver aims to make the message indistinguishable from the background noise of the environment.

An overt Degradation deception amounts to manipulating the noise term in Shannon’s capacity equation, such that N ≫ S and in turn C → 0, while a covert Degradation deception amounts to manipulating the signal term in Shannon’s capacity equation, such that S ≪ N and in turn C → 0.

Camouflage, concealment and hiding are covert forms of this model. Flooding a victim with non-sensical, redundant or irrelevant data to hide actual facts are overt forms of this model.

The Corruption deception model produces a false belief by replacing a real message with a similar, but false message, contrived to be very difficult to distinguish from a real message. The false message thus mimics a real message. Successful corruption is inherently covert, as the victim remains unaware that the information is misleading [35, 39].

A Corruption deception amounts to fabricating a deceptive message sufficiently similar to a real message, that the victim cannot recognise the difference, so S → 1 inside the victim’s cognitive system, where S is information-theoretic similarity. Any deception in which a falsehood is contrived to mimic a truth is represented by this model. An improved formulation based on information-theoretic similarity is described in Appendix 1.

The Denial deception model increases uncertainty by preventing the victim from collecting information by disrupting or damaging the means employed to collect information. This model is always overt, as the victim is aware that the means has been denied, either in a temporary or persistent manner [35, 39]. A Denial deception amounts to manipulating the bandwidth term in Shannon’s capacity equation, such that W → 0, yielding in turn C → 0.

A card player seating himself in front of a brightly lit window, so that his opponent cannot easily read any tells, would be an instance of this model. Denial of service attacks in the cyber domain are another instance of this model.

The Subversion model involves actions where the victim’s information processing method or algorithm is altered to the advantage of the deceiver. This model is commonly employed for deceptions, but also is employed by parasites to compromise the basic objectives pursued by the victim. Most known instances of Subversion are combined with an initial Corruption attack to first insert the self-destructive message into the victim’s cognitive or decision cycle [39].

Some of the best illustrations of Subversion are political or commercial deceptions using “spin”, where the victim is encouraged to change the manner in which they interpret a message, to the advantage of the deceiver. It is important to note that Denial via Subversion can be employed for purposes outside the scope of altering perceptions.

Fig 1 depicts the relationships between the deception models and the components of system they are employed to compromise.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The means of executing the four information-theoretic deception models.

The Degradation, and Denial deceptions attack the means of collecting information, which are the communication or perceptual channels. Corruption and Subversion attack the means of perception or processing, respectively.

https://doi.org/10.1371/journal.pone.0207383.g001

Fig 2 depicts the respective relationships between the deception models, when employed to produce deception effects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The effects of the four information-theoretic deception models.

The deception effect of increased uncertainty is produced by the overt form of Degradation, and by Denial. False beliefs resulting from false interpretation or false perception are produced by the covert form of Degradation, Corruption and Subversion.

https://doi.org/10.1371/journal.pone.0207383.g002

Earlier studies have shown the ubiquity of these models in all domains where information is employed to gain a competitive advantage. A large number of empirical instances of known deceptions across the domains of social systems, cyber and biology were tested against this model, and invariably found to map into one of the four models, or some combination of these [37, 38, 40, 42, 46, 50].

The four canonical models have a number of interesting properties. The first of these is analogous to atomicity, in the sense that since each involves manipulation of different parameters, there can be no simpler models for a player to employ [49].

The second interesting property is analogous to orthogonality, and arises because these models can be applied separately, or in arbitary combinations, by an attacker. The latter leads inevitably to compound deception models, where the victim might be subjected to multiple serial and parallel deception attacks, the intent of which is to drive the victim into a pre-determined internal state desired by the attacker. A compound model can then be modelled as a directed graph, in which states of belief in the victim are represented as vertices, and deceptions which change the state of belief as arcs in the graph [37].

An integrated framework for modelling deception

The central problems in games with deceptions arise from how the deception alters the victim’s beliefs, and how this in turn alters the victim’s decisions. The alteration of a belief is the effect produced by a deception. The information-theoretic models of deception map the deception into an effect.

Greenberg studied the problem of how deceptions impact players’ decisions in games. In Greenberg’s model a rational player will make the choice that maximises payoff, according to the subjective probabilities of payoffs for specific actions. These probabilities are derived from observations and prior beliefs, either or both of which may have been altered by a deception [23].

What the information-theoretic model of deception shows is that a player must make decisions when interpreting perceived inputs, before these inputs can be incorporated into the subjective model of the game, to estimate payoffs and risks, and to make decisions in the game. Decisions about perceived inputs are typically embedded in the perceptual and information processing mechanisms of the player, the behaviours of which are non-ideal and may or may not be readily altered by the player.

All four information-theoretic deception models are designed to defeat the mechanisms used by a player to develop correct or reasonable beliefs, which are subsequently employed to construct a payoff matrix for a decision, with the caveat that a Subversion deception may also alter the manner in which a player makes a decision, or acts upon the decision, as it may also alter utilities or decision algorithms.

The covert Degradation model is intended to transform a game of complete information into a game of incomplete information, by hiding facts, options or possible strategies from the victim.

The overt Degradation and Denial models are intended to introduce uncertainty into the victim’s decision process, to reduce the quality of the victim’s subjective probability estimates of payoffs or risks for specific actions. The victim knows that a deception is under way, but degraded or denied information produces uncertainty.

The Corruption model, and many instances of the Subversion model, are intended to introduce false beliefs by replacing facts, options or possible strategies with contrived alternatives, to the advantage of the attacker.

Seamless integration of the four information-theoretic deception models into the established game theoretic and decision theoretic constructs employed to model incomplete information, uncertainty, and false information, does present some practical challenges, mostly due to the immense diversity empirically observed in complex compound deceptions, the challenges of mapping perceptual models into subjective probabilities, but also due to the diverse foci in game and decision theoretic models, which may be oriented to understanding the strategies available, the specific decision, utility, payoff and risk models, or the possible equilibrium states, or lack thereof.

Fig 3 depicts the integrated framework for modelling deceptions. The deceptions produce effects, and these effects are employed as inputs to game or decision models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Deception framework.

The mapping of deception models into deception effects for use in game and decision models. This mapping provides an integrated framework for modelling deceptions.

https://doi.org/10.1371/journal.pone.0207383.g003

Experimental modelling of the “fake news” problem

The term “fake news” is the most widely accepted label for the empirically observed problem of the mass distribution of deceptive content across mostly digital media. Wardle aptly describes this term as unhelpful, as its conflates misinformation and disinformation of various forms in various media. The “Misinformation Matrix” defined by Wardle respectively maps seven means and eight motives for the production and distribution of misinformation, based on empirical observation of social and mass media “fake news” [47].

Lazer et al defined “fake news” as “fabricated information that mimics news media content in form but not in organizational process or intent.” and observe that “Fake news overlaps with other information disorders, such as misinformation (false or misleading information) and disinformation (false information that is purposely spread to deceive people).” [54]

Campan et al defined “fake news” in a manner closer to Wardle, mapping it into categories of clickbait, propaganda, commentary/opinion, and humour/satire, dividing it further into mis-information, where the propagating party is unaware of the falsehood, and disinformation, where the falsehood is known to be false [55].

Notably, the motives for “fake news” production and distribution always involve some profit or gain by the players involved, be it monetary, political, ideological or psychological. Misinformation is mostly employed to produce specific effects in the victim audience.

Many instances fall under the common label of “clickbait”, the sole aim of which is generating monetary profit from web based advertising, regardless of other collateral damage effects inflicted on the hapless audience, such as producing a state of confusion. The deception is focussed on attracting attention and promoting or compelling further distribution of the misinformation, typically via social media.

Much more interesting are instances where the aim is political, ideological, or psychological, as the gain sought by an attacker is a change of perception or belief in the victim audience. Paul and Matthews explore this specific problem in a case study, and observe for that instance that producing confusion in the audience is now a common aim, counter to the past practice in political influence operations, “which traditionally emphasize the importance of truth, credibility, and the avoidance of contradiction”, the latter intended to introduce coherent beliefs in a victim audience aligned with the political or ideological agenda of the attacker [56]. The traditional practice is well studied, and has been previously mapped into the information-theoretic models [37, 38].

As the works of Wardle, Paul and Matthews, Lazer et al, and Campan et al show, the deceptions employed in “fake news” are predominantly of types that are represented by the Degradation and Corruption information theoretic models, with some of the more traditional propaganda constructs employing the Subversion information theoretic model. This is entirely consistent with earlier analysis by Kopp, studying empirical instances of political and commercial deceptions, where these three deception models were found to be most commonly used [37, 38].

Digital media with their inherent capability to amplify “fake news” traffic volumes add an additional dimension, as they facilitate “saturation” or “flooding” attacks, in which the victim is inundated with deceptive message traffic, possible via multiple channels. This type of deception attack was found to have multiple effects. Where the victim is unable to cope with the volume of messages, the effect becomes that of a Denial attack as the channel is effectively disabled; where the victim is able to cope with the volume of messages, the attack becomes an instance of overt Degradation where the deceptive messages are not coherent in content, and Corruption or Subversion where they are coherent in content [38].

Which of the four deceptions and respective effects are produced in the victim of a “saturation” or “flooding” attack depends on how the victim processes information. Does the victim ignore messages beyond some volume? Does the victim attempt to infer message veracity from the respective quantities of messages, employing the argumentum ad populum fallacy, and fall for a Corruption or Subversion attack? Does the victim simply become confused, suffering the effect of a Degradation attack?

To fully address the problem of “saturation” or “flooding” attacks in digital media requires a more complex model for a participant, in which different information processing schemas are employed to capture different types of “fake news” victim. However, each victim type will suffer effects defined by one or more of the four deception models, and therefore even a simpler study of the kind we conducted will provide some useful insights into the effects of such attacks.

Digital media are also characterised by message forwarding, such as “retweeting” or “sharing”, where victims of a deceptive message propagate the message to others, a model that fits the definition of a Chained Compound Attack, in which the party propagating the message becomes a proxy for the attacking party, knowingly or not [38].

To date most modelling effort dealing with messaging in social media has focussed on the diffusion of messages, mostly employing epidemiological models. Nekovee et al applied this approach to random and scale-free networks, relating propagation to network topologies [57]. A empirical study by Jin et al, using the SEIZ model developed by Bettencourt et al from the earlier SIR model, showed excellent agreement with Twitter social media traffic [58, 59]. Isea and Lonngren extended the SEIZ model to describe rumour propagation [60]. Zhao et al have studied other variations of the SIR model in rumour spreading [61–63]. More recently, Mussumeci and Coelhoa applied the SIR model to study the propagation of news [64].

Zubiaga et al studied empirically the behaviour patterns of social media users propagating rumours, showing that false rumours persisted longer due to the difficulty in debunking them, and showing that prevalent behaviour was to propagate rumours regardless of veracity [65].

The characteristic of the epidemiological models is that they capture diffusion behaviour, but are not intended to model the underlying mechanisms that determine the behaviour of the population, as an agent-based model is intended to do. For instance, in the SEIZ model, agents in a population can be susceptible or infected, and parameters in the model determine the rate at which the infected will infect the susceptible and the message diffuse through the agent population. In the SEIZ model, the population size remains unchanged, but the size of the compartments of the population comprising susceptible or infected agents change over time. The model cannot capture the internal causes for agents in the population to propagate a message as it was derived from epidemiology models intended to describe the dynamics of an infection, where infectious pathogens implicitly propagate themselves.

Recent studies by Petrov et al and Mikhailov et al used differential models to study the effects of propaganda on populations divided into groups choosing opposing viewpoints [66, 67], while Conover et al showed how intensive cooperation is central to the activity of online political communities [68].

Not well studied to date is the effect of “fake news” on cooperation in such communities. Political debate and voter choices often reflect consensus within a population on the suitability of competing political alternatives. Traditional political propaganda aims to alter beliefs, reinforcing consensus in supporters, and seducing the undecided and opposed, mostly employing Corruption and Subversion deceptions [37, 38]. Even less studied is the effect of political propaganda contrived to create confusion and thus uncertainty, using the Degradation deception. The effects of such confusion were recently explored by Flynn et al, who showed that misperceptions aligned with prior beliefs or agendas were more often accepted [10].

Comprehensive modelling of the whole gamut of effects empirically observed in “fake news”, especially in a contemporary digital environment, is a major challenge, due to the diversity of these effects, the presence of simple and compound deceptions, and the potential for different propagation topologies and social media participant behaviours.

Our experimental modelling explored the effects of the Degradation and Corruption deceptions on populations, emulating the two styles of political propaganda currently prevalent in social media. The experimental platform was an evolutionary Iterated Prisoner’s Dilemma (IPD) simulation, in which agents can evolve a range of well known IPD strategies, and constrained models of deception. The IPD was chosen specifically as it is widely understood, providing a good basis for comparisons and interpretation of results, and because it captures relative performance of cooperative and uncooperative strategies well [69, 70].

There are few studies that explore the effects of deceptions in populations using evolutionary simulations. Számadó et at in their study of the effects of deceptive messaging in an evolutionary simulation, employed a variant of Ohtsuki’s donor-recipient reciprocity model. This model is not an IPD, and randomly pairs donors and recipients, who can cooperate, defect or punish. Dishonest signalling was employed to manipulate victim perceptions of player reputations, in which good or bad reputations could be misrepresented as the opposite. In terms of the information theoretic models, this misrepresentation maps into the Corruption deception. Our experiment on the effects of Corruption differs in a number of respects, primarily in the use of a different game, and in the inability of players to identify other players by past reputations, as in our model players remember only their own experienced outcomes, and not the identity of the past opponents who produced them [71, 72].

A particular focus in our modelling was to assess the impact of the cost of deceptions on population behaviours, as this reflects a real world scenario, where the effectiveness of a deception may be improved by increased effort while incurring increased costs. We did not model variable deception effectiveness as a function of cost.

Experimental design for agent based modelling of the “fake news” problem

The two series of experiments we conducted show the emulation of behaviours observed empirically in social media, by using an agent based IPD simulation with random pairing, but also show that information-theoretic models of deception can be employed in simulations to good effect.

The first experiment was designed to show that even a small fraction of agents in a population that conducts Degradation deceptions that introduce uncertainty can disrupt cooperation across a much larger population of agents, while also exploring the cost dependencies of these behaviours.

The second experiment was designed to show that in a population where agents conducting the Corruption deception are allowed to evolve and invade the population, this agent behaviour follows very similar diffusion behaviour within the population to that observed by Jin et al, Mussumeci et al and others in social media propagation of messages, while also exploring the cost dependencies of these behaviours.

Both experiments were thus designed to capture the propagation and amplification of deceptive messages, the effect of deceptive messages on an actively engaged population that is mutually interacting, and to explore sensitivity to the cost of deceptions.

The simulation was implemented using Netlogo turtles as agents [73]. We ran this simulation with a population of 50 agents, that is of the proper order of magnitude for Dunbar’s number in social media populations [74].

This is significant insofar as debates in social media typically involve smaller groups, with very much larger numbers of passive participants observing the debate. Campan et al studied “fake news” distribution mechanisms and identified a recurring practice pre-dating the digital social media, in which producers of fake news content target highly engaged and visible groups or individuals, who then become proxies that distribute the deceptive messages, thus implementing a Chained Compound Attack [55], [38].

Our initial assessment was that providing a population size of the order of Dunbar’s number should produce representative results for the social media context explored, as Dunbar’s number provides a reasonable bound on the size of a highly engaged and visible group in social media.

As the results of both experiments using this constrained population size demonstrated behaviours empirically observed in much larger real world environments, we considered that the additional effort in migrating the simulation to an environment compatible with much larger population sizes was not easily justified. It would be easily justified for more complex experiments capturing deceptive message propagation in complex and constrained network topologies. Given both the diversity and variability of social media and other environments subject to “fake news” attacks, the generality of the results will depend on how closely the environment resembles the types we compare our results against.

As the simulation is very computationally intensive, the population size of 50 required 14 days of computation time on the fastest four core processor available at that time, specifically an overclocked Core i7-6700K. Considerable effort was invested in profiling the performance of the simulation, which was constrained to four concurrent simulation runs. This effort showed that the Netlogo runtime environment Java Virtual Machine represented a serious performance bottleneck.

Because the computation time of the simulation scales with the square of the population size, simulating fully sized populations representative of real world social media environments in reasonable time was not feasible, without a different simulation environment that is suitable for a large parallel processing platform.

Agent reproduction in this simulation employs a two-point crossover at random locations, followed by a probabilistic mutation using an evolved mutation probability initially common to all agents.

In each generation, we remove the two agents with the lowest score from the population. We then select an agent probabilistically weighted on score and breed this agent with another randomly selected agent to produce two new offspring that replace the previously removed agents. This bounds the diffusion rate of invading strategies into a population.

The cumulative score determines agent fitness, and whether the agent will reproduce or die out. Agent fitness is a useful measure of effectiveness in modelling this problem area, as its effect can be mapped on to the popularity of “fake news” messages. If they are popular they propagate better thus increasing exposure of the victim audience to the deceptive message.

We employed the extant internal Netlogo turtle prisoner’s dilemma game that employs non-negative payoffs, which satisfy the prisoner’s dilemma condition of T > R > P > S, and can be remapped into the donation game parameters b, b − c:

• Temptation (b): T = 5
• Reward (b − c): R = 3
• Punishment: P = 1
• Sucker: S = 0

As the simulation employs a cumulative score as a measure of agent fitness, a form with non-negative payoffs is convenient as the fitness value is non-negative. We did not assess alternative values of Temptation and Reward in this simulation, as N such parameterizations would have increased required computational effort N-fold. Moreover, other parameterizations in the IPD would deny simple comparisons against earlier modelling using the default Netlogo IPD parameters.

The agent population was initialised with seven well known IPD strategies, specifically TFT, TF2T, Pavlov, Always Cooperate, Always Defect, Random and Probabilistic. The mix of strategies was intentional, to explore how the deceptions impacted susceptible strategies. In part this approach was also employed to provide control cases as some strategies, such as Always Cooperate and Always Defect will not be impacted by deceptions.

Each agent has memory to retain a history of three previous opponent moves, employed to determine its next move, given its evolved strategy.

The strategies differ in how they examine their history to select the next move. Always Defect, Always Cooperate and Random ignore the previous moves of the game. Tit for Tat considers only the opponent’s previous move. Tit for Two Tats considers the two previous observed moves. Pavlov considers the opponents’ previous move and the player’s previous move. Probabilistic considers all the previous moves of past opponents, choosing to defect or cooperate with equal probabilities to previously encountered opponents.

The cooperative strategies of most interest consider only the previous or two previous opponent moves, therefore additional depth in the history would not provide a benefit. This was confirmed by experimentation with early variants of the simulation that showed that increasing the depth of the memory to more moves, specifically four and five, did not appreciably alter results, but did increase simulation execution time appreciably, due to the high average frequency of operations involving alteration of agent memory. Therefore the final simulation was run with agents remembering only three previous moves, as this was assessed to provide sufficient sensitivity to deception effects in reasonable computation time.

For the experiments we report here, we ran the simulations in both sets of experiments for 5001 time steps. To provide control cases, some simulations were run in the first experiment with an initial population of deceiving agents, and some were allowed to evolve deceptions.

Experiments on the effects of degradation

The purpose of this set of experiments was to establish whether a small population of players introducing uncertainty into the memories of a large population of non-deceiving players could significantly alter the frequency of cooperative behaviours in the non-deceiving population. This experiment was in effect intended to explore the impact of confusing “fake news” being injected into a community of voters, who are intending to vote a particular way, and thus cooperate in public discourse by agreeing with each other to reinforce their subjective certainty in a particular voting choice. The introduction of confusing “fake news” has been claimed to produce dischord and increase the degree of uncertainty in voters making up their minds [56].

The problem of noise disrupting the TFT strategy, first described by Molander, has been well studied, but has been previously framed as a result of memory errors or decision errors, which are not correlated with player intent [75–77].

Our hypothesis was that the use of the Degradation deception to increase the level of uncertainty in decisions across a population of players using cooperative strategies such as TFT and TF2T would produce identical effects to the well studied problem of noise being injected into the decisions of agents employing strategies such as TFT. The experiment was intended to not only demonstrate this behaviour, but also provide measures of sensitivity to deception cost, the impact of the size of the population that is employing the Degradation deception, and the effect on strategies other than TFT and TF2T.

In defining this experiment, we considered the mapping of strategies in the simulation to the respective roles deceivers and victims of deceptions play in social media interactions. The behaviours of highly “polarised” participants will map into the Always Defect strategy, while the behaviours of consensus seeking participants will map into strategies such as Always Cooperate, and variants of Tit-For-Tat.

We parametrised cost across a range of values (0.05, 0.1, 0.15, 0.2, 0.25, 0.3), as earlier calibration runs of the simulation indicated that Degradation costs in excess of 0.2 were not sustainable in this simulation.

The populations for all simulations in this experiment were initialised with cooperative strategies, i.e. Tit for Tat, Tit for 2 Tats, Always Cooperate, Pavlov or Probabilistic, so we could observe non-cooperative strategies, i.e. Always Defect and Random, evolving and invading the population if their fitness permits this to occur. An agent playing any strategy can employ deception, with the consequence that strategies for which concurrent deception on average reduces payoffs will be unable to become established in the population due to reduced fitness.

Experiments on the effects of corruption

The purpose of this set of experiments was to establish the manner in which a population of agents performing Corruption deceptions, given some cost, would expand into a larger population of agents that are not deceiving. This experiment was in effect intended to emulate the diffusion of “fake news” in social media, for a population of agents that can derive a payoff from the deception.

The “fake news” problem is characterised by agents who propagate a message to gain an implicit reward, for instance by subjective gratification, or an explicit reward by advertising revenues. If the message is not “liked” or not propagated further by “retweeting”, the agent does not earn a reward [47].

This represents, as noted earlier, an instance of the Chained Compound Attack in which the entity propagating the confusing message becomes a proxy for the entity conducting the deception. A victim of a deception becomes a proxy of the deceiver the instant this victim propagates the deceptive message, and whether the deception is actually believed by the victim might not matter.

Our evolutionary simulation cannot propagate a pathogen or a belief, but agents can produce offspring that propagate characteristic properties, comprising a combination of a strategy and the use or otherwise of a deception. Notionally this is a remapping of the problem of a population shifting between states into a birth-death process representation, in which deaths and offspring are used to represent a state change as occurs in an epidemiological model such as SEIZ. This permits capture of the diffusion behaviour observed in epidemiological models.

Unlike the SIR and SEIZ epidemiological models, the IPD simulation provides control of a wider range of parameters, especially the payoffs and costs to players in the population.

This is important as in social systems, imperatives for propagating a message, false or true, derive from payoffs to the players. Fitness to produce offspring becomes a proxy variable to capture the transition from uninfected to infected states in an epidemiological model. We were especially interested in the sensitivity of this simulation to the cost of the deception, given its importance in social systems.

To test the simulation results for the characteristic diffusion behaviours previously observed in social media message propagation we devised an algorithm to fit a differential epidemiological model akin to SIR [59, 61–64].

This model describes the dynamics in terms of only the population fractions exhibiting specific behaviours s_i of the entire agent population, in the manner of compartments in epidemiological models. Each population is assumed large enough that s_i is continous.

For a locally conserved population of identifiable agents, the birth, death, immigration, and emigration rates are set to zero. And conversions form a point process with a constant rate. Let β_jk be the per capita rate of spontaneous conversion from trait i to trait j and μ_ijk the per capita per meeting rate from j to k induced by a meeting with i.

Given time series data [s_k]_t where t is a discrete time of a discrete simulation. There is also and [s_i s_j]_t for each i, and j. The problem of model fitting is to determine as a linear combination of the [s_k]_t and [s_i s_j]_t. Given m agent behaviours there are m distinct [s_k] and m(m+ 1)/2 distinct [s_i s_j]. Place these in a matrix F. Let U = F^T F, almost always non singular, and , the optimal parameters are , and the differential estimation is .

Population decays were also fitted, to compare as a control case against other work on decay transients in social media traffic [78].

We include a more detailed discussion of the fitting method in S4 Appendix.

Simulation results for experiments on degradation

Figs A-N in S3 Appendix show the evolution over time of the average populations of agents playing IPD strategies, parametrised by cost.

In assessing the equilibrium behaviours observed in an evolving population with a mix of different strategies, we employ the dynamical outcome descriptions employed by Le and Boyd in their study of evolutionary dynamics in a continuous IPD, noting that their simulations employed a population of 8,000—10,000 agents for 1,000 to 3,000 generations, while we employed a population of 50 agents for 5000 generations [79]. Le and Boyd identify four dynamical outcomes, labeled as stable cooperative equilibrium, cyclical polymorphism, stable polymorphism, and collapse to a non-cooperative equilibrium.

Other than a set of early simulation test runs intended as a control, the simulations were all initialised with an equal mix of cooperative strategies, allowing the exploitative All Defect strategy to randomly evolve and invade the population. There were no significant differences between the simulations initialised with a population of deceiving agents, and simulations initialised without deceiving agents. S1 Appendix presents data for simulations initialised without deceiving agents.

The first set of plotted data show the average sub-populations of agents, grouping them into six sub-populations, determined by the use of IPD strategy and the use or non-use of the Degradation deception, with the Random strategy as a control. These sub-populations may be broadly described as either cooperative or exploitative, based upon their use or otherwise of a cooperative IPD strategy. Where a combination of a strategy and deception yields on average a low fitness, that combination extinguishes itself very quickly. Examples for a cost of 0.05 and 0.3 are shown in Figs 4 and 5 respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Population map for degradation experiments at low cost.

Population map, with grouped cooperative and exploitative strategies, for Cost = 0.05.

https://doi.org/10.1371/journal.pone.0207383.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Population map for degradation experiments at high cost.

Population map, with grouped cooperative and exploitative strategies, for Cost = 0.3.

https://doi.org/10.1371/journal.pone.0207383.g005

The observed equilibrium behaviours fit the stable polymorphism identified by Le and Boyd, with an initial population of non-deceiving agents playing cooperative strategies invaded by a mix of deceiving and non-deceiving exploitative agents, with a stable but noisy equilibrium becoming established between 500 and 2,000 generations.

There is no evidence of periodicity in the equilibrium leading to the cyclical polymorphism, or a trend within the 5000 generation interval to a state of collapse to a non-cooperative equilibrium, but the stability of the equilibrium is continuously challenged by the random invasion of a small population of deceiving agents.

The results show in a convincing manner that agents using Degradation paired with the Always Defect strategy will successfully invade the population, and remain in the population long term, even through deceiving agents using Degradation struggle. The population size of such agents progressively declines, due to competition with agents that play the exploitative Always Defect strategy without deception, as the latter do not incur the cost of the deception. The small population of deceiving players is nevertheless able to disrupt the cooperative players, significantly reducing their numbers in the population, depicted in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Cost dependency of deceiver population size.

Fraction of population deceiving and playing exploitative IPD strategies versus the cost of deception, between 2000 and 5000 generations.

https://doi.org/10.1371/journal.pone.0207383.g006

These results are significant in two ways. The first is that even a very small population of exploitative deceivers that inject uncertainty in decision-making into a large population produce a major advantage for all players of exploitative strategies, that dominate the population while the cost of deception is low. The second is that deceiving exploitative agents suffer a significant disadvantage in fitness against non-deceiving exploitative agents who incur no costs.

Put simply, even very small and transient populations of exploitative deceiving agents drive cooperating agents out of the population, allowing them to be displaced by agents playing exploitative strategies, while damaging the overall fitness of the population by reducing the fraction of cooperating agents. This result has implications beyond the “fake news” problem, and is discussed later in this paper.

This experiment also included a Cost of C = 0 where deceiving agents do not suffer a disadvantage against non-deceiving agents. In this unique situation, plotted in Figs A and B in S3 Appendix, the deceiving agents playing the exploitative All Defect strategy invade the population and displace most strategies, other than non-deceiving All Defect in just over 2600 generations, producing a collapse to a non-cooperative equilibrium.

This experiment also displayed pronounced dependency of the size of the population exploiting the effects of the deception on the the cost of deception, depicted in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Cost dependency of exploitative population size.

Fraction of population playing exploitative IPD strategies versus the cost of deception, between 2000 and 5000 generations.

https://doi.org/10.1371/journal.pone.0207383.g007

Simulation results for experiments on corruption

Figs A-I in S4 Appendix show that the diffusion behaviour closely resembles the behaviour observed with differential epidemiological models such as SIR. The fit was remarkably good. This was determined using the differential model, where integrating E obtained compartment estimates that were conserved to 4 decimal places, a measure of the quality of the model, and behaviour that was very similar to the simulation, including decay transients. The general behaviour observed accurately emulates initial diffusion, followed by a decay transient, in repeated patterns as observed in social media, with each cycle of diffusion and decay corresponding to the release and propagation of a popular item of “fake news”.

No differently than in the previous experiment, the short duration effect of a deception is to produce a persistent impact on the population ratios. Agents playing exploitative strategies dominate the population for many generations following the transient presence of the deception, until cooperative strategies recover due to higher average fitness and displace the exploitative strategies. This shows the disruptive effect of a popular deception as observed empirically in social media debates. At some point the population of agents using the Corruption deception reappears, and the cycle is repeated again, as shown in Figs 8 and 9.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Comparison of transient and persistent population behaviours simulation run 101.

Population map showing transient and persistent behaviours for simulation run 101.

https://doi.org/10.1371/journal.pone.0207383.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of transient and persistent population behaviours simulation run 148.

Population map showing transient and persistent behaviours for simulation run 148.

https://doi.org/10.1371/journal.pone.0207383.g009

Similar behaviour was observed by Számadó et al in modelling deceptive messaging in a game of indirect reciprocity with cooperation, defection and punishment [72].

While the dependency on cost was pronounced as in the previous experiment, observed behaviour showed a weak initial cost dependency, followed by an abrupt and stronger dependency above a cost threshold, that for this simulation and its parameters occured at a cost of around 0.9, refer Fig 10. At higher costs, the population using the Corruption deception cannot gain a foothold in the population, and behavior is dominated by the conventional contest between populations playing cooperative and exploitative strategies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Stability of diffusion behaviour versus the cost of deception.

Fraction of population exhibiting stable diffusion behaviour versus the cost of deception.

https://doi.org/10.1371/journal.pone.0207383.g010

For comparison with the experiment exploring the Degradation deception, we include evolving average population data for this experiment, displayed in Figs A-L in S5 Appendix. The equilibrium behaviours observed again mostly fit the stable polymorphism identified by Le and Boyd, with an initial population of non-deceiving agents playing cooperative strategies invaded by a mix of deceiving and non-deceiving exploitative agents, with a very similar stable but noisy equilibrium becoming established between 500 and 2,000 generations.

An interesting comparison is the effectiveness of Corruption, that deterministically changes the victim belief, against Degradation, that randomly alters victim belief. The evolving average population data shows that Corruption produces larger populations of exploitative agents over a much wider range of costs. This behaviour reflects empirically observed effects in propaganda and the preference for Corruption over Degradation in propaganda predating the digital age [37, 38].

Notably, this experiment included a Cost of C = 0 where deceiving agents do not suffer a disadvantage against non-deceiving agents. In this unique situation, plotted in Fig A in S5 Appendix, the deceiving agents playing the exploitative All Defect strategy invade the population and displace most strategies, other than non-deceiving All Defect in just over 500 generations, producing a collapse to a non-cooperative equilibrium.

Applicable information theory concepts and an improved model of corruption

Shannon’s model of information, and the transmission of information through a channel, is a well proven representation, ubiquitous in modelling data transmission and storage, human cognitive loads, as well as other information-centric problems [95, 96].

The Shannon model is constructed around the assumption of two entities communicating symbols through a channel, the function of which is impaired by the effects of additive white Gaussian noise (refer Fig 11). The symbols form an alphabet, which is assumed to be known and understood by all entities using the channel. The effect of additive noise in the channel is to change some symbols, thus introducing errors in transmission through the channel.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Shannon’s model for a communication channel.

Shannon’s model for a communication channel, comprising a source, transmitter, channel with noise source, receiver, and destination (Kopp, per Shannon, 1948).

https://doi.org/10.1371/journal.pone.0207383.g011

The Shannon model makes two assumptions, which are important for the study of deception. The first and weaker constraint in practice, is that noise in the channel is assumed to be Gaussian. The more important assumption is that the symbols forming an alphabet are understood by both entities, although the model makes no assumptions about the meaning of the messages encoded by the alphabet. The model assumes both entities have prior probability distributions for messages sent and received. We will show that deceptions may often involve manipulation of the channel, or misrepresentation of the meaning of messages, the latter involving manipulation of the alphabet.

Shannon’s model is quantitative, and centred on the idea of entropy. If a message contains information, an entity receiving it and understanding it will experience a state change which alters its level of uncertainty. The less likely the message, the greater its information content, noting that the prior probabilities may be unique to each entity in the channel. In particular: where I(m) is the information in message m, and p(m) is the probability of the message.

In the context of deception, Shannon’s channel capacity theorem is much more useful. It states that the capacity of a channel to carry information depends on the magnitude of the signal encoding the symbols, the magnitude of the interfering noise in the channel, and the bandwidth of the channel:

The channel capacity theorem is defined in its basic form for a physical channel, with the properties of capacity C in bits/s, bandwidth W in Hertz or cycles/s, and signal power S and noise power N in Watts, reflecting the definition of the theorem for a physical communication channel. Actions by a transmitting entity to manipulate the terms in the capacity equation will, in turn, manipulate the internal state of the receiving entity.

The application of models of deception based on the channel capacity theorem is now well established in areas involving electrical transmission or detection of data [46]. This is because the channel capacity theorem can be easily and directly mapped into widely used models in this area.

In other systems, additional mappings to transform variables into a form suitable for the capacity theorem model are required. An example might be a victim flooded with a large number of text documents, mostly containing repetitive or irrelevant content, intended to reduce the “signal to noise ratio” seen by the victim. An applicable and context sensitive mapping into Shannon’s S/N ratio must be performed to produce a quantitative measure, although this may not be required if the intent is simply to understand how the deception alters a game, and perform ordinal ranking of outcome preferences.

Another important concept in information theory for understanding deception is that of measures of similarity or difference, since similarity to or differences from a known message may be used as a means of distinguishing valid from invalid messages, or as a means of deception, making a false message appear to be real. This is the basis for the model employed to describe Corruption deceptions.

Measures of similarity or difference remain an area of active research in information theory, and at this time a number of measures have been proposed, with varying degrees of generality. For instance, Vitanyi and Li et al have proposed measures based on information distance, while Lin proposed a similarity theorem that defines a measure of similarity by the ratios of information [97–99].

Vitanyi’s measure of difference is presented as one example. How different Y is from X is measured by the size K(Y|X) of the description of how to edit Y to turn it into X. If Y is the same as X, then there is no editing and the difference is 0. Since the edits required to turn Y into X might be different from those required to turn X into Y, the maximum is taken. However, this is divided by the maximum size of the edits K(X) and K(Y) required to construct X and Y from scratch. So, difference is taken in proportion to the complexity of X and Y alone. The precise formula used by Vitanyi is as follows:

Where S is similarity, D is difference, and K is the editing function applied to X, Y. Vitanyi uses a theoretically optimal editing process. But, for practical purposes he suggests the use of any common compression algorithm, such as ZIP. The typical compression algorithm is dictionary based. It stores common words and phrases from X and writes the list of code numbers. If Y contains few words or phrases that do not already exist in X, then the compression of XY will be only slightly larger than of X. Such an optimal process may or may not be realised in practice, when a player attempts to determine similarity.

A number of other widely employed measures of similarity or difference exist and could be applied here. For instance Kullback-Leibler divergence or relative entropy, or its second derivative, the Fisher information metric, are often employed to gauge differences between two probability distributions [100, 101].

Kullback-Leibler divergence is: where D_KL(P||Q) is the KL divergence in bits, and P(X) and Q(X) are some distributions of random variable X. If the distributions are not identical, divergence is non-zero:

KL divergence is not symmetrical, but still provides a useful measure of difference. Mutual information is also a useful means of establishing differences, and is of interest given its implicit relationship with Shannon information. A common definition is [102]:

Where P(X, Y) is the joint distribution, and P(X) and P(Y) are the respective distributions for X, Y.

The empirical problem in modelling deceptions will often lie in determining how an entity measures similarity, and which measure or metric best captures or approximates this behaviour.

Shannon’s channel capacity, and information-theoretic measures of similarity are bounds, respectively, on systems which transmit or compare messages. An actual non-ideal physical system may not be capable of achieving these bounds, and may have a decision threshold well below these bounds. As a result, a valid message may be lost in noise internal to the system, or two similar messages might be interpreted to be identical, despite being very different in some way.

Deceptions exploit these limitations of non-ideal systems. Put quantitatively: Where C_receiver is the channel capacity actually available to the victim, C_objective the achievable channel capacity, S_receiver(A, B) the similarity between A and B as perceived by the receiver, while S_objective(A, B) is the objective similarity, assuming an actual objective or “ground truth” rather than subjective probability distribution. Successful deceptions arise when the deceiver manipulates the channel or message in a manner, which the receiver is unable recognise or overcome, altering the receiver’s perception, and thus subjective probabilities. For simplicity, we further assume that the outcome of a deception is a discrete state change in the receiver’s perception, a reasonable assumption since in practice a successful deception typically captures the receiver.