The landscape and flux quantification for game theory

The landscape and flux theory as well as the non-equilibrium thermodynamics for the general dynamical systems have been explored in several different fields [20–27]. They can be used to address the issues of global stability, function, robustness for dynamical systems. Here, we will apply the landscape and flux theory to quantify the global stability and robustness of the game theory.

We consider a large finite population of players who play a finite set of pure strategies with random matching. Each player chooses a pure strategy from the strategy set S = 1, 2, 3, …, n [1, 2]. The aggregated behaviors of these players are described by a population state x, with x_i representing the proportion of players choosing pure strategy S_i. And x_i represents the frequency of strategy S_i. Due to the intrinsic and the extrinsic fluctuations [8], the deterministic dynamics described by a set of ordinary differential equations are supplemented with the additional fluctuation force. Then the stochastic dynamics emerges [28, 29]: dx = F(x)dt + g ⋅ dW, where x is the state vector representing the strategy density in game dynamics), F(x) is the driving force, W as a vector coupled through the matrix g represents an independent Wiener process. The evolution of the stochastic dynamics is thus more appropriately described by the probability evolution. The probability distribution P(x, t) in time can be obtained by solving the corresponding Fokker-Planck diffusion evolution equation [28, 29]: (1) We set DD = (1/2)(g ⋅ g^T), where D is a constant describing the scale of the fluctuations and D represents the anisotropy diffusion matrix of the fluctuations. The Fokker-Plank diffusion equation describes the game dynamics as a conservation law of probability. The change in local probability is equal to the net flux in or out.

The game process can be treated as a binomial sampling process: N players play the games with different strategies from a large population of players, just like n alleles in a diploid population of constant size N [24]. So we can set D_ij = x_i(δ_ij − x_j) coming from the sampling nature of the game which is widely used in evolutionary population dynamics [24].

The matrix D_ij has some special features [24]. The first is (2) so that . The second is its inverse matrix is known to have the feature [24]: (3) where .

The steady state probability distribution P_ss can be derived from the long time limit of the Fokker-Planck equation ∂P/∂t = 0. The steady state probability flux is defined as J_ss = FP_ss − D∇ ⋅ (DP_ss). The steady state flux is divergent free from Eq 1 and therefore a rotational curl. The population landscape is defined as U = − lnP_ss. Then, the deterministic driving force F can be decomposed as: F = −DD ⋅ ∇U + J_ss/P_ss + D∇ ⋅ D. The flux J_ss = 0 denotes the equilibrium with detailed balance since net flux is zero while the net flux J_ss ≠ 0 measures the degrees of detailed balance breaking and therefore the degree of one-equilibriumness. Therefore, the dynamics of the equilibrium game system is determined only by the population landscape gradient. The dynamics of the non-equilibrium system is determined by both the potential landscape and non-zero flux. [20]

The intrinsic landscape at the zero fluctuation limit has the the feature of the Lyapunov function [24, 30–32] and can be used to quantify the global stability and function of the game theory systems. The intrinsic landscape ϕ₀ follows the Hamilton—Jacobi equation as below [23, 24, 33]: (4) and (5) As seen, the intrinsic landscape ϕ₀ is a Lyapunov function monotonically decreasing along a deterministic path for the game dynamics [24, 33]. The intrinsic landscape ϕ₀ quantifies the global stability for general dynamical systems including game dynamics either with or without detailed balance and can be solved by the level set method [34].

The steady state probability distribution P_ss and the population landscape U have the relationship of P_ss(N) = exp(−U)/Z. Z is the partition function Z = ∫ exp(−U)dN. The entropy of the non-equilibrium game system is given as [24, 35–38]: S = −∫ P(N, t)lnP(N, t)dN. The energy can be defined as: E = D ∫ UP(N, t)dN = −D ∫ ln[ZP_ss]P(N, t)dN. Thus, the nonequilibrium free energy is: (6) The nonequilibrium free energy as the combination of energy and entropy reflects the first law of non-equilibrium thermodynamics. The free energy decreases in time monotonically while reaching its minimum value, [24, 35–38]. This reflects the second law of non-equilibrium thermodynamics. The free energy as a Lypunov functional can be used to explore the global stability of the stochastic non-equilibrium systems with finite fluctuations.

Game theory systems are often non-equilibrium open systems. They often exchange energy and information from the environments. This leads to dissipation. The evolution of the entropy in time can be separated into two terms [20, 24, 39, 40]: , where is entropy production rate which is positive or zero, and is heat dissipation rate which represents the entropy flow rate to the non-equilibrium system from the environments. It can be either positive or negative. The effective force is shown as F′ = F − D∇ ⋅ D. can be interpreted as the entropy change of the non-equilibrium system, and the can be interpreted as the total entropy change of the system and environments. is always non-negative according with the thermodynamic second law. The entropy production rate and heat dissipation rate are equal in steady state [20, 24, 39, 40]. Thus, the entropy production rate is another global physical characterization of a non-equilibrium system.

The repeated Prisoner’s Dilemma game theory model with mutations

We set A as the payoff matrix [6, 13–16]. The scalar A_i(x) represents the payoff to strategy S_i when the population state is represented by x. Since the sum of all frequencies is equal to 1: ∑_i x_i = 1, the system becomes n − 1 dimensional. The average fitness (pay-off) is obtained as by the players of the population [1, 2, 6]. The fitness denotes the individual’s evolutionary success [9]. In the game theory, the payoff of the game is the fitness. The fitness to strategy i becomes f_i = (Ax)_i [1, 2]. In this study, we use simple three-strategy game which can be reduced to two dimensional dynamics.

The players in standard Prisoner’s Dilemma model play either cooperate strategy or defect strategy simultaneously [41]. The players earn their payoff depending on their choices of the strategies. The mutually aided cooperators will acquire the reward R when the cooperators are encountered. The defectors will obtain a punishment Pu when the defectors are encountered. A cooperator acquires a sucker payoff S and a defector acquires a temptation payoff T when they encountered [18, 42]. The payoff matrix of the Prisoner’s Dilemma model should satisfy the relationship T > R > Pu > S. Mutual cooperative strategy is better than mutual defective strategy, so the reward R should be greater than the punishment Pu. Defectors gain more temptation T than the reward R that cooperators gain if the partner cooperates. The defectors will lose less punishment Pu than the lost sucker S of cooperators if the partner defects.

We explored a repeated Prisoner’s Dilemma game model with mutation using the replicator dynamics. Replicator dynamics was introduced by Taylor and Jonker (1978) [43], which is the best-known dynamics in the models of biological evolution. The mean dynamic evolutionary equation is the replicator dynamics shown as below [1, 2] (7)

In the presence of mutations, ALLD and ALLC players may mutate to strategy TFT. Therefore, more players choose TFT strategy, and help ALLC players to win the game. Then the ALLD players will emerge again to obtain more profit [6]. This dynamics can be used to quantify the peace and war oscillation [6].

We define the probability that agents with strategy S_i mutating to strategy S_j as q_ij, which satisfies ∑_j q_ij = 1. Thus, mutation matrix is Q = [q_ij] [6, 13, 17, 19]. The elements of the mutation matrix Q are defined in terms of a mutation parameter μ satisfying 0 ≤ μ ≤ 1. The mutation μ denotes the error probability in the process of replication. μ = 0 denotes no mutation with perfect replication while μ = 1 denotes the mutation completely. The replicator-mutator dynamics describes the dynamics of the population distribution x as a result of replication driven by fitness f and mutation driven by Q [6, 15, 16]: (8)

This equation is the quasi-species equation proposed by Manfred Eigen and Peter Schuster [6, 15, 16]. We set a uniform probability of mutation from one strategy to another strategy with q_ii = 1 − 2μ which shows the players with the same strategy can get profits 1 − 2μ for each player, and q_ij = μ which shows the player chooses strategy i will get profits μ when he encounters the player with strategy j. Then the matrix Q can be shown as follows [6, 15, 16]: (9)

Let x₁, x₂, x₃ represent the fractions of the population choosing the strategies ALLD, TFT, ALLC, respectively. We substitute Q into Eq 8, then the replicator-mutator dynamics are shown as the following simplified equations [18, 42]: (10)

We considered the players playing with infinite number of rounds. So in these limits, the average payoff matrix with the cost for the strategies ALLD, TFT, ALLC are shown as [17, 18]: (11)

The mutually aided cooperators will acquire the reward R when the cooperators are encountered. The defectors will obtain a punishment Pu when the defectors are encountered. A cooperator acquires a sucker payoff S and a defector acquires a temptation payoff T when they encountered [18, 42]. And we set there is a small complexity cost c to playing strategy TFT [18, 42].

TFT strategy is conditional while ALLD and ALLC strategies are unconditional. Thus, the payoff value of strategy TFT may have a small complexity cost [17, 44].

The parameters are set as: T = 5,R = 3,Pu = 1, S = 0 [6, 18, 45]. μ is the average mutation rate between each two of the three strategies. c is a complexity cost for playing TFT. Larger c represents less players who play the strategy TFT.

Phase diagram, Hopf bifurcations and the global stability of evolutionary game dynamics upon TFT cost changes

Fig 2(A) shows the phase diagram which shows a S shape for the repeated Prisoner’s Dilemma with the constant parameter μ = 0.006 and changing parameter c. There are five regions in this phase diagram. When the cost for TFT c is smaller, the game theory system has only one stable state, which denotes the Peace state in the peace and war game shown in the left Region I. As c increases, the stable state becomes a limit cycle in Region II. Then, as c increases further, another new stable state (can be viewed as War state) and an unstable saddle state emerge beside the limit cycle. This is a saddle-node bifurcation shown in Region III. As cost c increases furthermore, the limit cycle diminishes and becomes an unstable state, along with a stable state (War state) in Region IV. As cost c increases even further, there is only one stable state (War state) in the right Region I.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Phase diagram, entropy production rate, and free energy versus different parameter c, μ.

A: the phase diagram for repeated Prisoner’s Dilemma model with the constant parameter μ = 0.006 and changing cost parameter c. C: the entropy production rate versus parameter c. E: the free energy versus parameter c. B: the phase diagram for the repeated Prisoner’s Dilemma model with the constant parameter c = 0.22 and changing parameter μ. D: the entropy production rate versus parameter μ. F: the free energy versus parameter μ. The other parameters are T = 5, R = 3, Pu = 1, S = 0.

https://doi.org/10.1371/journal.pone.0201130.g002

It is interesting to note that the two-phase dynamics of p53 in the DNA damage response was intensively explored by Zhang et al [46]. They revealed that a sequential predominance of distinct feedback loops can lead the system to multiple-phase dynamical behaviors [46]. This shows another similar behavior as our game theory model where the variation of the parameters can also lead the system to multiple-phase dynamical behaviors. It is clear that the environmental changes and internal variations can lead the system to phase transitions.

Fig 2C shows the entropy production rate (EPR) versus cost c. We can see EPR has a bell shape in Region II and Region III under the limit cycle behavior of the system. It indicates that the limit cycle costs more energy to maintain its coherent oscillation.

By solving the Fokker-Planck diffusion equation, we obtain the steady distribution of the probability. Thus, the population landscape of the game system can be obtained as: U = −lnP_SS. We solved the Hamilton-Jacobi equation to obtain the intrinsic landscape ϕ₀ by the level set method [34]. Fig 3 shows the 3 dimensional non-equilibrium population landscape U at the top row and intrinsic potential landscape ϕ₀ at the bottom row for the repeated Prisoner’s Dilemma game dynamics as the parameter c increases, when the other parameters are set as μ = 0.006, T = 5, R = 3, Pu = 1, S = 0, D = 5 × 10⁻⁴. We can see the underlying population landscape and the intrinsic landscape have similar shapes. The population landscape and the intrinsic landscape both have a basin shown in Fig 3(A) and 3(E) with small cost c = 0.1. Fig 3(B) shows the population landscape U with a closed inhomogeneous Mexican hat ring valley which is not uniformly distributed, while Fig 3(F) shows that the intrinsic landscape ϕ₀ with Lyapunov feature has a closed homogeneous Mexican hat ring valley with cost c = 0.2. The value of ϕ₀ along this ring valley is almost a constant. As cost c = 0.24, a new basin emerges at the right corner of the state space shown in Fig 3(C) and 3(G). It denotes the War state with larger probability of all defecting strategy ALLD. Then at the cost c = 0.35, the closed ring valley disappears, only the stable War state basin is left shown in Fig 3(D) and 3(H). We notice that the oscillations between Peace and War can be explained as: when Peace state sustains for a long time, the population increases and the resources relatively reduce. In order to survive, the populations fight for the resources, to get better livings. After a war, the populations do not engage in the production, livelihood, and fall into a long-term state of tension. Then the Peace state emerges again, the population regrows. The oscillations will thus circulate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The 3 dimensional population landscapes and The 3 dimensional intrinsic energy landscapes.

The 3 dimensional population landscapes U with increasing parameter c are shown in A, B, C, D. Purple arrows represent the flux velocity(J_ss/P_ss) while the black arrows represent the negative gradient of population potential(−∇U). The 3 dimensional intrinsic energy landscapes ϕ₀ with increasing parameter c are shown in E, F, G, H. Purple arrows represent the intrinsic flux velocity (V = (J_ss/P_ss)_D→0) while the black arrows represent the negative gradient of intrinsic potential(−∇ϕ₀).

https://doi.org/10.1371/journal.pone.0201130.g003

−∇U is the negative gradient of the population landscape and −∇ϕ₀ is the negative gradient of the intrinsic landscape. −∇U at the top row and −∇ϕ₀ at the bottom row are represented by black arrows. J_ss/P_ss is the steady state flux divided by steady state probability and V = J_ss/P_ss)|_D→0 is the intrinsic flux velocity. The flux velocity satisfies V ⋅ ∇ϕ₀ = 0 [24]. This shows that the intrinsic flux velocity is perpendicular to the gradient of the non-equilibrium intrinsic potential ϕ₀ in the zero-fluctuation limit [24]. J_ss/P_ss at the top row and V at the bottom row are represented by purple arrows in Fig 3. We can see that the flux with purple arrows are nearly but not exactly perpendicular to the negative gradient of U with the black arrows around the basins or the closed ring valley shown on the top row. The region with higher population potential has some disordered oriented arrows due to the lower probability and limit of the computational accuracy. The flux velocities with purple arrows are exactly perpendicular to the negative gradient of ϕ₀ with black arrows at the bottom row. It is due to V ⋅ ∇ϕ₀ = 0. The landscape’s gradient force ∇U or the ∇ϕ₀ attracts the system down to the basin or the closed ring valley, while the flux drives the periodical oscillation flow or spiral descent to the basin. It is necessary to characterize this non-equilibrium repeated Prisoner’s Dilemma with both landscape and flux.

We show the 2 dimensional population landscape U for increasing parameter cost c with constant μ = 0.006 in Fig 4. Fig 4(A) shows that the population landscape U has a stable basin near the middle of TFT axis with the cost parameter c = 0.1. It means that most players choose strategies of ALLC and TFT when the cost parameter for TFT is small. The fluxes represented by the purple arrows rotate anticlockwise around this stable state. This state can be viewed as “Peace” state in peace and war game. As the cost parameter for TFT increases to c = 0.2, a limit cycle emerges and replaces the stable state in Fig 4(B). The population landscape has a blue ring valley along the deterministic trajectory. More players choose TFT and ALLC in the Peace state. As more players mutate to ALLC, a small number of ALLD players emerge. This leads to a state with more ALLD players. As the ALLD players become more, the profit obtained from the game becomes less. Some ALLD players convert their strategy to TFT. This makes a circle in the state space of strategy probability. Notice that the ring valley is not homogeneous in landscape depth. There is a deeper area on the left side of the limit cycle, which is still close to TFT axis. This indicates that the Peace state with deeper depth is more stable than other states. The system will stay in Peace state much longer than any other state. Fig 4(C) shows the ring valley of the oscillation expands its amplitude in the strategy-frequency space. A stable state War emerges at the right corner of the triangle, which is close to ALLD → 1. The stable War state is the one where most of players choose the ALLD strategy. We can see that the limit cycle and the stable state coexist in the strategy-frequency space. It indicates that the system is sometimes in the limit cycle and sometimes in the stable state. The game system can switch between the Peace and War attractor basins under the fluctuations and the mutations. When the cost for TFT increases to c = 0.24 and c = 0.25 shown in Fig 4(D) and 4(E), the ring valley becomes shallower and shallower while the basin of the stable War state becomes deeper and deeper. As c increases further, the profits obtained from the TFT strategy decreases in the whole game, more players give up TFT strategy and choose ALLD strategy to earn more, which leads to more stable and deeper War state basin. When the cost for TFT increases to c = 0.35 shown in Fig 4(F), the oscillation ring valley disappears while the War state survives and becomes deeper and more stable. Fig 4 also shows the changes in both direction and the scale of each flux at the area with higher probability (purple arrows) when the cost c increases. We can see that the fluxes have a anticlockwise rotational nature along the limit cycle. The flux is the driving force for the stable oscillation in game theory.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The 2 dimensional population landscape U with different parameter c and constant parameter μ = 0.006.

A:c = 0.1, B:c = 0.2, C:c = 0.22, D:c = 0.24, E:c = 0.25, F:c = 0.35. The purple arrows represent the probability flux.

https://doi.org/10.1371/journal.pone.0201130.g004

We quantify the landscape topography and show barrier heights versus cost c with mutation parameter μ = 0.006 in Fig 5(A). We first set U_o as the value of population landscape U at the maximum point at the center island of the limit cycle. U_s is the value of population landscape U at the saddle point between the limit cycle valley and the stable War state basin. U_p is the minimum value of population landscape U along the limit cycle near y axis, which is the Peace state. U_w is the minimum value of population landscape U at the stable War state. We set the barrier height for the oscillation ring valley as ΔU_Limit = U_o − U_p, the barrier height between the saddle point and the oscillation as ΔU_sp = U_s − U_p and the barrier height between the saddle point and the War stable state as ΔU_sw = U_s − U_w. We can see as the cost c increases, barrier height ΔU_sw increases, barrier height ΔU_sp decreases first then increases, barrier height ΔU_Limit increases first then decreases. It indicates that the oscillation itself relative to the maximum point in the center of limit cycle becomes more stable first then becomes less stable. It has a turning point during the process of c increasing. The War state becomes more robust, and the barrier height from oscillation to War state ΔU_sp becomes less than that of ΔU_sw with larger cost c value. This implies that the War attractor state becomes more preferred than that of the oscillation, as the cost increases further.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The barrier heights versus parameter c and μ.

A: The barrier heights versus parameter c with the parameter μ = 0.006. B: The barrier heights versus parameter μ with the parameter c = 0.22. Here, ΔU_Limit = U_o − U_p, ΔU_sp = U_s − U_p, ΔU_sw = U_s − U_w. U_o is the value of population potential landscape U at the maximum point on the center island of the limit cycle. U_s is the value of population potential landscape U at the saddle point between the limit cycle valley and the stable state basin War. U_p is the minimum value of the population potential landscape U along the limit cycle near the y axis, which is the Peace state. U_w is the minimum value of population potential landscape U at the stable state War.

https://doi.org/10.1371/journal.pone.0201130.g005

Phase diagram, state switching, landscapes and fluxes upon mutations

Fig 2(B) shows the phase diagram for the repeated Prisoner’s Dilemma model with changing mutation parameter μ at the constant cost parameter c = 0.22. There are four regions in this phase diagram. When the mutation rate μ is small, the system has a stable state (can be viewed as War state) in Region IV. As μ increases, a stable state War, an unstable saddle state and the limit cycle coexist in Region III. As the mutation rate μ increases further, the stable state and saddle point disappear after the saddle-node bifurcation. There is only limit cycle left in Region II. As the mutation rate μ keeps on increasing, the limit cycle becomes a stable mixed state (with moderate probability of more than 20% of the players with ALLD strategy) in Region I. The mixed state is the combination of these three strategies. We can see entropy production rate characterizing the heat dissipation EPR shown in Fig 2(D) has a bell shape as the mutation rate μ increases when the cost c is moderate. This is because that the state of oscillation costs more energy in the strategy probability state space and one stable state costs less energy.

We can also see this process of transition in Fig 6 which shows that the population landscape U and the flux change with the increasing mutation rate μ at constant cost c = 0.22. Fig 5(B) shows that ΔU_sp increases and ΔU_sw, ΔU_Limit decrease as mutation rate μ increases. This shows that when mutation rate is small, the limit cycle ring valley is very stable relative to its oscillation center, but has less probability relative to the War state since the War state is much deeper. It indicates that more players choose strategy ALLD, and the players do not like to mutate to the other two strategies. This leads to more stable War state. As μ increases, the limit cycle ring valley becomes less stable relative to its oscillation center, but becomes more stable relative to the War state. Eventually a much more stable state Peace emerges. The state War becomes shallower and less stable, and finally diminishes as μ increases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The flux (purple arrows) on the population potential landscape U with different parameter μ and constant parameter c = 0.22.

A:μ = 0.002, B:μ = 0.003, C:μ = 0.004, D:μ = 0.006, E:μ = 0.008, F:μ = 0.012.

https://doi.org/10.1371/journal.pone.0201130.g006

Phase diagram, state switching, landscapes and fluxes upon temptation payoff

Fig 7 shows the phase diagrams, EPRs, and free energies for the repeated Prisoner’s Dilemma model versus parameter T, R, Pu in three columns respectively. We put these figures together for comparison and orderliness. Figs 8, 9 and 10 show the population landscape U and flux for each parameter T, R, Pu respectively. Firstly, we illustrate Figs 7(A) and 7(D) and 8 for the same parameter T. Next we illustrate Figs 7(B), 7(E) and 9 for the same parameter R. And then we illustrate Figs 7(C), 7(F) and 10 for the same parameter Pu. At last, we focus on discussing the tendencies and the common trends of the free energies with respect to parameters c, μ, T, R, Pu in Figs 2(E), 2(F) and 7(G), 7(H) and 7(I).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The phase diagrams and the entropy production rate versus parameter T, R, Pu.

The phase diagrams for the repeated Prisoner’s Dilemma game model with different parameter T (A) R (B), Pu (C). The entropy production rate versus parameter T (D), R (E), Pu (F). The free energies versus parameter T (G), R (H), Pu (I). The other parameters are S = 0, μ = 0.006, c = 0.22.

https://doi.org/10.1371/journal.pone.0201130.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. The population landscape U and flux for T.

The population landscape U and flux with different parameter T at the constant parameters μ = 0.006, c = 0.22, R = 3, Pu = 1.0, S = 0.0.

https://doi.org/10.1371/journal.pone.0201130.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The population landscape U and flux for R.

The population landscape U and flux with different parameter R at the constant parameters μ = 0.006, c = 0.22, T = 5, Pu = 1.0, S = 0.0.

https://doi.org/10.1371/journal.pone.0201130.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. The population landscape U and flux for Pu.

The population landscape U and flux with different parameter Pu and the constant parameters μ = 0.006, c = 0.22, T = 5, R = 3, S = 0.0.

https://doi.org/10.1371/journal.pone.0201130.g010

Fig 7(A) shows the phase diagram for the repeated Prisoner’s Dilemma game model with the constant parameters μ = 0.006, c = 0.22, R = 3.0, Pu = 1.0, S = 0.0 and different parameter T. T is the temptation payoff earned by the defector while the cooperator gets sucker payoff S. There are three regions in this phase diagram. When the temptation T is small, the game theory system has two stable states which are denoted as Peace and War in the peace and war game shown in Region left V. As the temptation T increases, a limit cycle emerges with the coexistence of War state in Region III. As the temptation T increases further, the limit cycle diminishes and a stable mixed state emerges, along with a stable state (War state) at the right of Region V. We can see EPR shown in Fig 7(D) also has a bell shape as the temptation T increases. This also implies that limit cycle state cost more energy to maintain its oscillation.

We show that the population landscapes U and the flux change with the increasing temptation T of payoff matrix (which denotes a defector acquires a temptation payoff T when they encounter cooperators) in Fig 8. The game system has two stable states when temptation is very small as T = 3. Note that the relationship T > R should hold in this model. The game system covers a large area of state space with a Peace state and a War state. We can see that the basin of Peace state is very stable and deeper while the War state is much shallower and less stable. This illustrates that majority of players choose the cooperation ALLC or TFT strategy. It leads to a more stable Peace state, rather than War state with defection strategy ALLD, when the temptation is less. Defection can not earn more profits. As temptation increases, the Peace state is unstable and becomes a limit cycle state. The limit cycle state adopts the mixed strategy TFT and ALLD. The limit cycle valley representing the Peace state becomes shallower while the War state becomes more stable shown in Fig 8(B), 8(C) and 8(D). When the temptation for this game increases even further, more and more players choose defection strategy ALLD to earn more profits rather than the strategy ALLC. It reflects that more temptation from the defection strategy can lead to more stable War state.

Phase diagram, state switching, landscapes and fluxes upon cooperation reward

Fig 7(B) shows the phase diagram for the repeated Prisoner’s Dilemma game model at the constant parameter μ = 0.006, c = 0.22, T = 5.0, Pu = 1.0, S = 0.0 and different parameter R. The element R of payoff matrix denotes the reward that cooperators will acquire from the mutual aid when the cooperators encounter. The value of reward R should satisfy R > (T + S)/2. If this relationship is not satisfied, the agreement of alternate cooperation and defection will earn more payoff than that of pure cooperation in a repeated Prisoner’s Dilemma game [6]. There are three regions in this phase diagram. When the reward R is small, a limit cycle and the stable War state coexist in Region III. As the reward R increases, the War state disappears after the saddle-node bifurcation and the limit cycle is left in Region II. As the reward R increases further, the limit cycle diminishes and a stable Peace state emerges in Region I. Fig 7(E) also shows EPR versus reward R, which implies that the limit cycle state costs more energy than that of one stable state.

Fig 9 shows the population landscape U with increasing cooperation R. The system has one deep stable War basin state and a shallower limit cycle ring valley for small parameter R = 2.5, since the reward for cooperation strategy is small. The majority players choose the defection strategy leading to the War state shown in Fig 9(A). When the reward increases to R = 3, a limit cycle valley becomes deeper and stable while the War state becomes shallower and less stable as shown in Fig 9(B). It shows that more and more players prefer the strategy ALLC rather than strategy ALLD since more reward can be obtained from the cooperation. As reward R increases further, the limit cycle valley becomes deeper, and the War state vanishes shown in Fig 9(C). As reward R increases even further, the limit cycle ring valley shrinks into a stable Peace state shown in Fig 9(D). The majority players choose cooperation strategy. It is because more reward from the cooperation strategy can lead to more stable Peace state. It turns out that more reward from cooperation leads to Peace with win-win outcome.

Phase diagram, state switching, landscapes and fluxes upon defector punishment

Fig 7(C) shows the phase diagram for the repeated Prisoner’s Dilemma model at the constant parameter μ = 0.006, c = 0.22, T = 5.0, R = 3.0, S = 0.0 and different parameter Pu. The element Pu of payoff matrix denotes that the defectors obtain a punishment Pu when the defectors encounters. The value of punishment Pu should satisfy R > Pu > S [6]. There are five regions in this phase diagram. When the punishment Pu is small, only the stable War state exists in Region I on the left. As the punishment Pu increases, two unstable states emerge after the saddle-node bifurcation in Region IV. As the punishment Pu increases further, a limit cycle emerges and coexists with the stable state War in Region III; as the punishment Pu increases even further, the limit cycle diminishes and a stable Peace state coexists with a stable War state in Region V. As the punishment Pu approaches to 3, the stable Peace state diminishes after the saddle-node bifurcation, and the stable War state is left in Region I on the right. Fig 7(F) also shows EPR versus punishment Pu, which shows that the system with limit cycle dissipates more energy.

Fig 10 shows the population landscape U with increasing element Pu of payoff matrix. When the punishment Pu is small representing that the defector can earn less from the game and almost all players choose strategy ALLD shown in Fig 10(A). ALLD is the only strict Nash solution and the only evolutionary stable strategy. And when the punishment (Pu = 0.3) is small, the value of the first element in the left column of the payoff matrix for ALLD is Pu while the second of that for TFT is Pu − c = 0.08. Thus the fitness for ALLD is much higher than that of TFT, and that of ALLC (The sucker’s payoff S = 0). As punishment Pu increases, more and more players give up ALLD since the profits of TFT players are catching up with that of ALLD players when they both encounter the ALLD players. As the punishment Pu increases to Pu = 0.5, a shallow limit cycle valley emerges accompanied with the stable War state shown in Fig 10(B). When the punishment Pu increases further, the limit cycle valley shrinks its size but becomes deeper and more stable while the War state becomes shallower and less stable. This is shown in Fig 10(C), 10(D) and 10(E). Then the limit cycle ring valley shrinks to a stable Peace state where more players choose TFT strategy as shown in Fig 10(F), 10(G) and 10(H). This shows that as the punishment Pu is increased further, the Peace state becomes more stable first and then loses its stability while the War state becomes more stable. It demonstrates that the punishment Pu has an optimal value to lead to a more stable Peace state. At last, when the value of the punishment Pu approaches to the value of reward R, the stable Peace state diminishes and the stable War state is left in Fig 10(I). This shows that when the punishment Pu and the reward R are almost the same, ALLD is dominant than TFT since ALLD is the only strict Nash equilibrium solution and the evolutionary stable state. These results show that the strategy ALLD can be a dominant strategy and has an advantage for selection.

We explored the free energy versus the parameters of the repeated Prisoner’s Dilemma game model in Fig 2(E) (the free energy versus cost c), Fig 2(F) (the free energy versus the mutation rate μ), Fig 7(G) (the free energy versus temptation T), Fig 7(H) (the free energy versus reward R), Fig 7(I) (the free energy versus punishment Pu). We can see this five free energy profiles have some similarity that each has the opposite tendency with that of the corresponding EPR. These free energies link to the different phases and phase transitions. The first derivative of the free energies are discontinuous at the transition points from a stable state to a limit cycle oscillation state and vice versa [24]. This implies that non-equilibrium thermodynamic phase transition has certain similarities as that of the equilibrium thermodynamic phase transition. Free energy profiles can manifest the phase transitions and can be used to explore the global stability and robustness of the game system.

Kinetic speed and optimal paths of switching between the two stable states

We also explored the kinetic optimal paths of the repeated Prisoner’s Dilemma game model. We can obtain the relative probabilities of each path by the quantification of the path weights with path integrals. Path integral weights can be calculated by the action, which is analogous to the classical mechanical systems. The dominant optimal paths with the largest weights can be viewed as the major pathways. The path integral for the probability of (x_final, t) starting at initial condition (x_initial, 0) is given as [21, 24]: (12)

The above probability describes the chance of starting at the state x_initial at initial time and ending at the state x_final at the final time. The probability is the result of the sum of the weights from all possible paths. A(x) is the action for each path. Each weight is exponentially related to the action which has two contributions, one from the stochastic equation of motion for the dynamics and the other is from the variable transformation from the stochastic force to the system variable. Not all the paths contribute equally to the weight. Due to the exponential nature, the optimal path is exponentially larger in weight than the suboptimal ones. Therefore, we can identify the optimal path with the most probable weight.

We studied the repeated Prisoner’s Dilemma game model with the parameters μ = 0.006, c = 0.22, T = 5, R = 3, Pu = 2.4, S = 0.0, which has two stable state Peace and War. Fig 11 shows the optimal paths on the population landscape U with different diffusion coefficient D. We can see there are two stable states: War and Peace on the population landscapes. The purple lines represent the optimal paths from the War state to Peace state. The black lines represent the optimal paths from the Peace state to War state. The white arrows represent the steady state probability fluxes which guide the optimal paths apart from the steepest descent path from the landscape. Therefore, the optimal path from War state to Peace state and the optimal path from Peace state to War state are apart from each other. Under more fluctuations (bigger diffusion coefficient D shown in Fig 11(B)), the two optimal paths are further apart from each other due to larger probability fluxes than those in Fig 11(A). We can see the purple lines and black lines are irreversible in both two sub figures. The optimal paths are deviated from the naively expected steepest descent paths based on the potential landscape. These lines are apart from each other due to the non-zero flux. We can clearly see the fluxes have spiral shapes which show the dynamic feature of non-equilibrium system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. The pathways on the population potential landscape U with different diffusion coefficient D at μ = 0.006, c = 0.22, T = 5, R = 3, Pu = 2.4, S = 0.0.

The purple lines represent the optimal paths from the War state to Peace state. The black lines represent the optimal paths from the Peace state to War state. The white arrows represent the steady state probability fluxes.

https://doi.org/10.1371/journal.pone.0201130.g011

We show the population landscape U and the optimal paths with different parameter temptation T at the constant parameters μ = 0.006, c = 0.22, R = 3, Pu = 2.4, S = 0.0, D = 5 × 10⁻⁴ in Fig 12. The purple lines represent the optimal paths from the War state to Peace state. The black lines represent the optimal paths from the Peace state to War state. The white arrows represent the steady state probability fluxes. The optimal paths are deviated from the naively expected steepest descent optimal paths based on the potential landscape, and they are irreversible. The two optimal paths become more closer from each other as T increases. Fig 13(A) shows the barrier heights versus temptation T. We set ΔU_sp = U_saddle − U_Peace as the barrier height between state Peace and the saddle point and ΔU_sw = U_saddle − U_War as the barrier height between state War and the saddle point, where U_saddle represents the value of landscape U at the saddle point between state Peace and War, U_Peace represents the minimum value of landscape U at state Peace, U_War represents the minimum value of landscape U at state War. We can see that the barrier height ΔU_sw increases while the barrier height ΔU_sp decreases. It shows the Peace state loses its stability as the War state becomes more stable as the temptation T increases. It denotes that the temptation guides more players to choose strategy ALLD. The path weight represents the probability of each route. The path probability can be obtained by action A(x) between Peace and War. We labeled A_wp as the action of the dominant optimal path from War state to Peace state, and A_pw as the action of the dominant optimal path from Peace state to War state. Fig 13(B) showed the logarithm of the probability of the Peace to War optimal path divided that of War to Peace optimal path increases as the temptation T increases. This shows that the optimal path from Peace state to War state has more weight or chance than that of War state to Peace state due to the losing stability of Peace state which lowers the barrier from Peace state to War state. This originates from the increasing temptation which guide more players to choose strategy ALLD.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. The pathways on the population landscape U with different parameter T at the constant parameters μ = 0.006, c = 0.22, R = 3, Pu = 2.4, S = 0.0.

The purple lines represent the optimal paths from the War state to Peace state. The black lines represent the optimal paths from the Peace state to War state. The white arrows represent the steady state probability fluxes.

https://doi.org/10.1371/journal.pone.0201130.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. The barrier heights, the logarithm of the ratio of the probability of dominant optimal path, the logarithm of the escape time MFPT, entropy production rate versus T, R, Pu.

A: the barrier heights versus T, B: the logarithm of the ratio of the probability of dominant optimal path from Peace state to War state and the probability of dominant optimal path from War state to Peace state versus T, C: the logarithm of the escape time MFPT versus T, D: entropy production rate versus T, E: the barrier heights versus R, F: the logarithm of the ratio of the probability of dominant optimal path from Peace state to War state and the probability of dominant optimal path from War state to Peace state versus R, G: the logarithm of the escape time MFPT versus R, H: entropy production rate versus R, I: the barrier heights versus Pu, J: the logarithm of the ratio of the probability of dominant optimal path from Peace state to War state and the probability of the dominant optimal path from War state to Peace state versus Pu, K: the logarithm of the escape time MFPT versus Pu, L: entropy production rate versus Pu.

https://doi.org/10.1371/journal.pone.0201130.g013

We show the population landscape U and the optimal paths with different parameter reward R at the constant parameters μ = 0.006, c = 0.22, T = 5, Pu = 2.4, S = 0.0 in Fig 14. The optimal paths do not follow the steepest descent paths based on the potential landscape. The two optimal paths depart far away from each other as R increases. Fig 13(E) shows the barrier height ΔU_sw decreases while the barrier height ΔU_sp increases as reward R increases. Fig 13(F) showed the logarithm probability of Peace to War optimal path divided that of War to Peace optimal path decreases as reward R decreases. This shows that as the reward increases, more and more players choose strategy ALLC which makes the Peace state more stable and the War state less stable. Thus, the optimal path from state War to state Peace has more probability or weight (chance) than the opposite path.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. The pathways on the population landscape U with different parameter R at the constant parameters μ = 0.006, c = 0.22, T = 5, Pu = 2.4, S = 0.0.

The purple lines represent the optimal paths from the War state to Peace state. The black lines represent the optimal paths from the Peace state to War state. The white arrows represent the steady state probability fluxes.

https://doi.org/10.1371/journal.pone.0201130.g014

We show the population landscape U and the optimal paths with different parameter punishment Pu at the constant parameters μ = 0.006, c = 0.22, T = 5, R = 3, S = 0.0 in Fig 15. The two optimal paths do not follow the steepest descent optimal paths based on the potential landscape and come closer from each other as Pu increases. Fig 13(I) shows that the barrier height ΔU_sw increases while the barrier height ΔU_sp increases first and then decreases as punishment Pu increases. Fig 13(J) showed that the logarithm of Peace state to War state optimal path probability divided that of War state to Peace state optimal path decreases first and then increases as punishment Pu increases. This shows that as the punishment increases, more players choose strategy ALLC and ALLD which makes the Peace state and War state both become more stable. As the punishment grows bigger, the War state becomes more stable than that of the Peace state. This shows that the punishment Pu has an optimal value to make the Peace state stable.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. The population landscape U with different parameter Pu at the constant parameters μ = 0.006, c = 0.22, T = 5, R = 3, S = 0.0.

The purple lines represent the optimal paths from the War state to Peace state. The black lines represent the optimal paths from the Peace state to War state. The white arrows represent the steady state probability fluxes.

https://doi.org/10.1371/journal.pone.0201130.g015

We explored the escape time for the evolutionary game system. We used the following equation to determine the escape time τ: F ⋅ ∇τ + D ⋅ ∇²τ = −1 [28]. The escape time can be viewed as the average time spent for a system from one state to another [24]. We can set the mean first passage time (MFPT) τ_pw representing the MFPT from Peace state to War state and τ_wp representing the MFPT from War state to Peace state.

Fig 13(C), 13(G) and 13(K) shows that the logarithm of the MFPT increases as their corresponding barrier heights increase corresponding to Fig 13(A), 13(E) and 13(I). The logarithm of the MFPT and barrier heights have the positive correlation as: τ exp(ΔBa). The average kinetic speeds of the state switching along the corresponding optimal paths can be measured by the 1/τ. As the barrier height becomes higher, the escape time becomes longer and the kinetic speed becomes slower. Therefore, the state is more stable with higher barrier height. It is more difficult to switch from one basin of attraction to another with higher barriers between two stable states. It takes more time for switching and the kinetic speed is slower. The MFPT, kinetic speed and barrier height can provide the measurements for quantifying the stability of game theory systems. They can also give more quantitative information about the dynamics of a game theory system. This can help to uncover the underlying mechanisms of the state transitions in the game theory systems.

Fig 13(D), 13(H) and 13(L) shows the entropy production rate versus parameters T, R, Pu among the phase range of the two stable states. EPR decreased as temptation T increases, while Peace state loses its stability and War state becomes more stable shown in Fig 13(D). EPR increased as reward R increases, while War state loses its stability and Peace state becomes more stable shown in Fig 13(H). EPR decreased as punishment Pu increases, while Peace state loses its stability and War state becomes more stable shown in Fig 13(L). We can clearly see that the system with dominant Peace state will cost more dissipation quantified by EPR than that with dominant War state. In order to illustrate this phenomenon, we should explore the linear stability of the repeated Prisoners’ Dilemma game model. The eigenvalues of Peace state have negative real parts and two opposite imaginary parts. This indicates that Peace state is a stable focus which oscillates and spirals to its destiny. Stable focus can switch to an unstable focus which represents a limit cycle. The eigenvalues of War state have negative real parts and no imaginary parts. This indicates that War state is a stable node. Since the stable focus costs more dissipation. This shows that keeping the peace will consume more energy.