Cooperation and Stability through Periodic Impulses

Basic games, where each individual chooses between two strategies, illustrate several issues that immediately emerge from the standard approach that applies strategic reasoning, based on rational decisions, to predict population behavior where no rationality is assumed. These include how mutual cooperation (which corresponds to the best outcome from the population perspective) can evolve when the only individually rational choice is to defect, illustrated by the Prisoner's Dilemma (PD) game, and how individuals can randomize between two strategies when neither is individually rational, illustrated by the Battle of the Sexes (BS) game that models male-female conflict over parental investment in offspring. We examine these questions from an evolutionary perspective where the evolutionary dynamics includes an impulsive effect that models sudden changes in collective population behavior. For the PD game, we show analytically that cooperation can either coexist with defection or completely take over the population, depending on the strength of the impulse. By extending these results for the PD game, we also show that males and females each evolve to a single strategy in the BS game when the impulsive effect is strong and that weak impulses stabilize the randomized strategies of this game.


Introduction
A great deal of game-theoretic research has been devoted to explain the prevalence of cooperation in biological systems as well as in human society. One reason for the vast literature from members of the game theory community on this topic is that their methods do not work for the underlying stage game, the symmetric Prisoner's Dilemma, which pits cooperative behavior against its nemesis of defection. In particular, the only rational option in this PD stage game is to Defect since this strategy strictly dominates Cooperate (i.e. a player is better off defecting than cooperating no matter what the opponent does).
On the other hand, cooperation can be rational when the payoffs of the PD game are modified by assuming some relatedness between the players [1,2], by them playing the game an uncertain number of times [3], or by extending the model to a multi-player (i.e. more than two) public goods game [4]. These predictions are often based on applying either static (e.g. evolutionarily stable strategy (ESS)) or dynamic (e.g. the replicator equation) methods from evolutionary game theory [5] that assumes a large population of agents paired at random to play the game. Population interactions that are structured either spatially (e.g. through nearest neighbors on a lattice) or socially (e.g. through adjacent nodes in a graph) also enhance the evolution of cooperation [6][7][8][9][10][11] as do the stochastic effects of finite populations [12].
To a lesser extent, the question of stability of mixed strategy equilibrium solutions (and their interpretation) has also created controversy in the game theory community [13,14]. This is especially true of two-player non symmetric games due to the result that, at any evolutionarily stable state of such games, players must use pure strategies [15]. The controversy here is clearly demonstrated through typical payoffs used in the Battle of the Sexes game [5,16,17] introduced into biology by Dawkins [18] to model the conflict between males and females concerning their respective contributions to parental investment (see also the Buyer-Seller game [19] that has the same qualitative payoff structure). In the BS stage game, each player has two pure strategies and the only equilibrium solution is for both players to use a mixture of their strategies. Furthermore, the replicator equation applied to this game yields periodic solutions around this mixed strategy equilibrium pair even though Maynard Smith [17] (Chapter 11C) states that ''I am unable to offer illustrative examples, or evidence that such cycles occur.'' In this article, we re-examine the PD and BS stage games from the dynamic perspective where, in addition to the continuous trajectories of evolutionary game theory, there are periodic jumps in the population size. In biological systems, these latter impulsive perturbations may be due to sudden changes in the physical environment (e.g. the effects of climate change or natural disaster) or to intrinsic diurnal/nocturnal and seasonal life history effects in the physiological and reproductive mechanisms of individuals in the population. Impulsive perturbations have also been used to model the effect on human behavior of sudden market corrections or of sudden shifts in the business cycle [20]. We assume that the impulsive ''coefficient'' for an individual depends only on its strategy and analyze the resultant dynamics. In particular, we give analytic conditions for the coefficients in the PD game for the successful initial invasion of Cooperators into a population of Defectors as well as conditions based on stronger impulsive effects for Cooperators to completely take over the system. We also show that these latter conditions applied to the BS game imply global convergence to a monomorphic system where all males use one pure strategy as well as all females. Moreover, when impulsive effects are weak in the BS game, a globally attracting polymorphic state emerges near the mixed equilibrium pair.

Prisoner's Dilemma
The PD stage game is ubiquitous in the game theory literature and so needs no introduction. We follow the standard notation by taking its payoff matrix as where TwRwPwS. The entries in this matrix give the payoff to the row player in a two-player interaction with the column player (e.g. if a player Defects against an opponent who Cooperates, his payoff is T). Since TwR and PwS, Defect strictly dominates Cooperate and so it is the only rational outcome of this one shot game.
To illustrate how periodic impulses can be combined with an evolutionary dynamics, suppose that the replicator equation (see Eq.4a below) models behavioral evolution. From a biological perspective, these dynamics result from a direct correspondence between expected payoff and reproductive success [21]. Specifically, if n 1 and n 2 are the numbers of Defectors and Cooperators respectively in the population at time t, then where N~n 1 zn 2 is the total population size. Here Pn 1 =NzTn 2 =N is the expected payoff to a Defector in a single interaction with a randomly chosen opponent assuming that population size is large. From Taylor and Jonker [21], the population dynamics (Eq.2) implies the frequency p(t) of Defectors in the population (i.e. p~n 1 =N) evolves according to the replicator equation (Eq.4a). Now suppose that individuals die at periodic intervals t~kt for k~1,2,3 ::: . That is, reproductive success (i.e. fitness) in Eq.2 refers only to births. Deaths cause a jump in the solution trajectories of Eq.2 of the form Dn i kt is the number of Defectors immediately after (respectively before) the jump. If the death rate of each strategy type is independent of population size, then for some 0ƒc i ƒ1. The parameter {c i is called the ''impulsive coefficient'' for the i-th strategy. The dynamical system with periodic impulsive effects combines Eq.2 when t=kt with Eq.3 when t~kt. In particular, for every nonnegative initial condition n 1 (0),n 2 (0) ð Þ , this impulsive dynamical system has a unique nonnegative solution for all t §0.
Evolutionary game theory is more concerned with the evolution of strategy frequencies than with how the absolute numbers of strategy users evolve over time. It is shown in File S1 (see also [22]) that dp dt~p (1{p) (P{S)pz(T{R)(1{p) ð Þ for t=kt ð4aÞ for t~kt, k~1,2,3 ::: ð4bÞ where Dp kt ð Þ~p kt z ð Þ{p kt { ð Þ denotes the jump in p at moment t~kt. That is, p kt z ð Þ (respectively, p kt { ð Þ) is the frequency of Defector immediately after (respectively before) the jump. For the analysis of this dynamical system, it is important to notice that the frequency dynamics Eq.4 is independent of population size N. When there are no impulsive effects, we have the standard replicator equation Eq.4a [21]. For this reason, we call Eq.4 the replicator equation with periodic impulses.
Remark. The replicator equation with periodic impulses developed above is based on continuous births throughout the season and deaths only at the end. One consequence of our assumptions is that birth and death rates are independent of population size. It can be shown [22] that dynamics Eq.4 also emerges when birth rates are altered by any strategy-independent background fitness (which is usually assumed to decrease as population size increases). This background fitness can be used to investigate the dynamics of total population size and not only the frequency dynamics as in the standard approach to evolutionary game theory [21]. The dynamics Eq.4 also models other periodic impulses in biological systems such as regular perturbations in the physical environment. It is well-documented [23][24][25] that humans (and other biological species) exhibit more cooperation in the face of natural disasters (also called the disaster syndrome). Such shifting of aggregate population behavior through individuals changing their strategy becomes a positive jump in the proportion of Cooperators. In fact, any impulsive coefficients in Eq.3 satisfying c i ƒ1 are suitable since n i (t) §0 for all t in this case. The interpretation of c i v0 is that the impulse is then beneficial to the i-th strategy.
From Eq.4a, p strictly increases during the season (i.e. dp=dtw0) if 0vpv1 since PwS and TwR. Thus, if the Defector death rate is no higher than the Cooperator (i.e. c 1 ƒc 2 ), the population must evolve to all Defect since p also increases at the end of each season. However, if the death rates benefit Cooperators (i.e. if c 1 wc 2 ), the effect of Defector deaths may offset their higher birth rates and so it is unclear which effect dominates (see Figure 1). In the extreme case where c 1~1 , all Defectors die at the end of the first season and the population is all Cooperate thereafter.
For these reasons, we will assume that 1wc 1 wc 2 for the remainder of this section. The key to understanding the outcome in this scenario is to determine the stability of the boundary equilibria p~0 and p~1 of Eq.4. Heuristically, when p is near 0, the trajectory p(t) during the first season is approximated by p(t)~p(0)e (T{R)t since, from Eq.4a, dp=dt&(T{R)p. Also, from Eq.4b, the jump at the end of this season is It is proved analytically in File S1 (see also [22]) that p~0 is stable if inequality Eq.5 is true and unstable (i.e. p t ð Þ diverges from 0 if it is initially close) if this inequality is reversed (i.e. T{R ð Þtw ln 1{c 2 ð Þ= 1{c 1 ð Þ ð Þ ). It is also shown there that all Defect is stable if and unstable if the inequality is reversed. In fact, the dynamic stability of the boundary equilibria for threshold parameters when Eq.5 or Eq.6 is an equality is also characterized there. Besides one exceptional case discussed in the final paragraph of this section when both Eq.5 and Eq.6 are equalities, there is a unique interior t-periodic trajectory (i.e. a p Ã (t) with 0vp Ã (t)v1 for all t §0 and p Ã kt z ð Þ~p Ã (0) for all k~1,2,3, ::: ) if and only if either both boundary equilibria are unstable or both are stable. In the first (respectively, second) case, p Ã (t) is globally stable (respectively, unstable). Finally, if exactly one boundary equilibrium is stable, then it is globally stable in that it attracts all interior trajectories.
These analytic results from File S1, that are summarized in the preceding paragraph, are illustrated in Figures 1 and 2 for non threshold cases. In Figure 1, trajectories of Eq.4 are given for fixed payoff parameters and four different values of In Figure 2, Cw0 is fixed and the stability of boundary equilibria and interior t-periodic trajectories is characterized in different regions of the space with parameters T{R and P{S.
For small values of C, the population evolves to all Defect (Figs. 1a and 1b). An initial small population of Cooperators can successfully invade a population of Defectors once P{S ð ÞvC. Cooperators then completely take over the population if inequality T{R ð ÞvC also holds ( Fig. 1d and quadrant III in Fig. 2); otherwise the system approaches a globally stable t-periodic state consisting of a mixture of Cooperators and Defectors (Fig. 1c and quadrant IV in Fig. 2). If P{S ð ÞwC as in quadrants I and II of Figure 2, either all Defect completely takes over or we have a bistable situation where all Cooperate can persist if their initial frequency is sufficiently high.
It is instructive to consider the case of small impulsive effects (i.e. when the impulsive coefficients c 1 wc 2 are both close to 0). By Taylor  favor of Cooperate. By inequalities Eq.5 and Eq.6, if the positive payoff advantage during the season to Defect over Cooperate is always less than this average impulsive effect (i.e. if T{R and P{S are both less than C), Cooperators invade and take over the population. This intuitive result can be generalized to all situations where impulsive effects are nearly equal (i.e. c 1 is close to c 2 but they are not necessarily close to 0) (see also File S1). The central message here is that a small difference in the death rates of Cooperators and Defectors can have a major impact on the evolution of cooperation.
In fact, if we call C defined in Eq.7 the average impulsive effect over one season in favor of Cooperate for any choice of c 1 wc 2 , the intuitive result of the previous paragraph remains true.
In the exceptional case where T{R~C~P{S, the boundary equilibria are neutrally stable and every trajectory p(t) for any initial condition 0vp(0)v1 is t-periodic. Interestingly, the special payoffs for the PD game that satisfy T{R~P{S have attained prominence recently since this class includes the simplified PD games [1,26] with payoff matrix Here b is the payoff benefit a player interacting with a Cooperator gains and c is the cost paid by Cooperators. Under the usual assumption that bwcw0, the entries in this simplified payoff matrix have the same ordering as in Eq.1. From Eq.5 and Eq.6, if cvC (respectively, cwC), then all Cooperate (respectively, all Defect) is the final outcome. Unlike other studies on the simplified PD game [26] where the emergence of Cooperative behavior often depends on the cost-benefit ratio b=c, here it depends only on the cost of Cooperation. The size of the payoff benefit has no impact on our results since neither the replicator equation Eq.4a nor the impulse Eq.4b depends on b. In our model, it is the impulsive benefit C that replaces the payoff benefit b. In particular, Cooperation emerges if and only if the impulsive benefit to Cooperators is greater than the cost paid by Cooperators. That is, periodic impulses that favor cooperation provide a mechanism that promotes the evolution of cooperation.
If parental investment costs C, the benefit gained from an offspring is G and the cost of a long engagement is E, then the payoffs to males and females are given in the following bimatrix (e.g. a philanderer receives the benefit G against a fast female whose net payoff is then G{C).
coy fast philander faithful With the usual assumptions that these payoffs satisfy 0vEvGvCv2 G{E ð Þ, the characteristic feature of the BS game is the cyclical character of male and female best responses. If females are coy, it pays males to be faithful; if males are faithful, it pays females to be fast; if females are fast, it pays males to philander; and if males philander, it pays females to be coy. This characteristic also leads to cycling in the standard evolutionary dynamics that is concerned with the evolution of strategy frequencies.
Let p be the frequency of philanders in the male population and q be the frequency of coy females in their population. The bimatrix replicator equation is then This two-dimensional dynamics on the unit square has the unique interior equilibrium p Ã ,q Ã ð Þ~E= C{GzE ð Þ , C=2 G{E ð Þ ð Þ and all trajectories are periodic orbits surrounding p Ã ,q Ã ð Þ [5,16,17].  considered by Dawkins [18] that has p Ã ,q Ã ð Þ~(3=8,5=6). We again assume that trajectories of the replicator equation are based on male and female births throughout the season and that there are jumps at t~kt for k~1,2,3, ::: due to deaths at the end of the season. If the male (respectively, female) death rate is c i (respectively, y i ) for their i-th strategy, these latter periodic impulses are Dq~q 1{q ð ÞV q ð Þ ð11bÞ  and the neutral interior equilibrium is p Ã ,q Ã ð Þ~(3=8,5=6). The minimum period T min is about 4p s &3:55. In Figure 3a-b, c 1~y1~0 (no impulses) and the time step is from 1 to 15. All the interior trajectories are periodic orbits surrounding p Ã ,q Ã ð Þ. In Figure 3c, c 1~0 :1 and y 1~{ 0:5 (strong impulses) and the time step is from 1 to 3. Since G{ C 2 {Ew{C m and Ev{C f , from Table 1, boundary equilibrium 0,1 ð Þ is stable and all the interior trajectories converge to 0,1 ð Þ. In Figure 3d, c 1~y1~0 :005 (weak impulses) and the time steps are respectively 1 to 10, 1z10 5 to 10z10 5 and 1z10 6 to 10z10 6 . All the interior trajectories converge to an attracting set within 0.015 of the interior equilibrium p Ã ,q Ã ð Þ. In Figure 3e-f, c 1~y1~0 :3 (intermediate impulses) and the time step is from 1 to 15. Clearly, interior trajectories do not always evolve to either a boundary equilibrium or to a set close to p Ã ,q Ã ð Þ. doi:10.1371/journal.pone.0009882.g003 at t~kt where U p ð Þ: c 2 {c 1 ð Þ= 1{c 1 p{c 2 1{p ð Þ ð Þ and V q ð Þ: y 2 {y 1 ð Þ = 1{y 1 q{y 2 1{q ð Þ ð Þ . The bimatrix replicator equation with periodic impulses combines Eq.10 at t=kt with Eq.11 at t~kt.
The analysis of this impulsive dynamical system is more difficult than the replicator equation with periodic impulses for the PD game. We will assume that c i v1 and y i v1 for i~1,2. Each edge of the unit square is then invariant (as is the interior of the square). For example, on the edge where q~0, we have dp dt~p This impulsive dynamics is the same as that for the PD game with simplified payoffs Eq.8 given by bwc~C=2.
That is, on the edge where all females are fast, the game is one where males are playing a simplified PD game among themselves. Thus, all males will eventually be faithful if they have a lower death rate than philanderers (i.e., c 2 vc 1 ) that satisfies C=2vC m where C m :ln 1{c 2 ð Þ= 1{c 1 ð Þ ð Þ =t is the impulsive benefit for faithful males (cf. the discussion in the PD section about Eq.7). If C m v0, it is an impulsive benefit for philandering males. On the other hand, philanderers will take over if their death rate is lower or if C=2wC m . In this section, we will not consider threshold parameter values where all trajectories on an edge are t-periodic. Similarly, on the edge where all males are faithful, there is a simplified PD game among females (where c is now identified with {E). Let C f be the impulsive benefit for fast females (i.e., C f :ln 1{y 2 ð Þ= 1{y 1 ð Þ ð Þ =t). Thus, all females will eventually be fast (respectively, coy) if the cost of a long engagement is greater than (respectively, less than) the impulsive benefit for coy females, i.e. Ew{C f ) (respectively, Ev{C f ).
In fact, p,q ð Þ~0,0 ð Þ will be stable for the bimatrix replicator equation with periodic impulses (Eq.10 and Eq.11) on the unit square if males are eventually faithful on the first edge and females are eventually fast on the second edge. This result is indicated in the first row of Table 1 that summaries the stability of all four vertices of the unit square. The proof is in File S1 where it is also shown that, if one vertex is stable, then it is globally stable (i.e. all trajectories in the interior of the unit square converge to it). In particular, at most one vertex can be stable.
From Table 1, if impulsive effects are strong enough on any edge to reverse the flow of the replicator equation Eq.10 there (i.e., if any of the inequalities C m wC=2, C f wC{G, C m v{ G{C=2{E ð Þ , or C f v{E are true), then there is a globally stable pure strategy pair for males and females. In particular, these periodic impulses have removed the characteristic interior cycles of the BS game and replaced them with global stability at a vertex. Figure 3c illustrates a typical trajectory of Eq.10, Eq.11 when one of the stability conditions for the vertex (1,0) in Table 1 vC=2 and {EvC f vC{G), the limiting behavior of each trajectory on the boundary of the unit square is the same whether or not periodic impulses Eq.11 are combined with the bimatrix replicator dynamics; namely, all these trajectories evolve to the first vertex encountered in a counterclockwise direction. On the other hand, no interior trajectory evolves to a point on the boundary. Figure 3e-f illustrates a typical trajectory when impulses are of intermediate strength. Notice that this trajectory does not surround (p Ã ,q Ã ) since the impulses occur before it has enough time to do so. In fact, through simulations (see File S1), it is apparent that the properties of such trajectories are quite complex with multiple t-periodic solutions possible. The number N of these solutions appears to depend linearly on the ratio of t to the minimum period T min of interior periodic cycles of the bimatrix replicator dynamics Eq.10 in that it is approximated by N~2 t=T min ½ z1 where [ ] is the integer part of a positive real number. Proofs of these conjectures suggested by simulations are beyond the current techniques available to analyze these impulsive systems. We are hopeful that game-theoretic intuition will extend these techniques in future research and, in the process, improve our understanding of how arbitrary periodic impulses affect the evolutionary dynamics, especially for non symmetric two-player games (such as the BS game) with a unique equilibrium in the interior of the strategy space.
Finally, for weak impulses (i.e. for C m and C f both close to 0 and at least one nonzero) with tvvT min , simulations show that all interior trajectories evolve to a small neighborhood of (p Ã ,q Ã ) (see Figure 3d). Specifically, although the limiting behavior is not exactly the interior equilibrium, the large periodic cycles of the bimatrix replicator dynamics are replaced by orbits that become arbitrarily close to this equilibrium as the impulsive effect weakens. In fact, except for the one t-periodic solution, it appears that all trajectories approach the single orbit in Figure 3d (which is not t-periodic).
In summary, both strong and weak impulsive effects promote stability in BS games. Strong impulses imply the system has a globally stable outcome where all males adopt the same strategy as do all females (i.e. both sexes exhibit monomorphic behavior). On the other hand, weak impulses eliminate the wild fluctuations typical of the periodic cycles of the bimatrix replicator dynamics Eq.10 and replace them with an attracting set near its interior equilibrium that consists of a stable polymorphic population of males and females.

Results and Discussion
Our model combining periodic impulses with an evolutionary dynamics is based on several simplifying assumptions. First, population sizes are assumed to be sufficiently large that stochastic effects due to finite populations are ignored in the deterministic dynamics and fitness is given by expected payoff as in the original development of the replicator equation [21]. Our assumption that death rates depend only on strategy type then implies total population size does not influence the evolution of strategy frequencies given by our replicator equation with periodic impulses. Moreover, in the BS game, we have adopted the unstated common assumption of evolutionary game theory applied to asymmetric games with a bimatrix payoff matrix [5] that Table 1. Stability of Boundary Equilibria p,q ð Þ (p is the frequency of males who philander and q is the frequency of coy females). individual fitness is based on one random interaction per unit time between different types of individuals. The analysis of the effects of periodic impulses becomes more complex when finite populations and/or unequal population sizes of different types in asymmetric games are included. In particular, the dynamics of total population size must then be taken into account. From this perspective, our analysis of the frequency effects of periodic impulses can be viewed as a base model against which these more complex systems can be compared, in much the same way that the replicator equation of evolutionary game theory has served as a means to gain an intuitive understanding of behavioral evolution. Our analytic results, that characterize when periodic impulses favoring cooperation in the PD game can overcome the selective advantage of defection and when both strong and weak impulses have a stabilizing effect in the BS game, can then be tested (perhaps numerically) to see if they continue to hold in more complicated models that do not satisfy our simplifying assumptions.