## Figures

## Abstract

To be the fittest is central to proliferation in evolutionary games. Individuals thus adopt the strategies of better performing players in the hope of successful reproduction. In structured populations the array of those that are eligible to act as strategy sources is bounded to the immediate neighbors of each individual. But which one of these strategy sources should potentially be copied? Previous research dealt with this question either by selecting the fittest or by selecting one player uniformly at random. Here we introduce a parameter that interpolates between these two extreme options. Setting equal to zero returns the random selection of the opponent, while positive favor the fitter players. In addition, we divide the population into two groups. Players from group select their opponents as dictated by the parameter , while players from group do so randomly irrespective of . We denote the fraction of players contained in groups and by and , respectively. The two parameters and allow us to analyze in detail how aspirations in the context of the prisoner's dilemma game influence the evolution of cooperation. We find that for sufficiently positive values of there exist a robust intermediate for which cooperation thrives best. The robustness of this observation is tested against different levels of uncertainty in the strategy adoption process and for different interaction networks. We also provide complete phase diagrams depicting the dependence of the impact of and for different values of , and contrast the validity of our conclusions by means of an alternative model where individual aspiration levels are subject to evolution as well. Our study indicates that heterogeneity in aspirations may be key for the sustainability of cooperation in structured populations.

**Citation: **Perc M, Wang Z (2010) Heterogeneous Aspirations Promote Cooperation in the Prisoner's Dilemma Game. PLoS ONE 5(12):
e15117.
https://doi.org/10.1371/journal.pone.0015117

**Editor: **James A. R. Marshall, University of Sheffield, United Kingdom

**Received: **August 9, 2010; **Accepted: **October 22, 2010; **Published: ** December 6, 2010

**Copyright: ** © 2010 Perc, Wang. This is an open-access article distributed under the
terms of the Creative Commons Attribution License, which permits unrestricted use,
distribution, and reproduction in any medium, provided the original author and
source are credited.

**Funding: **MP acknowledges support from the Slovenian Research Agency (Grant No. Z1-2032).
ZW acknowledges support from the Center for Asia Studies of Nankai University
(Grant No. 2010-5) and from the National Natural Science Foundation of China
(Grant No. 10672081). The funders had no role in study design, data collection
and analysis, decision to publish, or preparation of the manuscript.

**Competing interests: ** The authors have declared that no competing interests exist.

## Introduction

Understanding the evolution of cooperation among selfish individuals in human and animal societies remains a grand challenge across disciplines. Evolutionary games are employed frequently as the theoretical framework of choice in order to interpret the emergence and survival of cooperative behavior [1]–[6]. The prisoner's dilemma game, in particular, has attracted considerable interest [7]–[11] as the essential yet minimalist example of a social dilemma. In the original two-person one-shot game the two players have two strategies to choose from (cooperation and defection), and their payoffs depend on the simultaneous decision of both. If they choose to cooperate they will receive the highest collective payoff, which will be shared equally among them. Mutual defection, on the other hand, yields the lowest collective payoff. Yet to defect is tempting because it yields a higher individual payoff regardless of the opponent's decision. It is thus frequently so that both players choose not to cooperate, thus procreating the inevitable social dilemma. In reality, however, interactions may be repeated and the reputation of players compromised [12], [13]. Additionally, individuals may alter with whom they interact [14], and different behaviors may be expressed when participants in a social interaction occupy different roles [15]–[18]. Such and similar considerations have been very successful in elucidating why the unadorned scenario of total defection is often at odds with reality [19], where it is clear that both humans and animals cooperate to achieve what would be impossible by means of isolated efforts. Mechanisms supporting cooperation identified thus far include kin selection [20] as well as many others [6], [21]–[23], and there is progress in place aimed at unifying some of these approaches [24], [25].

Probably the most vibrant of all in recent years have been advances building on the seminal paper by Nowak and May [26], who showed that spatial structure may sustain cooperation without the aid of additional mechanisms or strategic complexity. Although in part anticipated by Hamilton's comments on viscous populations [20], it was fascinating to discover that structured populations, including complex and social networks [27]–[29], provide an optimal playground for the pursuit of cooperation. Notably, a simple rule for the evolution of cooperation on graphs and social networks is that natural selection favors cooperation if , where is the benefit of the altruistic act, is its cost, while is the average number of neighbors [30]. This is similar to Hamilton's rule stating that should be larger than the coefficient of genetic relatedness between individuals [20]. In fact, on graphs and social networks the evolution of altruism can thus be fully explained by the inclusive fitness theory since the population is structured such that interactions are between genetic relatives on average [25], [31], [32].

According to the “best takes all” rule [33], [34] players are allowed to adopt the strategy of one of their neighbors, provided its payoff is higher than that from the other neighbors as well as from the player aspiring to improve by changing its strategy. Based on this relatively simple setup, it was shown that on a square lattice cooperators form compact clusters and so protect themselves against being exploited by defectors. The “best takes all” strategy adoption rule is, however, just one of the many possible alternatives that were considered in the past. Other examples include the birth-death and imitation rule [35], the proportional imitation rule [36], the reinforcement learning adoption rule [37], or the Fermi-function based strategy adoption rule [38]. The latter received substantial attention, particularly in the physics community, for its compatibility with the Monte Carlo simulation procedure and the straightforward adjustment of the level of uncertainty governing the strategy adoptions . However, with this rule the potential donor of the new strategy is selected uniformly at random from all the neighbors. This is somewhat untrue to what can be observed in reality, where in fact individuals typically aspire to their most successful neighbors rather than just somebody random. In this sense the “best takes all” rule seems more appropriate, although it fails to account for errors in judgment, uncertainty, external factors, and other disturbances that may vitally affect how we evaluate and see our co-players. Here we therefore propose a simple tunable function that interpolates between the “best takes all” and the random selection of a neighbor in a smooth fashion by means of a single parameter . In this sense the parameter acts as an aspiration parameter, determining to what degree neighbors with a higher payoff will be considered more likely as potential strategy sources than other (randomly selected) neighbors.

Aiming to further disentangle the role of aspirations, we also consider two types of
players, denoted by type and
, respectively. While players of type
conform to the aspirations imposed by the value of the
aspiration parameter , type
players choose whom to potentially copy uniformly at random
irrespective of . We denote the fraction of type
and players by
and , respectively. This
additional division of players into two groups is motivated by the overwhelming
evidence indicating that heterogeneity, almost irrespective of its origin, promotes
cooperative actions. Most notably associated with this statement are complex
networks, including small-world networks [39]–[41], random regular graphs [5], [42], scale-free
networks [43]–[48], as well as adaptive and growing networks [49]–[56]. Furthermore, we
follow the work by McNamara *et al.* on the coevolution of choosiness
and cooperation [57], in particular by omitting the separation of the
population on two types of players and introducing the heterogeneity by means of
normally distributed individual aspiration levels that are then also subject to
evolution.

At present, we thus investigate how aspirations on an individual level affect the evolution of cooperation. Having something to aspire to is crucial for progress and betterment. But how high should we set our goals? Should our role models be only overachievers and sports heroes, or is it perhaps better to aspire to achieving somewhat more modest goals? Here we address these questions in the context of the evolutionary prisoner's dilemma game and determine just how strong and how widespread aspirations should be for cooperation to thrive best. As we will show, a strong drive to excellence in the majority of the population may in fact act detrimental on the evolution of cooperation, while on the other hand, properly spread and heterogeneous aspirations may be just the key to fully eliminating the defectors. We will show that this holds irrespective of the structure of the underlying interaction network, as well as irrespective of the level of uncertainty by strategy adoptions . In addition, the presented results will be contrasted with the output of a simple coevolutionary model, where individual aspirations will also be subject to evolution by means of natural selection. We will conclude that appropriately tuned aspirations may be seen as a universally applicable promoter of cooperation, which will hopefully inspire new studies along this line of research.

## Results

Depending on the interaction network, the strategy adoption rule and other simulation
details (see *e.g.*
[4], [6], [23], [58]), there always
exists a critical cost-to-benefit ratio at which cooperators
in the prisoner's dilemma die out. This is directly related to Hamilton's
rule stating that natural selection favors cooperation if
is larger than the coefficient of genetic relatedness
between individuals [20]. If the aspiration parameter
(note that then the division of players to those of type
and those of type is irrelevant),
, and the interaction network is a square lattice, then, in
our case, . In what follows, we will typically set
slightly below this threshold to
and examine how different values of
, ,
, as well as different interaction networks influence the
outcome of the prisoner's dilemma game.

It is instructive to first examine characteristic snapshots of the square lattice for
different values of and
. Results presented in Fig. 1 hint to the conclusion that heterogenous
aspiration to the fittest promotes cooperation, although the details of this claim
depend somewhat on the value of the aspiration parameter
. For small values of it is best if all the
players, *i.e.*
, aspire to their slightly (note that
is small) fitter neighbors and thus none actually choose the
potential strategy sources uniformly at random. This can be deduced from the top
three panels of Fig. 1 if
compared from left to right. For large , however, it is best
if only half of the players, *i.e.*
, aspire to their most fittest neighbors, while the other
half chooses their role models randomly. This can be observed if one compares the
bottom three panels of Fig. 1
with one another, although the difference in the overall density of cooperators
(depicted green and blue) between the middle and the right panel is fairly small.
Finally, the role of the aspiration parameter is more clear cut since larger
clearly favor the cooperative strategy if compared to small
. This can be observed if comparing the snapshots presented
in Fig. 1 vertically.

Top row depicts results for the aspiration parameter while the bottom row features results for . In both rows the fraction of type players is , and from left to right. Cooperators of type and are colored green and blue, respectively. Defectors of type and , on the other hand, are colored red and yellow. If comparing the snapshots vertically, it can be observed that larger values of (top , bottom ) clearly promote the evolution of cooperation. The scenario from left to right via increasing the fraction of type players is not so clear cut. For (top row) we can conclude that larger favor cooperative behavior, as clearly the cooperators flourish more and more from the left toward the right panel. For (bottom row), however, it seems that for (bottom middle) cooperators actually fare better then for both (bottom left) and (bottom right). Hence, the conclusion imposes that for higher values an intermediate (rather than the maximal, as is the case for lower ) fraction of type players (those that aspire to their most fittest neighbors only) is optimal for the evolution of cooperation. Results in all panels were obtained for and .

Since the snapshots presented in Fig. 1 can be used primarily for an initial qualitative assessment of the impact of heterogeneous aspirations, we present in Fig. 2 the fraction of cooperators (left) and the critical cost-to-benefit ratio (right) in dependence on for different values of . It can be observed that the promotion of cooperation for the optimal combination of the two parameters, being and , is really remarkable. The fraction of cooperators rises from to , while the critical cost-to-benefit ratio rises a full order of magnitude from to . As tentatively deduced from the lower three snapshots in Fig. 1, it can also be observed that for high values of an intermediate fraction of type players is optimal for the evolution of cooperation. Conversely, for low the fraction of cooperators and the critical cost-to-benefit ratio both increase monotonously with increasing . If, however, selecting a particular value of , then the impact of the aspiration parameter is always such that cooperation is the more promoted the larger the value of . This can be observed clearly from both panels, and indeed seems like the main driving force behind the elevated levels of cooperation. Fine-tuning the fraction of players making use of the aspiration to the fittest (from downwards since the limit trivially returns the random selection of potential strategy sources) at high can rise the cooperation level further, but more in the sense of minor adjustments, similarly as was observed in the past for the impact of uncertainty by strategy adoptions [42] or the impact of noise [59].

Left panel depicts the density of cooperators in dependence
on the fraction of type players
for different values of the aspiration parameter
. Right panel depicts the critical cost-to-benefit
ratio at which cooperators die out, *i.e.*
, in dependence on for different
values of . Results in both panels convey the message that low
values of require a high fraction of type
players for cooperation to flourish. Conversely,
higher values of sustain
cooperation optimally if only half () of the
players aspires to their most fittest neighbors while the rest chooses whom
to potentially imitate uniformly at random. Optimal conditions for the
evolution of cooperation thus require and
to be fine-tuned jointly. Depicted results in both
panels were obtained for and
.

Aiming to generalize the validity of our results, we present in Fig. 3 the fraction of cooperators in dependence on for different values of as obtained on the random regular graph (left) and the small-world network (right). The goal is to test to what extend above conclusions hold also on interaction networks other than the square lattice, in particular such that are more complex and spatially heterogenous. If comparing the obtained results with those presented in the left panel of Fig. 2, it seems save to conclude that they are to a very large degree qualitatively identical. Some differences nevertheless can be observed. The first is that what constitutes a high limit is a bit higher on complex networks than on the regular lattice. Note that for the optimal fraction of type players is practically still . Even for the bell-shaped dependence on is far less pronounced than on the square lattice, and the optimal (the peak of ) is closer to than . The second difference is, looking relatively to the starting point at , that the promotion of cooperation due to positive and is somewhat less prolific. This is, however, not that surprising since complex networks in general promote cooperation already on their own [6], and thus secondary promotive mechanisms may therefore become less expressed. Aside from these fairly mild differences though, we can conclude that heterogenous aspirations do promote cooperation irrespective of the underlying interaction network, and that the details of the promotive effect are largely universal and predictable.

Left panel depicts the density of cooperators in dependence on the fraction of type players for different values of the aspiration parameter for the RRG. Right panel depicts in dependence on for different values of for the Watts-Strogatz SW network with the fraction of rewired links equalling . These results are in agreement with those presented in Fig. 2, supporting the conclusion that the impact of heterogenous aspirations on the evolution of cooperation is robust against alterations of the interaction network. As on the square lattice, low, but also intermediate, values of require for cooperation to thrive, while higher values of sustain cooperation optimally only if . Depicted results in both panels were obtained for and .

Next, we proceed with examining how positive values of and fare under different levels of uncertainty by strategy adoptions. The latter can be tuned via [see Eq. (3)], which acts as a temperature parameter in the employed Fermi strategy adoption function [38]. Accordingly, when all information is lost and the strategies are adopted by means of a coin toss. Note that this aspect has thus far not received any attention here as was fixed. The matter is not trivial to address because uncertainty and noise can have a rather profound impact on the evolution of cooperation [40], [42], [59]–[62], and thus care needs to be exercised. The safest and most accurate way to approach the problem is by means of phase diagrams. Since we have two additional parameters ( and ) against which we want to test the impact of , we determined full phase diagrams for six characteristic combinations of and on the square lattice. Obtained results are presented in Fig. 4. Notably, the phase diagram presented in the top left panel of Fig. 4 is well-known, implying the existence of an optimal level of uncertainty for the evolution of cooperation, as was previously reported in [42], [59]. In particular, note that the transition line is bell shaped, indicating that is the optimal temperature at which cooperators are able to survive at the highest value of . Importantly though, this phenomenon can only be observed on interaction topologies lacking overlapping triangles [63], [64]. Interestingly, increasing from (top row) to (bottom row) completely eradicates (as do interaction networks incorporating overlapping triangles) the existence of an optimal , and in fact qualitatively reverses the dependence. The transition line has an inverted bell-shaped outlay, indicating the existence of the worst rather than an optimal temperature for the evolution of cooperation. The qualitative changes are less profound if is kept constant at (top row) and increases (from left to right). Still, however, the bell-shaped outlay of the transition gives way to a monotonically increasing curve, saturating only for high . These qualitative changes in the phase diagrams imply that increasing the aspiration parameter or the fraction of players abiding to it (type ) effectively alters the interaction network. While the square lattice obviously lacks overlapping triangles and thus enables the observation of an optimal for small enough values of and (or a combination thereof, as is the case in the top left panels), trimming the likelihood of who will act as a strategy source and how many players will actually aspire to their fittest neighbors seems to effectively enhance linkage among essentially disconnected triplets and thus precludes the same observation. It is instructive to note that a similar phenomenon was observed recently in public goods games, where the joint membership in large groups was also found to alter the effective interaction network and thus the impact of uncertainly on the evolution of cooperation [64].

Top row depicts results for the aspiration parameter
while the bottom row features results for
. In both rows the fraction of type
players is
, and
from left to right. The outline of panels thus
corresponds to the snapshots presented in Fig. 1. Thin blue and thick red lines
mark the border between stationary pure and
phases and the mixed phase,
respectively. In agreement with previous works [42], [63], it can be observed that
for and (top left)
there exists an intermediate uncertainty in the strategy adoption process
(an intermediate value of ) for which the
survivability of cooperators is optimal, *i.e.*
is maximal. Conversely, while the borderline
separating the pure and the mixed
phase for all the other combinations of
and exhibits a
qualitatively identical outlay as for the and
case, the transition is
qualitatively different and very much dependent on the particularities
players' aspirations. Note that in all the bottom panels there exist an
intermediate value of for which
is minimal rather than maximal, while towards the
large limit increases,
saturating only for (not shown).
In the top middle and right panel, on the other hand, the bell-shaped outlay
of the transition gives way to a monotonically increasing
curve, again saturating only for . It can thus
be concluded that, while the aspiration based promotion of cooperation is
largely independent of , the details
of phase transition are very much affected, which can be attributed to an
effective alterations of the interaction network due to preferred strategy
sources (see also main text for details).

In terms of the facilitation of cooperation, however, it can be concluded that the
promotive impact of positive values of and
prevails irrespective of . By comparing the
extend of pure and mixed regions for different
pairs of the two parameters, we can observe that for small values of
(top panels in Fig. 4) it is best if all the players, *i.e.*
, aspire to their slightly (note that
is small) fitter neighbors, while for large
(bottom panels in Fig. 4) it is best if only approximately half of
the players, *i.e.*
, aspire to their most fittest neighbors. The same
conclusions were stated already upon the inspection of results presented in Figs. 2 and 3, and with this we now affirm that not only is
the promotion of cooperation via heterogeneous aspirations robust against
differences in the interaction networks, but also against variations in the
uncertainty by strategy adoptions.

It remains of interest to elucidate why then cooperative behavior is in fact promoted
by positive values of and
. To provide answers, we show in Fig. 5 time courses of
for different characteristic combination of the two main
parameters that we have used throughout this work. What should attract the attention
is the fact that in the most early stages of the evolutionary process (note that
values of were recorded also in-between full iteration steps) it
appears as if defectors would actually fare better than cooperators. This is
actually what one would expect, given that defectors are, as individuals, more
successful than cooperators and will thus be chosen more likely as potential
strategy donors if is positive. This
should in turn amplify their chances of spreading and ultimately result in the
decimation of cooperators (indeed, only between 20–30% survive). Quite
surprisingly though, the tide changes fairly fast, and as one can observe from the
presented time courses, frequently the more so the deeper the initial downfall of
cooperators. We argue that for positive values of and
a negative feedback effect occurs, which halts and
eventually reverts what appears to be a march of defectors toward dominance. Namely,
in the very early stages of the game defectors are able to plunder very efficiently,
which quickly results in a state where there are hardly any cooperators left to
exploit. Consequently, the few remaining clusters of cooperators start recovering
lost ground against weakened defectors. Crucial thereby is the fact that the
clusters formed by cooperators are impervious to defector attacks even at high
values of because of the positive selection towards the fittest
neighbors acting as strategy sources (occurring for ). In a sea of
cooperators this is practically always another cooperator rather than a defector
trying to penetrate into the cluster. This newly identified mechanism ultimately
results in fairly widespread cooperation that goes beyond what can be warranted by
the spatial reciprocity alone (see *e.g.*
[6]), and this
irrespective of the underlying interaction network and uncertainty by strategy
adoptions.

Results are presented for the aspiration parameters (solid lines) and (dashed lines), each for three different fractions of type players , as depicted on the figure. The crucial feature of all time courses is the initial temporary downfall of cooperators, which sets in for all depicted combinations of and . Quite remarkably, what appears to become an ever faster extinction eventually becomes a rise to, at least in some cases, near-dominance. Note that the horizontal axis is logarithmic and that values of were recorded also in-between full iteration steps to ensure a proper resolution. Depicted results were obtained for and .

Finally, it is instructive to examine whether an optimal intermediate value of , determining the aspiration level of player , can emerge spontaneously from an initial array of normally distributed values. This would imply that natural selection indeed favors individuals with a specific aspiration level, which would in turn extend the credibility of thus far presented results that were obtained primarily in a top-down manner [by optimizing a population-level property (cooperation) by means of an appropriate selection of parameters determining the aspiration level of individuals]. For this purpose we omit the division of the population on players of type and , and initially assign to every player a value that is drawn randomly from a Gaussian distribution with a given mean and standard deviation . Then if player adopts the strategy from player also becomes equal to (see Methods for details). Results obtained with this alternative coevolutionary model are presented in Fig. 6. It can be observed that the initial Gaussian distribution sharpens fast around an intermediate value of , which then gradually becomes more and more frequent in the population as the natural selection spontaneously eliminates the less favorable values that warrant a lower individual fitness. The final state is a population where virtually all players have an identical aspiration level , and accordingly, the outcome in terms of the stationary density of cooperators is equal to that obtained with the original model having and . In this sense the preceding results are validated and their generality extended by means of a bottom-up approach entailing a spontaneous coevolution towards an intermediate individually optimal aspiration level. We note, however, that with this simple coevolutionary model the result that heterogeneous aspirations promote cooperation is not exactly reproduced. Further studies on more sophisticated models incorporating coevolving aspirations are required to arrive spontaneously at a heterogeneous distribution of individual aspiration levels. Inspirations for this can be found in the recent review on coevolutionary games [23], and we are looking forward to further developments in this direction.

Presented results were obtained with the alternative model where players are not divided into two groups and initially every player is assigned a random aspiration level drawn from a Gaussian distribution with the mean and standard deviation . The main panel depicts the distributions of individual aspiration levels as recorded at (solid black line), (dashed red line) and (dotted green line) full iteration steps. The fixation towards a dominant average value due to natural selection is evident since the interval of values still present in the population becomes more and more narrow as time progresses. The inset shows the convergence of (solid gray line) and (dotted blue line). The initial temporary downfall of cooperators, followed by the rise to near-dominance, is well-expressed also in the coevolutionary setup, and the stationary density agrees well with the results obtained by means of the original model with and (compare with the dashed cyan line in Fig. 5). Note that in the inset the horizontal axis is logarithmic and that values of and were recorded also in-between full iteration steps to ensure a proper resolution. Depicted results were obtained for and .

## Discussion

We have shown that heterogenous aspiration to the fittest, *i.e.* the
propensity of designating the most successful neighbor as being the role model, may
be seen as a universally applicable promoter of cooperation that works on different
interaction networks and under different levels of stochasticity. For low and
moderate values of the aspiration parameter cooperation thrives
best if the total population abides to aspiring to the fittest. For large values of
, however, it is best if only approximately half of the
players persuasively attempt to copy their most successful neighbors while the rest
chooses their opponents uniformly at random. The optimal evolution of cooperation
thus requires fine-tuning of both, the density of players that are prone to aspiring
to the fittest, as well as the aspiration parameter that determines how fit a
neighbor actually must be in order to be considered as the potential source of the
new strategy. In addition, by studying an alternative model where individual
aspiration levels were also subject to evolution, we have shown that an intermediate
value of the aspiration level emerges spontaneously through natural selection, thus
supplementing the main results by means of a coevolutionary approach.

Notably, the extensions of the prisoner's dilemma game we have considered here seem very reasonable and are in fact easily justifiable with realistic examples. For example, it is a fact that people will, in general, much more likely follow a successful individual than somebody who is just struggling to get by. Under certain adverse circumstances, like in a state of rebelion or in revolutionary times, however, it is also possible that individuals will be inspired to copy their less successful partners or those that seem to do more harm than good. In many ways it seems that the ones who are satisfied with just picking somebody randomly to aspire to are the ones that are most difficult to come by. In this sense the rather frequently adopted random selection of a neighbor, retrieved in our case if (or equivalently ), seems in many ways like the least probable alternative. In this sense it is interesting to note that our aspiring to the fittest becomes identical to the frequently adopted, especially in the early seminal works on games on grids [26], [33], [34], “best takes all” adoption rule if , in Eq. (2), and in Eq. (3). Although in our simulations we never quite reach the “best takes all” limit, and thus a direct comparison with the seminal works is somewhat circumstantial, we find here that the intermediate regions of heterogenous aspirations offer fascinating new insights into the evolution of cooperation, and we hope that this work will inspire future studies, especially in terms of understanding the emergence of successful leaders in societies via a coevolutionary process [23].

## Methods

An evolutionary prisoner's dilemma game with the temptation to defect (the highest payoff received by a defector if playing against a cooperator), reward for mutual cooperation , the punishment for mutual defection , and the sucker's payoff (the lowest payoff received by a cooperator if playing against a defector) is used as the basis for our simulations. Without loss of generality the payoffs can be rescaled as , , and , where is the cost-to-benefit ratio [5]. For positive we have , thus strictly satisfying the prisoner's dilemma payoff ranking.

As the interaction network, we use either a regular square lattice, the random regular graph (RRG) constructed as described in [65], or the small-world (SW) topology with an average degree of four generated via the Watts-Strogatz algorithm [66]. Each vertex is initially designated as hosting either players of type or with the probability and , respectively. This division of players is performed uniformly at random irrespective of their initial strategies and remains unchanged during the simulations. According to established procedures, each player is initially also designated either as a cooperator () or defector () with equal probability. The game is iterated forward in accordance with the sequential simulation procedure comprising the following elementary steps. First, player acquires its payoff by playing the game with all its neighbors. Next, we evaluate in the same way the payoffs of all the neighbors of player and subsequently select one neighbor via the probability(1)

where the sum runs over all the neighbors of player . Importantly, is the so-called selection or aspiration parameter that depends on the type of player according to(2)

Evidently, if the aspiration parameter then irrespective of (density of type players) the most frequently adopted situation is recovered where player is chosen uniformly at random from all the neighbors of player . For and , however, Eqs. (1) and (2) introduce a preference in all players of type (but not in players of type ) to copy the strategy of those neighbors who have a high fitness, or equivalently, a high payoff . Lastly then, after the neighbor that is aspired to by player is chosen, player adopts the strategy from the selected player with the probability(3)

where denotes the amplitude of noise or its inverse () the so-called intensity of selection [38]. Irrespective of the values of and one full iteration step involves all players having a chance to adopt a strategy from one of their neighbors once.

An alternative model, allowing for individual values to be subject to evolution as well, entails omitting the division of the population on two types of players and assigning to every individual an initial value that is drown randomly from a Gaussian distribution having mean and standard deviation , as was done recently in [57], for example. Subsequently, if player adopts the strategy from player following the identical procedure as described above for the original model, then the value of changes to that of as well. The key question that we aim to answer with this model is whether a specific aspiration level is indeed optimal for an individual to prosper, and if yes, does the selection pressure favor it spontaneously. Essentially, we are interested in the distribution of values after the stationary fraction of strategies in the population is reached. A link with the original model can be established by considering in this case to equal one and .

Results of computer simulations were obtained on populations comprising to individuals, whereby the fraction of cooperators was determined within full iteration steps after sufficiently long transients were discarded. Moreover, since the heterogeneous preferential selection of neighbors may introduce additional disturbances, final results were averaged over up to independent runs for each set of parameter values in order to assure suitable accuracy.

## Acknowledgments

We have benefited substantially from the insightful comments of the Editor James Arthur Robert Marshall and an anonymous referee, which were instrumental for elevating the quality of this work. Helpful discussions with Professor Lianzhong Zhang are gratefully acknowledged as well.

## Author Contributions

Conceived and designed the experiments: MP ZW. Performed the experiments: MP ZW. Analyzed the data: MP ZW. Contributed reagents/materials/analysis tools: MP ZW. Wrote the paper: MP ZW.

## References

- 1.
Hofbauer J, Sigmund K (1998) Evolutionary Games and Population Dynamics. Cambridge, UK: Cambridge University Press.
- 2.
Nowak MA (2006) Evolutionary Dynamics. Cambridge, MA: Harvard University Press.
- 3.
Maynard Smith J (1982) Evolution and the Theory of Games. Cambridge, UK: Cambridge University Press.
- 4. Roca CP, Cuesta JA, Sánchez A (2009) Evolutionary game theory: Temporal and spatial effects beyond replicator dynamics. Phys Life Rev 6: 208–249.
- 5. Hauert C, Szabó G (2005) Game theory and physics. Am J Phys 73: 405–414.
- 6. Szabó G, Fáth G (2007) Evolutionary games on graphs. Phys Rep 446: 97–216.
- 7. Nowak M, Sigmund K (1989) Game-dynamical aspects of the prisoner's dilemma. Appl Math Comp 30: 191–213.
- 8. Moyano LG, Sánchez A (2009) Evolving learning rules and emergence of cooperation in spatial prisoner's dilemma. J Theor Biol 259: 84–95.
- 9. Doebeli M, Hauert C (2005) Models of cooperation based on prisoner's dilemma and snowdrift game. Ecol Lett 8: 748–766.
- 10. Szabó G, Szolnoki A, Sznaider GA (2007) Segregation process and phase transition in cyclic predator-prey models with even number of species. Phys Rev E 76: 051921.
- 11. Szolnoki A, Perc M, Szabó G, Stark HU (2009) Impact of aging on the evolution of cooperation in the spatial prisoner's dilemma game. Phys Rev E 80: 021901.
- 12. Milinski M, Semmann D, Krambeck HJ (2002) Reputation helps to solve the ‘tragedy of the commons’. Nature 415: 424–426.
- 13. Fehr E (2004) Don't lose your reputation. Nature 432: 449–450.
- 14. Van Segbroeck S, Santos FC, Lenaerts T, Pacheco JM (2009) Reacting differently to adverse ties promotes cooperation in social networks. Phys Rev Lett 102: 058105.
- 15. Maynard Smith J, Parker GA (1976) The logic of asymmetric contests. Anim Behav 24: 159–175.
- 16. Selten R (1980) A note on evolutionary stable strategies in asymmetric animal conflicts. J Theor Biol 84: 93–101.
- 17. Hammerstein P (1981) The role of asymmetries in animal contests. Anim Behav 29: 193–205.
- 18. Marshall JAR (2009) The donation game with roles played between relatives. J Theor Biol 260: 386–391.
- 19. Milinski M (1987) TIT FOR TAT in sticklebacks and the evolution of cooperation. Nature 325: 533–535.
- 20. Hamilton WD (1964) Genetical evolution of social behavior I & II. J Theor Biol 7: 1–52.
- 21. Nowak MA (2006) Five rules for the evolution of cooperation. Science 314: 1560–1563.
- 22. Schuster S, Kreft JU, Schroeter A, Pfeiffer T (2008) Use of game-theoretical methods in biochemistry and biophysics. J Biol Phys 34: 1–17.
- 23. Perc M, Szolnoki A (2010) Coevolutionary games – a mini review. BioSystems 99: 109–125.
- 24. Lehmann L, Keller L (2006) The evolution of cooperation and altruism - a general framework and a classification of models. J Evol Biol 19: 1365–1376.
- 25.
Marshall JAR (2010) Ultimate causes and the evolution of altruism. Behav Ecol Sociobiol: in press.
- 26. Nowak MA, May RM (1992) Evolutionary games and spatial chaos. Nature 359: 826–829.
- 27. Albert R, Barabási AL (2002) Statistical mechanics of complex networks. Rev Mod Phys 74: 47–97.
- 28. Newman MEJ (2003) The structure and function of complex networks. SIAM Review 45: 167–256.
- 29. Boccaletti S, Latora V, Moreno Y, Chavez M, Hwang D (2006) Complex networks: Structure and dynamics. Phys Rep 424: 175–308.
- 30. Ohtsuki H, Hauert C, Lieberman E, Nowak MA (2006) A simple rule for the evolution of cooperation on graphs and social networks. Nature 441: 502–505.
- 31. Lehmann L, Keller L, Sumpter D (2007) The evolution of helping and harming on graphs: the return of the inclusive fitness effect. J Evol Biol 20: 2284–2295.
- 32. Grafen A (2007) An inclusive fitness analysis of altruism on a cyclical network. J Evol Biol 20: 2278–2283.
- 33. Nowak MA, May RM (1993) The spatial dilemmas of evolution. Int J Bifurcat Chaos 3: 35–78.
- 34. Nowak MA, Bonhoeffer S, May RM (1994) More spatial games. Int J Bifurcat Chaos 4: 33–56.
- 35. Ohtsuki H, Nowak MA (2006) Evolutionary games on cycles. Proc R Soc Lond B 273: 2249–2256.
- 36. Schlag KH (1998) Why imitate, and if so, how? a bounded rational approach to multi-armed bandits. J Econ Theory 78: 130–156.
- 37. Wang S, Szalay MS, Zhang C, Csermely P (2008) Learning and innovative elements of strategy adoption rules expand cooperative network topologies. PLoS ONE 3: e1917.
- 38. Szabó G, Tőke C (1998) Evolutionary prisoner's dilemma game on a square lattice. Phys Rev E 58: 69–73.
- 39. Abramson G, Kuperman M (2001) Social games in a social network. Phys Rev E 63: 030901(R).
- 40. Ren J, Wang WX, Qi F (2007) Randomness enhances cooperation: coherence resonance in evolutionary game. Phys Rev E 75: 045101(R).
- 41. Chen XJ, Wang L (2008) Promotion of cooperation induced by appropriate payoff aspirations in a small-world networked game. Phys Rev E 77: 017103.
- 42. Vukov J, Szabó G, Szolnoki A (2006) Cooperation in the noisy case: Prisoner's dilemma game on two types of regular random graphs. Phys Rev E 73: 067103.
- 43. Santos FC, Pacheco JM (2005) Scale-free networks provide a unifying framework for the emergence of cooperation. Phys Rev Lett 95: 098104.
- 44. Santos FC, Rodrigues JF, Pacheco JM (2006) Graph topology plays a determinant role in the evolution of cooperation. Proc R Soc Lond B 273: 51–55.
- 45. Poncela J, Gómez-Gardeñes J, Floría LM, Moreno Y (2007) Robustness of cooperation in the evolutionary prisoner's dilemma on complex systems. New J Phys 9: 184.
- 46. Assenza S, Gómez-Gardeñes J, Latora V (2008) Enhancement of cooperation in highly clustered scale-free networks. Phys Rev E 78: 017101.
- 47. Szolnoki A, Perc M, Danku Z (2008) Towards effective payoffs in the prisoner's dilemma game on scale-free networks. Physica A 387: 2075–2082.
- 48. Perc M (2009) Evolution of cooperation on scale-free networks subject to error and attack. New J Phys 11: 033027.
- 49. Zimmermann MG, Eguíluz V, Miguel MS (2004) Coevolution of dynamical states and interactions in dynamic networks. Phys Rev E 69: 065102(R).
- 50. Pacheco JM, Traulsen A, Nowak MA (2006) Coevolution of strategy and structure in complex networks with dynamical linking. Phys Rev Lett 97: 258103.
- 51. Szolnoki A, Perc M, Danku Z (2008) Making new connections towards cooperation in the prisoner's dilemma game. EPL 84: 50007.
- 52. Poncela J, Gómez-Gardeñes J, Floría LM, Sánchez A, Moreno Y (2008) Complex cooperative networks from evolutionary preferential attachment. PLoS ONE 3: e2449.
- 53. Szolnoki A, Perc M (2009) Resolving social dilemmas on evolving random networks. EPL 86: 30007.
- 54. Poncela J, Gómez-Gardeñes J, Traulsen A, Moreno Y (2009) Evolutionary game dynamics in a growing structured population. New J Phys 11: 083031.
- 55. Szolnoki A, Perc M (2009) Emergence of multilevel selection in the prisoner's dilemma game on coevolving random networks. New J Phys 11: 093033.
- 56. Wu B, Zhou D, Fu F, Luo Q, Wang L, et al. (2010) Evolution of cooperation on stochastic dynamical networks. PloS ONE 5: 1–7.
- 57. McNamara JM, Barta Z, Fromhage L, Houston AI (2008) The coevolution of choosiness and cooperation. Nature 451: 189–192.
- 58. Hauert C (2002) Effects of space in 2×2 games. Int J Bifurc Chaos 12: 1531–1548.
- 59. Perc M (2006) Coherence resonance in spatial prisoner's dilemma game. New J Phys 8: 22.
- 60. Perc M, Marhl M (2006) Evolutionary and dynamical coherence resonance in the pair approximated prisoner's dilemma game. New J Phys 8: 142.
- 61. Tanimoto J (2007) Promotion of cooperation by payoff noise in a 2×2 game. Phys Rev E 76: 041130.
- 62. Traulsen A, Nowak MA, Pacheco JM (2007) Stochastic payoff evaluation increases the temperature of selection. J Theor Biol 244: 349–356.
- 63. Szabó G, Vukov J, Szolnoki A (2005) Phase diagrams for an evolutionary prisoner's dilemma game on two-dimensional lattices. Phys Rev E 72: 047107.
- 64. Szolnoki A, Perc M, Szabó G (2009) Topology-independent impact of noise on cooperation in spatial public goods games. Phys Rev E 80: 056109.
- 65. Szabó G, Szolnoki A, Izsák R (2004) Rock-scissors-paper game on regular small-world networks. J Phys A: Math Gen 37: 2599–2609.
- 66. Watts DJ, Strogatz SH (1998) Collective dynamics of ‘small world’ networks. Nature 393: 440–442.