Introduction

Human society is built upon cooperation among individuals. However, our society is full of social dilemmas, where cooperative actions, which are costly to individuals, appear to be superseded by non-cooperative, selfish actions that exploit cooperative others [1–3]. There are several mechanisms that explain cooperative behavior in social dilemma situations [4–6]. The evolutionary game theory has provided firm evidence that static networks enhance cooperation as compared to well-mixed populations under generous conditions, with the effect being called spatial reciprocity (in the case of finite-dimensional networks) and network reciprocity (in the case of general networks) [4, 7–10]. This finding is in alignment with broadly made observations that humans as well as animals interact on contact networks where a node is an individual [11–13].

However, a series of laboratory experiments using human participants involved in the prisoner’s dilemma game (PDG) has produced results that are not necessarily consistent with spatial and network reciprocity. In fact, the structure of networks (e.g., scale-free, random, and lattice) did not correlate with the propensity of human cooperation in the PDG [14–20]. In contrast, Rand et al. have shown that humans present network reciprocity if the benefit-to-cost ratio, a main parameter of the PDG, is larger than the degree of nodes in the network (i.e., number of neighbors per player) [21], which is consistent with the prediction of evolutionary game theory [8]. Note that the earlier experimental studies used smaller benefit-to-cost ratio values [14–20].

The theoretical results in Ref. [8] are derived from the probability of fixation of cooperation, i.e., the probability that a unanimity of cooperation is reached before that of defection under weak selection (i.e., the difference between the strength of cooperator and that of defector is assumed to be small). While theoretically elegant, unanimity under weak selection may be different from the population dynamics taking place in laboratory experiments with human participants, such as those in Ref. [21]. (However, see [22] for conditions for cooperation that are derived in the case of infinite populations and therefore no fixation, assuming replicator dynamics.) In laboratory experiments, the unanimity of cooperators is hard to be reached. The aim of the present study is to look for an alternative mechanism that explains behavioral results under the PDG on networks.

We hypothesize that a type of reinforcement learning implemented as a strategy of players produces game dynamics that are consistent with the aforementioned experimental results regarding network reciprocity. In particular, aspiration-based reinforcement learning [23–27], with which players modulate their behavior based on the magnitude of the earned reward relative to a threshold, has been successful in explaining conditional cooperation behavior and its variants called moody conditional cooperation [28, 29]. Furthermore, aspiration-based reinforcement learning, not evolutionary game theory, yielded the absence of network reciprocity in numerical simulations [30]. This result is consistent with those showing that outcomes of aspiration-based learning and those of evolutionary dynamics are intrinsically different [31, 32]. In the present paper, we vary the benefit-to-cost ratio and the node’s degree, two key parameters in the discussion of network reciprocity in the literature, to show that aspiration-based reinforcement learning gives rise to network reciprocity under the conditions consistent with the previous experimental study [21]. In this way, we significantly extend the previous numerical results [30].

Model

Static- and shuffled-network treatments

We compare the propensity of cooperation between static and dynamically shuffled networks, mimicking the situation of a laboratory experiment [21]. In both static- and shuffled- network treatments, the network in each round is a ring network in which each node has k neighbors, where k is an even number (Fig 1). Each player is adjacent to k/2 players on each side on the ring. In the static-network treatment, the position of the players is fixed throughout all the rounds. In the shuffled-network treatment, while the network structure is fixed over rounds, we randomize the position of all the players after each round.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Ring networks composed of N = 20 players.

The player represented by a black circle is adjacent to k players represented by gray circles. (a) k = 2. (b) k = 6.

https://doi.org/10.1371/journal.pone.0189220.g001

Results

We consider two values of b/c, i.e., b/c = 2 and 6 by setting (b, c) = (2, 1) and (b, c) = (6, 1), respectively.

Numerically calculated fractions of cooperative players are compared between the two treatments in Fig 2. When the node’s degree, k, is small (i.e., k = 2) and b/c is large (i.e., b/c = 6), cooperation is more frequent in the static-network treatment than the shuffled-network treatment. This result is consistent with the previous experimental results [21]. When b/c = 2, this effect is not observed, which is also consistent with the experimental results [14–21].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Fraction of cooperative players in each round, averaged over 10³ simulations.

We set k = 2, N = 100, t_max = 50, β = 0.2, A = 1.0, and ϵ = 0.05. (a) b/c = 6. (b) b/c = 2.

https://doi.org/10.1371/journal.pone.0189220.g002

To examine the robustness of the results shown in Fig 2, we carried out simulations for a region of the A–ϵ parameter space and four values of k. We did not vary β (= 0.2) because β did not considerably alter the behavior of the players unless it took extreme values [29]. With b/c = 2, the fraction of cooperative players averaged over the first 25 rounds is shown in Fig 3(a) and 3(b) for the static and shuffled networks, respectively. The difference between the two types of networks, shown in Fig 3(c), is small in the entire parameter region, in particular for large k, suggesting a marginal effect of network reciprocity. In contrast, when b/c = 6, the fraction of cooperators is larger in the static-network than the shuffled-network treatment in a relatively large region of the A–ϵ parameter space [Fig 3(e), 3(f) and 3(g)]. As k increases, the difference between the two treatments decreases. In summary, a static as opposed to shuffled network promotes cooperation only when b/c is large and k is small. These results are consistent with the experimental findings [21].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Fraction of cooperative players under the static-network treatment [(a) and (e)] and the shuffled-network treatment [(b) and (f)].

The difference between the fraction of cooperation in the static and shuffled networks is shown in (c) and (g). The assortment for the static networks is shown in (d) and (h). We set N = 100 and β = 0.2. (a)–(d) b/c = 2. (e)–(h) b/c = 6. To calculate the fraction of cooperators and the assortment, we take averages over the first 25 rounds and 10³ simulations.

https://doi.org/10.1371/journal.pone.0189220.g003

Network reciprocity is attributed to assortative connectivity between cooperative players [7–10]. In other words, cooperation can thrive if a cooperator tends to find other cooperators at the neighboring nodes. To measure this effect, we defined the assortment by P(C|C; t) − P(C|D; t), where P(C|C; t) is the probability that a neighbor of a cooperative player is cooperative in round t, and P(C|D; t) is the probability that a neighbor of a defective player is cooperative in round t [21, 36]. For various values of A and ϵ, the assortment values in the static-network treatment averaged over the first 25 rounds are shown in Fig 3(d) and 3(h) for b/c = 2 and b/c = 6, respectively. The figures indicate that the assortment tends to be positive when cooperation is more abundant in the static-network than shuffled-network treatment regardless of the value of b/c, suggesting that cooperative players are clustered in these parameter regions. In the shuffled treatment, we confirmed that the assortment was ≈ 0 in the entire parameter region.

Conclusions

We have numerically shown that an aspiration-based reinforcement learning model, the BM model, produces network reciprocity if and only if the benefit-to-cost ratio in the donation game is large relative to the node’s degree. The results are consistent with the previous experimental findings [14–21]. In addition to network reciprocity, the BM model also accounts for the conditional cooperation, which is hard to explain by evolutionary game theory [28, 29, 37]. Aspiration-based reinforcement learning may be able to describe cooperative behavior of humans and animals in broader contexts. Finally, we remark that, although network reciprocity is not observed in the shuffled-network treatment, in both theory and experiments, dynamic linking treatments that allow players to strategically sever and create links promote cooperation in laboratory experiments [17, 38–41]. Evolutionary game theory predicts cooperation under dynamic linking [42–48]. Reinforcement learning may also account for enhanced cooperation under dynamic linking.

Acknowledgments

We acknowledge Hisashi Ohtsuki for valuable comments on the manuscript. TE acknowledges the support provided through PRESTO, JST (No. JPMJPR16D2) and Kawarabayashi Large Graph Project, ERATO, JST (No. JPMJER1201). NM acknowledges the support provided through, CREST, JST (No. JPMJCR1304) and Kawarabayashi Large Graph Project, ERATO, JST (No. JPMJER1201).