Genotype dynamics in sexual diploid population

We consider a very large diploid population consisting of individuals. Here by “very large”, we mean that during sexual reproduction all possible genotypes occur in the population. Females and males differ only in their sex, the sex ratio is 1:1, and there is no sexual selection. We track the evolution of the genotype frequencies over time. Let m denote the number of distinct genotypes; then, the population state at any given time is represented by a point in the -dimensional simplex

where x_i denotes the frequency of the i-th phenotype. Our dynamics is based on the following assumptions.

Assumption A) The mating system is fixed. Let the probability of mating of genotypes and be , e.g., under panmixia.

Assumption B) The genotypes of the parents determine the genotypes of their offspring.

The values , representing the probability that a mating between genotypes and produces an offspring of genotype , are organized into a mating table (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Genotype survival table based on the mating table for a general matrix game.

https://doi.org/10.1371/journal.pone.0331044.t001

Remark 1. This framework allows us to consider arbitrary genetic architectures. For instance, the probabilities can be derived for systems with multiple autosomal loci located on the same or different chromosomes, with or without recombination. Furthermore, if each allele can mutate to any other allele at the same locus, then each entry of the mating table is strictly positive.

Since natural selection acts at the level of the phenotype, it is essential to specify the genotype-phenotype map.

Assumption C) The phenotype is genetically determined. Under this assumption, we can consider various selection regimes and calculate the survival rate of each sibling based on game-theoretical conflicts (see Remark 2). This condition ensures that the choice of a strategy is not an individual decision but is entirely determined by genotype.

Assumption D) The survival rate of each offspring depends on the genotypes of the parents. The parents’ genotypes determine the genotypes of their offspring, which in turn determine their phenotypes and hence their survival. Assumptions C and D allow us to model survival games between siblings, where the outcome is determined by the parental genotypes. Let denote the number of surviving offspring of genotype resulting from a mating between genotypes and .

Remark 2. The parameters and are general in the sense that they can accommodate a wide range of selection situations. For instance, the number of offspring may depend on the genotypes of the parents [20] as well as on the genotype distribution x. Similarly, the survival probability of an offspring with genotype born to parents with genotypes and may also depend on x.

Let denote the number of offspring produced by a mating pair with genotypes and , and let be the survival probability of an offspring with genotype from such a mating pair, in a population with genotype distribution . Then the number of surviving offspring of genotype is given by

Nevertheless, after presenting the general formulation of the genotype dynamics, for the remainder of the paper we assume and are independent of x and hence we write them simply as and .

To follow the evolution of genotype frequencies over time, we need to know the relative frequency of mating pairs , denoted by , and, for each such pair, the number of surviving offspring with genotype , i.e., , for any .

The total number of offspring of genotype produced by the entire parental population in state is given by:

If , the average production rate of genotype is

and the average production rate of the entire diploid population is

To avoid degenerate cases, we assume for all , that is, for every , there exists at least one such that .

Genotype dynamics: Following the reasoning that precedes the introduction of the replicator dynamics in Chapter 7.1 of [41], we consider a sufficiently large population in which generations blend continuously into each other. In this setting, the frequency of a given genotype can be treated as a differentiable function of time, and its change can be described by a system of differential equations. Analogous to the replicator dynamics, the rate of change is given by the difference between the production rate of the genotype and the average production rate of the population:

(1)

This dynamical system is referred to as the genotype dynamics (for a detailed derivation, see Supplementary Information C in [3]). The foundation of dynamics (1) lies in the basic tenet of Darwinism: the relative frequency of a genotype increases if its production rate exceeds the average production rate of the population; in other words, when it has a positive relative advantage [42].

A key distinction between the replicator dynamics and the genotype dynamics is that the former describes asexual populations, where a parent of type produces only offspring of the same type, whereas dynamics (1) applies to sexual populations, where parents of type may produce offspring with genotypes different from (see Tables 1 and 4–6).

However, if the population is large enough such that all genotypes are present (i.e., for every ), then dynamics (1) can be rewritten in the following replicator-dynamics-like form:

Evolutionary stability of genotype distributions.

We now ask under what conditions a genotype distribution qualifies as an evolutionarily stable genotype distribution (ESGD) in an arbitrarily large sexual population. If every individual in the resident population has the same homozygous genotype, then, following the classical definition by Maynard Smith and Price [43], this genotype is said to be evolutionarily stable if no mutant genotype can invade the population through natural selection. Mathematically, this means that the (relative) frequency of the resident homozygotes increases from generation to generation, provided mutations are rare. The formal definition of ESGD for such homozygous populations was introduced in [3].

We now extend this concept to mixed genotype distributions. However, the original notion by Maynard Smith and Price [43] is not directly applicable when the genotype population exhibits polymorphism. Adopting a dynamic stability perspective, we define a polymorphic state as an ESGD if the system returns to this state after a small perturbation, such as the appearance of a rare mutant [44]. From a static, evolutionary viewpoint, a genotype distribution is considered to be an ESGD if the average production rate of the entire genotype population (as on the right-hand side of inequality (2) below) is strictly less than that of the genotype subpopulation in the ESGD state (on the left-hand side of inequality (2)). In other words, the ESGD subpopulation enjoys a relative advantage over the total population.

Definition 1. A genotype distribution is evolutionary stable if

(2)

provided that is sufficiently close to (but not equal to) .

Clearly, inequality (2) can be rewritten as

(3)

and it follows that is an ESGD if and only if

(4)

whenever is sufficiently close to (but not equal to) . Note that represents the average production rate of genotype and thus plays a role analogous to that of fitness in asexual populations.

In asexual populations, coexisting types at an evolutionarily stable state exhibit equal fitness. A similar condition holds in our model. In section SI A of S1 File, we show that the following equilibrium condition (interpretable as a Nash equilibrium in game theoretical terms) holds.

Equilibrium condition. If is an ESGD and is an arbitrary state in then

(5)

Furthermore, a key property of interior Nash equilibria in asexual matrix games also holds in this model. Namely, at an interior ESGD, the average production rates of all genotypes must be equal. Formally, if is an ESGD, then for every we have

(6)

This equality implies that an ESGD is a rest point of the genotype dynamics (1).

Following the reasoning in Chapter 7.2 of [41], we can establish the ensuing result (see Theorem SI.1 in section SI B of S1 File).

Theorem 1. Assume that is an ESGD. Then is a locally asymptotically stable rest point of the genotype dynamics (1).

We also need to define ESGD for a vertex of the simplex, particularly when all individuals share the same homozygous genotype. This definition was introduced in Supplementary Information B in [3], and we recall here. We say that is an ESGD if

for every sufficiently close to (but not equal to) . (We note that equation (4) remains applicable if we agree that, in cases where , the expression is interpreted as 0 while is interpreted as 1.) It has been shown that if the vertex is an ESGD, then it is a locally asymptotically stable equilibrium point of the genotype dynamics (1) (see Supplementary Information C in [3]).

Application: Filial selection with a matrix game between full siblings

We consider a diploid species with internal fertilization. The following assumptions are made: mutations are sufficiently rare; the species practices monogamy, resulting in a relatedness of 1/2 between full siblings; and mating is panmictic, meaning that the probability of mating between genotypes and is . The population size N is assumed to be very large, with mating pairs are formed randomly, ensuring random mating and preventing inbreeding.

As stated in the introduction, our goal is to generalize Haldane’s concept of “familial selection” [1]. For this purpose, we adopt the simplifying assumption that each mating pair produces a fixed number n of offspring, that is, for all . Furthermore, an offspring’s probability of survival depends solely on the behaviour of its siblings and is independent of the overall genotype distribution in the population. These simplifications ensure that the frequency-dependent interaction between mating pairs and the population-dependent juvenile survival rate cannot obscure the effect of the survival game between full siblings. As a result of differing survival probabilities among genotypes, the diploid parental population generally deviates from Hardy-Weinberg equilibrium.

For simplicity, we focus on a single autosomal locus with two alleles, denoted by [a] and [A], resulting in three possible genotypes, , , and with corresponding frequencies , and . Thus the population state is represented by the vector , which belongs to the simplex

Genotype-phenotype mapping. We assume that each genotype uniquely determines a phenotype. Let , and denote the phenotypes associated with genotypes , and , respectively. We consider both dominant-recessive and intermediate Mendelian inheritance systems.

We assume that all interactions relevant to the model occur between full siblings and are described by the same two-player one-shot matrix game, where the entries of the payoff matrix represent survival probabilities. Each sibling’s survival rate is then the average payoff from interactions with its siblings. Accordingly, we consider a two-dimensional matrix game within each family, with a common payoff matrix ,

where for every .

The phenotypes are pure or mixed strategies from the simplex

Under recessive-dominant inheritance, each individual adopts a pure strategy, either or . Under intermediate inheritance, the homozygotes follows a pure strategy, while each heterozygote employs the mixed strategy. Note that the survival rate of an offspring depends solely on its own phenotype and the phenotypes of its full siblings, which are determined by the parents’ genotypes. Consequently, the family unit acts as a genetically well-defined interacting group, thereby introducing a “group selection ingredient” to our model.

Arbitrary inheritance.

First, we give an overview of payoffs in the general case for two alleles at a single autosomal locus (see Table 1). Genotype is associated with phenotype , genotype with phenotype and genotype with phenotype . That is, all heterozygotes exhibit the same phenotype, irrespective of the parental origin of each allele.

Above we defined a selection regime in Haldane’s familial selection framework, where the interaction between full siblings is modelled by a two-player matrix game. In this setting, the expected number of surviving offspring of genotype from a family, denoted by , is summarized in Table 1.

Within each family, the game theoretical interaction is assumed to be well-mixed. This means that the genotypes of the offspring are independent of one another, and siblings interact in randomly formed pairs. Thus, in any interacting pair, the genotypes of the two individuals are independent. For instance, in a family, a zygote has genotype , and with probabilities 1/4, 1/2 and 1/4, respectively. As a result, a juvenile of genotype interacts with phenotypes , and with probabilities 1/4, 1/2 and 1/4. Consequently, the survival rate of a focal juvenile is given by

Since the expected proportion of offspring in a family is 1/4, the average (expected) number of surviving juveniles in such families is

(For further details, see section SI C of S1 File.)

Based on Table 1, the total number of individuals of genotypes , and in the next generation can be calculated as follows:

Using these genotype-specific production rates, we obtain the explicit form of the genotype dynamics (1) under the selection regime considered.

We focus on the following questions in the general case when the phenotype of homozygotes is (1,0) while the phenotype of homozygotes is (0,1).

Question 1. What is a necessary condition for the stability of the homozygote sates?

(Here we present only the first-order condition. The detailed mathematical analysis is provided in section SI C of S1 File.) Let allele [a] be recessive and allele [A] dominant.

The state (1,0,0), in which every individual is a recessive homozygote , is an ESGD if

(8)

The state (0,0,1), in which every individual is a dominant homozygote , is an ESGD if

(9)

Question 2. What is a necessary condition for the coexistence of all the genotypes?

Clearly, the homozygote states are the only possible rest points of the genotype dynamics on the boundary of the simplex , and is positively invariant under the genotype dynamics (see Supplementary Information C in [3]). Therefore, by the two-dimensional Poincaré-Bendixson theorem, if both homozygote states are repellors, then (in biological terms) the system exhibits either stable or cyclic coexistence of the two phenotypes.

The state (1,0,0), where all individuals are recessive homozygotes _, is a repellor (see Theorem SI.5 in section SI C of S1 File), if

(10)

The state (0,0,1), where all individuals are dominant homozygotes , is a repellor (see Theorem SI.6 in section SI C of S1 File), if

(11)

We note that if inequalities (10) and (11) are both satisfied, then neither homozygote state is an ESGD. Based on our earlier results (see Supplementary Information C in [3]), this implies that both are repellor. Thus, the combination of these two inequalities provides a sufficient condition for the coexistence of all three genotypes (for additional details, see section SI D in S1 File). It is worth noting that conditions (8)-(11) apply to arbitrary payoff matrices, provided that allele [a] is recessive.

Remark 3. For the reader’s convenience, we provide a helpful guide for analysing an arbitrary payoff matrix. Condition (2) of ESGD in Definition 1 is non-linear in the genotype distribution (see Supplementary Information C in [3]). When the linear term of the function (see equation (4)) is non-zero, its sign determines the evolutionary stability of the corresponding homozygote: a positive sign implies local stability, while a negative one implies instability. If the linear term vanishes, the second-order terms must be examined, which requires more detailed analysis.

Although one might consider this latter case to be exceptional and therefore less important, this is not so. As already observed in [3], if allele [a] is dominant, the linear term vanishes near the homozygote vertex (1,0,0). This occurs because, in that neighbourhood the most common family type is , followed by , with all other families being rare. The genotype production rates in this neighbourhood therefore depend primarily on interactions within these two family types. However, due to the dominance of allele [a], all individuals in both families express phenotype (since ), causing the first-order term to cancel. A similar argument applies when allele [A] is dominant: the linear term then vanishes near the homozygote vertex (0,0,1).

This phenomenon is reminiscent of the classical definition of ESS for matrix games, where the second-order condition becomes necessary to determine the evolutionary stability of a mixed Nash equilibrium (see condition (6.12) on p 63 in [41]).

Since second-order conditions are typically quadratic in the entries of the payoff matrix, it is useful to identify simple linear inequalities that imply (or at least suggest) that a vertex is a repellor (see Tables 2 and 3). This practical need was a key motivation behind our introduction of genotype dynamics: the use of differential equations allows a more effective analysis of the stability of the vertices.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Conditions when states (1,0,0) and (0,0,1), respectively, are ESGD-s.

https://doi.org/10.1371/journal.pone.0331044.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Conditions when states (1,0,0) and (0,0,1), respectively, are repellors.

https://doi.org/10.1371/journal.pone.0331044.t003

The results presented in Tables 2 and 3 are derived in SI C of S1 File and build on our earlier work in [3], where we provided first- and second-order conditions (in the form of inequalities) the evolutionary stability of a genotype. If the inequality in the first-order condition is reversed, the corresponding vertex becomes a repellor. However, this logic does not extend to second-order conditions: reversing such an inequality provides only a necessary, but not sufficient, condition for instability. Since the inequalities in Table 3 stem from second-order considerations, they merely suggest that the vertex may be a repellor, but further analysis is required to confirm this.

In what follows, we illustrate how the joint effect of the phenotypic payoff matrix and the genotype-phenotype mapping determines the solution of the genotype dynamics. It is important to note that our aim is not to provide a rigorous topological classification for the genotype dynamics, but rather to present numerical examples that offer insights into the underlying mechanisms.

To this end, we assume that the interaction between siblings is modelled by a prisoner’s dilemma game, in which the two pure strategies are cooperation and defection. The corresponding payoff matrix is

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

https://doi.org/10.1371/journal.pone.0331044.t008

where

From the viewpoint of group selection, we distinguish two subclasses of the prisoner’s dilemma depending on how siblings can maximize the “welfare of the family” (i.e., the sum of their payoffs).

Coordinated case: If , then both siblings should cooperate to maximize their total survival rate. Note that although this condition is required in the iterated version of the game, our focus here is solely on the one-shot interaction.

Anti-coordinated case: If , then the highest collective payoff is achieved when one sibling cooperates and the other defects.

These terms are introduced in analogy with the coordination game (in which each player’s individual payoff is maximized when both use the same pure strategy) and the anti-coordination game (in which individual payoffs are maximized when the players use different pure strategies), as the underlying ideas show a clear conceptual parallel (cf., [45]). However, the cases are not identical: in our setting, the joint payoff of the players is what matters, whereas in coordination and anti-coordination games, the focus is on maximizing individual payoffs.

A key question of the article is whether the evolutionary process indeed promotes cooperation in the coordinated case, or a division of roles (cooperator and defector) in the anti-coordinated case. As shown in Figs 10 and 11, which summarize the main subcases, the long-term outcomes do not necessarily align with the intuitive expectations suggested by these labels. For example, in the coordinated case, it is possible that only the defector homozygote state is stable. Unfortunately, we are unable to provide an intuitive explanation for this outcome, though it does emerge in analytical investigations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Examples of prisoner’s dilemma games in which cooperation is recessive and the total payoff is maximized by the cooperator-cooperator strategy pair (coordinated case).

Black dots indicate asymptotically stable rest points, while empty dots represent unstable equilibria. The bottom-left vertex of the triangle corresponds to the state (1,0,0), in which all individuals are cooperator homozygotes with genotype ([a],[a]). The bottom-right vertex represents the state (0,1,0), where all individuals are heterozygotes with genotype ([a],[A]). The top vertex corresponds to the state (0,0,1), where all individuals are defector homozygotes with genotype ([A],[A]).

https://doi.org/10.1371/journal.pone.0331044.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Examples of prisoner’s dilemma games in which cooperation is recessive and the total payoff is maximized by the cooperator-defector strategy pair (anti-coordinated case).

See the caption at Fig 1.

https://doi.org/10.1371/journal.pone.0331044.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Definition 1 is not empty.

Left panel: The function around . The black dot marks the graph point corresponding to . It is clearly visible that in variable attains a minimum at this point. Right panel: Phase portrait of dynamics (1). The system is governed by the phenotypic payoff matrix from Fig 2. The interior rest point is globally asymptotically stable; the vertices and corresponding to monomorphic population of homozygotes are unstable rest points. Black dots and circles indicate asymptotically stable and unstable rest points, respectively. Note that the simplex is only positively invariant under the genotype dynamics (1), in general. For instance, while the homozygote vertices (1,0,0) and (0,0,1) are trivial rest points of the genotype dynamics (1), the heterozygote vertex (0,1,0) is not. It becomes a rest point of the genotype dynamics (1) if and only if both homozygotes are lethal.

https://doi.org/10.1371/journal.pone.0331044.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Examples of prisoner’s dilemma games in which cooperation is dominant and the total payoff is maximized by the cooperator-cooperator strategy pair (coordinated case).

Black dots indicate asymptotically stable rest points, while empty dots represent unstable equilibria. The bottom-left vertex of the triangle corresponds to the state (1,0,0), in which all individuals are cooperator homozygotes with genotype ([a],[a]). The bottom-right vertex represents the state (0,1,0), where all individuals are heterozygotes with genotype ([a],[A]). The top vertex corresponds to the state (0,0,1), where all individuals are defector homozygotes with genotype ([A],[A]).

https://doi.org/10.1371/journal.pone.0331044.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Examples of prisoner’s dilemma games in which cooperation is dominant and the total payoff is maximized by the cooperator-defector strategy pair (anti-coordinated case).

See the caption at Fig 4.

https://doi.org/10.1371/journal.pone.0331044.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Examples of prisoner’s dilemma games with intermediate inheritance and in which the total payoff is maximized by the cooperator-cooperator strategy pair (coordinated case).

Black dots indicate asymptotically stable rest points while empty dots represent unstable equilibrium points. The bottom-left vertex of the triangle corresponds to the state (1,0,0), in which all individuals are ([a],[a]) cooperator homozygotes. The bottom-right vertex represents the state (0,1,0), where all individuals are heterozygotes with genotype ([a],[A]). The top vertex corresponds to the state (0,0,1), where all individuals are defector homozygotes with genotype ([A],[A]).

https://doi.org/10.1371/journal.pone.0331044.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Examples of prisoner’s dilemma games with intermediate inheritance and in which the total payoff is maximized by the cooperator-defector strategy pair (anti-coordinated case).

See the caption at Fig 6.

https://doi.org/10.1371/journal.pone.0331044.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Example for Haldane monotonicity.

Here we consider the payoff matrix and assume that allele [a] is recessive to [A]. Notice that the orbits, as they move toward the vertex (representing the state where all individuals are recessive homozygotes ([a],[a])), intersect the level lines (displayed as parallel lines in the figure) of the function , which describes the frequency of the recessive allele, exactly once. The vertex corresponds to the state in which all individuals are heterozygotes ([a],[A]), while corresponds to the state in which all individuals are homozygotes ([A],[A]). The black dot indicates an asymptotically stable rest point, while the empty dot marks an unstable equilibrium.

https://doi.org/10.1371/journal.pone.0331044.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Global stability of a homozygous state without Haldane monotonicity.

Consider a donation game with parameters , and , leading to the payoff matrix and assume that allele [a] is recessive to allele [A]. The parallel lines are the level lines of the function , which describes the frequency of the recessive allele. Observe that, along the orbits starting near the heterozygote vertex (which is not an equilibrium), this function first decreases and then increases. This demonstrates that the dynamics is not globally Haldane monotone. However, if we exclude an appropriate neighbourhood of the pure heterozygote vertex, the system exhibits Haldane monotonicity. In particular, near the vertex , where all individuals are of genotype ([a],[a]), the system is locally Haldane monotone. The vertex corresponds to the state in which all individuals are homozygotes with genotype ([A],[A]). The black dot indicates an asymptotically stable rest point, while the empty dot represents an unstable equilibrium.

https://doi.org/10.1371/journal.pone.0331044.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Summary of prisoner’s dilemma games in which the sum of the payoffs is maximized by the cooperator-cooperator strategy pair (coordinated case).

In the left column, the payoff matrices are given. The other columns contain the phase portraits of the corresponding genotype dynamics (1). Black dots indicate asymptotically stable rest points, while empty dots represent unstable equilibrium points. The bottom-left vertex of the triangle corresponds to the state (1,0,0), in which all individuals are ([a],[a]) cooperator homozygotes. The bottom-right vertex represents the state (0,1,0), in which all individuals are heterozygotes with genotype ([a],[A]). The top vertex corresponds to the state (0,0,1), where all individuals are defector homozygotes with genotype ([A],[A]).

https://doi.org/10.1371/journal.pone.0331044.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Summary of prisoner’s dilemma games in which the sum of the payoffs is maximized by the cooperator-defector strategy pair (anti-coordinated case).

See the caption at Fig 10.

https://doi.org/10.1371/journal.pone.0331044.g011

We emphasize that in the inequalities above, the multiplier 2 refers the number of players in the interaction, and is not related to the coefficient of genetic relatedness between full siblings.

In the next subsection we analyse how the outcome of the familial selection model depends on whether the cooperator phenotype is recessive, dominant or intermediate.

The cooperator phenotype is recessive.

If the cooperator phenotype is recessive then and . For the corresponding table, see Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Genotype survival table based on Table 1 if the cooperator strategy is recessive.

https://doi.org/10.1371/journal.pone.0331044.t004

In this setting, inequalities (8) and (9) ensure the evolutionary (and so asymptotic) stability of the homozygous states and while inequalities (10) and (11) provide sufficient conditions for the stable coexistence of all genotypes. Based on these conditions, four subcases can be identified in the coordinated case: both homozygous states are stable, only one is stable, or both of them are unstable. In contrast, in the anti-coordinated case, state can never be stable so only two subcases are possible: is unstable and is stable or both homozygous states are unstable. These subcases are illustrated by numerical examples in Figs 1 and 2.

Although coexistence does not necessarily imply the existence of an interior equilibrium, every numerical example in this article that satisfies the conditions for coexistence exhibits an (interior) ESGD. This demonstrates that Definition 1 is non-empty. To illustrate this, consider the example of coexistence from Fig 2. The payoff matrix is

The corresponding interior Nash equilibrium, which satisfies (6), is . This equilibrium is evolutionarily stable, as shown in the left panel of Fig 3, where the function is plotted in around , clearly indicating that attains a minimum in variable at . The right panel of Fig 1 shows the trajectories of the dynamics (1), demonstrating that the interior equilibrium point is asymptotically stable, whereas the two boundary equilibria and are unstable.

The cooperator phenotype is dominant.

If the cooperator phenotype is dominant then and . For the corresponding table, see Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Genotype survival table based on Table 1 if the cooperator strategy is dominant.

https://doi.org/10.1371/journal.pone.0331044.t005

Observe that Tables 4 and 5 illustrate different selection scenarios. For instance, in families, the survival rates vary. In particular, for each , the value of differs. According to the genotype-phenotype mapping, if the cooperator phenotype is dominant, the number of cooperators will be higher in families with the same genotype distribution.

If allele [a] responsible for cooperation is dominant, then the conditions of the stability of the homozygous states are as follows (see section SI C in S1 File).

State , in which all individuals are homozygotes , is an ESGD if

(12)

State , in which all individuals are homozygotes , is an ESGD if

(13)

Similarly to the recessive case, reversing the inequality (12) ensures that state is a repellor and not an ESGD. Likewise, reversing inequality (13) implies that state is a repellor and not an ESGD. It is important to note that conditions (12) and (13) apply to arbitrary payoff matrices, provided that allele [a] is dominant.

When the interaction among siblings is described by the prisoner’s dilemma, we can again distinguish four subcases in the coordinated case and two subcases in the anti-coordinated case, based on the stability properties of states and . These subcases are illustrated with particular examples in Figs 4 and 5.

Intermediate inheritance.

Finally, we turn to the case of intermediate inheritance. In this scenario, the phenotype of the homozygotes is the pure strategy , the phenotype of the homozygotes is the pure strategy , while the phenotype of the heterozygotes is the mixed strategy (see Table 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Genotype survival table based on Table 1 in case of intermediate inheritance.

https://doi.org/10.1371/journal.pone.0331044.t006

As in the recessive and the dominant cases, we provide mathematical conditions for the stability of the homozygote states and . In the main text, we only present the first-order condition. The detailed mathematical analysis is available in section SI C of S1 File.

State , in which all individuals are homozygotes , is an ESGD if

(14)

State , in which all individuals are homozygotes , is an ESGD if

(15)

Reversing inequality (14) implies that state is a repellor and not an ESGD, and reversing inequality (15) implies that state is a repellor and not an ESGD. These conclusions hold for arbitrary matrices under intermediate inheritance.

When applying inequalities (14)-(15) to the context of the prisoner’s dilemma, we again obtain four possible subcases in the coordinated case and two subcases in the anti-coordinated case. These scenarios are illustrated with concrete numerical examples in Figs 6 and 7.

Donation game in diploid Mendelian population and classical Hamilton’s rule

A special case of the prisoner’s dilemma is the so-called “donation game” [17], defined by the payoff matrix

where and denote the benefit and the cost of an interaction, respectively, with . The donation game serves as a link between the prisoner’s dilemma and Hamilton’s rule [5,46,47]. Since in a monogamous diploid family the relatedness between full siblings is , the classical Hamilton’s rule predicts cooperation between siblings when .

Although the donation game is formally not a survival game (the entries of the payoff matrix are not survival probabilities), starvation can be one of the primary causes of mortality. In times of famine, when survival depends on reaching a minimum level of food intake during a critical period, siblings may enhance each other’s survival through food sharing. Therefore, an additive model (where the outcomes of individual interactions are summed) is more suitable for modelling food sharing than a multiplicative model (where outcomes are multiplied).

It is worth noting that only non-negative payoff matrices make sense in the model. However, it is not a real limitation because adding the same constant to all entries of the matrix does not affect the relative fitness and thus leaves game-theoretical predictions unchanged. Therefore, instead of using the above payoff matrix, we can equivalently use the shifted matrix:

where can be interpreted as a baseline, interaction-independent payoff.

This raises the question: under what condition is the cooperator homozygotes locally stable in the game between diploid siblings? Since , the payoff matrix H satisfies and , indicating that the donation game corresponds to a special case of the prisoner’s dilemma, one in which the cooperator-cooperator pair maximizes total payoff (i.e., the coordinated case).

It is easy to see that inequalities (8), (12) and (14) correspond to the classical Hamilton’s rule (), inequalities (9), (13), and (15) are equivalent to . Therefore, within the inheritance systems considered, when survival is determined by the donation game, there is neither bistability nor coexistence. Only two outcomes are possible: if then the cooperator homozygotes become fixed through selection; if then the defector homozygotes do.

Haldane monotonicity

Price [48] was interested in how the relative frequency of an allele changes between the parental and the offspring population. Hamilton’s rule [5,6] and Haldane’s arithmetic [4] describe conditions under which the frequency of the altruistic allele strictly increases in well-defined selection scenario, thereby ensuring the evolutionary stability of the population of altruistic homozygotes. (Haldane’s arithmetic claims [4]: “The relative frequency of the altruistic gene increases, if by ‘self-sacrifice’, it has ‘rescued’ on average more than one copy of itself in its lineage.” We note that Haldane’s arithmetic is equivalent to evolutionary stability—in ESGD sense—of recessive homozygotes in the one-person altruistic game. Moreover, this equivalence holds even when the interaction between siblings is governed by a non-linear game [4].) These rules are applicable when fitness can be expressed in terms of explicit cost and benefit parameters. However, in the case of prisoner’s dilemma games, at least three parameters are typically needed, making the application of both Hamilton’s rule and Haldane’s arithmetic less straightforward.

Instead, we adopt a different approach to investigate when the frequency of a given allele increases along every trajectory of the genotype dynamics. Mathematically, this corresponds to the allele frequency functioning as a Lyapunov function. For simplicity, we focus on the case in which three genotypes are possible. The frequency of the allele [a] in homozygote is given by

The derivative of function with respect to the genotype dynamics (1) is

Since the genotype dynamics satisfies it follows that

Thus, the frequency of the [a] allele strictly increases if

The biological interpretation of this inequality is the following: we say that the genotype dynamics (1) Haldane monotonic if the relative advantage of a homozygote over the entire population is greater than that of a homozygote. The key difference between Hamilton’s rule, Haldane’s arithmetic, and Haldane monotonicity lies in their respective focuses. Hamilton’s rule emphasizes the interaction between two siblings, Haldane’s arithmetic considers how many parental genes are saved through the “self-sacrifice” of a cooperating sibling, whereas Haldane monotonicity centres on the relative advantage of a particular homozygote taking into account production rates within the entire population.

Remark 4. Mathematically, Haldane monotonicity provides only a sufficient condition for the global stability of a homozygous state; global stability can still arise even when Haldane monotonicity is not satisfied. For example, it is possible for a trajectory to initially move away from the homozygous vertex but later reverse direction and onverge to it (cf. Fig 9).

We first illustrate, through a numerical example, the existence of a matrix game in which the genotype dynamics is Haldane monotone, see Fig 8.

Second, one may ask whether the genotype dynamics is necessarily Haldane monotone when a donation game governs interaction between siblings? The answer is negative, see Fig 9. The intuitive explanation is as follows. When the whole population consists entirely of heterozygotes, Mendelian inheritance dictates that diploid embryos will follow Hardy-Weinberg proportions, i.e., 1/4, 1/2 and 1/4 are the respective relative frequencies of the recessive homozygotes, heterozygotes and dominant homozygotes. If phenotypic selection does not substantially alter these Hardy-Weinberg proportions, then the frequency of the cooperative allele cannot increase. The key insight is that changes in genotype frequencies are determined jointly by the genetic system and phenotypic selection. We note that the heterozygote vertex is a rest point of the genotype dynamics only in the extreme case where both homozygotes are lethal in all family types.

In diploid populations, the payoff matrix and the genotype-phenotype mapping together determine the evolutionary outcome

We first examine the conditions under which the homozygous states are evolutionarily stable, depending on the inheritance system and the (arbitrary) survival payoff matrix (i.e., where each matrix element is between 0 and 1). Applying the results in Supplementary Information B in [3] for matrix games, we obtain the following Table 7 (see SI C of S1 file).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Summary of the conditions on a payoff matrix for the homozygous states to be ESGD, and so asymptotically stable.

https://doi.org/10.1371/journal.pone.0331044.t007

Several noteworthy observations follow:

• The condition for the evolutionary stability of the dominant is the same as that for intermediate homozygote. Hence in the case of dominant-recessive and intermediate inheritance, the evolutionary stability (in ESGD sense), instability, or degeneracy of the state associated with coincide.
• Similarly, the condition for the dominant homozygote is identical to that for intermediate homozygote. Consequently, in both dominant and intermediate inheritance systems, the evolutionary stability, instability, or degeneracy of the state associated with coincide.
• It follows that the recessive and dominant cases (with respect to the allele in ) unambiguously determine the evolutionary outcome in the intermediate case. In particular, the stability of the state associated with in the intermediate case matches that in the dominant case, while the stability of the state associated with is the same as in the recessive case.

These coincidences are somewhat surprising, given the selection regimes defined in Tables 4–6 are quite different.

Secondly, based on these observations, we classify prisoner’s dilemma payoff matrices according to the evolutionary stability of the two homozygous states under different inheritance systems. In the coordinated case, where the cooperator-cooperator strategy pair maximizes the total payoff of the siblings, we find eight distinct evolutionary scenarios. In the anti-coordinated case, where the cooperator-defector strategy pair maximizes the sum of the siblings’ payoffs, there are only three. These cases are summarized in Figs 10 and 11, each accompanied by a concrete numerical example.

The numerical examples shown in Figs 10 and 11 highlight several important points:

• Both the cooperator and defector homozygote states can be the unique stable state by natural selection, see, e.g., and , respectively.
• Bistable scenarios can arise (e.g., ) and, in such cases, there is exactly one unstable interior rest point.
• Cooperator and defector genotypes can coexist (e.g., ), with a globally stable interior rest point in such cases. The genotype-phenotype map can change the coordinates of this stable rest point of the genotype dynamics.
• Although the phenotypic payoff matrix alone is not sufficient to predict the asymptotic stability of cooperation, it plays a decisive role in shaping the evolutionary outcome. For instance, in the anti-coordinated case, the population of cooperator homozygotes cannot be stable, yet coexistence between cooperators and defectors is still possible (see ).
• Despite the 1/2 genetic relatedness between siblings, and even when cooperation maximizes the total survival probability of siblings in the coordinated case, the population of defector homozygotes can be the unique asymptotically stable rest point of the genotype dynamics (e.g., ).
• The phenotypic payoff matrix and the genotype-phenotype mapping together determine the outcome of the natural selection. When the phenotypic payoff matrix is fixed, the genotype-phenotype mapping cannot only influence the coordinates of the stable rest point of the genotype dynamics, but can also determine whether both homozygotes are present or one of them is absent in the stable equilibrium (e.g., ).
• However, there is no case in which one inheritance system exhibits bistability while another leads to coexistence. At most, the stability of a single homozygous vertex can change. This suggests that although the genotype-phenotype mapping can influence the outcome of natural selection, the payoff matrix still imposes structural constraints on evolutionary possibilities.

Our results from the viewpoint of group selection

In our classification of prisoner’s dilemma games, the principle of “family welfare,” which means that siblings aim at maximizing the sum of their survival rates, proved to be a key factor. We found that this principle is essential for the evolutionary stability of a population of cooperator homozygotes. Specifically, such population can never be stable in the anti-coordinated case, in which the cooperator-defector strategy pair yields the highest combined survival rate of siblings, regardless of the inheritance system (cf. Fig 11). In contrast, in the coordinated case in which the cooperator-cooperator strategy pair maximizes the sum of the siblings’ survival rates, evolutionary stability of the cooperator genotype is possible (e.g., in Fig 10) but not guaranteed. Other outcomes such as bistability (e.g., ) and coexistence (e.g., ), or even global stability of the population of defector homozygotes may also arise (e.g., ). Thus, while the principle of “family welfare” plays a crucial role, the evolutionary stability of the population of cooperator homozygotes does not depend solely on it.

It is well known that theories based on group selection typically require at least two components for the spread of cooperation [49]: a) the synergistic effect of the number of cooperators on the fitness of group members, and b) the non-random group formation process, including composition.

In the context of familial selection discussed in this article, the family naturally defines the group, and its composition is determined by the genetic system. However, no synergistic effects are assumed, highlighting that group formation alone may be sufficient for the evolutionary stability of the population of cooperative homozygotes.

It is worth noting that standard models of group selection typically focus on asexual populations (e.g., [50,51]), and therefore are generally not suitable, or only limitedly applicable, for studying effects that emerge from group formation grounded in genetic foundations.

Our results from the viewpoint of kin selection

The donation game provides a link between the prisoner’s dilemma and the classical Hamilton’s rule (e.g., [5,46,47]). It belongs to the coordinated case and involves only two parameters: cost and benefit. We found that regardless of the genotype-phenotype mapping, the genotype dynamics leads to the asymptotically stable fixed point representing a population composed entirely of cooperator homozygotes when the classical Hamilton’s rule is satisfied.

However, in general prisoner’s dilemma games which typically involve more than two parameters, the evolutionary outcome is more complex. A population of homozygous defectors may be the unique evolutionarily stable state (see in Fig 10), but bistability and stable coexistence may also occur. Moreover, even a change in the genotype-phenotype mapping alone can alter the stability of the population of cooperator homozygotes (see in Fig 10), despite the payoff matrix remaining unchanged and relatedness being 1/2. It is also noteworthy that relatedness is not always a necessary condition for the stability of a homozygous cooperator population. For instance, inequalities (12) and (14) yield stability conditions without the usual multiplicative factor of 2 that reflects the 1/2 relatedness typical of siblings in monogamous population.

From the perspective of inclusive fitness, the cooperative gene can enhance its evolutionary success by indirectly increasing the survival probability of siblings who have a 1/2 chance of carrying the same allele. This indirect effect also operates under familial selection. The consideration of inclusive fitness thus explains the spread of the cooperative trait in models of kin selection. In line with this reasoning, we asked whether there exists a mechanism or condition within the framework of the genotype dynamics that guarantees an increase in the frequency of the cooperative allele along every trajectory. This led to introduce the concept of Haldane monotonicity. Unlike the inclusive fitness approach, however, the genotype dynamics not only allow us to characterize when a population of particular homozygotes can be asymptotically stable, but also to analyse phenomena such as bistability and the stable coexistence of different genotypes.

Van Veelen made an effort to identify conditions under which the classical Hamilton’s rule provides accurate predictions. Using replicator dynamics, he showed that in linear models, the classical Hamilton’s rule holds [47]. While evolutionary matrix games are indeed linear models, our focus differs from Van Veelen’s: we study the genotype dynamics within the framework of Haldane’s familial selection. In this setting, the framework of the classical Hamilton’s rule cannot be sufficient (mathematically speaking) for deciding evolutionary stability. While the classical rule relies on two parameters (cost and benefit) in addition to the relatedness, our model generally requires three parameters to formulate the inequalities that determine evolutionary stability, based on the entries of the payoff matrix.

In summary, some of our results align with those of detailed models of group selection and kin selection, for instance, in the case of the donation game. However, there are notable differences. One of the most important distinctions is that our model highlights the substantial impact of the genotype-phenotype mapping on the outcome of natural selection. This difference may stem from a key feature of our approach, which explicitly incorporates diploidy, whereas most detailed models of kin selection, as noted by Van Veelen et al. [5, p. 180], are based on haploid assumptions (see, e.g., [6]).

Genotype dynamics integrates core principles of kin and the group selection theories in familial selection

As noted above, the “welfare of the family” (that is, the sum of the survival rates of siblings) is a crucial determinant of the selection outcome. For instance, in the anti-coordinated case, where the cooperator-defector strategy pair maximizes the sum of the siblings’ payoffs, the population of homozygous cooperator can never be evolutionary stable (as the total payoff is higher when one of the interacting siblings defects). On the other hand, “family welfare” alone is not sufficient to predict the evolutionary stability of the population of homozygous cooperators: as shown by the payoff matrix in Fig 10, where the population always evolves toward the homozygous defector state, regardless of the genotype-phenotype mapping.

In the case of the donation game, we have shown that the classical Hamilton’s rule correctly predicts the evolutionary stability of a population consisting only of cooperator homozygotes. However, this arises the question: what is the precise role of the multiplier 2 in the inequality [52,53]? According to the classical Hamilton’s rule, it reflects genetic relatedness. From the perspective of group selection, it corresponds to the number of players in the matrix game. In the context of the donation game, both interpretations appear reasonable.

In a diploid population, when two siblings interact with the aim of maximizing the “welfare of the family”, both direct and indirect fitness effects come into play. In other words, under familial selection, the mechanisms of both kin selection and group selection are simultaneously at work. Thus, it can be concluded that the present model of familial selection, which focuses only on changes in genotype frequencies, bridges the foundational ideas of both theories within the framework of an “orthodox” Darwinian view.