Evolution of Assortative Mating in a Population Expressing Dominance

In this article, we study the influence of dominance on the evolution of assortative mating. We perform a population-genetic analysis of a two-locus two-allele model. We consider a quantitative trait that is under a mixture of frequency-independent stabilizing selection and density- and frequency-dependent selection caused by intraspecific competition for a continuum of resources. The trait is determined by a single (ecological) locus and expresses intermediate dominance. The second (modifier) locus determines the degree of assortative mating, which is expressed in females only. Assortative mating is based on similarities in the quantitative trait (‘magic trait’ model). Analytical conditions for the invasion of assortment modifiers are derived in the limit of weak selection and weak assortment. For the full model, extensive numerical iterations are performed to study the global dynamics. This allows us to gain a better understanding of the interaction of the different selective forces. Remarkably, depending on the size of modifier effects, dominance can have different effects on the evolution of assortment. We show that dominance hinders the evolution of assortment if modifier effects are small, but promotes it if modifier effects are large. These findings differ from those in previous work based on adaptive dynamics.


S1 Invasion and fixation of assortment modifiers
Here, we derive invasion and fixation conditions for modifiers inducing stronger or weaker assortment. We label the genotypic values and the genotype frequencies according to Table S1.
In all cases, we assume that fitness is given by equation (8) in the main text and that population size is constant and close to demographic equilibrium. To derive invasion conditions we can neglect matings between individuals carrying a modifier allele, i.e., O(p i p j ) terms (i, j ∈ {4, . . . , 10}). Moreover, the genetic composition of the population is adequately described by the vector (p 1 , p 2 , p 3 , p 4 , p 5 , p 6 , p 7 ) T . It is easily verified that the linearized recursion matrix of the vector (p 4 , p 5 , p 6 , p 7 ) T is where Recall that an asterisk indicates that genotype frequencies after selection are used, and that r denotes the recombination rate between ecological locus and modifier locus. The genotype frequencies of individuals carrying exactly one copy of the modifier in the next generation are then given by U · (p 4 , p 5 , p 6 , p 7 ) T . If the leading eigenvalue of U is larger than 1 in modulus, the modifier will spread, otherwise it will vanish. Similarly, one can determine whether a sufficiently frequent modifier will rise to fixation or not. Clearly, forã = 0 the modifier allele is selectively neutral and the leading eigenvalue λ of U is 1. For a modifier with small effect, i.e.,ã − a is small, the leading eigenvalue of U can be written as

S2
The sign of φ determines whether a modifier can invade or not. A rare modifier spreads if and only if φ > 0. Note that it follows from (S3) that a modifier decreasing assortment invades if and only if φ < 0.

S1.1 No dominance, small modifier effect
We start by observing that the case of no dominance allows a few simplifications of the matrix U . We restrict our attention to symmetric equilibria, i.e.,p 1 =p 3 , since it seems infeasible to calculate polymorphic asymmetric equilibria explicitly. However, this assumption is justified. If the modifier locus is fixed and codes for random mating, the symmetric equilibriump 1 =p 3 = 1/4 is globally stable, as we shall argue below. If the modifier locus is fixed and codes for complete assortment, the symmetric equilibriump 1 =p 3 = 1/2 is the only locally stable equilibrium (see below). For all other cases, we assume that assortment is either weak or almost complete such that stable equilibria with large regions of attraction exist sufficiently close to the symmetric equilibria. At a symmetric equilibrium, the matrix U simplifies to since holds at a symmetric equilibrium with no dominance. Furthermore, W 1 = W 3 . Then, the characteristic polynomial of W U is As shown in [1], the leading eigenvalue of U is the larger of the two solutions of Since U is irreducible, the leading eigenvalue is positive and simple. For symmetric equilibria, i.e.,p 1 =p 3 , it is also shown in [1] that the leading eigenvalue is larger than one if and only if To derive useful invasion conditions from (S7) one needs to know the gene frequencies at equilibrium.

S1.1.1 Weak initial assortment
Assume that the wild-type allele is fixed at the modifier locus. If the wild-type allele codes for random mating, we obtain the following equilibrium This equilibrium is the only polymorphic equilibrium, and it is locally asymptotically stable, whereas the monomorphic equilibria are unstable [2]. Since the case of random mating falls into the class of models studied in [3] with non-negative interaction coefficients, it follows from there that the polymorphic equilibrium is globally asymptotically stable. Now, consider a small degree of initial assortment a. If a 1, we obtain the following equilibrium More precisely, there necessarily exists an asymptotically stable equilibrium sufficiently near (S9). Substitution of (S9) into (S7) yields Thus, under the assumption of weak selection and weak assortment the invasion condition becomes (S11) This result includes the case of random mating for a = 0. Notably, in this case the leading eigenvalue can be calculated directly to be Moreover, we infer from (S12) that a sufficiently frequent modifier with small effectã goes to fixation in a population in which the wild type codes for random mating if and only if c < s +ã.

S1.1.2 Strong initial assortment
We set ε = exp(−a) and assume ε ≈ 0, so that terms of order O(ε 5 ) can be neglected. Up to fourth order in ε, the symmetric equilibrium with the modifier locus fixed for the wild-type allele iŝ More precisely, for a fixed modifier locus there necessarily exists an asymptotically stable equilibrium sufficiently near (S13). This follows because the symmetric equilibrium is globally stable under complete assortment, as we will show in Section S1.5. The invasion criterion (S7) simplifies to which is always satisfied since a, c ∈ [0, 1/4].

S1.2.1 Invasion
Assume no dominance and a population fixed for the wild-type allele at the modifier locus. The wild-type allele codes for random mating and the population is at equilibrium (S8). Moreover, we assume an a little more general function for the mating probabilities (9). Namely, if genotype g carries exactly one copy of the modifier, we set

S4
where 0 ≤ k, K < 1. If π(g, h) is given by (9), we have k = exp(−ã) and K = k 2 . The leading eigenvalue of U , i.e, the larger of the two solutions of (S6), is Thus, a modifier increasing assortment can invade a randomly mating population if and only if c > s. Similarly, in an initially weakly assortatively mating population at a symmetric equilibrium close to (S8), the leading eigenvalue of U is Consequently, a modifier increasing assortment can invade a population that mates weakly assortatively if and only if c > s + a/2. Note, that for the limit, we assumed that a k, K, i.e., that the degree of assortative mating caused by the modifier is large compared to the initial degree of assortment.

S1.2.2 Fixation
We consider a modifier with large effectã, i.e., such that ε 2 1, where ε := exp(−ã) 1. The wildtype allele at the modifier locus codes for random mating. If the modifier is fixed, the equilibrium gene frequencies at equilibrium are given by (S13). The leading eigenvalue of U at (S13) is Because a, s ∈ [0, 1/4], λ < 1 and hence the modifier goes to fixation if it is sufficiently frequent.

S1.3 Weak dominance, random mating
The above described invasion criterion is valid only in the absence of dominance. The reason is that the symmetry of the model without dominance is crucial in the derivation of (S9). Thus, we have to derive the leading eigenvalue directly as a Taylor Series in (ã − a). Assume that the population is fixed for the wild-type allele at the modifier locus and that the wild type codes for random mating. As shown in [2] only one polymorphic equilibrium (with fixed modifier locus) exists. This equilibrium is locally stable, whereas the monomorphic equilibria are unstable. As in Section S1.1.1 it follows from [3] that the polymorphic equilibrium is globally stable.
To calculate the globally stable equilibrium with monomorphic modifier locus coding for random mating, we assume dominance to be sufficiently weak to ignore terms of order O(d 3 ). We obtain , Consider the characteristic polynomial P of U at the equilibrium (S19). Clearly, P (1) = 0 forã = 0. Thus, ifã = 0, the leading eigenvalue of U can necessarily be written as asã → 0 for some φ which is independent ofã. By neglecting terms of order O(ã 2 ), the Taylor expansion of P atp leads to The strength of selection for assortment modifiers consequently decreases under weak dominance. However, the invasion condition is not affected by a small degree of dominance.

S1.4 Strong dominance, random mating
We set δ = (1 − d). By strong dominance we mean that d ≈ 1 such that terms of order O(δ 2 ) can be neglect. If the modifier locus is fixed and codes for random mating, the same argument as in Section S1.3 yields the existence of a globally stable polymorphic equilibrium. It is approximately given bŷ Moreover we have Since 3 √ 2 > 4, and it follows that modifiers increasing assortment can invade if c > s.

S1.5 Intermediate dominance and complete assortment
It is easily verified that only homozygotes exist for complete assortment for any level of dominance, i.e., p 1 +p 3 = 1. Moreover, the equilibrium condition simplifies to W 1 = W 3 = W . This condition implies that the only polymorphic equilibrium is the symmetric equilibriump 1 =p 3 = 1 2 . To determine the local stability of this equilibrium we need to derive the Jacobian matrix. This can be done similar as in [4]. From this it is easily verified that the symmetric polymorphic equilibrium is locally asymptotically stable. Moreover, it is easily verified that the monomorphic equilibria are unstable (cf. [4]).
Consider an assortment modifier that leads to the mating probabilities Because we assume 0 ≤ d < 1, we can make the natural assumption 0 ≤ ξ 3 ≤ ξ 2 ≤ ξ 1 < 1, which will hold as long as π is a monotone decreasing function of the differences between trait values. Then atp 1 =p 3 = 1 2 (S1) simplifies to

S7
where α 1 = 2+ξ 2(1+ξ) . Straightforward calculation yields the eigenvalues It is easily verified that −1 ≤ λ 3,4 ≤ 1, so that the leading eigenvalues equals one. Hence, modifiers decreasing assortment are neutral. It should be mentioned that we can also study a modifier decreasing the degree of dominance. Form eqs. (B2) to (B6) in [5], it becomes clear that a modifier decreasing dominance is also selectively neutral.

S1.7 Assortment vs. dominance
The invasion fitness of a modifier that induces an arbitrary degree of dominance in a randomly mating population is derived in [5]. The leading eigenvalue of the linearized transition matrix for a rare dominance modifier with effect d is given by For an assortment modifier with effectã 1 in an initially randomly mating population (S12) implies If the modifier effects d andã go to 0, λ d and λã behave qualitatively differently. Because λ d = 1 + O(d 2 ) and λã = 1 + O(ã), the strength of selection for a dominance modifier decreases faster than the strength of selection for an assortment modifier. Tables   Table S1. Notation for genotypes in Section S1.