Transmissible cancers and the evolution of sex under the Red Queen hypothesis

The predominance of sexual reproduction in eukaryotes remains paradoxical in evolutionary theory. Of the hypotheses proposed to resolve this paradox, the ‘Red Queen hypothesis’ emphasises the potential of antagonistic interactions to cause fluctuating selection, which favours the evolution and maintenance of sex. Whereas empirical and theoretical developments have focused on host-parasite interactions, the premises of the Red Queen theory apply equally well to any type of antagonistic interactions. Recently, it has been suggested that early multicellular organisms with basic anticancer defences were presumably plagued by antagonistic interactions with transmissible cancers and that this could have played a pivotal role in the evolution of sex. Here, we dissect this argument using a population genetic model. One fundamental aspect distinguishing transmissible cancers from other parasites is the continual production of cancerous cell lines from hosts’ own tissues. We show that this influx dampens fluctuating selection and therefore makes the evolution of sex more difficult than in standard Red Queen models. Although coevolutionary cycling can remain sufficient to select for sex under some parameter regions of our model, we show that the size of those regions shrinks once we account for epidemiological constraints. Altogether, our results suggest that horizontal transmission of cancerous cells is unlikely to cause fluctuating selection favouring sexual reproduction. Nonetheless, we confirm that vertical transmission of cancerous cells can promote the evolution of sex through a separate mechanism, known as similarity selection, that does not depend on coevolutionary fluctuations.


Differential equations
The following equations control the changes in densities of susceptible hosts (S), hosts that developed a neocancer by neoplasia (I 0 ), and hosts that are infected by a transmitted cancer (I T ): With birth rate b > 0, carrying capacity K > 0, mortality rates µ > 0 and ν > 0, rate of neoplasia λ 0 > 0, transmission rate β > 0, and rate of changes in infection status θ ∈ [0, 1].
With the change of variable N = S + I 0 + I T and I = I 0 + I T , the system of equations is equivalent to: Below, we derive the equilibrium state of this system of equations.

Equilibrium
At equilibrium, we get: With P * the prevalence of transmissible cancers at equilibrium: To get N * > 0, the condition b > µ + P * ν must be satisfied. If b ≤ µ + P * ν, then N * ≤ 0 and the host population gets extinct.

Local stability
Unlike in the population genetic model analyzed in S1 Appendix, we assume that time is continuous in the epidemiological model. At the equilibrium point, the local stability can therefore be inferred from the sign of the real part of the eigenvalues of the Jacobian matrix J at the equilibrium point (N * , I * , I * 0 ), the expression of which is: Notably, the terms of the Jacobian matrix do not depend on I * and I * 0 . They depend however on N * , and on P * the prevalence of transmissible cancers at equilibrium, calculated as: The last column of the Jacobian matrix J (N * ,I * ,I * 0 ) is all zeros except for the element along the diagonal. This element, − (λ 0 θ + µ + ν) − θ β P * , is therefore one of the eigenvalues. This eigenvalue is negative. The remaining eigenvalues are the eigenvalues of the smaller matrix: We can infer the sign of real parts of the remaining eigenvalues from the signs of the determinant and the trace of the matrix. In particular, the real parts of the remaining eigenvalues are both negative (which is the condition for equilibria to be stable) if: det J (N * ,I * ,I * 0 ) > 0 and trace J (N * ,I * , , and we can calculate: Following the condition of existence of the equilibrium (Equation 16), the second term of this expression is positive, and therefore: We also get the expression of the trace of the matrix: trace J (N * ,I * , The trace can also be expressed as: As detailed below, we can show that ν P is always negative under the condition of existence of the equilibrium. Given that det J (N * ,I * ,I * 0 ) > 0 and trace J (N * ,I * ,I * 0 ) < 0 under the condition of existence of the equilibrium, the real parts of the two remaining eigenvalues of the Jacobian matrix at equilibrium are both negative (just like the first eigenvalue; determined directly from the expression of the Jacobian matrix, Equation 22). The equilibrium point is therefore locally stable.
Here we use the notation X = (β − λ 0 − µ − ν) 2 + 4βλ 0 , and we aim at showing that ν P * − √ X < 0. ν P * − √ X has the same sign as: ν 2 P * 2 − X. We can calculate: We call Q the polynomial function: Therefore, ν P * − √ X has the same sign as the polynomial fonction Q at x = √ X. The discriminant of this polynomial function Q is positive: And there are therefore two values x 1 and x 2 that are solution to the equation Q(x) = 0: For ν − 2β ≥ 0, we get: This means that if ν − 2β ≥ 0, we get √ X ∈ [x 1 , x 2 ] and Q( √ X) < 0. Therefore, in that condition, we get ν P * − √ X < 0.

Proportion of neocancers at equilibrium
At equilibrium, we calculate the proportion of neocancers as: Therefore:α

Selection coefficient due to transmissible cancers at equilibrium
At equilibrium, we infer the selection coefficient as the lifespan reduction due to the risk of being infected by transmissible cancers:ŝ host = 1 1 + µ ν 1 P * (41)

Calculation:
The mean time spent in a each state (S, I 0 , I T , cf. Equation 1) is: As a newly infected individual (by either a neocancer or a transmitted cancer), you live: Which gives us: As a new born, you live: Without transmissible cancers, an individuals has an average lifespan equal to 1/µ. The lifespan reduction due to the risk of being infected by transmissible cancers is therefore: Yet, given that dI dt = 0 at equilibrium, then λ 0 + β I * N * (N * − I * ) − (µ + ν) I * = 0, and β I * N * = (µ+ν)I * N * −I * − λ 0 and therefore:ŝ This means that the lifespan reduction due to the risk of being infected by transmissible cancers is equivalent to the relative mortality rate caused by cancer in the population.
Finally, given that I * = P * N * at equilibrium:ŝ

Effect of b, K and θ
While parameters b and K change the densities N * , I * , I 0 * at equilibrium, they do not affect P * ,α andŝ host . Parameter θ has no effect on the densities N * , I * , I 0 * at equilibrium, and does not affect P * ,α andŝ host .

Effect of ν
On the prevalence of transmissible cancers: We determine the effect of ν on the prevalence of transmissible cancers: An increase in the cancer-associated mortality rate increases the mortality rate of infected hosts, thereby decreasing the prevalence of transmissible cancers at equilibrium.

On the proportion of neocancers:
We determine the effect of ν on the proportion of neocancers at equilibrium: The cancer-associated mortality rate decreases both the densities of hosts infected by a neocancers (I 0 ) and hosts infected by transmitted cancer (I T ) at equilibrium. As shown by the signs of the terms ∂ 1 µ+ν ∂ν < 0 and ∂( 1 P * −1) ∂ν > 0, the cancerassociated mortality rate affects the proportion of neocancerα depending on those effects on I 0 and I 0 + I T .
And therefore: A high cancer-associated mortality rate decreases the density of hosts infected by transmitted cancers relatively more than the density of hosts infected by neocancers. This makes sense because a high mortality rate of infected hosts decreases the density of hosts infected by transmitted cancers via both increased mortality and reduced transmission. Overall, increased cancer-associated mortality rate increases the proportion of neocancers.
On the selection coefficient: The cancer-associated mortality rate has antagonistic effects on the selection coefficient due to transmissible cancers: As emphasized by the term ∂(νP * ) ∂ν , a high cancer-associated mortality rate increases mortality of infected hosts, but also decreases the cancer prevalence (which increasesŝ host ). The balance between those antagonistic effects will determine whether increased cancer-associated mortality rate decreases or increases the selection coefficient due to transmissible cancers -i.e, whether ∂ŝ host ∂ν < 0 or > 0.
Therefore, ∂ŝ host ∂ν has the same sign as: and ∂ŝ host ∂ν > 0 for: Yet, because µ + ν > µ, we can show that: And therefore, ∂ŝ host ∂ν > 0 for : Increased cancer-associated mortality rate increases the selection coefficient as long as the mortality rate of infected individuals (µ + ν) is low enough. Otherwise the reduction of prevalence decreases the selection coefficient due to transmissible cancers.

Effect of µ
Given that the expressions of P * andα depends on µ + ν. The parameter µ has the same effect on P * andα as parameter ν by affecting the mortality rate of infected individuals.
On the prevalence of transmissible cancers: An increase in the baseline mortality rate increases the mortality rate of infected hosts, thereby decreasing the prevalence of transmissible cancers at equilibrium.
On the proportion of neocancers: An increase in the baseline mortality rate increases the mortality rate of infected hosts, thereby increasing the proportion of neocancers.
On the selection coefficient: Increased baseline mortality rate decreases the selection coefficient due to cancer by reducing both the prevalence of transmissible cancers (P * ) and the relative mortality cost associated with cancer (ν/µ).
Effect of λ 0 On the prevalence of transmissible cancers: We determine the effect of λ 0 on the prevalence of transmissible cancers: Yet, Therefore: The rate of neoplasia increases the prevalence of transmissible cancers at equilibrium.

On the proportion of neocancers:
We determine the effect of λ 0 on the proportion of neocancers at equilibrium: The rate of neoplasia increases both the densities of hosts infected by a neocancers (I 0 ) and hosts infected by transmitted cancer (I T ) at equilibrium. As shown by the sign of the terms 1 P * − 1 > 0 and λ 0 ∂( 1 P * −1) ∂λ0 < 0, the rate of neoplasia affects the proportion of neocancerα depending on those effects on I 0 and I 0 + I T .
And therefore: A high rate of neoplasia increases the density of hosts infected by neocancers relatively more than the density of hosts infected by transmitted cancers. Overall, increased rate of neoplasia increases the proportion of neocancers.

On the selection coefficient:
A high rate of neoplasia increases the selection coefficient due to transmissible cancers by increasing the prevalence:

Effect of β
On the prevalence of transmissible cancers: We determine the effect of β on the prevalence of transmissible cancers: Therefore: The transmission rate increases the prevalence of transmissible cancers at equilibrium.

On the proportion of neocancers:
We determine the effect of β on the proportion of neocancers at equilibrium: A high transmission rate increases the density of hosts infected by transmitted cancers. Therefore, increased transmission rate decreases the proportion of neocancers.