Mechanistic model of evolutionary dynamics

To investigate the evolution of foraging behavioral strategies, we focus on modeling a population of organisms that must navigate their environment in search of locations favorable for reproduction. For simplicity, we will assume the favorability of each location is determined by the amount of a non-depleting “resource” at this location. Additionally, each location has a fixed carrying capacity, allowing only a single individual to occupy the location at any given moment. Organisms are able to detect the resource level only at their current locations, and must decide whether to remain where they are or explore in search of regions with more resources (see Fig 1A); this is referred to as the “exploration-exploitation trade-off” [22]. In this model the organisms’ movements are not deterministic, but instead each individual will attempt to relocate stochastically at rates determined by the amount of resource at the individual’s current location; the value of these rates is determined by the individual’s phenotype. The resource level at each location is predetermined and sampled from a discrete distribution; see Fig 1B, depicting a discrete uniform resource distribution. The set of relocation rates (used interchangeably with the term “hopping rates”) for each resource level comprises an organism’s phenotype in this model; see Fig 1C.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic diagram of the mechanistic evolutionary dynamics model.

A. The model consists of a population of agents foraging in an environment with a random distribution of resources, in which each agent can either asexually produce an offspring (as shown by the orange arrow) or relocate to one of the vertical or horizontal neighboring locations (as shown by the dashed blue arrow). Different colors here distinguish agents with different phenotypes. B. The environment is a two dimensional discrete lattice on which each location has a predetermined resource level S_k, k ∈ {1, 2, 3, 4, 5}, chosen from a distribution (a discrete uniform distribution shown here). A 20 × 20 subset of the environment is shown here. Each site on the lattice is colored according to its resource level S_k. Agents at locations with resource level S_k will reproduce at a birth rate b_k. C. Each agent has a phenotype vector with 5 components, corresponding to the relocation or hopping rates of the agent at 5 different resource levels. These components are initially chosen randomly for each agent and are shaped over the course of evolution. In the example shown here, agent 2 has a relocation rate γ₃ ≈ 4 when it is at a site with resource level S₃. In the agent-based stochastic model, an offspring inherits a noisy copy of its parent’s phenotype, as shown here for the phenotypes of agent 1 and its offspring.

https://doi.org/10.1371/journal.pcbi.1009934.g001

In addition to determining an organism’s relocation rate, the resource level at each site directly determines the organisms’ birth rates at that location. Importantly, the birth rates do not depend on the organisms’ phenotype—the only way the phenotype influences the organism’s reproductive success is by improving the organism’s odds of encountering high-resource sites and staying there long enough to produce sufficiently large numbers of offspring. That is, the selective advantage of a phenotype is due only to its dynamical success in the environment, not any intrinsic advantage. For simplicity, we model all reproduction as asexual. In the agent-based model to be defined shortly, the offsprings’ phenotypes are taken to be noisy copies of the parents’ phenotype (see “Agent 1” and “Agent 1 offspring” in Fig 1C). In the mean-field model discussed in the next section, the offsprings’ phenotypes will be direct copies of the parents’.

The final ingredient the model needs is a means of selection. Typically, one would allow for stochastic death of individuals, such that phenotypes that do not confer a sufficiently competitive advantage will eventually die out, leaving the more competitive phenotypes to dominate the population. However, a technical downside of stochastic death is that extinction becomes not only possible but inevitable, potentially complicating investigation of long-term phenotype distributions. To avoid the technical complications of stochastic death, we instead induce selection by subjecting the population to a periodically-occurring “catastrophe” that randomly culls individuals until the population is reduced to its initial size. In doing so, the population is never at risk for extinction, but selection still occurs because the phenotypes that are able to produce large numbers of offspring have better odds of surviving each catastrophe.

To formalize this schematic picture, we first build an agent-based stochastic model. In this model the population consists of individual organisms, or “agents,” each of which is assigned a location within a 2d environment, modeled as a discrete lattice, and the values of its hopping rates γ at each resource level. In the simulations we investigate here, we assume the resource level of site at location x, labeled s_x, is drawn from 5 distinct resource levels, s_x ∈ {S₁, S₂, S₃, S₄, S₅}. Thus each agent has 5 corresponding hopping rates γ = {γ₁, γ₂, γ₃, γ₄, γ₅}, one for each of the five resource levels. The values of each of these hopping rates are initially chosen randomly from the range [γ_min, γ_max], where γ_min is the minimum rate at which an organism relocates and γ_max is the fastest rate an organism can relocate—we interpret this upper bound on relocation rates as being set by metabolic constraints that we do not model or allow to change during the evolution of the population.

The state of the population of agents is updated in continuous time. At any given moment, an agent can either relocate to or asexually give birth to an offspring into one of the vertical or horizontal neighboring locations, as schematically shown in Fig 1A (because only one agent is permitted per site, an agent cannot give birth to a child on its current site). The target site is chosen randomly with a probability 1/4 in a 2d environment. The per-capita birth rate of an agent, b(s_x), is a function of the local resource level s_x at location x. Because the resources are not depleted in this model the actual numerical values of the s_x are unimportant; the meaningful values are the birth rates b(s_x). However, we assume that the values of the birth rates are ordered according to resource level, such that a site x with resource level s_x = S₁ has the lowest birth rate and a site with resource level s_x = S₅ has the highest birth rate. Again, we stress that the birth rates are entirely independent of an agent’s phenotype γ.

Agents with very different phenotypes will navigate the environment in different ways. An agent with a small value of γ₁, for example, will tend to remain in sites of resource level S₁ for long periods of time, while agents with large values of γ₁ will tend to leave sites of this resource level rather quickly. Because the birth rate in sites of resource level S₁ is low, we might expect that agents with small γ₁ may be less successful at proliferating than agents with larger γ₁, and will therefore be less likely to survive the periodic catastrophes that occur, creating a selection pressure toward larger values of γ₁. Similarly, agents with small (large) values of γ₅ will tend to remain in these high-resource sites for long (short) periods of time. Agents with small γ₅ may therefore produce more offspring because b(S₅) is the highest, increasing their odds of surviving the catastrophes. This line of thinking makes it seem intuitively “obvious” that the most competitive phenotype ought to have large hopping rates out of low resource sites and minimal hopping out of the highest resource sites. Moreover, we might expect we can identify an objective function that yields this phenotype as its optimum. In the next sections we present the results of our stochastic simulations as well as the numerical solutions of the mean-field model, showing that this intuition is largely correct when the population density is low—in which case we can indeed derive an effective objective function—and when the distribution of resource levels are drawn from a discrete uniform distribution, but this picture is not always correct when high resource sites are rare or population density is high.

Distributions of competitive phenotypes are selected by birth-death dynamics

For the results presented in this section, we simulate our agent-based model using the Gillespie algorithm [23], starting with 500 agents on a 128 × 128 2d lattice. This corresponds to an initial population density of 3.1%. Each lattice site belongs to one of the 5 resource levels {S₁, S₂, S₃, S₄, S₅}, which is drawn from a discrete uniform distribution. Initially, each agent is assigned a spatial location x = (x, y) and a phenotype vector γ = (γ₁, γ₂, γ₃, γ₄, γ₅) uniformly at random. Catastrophes occur after a simulation runtime T, which defines an iteration of the simulation. While the catastrophe randomly picks which agents to cull, it always reduces the total population back to 500 agents. In Distribution of resources shapes exploration-exploitation trade-offs we consider non-uniform distributions, and in Ranges of population density allow for long-lasting phenotype coexistence we investigate different lattice sizes as a means of changing the initial population density. The details of the simulation setups are presented in the Methods Section.

Fig 2A–2D displays the evolution of the distribution of phenotypes over several thousand catastrophes, with snapshots at (A) 0, (B) 50, (C) 500, and (D) 20000 iterations; these distributions represent averages over 20 trials of the dynamics starting from random initial conditions generated from the same distribution. The initially uniform distributions of each of the five hopping rates (phenotype components) are gradually reshaped by births and deaths. At long times, the distributions are approximately stable; we do not expect the distributions to converge to a single value for each γ_k due to the reproduction noise that accompanies each birth. See S1A–S1C Fig for additional examples for different realizations of the resource levels randomly assigned to each site. In Fig 2E we plot the evolution of the mean value of each hopping rate over the 20000 catastrophes, averaged over 20 trials. The shaded regions of the plot correspond to the width of the distributions (the standard deviation of each hopping rate component). These dynamics are qualitatively reproduced by a mean-field model, shown in Fig 2F, and extensively discussed in A minimal mean-field model explains variability in competitive phenotypes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Emergent competitive phenotypes from evolutionary birth-death dynamics.

A-D. Violin plots of distributions of phenotype components in the agent-based stochastic simulations at iterations 0, 50, 500, and 20000, respectively. Each violin plot combines data from 20 independent trials. The connected white dots are the median values of each relocation rate component, and the black boxes confine the first and third quartiles of each phenotype component (i.e., box plot). The colored areas are the kernel density estimations of the underlying distributions. Panel A shows a uniform relocation rate distribution for each phenotype component as the initial condition at iteration 0. The phenotype distributions evolve as the birth-death dynamics play out in the agent-based model and approach a steady shape after around 20000 iterations as shown in panel D. E The evolutionary trajectory of each phenotype component, averaged over 20 independent trials. The shaded area represents the standard deviations of phenotype components. The stochastic birth-death dynamics selects out competitive phenotypes whose first three phenotype components γ_1,2,3 saturate at hopping rate of around 9, close to the maximum possible hopping rate of γ_max = 10, and whose last phenotype component γ₅ saturates at hopping rate around 0.5, close to the minimum possible hopping rate of γ_min = 0. In contrast, the fourth component γ₄ approaches a value around 8 with a relatively large variability throughout the simulation. F. Predictions of the mean relocation rate of each phenotype component and their standard deviations using the coupled 5-state mean-field model, discussed in A minimal mean-field model explains variability in competitive phenotypes. Note that the mean-field model uses a slightly different birth rate b₅ = 5, while the stochastic simulations use b₅ = 4, and the 5-state mean-field model converges at a much shorter timescale (700 iterations vs. 20000 iterations). These quantitative differences between the full agent-based model and the simplified 5-state model are discussed in detail in the Discussion, S5 and S6 Figs.

https://doi.org/10.1371/journal.pcbi.1009934.g002

We see that the long-time distribution appears to favor high hopping rates in sites with resource levels S₁ to S₃ and low hopping rates for sites with the highest resource level S₅, reflecting our intuition that the most competitive phenotypes would be those that result in agents leaving their location as fast as possible when at low resource sites and staying put as long as possible when at high resource sites. Notably, however, the distribution of the hopping rates γ₄ at the second-highest resource level S₄ is not narrow, but spans much of the entire possible range of hopping rate values. This indicates that the selection pressure on this phenotype component is relatively weak compared to the selection pressure on the other components. While the average value of γ₄ is closer to γ_max than γ_min, the large variance demonstrates that the most competitive phenotype does not conform tightly to our intuitive predictions, and the situation is more complex than might be naively expected. To gain insight into our results and investigate how selection pressure depends on our environments’ properties, namely the distribution of resources and the values of the birth rates tied to each resource level, we develop a deterministic minimal mean-field model that enables us to derive an effective objective function in a low-density limit. This fitness function predicts the optimal phenotypes and explains the lack of selection pressure on γ₄ observed in our stochastic simulations.

A minimal mean-field model explains variability in competitive phenotypes

Agent-based models are often difficult to formulate directly as mathematical models, owing to the combination of having individual with private properties (here, their phenotype) and the fact that agents may be born or culled at any time. However, a formalism that can approximate the agent-based model outlined here is a master equation formalism that keeps track of the probability of configurations of the number of agents with each phenotype at each spatial location. From such a formalism one can then derive a system of differential equations for the expected number of agents with each phenotype and location. Such a model serves as the starting point for the reduced model we investigate in this section. We focus only on the reduced mean-field model here; see the Methods for details on the full master equation model and the approximate reduction to the minimal mean-field model described below.

When agent density is low, such as in the simulations of the previous section (initial density of 3.1%), we may assume that events in which an agent is blocked from relocating to another site by another occupying agent are rare and may be neglected. In this approximation the growth dynamics of the different phenotypes decouple, and we can describe the dynamics of the population as a set of linear differential equations describing the expected number of agents at each site. To simplify the mean-field model further, we use the fact that each site belongs to one of the five resource levels and only count the number of agents at each level, eliminating the spatial dependence of the environment. This reduces the model to a 5-state model whose solution is the time course of the expected number of agents of each phenotype at each resource level; see the Methods for a detailed derivation of this reduction. The resulting system of differential equations reads (1) where n_kγ is the expected number of agents at a site with resource level k and phenotype γ. The parameter b_k is the per-capita birth rate of resource level k, a constant of the environment, and γ_k is the hopping rate out of state k, determined by the phenotype. Again, we note that in this approximation the dynamics of different phenotypes are decoupled, such that there is a system of differential equations of the form Eq (1) for each phenotype γ. For practical calculations we must discretize the phenotype space, giving a finite system of differential equations. Finally, the weights w_k correspond to the probability that a site in the agent-based model has resource level S_k. For resource levels drawn from a discrete uniform distribution, which we investigate first, w_k = 1/5 for all k = 1 to 5.

The first term, w_k∑_i b_in_iγ, in Eq (1) is the number of new agents being birthed into state k by agents in all other states. The second term w_k∑_i γ_in_iγ is the number of agents hopping into resource level k from all other resource levels, formally including resource level k itself to account for the possibility of an agent hopping from a site of resource level k to a neighboring site that also has resource level k. The last term −γ_kn_kγ is the number of agents hopping out of resource level k.

The system of differential Eq (1) can be written down in matrix form as , which has a solution in terms of the matrix exponential, . The between-catastrophe dynamics of each phenotype are therefore dominated by the eigenvectors and eigenvalues of . Although the dynamics of the phenotypes are independent, they are coupled through the periodic catastrophes, which reduce the population down to its original size after a time T. Let Λ_γ be the maximum eigenvalue of , which we interpret as a function of the phenotype parameters γ = (γ₁, γ₂, γ₃, γ₄, γ₅). In the Methods we show that the matrix has purely real eigenvalues and only the maximum eigenvalue Λ_γ is positive. Therefore, the phenotype γ that yields with the largest maximum eigenvalue Λ_γ overall will be the only phenotype to survive after a large number of catastrophes, with coexistence possible only if different phenotypes have the same value of Λ_γ. Because the hopping rates γ are just parameters in this formulation of the model, the eigenvalues of will simply be functions of the phenotype parameters (as well as environmental parameters), and therefore the maximal eigenvalue serves as an effective objective function of the system in the low-density limit.

If the maximal eigenvalue of this minimal model acts like an effective objective function, then we would expect the lack of selection pressure on the hopping rate γ₄ to be reflected in an insensitivity of this eigenvalue to varying γ₄, i.e., there would be a range of values of γ₄ for which the maximum eigenvalue approximately takes on its largest value. This is indeed what we find but only when the birth rate in the highest resource site b₅ is close to a particular critical value B_5c. As shown in S2 Fig, when b₅ > B_5c we find that the competitive population dynamics drive γ₄ → γ_max, and the eigenvalue Λ_γ is insensitive to variations in the value of γ₄ around this point. When b₅ < B_5c and is close to b₄, the birth rate in sites of resource level S₄, we instead find that selection drives γ₄ → γ_min, again with little variation in the maximal eigenvalue. This makes sense: if the birth rates b₄ and b₅ are close then there is little advantage in leaving a site with resource level S₄ to search for a site of level S₅. That is, the maximal eigenvalue predicts a transition from γ₄ → γ_min to γ₄ → γ_max as b₅ increases from b₄ and crosses B_5c. Near the transition at b₅ ≈ B_5c the maximum of the eigenvalue is less sensitive to γ₄—equivalently, fluctuations in γ₄ are large, as is often observed near phase transitions and bifurcations [24], such that selection is weak, resulting in coexistence of a range of γ₄ values near this transition, at least in a finite population. See S3A–S3C Fig for the distributions of phenotypes around this transition point.

Fig 3A, upper panel, demonstrates that the predicted transition occurs in both our minimal mean-field model and our full agent-based simulations. The critical value B_5c is different for the full stochastic simulation and the mean-field model, but this is to be expected due to the simplifications of the mean-field model as well as the lack of demographic noise. As seen in Fig 3A, upper panel, the distributions of γ₄ are indeed narrower when b₅ is far from the critical value B_5c, reflecting that the selection pressure indeed increases when there is a clear advantage to exploring (b₅ > B_5c) versus exploiting (b₅ < B_5c). In principle, we might expect to see similar transitions in the other hopping rates γ₁, γ₂, and γ₃ if we adjust the value of b₄, b₃, etc. Instead of varying these parameters to investigate whether such transitions exist, we shift focus to another ecologically important scenario in which these transitions will occur: the effect of the distribution of resources throughout the environment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The most competitive phenotypes are shaped by the rarity of the highest-resource sites.

The abundance of the five resource levels distributed with frequency w_k ∝ e^−ck, where k ∈ {1, 2, 3, 4, 5} and the exponential coefficient c determines the rarity of levels. The most competitive phenotype changes as a function of birth rates and resource rarity: A, top. For resources drawn from discrete uniform distribution (c = 0), the fourth component of the most competitive phenotype γ₄ transitions from low to high as the birth rate b₅ increases. All other phenotype components remain steady as γ_1,2,3 → γ_max and γ₅ → γ_max. The agent-based stochastic simulation and the mean-field prediction transitions are qualitatively similar, though the transition occurs at different values of b₅. A, bottom. A similar transition in γ₄ occurs for a resource distribution with exponential coefficient c = 0.5, which has rare high resource level sites. The transition persists but at a higher birth rate b₅. B. A heatmap plot of the fourth phenotype component γ₄ as a function of the exponential coefficient c and the birth rate b₅. The orange dashed line is the predicted transition boundary from the mean-field model, where γ₄ = 5 maximizes the effective fitness function derived in the low-density limit. The two white dotted lines show the range of b₅ for c = 0 and 0.5 in panel A.

https://doi.org/10.1371/journal.pcbi.1009934.g003

Distribution of resources shapes exploration-exploitation trade-offs

The distribution of resources throughout an environment will have an impact on the most successful phenotypes. For instance, when high resource sites are scarce, it may be advantageous to settle for lower resource locations. To explore this effect, we vary the distribution of resource levels within the random environment, in particular by selecting distributions in which the high-resource sites are much rarer than lower resources sites. In the previous sections, the probability that any site has resource level S_k, denoted w_k, was chosen to be w_k = 1/5; i.e., a discrete uniform distribution of the 5 resource levels. A simple choice of distribution in which high resources sites become rarer than low resource sites is an exponential probability w_k = Ae^−ck, where is the normalization constant and c is a parameter that determines the steepness of the distribution. The factor w_k is the same factor that appears in Eq (1). Note that c = 0 results in w_k = 1/5; a positive c results in a distribution that resource level rarity increases with k, that is, the high resource levels become rarer than the low resource levels.

As shown in Fig 3A, lower panel, our agent-based simulations confirm that the transition from an average value of 〈γ₄〉 = γ_min = 0 to γ_max = 10 still exists when c is, e.g., 0.5, but occurs at a much larger value of b₅ ≈ 11. Indeed, as seen in Fig 3B, there exists a transition line in the b₅-c plane that separates 〈γ₄〉 ≈ γ_min from γ_max = 10. The orange dashed curve is the boundary predicted by evaluating the maximal eigenvalue of the low-density model Λ_γ as c varies from 0 to 1 and solving for the value of b₅ that yields γ₄ = (γ_max + γ_min)/2 = 5 as the most competitive phenotype. This curve represents the line along which the population achieves a balance of the exploration-exploitation costs. See S1D–S1F Fig for examples of the phenotype distributions in different realizations of environments with exponentially distributed resource levels and S3D–S3F Fig for the phenotype distributions around the transition point.

Intuitively, agents staying at low resource levels need sufficient incentives to leave their current locations and start to explore new sites, at the risk of missing out reproductive opportunities while exploring. These results demonstrate that rarity of high resource sites can drive selection of phenotypes that settle for sites with lower resource levels.

Importantly, here the rarity of sites is only due to the intrinsic environmental distribution of resources. However, high resource sites could also become dynamically rare and unavailable as they become occupied, preventing multiple individuals from exploiting the same sites. So far we have been working within a low-density regime, such that it is relatively rare for organisms’ movements to be blocked by other individuals, and such that a sufficient fraction of high resource sites remain available for discovery and occupation. We now turn to investigating how the competitive phenotypes change with organism density.

Ranges of population density allow for long-lasting phenotype coexistence

In order to investigate how population density impacts the most competitive phenotype changes, we return to our agent-based stochastic simulations with resource levels drawn from a discrete uniform distribution. The simulations presented previously had 500 agents on a 128 × 128 2d lattice, resulting in an initial occupancy level of 3.1%. To study the effect of population density we keep the initial number of agents fixed and decrease the lattice size L × L to L = 96, 80, 64, 48, 40, 35, 32, 28, 26, 25, 24, and 23, giving initial occupancies of 5.4%, 7.8%, 12.2%, 21.7%, 31.2%, 40.8%, 48.8%, 63.8%, 74.0%, 80.0%, 86.8%, and 94.5%, respectively. One could of course also vary density by increasing the initial agent number and keeping the lattice size at 128 × 128, but increasing the number of agents dramatically increases the simulation load.

For each density we ran 20 independent trials of the agent-based stochastic simulation, the results of which are summarized in Fig 4. We found that as density—and hence interactions among agents—increases, the most competitive phenotypes also change. Notably, these changes qualitatively mimic the effects of increasing the rarity of high resource sites: the mean hopping rate 〈γ₄〉 saturates at a value γ_min < 〈γ₄〉 ≈ 5 < γ_max with a large variance at 12.2% initial density, and drops to 〈γ₄〉 ≈ γ_min with small variance at higher densities. See S4 Fig for examples of the phenotype distributions at increased densities, which show this transition take place.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Higher initial densities alter the most competitive phenotypes.

A-D. The evolution course for populations of agents starting with different initial densities: 12.2%, 21.7%, 48.8%, and 86.8%, respectively; c.f. Fig 2E at 3.1% density. The initial density is varied by fixing the initial population size at 500 and decreasing the environment size L × L with L = 128, L = 64, 48, 32, and 24. We observe a systematic shift of the most competitive phenotypes; in particular, the fourth phenotype component γ₄ decreases as the initial density increases. E. Mean values of γ₄ after 20000 iterations in the agent-based model as a function of the initial density, showing the change in the most competitive phenotype. At ultrahigh densities, e.g. 94%, there is no space for agents to move or give birth, resulting in little change in the initially uniform phenotype distribution. F. Mean-field model predictions of the most competitive phenotypes as a function of initial density after 700 iterations. A birth rate of b₅ = 5 is used in the mean-field model so that 〈γ₄(t)〉 converges to 10 at low densities.

https://doi.org/10.1371/journal.pcbi.1009934.g004

The results observed in our stochastic simulations are replicated by a 5-state mean-field model that incorporates the effects of site occupation: (2) where n_kγ is again the number of agents at sites of resource level k and M = L² is the total number of sites, with w_kM the expected number of sites of resource level k, which sets a maximum occupancy for each state. Note that when this model reduces to Eq (1). Importantly, Eq (2) is nonlinear, and unlike the low-density approximation of Eq (2), all phenotypes are dynamically coupled. Not only does this preclude writing down an analytical solution, but the dynamical coupling between phenotypes also suggests there may be no effective objective function of a single arbitrary set of the phenotype parameters γ = (γ₁, …, γ₅) that predicts the most competitive phenotype simply by maximizing it with respect to these values.

As in the low-density model, we solve Eq (2) for a discretized set of phenotypes. Solving the high density model is comparatively computationally demanding due to the necessity of solving for all phenotypes simultaneously, but is still feasible for the set of discretized phenotypes chosen in this study. As shown in Fig 4F, the mean-field predictions are consistent with our full stochastic simulations, and we indeed observe changes in the most competitive phenotype as a function of initial density N₀/M. At low densities the value of 〈γ₄〉 is close to γ_max (after 700 catastrophes), and as N₀/M increases we see a coexistence region in which multiple phenotypes are present at an observation time T_obs, before 〈γ₄〉 drops to γ_min at densities N₀/M ≳ 20%. As N₀/M increases towards 1, we observe an increase in 〈γ₄〉. This extreme high-density limit represents the scenario in which the population is so dense that there is little room for agents to give birth or move, leading to an extremely slow change in the composition of the phenotypes. In the extreme limit N₀/M = 1 we have 〈γ₄(t)〉 = (γ_max + γ_min)/2 for all times t, due to the fact that phenotypes are initially present in equal proportions and there is no room for any agent to move or give birth.

The mean-field model is useful for dissecting the dynamics of the phenotype changes. Initially, we hypothesized that the dominance of γ₄ = γ_min phenotypes at high densities is due to the S₅ resource sites filling up, leaving any γ₄ = γ_max agents that did not find an S₅ site to search fruitlessly for resource-level 5 sites instead of producing offspring, allowing the γ₄ = γ_min phenotypes to gain the reproductive advantage. However, as shown in Fig 5, it turns out that early in the evolutionary dynamics the γ₄ = γ_max agents lag behind the other phenotypes, as the γ₄ = γ_max agents leave S₄ sites more frequently and explore the neighboring locations while agents with lower γ₄ gain an advantage by tending to stay at S₄ longer and produce offspring at a relatively high birth rate b₄. At low densities the γ₄ = γ_max agents eventually discover S₅ sites and their reproduction rate is able to overcome their slow reproductive start. However, at higher densities the γ₄ = γ_max agents are increasingly less able to overcome the other phenotypes’ head start, and are eventually outcompeted completely by the γ₄ = γ_min phenotypes at sufficiently high densities.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Numerical solutions of the coupled mean-field model at different initial densities.

A-D show the evolution of the fraction of the population with each value of γ₄ in the mean-field model; insets at 3.1% and 21.7% initial densities show a zooming in of the initial transient dynamics from iteration 0 to 50. Initially in the evolution, γ₄ = γ_max = 10 agents show a reproductive disadvantage, intuitively because agents with larger γ₄ tend to leave S₄ resource levels (with a relatively high birth rate) more often and produce less offspring, while later in the evolutionary process this disadvantage is overcome by γ₄ = γ_max agents in low densities as they eventually find enough available S₅ sites to overcome their initial disadvantage. However, as density increases, other agents can quickly fill up S₅ sites and leave little opportunities for γ₄ = γ_max agents to overcome that initial disadvantage. Note that b₅ = 5 is used in these numerical solutions so that 〈γ₄(t)〉 converges to 10 at low densities.

https://doi.org/10.1371/journal.pcbi.1009934.g005

Unlike the low-density model, we cannot tractably derive an effective objective function similar to Λ_γ in the high-density regime—in particular, it is not clear whether there exists a function C(γ) of a single phenotype vector γ whose optima predict the most competitive phenotypes. We discuss this issue further in Evolution as an optimization or learning process. Even if no such cost function for the phenotypes exists, one might wonder whether the dynamics for the (expected) population sizes can be written as a dynamical system that follows the gradient of some cost function C(n), . In evolutionary models, the negative of the cost function −C(n) is often interpreted as fitness. However, the vector field defined by the right hand side of Eq (2) is not conservative, and therefore cannot be written as the gradient of an objective function C(n) (see The mean-field dynamics are not gradient-based in Methods). The evolution of the population sizes therefore cannot be interpreted as optimization problem, at least using gradient dynamics.

Summary and interpretation of results

In this work we have used a population dynamics approach to explore the evolution of the innate foraging behaviors of a population of organisms exploring an environment with randomly distributed non-consumable resources. These organisms have limited sensory capabilities and no memory, and must take actions to either continue exploiting the resources at their current location or leave in search of locations with more resources (i.e., more favorable to reproduction). The phenotype of the organisms comprises the rates at which the agents attempt to relocate. While an “intuitive” prediction—the agents should leave low-resource sites as quickly as possible and not move from the highest resources sites—is indeed correct when the population density is low and high resource sites are readily available or have high enough birth rates to compensate their scarcity, this intuitive prediction becomes incorrect when the difficulty of finding high resource sites increases due to intrinsic scarcity or occupation by other organisms in high-density populations. In particular, as the initial population density varies, we not only observe a change in the dominant phenotype at long times, but we also observe long-lived periods of phenotype coexistence.

We have used a combination of stochastic agent-based simulations and a simplified mean-field model to gain insight into the basic phenomenology of the evolutionary dynamics. In the low-density limit we derived an effective fitness function that predicts the dominant phenotype at long times, but could not identify such a function that predicts the dominant phenotypes in higher density environments. Similarly, the resulting dynamics of the average population sizes are not interpretable as gradient-based learning dynamics, as might be obtained from a normative approach in which the organism phenotypes are predicted by optimizing a hypothetical objective function. Thus, there is no guarantee that one can in general derive a function of the phenotype parameters that predicts the optimal phenotype—or distribution of phenotypes—without running the dynamics of the model.

We will discuss some of these points in detail in the following subsections, along with detailed remarks on Comparisons to related work and Limitations of the study, and we close with a brief discussion of Future directions.

Comparisons to related work

Several lines of work have used population dynamics approaches based on master equation models to study the evolution of species under various ecological situations [25, 32], including competition for resources [33–36], prey and predation [37–40], evolutionary arms races [41, 42], and more. Our model follows this methodology, and may be viewed as a generalization of logistic population growth to a spatial environment with multiple possible phenotypes competing for the same limited resources.

Because our model considers only organisms of the same trophic level (i.e., similar organisms compete for a common pool of resources, as opposed to, e.g., predator and prey dynamics), it is similar to the context of the neutral theory of biodiversity [35, 43–45]. The neutral theory of biodiversity assumes that the relative abundances of different species occupying the same environment is not necessarily due to relative evolutionary advantages or disadvantages, but potentially simply due to stochasticity of birth-death dynamics—or “luck.” This idea has been a subject of debate in ecology [35], but neutral models have been successful in explaining some observed trends in experimental data, although a more complete picture of ecological dynamics is thought to require non-neutral effects [33, 43]. While our model population has no intrinsic differences in terms of their raw birth rates, and hence may superficially be considered neutral, different phenotypes do have emergent advantages over one another because some phenotypes are certainly more successful than others at finding high resource sites and producing more offspring, and is therefore effectively not neutral.

Beyond master-equation based models, there are many other studies that take normative approaches to understanding the behaviors and structures of organisms performing similar foraging tasks. Work in this vein may take several forms of abstraction, ranging from models in which the quantities to be optimized are directly relevant quantities—such as expected numbers of offspring [5] or the expected time it takes organisms to find a stockpile of food [6, 7]—to more abstract “rewards” such as the information an organism has extracted from its environment [11, 17, 27, 46, 47]. In some cases these normative models consider just a single representative agent and its associated parameters that determine its dynamics over the course of a specified task, and hence phenotype. More recent work has considered the dynamics of multiple agents performing tasks through the lens of evolutionary game theory [17, 48–51], sometimes taking the limit of a continuous population density, effectively reducing the individual agent-based dynamics to a coarse-grained statistical description [27]. Many similar approaches are being developed in machine learning applications [28]. We note that the dynamics in these normative approaches are typically a single instantiation of a task, such as a single catastrophe period in our model, not a full evolutionary dynamics, though some studies do use evolution-inspired “genetic algorithms” [52], in which the parameters of a population are remixed and inherited by the next generation after a certain amount of simulation time (i.e., there are typically no mechanistic birth-death events as in the model considered in this work). Relatedly, multiple independent lines of work have begun formulating Bayesian approaches to bridge mechanistic and normative approaches to understanding biological structure [6, 9, 10]. As investigation of biological evolution from this multitude of approaches continues to develop, it would be interesting for future studies to directly investigate the alignment of the outcomes of agent-based birth-death dynamics and normative approaches in such multi-agent settings, especially as increases in computational efficiency allow for simulation and investigation of increasingly complex agents.

Limitations of the study

As with any model, there are many aspects of the phenomena we seek to study that cannot be fully captured by our approach or assumptions, and warrant discussion. We focus on discussing how some of the important model simplifications could potentially be relaxed, and consider possible implications for empirical data at the end of this section.

Future directions

In this work we have focused on agents with relatively simple sensory capabilities and behaviors in order to tractably compare agent-based simulations with mean-field approximations and calculate an effective fitness function in a tractable limit of the mean-field model. Two main avenues for future investigations of the models that we have not attempted to elucidate here include i) performing more advanced mathematical analyses of master equation models, for example by investigating the effect of stochastic fluctuations on the stability of phenotype coexistence in more detail, and ii) pushing the boundaries of agent and environment complexity, along with implementing other more realistic features of evolutionary dynamics.

Investigating the effects of stochasticity requires tools to go beyond the mean-field models introduced here. Several tools exist in the mathematical and statistical physics literature for this purpose, such as the Wentzel–Kramers–Brillouin (WKB) approximation [25, 32] or field-theoretic formulations of the master equation model [37–39, 61]. Because these cases explicitly deal with beyond-mean-field effects it would be more natural to use stochastic death in this setting, rather than the periodic catastrophe introduced in this work to control the efficacy of the mean-field approximation. For example, the WKB approximation could allow for a direct calculate of the large deviation functions determining the extinction transition. This large deviation function may serve as an effective cost function of the population, potentially allowing for the derivation of such a function, even in the high-density regime [61].

In terms of increasing organism complexity, a long-term goal of this line of work would be to endow agents with “brains”—small neural networks that take in some sensory information from the environment and then determine how the agent will attempt to move. In such a context evolution would act on the structure of the brain, similar to the case of Braitenberg vehicles [62, 63] (as opposed to determining the synaptic weights in neural network models such as in [64]). In practice, this will be a challenging direction due to the large phenotype space and the separation of timescales involved in sensory processing and evolution. A more tractable approach would be to endow the organisms with a simple “receptive field” that detects sensory information in its immediate environment, down to a modest modification of the current model that allows for agents to detect resource levels at neighboring sites in addition to its current location. Straightforward increases in environmental complexity include adding spatial correlations between resources levels at different locations, as well as introducing sensory signals distributed through the environment that are correlated with resource levels that agents cannot detect directly. Modeling the impact of consumable resources is also of ecological importance, and could be implemented in our models, e.g., by introducing a non-motile “species” at each site that is consumed by agents and replenished at a fixed rate. In this way one could also introduce more realistic metabolic limitations: actions such as giving birth or relocating could incur energy costs that must be replenished by consuming food in the environment. This is one way we could formally remove the artificial restriction of a maximum hopping rate γ_max, allowing the organisms to move as fast as possible between sites if they can acquire the necessary energy to maintain such a strategy.

Finally, one of the goals of this work was to attempt to derive functions that could serve as effective cost functions in normative approaches to modeling the biological structures and behavioral strategies that emerge from the more fundamental evolutionary dynamics. It would be interesting to perform a more thorough comparison of the predictions of birth-death dynamics and normative approaches to identify regimes where the predictions diverge, especially in approaches in which the objective functions are abstract quantities such as information-theoretic quantities related to random variables in an organism’s environment [9–11]. Doing so will provide valuable insight into the relationships between fundamental evolutionary processes and the more parsimonious normative approaches useful for modeling complex organisms.

Details of the agent-based stochastic simulations

In the agent-based stochastic simulation, we use an L × L lattice with periodic boundary conditions, with a standard value of L = 128. The resource level at each location is drawn from a discrete uniform distribution with 5 possible levels, labeled S₁, S₂, S₃, S₄, and S₅. Each agent has a phenotype vector γ = (γ₁, γ₂, γ₃, γ₄, γ₅) where the i^th component corresponds to the hopping or relocation rate when the agent is at a site with resource level S_i. The birth rate of agents at spatial coordinate x within the environment is a function of the resource level at that location, b(s_x) = b(S_i) ≡ b_i if s_x = S_i. Initially, 500 agents are introduced to the system, each of which has a random initial location and random initial phenotype vector with each component ranging from γ_max = 0 to γ_max = 10. This bound is set considering that the relocation rate cannot go negative and there are metabolic costs that constrain agents from relocating too frequently. An offspring inherits its parent’s phenotype vector with Gaussian noise added to each component. The noise term follows , with σ = 0.05 (arbitrary units) in all the stochastic simulations throughout this study. If the noise would push a relocation rate component outside of the range [γ_min, γ_max] then the component is set to the closest boundary value. Each location can hold at most 1 agent at any given time and thus the reproduction process has to leave the offspring at one of the vertical or horizontal neighboring locations in practice—if the agent attempts to give birth into an occupied site then this reproduction attempt is considered as unsuccessful. Similarly, hopping may only occur from an agent’s position to one of the four adjacent sites, with the hop being unsuccessful if the target site is occupied.

Importantly, a system-wide periodic random culling is applied to the system after a duration of T = 0.3 (arbitrary time units), which reduces the number of agents back to its initial state. This periodic catastrophe removes agents in a random manner without distinguishing its phenotype or location, functioning as the selection in the evolutionary dynamics. We expect that the most competitive phenotypes will be selected out after a series of growth and catastrophe events. A full iteration of the simulation consists of stochastic growth and relocation, as well as a random culling at the end. The next iteration starts off with the survived agents from the last iteration as the new initial condition.

As the carrying capacity of each location is set to 1, the 128 × 128 2d lattice can hold at most 16384 agents. For a typical run, in 0.3 units of simulation time the number of agents grows from 500 to around 850 before the random culling brings the number of agents back to 500. The initial density of agents is defined as initial number of agents divided by total possible number of agents in the system, which is 500/16384 ≈ 3.1% for this standard parameter set. When investigating the high density regime, we kept the number of agents to be 500 in all the initial conditions and reduced the lattice size from 128 × 128 to smaller size to achieve higher initial densities.

To reduce the potential variability from stochastic simulations, we run 20 independent simulations with the same set of model parameters and resource background for all the agent-based stochastic simulations in this study. Then the averaged phenotype components cross all trials and their standard deviation were calculated as the solid lines and shaded areas in Figs 2E and 4A–4D.

The stochastic agent-based model was simulated in the Python programming language using a direct method Gillespie algorithm [23]. The Gillespie algorithm computes the total rates of the possible events that may take place in the system to determine when and what the next “action” is. Here “action” refers to either birth or relocation of an agent in the system. The total rate is the sum of the birth and relocation rates of all the agents in the system; the algorithm first determines the time of the next action and then probabilistically chooses one of the available actions based on the relative rate of each action compared to the total rate. That is, at each step, the time of the next birth or relocation event is drawn from an exponential distribution with mean 1/total rate. The probability that the next event is agent A giving birth or relocating is A′s birth rate/total rate or A′s relocation rate/total rate. The target location of the birth or relocation event is then randomly chosen from one of the four possible neighboring locations of the agent’s current position. If the chosen target location is already occupied, the birth or relocation attempt is considered unsuccessful. Note that in this implementation of the algorithm, hopping or birth from site x to site y is a two-step process: the rates going into the total rate are just b(s_x) and γ_x, and then if one of these events is chosen to occur the target site is drawn with probability 1/2d = 1/4 in our two-dimensional environment. However, an alternative way we could have implemented these dynamics is via a one-step process in which we include the birth and hopping rates from site x to site y for all possible target sites as separate rates into the total rate (all of which happen to have the same numerical values). In this second possible implementation the birth and hopping rates must be scaled down by the number of target sites (2d = 4) in order to match the results of the two-step implementation. This technical detail is important when developing the mean-field model, as the one-step implementation corresponds more directly to the master equation formalism we introduce in Master equation model of the stochastic dynamics and derivation of the mean-field approximation, and scaling the birth and hopping rates is required to make the derived mean-field model match the agent-based simulations.

Multiple Python packages were used in the stochastic simulations, mean-field model analyses as well as data visualizations in this study, most notably, Numpy [65], Scipy [66], and Matplotlib [67]. In the stochastic simulations, each stochastic trajectory was run in series on a single CPU core. A computing server with two Intel Xeon Processor E5–2690 v4 CPU (14 cores for each CPU unit) was used to generate 20 stochastic trajectories for the same set of parameters in parallel with the aid of the Python Multiprocessing module. Typically, each run takes approximately 35 minutes for a system starting with 500 agents and b₅ = 4 in Fig 2. Higher b₅ led to more agents being produced, resulting in a higher simulation load, and thus simulations with higher b₅ value take longer to run. For example, the simulations with b₅ = 16 depicted in Fig 3 took over an hour to run.

Master equation model of the stochastic dynamics and derivation of the mean-field approximation

At low-density the most competitive phenotype has the largest maximum eigenvalue.

We now prove that the most competitive phenotype in the reduced mean field model is that whose hopping rates maximize the largest eigenvalue of , denoted Λ_γ.

Our proof begins by using the explicit solution of Eq (9) in terms of the matrix exponential, together with the reset condition Eq (10), to relate the final population sizes just before the next catastrophe by a system of iterative maps: (11) where is the vector of population sizes at the end of the ℓ^th iteration (i.e., just prior to the ℓ^th catastrophe). As ℓ → ∞ this iterative map may converge to a fixed point, , which is the stable distribution of the population in this dynamical system. The fixed point equation may be rewritten (12) for each phenotype γ, and where is the identity matrix. We expect the trivial solution to be obtained for phenotypes that are outcompeted by other phenotypes, such that these phenotypes comprise smaller and smaller fractions of the total population, eventually vanishing as ℓ → ∞.

Because extinction is not possible in this model, at least one phenotype must achieve a non-zero population size at the fixed point. For a non-trivial fixed point to exist, the determinant of must vanish for some phenotypes γ. This implies that is an eigenvalue of the matrices and is the corresponding eigenvector. The eigenvalues and eigenvectors of are related to the eigenvalues λ_γ and eigenvectors of the much simpler , which can be efficiently computed ([69]). As empirically observed in Fig 6, only one of the eigenvalues of is positive; the rest are negative. As the self-consistency condition requires that , the only possible choice of eigenvector is that which corresponds to the largest, positive eigenvalue, which we denote Λ_γ, as otherwise would be less than N₀, which is not possible in this model. (Moreover, the eigenvectors with negative eigenvalues often have negative components, which does not make sense as a population count).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Empirical demonstration that the low-density coefficient matrix has only one positive eigenvalue.

We numerically calculate the eigenvalues of the matrix by randomly drawing all birth and relocation rates from a uniform distribution in [0, 10). Across 50000 samples shown, we find that only the largest eigenvalue of this matrix is positive and all the other eigenvalues are always negative.

https://doi.org/10.1371/journal.pcbi.1009934.g006

Because we assume that all phenotypes are present in equal proportions in the mean field model, it follows that the phenotype with the largest overall maximum eigenvalue Λ_γ will be the sole survivor in the ℓ → ∞ limit. The fact that , a single scalar number, must be an eigenvalue of for every γ implies that the only way multiple phenotypes may coexist in the low-density limit is if their largest eigenvalues Λ_γ have the same value. In such a case the relative proportions of those phenotypes will depend on the overlap of n(0) with the eigenvector corresponding to Λ_γ, which could be different for different phenotypes even if their eigenvalues are the same, leading to different relative proportions in the long time limit.

The mean-field dynamics are not gradient-based

Here, we briefly prove the claim that the mean-field model cannot be written as a gradient-based dynamics for some effective fitness function −C(n). The proof uses the fact that the right-hand side of Eq (2) is not conservative, and hence cannot be written as such a gradient. Let the right-hand-side of Eq (2) be

If this vector field were conservative, then it must be the case that as if Q_kγ = ∂C(n)/∂n_kγ then this quantity would vanish by equality of mixed partial derivatives. We will demonstrate this is not the case. The derivative is

Then, the difference becomes

This quantity is non-zero in general, and hence the vector field Q_kγ is not conservative and cannot be written as a gradient of a cost function that predicts the steady-state configuration of the population. This holds true even in the low-density limit (M ≫ ∑_γ′ n_kγ′), for which the difference reduces to

Numerical solution and analysis of the 5-state mean-field model

With the simplified 5-state mean-field model we can evaluate the evolution of agents by numerically solving the system of differential equations in Eq (8). To do so, we discretize the continuous phenotype space γ = (γ₁, γ₂, γ₃, γ₄, γ₅) with γ_i ∈ [0, 10] into a space where each component can take discrete values from the set {0, 2, 4, 6, 8, 10}. This discretized phenotype space contains 6⁵ = 7776 possible phenotypes.

We numerically solve Eq (9) with the ODE solver from Scipy package [66], using explicit Runge-Kutta method of order 5(4). In practice, we solve the full mean-field model Eq (2) for the fully coupled phenotype system, even in the low density limit. The number of agents is reset following Eq (10) after a duration of simulation time T = 0.3, repeated for 2000 iterations. A typical run takes around 2 hours to finish. Note that in all the numerical evaluations of the mean-field model, the birth rates are taken to be [0, 1, 2, 3, 5] except for Fig 3 where the change of b₅ is explicitly investigated. This slightly different choice of b₅ = 5 in the mean-field model compared to the b₅ = 4 in the stochastic model is to compensate the minor quantitative difference of the critical value of b₅ at which the transition from 〈γ₄〉 = γ_max to γ_min occurs, as shown in Fig 3A, upper panel.

The mean phenotype and its standard deviation can be calculated from the fraction of agents with different phenotypes where N_γ(t) = ∑_i n_iγ(t) is the total number of agents with phenotype γ. Because there are no deaths in our model the numerator will never go to zero, and f_γ is therefore bounded between 0 and 1, so we can treat it as a probability and calculate the mean phenotype by for each component of the vector γ. We similarly estimate the standard deviation by