## Figures

## Abstract

Many fast renewing tissues are characterized by a hierarchical cellular architecture, with tissue specific stem cells at the root of the cellular hierarchy, differentiating into a whole range of specialized cells. There is increasing evidence that tumors are structured in a very similar way, mirroring the hierarchical structure of the host tissue. In some tissues, differentiated cells can also revert to the stem cell phenotype, which increases the risk that mutant cells lead to long lasting clones in the tissue. However, it is unclear under which circumstances de-differentiating cells will invade a tissue. To address this, we developed mathematical models to investigate how de-differentiation is selected as an adaptive mechanism in the context of cellular hierarchies. We derive thresholds for which de-differentiation is expected to emerge, and it is shown that the selection of de-differentiation is a result of the combination of the properties of cellular hierarchy and de-differentiation patterns. Our results suggest that de-differentiation is most likely to be favored provided stem cells having the largest effective self-renewal rate. Moreover, jumpwise de-differentiation provides a wider range of favorable conditions than stepwise de-differentiation. Finally, the effect of de-differentiation on the redistribution of self-renewal and differentiation probabilities also greatly influences the selection for de-differentiation.

## Author summary

How can a tissue such as the blood system or the skin, which constantly produces a huge number of cells, avoids that errors accumulate in the cells over time? Such tissues are typically organized in cellular hierarchies, which induce a directional relation between different stages of cellular differentiation, minimizing the risk of retention of mutations. However, recent evidence also shows that some differentiated cells can de-differentiate into the stem cell phenotype. Why does de-differentiation arise in some tumors, but not in others? We developed a mathematical model to study the growth competition between de-differentiating mutant cell populations and non de-differentiating resident cell population. Our results suggest that the invasion of de-differentiation is jointly influenced by the cellular hierarchy (e.g. number of cell compartments, inherent cell division pattern) and the de-differentiation pattern, i.e. how exactly cells acquire their stem-cell like properties.

**Citation: **Zhou D, Luo Y, Dingli D, Traulsen A (2019) The invasion of de-differentiating cancer cells into hierarchical tissues. PLoS Comput Biol 15(7):
e1007167.
https://doi.org/10.1371/journal.pcbi.1007167

**Editor: **Natalia L. Komarova,
University of California Irvine, UNITED STATES

**Received: **February 25, 2019; **Accepted: **June 7, 2019; **Published: ** July 1, 2019

**Copyright: ** © 2019 Zhou et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability: **All relevant data are within the manuscript and its Supporting Information files.

**Funding: **D.Z. is supported by the China Scholarship Council (No. 201806315038), the Max Planck Institute for Evolutionary Biology, and the Fundamental Research Funding for the Central Universities in China (No. 20720180005). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

**Competing interests: ** The authors have declared that no competing interests exist.

## Introduction

In multicellular organisms, it is important that the inevitable replication errors of cells do not persist and threaten the functioning of the organism as a whole. Many tissues that need to undergo continuous cell turnover are organized in a hierarchical multi-compartment structure, which reduces the risk of the persistence of such mutations [1–13]. Each compartment represents a certain stage of cellular differentiation (Fig 1). At the root of the cellular hierarchy are tissue specific stem cells (SCs), which are capable of self-renewal and differentiation into more mature cells [14]. It is often argued that cancers may have similar hierarchical structures, where cancer stem cells (CSCs) possess characteristics associated with SCs in normal tissues [14, 15]. The CSCs scenario assumes that some cancerous tissues are hierarchically organized, similar to normal tissues [16].

We illustrate our models by considering a four-compartment hierarchical structure. (**a**) Null model without de-differentiation. Each compartment represents a certain stage of cell differentiation. For example, compartment 1 represents stem cell which performs cell division with rate *r*_{1}. In each cell division, it can either give birth to two identical stem cells (self-renewal) with probability *p*_{1} or two identical daughter cells in adjacent downstream compartment 2 (differentiation) with probability *q*_{1}. Similar division pattern can also happen to cells in compartments 2 and 3 (with division rates *r*_{2} and *r*_{3} respectively). Compartment 4 represents terminally differentiated cells which cannot divide and are removed from the tissue at rate *d*. (**b**) Stepwise de-differentiation. Based on the hierarchical structure, we consider de-differentiation from downstream compartment *i* + 1 to the adjacent upstream compartment *i*. By introducing de-differentiation (with probability *δ*_{i}) in cell division, the self-renewal probability of each cell in compartment *i* is changed from *p*_{i} to *p*_{i} − *κδ*_{i}, while its differentiation probability is changed from *q*_{i} to *q*_{i} − (1 − *κ*)*δ*_{i}. Here, we have introduced the redistributing factor *κ* that captures the effect of de-differentiation on the self-renewal and differentiation probabilities. (**c**) Jumpwise de-differentiation, in which de-differentiation happens directly from compartment 3 to 1 without cells reaching the state in compartment 2. For each cell in compartment 3, its self-renewal probability is changed from *p*_{3} to *p*_{3} − *κδ*_{3}, and its differentiation probability is changed from *q*_{3} to *q*_{3} − (1 − *κ*)*δ*_{3}. (**d**) The four cell division patterns used in our models.

The hierarchical tissue architecture proposes a unidirectional cascade from less differentiated stages to more differentiated stages (Fig 1a). This would minimize the risk of the accumulation of genomic damage in the long-term self-renewing stem cells. However, there is significant evidence that the directional relation between different stages of differentiation could be broken in some tissues [17–22]. In other words, cells in later differentiated stages can, under some circumstances, revert to earlier differentiated stages, or even the stem cell stage, in a process called de-differentiation (Fig 1b and 1c). De-differentiation could play an important role in regeneration and tumorigenesis [17]. In particular, even though the origin of cancer stem cells is still an open question, growing evidence shows that non-stem cancer cells can reacquire stem-like characteristics in colon cancer [23], breast cancer [20, 21], melanoma [24], leukemia [25–28], glioblastoma [29], and other cancers. For example, expression of the MLL-AF9 gene in committed hematopoietic progenitor cells led to the development of a leukemic stem cell population where only four of these cells were able to result in disease in a mouse model that could be transferred from one mouse to another, confirming the presence of a stem cell population [27].

More recently, special attention has been paid to the effect of de-differentiation on the cellular hierarchy by mathematically modeling its impact [30]. Previous work has e.g. considered how de-differentiation influences the waiting time to carcinogenesis [31], the fixation probability of a mutant [32, 33], the phenotypic equilibrium [34–36], transient overshoots [37, 38], and radiation sensitivity [29]. However, the adaptive significance of de-differentiation is still poorly understood: Under which circumstances would de-differentiation arise in the first place and rise in abundance? Intuitively, de-differentiation contributes to a faster growth of stem cells, and note that stem cells are typically defined as having the greatest self-renewal potential, hence de-differentiation should benefit the growth of whole population and always be favored in the cellular hierarchy. However, reality seems even more complicated, as de-differentiation arises in only some tumors, but not in others. Therefore, it is still unclear whether de-differentiation is a crucial improvement or just an unintended consequence of cellular hierarchy. Moreover, the comparison between different patterns of de-differentiation has received little attention.

Here, we develop a matrix population model [39] of a stage-structured population for studying the evolution of de-differentiation. Two typical de-differentiation cases are taken into account in our model: One is stepwise de-differentiation which happens from a downstream compartment to an adjacent upstream compartment (Fig 1b), the other is jumpwise de-differentiation which is directly from a highly differentiated compartment into the stem cell compartment without any intermediate stages (Fig 1c). Given a hierarchically structured multi-compartment cell population, we are concerned about the selection of stepwise or jumpwise de-differentiating mutant cell population in the competition with non de-differentiating resident cell population. By comparing the growth rates of different cell populations, we analyze favorable conditions for different de-differentiation patterns to invade a tissue. However, we do not study the direct competition between de-differentiating and non de-differentiating cells. We hope that our work contributes to the theoretical understanding of the emergence of de-differentiation in multicellular tissues.

## Methods

### The matrix population model for cellular hierarchy

Consider a cell population composed of *n* compartments, each of which represents a certain stage of differentiation [10, 13] (Fig 1). For example, compartment 1 represents stem cells, and compartment *n* represents terminally differentiated cells. Each cell in compartment *i* (1 ≤ *i* ≤ *n* − 1) divides at rate *r*_{i}. With probability *p*_{i}, it divides symmetrically, giving birth to two identical cells in compartment *i* (Fig 1d). With probability *q*_{i}, it differentiates symmetrically, generating two identical daughter cells in compartment *i* + 1. The terminally differentiated cells in compartment *n* cannot divide and are removed from the tissue at rate *d*.

We use the vector to denote the cell numbers in different compartments. Then, the hierarchically structured population dynamics composed of non de-differentiating cells can be described as a matrix population model [39]
(1)
where *A*_{0} is the projection matrix which is given by
(2)
Here *r*_{i}(*p*_{i} − *q*_{i}) represents the effective self-renewal rate of compartment *i*, and 2*r*_{i}*q*_{i} represents the influx rate from compartment *i* to compartment *i* + 1 due to differentiation. It should be pointed out that, for simplicity, asymmetric division [40, 41] (giving birth to one daughter cell in compartment *i* and the other in compartment *i* + 1) is not taken into account here. It can be shown that our model is equivalent to a model with asymmetric division [42]. Actually, by introducing asymmetric division (e.g. with probability *s*_{i}) into our model, the effective self-renewal rate of compartment *i* is still given by *r*_{i}(*p*_{i} − *q*_{i}), while the influx rate from compartment *i* to compartment *i* + 1 is shifted from 2*r*_{i}*q*_{i} to 2*r*_{i}*q*_{i} + *r*_{i}*s*_{i}. We can see that the characteristics of matrix *A*_{0}, such as essentially non-negativity (all the off-diagonal elements are non-negative [43]) and lower triangular structure, remain unchanged. Therefore, our approaches and results are still applicable for the model with asymmetric division.

Let be the total cell number of the population. Note that *A*_{0} is an essentially non-negative and lower triangular matrix. According to the standard theory of matrix population models [39], the population approaches exponential growth, i.e.
(3)
where λ_{0} is the real largest eigenvalue. The largest eigenvalue hence characterizes the asymptotic growth rate of the whole population, which is often used as a measure of fitness in matrix population models [44, 45]. The whole population will expand if λ_{0} > 0, remain in homeostasis if λ_{0} = 0, or shrink if λ_{0} < 0. Here, we are interested in the cases when λ_{0} ≥ 0, i.e. we assess whether a mutant can invade an expanding or steady resident population by comparing their fitness measures. Besides, due to the intense inevitable internal and external noise in cellular dynamics [46] and experimental measurements, in reality it is quite unlikely for different compartments to have exactly the same observations of parameters, and therefore there is little chance for *A*_{0} to have multiple eigenvalues [37]. It is thus reasonable to assume that λ_{0} is unique (or simple).

### Stepwise and jumpwise de-differentiation

Let us now introduce de-differentiation processes given the non de-differentiating resident cell population Eq (1). Since it is biologically unclear how a non de-differentiating resident cell acquires the ability for de-differentiation, here we consider de-differentiation as a result of certain genetic or epigenetic alterations (jointly referred to as mutations). It is assumed that the mutant cells are provided with the additional ability of de-differentiation. More specifically, when these mutant cells divide, besides symmetric division and symmetric differentiation, they can also perform symmetric de-differentiation (Fig 1d) with a small probability. In principle, there are two different ways to do this: (i) stepwise de-differentiation, where cells de-differentiate to the previous compartment, and (ii) jumpwise de-differentiation, where de-differentiation happens across multiple compartments at a time. These are the most extreme cases and a mixture between them is possible.

For stepwise de-differentiation, a mutant cell in compartment *i* gives rise to two daughter cells in its adjacent upstream compartment *i* − 1 (Fig 1b) when de-differentiation happens. Suppose that the de-differentiation probability from compartment *i* to *i* − 1 is *δ*_{i}. Then, the influx rate from compartment *i* to *i* − 1 due to de-differentiation is given by 2*r*_{i} *δ*_{i}. We denote the self-renewal and differentiation probabilities of each mutant cell in compartment *i* as and respectively. Note that , that is, the sum of the self-renewal and differentiation probabilities of each mutant cell is reduced from 1 to 1 − *δ*_{i}. Due to the current lack of knowledge regarding the effect of de-differentiation on the self-renewal and differentiation probabilities, there is no way to know how much the self-renewal probability or differentiation probability changes individually. In view of this, we introduce a parameter *κ* (0 ≤ *κ* ≤ 1) to capture how mutant cell redistributes the probabilities for self-renewal and differentiation when taking de-differentiation into account. We thus call *κ* the redistributing factor. In this way, the self-renewal probability of each mutant cell in compartment *i* is given by , and its differentiation probability is given by . Although currently we are unable to measure the specific value of *κ*, it would be very interesting to see if the redistributing factor affects the emergence of de-differentiation, and we will see that *κ* does deserve special attention.

It has been reported that de-differentiation is generally a rare event [21], we thus assume that *ρ*_{i} = 2*r*_{i}*δ*_{i} ≪ 1. As the occurrence of de-differentiation for different stages of differentiation is poorly understood, for simplicity we assume that all the *ρ*_{i} are the same, i.e. they are independent of index *i* and denoted as *ρ* for short. In this way, the population dynamics of the stepwise de-differentiating mutant cell population can be modeled with a projection matrix given by
(4)

Jumpwise de-differentiation provides an alternative pattern where even highly differentiated cells can directly revert to stem cells without being in intermediate stages (Fig 1c). Formally, it is assumed that the jumpwise de-differentiating mutant cell in compartment *n* − 1 can give birth to two daughter stem cells in compartment 1 (Fig 1d). Therefore, the projection matrix is given by
(5)

### Selection gradient for de-differentiation

In the following, we consider the competition between a non de-differentiating resident cell population and a stepwise de-differentiating mutant cell population (which is called *S* mutant cell population for short), as well as between a non de-differentiating resident cell population and a jumpwise de-differentiating mutant cell population (which is called *J* mutant cell population for short) by comparing their fitness measures, i.e. the largest eigenvalues λ_{0}, λ_{S} and λ_{J} of *A*_{0}, *A*_{S} and *A*_{J}, respectively. Note that *ρ* is very small, such that both *A*_{S} and *A*_{J} can be seen as matrix perturbations to *A*_{0}. According to the eigenvalue perturbation theory (see e.g. Theorem 4.4 in [47]), both λ_{S} and λ_{J} are differentiable with respect to *ρ* provided that λ_{0} is simple. In this way, we have
(6)
Here, Δλ_{S} and Δλ_{J} are given by
(7)
where and are the left and right eigenvectors associated with λ_{0} respectively (see S1 File).

For a given parameter set (*r*_{i}, *p*_{i}, *q*_{i}, *d*, *κ*), Δλ_{S} characterizes the selective difference between an *S* mutant cell population and a non de-differentiating cell population. If Δλ_{S} > 0, for example, the *S* mutant population is favored in this competition—a non de-differentiating resident cell population is invaded by an *S* mutant cell population. Therefore, Δλ_{S} corresponds to a selection gradient and acts as a comparative fitness measure of the *S* mutant cell population relative to the non de-differentiating resident cell population. A similar argument also applies for Δλ_{J}. We thus term Δλ_{S} and Δλ_{J} as selection gradients of the *S* mutant cell population and the *J* mutant cell population, respectively. Based on these quantities, we will analyze the favorable conditions for de-differentiation.

## Results

We infer whether de-differentiation leads to an increased fitness in the different cases (stepwise and jumpwise), both analytically and numerically.

Let us first focus on the null model without de-differentiation. In this case, the projection matrix *A*_{0} is a lower triangular matrix whose eigenvalues are just the diagonal elements. Note that the resident cell population in Eq (1) is assumed to be not shrinking, which implies that there exists at least one non-negative diagonal element in *A*_{0}. In this way, the largest eigenvalue λ_{0} is the largest among all the non-negative diagonal elements of *A*_{0}. Note that −*d* is always negative, such that λ_{0} is always in the form of , where *j*_{0} is the compartment that maximizes this quantity.

Next, we turn to stepwise de-differentiation, Eq (4). Given , the selection gradient (comparative fitness) of an *S* mutant cell population is given by (see S1 File for mathematical details)
(8)
where . Note that the largest eigenvalue λ_{0} is unique, which implies that is strictly larger than any other *r*_{j}(*p*_{j} − *q*_{j}) for *j* ≠ *j*_{0}. Thus, all the Γ_{j,k,l} in Eq (8) are positive. In particular, for *j*_{0} = 1, Δλ_{S} = Γ_{1,1,2} is positive. In other words, an *S* mutant cell population is always favored in the competition with non de-differentiating resident cell population provided that stem cells have the largest effective self-renewal rate among all cell compartments. We performed exact numerical solutions to verify our theoretical approximation and find a very good agreement. Fig 2 illustrates two different cases. One is for expanding populations, i.e. when the effective self-renewal rate of stem cells λ_{0} = *r*_{1}(*p*_{1} − *q*_{1}) is positive. The other is for the populations at steady state (homeostasis), i.e. when λ_{0} is zero. We can see that the selection gradient Δλ_{S} is always positive, even though different patterns of function relation are present for left and right panels. That is, the stepwise de-differentiation always provides a fitness advantage, regardless of whether the resident cell populations are expanding or at steady state. Actually, this result is quite in line with biological intuition. Given that stem cells have the highest self-renewal potential, i.e. the self-renewal potential is gradually lost in the process of differentiation, de-differentiation effectively leads to a faster growth rate of the population.

Illustration of the selection gradient (comparative fitness) of the *S* mutant cell population Δλ_{S} as a function of division rates and symmetric division probabilities, provided that the stem cell compartment has the largest effective self-renewal rate, i.e. λ_{0} = *r*_{1}(*p*_{1} − *q*_{1}). In both panels, colored lines represent analytical approximations from Eq (8) by using the eigenvalue perturbation method and symbols represent exact numerical solutions, which agree very well with each other. The common parameters are *n* = 4, *κ* = 0.1, *ρ* = 0.001, *d* = 0.05, *r*_{1} = 0.99, *r*_{3} = 0.3. (**a**) **Expanding case** (λ_{0} > 0). De-differentiation provides a fitness advantage for all values of *p*_{1} and *r*_{2}. Here *p*_{2} = 0.55, *p*_{3} = 0.6 and the range of *p*_{1} (0.55 < *p*_{1} < 1.0) ensures that *r*_{1}(*p*_{1} − *q*_{1}) is the largest eigenvalue. (**b**) **Homeostasis case** (λ_{0} = 0). De-differentiation also provides a fitness advantage for all values of *p*_{2} and *r*_{2}. Here *p*_{1} = 0.5, *p*_{3} = 0 and the range of *p*_{2} (0 < *p*_{1} < 0.3) ensures that λ_{0} = 0 is the largest eigenvalue.

In general, stem cells are defined as having the greatest potential for long term self-renewal. There is also evidence that stem cells replicate slowly and therefore in many tissues it is the progenitor cells that lead to amplification and maintenance of tissues due to a process of replication, self-renewal and differentiation [48, 49]. Previous modeling work has considered different relationships between differentiation stage and self-renewal rate [10, 50, 51], in which downstream compartments rather than the stem cells compartment were often assumed to have the largest effective self-renewal rate. Therefore, it is of significance and interest to consider the case of *j*_{0} > 1 in our model.

From Eq (8) we can see that Δλ_{S} is a linear combination of Γ_{j,k,l} and *κ* when *j*_{0} > 1. It is interesting to see that Δλ_{S} is negatively correlated with *κ*. Note that *κ* is the redistributing factor that characterizes how the introduction of de-differentiation reshapes the probabilities for self-renewal and differentiation. For *κ* = 0, Δλ_{S} is surely positive. With an increase of *κ*, Δλ_{S} could become negative. Hence, there are typically two scenarios of Δλ_{S}: either it is always larger than zero for any *κ*, or it changes from positive to negative at some critical point 0 < *κ** < 1. Fig 3 illustrates how Δλ_{S} changes with *κ* provided that compartment 2 has the largest effective self-renewal rate (*j*_{0} = 2). In the expanding case (left panel) both of these two scenarios are present, whereas in the homeostasis case (right panel) Δλ_{S} is always larger than zero. Actually, when the population is at homeostasis, i.e. λ_{0} = *r*_{2}(*p*_{2} − *q*_{2}) = 0, we can show that Γ_{1,2,1} is larger than 1 and note that Γ_{2,2,3} is always positive, then Δλ_{S} is shown to be positive for any 0 ≤ *κ* ≤ 1. Fig 4 illustrates Δλ_{S} as a function of both *κ* and *p*_{2} in the scenario that Δλ_{S} can change from positive to negative. It is shown that with the increase of *p*_{2}, the critical value *κ** decreases, which means it is getting less likely for the *S* mutant cell population to be favored. Note that Γ_{j,k,l} represents the effect of cellular hierarchy on de-differentiation, and *κ* represents how de-differentiation reshapes the cellular division patterns. Therefore, the selection of de-differentiation is a combined result of cellular hierarchy and de-differentiation pattern.

Illustration of the selection gradient (comparative fitness) of the *S* mutant cell population Δλ_{S} as a function of redistributing factor and division rates provided that λ_{0} = *r*_{2}(*p*_{2} − *q*_{2}). In both panels, colored lines represent the eigenvalue perturbation results from Eq (8) and symbols represent exact numerical solutions. The common parameters are *n* = 4, *ρ* = 0.01, *d* = 0.05. (**a**) **Expanding case** (λ_{0} > 0). In this case, there are two different scenarios: For , Δλ_{S} is always positive (blue color); For *r*_{1} > 0.1950, Δλ_{S} changes from positive to negative with the increase of *κ* (red color). Here *p*_{1} = 0.5, *p*_{2} = 0.95, *p*_{3} = 0.55, *r*_{2} = 0.44, and *r*_{3} = 0.17. (**b**) **Homeostasis case** (λ_{0} = 0). In this case, Δλ_{S} is always positive. Here *p*_{1} = 0.001, *p*_{2} = 0.5, *p*_{3} = 0.001, *r*_{1} = 0.99, and *r*_{3} = 0.8.

The curve represents the boundary with Δλ_{S} = 0, which is generated by the eigenvalue perturbation approximation from Eq (8). The symbols represent the exact numerical solutions for Δλ_{S} = 0. The parameters are *n* = 4, *ρ* = 0.01, *r*_{1} = 0.0885, *r*_{2} = 0.4145, *r*_{3} = 0.5555, *p*_{1} = 0.4723, *p*_{3} = 0.0727, *d* = 0.005.

We now turn our attention to the selection gradient (comparative fitness) of the *J* mutant cell population, which is given by (see S1 File for mathematical details)
(9)
Similar to Eq (8), here all the Γ_{j,k,l} in Eq (9) are positive. For the case of 1 ≤ *j*_{0} < *n* − 1, in particular, Δλ_{J} is always positive, i.e. the *J* mutant cell population is advantageous. Fig 5 illustrates the selection of jumpwise de-differentiation for the cases *j*_{0} = 1 and *j*_{0} = 2. For each case, it is shown that Δλ_{J} is positive, regardless of whether the resident cell populations are expanding or maintaining homeostasis.

Illustrations of the selection gradient (comparative fitness) of the *J* mutant cell population Δλ_{J} for the cases *j*_{0} = 1 and *j*_{0} = 2. In all panels, colored lines represent analytical approximations from Eq (9) by using eigenvalue perturbation and symbols represent exact numerical solutions. The joint parameters *n* = 4, *κ* = 0.1, *ρ* = 0.01, *d* = 0.05. (**a**) Δλ_{J} as a function of *p*_{1} provided an expanding population in which compartment 1 has the largest effective self renewal rate, i.e. λ_{0} = *r*_{1}(*p*_{1} − *q*_{1}) > 0. Here *p*_{2} = 0.55, *p*_{3} = 0.6, *r*_{1} = 0.2, and *r*_{3} = 0.3. (**b**) Δλ_{J} as a function of *p*_{2} provided an expanding population in which compartment 2 has the largest effective self renewal rate, i.e. λ_{0} = *r*_{2}(*p*_{2} − *q*_{2}) > 0. Here, *p*_{1} = 0.55 *p*_{3} = 0.6, *r*_{1} = 0.2, *r*_{3} = 0.3. (**c**) Δλ_{J} as a function of *p*_{2} provided a steady population in which compartment 1 has the largest effective self renewal rate, i.e. λ_{0} = *r*_{1}(*p*_{1} − *q*_{1}) = 0. Here *p*_{1} = 0.5, *p*_{3} = 0.1, *r*_{2} = 0.4, and *r*_{3} = 0.6. (**d**) Δλ_{J} as a function of *p*_{1}, provided a steady population in which compartment 2 has the largest effective self renewal rate, i.e. λ_{0} = *r*_{2}(*p*_{2} − *q*_{2}) = 0. Here, *p*_{2} = 0.5, *p*_{3} = 0.1, *r*_{1} = 0.4, and *r*_{3} = 0.6.

On the other hand, for *j*_{0} = *n* − 1, Δλ_{J} is negatively correlated with the redistributing factor *κ*. Fig 6 illustrates how Δλ_{J} changes with *κ* provided that cells in compartment 3 have the largest effective self-renewal rate (*j*_{0} = *n* − 1 = 3). The results are quite similar to Fig 3. In the expanding case (left panel), either Δλ_{J} is always positive (blue line), or it changes from positive to negative at some critical point 0 < *κ** < 1 (red line). Whereas in the homeostasis case (right panel), Δλ_{J} is always positive for all *κ* ∈ [0, 1]. Actually, when the largest eigenvalue becomes zero, theoretically we can show that Γ_{i,n − 1,i} is larger than 1, and then the product is also larger than 1. In this way, is always positive for any *κ* ∈ [0, 1]. By combining the results from Figs 3 and 6, we know that de-differentiation always provides a fitness advantage in the populations at homeostasis, regardless of how the redistributing factor *κ* affects the self-renewal and differentiation probabilities.

Illustration of the selection gradient Δλ_{J} as a function of the redistributing factor *κ* provided that λ_{0} = *r*_{3}(*p*_{3} − *q*_{3}). In both panels, colored lines represent eigenvalue perturbation results in Eq (9) and symbols represent exact numerical solutions. The common parameters are *n* = 4, *ρ* = 0.01, *d* = 0.05. (**a**) **Expanding case** (λ_{0} > 0). In this case, there are two different scenarios: For , Δλ_{J} is always positive (blue color). For *r*_{1} < 0.45, Δλ_{S} is changed from positive to negative with the increase of *κ* (red color). Here *p*_{1} = 0.5, *p*_{2} = 0.65, *p*_{3} = 0.85, *r*_{2} = 0.4, *r*_{3} = 0.6. (**b**) **Homeostasis case** (λ_{0} = 0). In this case, Δλ_{J} is always positive. Here *p*_{1} = 0.01, *p*_{3} = 0.5, *r*_{1} = 0.8, *r*_{2} = 0.7, and *r*_{3} = 0.2.

A comparison between Eqs (8) and (9) reveals some important differences between stepwise and jumpwise de-differentiation patterns. First of all, jumpwise de-differentiation provides a much wider range of favorable conditions for de-differentiation than stepwise de-differentiation in the sense that Δλ_{J} is always positive for any 1 ≤ *j*_{0} < *n* − 1, but Δλ_{S} is always positive only for *j*_{0} = 1. Secondly, Δλ_{S} only depends on the parameters related to the neighborhood compartments of *j*_{0}, but Δλ_{J} depends on the parameters related to all compartments, ranging from the stem cell stage to the stage where de-differentiation occurs. This implies that, the total number of compartments does matter in the jumpwise case, but not in the stepwise case. In other words, stepwise de-differentiation utilizes the local structure around the compartment with the largest effective self-renewal rate, whereas jumpwise de-differentiation utilizes the global structure throughout the multi-compartment hierarchy.

## Discussion

In this study, we have explored the adaptive significance of de-differentiation in hierarchical multi-compartment structured cell populations. Favorable conditions for de-differentiation have been presented by comparing the fitness measures between resident hierarchical structured cell populations without de-differentiation and mutant cell populations with different modes of de-differentiation.

In principle, there are two main factors that could influence the selection of de-differentiation: cellular hierarchy and the de-differentiation pattern. Cellular hierarchy refers e.g. to the number of cell compartments, the inherent cell division pattern, and the cell division rate. These correspond to the parameter landscape of (*n*, *p*_{i}, *q*_{i}, *r*_{i}) in our model. The de-differentiation pattern refers to different modes of de-differentiation (stepwise or jumpwise), as well as how de-differentiation reshapes the division pattern in the cellular hierarchy (corresponding to *κ* in our model). Interestingly, our results show that the selection gradients for de-differentiation (Δλ_{S} and Δλ_{J}) can generally be decomposed into a sum of a cellular hierarchy part and a de-differentiation part, showing that the selection of de-differentiation is a result of the linear combinations of these two factors.

Among all factors in the cellular hierarchy, the most important one is which of the cell compartments has the largest effective self-renewal rate. In general the stem cells are the cells with the highest potential for long term self-renewal. There is also agreement that stem cells replicate slowly and therefore in many tissues it is the progenitor cells that lead to amplification and maintenance of tissues. There is evidence that cells downstream of the stem cells can undergo self-renewal, albeit not long term or indefinite. In hematopoiesis, for example, erythroid progenitors that are committed to produce red blood cells undergo self-renewal that is regulated by Bm1-1 and PU-1 [52, 53]. Guibal et al have also shown that proerythroblasts in the bone marrow undergo self-renewal [54]. Mutations in cells downstream of the hematopoietic stem cell can transform such cells with long term self-renewal potential behaving like stem cells and able to transfer disease in serial transplantation experiments. Examples of these include AML-ETO expression in primary erythroid cells [55], PML-RARA in acute promyelocytic leukemia [54]. Krivtsov et al [27] also discuss how MLL-AF7 expression in progenitor cells leads to stem cell like behavior. Finally, Jamieson et al have shown how the CML blast crisis emerges from progenitor cells not CML stem cells and leads to self-renewal of such transformed progenitor cells [56].

According to our results, de-differentiation is more likely to be favored when earlier compartments have the largest effective self-renewal rate. For example, in the stepwise case, de-differentiation is favored provided that stem cells have the largest effective self-renewal rate. This result is quite intuitive. Stem cells are normally considered to have the greatest self-renewal potential, and due to de-differentiation the stem cells compartment receives the influx from differentiated cells. In this way, de-differentiation contributes to a faster growth rate of the whole population. In the jumpwise case, de-differentiation is favored in all cases except when the latest divisible cell compartment has the largest effective self-renewal rate. Interestingly, these results apply in both expanding and steady cell populations. For the expanding case, advantageous de-differentiation can speed up the growth rate of the whole population. For the steady case, de-differentiating mutant cell populations with fitness advantage can escape from the homeostasis and expand with time. A significant biological implication of this result is that de-differentiation could play a very important role in tumor initiation [57] during which the balance between self-renewal and differentiation of stem cells could be broken. Furthermore, it has been reported that de-differentiation also happens in normal tissues and contributes to the regenerative processes after injuries [17, 19, 20]. Our results suggest that the presence of de-differentiation could effectively speed up the recovery of tissues. It should be noted that, even though the characteristics of de-differentiation seem similar in both tumorigenesis and regenerative processes, their biological mechanisms should be highly different: The de-differentiation in regenerative processes must be tightly regulated, whereas the de-differentiation in tumorigenesis may be more difficult to control. Note that the differences between them are still poorly understood, it will be very interesting and enlightening to model and compare de-differentiation mechanisms in these two different scenarios.

Given all the factors in the cellular hierarchy, we are most concerned about how different de-differentiation patterns shape the evolution of de-differentiation. In particular the redistributing factor, i.e. the effect of de-differentiation on self-renewal and differentiation probabilities greatly influences the selection conditions. Our results suggest that de-differentiation is more likely to be favored if there is less effect on self-renewal than on differentiation. That is, the smaller the redistributing factor *κ* is, the larger the selection gradient of de-differentiation will be. Furthermore, it should be noted that in the homeostasis cases, the selection gradients for both stepwise and jumpwise de-differentiation are always positive for any *κ* ∈ [0, 1], which suggests that de-differentiation is always advantageous when invading the hierarchical tissues at homeostasis. In addition, the de-differentiation mode (stepwise or jumpwise) has enormous implications for the selection conditions. Our results suggest that de-differentiation is more likely to be favored in the jumpwise case than in the stepwise case. However, jumpwise de-differentiation seems to be biologically much more difficult to achieve, the overall incidence of it would still be very low. Perhaps an example of the differences between stepwise and jumpwise de-differentiation and the implications of the subsequent disease behavior can be illustrated by various types of leukemia. As already mentioned, MLL-AF9 expression in committed progenitor cells can lead to the development of leukemic stem cells that can result in disease transmission across mice [27, 58]. In general MLL expression is associated with a poor prognosis in acute myeloid leukemia [59, 60]. This may be an example of jumpwise de-differentiation. In contrast, acute promyelocytic leukemia (APL) is an example of acute leukemia that is highly curable [61]. It is therefore possible that in this disease, stepwise de-differentiation—or a situation where a mutant cell can stick in a compartment without differentiating, similar to a stem cell—is occurring that in part makes the disease still potentially curable.

Note that the presented study is based on matrix population models with constant elements, which in principle do not take any non-linearity into account. Even though there are still uncertainties regarding the growth patterns of cell populations in different contexts (cancer or normal, solid or hematologic tumor, in vivo or in vitro) [7, 62] and linear models are often considered to be unable to capture the biological processes in reality, they are widely employed as default models to describe steady or growing cell populations, especially in normal tissue at homeostasis and early cancer development [21, 63–66]. We followed this idea and used it as a starting point to explore the adaptive significance of de-differentiation. In the future, more complex biological mechanisms such as non-linear feedback [67, 68] could be taken into account. In pioneering work, Wodarz studied mathematical models by integrating feedback regulation with de-differentiation [33]. He showed that in the presence of non-linear feedback, de-differentiation can lower the rates of tumor initiation and progression. Interestingly, this prediction is opposite to the prediction by Shirayeh et al [32], in which they showed that de-differentiation can increase the rate of tumor initiation in the absence of non-linear feedback. The discrepancy between these two predictions actually reveals the complexities brought by the non-linear feedback which deserves special attention in future study. Moreover, while the hierarchical architecture of tissues is considered to have been selected to minimize the risk of retention of mutations, the risk of acquisition of stem cell like properties by the large population of progenitor cells introduces new dynamics—perhaps in such a scenario two additional considerations could reduce the risk of cancer—namely the low probability that specific mutations lead to acquisition of stem cell like behavior or the average survival of progenitor cells may be low enough to prevent the acquisition of the additional mutations needed to reach the full cancer phenotype. This could be an extension of this work in future.

## Acknowledgments

We would like to thank the Department for Evolutionary Theory at the MPI Plön for feedback.

## References

- 1. Michor F, Nowak MA, Frank SA, Iwasa Y. Stochastic elimination of cancer cells. Proceedings of the Royal Society of London B: Biological Sciences. 2003;270(1528):2017–2024.
- 2. Nowak MA, Michor F, Iwasa Y. The linear process of somatic evolution. Proceedings of the National Academy of Sciences. 2003;100(25):14966–14969.
- 3. Dick JE. Stem cell concepts renew cancer research. Blood. 2008;112(13):4793–4807. pmid:19064739
- 4. Fichelson P, Audibert A, Simon F, Gho M. Cell cycle and cell-fate determination in Drosophila neural cell lineages. Trends in Genetics. 2005;21(7):413–420. pmid:15927300
- 5. Michor F, Hughes TP, Iwasa Y, Branford S, Shah NP, Sawyers CL, et al. Dynamics of chronic myeloid leukaemia. Nature. 2005;435(7046):1267. pmid:15988530
- 6. Dzierzak E, Speck NA. Of lineage and legacy: the development of mammalian hematopoietic stem cells. Nature Immunology. 2008;9(2):129. pmid:18204427
- 7. Johnston MD, Edwards CM, Bodmer WF, Maini PK, Chapman SJ. Mathematical modeling of cell population dynamics in the colonic crypt and in colorectal cancer. Proceedings of the National Academy of Sciences. 2007;104(10):4008–4013.
- 8. Dingli D, Traulsen A, Pacheco JM. Compartmental architecture and dynamics of hematopoiesis. PLoS ONE. 2007;2(4):e345. pmid:17406669
- 9. Takizawa H, Regoes RR, Boddupalli CS, Bonhoeffer S, Manz MG. Dynamic variation in cycling of hematopoietic stem cells in steady state and inflammation. Journal of Experimental Medicine. 2011;208(2):273–284. pmid:21300914
- 10. Werner B, Dingli D, Lenaerts T, Pacheco JM, Traulsen A. Dynamics of mutant cells in hierarchical organized tissues. PLoS Computational Biology. 2011;7(12):e1002290. pmid:22144884
- 11. Rodriguez-Brenes IA, Wodarz D, Komarova NL. Minimizing the risk of cancer: tissue architecture and cellular replication limits. Journal of The Royal Society Interface. 2013;10(86):20130410.
- 12. Alvarado C, Fider NA, Wearing HJ, Komarova NL. Optimizing homeostatic cell renewal in hierarchical tissues. PLoS Computational Biology. 2018;14(2):e1005967. pmid:29447149
- 13. Böttcher MA, Dingli D, Werner B, Traulsen A. Replicative cellular age distributions in compartmentalized tissues. Journal of The Royal Society Interface. 2018;15(145):20180272.
- 14. Reya T, Morrison SJ, Clarke MF, Weissman IL. Stem cells, cancer, and cancer stem cells. Nature. 2001;414(6859):105. pmid:11689955
- 15. Jordan CT, Guzman ML, Noble M. Cancer stem cells. New England Journal of Medicine. 2006;355(12):1253–1261. pmid:16990388
- 16. Altrock PM, Liu LL, Michor F. The mathematics of cancer: integrating quantitative models. Nature Reviews Cancer. 2015;15(12):730. pmid:26597528
- 17. Tata PR, Mou H, Pardo-Saganta A, Zhao R, Prabhu M, Law BM, et al. Dedifferentiation of committed epithelial cells into stem cells in vivo. Nature. 2013;503(7475):218. pmid:24196716
- 18. Easwaran H, Tsai HC, Baylin SB. Cancer epigenetics: tumor heterogeneity, plasticity of stem-like states, and drug resistance. Molecular Cell. 2014;54(5):716–727. pmid:24905005
- 19. Tetteh PW, Farin HF, Clevers H. Plasticity within stem cell hierarchies in mammalian epithelia. Trends in Cell Biology. 2015;25(2):100–108. pmid:25308311
- 20. Chaffer CL, Brueckmann I, Scheel C, Kaestli AJ, Wiggins PA, Rodrigues LO, et al. Normal and neoplastic nonstem cells can spontaneously convert to a stem-like state. Proceedings of the National Academy of Sciences. 2011;108(19):7950–7955.
- 21. Gupta PB, Fillmore CM, Jiang G, Shapira SD, Tao K, Kuperwasser C, et al. Stochastic state transitions give rise to phenotypic equilibrium in populations of cancer cells. Cell. 2011;146(4):633–644. pmid:21854987
- 22. Meacham CE, Morrison SJ. Tumour heterogeneity and cancer cell plasticity. Nature. 2013;501(7467):328. pmid:24048065
- 23. Yang G, Quan Y, Wang W, Fu Q, Wu J, Mei T, et al. Dynamic equilibrium between cancer stem cells and non-stem cancer cells in human SW620 and MCF-7 cancer cell populations. British Journal of Cancer. 2012;106(9):1512. pmid:22472879
- 24. Quintana E, Shackleton M, Foster HR, Fullen DR, Sabel MS, Johnson TM, et al. Phenotypic heterogeneity among tumorigenic melanoma cells from patients that is reversible and not hierarchically organized. Cancer cell. 2010;18(5):510–523. pmid:21075313
- 25. Dorantes-Acosta E, Pelayo R. Lineage switching in acute leukemias: a consequence of stem cell plasticity? Bone Marrow Research. 2012;2012. pmid:22852088
- 26. Passegué E, Weisman IL. Leukemic stem cells: where do they come from? Stem Cell Reviews. 2005;1(3):181–188. pmid:17142854
- 27. Krivtsov AV, Twomey D, Feng Z, Stubbs MC, Wang Y, Faber J, et al. Transformation from committed progenitor to leukaemia stem cell initiated by MLL—AF9. Nature. 2006;442(7104):818. pmid:16862118
- 28. Haeno H, Levine RL, Gilliland DG, Michor F. A progenitor cell origin of myeloid malignancies. Proceedings of the National Academy of Sciences. 2009;106(39):16616–16621.
- 29. Leder K, Pitter K, LaPlant Q, Hambardzumyan D, Ross BD, Chan TA, et al. Mathematical modeling of PDGF-driven glioblastoma reveals optimized radiation dosing schedules. Cell. 2014;156(3):603–616. pmid:24485463
- 30. Jilkine A. Mathematical Models of Stem Cell Differentiation and Dedifferentiation. Current Stem Cell Reports. 2019;5(2): 66–72.
- 31. Jilkine A, Gutenkunst RN. Effect of dedifferentiation on time to mutation acquisition in stem cell-driven cancers. PLoS Computational Biology. 2014;10(3):e1003481. pmid:24603301
- 32. Mahdipour-Shirayeh A, Kaveh K, Kohandel M, Sivaloganathan S. Phenotypic heterogeneity in modeling cancer evolution. PLoS ONE. 2017;12(10):e0187000. pmid:29084232
- 33. Wodarz D. Effect of cellular de-differentiation on the dynamics and evolution of tissue and tumor cells in mathematical models with feedback regulation. Journal of Theoretical Biology. 2018;448:86–93. pmid:29605227
- 34. dos Santos RV, da Silva LM. A possible explanation for the variable frequencies of cancer stem cells in tumors. PLoS ONE. 2013;8(8):e69131. pmid:23950884
- 35. Niu Y, Wang Y, Zhou D. The phenotypic equilibrium of cancer cells: From average-level stability to path-wise convergence. Journal of Theoretical Biology. 2015;386:7–17. pmid:26365152
- 36. Zhou JX, Pisco AO, Qian H, Huang S. Nonequilibrium population dynamics of phenotype conversion of cancer cells. PLoS ONE. 2014;9(12):e110714. pmid:25438251
- 37. Zhou D, Wang Y, Wu B. A multi-phenotypic cancer model with cell plasticity. Journal of Theoretical Biology. 2014;357:35–45. pmid:24819463
- 38. Chen X, Wang Y, Feng T, Yi M, Zhang X, Zhou D. The overshoot and phenotypic equilibrium in characterizing cancer dynamics of reversible phenotypic plasticity. Journal of Theoretical Biology. 2016;390:40–49. pmid:26626088
- 39.
Caswell H. Matrix Population Models. John Wiley & Sons, Ltd; 2006.
- 40. Dingli D, Traulsen A, Michor F. (A) symmetric stem cell replication and cancer. PLoS Computational Biology. 2007;3(3):e53. pmid:17367205
- 41. Hu Z, Fu YX, Greenberg AJ, Wu CI, Zhai W. Age-dependent transition from cell-level to population-level control in murine intestinal homeostasis revealed by coalescence analysis. PLoS Genetics. 2013;9(2):e1003326. pmid:23468655
- 42. Hillen T, Enderling H, Hahnfeldt P. The tumor growth paradox and immune system-mediated selection for cancer stem cells. Bulletin of Mathematical Biology. 2013;75(1):161–184. pmid:23196354
- 43. Cohen JE. Convexity of the dominant eigenvalue of an essentially nonnegative matrix. Proceedings of the American Mathematical Society. 1981;81(4):657–658.
- 44. Metz JA, Nisbet RM, Geritz SA. How should we define ‘fitness’ for general ecological scenarios? Trends in Ecology & Evolution. 1992;7(6):198–202.
- 45. Pichugin Y, Peña J, Rainey PB, Traulsen A. Fragmentation modes and the evolution of life cycles. PLoS Computational Biology. 2017;13(11):e1005860. pmid:29166656
- 46. Paulsson J. Models of stochastic gene expression. Physics of Life Reviews. 2005;2(2):157–175.
- 47.
Demmel JW. Applied numerical linear algebra. vol. 56. SIAM; 1997.
- 48. Fuchs E, Chen T. A matter of life and death: self-renewal in stem cells. EMBO Reports. 2013;14(1):39–48. pmid:23229591
- 49. Visvader JE, Clevers H. Tissue-specific designs of stem cell hierarchies. Nature Cell Biology. 2016;18(4):349. pmid:26999737
- 50. Marciniak-Czochra A, Stiehl T, Ho AD, Jäger W, Wagner W. Modeling of asymmetric cell division in hematopoietic stem cells-regulation of self-renewal is essential for efficient repopulation. Stem Cells and Development. 2009;18(3):377–386. pmid:18752377
- 51. Werner B, Dingli D, Traulsen A. A deterministic model for the occurrence and dynamics of multiple mutations in hierarchically organized tissues. Journal of The Royal Society Interface. 2013;10(85):20130349.
- 52. Kim AR, Olsen JL, England SJ, Huang YS, Fegan KH, Delgadillo LF, et al. Bmi-1 regulates extensive erythroid self-renewal. Stem Cell Reports. 2015;4(6):995–1003. pmid:26028528
- 53. Back J, Dierich A, Bronn C, Kastner P, Chan S. PU. 1 determines the self-renewal capacity of erythroid progenitor cells. Blood. 2004;103(10):3615–3623. pmid:14739214
- 54. Guibal FC, Alberich-Jorda M, Hirai H, Ebralidze A, Levantini E, Di Ruscio A, et al. Identification of a myeloid committed progenitor as the cancer-initiating cell in acute promyelocytic leukemia. Blood. 2009;114(27):5415–5425. pmid:19797526
- 55. Tonks A, Pearn L, Tonks AJ, Pearce L, Hoy T, Phillips S, et al. The AML1-ETO fusion gene promotes extensive self-renewal of human primary erythroid cells. Blood. 2003;101(2):624–632. pmid:12393523
- 56. Jamieson CH, Ailles LE, Dylla SJ, Muijtjens M, Jones C, Zehnder JL, et al. Granulocyte—macrophage progenitors as candidate leukemic stem cells in blast-crisis CML. New England Journal of Medicine. 2004;351(7):657–667. pmid:15306667
- 57. Schwitalla S, Fingerle AA, Cammareri P, Nebelsiek T, Göktuna SI, Ziegler PK, et al. Intestinal tumorigenesis initiated by dedifferentiation and acquisition of stem-cell-like properties. Cell. 2013;152(1-2):25–38. pmid:23273993
- 58. Dong F, Bai H, Wang X, Zhang S, Wang Z, Xie M, et al. Mouse acute leukemia develops independent of self-renewal and differentiation potentials in hematopoietic stem and progenitor cells. Blood Advances. 2019;3(3):419–431. pmid:30733302
- 59. Stavropoulou V, Kaspar S, Brault L, Sanders MA, Juge S, Morettini S, et al. MLL-AF9 expression in hematopoietic stem cells drives a highly invasive AML expressing EMT-related genes linked to poor outcome. Cancer Cell. 2016;30(1):43–58. pmid:27344946
- 60. Scholl C, Schlenk RF, Eiwen K, Dohner H, Frohling S, Dohner K, et al. The prognostic value of MLL-AF9 detection in patients with t (9; 11)(p22; q23)-positive acute myeloid leukemia. Haematologica. 2005;90(12):1626–1634. pmid:16330435
- 61. Werner B, Gallagher RE, Paietta EM, Litzow MR, Tallman MS, Wiernik PH, et al. Dynamics of leukemia stem-like cell extinction in acute promyelocytic leukemia. Cancer Research. 2014;74(19):5386–5396 pmid:25082816
- 62. Gerlee P. The model muddle: in search of tumor growth laws. Cancer Research. 2013;73(8):2407–2411 pmid:23393201
- 63. Rodriguez-Brenes IA, Komarova NL, Wodarz D. Tumor growth dynamics: insights into evolutionary processes. Trends in Ecology & Evolution. 2013;28(10):597–604.
- 64. Weekes SL, Barker B, Bober S, Cisneros K, Cline J, Thompson A, et al. A multicompartment mathematical model of cancer stem cell-driven tumor growth dynamics. Bulletin of Mathematical Biology. 2014;76(7):1762–1782. pmid:24840956
- 65. Williams MJ, Werner B, Barnes CP, Graham TA, Sottoriva A. Identification of neutral tumor evolution across cancer types. Nature Genetics. 2016;48(3):238. pmid:26780609
- 66. Talkington A, Durrett R. Estimating tumor growth rates in vivo. Bulletin of Mathematical Biology. 2015;77(10):1934–1954. pmid:26481497
- 67. Lander AD, Gokoffski KK, Wan FY, Nie Q, Calof AL. Cell lineages and the logic of proliferative control. PLoS Biology. 2009;7(1):e1000015.
- 68. Stiehl T, Marciniak-Czochra A. Stem cell self-renewal in regeneration and cancer: insights from mathematical modeling. Current Opinion in Systems Biology. 2017;5:112–120.