Gene regulatory network models

We demonstrate the rare-event sampling method for two representative GRN models. A small, two-gene network serves as a model system to validate the simulations. We then apply the method to a more complex model of pluripotency in mouse Embryonic Stem Cells (mESCs).

Theoretical background: The chemical master equation and stochastic transition-matrix

The mathematical framework of the network models is the discrete Chemical Master Equation (CME) [33], which gives the time-evolution of the probability to observe the system in a given state. In vector-matrix form, the CME can be written (1) where p(x, t) is the probability over the system state-space (x) at time t, and K is the reaction rate-matrix containing stochastic reaction propensities (diagonal elements k_jj = −∑_i k_ij, i.e., columns sum to 0). Eq 1 assumes a well-mixed system of reacting species, and assumes that the technically infinite state-space described by x (containing molecular species numbers/configurations) may be limited to some finite number of “reachable” states, (i.e., with non-negligible probability) for an enumeration of N states of the system, . The steady-state probability π(x) ≡ p(x, t → ∞) over N states satisfies (2) Thus, π(x) can be obtained from K as the normalized right-eigenvector corresponding to the zero-eigenvalue.

It is sometimes desirable to work with the time-dependent stochastic transition-matrix T(τ) rather than the time-independent stochastic rate matrix K [42]. For example, T(τ) may be more amenable to estimation by sampling (as we demonstrate in this work for the pluripotency network, for which K is impractical to enumerate). For a CME with rate matrix K, T(τ) is given by (3) where exp denotes the matrix exponential. then gives the conditional probability for the system to transition between each pair of states within a lagtime τ. That is, the elements T_ij give the probability that the system, if found in state i, will then be found in state j at a time τ later, and rows sum to 1. Using T(τ), the evolution of probability over discrete intervals of the lagtime τ is given by the Chapman-Kolmogorov equation: (4) Eigenvectors corresponding to dominant eigenvalues of the stochastic transition-matrix are associated with slow system processes. By Perron-Frobenius, for an irreducible stochastic matrix T(τ) with eigenvalues λ_i, there exists λ₁ = 1, and all other eigenvalues satisfy |λ_i| < 1. Analogous to Eq (2) for K, the steady-state probability can be obtained directly from T(τ) according to π^T(x) = π^T(x)T(τ), i.e., as the normalized left-eigenvector corresponding to λ₁. Eigenvalues λ_i are related to global system timescales t_i by (5) (with t₁ giving the infinite-time, stationary result) [42]. Additionally, the Mean First Passage Time for transitions from an individual state i to a region Y (MFPT_i,Y, where Y may be an individual state or a set of states) can be computed using the matrix elements T_i,j by [43, 44]: (6) MFPT_i,Y is defined as the expected time for the system to reach Y for the first time, having started in state i. The MFPTs may be computed by solving the linear system in Eq 6. Eq 6 computes the MFPT as a dimensionless quantity, the expected number of “steps” (of duration τ) required for the transition; multiplication by τ gives the MFPT in units of time. The MFPT starting from a region X (i.e., a set of states, rather than an individual state) and ending in a region Y is given by the stationary-probability-weighted sum: (7)

Weighted ensemble stochastic simulation

Stochastic reaction kinetics can be simulated by the Stochastic Simulation Algorithm (SSA) [33], which produces numerically exact realizations of the CME (Eq 1). Simulation circumvents the need for enumerating the exceedingly large system state-spaces typical of gene network models, but suffers from inefficiency due to rare events. The Weighted Ensemble (WE) rare-event sampling algorithm [45] redistributes computational resources from high-probability regions of state-space to low-probability regions, which tend to be under-sampled in conventional simulation. The method thereby reduces computational effort in sampling rare transitions and improves accuracy of estimating probability density in, e.g., barrier-regions or tails of distributions. The method can be applied to any stochastic dynamics framework; in recent years, it has been widely applied to atom-scale Molecular Dynamics. Details of the methodology are discussed in a recent review [35] and references therein. Both WE and a related method, Forward Flux Sampling, have been applied previously to the study of 2-gene networks [46, 47].

Briefly, the algorithm works as follows: state-space is divided up into bins that span transitions of interest. The number of bins, N_bins, is typically , and a variety of binning procedures can be used (we use an adaptive procedure described below). Initially, a single simulation trajectory, or “replica”, is assigned a weight of 1 and allowed to freely move within and between bins for a user-defined lagtime τ_WE. After each iteration of τ_WE, a splitting and culling procedure divides and/or combines replicas and their associated weights in such a way as to reach and maintain an equal target number of weighted replicas, M_targ, in each bin. Over the course of the simulation, the combined weights of the replicas in a bin (averaged over successive iterations) will evolve toward the probability of the system to reside in that bin. By maintaining the same number of replicas in each bin (M_targ), with weights proportional to probability, the algorithm devotes comparable computational time to low- and high-probability regions. Effectively, the algorithm computes long-time processes on the basis of many short-time simulated trajectories.

Coarse-graining procedure to classify phenotype-states

While the sampled N_bins × N_bins transition-matrix provides a global approximation of the epigenetic landscape and state-transitions, we apply a method to further coarse-grain dynamics, known as the Markov State Model framework [29, 36, 42]. This automated procedure produces a highly simplified representation of global dynamics in terms of a few (generally < 10) clustered sets and the transitions among them. Such highly-reduced models can be beneficial in terms of human intuition of system dynamics, comparison to experiments, and—in this application—automated designation of dynamic phenotype-states. The method utilizes the concept of metastability, i.e., system states that experience relatively fast transitions among them are clustered together into the same coarse-grained set. Collectively, the coarse sets experience relatively rare inter-cluster transitions and frequent intra-cluster transitions. We employ the metastability concept as a definition of cell phenotype, reasoning that a phenotype should be a relatively stable attribute of a cell, and stochastic inter-phenotype transitions should be relatively rare. In practice, we employ the Markov State Model framework to further reduce the sampled row-stochastic transition-matrix from size N_bins × N_bins down to C × C, where C is the number of coarse-grained clusters chosen by the user. As the Markov State Model (MSM) is itself a stochastic transition-matrix on a coarse-grained space, it implies a more severe Markovian approximation. It provides a way to describe global system dynamics in a highly simplified way while maintaining high accuracy to the slowest system dynamics as sampled by . In previous work, we demonstrated the application of this coarse-graining approach to automatically designate phenotypes in small gene networks [29]; here, we extend the applicability of the coarse-graining to large, complex networks by combining it with rare-event sampling.

The coarse-graining procedure is a spectral clustering method based on the Perron Cluster Cluster Analysis (PCCA+) algorithm [53], which optimizes the (nearly)-block-diagonal structure of for systems with metastability. The signature of such metastability is a separation-of-timescales for intra- and inter-basin dynamics, which may be seen as gaps in the eigenvalue spectrum [42]. As noted above, T(τ) (or its sampled counterpart, ) has λ₁ = 1, corresponding to the infinite time-limit. If a set of m dominant eigenvalues exists, such that for decreasing eigenvalues λ_i ⪅ 1, i ∈ {2,…, m}, and a gap is present, λ_j ≪ λ_m for j > m, this indicates the presence of m slow-timescale processes in the system, and further indicates that may be re-ordered to give m nearly-uncoupled blocks. In practice, the algorithm attempts to find a coarse-graining onto C clusters, where C may be user-defined, or may be determined algorithmically, e.g., according to the spectral gap [53]. Here, we choose C clusters, where the last significant gap in the spectrum is seen between λ_C and λ_C+1. For the GRNs studied here, this corresponds to choosing C such that λ_C/λ_C+1 > 10.

Visualization of epigenetic landscapes

Both the sampled transition-matrix and the coarse-grained MSM encode stationary and dynamic information about global dynamics—that is, they quantify the epigenetic landscape. For visualization, we use Gephi graph visualization software [54] using the Force Atlas algorithm. Every circle (or node) in the graph corresponds to a sampling bin or to a coarse-grained phenotype, and the area of a circle is proportional to its relative steady state probability according to ln(γP_SS), where P_SS is the steady state probability of the node and γ is a constant chosen to improve visibility of low probability regions of the landscape. Lines between circles (edges) correspond to transitions between sampling regions or coarse-grained phenotype. Their thickness and coloring correspond to their relative transition probability and source state, respectively.

Validation: Numerical solution of the chemical master equation

To validate the simulation method, we compare the simulated dynamics to the numerical solution to the CME. We choose the parameters of the ExMISA model in such a way as to restrict the effective state-space, so that a numerical solution of the CME is tractable. Building the reaction rate matrix requires enumeration of N system states. In general, if a system of S molecular species has a maximum copy number per species of n_max, then . In the ExMISA model, the state-vector is given by x = [A_ij, B_ij, n_a, n_b]. For enumeration, we neglect states with protein copy-numbers larger than a cutoff value which exceeds g₁₀/k (corresponding to the average number of transcription factors maintained in the system from a gene while in its active state). For example, with model parameters g₁₀ = 18 and k = 1, we truncate at n_a,max = n_b,max = 41 and assume that probability flux between states with n_a, n_b ≤ 41 and states with n_a, n_b > 41 is assumed to be 0 (i.e., the boundaries of the state-space are reflective). Including the gene-binding states, this gives N = 3 × 3 × 42 × 42 = 15876 states. This size is tractable for complete solution of the CME using matrix methods in MATLAB [55]. This truncation of the state-space introduces a small approximation error (see S2 Fig).

The pluripotency network has 8 genes with copy numbers of (determined by the parameters g_on/k = 3900). The number of distinct binding-promoter states for each gene are 16, 32, 8, 8, 2, 8, 4, and 2 for GATA6, NANOG, CDX2, OCT4, SOX2, KLF4, GCNF, PBX1, respectively (see S2 Table). Together these combinations enumerate a state-space of N > 10³⁰ ≈ 1000⁸ × 16 × 32 × 8 × 8 × 2 × 8 × 4 × 2. This size precludes solution of the CME, and we instead estimate the dynamics by WE sampling. Where possible, we validate the WE-sampling results by “conventional”, i.e., by direct simulation using SSA.

Implementation and software

Stochastic Gillespie (SSA) simulations were carried out using BioNetGen [56]. WE sampling was implemented with in-house software code written in MATLAB. Simulations were run on the high performance computing cluster (HPC) at the University of California, Irvine, and parallelization of BioNetGen SSA simulations was performed using the Sun Grid Engine scheduler. The coarse-graining procedure and transition path analysis was implemented in python scripts, adapted from MSMBuilder [57] and Pyemma [43], respectively. Transition-matrix and MSM visualization was carried out using Gephi software and the Force Atlas layout [54]. All simulation parameters can be found in the supplement S4 Table. Pseudo-code for the adaptive binning procedure can be found in S2 File and software can be found in https://github.com/Read-Lab-UCI/Rare-Event-Sampling-Gene-Networks.

Rare states and transitions in gene regulatory networks are accessible by rare-event sampling

We first apply the computational pipeline to a small two-gene model (the exclusive Mutual Inhibition, Self-Activation model, ExMISA, see Methods), exhibiting an archetypal motif for cell fate-decisions [37, 38]. The model is tractable for computation of full, discrete stochastic dynamics to within a small approximation error using matrix methods. Thus, the model provides a numerical benchmark for assessing the accuracy of the simulation method, before extension to larger systems where solution of the Chemical Master Equation (CME) is intractable. For the chosen parameters, the ExMISA model shows four peaks in the steady-state probability distribution (projected onto protein copy numbers, n_a and n_b). Peaks in probability correspond to basins in the so-called quasipotential landscape, defined by U = −ln(π(x)) (Fig 2). The four peaks/basins corresponds to four possible combinations of binarized A/B gene expression: hi/hi, hi/lo, lo/hi, and lo/lo. These four phenotype-states arise due to the combination of balanced repression and self-activation in the network, and the slow kinetic parameters (Supplementary S1 Table) for transcription factor binding and unbinding to promoters that effect changes in individual gene-activity states between low and high expression rates [29, 58].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Simulation results show good agreement with a theoretical benchmark for the 2-gene ExMISA (mutual inhibition, self-activation) cell-decision circuit.

The Chemical Master Equation for the 2-gene model, ExMISA, was solved numerically (see Methods) (top) and compared to simulation results from the computational pipeline presented in this paper (bottom). Shown for each are the Quasipotential Landscape (A), Eigenvalue Spectrum (B), and Markov State Model (C). (A) Quasipotential landscapes of the ExMISA network projected onto the two protein coordinates. Deep blue regions denote low potential (high probability) and yellow denote high potential (low probability). The four visible basins in both correspond to combinations of lo/hi expression for the two genes A and B. (For both rows, quasipotential surfaces estimated over discrete states/bins are smoothed for visualization). WE sampling captured both the basin structure and low probability edge and barrier regions. (B) Eigenvalue spectra and corresponding computed global transition timescales. Gaps in the eigenvalue spectrum indicate separation of timescales, i.e., the presence of metastability. C) Four-phenotype coarse-grained models automatically generated from the clustering algorithm (see Methods). Each colored circle represents a cell phenotype, sized proportionally to its probability. Edges are inter-phenotype transitions (colored by source-state, with width proportional to probability). The full CME and simulation pipeline identify similar metastable phenotype networks (see S11 Fig for details).

https://doi.org/10.1371/journal.pcbi.1006336.g002

The WE-based simulation method enabled estimation of global dynamics of the ExMISA model. By redistributing computational resources from relatively high-probability to low-probability regions (see Methods), the WE method enabled uniform sampling of the quasipotential landscape, i.e., mapping basins (high-probability regions) along with high barriers (low probability regions) (Fig 2a). The simulation estimated individual steady-state bin-probabilities as low as 1.3 × 10⁻⁶ and showed good global agreement with the numerical CME benchmark (see Fig 2 and Supplement, S3 Fig).

In addition to sampling global dynamics, the WE method can be used to estimate rate constants for individual, rare transitions of interest. The Mean First Passage Time of the global network switch from the center of one polarized phenotype-state to another, i.e., MFPT_X→Y from protein a/b expression level hi/lo to lo/hi was estimated from WE to be 1.82 × 10⁵ (units of k⁻¹) (see S6 Table), in agreement with the CME result.

Phenotype transitions can be approximated by Markovian jumps, enabling construction of coarse-grained models

A network transition-matrix over sampled bins (N_bins = 300) was constructed from WE sampling for ExMISA and used for subsequent analysis of global system dynamics. By comparison, a full network transition-matrix T(τ) over the enumerated system state-space was constructed from the CME (N = 15876, see Methods). The full, computed (T(τ)) and simulated () transition-matrices showed qualitatively similar eigenvalue spectra with four dominant eigenvalues, indicating the presence of metastability (separation-of-timescales between intra-basin and inter-basin transitions) (Fig 2b). The slow system-timescales predicted by the full CME model corresponding to eigenvalues λ₂, λ₃, λ₄ were t₂, t₃, t₄ = 6.8 × 10⁴, 4.2 × 10⁴, 1.0 × 10⁴ respectively, in units of k⁻¹ where k is the protein degradation rate (the Perron eigenvalue λ₁ = 1 is associated with the infinite-time (stationary) distribution). The corresponding values given by the WE-simulated were 6.1 × 10⁴, 3.5 × 10⁴, 9.4 × 10³, respectively. These numbers demonstrate how the sampled enables global approximation of slow system timescales to < 20% relative error. Error in these values (relative to the slowest timescales implied by the true eigenvalues) depends on both “spectral” (lagtime) and discretization error, i.e., improvements can be achieved only with a larger number of bins (finer discretization) and/or longer lagtime [42] (see S4 Fig). In contrast, WE sampling in “rate mode” (see Methods) enabled highly accurate estimation of MFPT_X→Y to within 2% error (S6 Table).

According to the Markov State Model framework, the presence of timescale separation indicates that a simplified model, retaining a few coarse-grained metastable states with Markovian transitions among them, can reasonably approximate the full system dynamics. Using this approach, we label the metastable sets as phenotypes accessible to the network, reasoning that a useful classification of cell phenotypes should be one that gives relatively stable, rather than transient, cell types. We apply the Markov State Model coarse-graining procedure to both the full T(τ) and simulated , yielding similar results. The coarse sets (or metastable phenotype-states) in the reduced models for both cases are generated automatically, and map directly onto the four basins seen in the quasipotential landscape (i.e., the gene A/B expression hi/hi, hi/lo, lo/hi, and lo/lo cell phenotypes). The reduced models are visualized by network graphs, in which node sizes are proportional to steady-state probability, and the thicknesses and lengths of edges are proportional to the transition probability between them (on lagtime τ) (Fig 2c). Some discrepancies can be seen visually in the network graphs. These discrepancies likely result in part from the slightly different mappings of the full state-space onto the four clusters (see S11 Fig for details), which could in turn result from the distance-metric-based binning, which is relatively insensitive to changes in promoter configuration. Numerical values for the reduced models can be found in S5 Table. The network graph can be considered to be an alternative representation of the global epigenetic landscape, which contains both stationary and dynamic information. (In contrast, the epigenetic landscape plotted as a quasipotential function does not explicitly contain dynamic information, due to non-gradient dynamics [16]).

Validation of the coarse-grained model can be carried out according to the Chapman-Kolmogorov test [42], which tests how well the relaxation dynamics initialized in the metastable phenotypes approximate the dynamics that are predicted either by the full model (CME) or simulated trajectories. According to this test, relaxation dynamics out of metastable phenotypes from WE sampling was predicted with relative error values between 0.02 and 0.12 for all phenotypes (S5 Fig). Together, these results indicate (i) that a Markovian model of phenotype transitions is a good approximation of the full system dynamics for the ExMISA model, and (ii) that the WE-simulation based computational pipeline predicts a quantitatively similar coarse-grained phenotype-network to the full CME model.

The method maps the epigenetic landscape and identifies dominant phenotypes in a pluripotency network model

We apply the computational pipeline to a pluripotent fate-decision network from mouse Embryonic Stem Cells (mESCs) introduced by Zhang et al. [28] (Fig 3A). The network comprises eight interacting genes: NANOG, GATA6, CDX2, SOX2, OCT4, GCNF, and PBX1. Three of these genes, NANOG, SOX2, and OCT4 have been suggested to maintain pluripotency [59], and NANOG inhibits the expression of differentiation markers [60]. The GATA6 and CDX2 genes have been used in experiments as markers of differentiation, with the GATA6 transcription factor being a marker of the primitive endoderm cell lineage, and the CDX2 transcription factor being a marker of the trophectoderm lineage [61].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Pluripotency network model and simulation results (Parameter Set I).

A)Wiring diagram for the eight-gene pluripotency network model, adapted from [28]. Arrowheads represent positive interactions, while flat lines denote repression. B) Simulation results: state-transition graph of sampled network states. Circles represent aggregate gene-expression states sampled during the Weighted Ensemble simulation. Circle areas are proportional to the steady-state probability π_i in each state according to ln(γπ_i) with scaling factor γ = 3.4. States are colored according to the gene expression levels of three of the genes; red, green, and blue correspond to high NANOG, GATA6, and CDX2 expression respectively, while black corresponds to low or no gene expression. Edges connecting the states indicate possible state-transitions, colored according to the originating state. The graph is produced using Gephi [54] using a force-directed layout algorithm (Force Atlas), therefore short inter-state distances reflect higher probability of transitioning. C) Full protein compositions of two representative states, with either high CDX2 expression (blue) or high NANOG expression (red). States in (C) correspond to yellow circles in (B).

https://doi.org/10.1371/journal.pcbi.1006336.g003

Using the WE-based computational pipeline, we estimate with a resolution of N_bins = 250. To visualize the global landscape as a graph network at this resolution, we plot the converged using a force-directed automated graph layout [54] (Fig 3B). The barbell shape of the network reflects the broad antagonism between pluripotency and differentiation genes, which is a general feature of the overall network topology. At the same time, each “pole” comprises multiple distinct patterns of gene expression (seen in the graph as different colors with full compositions in Fig 3C), hinting at the existence of multiple phenotypes associated with both pluripotency and lineage-specification. Moreover, the network representation reveals numerous links between pluripotent and differentiated states, pointing to both direct and indirect transitions, through a network of relatively transient intermediate states.

To further analyze the global dynamics of the pluripotency network, we apply the Markov State Model coarse-graining framework. The simulated shows gaps in the eigenvalue spectrum after four and after six eigenvalues (Fig 4a). The corresponding approximate timescales are given by t₂, t₃, t₄, t₅, t₆ = 1.1 × 10⁵, 95, 51, 12, 12 (k⁻¹), respectively. These values, though only approximate, indicate the presence of a single long timescale process (t₂) corresponding to transfer between differentiated and pluripotent states, while transitions within those basins (t₃, etc.) occur at least four orders of magnitude more quickly. Applying the coarse-graining algorithm to achieve six clusters results in a reduced model (Fig 4b), with the clusters representing metastable phenotypes. The phenotypes can largely be distinguished in the subspace of NANOG, GATA6, and CDX2 expression levels; the differentiated phenotypes show expression of either GATA6 (primitive endoderm, PE), CDX2 (trophectoderm, TE), or both (denoted an intermediate cell type, IM). Phenotypes associated with pluripotency do not express high levels of GATA6 or CDX2, and may express high levels of NANOG (stem cell, SC). The coarse-grained model reveals two separate pluripotent phenotypes that are low in NANOG expression: one which expresses other pluripotent factors OCT4, SOX2, and KLF4 (“Low NANOG 1” LN1), and one which has low expression of all factors (“Low NANOG 2” LN2) (Fig 4c). Overall, these phenotypes broadly match experimentally-determined categories, coincide with steady-states of the stochastic model computed previously by a CME-approximation method [28], and coincide with phenotype-states identified in related pluripotency GRN models [62]. The steady-state probabilities associated with the phenotypes are highly nonuniform, with 95% of the population divided nearly evenly between the IM and LN1 phenotypes, which are associated with differentiation and pluripotency, respectively. The LN2 state is rarest, comprising only 8 × 10⁻⁴% of the population, and was not identified previously [28]. Together, these results indicate that the clustering method identifies both common and exceedingly rare phenotypes in the in silico cell population modeled by simulation trajectories. Furthermore, the automated method identifies both expected phenotypes and one novel (albeit low probability) phenotype.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Simulation results for the pluripotency network (Parameter Set I).

The Computational Pipeline Uncovers Six Metastable Phenotypes and Irreversible Phenotype Transitions. A) Computed eigenvalue spectrum and global timescales indicating the presence of metastability in the network. The gap in the eigenvalue spectrum after the sixth eigenvalue suggests that a partitioning can be found into six metastable phenotypes. B) The coarse-grained network showing six algorithmically-identified phenotypes designated as Low NANOG 1 (LN1), Low NANOG 2 (LN2), Stem Cell (SC), Primitive Endoderm (PE), Trophectoderm (TE), and the Intermediate Cell (IM) state. C) The averaged gene expression levels (copy numbers) of each transcription factor for each phenotype and their respective steady-state probabilities. D) The four most probable transition pathways from the SC state to the TE state (differentiation) and from the TE state to the SC state (dedifferentiation). E) The highest probability transition paths projected onto three protein coordinates, NANOG, GATA6, and CDX2. Differentiation from SC to TE is visibly irreversible, i.e., the system returns by a separate route.

https://doi.org/10.1371/journal.pcbi.1006336.g004

The method reveals multiple, irreversible pathways for phenotype transitions in the pluripotency network

Previously, Markov State Models constructed on the basis of Molecular Dynamics simulations were used to analyze the ensemble of distinct pathways of protein-folding [36]. Here, we utilize the coarse-grained model of phenotype transitions in the pluripotency GRN in a similar manner, to analyze pathways of cell differentiation and dedifferentiation. Using Transition Path Theory, the method identifies the pathways that carry the greatest fraction of net probability flux, among sequences associated with successful SC→TE transitions (and reverse) (Fig 4d and 4e). Transition paths between the stem cell (SC) and PE phenotypes can be found in S6 Fig. For Parameter Set I, the method identifies three pathways encompassing > 98% of the probability flux for both forward and reverse transitions. While the SC→ TE transition is most likely to occur directly through the LN1 state (i.e., NANOG expression will shut off, followed by turning on CDX2), the reverse transition shows a different route through the IM and PE states (i.e., GATA6 expression turns on, then CDX2 turns off, then GATA6 turns off, and finally NANOG turns on).

Dynamic analysis of the coarse-grained model, including analysis of transition paths, relies on the Markovian approximation for inter-phenotype transitions. In the pluripotency network, stochastic transitions between pluripotency (SC, LN1, LN2) and differentiation (TE, IM, PE) basins are infrequent relative to transitions within those basins, justifying the Markovian assumption, since the system equilibrates within those basins much more rapidly than inter-basin transitions occur. However, the Markovian assumption may be less accurate for describing intra-basin transitions between phenotypes, which occur much more frequently. Despite the coarse-grained model encompassing transitions on highly disparate timescales, the qualitative results of transition path analysis were validated by collected conventional simulation trajectories (not subject to any Markovian assumption), which identified the same dominant transition paths (S7 Fig). Overall, these results indicate that a stochastic excursion of a cell from the SC to TE phenotypes and back maps a cycle in gene-expression space, echoing previous studies indicating nonequilibrium dynamics in GRNs [16, 23]. The results further indicate that the Markov State Model, while a highly coarse-grained approximation, can provide an accurate estimation of inter-phenotype transition dynamics.

Cell phenotype landscape and transition dynamics are sensitive to kinetic parameters

We applied the computational pipeline to the pluripotency network using two different rate parameters sets (see S1 File), which differ in rates of transcription factor binding and unbinding to DNA. In line with previous studies [23, 24, 29], we found that increasing the so-called adiabaticity (i.e., increasing h and f, or the rates of TF-binding relative to protein production and degradation, Parameter Set II) led generally to rarer inter-phenotype transitions (see Table 1). For example, in Parameter Set I, the Mean First Passage Time (MFPT) for transitions from SC → TE was calculated to be 1.36 × 10⁵ in units of k⁻¹, as compared to 8.13 × 10⁸ for Parameter Set II. The MFPTs of the reverse transition TE → SC for each set were 2.70 × 10⁵ and 5.82 × 10⁹, respectively (see Table 1 and S7 Table). These differences in magnitude broadly reflect that moving toward the adiabatic regime leads to increased epigenetic barriers between phenotypes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Computed mean first passage times (MFPTs) of phenotype transitions in the pluripotency network.

MFPTs are shown for transitions between the pluripotency (high NANOG) state (SC) and low NANOG expression states (LN(1)) (left columns) and for transitioning between the pluripotency state (SC) and the trophectoderm state (TE) (right columns), in units of the inverse transcription factor decay rate, k⁻¹. Transitions for Parameter Set I were computed using the WE method in rate mode while transitions for Parameter Set II were estimated from the sampled transition matrix. The definitions of SC and LN(1) are analogous to the high NANOG production (N^hi) and low NANOG production (N^lo) transitions measured in experiments [8, 9]. Increasing the adiabaticity (i.e., the rates of DNA-(un)binding, h, f), leads to rarer inter-phenotype transitions. The simulations also show that, within the same gene network for a given parameter set, inter-phenotype transition times span four orders of magnitude.

https://doi.org/10.1371/journal.pcbi.1006336.t001

In addition to generally slowing transitions, the increased adiabaticity of Parameter Set II gives rise to an epigenetic landscape structure that is distinct from that of Parameter Set I, with altered steady-state phenotype probabilities (Fig 5a). The eigenvalue spectrum shows qualitatively distinct features as well, with a gap after five values (Fig 6a). As such, the Markov State Model framework identifies five dominant phenotypes in the network, which correspond broadly to those of Parameter Set I, except that only a single Low-NANOG (LN) phenotype is identified (Fig 6b). Most of the steady-state probability is contained in the IM state (Fig 6c). In addition to altering the transition rates and relative phenotype probabilities, the kinetic parameters altered the dynamics of differentiation and dedifferentiation. The two likeliest pathways of forward (and reverse) SC → TE transitions follow the same route through LN and IM phenotypes (Fig 6d and 6e). Alternative differentiation pathways of forward (and reverse) SC → PE transitions can be found in S9 Fig. These results indicate that, while the same GRN model with different kinetic parameters may give rise to qualitatively similar phenotypes, they differ in quantitative stationary and dynamic features, including relative steady-state probabilities, transition times, and likeliest transition pathways.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The rare-event sampling pipeline makes rare states and transitions accessible to simulation.

A) The global state-transition graph computed with the computational pipeline for the Pluripotency Network with rare transitions (Parameter Set II). The states are colored according to the coarse-grained (algorithmically-identified) phenotypes. In this parameter regime (f = 50) the differentiated (TE, PE, IM) and pluripotent phenotypes are cleanly separated, reflecting exceedingly rare transitions between the two phenotypes (O(10⁹), see Table 1). (B) States visited in conventional SSA simulation (using the same initialization, definitions, and placement as in (A)). In the conventional simulation, a transition out of the IM phenotype was never observed.

https://doi.org/10.1371/journal.pcbi.1006336.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Simulation results for the pluripotency network (Parameter Set II).

Changing DNA-Binding Kinetics Alters the Epigenetic Landscape. A) Computed eigenvalue spectrum and global timescales. B) The coarse-grained Markov State Model showing five phenotypes corresponding to the LN1, SC, PPE, TE, and IM phenotypes of Parameter Set I. The majority of the steady state probability is in the IM phenotype (0.98). C) The gene expression levels for each phenotype and their respective steady-state probabilities. D) The four most probable differentiation pathways between SC and TE phenotypes. E)The dominant pathways of (de)differentiation projected onto the GATA6, CDX2, and NANOG coordinates. The change in DNA-binding kinetics shows different transition dynamics from Parameter Set I. Here, the forward and reverse paths are the same.

https://doi.org/10.1371/journal.pcbi.1006336.g006

Efficiency of rare-event sampling compared to conventional SSA

Rare phenotype transitions can be difficult to observe with conventional SSA simulation. We compared simulated landscapes (based on estimated ) from the computational pipeline for the Pluripotency network (Parameter Set II) to those obtained from an equivalent (large) number of SSA simulation steps (Fig 5a and 5b). This comparison revealed that the WE-based method uncovers multiple phenotypes and associated transitions that are effectively invisible to conventional simulation due to the rarity of exiting metastable basins.

Quantitative estimates of efficiency gains for WE have been based on comparing the number of simulation steps required to estimate a desired quantity (such as a rate constant) using WE versus conventional simulation [47]. Treating as the desired output (as it contains holistic dynamic information for the system), we estimate the efficiency gain of our pipeline by computing: (10)

The denominator of Eq 10 is given by N_bins × N_iterations × τ × M_targ, thus accounting for all individual replica-steps in the total WE simulation time. The numerator is computed by asking how many steps of a conventional simulated trajectory are required to estimate . It is generally prohibitive to collect enough conventional simulation steps to estimate to a similar resolution as WE. However, given a estimated from WE, it is in principle possible to estimate how many steps would be necessary to achieve the same by conventional simulation. We used an approximate, conservative estimate given by: (11) where P_5%,i denotes the 5th percentile over nonzero elements of row i. Justification of Eq 11 is given in the Supplement, S3 File. Briefly, Eq 11 reflects the fact that the required simulation time should be dominated by the rare transitions (i.e., the smaller elements of ), while attempting to avoid over-dependence on individual estimates of small T_ij, which generally have unknown error. The error versus simulation time in WE- and Conv.-estimated are plotted in S12 Fig.

According to Eq 11, we estimate that our pipeline provided efficiency gains of 2 for ExMISA (Fig 2), 900 for Pluripotency Parameter Set I (Fig 3), and 1 × 10⁶ for Parameter Set II (Fig 6). These numbers show that the pipeline can afford a significant speedup over conventional simulation in providing global dynamic information. The numbers further show that the efficiency gain is most pronounced for the Pluripotency network with exceedingly rare inter-phenotype transitions.

Insights from the pluripotency network simulations

We used the pluripotency network as a model system to develop and demonstrate the simulation approach, but the results also yielded biological insights. For example, the simulations revealed a hierarchical structure of the epigenetic landscape. The network—exhibiting 5-6 metastable phenotypes—occupies a limited subspace from the vast possible gene combinations (e.g., 2⁸ = 256 possible distinct on/off combinations of gene expression states). The dominant feature of the global landscape is a high barrier/slow timescale between pluripotent and differentiated phenotypes. Within each of these categories, further sub-states were identified. The model revealed multi-timescale dynamics of phenotype transitions; the pluripotency network showed relatively rapid transitions between phenotype-states that differed in the expression-level (high vs. low) of a single gene, e.g. the high NANOG to low NANOG transition, whereas phenotype transitions involving a change in expression level of seven genes, e.g. the SC macrostate to the TE macrostate, occurred five orders of magnitude more slowly on average.

While the accessible phenotypes appear broadly similar across parameter sets, the relative stability and transition dynamics among phenotypes were sensitive to kinetic parameters governing transcription factor binding/unbinding. A global change in these parameters (affecting all individual transcription factor-DNA interactions equally) changed the shape of the landscape, altering the relative steady-state probabilities of different phenotypes and the likely transition pathways linking them. The DNA binding parameters capture the local epigenetic mechanisms that enable/disable transcription factors from accessing regulatory elements. A global rate change nevertheless has a varying influence on different genes because the number of regulators differs, as does the molecular logic by which activators and repressors exert combinatorial control on different genes. These results echo findings that global modification of chromatin regulators often have lineage-specific effects [63]. These results highlight both the need for, and the challenge of, informing cell reprogramming strategies with quantitative network models, as they suggest that the dynamic response of cellular networks to perturbations is governed by the detailed kinetics of molecular regulatory mechanisms, which are generally difficult to parameterize.

Dynamic definition of cell phenotype

The Markov State Model framework implicitly imposes a dynamic definition of cell phenotypes; the number of phenotypes was determined using spectral gap-analysis, and the coarse-graining algorithm automatically identified metastable aggregates (i.e., grouped sampled network states into larger clusters). This is different from the classifications of phenotypes that are generally used in analyzing experimental data, where gene expression or marker levels are often used to categorize cells. However, experiments have also revealed the potential need for a dynamic definition of cell phenotype, based not only on single-timepoint measurements of gene expression or phenotype-markers, but also on information from past or future timepoints [4, 8]. For example, Filipczyk et al. [8] identified distinct subpopulations within a compartment of NANOG-negative cells in mESCS, which differed in their propensity to re-express NANOG. At the same time, fluctuations between low- and high-NANOG expressing cells were not necessarily associated with any functional state change. The Markov State Model approach, based on kinetic/dynamic coarse-graining, thus provides a quantitative approach for classifying phenotype-states that is both completely generalizable rather than ad hoc (it requires no a priori knowledge or designation of markers/genes) and is in line with these recent experiments revealing the need for a dynamic definition of phenotype.

Timescales of stochastic phenotype transitions

Markovian transitions (i.e., memoryless “hops”) among cell phenotypes have been observed experimentally: examples include transitions among phenotypes in cancer cells, as measured by flow cytometry [10], and among pluripotency-states in mESCs, as measured by time-lapse microscopy of fluctuating gene expression [7–9]. The compact nature of these data-inferred networks—showing hops among a limited set of broad phenotypes—suggests that the computed MSM framework advanced in this study provides an appropriate level of resolution at which to analyze GRN dynamics and may serve as a useful tool for comparing models to experimental data.

Experimental studies have quantified the timescales of Markovian transitions between NANOG-high and NANOG-low states in mESCs [8, 9]. From Hormoz et al., the probability of transitioning from NANOG-high to NANOG-low in mESCs is 0.02 per cell cycle, while that of the reverse transition is 0.08. These values represent a relatively rapid transition rate, since NANOG expression is known to be particularly dynamic [60]. Similarly, plasticity has been observed in cancer cells where quantitative estimates of stochastic cell transitions between a stem cell cancer cell phenotype to a basal cancer cell phenotype were observed to be roughly on the order of 0.01 to 0.1 per cell cycle [10]. We can translate our model results to approximate biological timescales: the degradation rate, which sets the timeunit for model results (i.e., k is taken to be 1) was experimentally determined to be on the order of a few hours (in the E14 mouse embryonic stem cell line, the half-lives of NANOG, OCT4, and SOX2 are approximately 4.7, > 6, and 1.6 hours, respectively [64]). Assuming that degradation is unimolecular, k = ln(2)/t_[NANOG]1/2, and the half-life of NANOG, t_[NANOG]1/2 = 5 hours, the degradation rate is k = 0.1. Using a mESC cell cycle time of 12 hours [65], the simulations for Parameter Set I then predict NANOG-high to NANOG-low transitions occurring with a rate of 0.03 per cell cycle, and of 3 × 10⁻³ for the reverse. For Parameter Set II, the computed rates were 8 × 10⁻⁶ and 5 × 10⁻⁵, respectively. Comparison of these computed and experimental rates of NANOG transitions indicates that Parameter Set I (f = 10) is more in line with experimental observations, while Parameter Set II (f = 50) gives transition rates that are three orders of magnitude too slow. These results are in agreement with previous findings from theoretical studies that GRNs in pluripotency networks operate in a so-called “weakly-adiabatic” regime [24, 27, 28], in which the timescale of DNA-binding by transcription factors is on the order of transcription factor production and degradation.

Comparison to other models and computational approaches

A number of theoretical studies have elucidated dynamics of stochastic molecular-detailed GRN models (i.e., models that include molecular fluctuations and regulatory mechanisms, in contrast to Boolean models [66]). These studies have largely focused on small 1- or 2-gene motifs[[21–25, 32, 39]]. In the limit of slow DNA-binding/unbinding, it was shown that the stationary distribution of the stochastic model can be solved exactly [41]. Recent years have seen extension of stochastic methods to studies of more complex, experimentally derived GRN models encompassing genes. For example, determination of global dynamic properties of such networks has been achieved by combining information from long stochastic simulations of discrete models [27, 62], or of continuum SDE models, in combination with path integral approaches [58, 67]. The pluripotency network studied herein was developed by Zhang and Wolynes [28]; in their work, the authors developed a continuum approximation to the Chemical Master Equation that enabled quantitative construction of the epigenetic landscape. Here, we present an alternative approach that is unique in two major aspects: (1) the use of stochastic simulations (i.e., SSA [33]), which is enabled by use of the WE rare-event sampling algorithm, and (2) the automated Markov State Model framework for designating phenotypes and constructing a coarse-grained view of the epigenetic landscape. While we utilize a different framework (that of coarse-grained, discrete stochastic models) from Zhang and Wolynes to approximate and interpret dynamics, our results are broadly consistent with theirs. For example, the dominant identified phenotypes we found are the same as in their work (the only exception being the exceedingly rare LN2 phenotype identified by the coarse-graining algorithm for Parameter Set I).

Current challenges and future directions

Our approach is uniquely suited to extracting global dynamics information for stochastic systems with metastability, using simulations. An advantage of this approach is that both the WE and coarse-graining algorithms are“dynamics-agnostic” [47], meaning that they can be applied to any type of stochastic dynamics framework. In the context of computational biology, our pipeline could be extended to other types of stochastic biochemical systems, such as systems with hybrid discrete-continuum dynamics [68], systems with spatial heterogeneity [69], or multi-level models [70]. In addition to this flexibility, simulation-based methods have the advantage of being able to leverage existing, widely-used open-source packages, which in turn facilitate model specification and model sharing. For example, BioNetGen [56] can interpret models specified in the Systems Biology Markup Language [71].

Several challenges and potential weaknesses with the pipeline exist, both with regard to sampling rare events, and in determining an appropriate coarse-grained model. Potential challenges with the WE algorithm itself have been described elsewhere [35, 69], and include the difficulty of determining a binning that captures slow degrees of freedom and the existence of time-correlations between sampled iterations of the simulation, which can impede unbiased sampling. The Voronoi-based binning procedure we employ here is related to a number of similar approaches [24, 48–50], and has the advantage of effectively tiling a high-dimensional space without the need for a priori knowledge. However, in practice, according to others and our own studies, the method is effective up to about 10 degrees of freedom. Therefore, in larger gene networks (as in other complex systems) an ongoing challenge will be to identify optimal binning methods to effectively partition slow degrees of freedom and thus enable efficient enhanced sampling. New adaptive partitioning methods could also have the effect of improving the accuracy of coarse-grained Markov models, as finer partitioning of transition regions has been found to reduce errors in the Markovian approximation [42]. Additional improvements to efficiency, which could aid in scaling the method to larger networks, could be achieved in the future by using alternatives to the direct SSA algorithm (see e.g., [72]) or improved parallelization techniques.