A Bayesian model of reticulate phylogenies

The data S = {S₁, …, S_m} consists of the sequence alignments of m loci that we assume to be independent and recombination-free (S_i is the sequence alignment that corresponds to locus i). Our model consists of Ψ, the phylogenetic network (topology and branch lengths), and Γ, the inheritance probabilities matrix (see Methods). The posterior of the model is then given by (1) where the integration is taken over all possible gene trees, p(S_i|g) is the probability of the sequence alignment S_i given a particular gene tree g [41], and p(g|Ψ, Γ) is the density of the gene tree (topologies and branch lengths) given the model parameters [30] (see S1 Text for details of the density function).

If the gene tree estimates (or, a posterior distribution thereof) of the individual loci are also of interest, the formulation above can be modified to co-estimate gene trees, in addition, as follows: (2) where g_i is the estimated gene tree for locus i.

Indeed, this co-estimation approach is adopted by two popular species tree inference methods, *BEAST [8] and BEST [7], with the major difference being that in these methods Ψ is a tree and Γ is therefore redundant.

Inference based on Eq (1) is computationally infeasible due to the integration over all gene trees. Even Monte Carlo integration techniques would fail at estimating the integral, except for very small data sets. While sampling gene trees, as is done in *BEAST and BEST, has been shown to yield very good estimates of species and gene trees, these methods are computationally prohibitive for large data sets. Consequently, a wide array of methods for inferring species trees from gene tree estimates, rather than sequence alignments, have been introduced. In the case of networks, maximum likelihood inference of networks that uses gene tree estimates has been shown to provide good results as well [30]. If we assume that a set G of gene trees has been estimated for the m loci, then we get (3) where g_i is the estimated gene tree for locus i (with or without branch lengths). While inference from sequences directly accounts naturally for gene tree uncertainty, one has to account for this uncertainty explicitly when using gene tree estimates. Here, we will adopt the same strategy as in [30], where for each locus i, a set of gene tree estimates are obtained (e.g., the set of gene trees in a bootstrap analysis or the set of gene trees obtained from sampling the posterior of trees for that locus). Furthermore, while the method is applicable to data that consist of gene tree with branch lengths, we focus here on gene tree topologies alone (see results and discussion below for more on this point). The mass function for gene tree topologies given a phylogenetic network was derived in [35, 37] and is given in S1 Text.

To fully specify the model given by Eq (1), we need two priors p(Ψ) and p(Γ). For the phylogenetic network, we define a prior that is similar to that defined on ancestral recombination graphs in [39]. Given a phylogenetic network Ψ, we denote by Ψ_top, Ψ_λ, and Ψ_ret the topology, branch lengths vector, and number of reticulation nodes, respectively, of Ψ. We have (4) where , where T_{n, m} is the number of phylogenetic network topologies with n leaves and m reticulation nodes, , and . For the inheritance probabilities Γ, we use a uniform prior on [0, 1], though a Beta distribution would also be appropriate in general cases (see S1 Text for full details).

A reversible-jump Markov chain Monte Carlo sampler

As computing the posterior distribution given by Eq (3) is computationally intractable, we implement an Markov chain Monte Carlo (MCMC) Metropolis-Hastings algorithm. While we introduced the inheritance probabilities Γ as one parameter per reticulation node and locus, in practice, this results in a scenario where the number of parameters grows with the number of loci. Therefore, we make the simplifying assumption that there is one inheritance probability per reticulation node that is the same across all loci. In this case, Γ is a vector of length k, where k is the number of reticulation nodes in Ψ.

The description given hereafter assumes that the gene trees in the input are given by their topologies and their branch lengths are ignored. When branch lengths of the gene trees are taken into account, they pose temporal constraints on the phylogenetic network and change the moves allowed, as well as some of the quantities computed in the algorithm below. However, it is important to note that, in practice, coalescent times tend to be underestimated and that this underestimation results in biased phylogenetic estimates when sampling of loci is increased [42]. For the three biological data sets we consider below, we computed the branch lengths of the gene trees and plotted their distributions. In agreement with [42], the estimated branch lengths are very low and would result in phylogenetic networks all of whose nodes are roughly at the same level as that of the leaves. We further discuss this issue below.

In each iteration of the sampling, a new state (Ψ_i, Γ_i) is proposed and either accepted or rejected according to the Metropolis-Hastings ratio r, which is composed of the likelihoods, priors, and Hastings ratio. When the new sample changes dimensionality with respect to the current sample (which occurs only under two moves: adding a new reticulation or removing an existing reticulation), the Jacobian is also taken into account in the ratio, which results in a reversible-jump MCMC, or RJMCMC [43]. To compute the Hastings ratio, we follow the technique of [36] and illustrated by [44] for phylogenetic trees. Using this technique, given the current state x, a set of random numbers u is generated using a probability distribution with the joint probability density g(u). A deterministic function generates the new proposed state x^′ = h(x, u). To calculate the Hastings ratio, we need to account for the move that would reverse the effects of the move x → x^′. To propose x given state x^′, a new set of random numbers, u^′ is generated according to a distribution with density g^′(u^′). Then, x = h^′(x^′, u^′) where h^′ is another deterministic function. Green [36] replaced the Hastings ratio by where J is the Jacobian of the transformation from {x, u} to {x^′, u^′}.

Our algorithm employs seven moves to propose a new state of the Markov chain, illustrated in Fig 2 and the Methods section, along with their respective Hastings ratios whose full derivation is given in S1 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The seven moves that the MCMC sampler utilizes can be classified into ones that do not modify the topology of the phylogenetic network (moves 1 and 2), ones that modify the topology but do not change the model’s dimensions (moves 3, 4, and 5), and ones that modify the topology and model’s dimensions (moves 6 and 7).

The current-state phylogenetic network is shown at the center in dark blue, and the resulting next-state phylogenetic network after each moves is show in black lines. Moves 1 and 2 modify branch lengths and inheritance probabilities, respectively. Moves 3–5 relocate one of the children of a tree node, relocate the head of a reticulation edge, and reverse the direction of a reticulation edge, respectively. Moves 6 and 7 add and remove a reticulation edge, respectively. The probabilities κ and ω determine which of the three groups of moves is selected in an iteration. Within each group, an edge is selected and a move is selected uniformly at random among all the ones that are applicable to the selected edge within that group.

https://doi.org/10.1371/journal.pgen.1006006.g002

We implemented our method in PhyloNet [40], which is a publicly available, open-source software package for phylogenetic network inference and analysis. We studied the performance of the method on simulated data and three biological data sets.

Performance on simulated data

For the simulations, we used three model phylogenetic networks whose topologies and branch lengths were inspired by the estimated phylogenetic networks of the mosquito data set in [31]. Each of these networks has seven taxa, one of which is designated as an outgroup. Unfortunately, due to the prohibitive running times of computing likelihoods of networks, we currently cannot experiment with much larger (in terms of the number of taxa and/or number of reticulations) networks. The branch lengths vary from very short (about 0.5 coalescent units) to longer ones (about 1.5 coalescent units). The networks differ in the numbers of reticulations they posses (1, 2, and 3), as well as in the inheritance probabilities associated with them.

We generated gene trees for varying numbers of loci (128, 320, 800, and 2000) within each of the three networks under the multispecies coalescent process. Then, using a population mutation rate θ = 0.036, we simulated 1000-nucleotide sequences on the generated gene trees under the general time-reversible (GTR) model. Finally, for each generated alignment, we inferred 100 bootstrap trees under maximum likelihood and used those estimated gene trees as the data for our method. It is important to note that the estimated gene trees differed from the true gene trees on average in about 10% of their branches, with a standard deviation of about 10% as well. See S1 Text for the exact details of the model phylogenetic networks as well as the generated data.

Our results show that for the 1-reticulation model phylogenetic network, the 95% credible set consists of only one topology that is identical to the model network, regardless of the number of loci used. For the 2-reticulation model phylogenetic network, the 95% credible set on 128 and 320 loci consists of a single network that differs from the true network only in missing one of the two reticulations, whereas the 95% credible set on 800 and 2000 loci consists of the true phylogenetic network alone. For the 3-reticulation model network, using 128 and 320 loci resulted in 95% credible sets with one and two reticulations, respectively, of the true set of three reticulations. For 800 and 2000 loci, the 95% credible set consists of three different phylogenetic networks. However, these three networks are indistinguishable based on likelihood using the data, in the sense that their branch lengths and inheritance probabilities could be optimized to yield the same gene tree distributions. In this case, the differences among the topologies stem from different temporal orderings of the reticulation events involving the same pair of taxa. To summarize these results, the method performs very well in terms of recovering the true evolutionary history, including the number and placement of the reticulation events. For smaller numbers of loci, the method obtained networks that are missing one or two of the reticulations, but the rest of the evolutionary history was correct. In other words, for these smaller numbers of loci, the false positive rate was effectively 0.

In terms of runtime, the method took about 2.8 hours to run for 5 million iterations on the smallest data set (one reticulation and 128 loci) and about 9.2 hours for the same number of iterations on the largest data set (three reticulations and 2000 loci). The bottleneck in these computations comes from the likelihood calculations. The phylogenetic network topology and gene tree topology both play a role in a large variations in likelihood computation times, as reflected by large standard deviations when averaging times over the different distinct gene trees.

Full results of the performance of the method on the simulated data in terms of phylogenetic network quality and runtimes are given in S1 Text.

Reticulate phylogenies of wheat, mosquito, and mouse genome data sets

In addition to the synthetic data, we analyzed a bread wheat genome data set from [29], a mosquito genome data set from [31], and a house mouse genome data set from [30]. It is important to note that the wheat and mosquito data sets consist of the genomes of different species, whereas the house mouse data set consists of multiple genomes from the same species. Thus, these analyses highlight the applicability of our method to the detection of intra- and inter-specific hybridization, as well as the challenges that arise in the different evolutionary scenarios.

The bread wheat data set consists of three subgenomes of wheat: Triticum aestivum, TaA (A subgenome), TaB (B subgenome) and TaD (D subgenome). Marcussen et al. found that each of the A and B lineages is more closely related to D than to each other, as represented by the phylogenetic network in Fig 3A that they inferred using the parsimony approach of [45]. Based on this network, they proposed an evolutionary history of Triticum aestivum, where the D genome originated from the A and B genome lineages, AABB originated from AA and BB, finally AABB and DD led to origination of AABBDD by polyploidizations and hybridizations, as shown in Fig 3 in [29].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Phylogenetic history of the bread wheat.

(A–C) The three phylogenetic networks that comprise the 95% credible set, (D) and a plausible summary of the three networks that is consistent with the model of phylogenetic history of bread wheat (Fig 3 in [29]).

https://doi.org/10.1371/journal.pgen.1006006.g003

To analyze this data set, we constructed bootstrap trees from the sequences of 2269 genes provided in [29]. The MCMC chains converged fast within a short period of burn-in, as indicated by the trace plot. The three phylogenetic network topologies in the 95% credible set are shown in Fig 3A–3C. A plausible summary of the three networks is shown in Fig 3D, which is consistent with the model of bread wheat proposed by Marcussen et al. In terms of runtime, the five million iterations of the MCMC sampling took about 2.2 hours.

While we only used gene tree topologies here, we also inferred gene trees with branch lengths under maximum likelihood. We then estimated for every pair of species the coalescence times based on all 2269 gene trees. We observed that the median pairwise distance for each pair of taxa was around 0.025, and with minimum distances of 0. Since each pairwise distance poses an upper bound on the time of the most recent common ancestor (MRCA) of that pair of species (considering the time at the leaves to be 0), then inference of phylogenetic networks using the likelihood function employed here would result in sampling only phylogenetic networks all of whose nodes have time 0. In other words, using gene tree branch lengths here would result in uninformative phylogenetic networks.

The mosquito data set [31] consists of the four autosomes and X chromosome of six species from the An. gambiae complex: An. gambiae (G), An. coluzzii (C), An. arabiensis (A), An. quadriannulatus (Q), An. merus (R) and An. melas (L). This data set was collected and analyzed by Fontaine et al. [31]. In that study, the authors inferred a species tree based on the X chromosome and postulated two major hybridization events to explain the extensive introgression. More recently, Wen et al. [33] reanalyzed the data set using the maximum likelihood method of [30] while restricting the number of reticulations to three, due to computational requirements. Their results corroborated parts of the evolutionary history presented in [31] and provided a different scenario for other parts.

We reanalyzed this data set using our Bayesian method. We first analyzed the X chromosome data alone. The 95% credible set consists of a single phylogenetic network, shown in Fig 4A, that agrees with [31, 33]. We then analyzed the autosome data. The 95% credible set consists of three phylogenetic networks that have the same three reticulation events but differ in terms of their temporal orders. As discussed above, these three are indistinguishable under our model, and their summary is given in Fig 4B. The result is in agreement with that in [33]. However, an important point here is that in this analysis, we did not bound the number of reticulations. This number was inferred as a function of the data used and the prior setting. In contrast, in [33], the number of reticulations was bounded by three, for computational reasons. In terms of runtime, this analysis took about 7.65 hours.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Phylogenetic history of the six mosquito genomes.

(A) The single phylogenetic network in the 95% credible set sampled on the X chromosome data. (B) A summary of the three phylogenetic networks in the 95% credible set sampled on the autosome data. The dotted circle indicates the temporal order of the two reticulation events involving R cannot be discerned with confidence from the data.

https://doi.org/10.1371/journal.pgen.1006006.g004

Given that the data set was larger than the wheat data set, we also experimented here with the prior and amount of data. In particular, we tried three values for the Poisson prior mean: 0.1, 1.0, and 10. Furthermore, while the full data set used here consists of 2791 loci, we also sampled 311 and 931 loci to create two additional data sets with smaller numbers of loci. We then conducted sampling under each combination of prior setting and data set size, our hypothesis being that as the amount of data increases, the effect of the prior setting would diminish. Indeed, we found that as the number of loci increases, the 95% credible set becomes the same regardless of the Poisson prior mean value. For the smallest data of 311 loci, a mean value of 0.1, resulted in a 95% credible set with one phylogenetic network. Yet, when the mean value was changed to 1.0 or 10, the 95% credible set contained 4 phylogenetic networks. These results further demonstrate a desirable behavior of the method.

It is worth mentioning that when setting the Poisson prior mean value to 10, the runtime increased significantly. For example, on the data set with 2791 loci, it took about 30 hours for the 5 million iterations. This is because the chain samples in this case networks with larger numbers of reticulations and whose likelihood computation time is large.

Just as in the case of the wheat data set, we also computed the pairwise distances among species based on estimated gene tree branch lengths. Once again, the results point to minimum pairwise estimates that are very close to 0 (they equal 0 for some pairs). In this case, inference the uses our likelihood formulation and gene tree branch lengths would result in an uninformative network.

Finally, the mouse data set consists of individuals sampled from five Mus musculus populations: two samples of M. m. domesticus from France (DF) and Germany (DG), and three samples of M. m. musculus from the Czech Republic (MZ), Kazakhstan (MK), and China (MC). For the gene genealogies, 20,639 trees were inferred for sampled loci; see [30] for details. Yu et al. found two main introgressions by maximum likelihood regularized by cross-validation. One involves the MRCA of {DF, DG} as a recipient population and MK, MC, or their MRCA as the donor population. The other involves MZ as a recipient population and DF, DG, or their MRCA as the donor population.

This data set differs from the previous two data sets in a significant way: The five genomes are obtained from individuals of the same species (M. musculus). That is, this is a data set with very low divergence. Indeed, when estimating gene trees with branch lengths, we found that all pairwise distances among the five individuals were 0; in this case, even the medians were mostly 0. Since the number of loci is very large in this case, we first analyzed the estimated gene trees for resolution. For each locus, we computed the majority-rule consensus of the 100 bootstrap trees of that locus, and counted the number of internal branches in the resulting tree. 11,457 loci had fully-resolved majority-rule consensus trees. Within this set of trees, and out of all the 105 possible binary trees on 5 taxa, 3 trees appeared with a frequency greater than 2000 each, 7 trees appeared with a frequency in the range [200, 399] each, and 11 trees appeared with a frequency in the range [50–199] each. The other 84 binary trees on 5 taxa appeared with a frequency smaller than 50 (7 trees did not appear at all in this set). In our reanalysis of this data set, we used the first set of 3 + 7 + 11 = 21 trees. In total, this amounted to using 10,575 loci.

Since the taxa correspond to individuals from the same species, more extensive gene flow is expected. Indeed, the analysis resulted in a 95% credible set with six different phylogenetic network topologies. A plausible summary of these six networks is shown in Fig 5 (the actual six networks are shown in S1 Text). This network indicates significant hybridization involving MZ. Furthermore, MC and DG are not involved in any of the hybridization events, while their ancestors and sibling taxa are involved. The network gives rise to a hypothesis of hybridization involving the ancestors of {MC,MK} and {DF, DG}, which are two different subspecies, M. m. musculus and M. m. domesticus, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Phylogenetic history of the five mouse genomes.

A summary of the six phylogenetic networks that comprise the 95% credible set. The dashed circle indicates that the different networks in the 95% credible set resolve the order of the hybridization events involving MZ in different ways.

https://doi.org/10.1371/journal.pgen.1006006.g005

In terms of computational runtime, this analysis too about 45 hours, which is much longer than the other two data sets. This is a reflection of the larger size of the data, and the complexity of the networks visited during the MCMC walk.

Phylogenetic networks

A reticulate, i.e., non-treelike, evolutionary history that arises in the presence of processes such as hybridization and horizontal gene transfer is best represented by a phylogenetic network.

Definition 1 A phylogenetic -network, or -network for short, Ψ is a directed, acyclic graph (DAG) with V = {r} ∪ V_L ∪ V_T ∪ V_N, where

• indeg(r) = 0 (r is the root of Ψ);
• ∀v ∈ V_L, indeg(v) = 1 and outdeg(v) = 0 (V_L are the external tree nodes, or leaves, of Ψ);
• ∀v ∈ V_T, indeg(v) = 1 and outdeg(v) ≥ 2 (V_T are the internal tree nodes of Ψ); and,
• ∀v ∈ V_N, indeg(v) = 2 and outdeg(v) = 1 (V_N are the reticulation nodes of Ψ),

E ⊆ V × V are the network’s edges, including reticulation edges whose heads are reticulation nodes, and tree edges whose heads are tree nodes., and ℓ: V_L → is the leaf-labeling function, which is a bijection from V_L to .

We use V(Ψ) and E(Ψ) to denote the set of nodes and edges of phylogenetic network Ψ respectively. In addition to the topology of a phylogenetic network Ψ, each edge b = (u, v) in E(Ψ) has a length λ_b measured in coalescent units, which is the number of generations divided by effective population size on that branch.

The multispecies network coalescent (MSNC)

As an orthologous genomic region from a set of species evolves within the branches of the species phylogeny of , the genealogy of this region, also called gene tree, can be viewed as a discrete random variable whose values are all possible gene tree topologies on the set of genomic regions. When the gene tree branch lengths are also taken into account, the random variable becomes continuous. Yu et al. [35] gave the probability mass function (pmf) for this discrete random variable given the phylogenetic network Ψ and an additional parameter Γ that contains the inheritance probabilities associated with reticulation nodes, which we now describe briefly.

Let E_R ⊆ E(Ψ) be the set of reticulation edges, and ρ = |E_R|, and consider a data set that consists of m independent loci. The inheritance probabilities are given by a ρ × m matrix Γ such that for every 1 ≤ j ≤ m:

1. Γ[b, j] ∈ [0, 1] for every b ∈ E_R, and
2. Γ[b, j] + Γ[b^′, j] = 1 for every distinct pair b, b^′ ∈ E_R such that b and b^′ are incident into the same reticulation node.

For an edge b incident into node v in Ψ, the entry Γ[b, j] denotes the probability that a sample from locus i tracks branch b when “entering” the population represented by node v.

The mass and density functions of gene trees given a phylogenetic network, its branch lengths, and inheritance probabilities were derived in [30, 34, 35, 37]; see S1 Text for a brief discussion of these two functions. Furthermore, Yu et al. discussed unidentifiability issues of phylogenetic networks and their parameters from gene tree topologies [35].

Inference using RJMCMC

The general form of the Metropolis-Hastings algorithm that we implement is given in Algorithm 1. While the algorithm is described in a way that all accepted samples are returned, in the actual implementation all samples from a burn-in period are discarded, and only a small percentage of the samples beyond that are stored (the burn-in period and percentage of samples to store are both user-defined in our implementation).

Algorithm 1: Metropolis-Hastings.

Input: A set of gene trees G and the number of iterations N.

Output: A collection S of (Ψ, Γ) samples.

Initialize Ψ₀, Γ₀;

for i ← 1 to N do

  Ψ_i, Γ_i ← propose(Ψ_i−1, Γ_i−1);

 if proposal does not change dimensionality then

   ;

 end

 else

   ;

 end

 if r < random(0, 1) then

  Ψ_i ← Ψ_i−1;

  Γ_i ← Γ_i−1;

 end

 S ← S ∪ {(Ψ_i, Γ_i)};

end

return S;

The function propose(Ψ, Γ) proposes a sample based on the current sample and set of predefined moves that are listed in Table 1 and illustrated in Fig 2 above. As we described above, the moves might generate a phylogenetic network topology that deviates “slightly” from the conditions of Definition 1. What we mean by “slightly” is that the network could violate Definition 1 in one of the following ways:

• the proposed network topology has a cycle (moves 3–5 in Table 1 might cause this);
• the proposed network topology has two edges with the same tail and head (moves 3 and 6 in Table 1 might cause this); or,
• the proposed network has more than a single node of in-degree 0 (move 3 in Table 1 might cause this).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The 7 moves that the Metropolis-Hastings algorithm employs.

https://doi.org/10.1371/journal.pgen.1006006.t001

Therefore, in computing the Metropolis-Hastings ratio, our implementation explicitly tests whether the proposed network topology has any of these properties; if it does, the phylogenetic network prior is set to 0, and otherwise, the prior is set based on Eq 4.

The function propose selects the move as follows:

• With probability κ, one of the two dimension-changing moves (moves 6 and 7 in Table 1) is selected. If the current network has at least one reticulation edge, then both moves 6 and 7 are possible. Add-Reticulation is selected with probability κ₁ (and Delete-Reticulation is selected with probability 1 − κ₁). If the current network has no reticulation edges (i.e., it is a tree), then Add-Reticulation is selected.
• With probability 1 − κ, a non-dimension-changing move (moves 1–5 in Table 1) is selected.
  • With probability ω a non-topology-changing move (moves 1 and 2 in Table 1) is selected. If the current network has at least one reticulation edge, then Change-Length is selected with probability ω₁ and Change-Inheritance is selected with probability 1 − ω₁. If the current network has no reticulation edges, then Change-Length is selected.
  • With probability 1 − ω a topology-changing move (moves 3–5 in Table 1) is selected. Moves 3, 4, and 5 are selected with probabilities ξ₁, ξ₂, and ξ₃, respectively, where ξ₁ + ξ₂ + ξ₃ = 1.

Full derivation of the Hastings ratios for all seven moves is given in S1 Text.