Mixture models.

Let be the observed data, with for each item being considered, where each observation has P variables. We wish to model the data using a mixture of densities:

(1)

independently for each . Here, is a probability density function such as the Gaussian density function or the categorical density function, and each component has its own weight, , and set of parameters, . The component weights are restricted to the unit simplex, i.e., . To capture the discrete structure in the data, we introduce an allocation variable, , to indicate which component a sample was drawn from, introducing conditional independence between the components,

(2)

(3)

The joint model can then be written

(4)

We assume conditional independence between certain parameters such that the model reduces to

(5)

In terms of the hierarchical model this is:

(6)

(7)

(8)

(9)

where F is the appropriate distribution (e.g., Gaussian, Categorical, etc.) and G⁽⁰⁾ is some prior over the component parameters. In practice, represents the parameters characterizing each cellular compartment (e.g., mean protein abundance profiles), π captures the relative sizes of compartments, and c_n indicates which compartment each protein belongs to. The hierarchical structure allows uncertainty to propagate through all levels of the model. We use the empirical priors suggested by Fraley and Raftery [43] for the Gaussian mixture models. For the Categorical model we use the average proportion of each class for each feature to define our prior density.

T-augmented Gaussian mixture model.

Crook et al. [23] proposed capturing outliers within the Gaussian mixture model using a heavy-tailed multivariate t (MVT) distribution in their t-augmented Gaussian mixture (TAGM) model. They consider each component a mixture of a multivariate normal (MVN) density and this MVT which has common parameters across all components, i.e.,

(10)

where and are the mean vector and covariance matrix of the k^th component, and M,V and η are the parameters of the MVT which are set empirically and not sampled. Being considered an outlier is an inferred quantity indicated through the binary variable. Note that the items of known label (in our applications, the marker proteins) are never considered outliers. This extends the hierarchical model described above to:

(11)

(12)

(13)

(14)

(15)

where we set η to 4 to ensure heavy-tails, and set M to empirical mean and V to half of the global variance of the data following Crook et al. [23].

Gaussian process.

The Gaussian Process (GP) is a generalisation of the Gaussian probability distribution over functions or infinite vectors rather than finite vectors. The GP is immensely flexible and can describe measurements of arbitrary proximity, most relevant in modelling temporal or spatial data. Specifically, the GP is a continuous stochastic process with index set T such that its realisation at any finite collection of points is jointly Gaussian. It is uniquely specified by a mean function m(t) and a positive semi-definite covariance kernel , which determine the mean vectors and covariance matrices of the associated multivariate Gaussian distributions. That is, f(x) is a GP with mean function m(t) and covariance kernel if for any

(16)

(17)

GP models have been used in a range of problems such as systems biology analyses [44–48] environmental and health studies [49,50], and monitoring patient health [51,52] and are built upon rigorous theoretical foundations in terms of support and contraction rates [53–55].

In practice, a prior mean function of is commonly used and the covariance function is chosen from a small family of easy to implement functions such as the squared exponential function [56]. Lacking any strong belief about periodicity, symmetry or other constraints on the functions, we use the squared exponential function which is defined for two time points/locations , as

(18)

where a and l are the amplitude and length-scale respectively. We use the most common choice for the metric , Euclidean distance or the l²-norm.

Gaussian process mixture models.

If we consider the mean vector of a Gaussian density as the finite realisation of an infinite function in our feature space, we can place a Gaussian process prior over the space of functions it is drawn from. Thus, for the k^th component of this mixture, mean vector and observed data which has been transformed to a vector, ,

(19)

(20)

As we assume exchangeable data for ease of notation we permute the original dataset such that we can denote X_k with contiguous indices, i.e., . The posterior distribution can be derived by using the properties of the multivariate Gaussian relating to conditional distribution of two jointly distributed Gaussian variables [57],

(21)

(22)

We can then say

(23)

(24)

(25)

This involves an expensive calculation, taking the inverse of an N_kP × N_kP matrix. There are methods to scale GP models, e.g., sparse approximations [58] and low-dimensional approximations [59], see Liu et al. [60] for a review of the subject. However, in our application we have rich structure in the data as all items within a modality have the same time/spatial measurements with consistent distance between each measurement and we can use this to reduce the computational load without using any of these more complex methods [61], we can instead rewrite the posterior parameters as

(26)

(27)

Please see section A.1 in S1 Text for this derivation.

We sample the hyper-parameters according to a Metropolis-Hastings scheme. Standard log-normal hyperpriors were used.

Multiple dataset integration.

MDI is a Bayesian integrative or multi-modal clustering method. Bayesian statistics allows quantification of uncertainty through the use of probability distributions and offers a modular framework for data analysis by making dependencies between data and parameters explicit [62]. In MDI, signal sharing is defined by the prior on the cluster label of the n^th observation in the M modalities/datasets:

(28)

where is the label of the n^th observation in the m^th modality, is the weight of the k^th cluster in the m^th modality, is the similarity between the clusterings of the m^th and l^th modalities and is the indicator function returning 1 if x is true and 0 otherwise. Attractively, is inferred from the data and if there is no shared signal it will tend towards 0 giving us no dependency between these modalities in which case each dataset will have independent clusters.

Within each dataset/modality MDI uses a specific density to model the probability of allocation to a specific cluster. For example, one might wish to model continuous data with Gaussian densities, categorical data with Categorical distributions and/or time-series data with Gaussian processes. Thus we have flexibility in how we model each dataset within the MDI framework.

We implement this in C++ with the novel functionality to allow for observed classes and labels in a subset of datasets/modes and make this available through the R package MDIr. In the Bayesian paradigm, moving from an unsupervised cluster method to a semi-supervised method can be phrased as a missing data problem. Let be a vector indicating if the label c_n is observed or not. If i_n = 0 (i.e. the label is unobserved), then c_n is sampled as usual, but if i_n = 1 then c_n is known and held fixed. Observing labels means that we can no longer arbitrarily switch labels as the labels are inherently linked to a specific class (in our case, subcellular niches). The Gibbs sampler to perform inference on the model is implemented in C++ and includes a correction to the conditional distribution of the parameters which was incorrect in previous implementations as observed by Stephen Johnson, Daniel Henderson, and Richard Boys in 2017 (private correspondence; see section A.2 of S1 Text).

As per Kirk et al. [40], we use an overfitted mixture model [63] to approximate a Dirichlet process mixture model [64–67] in each unsupervised modality, allowing inference of the number of clusters/classes present. If a number of classes are observed but one believes that additional classes might be present in the data one may use a semi-supervised overfitted mixture to detect novel classes as per Crook et al. [68].

Markov chain Monte Carlo methods.

We use Markov chain Monte Carlo (MCMC) methods [69–72] to perform inference on our Bayesian models. MCMC is the gold standard for Computational Bayes [73–76], constructing a Markov chain of sampled parameters to describe the conditional posterior density for model parameters, with asymptotic guarantees that this description is exact. However, in practice MCMC is prone to becoming trapped in local modes of the target density. Designing algorithms that perform better in this scenario is an area of active research [77–83], but implementing these methods can be time-consuming and with no guarantee of overcoming the problem. We instead use Gibbs sampling [84] in our implementation of MDI and the consensus clustering approach to circumvent multi-modality [85]. We can use the collection of sampled values to infer a point estimate and provide a quantification of uncertainty about this. Common choices of point estimate for continuous parameters are the median or the mean, but to obtain a summary clustering we instead use that which minimises the variation of information [86], which has been found to have desirable properties [87], though alternative approaches exist [88,89]. To summarise the uncertainty about a partition, the posterior similarity matrix (PSM) is often used. This considers the average co-clustering probability of each pair of items based on the sampled clusterings. It is an N × N symmetric matrix. For a classification, we assign the samples to the class with the highest average probability of allocation across the retained MCMC samples.

Transfer learning using a k-nearest neighbours framework.

The k-nearest neighbours (KNN) classifier is a supervised method in which predictions are made using the majority vote of an item’s k nearest labelled neighbours. The KNN-transfer learning (KNN-TL) classifier extends this to transfer information from an auxiliary modality to a primary modality [41,90]. To use the transfer learning algorithm, the best choice of , the number of neighbours considered when classifying items with missing labels for each modality, must be estimated from the KNN classifier to be used as inputs into the KNN-TL classifier. This is performed via a grid search on a pre-specified grid, with the optimal value selected through cross-validation. Breckels et al. [41] adapted this method to predict protein localisation using LOPIT data as the primary dataset and GO terms as the auxiliary dataset (in contrast to MDI, the KNN-TL classifier does consider an ordering of datasets). In this context, this algorithm finds the k₁ nearest marker proteins to the n^th protein in the LOPIT data and the k₂ nearest marker proteins in the GO data. The representation of each class among these neighbours is calculated, i.e., if are the sets of neighbours in the first and second modalities respectively,

(29)

from which a localisation is predicted

(30)

where is estimated using a grid search (again, the optimal value is selected through cross-validation on the marker proteins). The final estimate is .

Simulation study.

Our simulation study investigates the benefit of having observed labels available in a modality (such as marker proteins in a hyperLOPIT experiment), and the advantage of performing an integrative analysis rather than independent analyses of each modality. To achieve this, we construct three scenarios. Each scenario is defined by the density used to generate the clusters (the generative model) and by the ϕ vector which determines the similarity of the clustering structure across modalities. We first generate the labels indicating the cluster an item is generated from in each modality and then sample measurements based on this cluster’s parameters. In every simulation, the first dataset is generated from six clusters, the second from seven clusters and the third from eight clusters. We generate two hundred labels and data points in each datasets from this model:

(31)

(32)

There are fifteen measurements/feature for each data point (i.e., in each dataset in all simulations). For the semi-supervised models, a random 30% of the labels are given as observed in the first dataset with no labels or classes observed in the second and third datasets in all models. Within a generating seed the labels in each modality are the same across all scenarios and each dataset is informative of the others (). The Adjusted Rand Index (ARI) [91] between the true labels of each pair of datasets across simulations is shown in Fig 1A. The ARI is a score comparing two partitions, adjusted for chance. A value of 0 means two partitions are no more similar than a pair of random partitions would be expected to be, 1 means they are identical.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A. ARI between the generating labels for each combination of dataset pairings within a given simulation across all scenarios.

For example, a point contributing to the (1, 2) boxplot is the ARI between the generating clusterings for the first and second datasets for a given random seed and scenario. B. Example of simulated data from each scenario and the generating labels all ordered by the data in the first modality for each scenario.

https://doi.org/10.1371/journal.pcbi.1013799.g001

We consider three different generating models which give the scenarios their names. The first two are

(33)

(34)

Scenario 3, the Log-Poisson case, is more complex.

(35)

(36)

(37)

This is inspired by the simulation study of Chandra et al. [92], but the Gaussian noise is added after the log-transform to ensure this transform is always possible. Each cluster strongly deviates from Gaussian in this case. An example of the simulated data is shown in Fig 1B. These distributions were selected to test robustness under different model assumptions: Gaussian represents ideal conditions matching model assumptions (often assumed for gene co-expression data), MVT tests heavy-tailed outlier scenarios common in proteomics, and Log-Poisson evaluates performance under severe distributional misspecification (based on log-normalization of count data such as RNA-seq). A full description of the generating mechanisms and choice of parameters to differentiate the datasets are given in section B of S1 Text.

We fit five different models in each simulation. These are distinguished with the information available to them in the first dataset and whether they jointly model the datasets (i.e., MDI) or model each dataset separately using a mixture model, and if the model has access to class labels and the true number of classes present (e.g., when the number of subcellular niches is unknown). This comparison allows us to isolate the benefits of: (1) integrative vs. independent modeling, (2) supervised vs. unsupervised learning, and (3) known vs. inferred cluster numbers. In all cases datasets 2 and 3 are unsupervised and the number of clusters present is inferred. For the overfitted approaches, we use the overfitted mixture model framework of [63], which includes more mixture components than expected and allows the data to determine the effective number of clusters through Bayesian model selection. This creates two variants of semi-supervised MDI: (1) semi-supervised MDI where the true number of subcellular compartments is provided as prior knowledge, and (2) overfitted semi-supervised MDI where this number must be inferred from the data alone. The overfitted approach is often relevant to real applications where the total number of subcellular niches may be unknown, while the standard approach provides an upper bound on performance when complete prior knowledge is available.

• Unsupervised MDI: MDI with no observed labels and the number of clusters unknown,
• Overfitted semi-supervised MDI: MDI with known labels, but the true number of clusters is hidden,
• Semi-supervised MDI: MDI with both observed labels and the true number of clusters known,
• Overfitted semi-supervised mixture model: mixture model with known labels, but the true number of clusters is hidden,
• Semi-supervised mixture model: mixture model with both observed labels and the true number of clusters known,

We run 10 chains of each method to each simulated dataset. Each chain is 15,000 iterations long with the first 7,500 discarded as burn-in. We thin to every 100th sample to reduce autocorrelation. The best chain for the MDI methods is determined by the chain with the highest mean ARI between the point estimate clustering and the truth across datasets whereas for the mixture model the chains are selected based on the highest ARI between the point estimate and the truth in each dataset (rather than using the same seed across datasets).

Validation study.

We compare our approach to a state-of-the-art transfer learning method previously applied to LOPIT data, the KNN-TL classifier, using the marker proteins from four of the same datasets analysed by Breckels et al. [41]. This includes two datasets for Arabidopsis thaliana, the Root Callus [16] dataset (referred to hereafter as Callus to distinguish it from the second A. thaliana dataset) which contains 261 marker proteins and 16 fractions, and the Root dataset [93] which contains 185 marker proteins and six fractions. a dataset for human embryonic kidney fibroblast cells [94] which contains 404 marker proteins and 8 fractions, and a mouse pluripotent stem cell dataset [12] which contains 8 fractions for 387 marker proteins. These are available through the pRolocdata Bioconductor package. Fig 2 shows PCA plots of these proteins. These are paired with GO term data. GO terms record observed and inferred characteristics of gene and gene products in terms of three sub-ontologies. The cellular component (CC) the product localises to; its molecular function (MF), such as binding or catalysis; and the biological processes (BP) the product is involved, e.g., chromatin binding. The data we use are the same as are available through the pRolocdata Bioconductor package [95]. This data was prepared by Breckels et al. [41] by using all possible GO CC terms associated to the proteins in the experiment to construct a binary matrix representing the presence/absence of a given term for each protein, for each experiment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The T. gondii data.

A) A tSNE of the hyperLOPIT data from Baryluk et al. with the marker proteins of each organelle highlighted. B) The gene expression data from Behnke et al. C) Fig 1 from Behnke et al. showing progression of cell number per vacuole (vacuole size, squares) and portion of cells undergoing division (triangles) at different timepoints in the gene expression data.

https://doi.org/10.1371/journal.pcbi.1013799.g002

We model the proteomic datasets with a TAGM model and the GO terms using a categorical mixture model (see the supplementary material of [40] for a description of this model) within the MDI framework. We run five chains of the TAGM and MDI models for 15,000 iterations, discard the first 5,000 as warm-up, thin to every 50th and then combine samples across chains (i.e. construct an ensemble). We follow the vignette of Breckels et al. [94] for the KNN-TL algorithm. This involves identifying the inputs and θ in a grid search using cross-validation. We consider a range of and for . As all the combinations of θ must be considered this becomes computationally intensive as K increases. We considered the smaller of either the full combination matrix or a random sample of 20,000 rows. We inspect performance across ten folds with approximately 30% of the marker proteins observed in each organelle and 70% of the localisations predicted.

We compare performance using the Brier loss, macro F1 score and accuracy (defined below). Accuracy is the number of predictions that match the ground truth divided by the total number of predictions; this is the simplest score we use as it does not account for either uncertainty or class size. The F1 score is a common measure of performance in binary classification examples that corrects for sensitivity and specificity. The variation considered here is an extension to the multi-class setting. Letting denote the predicted class for the n^th item and c_n denote the true class, then

(38)

(39)

(40)

(41)

(42)

(43)

(44)

where z_n,k is the (n,k)^th entry in the one-hot-encoding of the true allocation matrix, and is the inferred probability of the n^th item being allocated to the k^th class. The KNN-TL classifier does not provide a measure of uncertainty and thus we use for and 0 otherwise. The Brier score captures how well the Bayesian methods quantify the uncertainty about the classification.

To explore performance on more modern datasets and show that MDI has application beyond analysing pairs of datasets (in contrast to the KNN-TL classifier) we also investigate performance in predicting allocation of the marker proteins in data from Beltran et al. [96] and Orre et al. [14]. The first is spatial proteomics data for human cells infected by Human Cytomegalovirus (HCMV) across 5 time points (24, 48, 72, 96 and 120 hours post infection) and contains 295 marker proteins with measurements in every time point. The second contains measurements for 3,330 marker proteins across five different human cancer cell lines (epidermoid carcinoma A431, glioblastoma U251, breast cancer MCF7, lung cancer NCI-H322, and lung cancer HCC-827). We use a TAGM model for each dataset within the MDI model and as in the previous datasets we consider performance over ten different splits of 30% localisations observed and the remaining 70% inferred.

Toxoplasma gondii.

The Apicomplexa phylum of highly successful obligate parasites that can invade practically all warm-blooded animals [97–101]. As such, the apicomplexans are the source of much scientific interest, particularly understanding the mechanisms by which they infect a host and evade the innate and adaptive immune responses. However, this phylum is deeply divergent from the canonical models of molecular biology [31]. This means that results and knowledge do not always translate well from these well-studied organisms to the Apicomplexa. T. gondii is a natural candidate for the model apicomplexan as it can be cultured continuously in a lab and is readily amenable to genomic engineering for targeted experiments [102]. Beyond implications for other apicomplexans such as Plasmodium falciparum, the cause of malaria, T. gondii is also of medical interest in and of itself being the cause of Toxoplasmosis which, in immunocompromised individuals and as a congenital disease in infected infants, can be life-threatening [103]. From a proteomic perspective, the apicomplexans are a fascinating group; they have a myriad of unique cell structures and compartments that play a key role in their success. For example, they contain an apical structure central to penetration and invasion of animal cells; this contains secretory compartments including the micronemes, rhoptries, and dense granules for the staged release of molecules required for successful invasion and establishment within the host cell [104–107]. Some of the de novo subcellular niches are formed during the invasion of the host cell by the parasite [108] and it is hoped that better understanding of these organelles and their formation during invasion might lead to novel, targeted treatments [109].

Motivated by the importance of these organisms and their complex cellular structure, we perform an multi-omic analysis of the model apicomplexan, T. gondii, combining hyperLOPIT data from Barylyuk et al. [31] and transcriptomic data for the cell-cycle over twelve hours of post-invasion parasite replication [110] (Fig 2). We use cell cycle time-series gene expression data due to recent evidence that tightly co-ordinated transcriptional programs are essential for proper organelle [111,112], and we expect this to reveal reveal the transcriptional programmes of the genes involved in organelle formation and the coexpression patterns of the secreted proteins [113]. Additionally, our MDI framework does not force proteins with strong spatial proteomics evidence to relocalize based purely on temporal data - proteins most likely to benefit from integration are those with ambiguous spatial signals but strong temporal co-clustering with organelle-specific genes. The hyperLOPIT data consists of 3 experimental iterations, each containing 10 fractions. By jointly modelling gene expression data with hyperLOPIT data annotated using marker proteins we hope to draw on the temporal programmes of the cell that are specific to spatial niches, and in doing so further resolve protein functional cohorts. However, jointly modelling the two datasets might also shed light on the localisation of proteins. As it stands, proteomes of most parasite compartments remain poorly characterised. Even the locations of proteins of predicted function based on conserved sequences in other organisms are largely untested in apicomplexans, though recent work from Barylyuk et al. [31] has improved our knowledge enormously. To generate both datasets the parasites were grown in and purified from human foreskin fibroblasts. For the gene-expression dataset parasites were synchronised using thymidine following the protocol developed by Radke and White [114]. We consider only the genes which have products in both datasets, reducing to 3,643 genes modelled.

Within the MDI model we choose data-specific probability densities to capture the structure in each dataset. We model the hyperLOPIT data using a TAGM model, setting the number of components to 26 which is the number of subcellular niches with associated marker proteins. In the cell cycle data, we are interested in co-expression patterns. We therefore mean-centre and standardise the measurements for each gene rather than considering absolute transcript abundance. To capture the temporal nature of the data, this modality is modelled using a mixture of MVNs with a GP prior on the mean function, as described in section Gaussian process mixture models. The proposal window for the Metropolis-Hastings sampling of the Gaussian process hyperparameters was set by achieving an acceptance rate between 0.2 and 0.8 in a short chain. For the gene expression data we use 125 components to infer the number of clusters present (approximating a Dirichlet process mixture model [63] in this modality). We also model this data with an unsupervised mixture model with the same choices and parameters as the corresponding submodel in MDI, but the number of components required to be overfitted is much larger and we use 300 components in the mixture model. Due to the complexity of the data and a finite computational budget we used the consensus clustering approach of Coleman et al. [85] to circumvent the issue of poor mixing in individual chains and used the 15,000th sample from 150 randomly initialised chains (see section C in S1 Text for details of assessing stability). We inferred a point estimate partition for the hyperLOPIT data by using the localisation with the highest average allocation probability across MCMC samples, in the transcriptomic data we used the estimate which minimised the lower bound to the posterior expected Variation of Information with a credible ball around the estimate as per the method of Wade and Ghahramani [86].