Basic model

The methodology of this study is based on the Gaussian graphical model (GGM), a probability model that characterizes the conditional dependence structure of a set of random variables by a graph. Let X = (X₁,X₂,⋯,X_p)′ be a multivariate Gaussian random vector with mean μ (we assume μ = 0 hereafter) and covariance matrix Σ. A GGM associated with X is a graph G = (V, E). The node set V = {X₁,X₂,⋯,X_p} has p components and the edge set E consists of pairs (i, j), where (i, j) ∈ E if there is an (undirected) edge between X_i and X_j. There is an edge between two nodes X_i and X_j if and only if X_i and X_j are conditional dependent given all other variables {X_K, k ≠ i, j}. It is well known that the conditional independence between X_i and X_j given other variables is equivalent to that the corresponding element in the precision matrix is zero [11,12], i.e., ω_ij = 0, where the precision matrix is the inverse covariance matrix Ω = (ω_ij) = Σ⁻¹. Given n i.i.d and samples X⁽¹⁾,⋯,X⁽ⁿ⁾, the goal is to make statistical inference of each edge in the graph, or equivalently of each ω_ij.

In the classical low-dimensional setting, in which p is fixed while n goes to infinity, it is natural to apply the maximal likelihood estimator, the inverse of sample covariance matrix, to obtain the asymptotic normality estimation of each edge at rate. However, in modern applications of biological network, the dimension p is often far larger than n, so the inverse sample covariance matrix does not exist or is inconsistent. Motivated by the sparseness assumption of the graph, i.e., most ω_ij are zeros, Ren et al. and Sun and Zhang tackled the inference problem from another point of view of the model [1,13].

Assume we are interested in the partial correlation between the ith and jth variables, we model the conditional Gaussian distribution with index set A = {i, j}, , where X_C represents the coordinates of X indexed by C and Ω_C,D denotes the submatrix of Ω with rows and columns indexed by C and D respectively. This observation motivates us to consider the estimation of individual element ω_ij by estimating the noise level Ω_A,A in the bivariate regression of X_A against . To be specific, we regress the ith and the jth columns X_A of the n × p dimensional data matrix X = (X⁽¹⁾,⋯,X⁽ⁿ⁾)′ against the remaining columns based on the equation , where the true coefficients is sparse due to the sparseness structure of the graph and rows of ϵ_A are i.i.d. Gaussian vectors with mean zero and covariance . A tuning-free penalized approach, scaled Lasso was used for this regression to obtain an estimator of β_A as well as the residue . Then the final estimator can be formally written as . It is worthwhile to note that the method is tuning-free and thus can avoid the statistical problems arisen from the cross-validation procedure, and the estimator is proven optimal in the decision theory framework. In particular, when the network graph is sufficiently sparse, the number of active variables or non-zero edges is not too large relative to the sample size, i.e., the maximum node degree of the graph satisfies , besides the bounded eigenvalue condition on Ω, the estimator is asymptotically efficient in the sense that . A closely related quantity, partial correlation, which has a natural interpretation for the strength of conditional dependence, is formally written as with the property .

Ren et al.’s theoretical study provides asymptotically normal estimation of each edge in GGM under a minimal sparseness assumption, hence can provide p-value and confidence interval for each ω_ij [1].

FastGGM algorithm

Although theoretically Ren et al. have developed a justified tuning-free inference methodology [1], it is still not very clear if it can handle real high-dimensional datasets in a reasonable time. While no software was provided by Ren et al., later Chen et al. developed an R package named “ANTAC” for a naïve implementation of the methodology [14]. However, as the procedure of the method implies, in order to obtain asymptotically normal estimators of all edges, an order of O(p²) runs of scaled Lasso regressions need to be implemented, which make the computation with ANTAC for tens of thousands of variables extremely challenging and even infeasible.

In this study, we develop a tuning-free and efficient algorithm to accelerate the implementation and make the method of Ren et al. computationally attractive and feasible with tens of thousands of variables. Specifically, by focusing on the cyclical coordinate descent method for Lasso regression [15], our new algorithm pre-calculates and saves the sample covariance matrix which is used for each single run of scaled Lasso regression later. As a result, the new algorithm is able to avoid repetitive computation of this major step for all of O(p²) runs (or O(sp) runs, see Discussion below the flowchart) of Lasso and thus its computational cost is significantly reduced. Our fast algorithm is not an approximate version but provides exact solution of the method proposed by Ren et al. and thus can accurately generate partial correlation, p-value and confidence interval of the edge between any two nodes in the studied network. We outline a detailed procedure below (Fig 1), followed by some discussion on the advantage of our algorithm:

1. Standardize each variable in data matrix X:
   1. Center each column of X to have mean zero and get matrix X_c;
   2. Scale each column of X_c by for all i = 1,⋯,p, where ‖⋅‖ denotes the vector norm, and get matrix X_s;
2. Calculate sample covariance matrices for the purpose of accelerating Lasso regression using coordinate descent optimization based on covariance updates [16,17], they are:
   1. IP_YX, where IP_YX_ij is the inner product between the column i of X_c and the column j of X_s;
   2. IP_XX, where IP_XX_ij is the inner product between the column i and column j of X_s. It can be rapidly calculated via multiplying each column of IP_YX by ;
3. Apply scaled Lasso regression (SLR) to each variable i against all other variables i^c:(1)
   1. Fix σ and solve a fast Lasso regression based on the pre-calculated covariance matrices to get β (See the pseudo code of fast Lasso solver in S1 File);
   2. Fix β and update ;
   3. Iterate steps a and b until σ and β converge to get and ;
   4. Save and residual for variable i respectively into the i column of a p × p matrix B = (b_ij) with 0 diagonal and a n × p matrix ;
4. Estimate conditional dependence relationship for each pair of variables (e.g. i and j, i ≠ j). Results from step 3 allow reducing computations of SLR due to the sparse assumption:
   1. For the variable i, get residual of SLR against all other variables except variables i and j (i.e. {i, j}^c):
      1. If b_ji = 0 in the saved B matrix, directly get from column i of the pre-saved matrix;
      2. If b_ji ≠ 0, apply SLR (2) in the same way as that in step 3 to obtain the residue ; (2)
   2. For variable j, similarly get residual of regression against variables {i, j}^c:
      1. If b_ij = 0 in the saved B matrix, directly get from column j of the pre-saved matrix;
      2. If b_ij ≠ 0, apply SLR (3) to obtain residue ; (3)
   3. Calculate precision matrix for variables i and j: (4)
   4. Estimate p-value and confidence interval for ω_ij from distribution: (5)
   5. Calculate partial correlation ;
   6. Estimate p-value and confidence interval for γ_ij from distribution:(6)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of FastGGM algorithm.

https://doi.org/10.1371/journal.pcbi.1004755.g001

The advantage of our fast algorithm over the algorithm in the software developed by Chen et al. is mostly due to steps 2 and 3. First of all, step 2 is completely new and can save computation in all runs of SLR in steps 3 and 4. To see the reason, we take a close look at the covariance updates in the coordinate descent optimization for Lasso regression, which is reflected in the line 15 of the pseudo code (See S1 File). For each single run of Lasso regression, inner products of response and design matrix as well as sample covariance of the design matrix need to be computed in advance (O(np²) operations). Although initial computation procedures for different Lasso regressions in steps 3 and 4 are slightly different, they all rely on a single sample covariance matrix of the data matrix X. Realizing this fact, the number of operations can be significantly reduced from O(np⁴) (or O(nsp³)) to only O(np²) since there are O(p²) (or O(sp)) runs of Lasso in steps 3 and 4. Secondly, as briefly commented by Ren et al. [1], the extra p runs of scaled Lasso in step 3 can further significantly save the computation in step 4 and reduce the required number of runs of scaled Lasso from O(p²) to O(sp). This is because whenever b_ji = 0, the solution of scaled Lasso applied to the ith variable against all others (saved in step 3) is equal to that of SLR in (2). Since the maximum node degree is s, we expect there are no more than s nonzero elements in each column of B.

Besides the fast algorithm, we develop FastGGM package with R language and two libraries, “Rcpp” [18] and “RcppParallel”, which help implement C++ functions in R and significantly speed up the loop operation. The package has three main functions: “FastGGM” is for analyzing GGM with one CPU, “FastGGM_parallel” is for analyzing with parallel computation, and “FastGGM_pairs” is for analyzing the conditional dependence for specific variable pairs. The first two functions output matrices of precision and partial correlation as well as the corresponding p-value and confidence interval for every pair of variables. The last function outputs vectors of precision, partial correlation, p-value, and confidence interval for the specified variable pairs. According to Section 5.1 in Ren et al.’s work, the default value of penalty parameter λ in our package is set as .

Simulation analysis

To illustrate the efficiency of FastGGM, we simulated data as follows: Firstly, we generated the upper triangular of a p × p sparse precision matrix Ω with diagonal element being 4, where the probability of each off-diagonal element being nonzero was π and its value was randomly set as 0.3, 0.6 or 1 with equal probability whenever the element is nonzero. Next, we made Ω symmetrical and set the elements of its bottom right block, whose size was , as twice of the elements of its top left block. Finally, we inversed Ω to get a covariance matrix, and generated a n × p data matrix X by randomly sampling from a multivariate Gaussian distribution N(0, Ω⁻¹). Note that Ω is invertible with high probability by our construction. Eleven models with different combinations of parameters {π, p, n} were considered in our data simulation (See Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Performance of estimating the precision matrix.

https://doi.org/10.1371/journal.pcbi.1004755.t001

Next, we estimated the underlying precision matrices from the simulated data matrices. For each precision matrix, we randomly generated 100 data matrices and respectively applied FastGGM on them. Then we compared differences between the true precision matrix and 100 estimated ones. We calculated Pearson correlation coefficient (PCC) and Chebyshev distance (ChebDist, defined as the greatest difference along any coordinate dimension) between two vectors consisting of the upper triangular entries (because the matrices are symmetric) to measure the differences. Ideally, a good estimation should present large PCC and small ChebDist. Table 1 shows the mean differences of 100 estimations for each precision matrix. It is shown that when p is fixed (e.g. 400) and n increases, PCC gets larger and ChebDist gets smaller; while n is fixed (e.g. 400 or 800) and p increases, PCC turns smaller and ChebDist becomes larger. This means that data with more samples or less variables are better for the inference of GGM. Besides, we evaluated the estimation of nonzero entries in the precision matrix, which could be considered as conditional interactions between the variables. We calculated mean type I error of 100 estimations at the level of 0.01 using the output p-values. The results in Table 1 show that FastGGM controls type I error well. In addition, we generated Receiver Operating Characteristic (ROC) curve, which presents the true-positive rate against false-positive rate at various threshold settings, using the estimated p-values and the truth of zero and non-zero entries in the simulated precision matrix, and then computed the Area Under the Curve (AUC) to measure the performance of estimation. The larger AUC represents the better estimation. Table 1 also lists the mean AUC of 100 estimations for each precision matrix model, all AUCs are large enough to demonstrate that the estimations are accurate, and when p is fixed AUC consistently goes higher as n increases, which again indicates that large sample size is beneficial to the estimation of conditional interaction relationship among variables. Note that Ren et al. had compared the estimator of the entire sparse precision matrix via the current available algorithm after certain data-driven thresholding procedure with the existing penalized methods, such as GLasso and CLIME, and reported better performance [1], we additionally compared with another Spectral method proposed by Honorio and Jaakkola with an penalty [19]. As a result, FastGGM outperformed Honorio’s method on estimating the sparse precision matrix after the thresholding procedure (See results in S2 File).

FastGGM is designed to accelerate the inference of GGM on large-scale data sets based on the theoretical work proposed by Ren et al [1]. Note that Chen et al. developed the R package ANTAC, in which a function ANT_GGM is a naïve implementation of the original method [14]. In simulation studies, we compared the computational time of FastGGM on each simulation data set with the ANT_GGM and another fast stock method GLasso [4] that was implemented in R package “huge” [20]. Table 2 shows that FastGGM is much faster than ANT_GGM on all data sets. In particular, our algorithm is still feasible for even p = 5000 or 10,000 while ANT_GGM clearly cannot handle the settings. It is worthwhile to point out that besides the structural advantages of our algorithm over the naïve one implemented by ANT_GGM, there is another important aspect for saving the computational time complexity: we implemented C++ functions in R environment for efficient loop computation with the help of Rcpp and RcppParallel libraries. On the other hand, Table 2 also shows that the computational time of FastGGM is comparable with huge_glasso and even faster when applying parallel computing with 10 CPUs. The parallel computation is another big advantage of FastGGM and makes it more feasible in real applications. Besides, note that GLasso is only able to provide point estimation of the precision matrix without inference results, while FastGGM can additionally provide p-values and confidence intervals.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of computational time.

https://doi.org/10.1371/journal.pcbi.1004755.t002

Global gene association networks in childhood asthma

We applied our algorithm on a public gene expression microarray data set, E-MTAB-1425 [21], which was measured from the human lymphoblastoid cells of 258 asthmatic children and 134 healthy children, to estimate the global gene association networks in normal and disease cells and compare the differences between the two networks.

We downloaded the normalized data from the EMBL-EBI ArrayExpress database [22], and averaged the expression values of probes that represented the same genes in each sample. Then we removed genes that had a mean or standard deviation ranked below 40%, and remained the expression data of 10,049 genes. Next, we filtered 111 genes that were highly associated with either age or gender (FDR < = 0.01 under univariate regression), and generated two expression matrices of 9,938 genes for the disease and normal samples. After this preprocessing, we respectively utilized FastGGM on the two matrices to estimate the corresponding GGMs of genes. From the results we selected significant gene pairs, whose partial correlations had FDR < = 0.01, and constructed the gene association networks under the two conditions. A Venn diagram of the edges and nodes in the two networks (Fig 2A) showed that the disease network had more edges than the healthy network (11,139 vs 8,373), and only 561 edges were overlapped, although there was a large proportion of nodes overlapped between the two networks. These indicated the associations among genes changed a lot when the disease occurred. Comparing the vertex degrees (i.e. the number of nearest neighbors of a vertex) of the two networks, we found that the disease network had more genes with large degrees (Fig 2B). Besides, the disease network consisted of 276 clusters, which represented the maximal connected sub-networks, while the healthy network had 453 clusters. These observations indicated that the connection of genes in disease samples was stronger than that in healthy samples.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparing gene association networks under asthmatic and healthy conditions.

A) Venn diagram of the edges and nodes in the asthmatic and healthy networks. B) Distributions of vertex degree in the two networks.

https://doi.org/10.1371/journal.pcbi.1004755.g002

Analyzing differences of gene networks between the asthmatic and healthy conditions could be helpful for understanding the genetic mechanisms of the disease. Therefore, we merged 18,390 unique edges that only belonged to one network to construct a differential gene association network. For simplification and visualization, we selected the top 3 hub genes and extended with their nearest neighbor genes to generate a differential sub-network, which included 80 genes and 84 edges (Fig 3). In this network, many genes have been reported relevant to asthma. For example, TNF/TNF-α is a pro-inflammatory cytokine that has been implicated in many aspects of the airway pathology in asthma, and anti-TNF-α has been demonstrated as a potential therapy in severe refractory asthma [23]; MIF plays important function in the immune pathogenesis of asthma via the promotion of TH2 responses, and its inhibition may be therapeutically beneficial to asthma [24]; The hub gene CLK1 has been shown significantly down regulated in alveolar macrophages by diesel exhaust particles, which can induce or aggravate pulmonary diseases including asthma [25]. In addition, functional analysis on this sub-network indicated the most significantly enriched KEGG pathway of these inclusive genes was “Protein processing in endoplasmic reticulum” (Fisher’s exact test, p-value = 2e-8). It has been approved that endoplasmic reticulum stress and the related signaling networks are important modulators of inflammatory and immune responses in the development of allergen-induced bronchial asthma, especially severe asthma [26]. In short, our analysis on the gene expression data of asthma manifests that FastGGM is useful for studying large-scale biological networks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Sub-network of differential gene-gene associations between asthmatic and healthy conditions.

The sizes of nodes are proportional to their degrees.

https://doi.org/10.1371/journal.pcbi.1004755.g003

Synaptic protein networks in Alzheimer’s disease (AD)

Then we applied our algorithm on a proteomic data set coming from 59 AD subjects with mild to moderate dementia severity. The objective of this study was to use targeted mass spectrometry to determine levels of a number of synaptic proteins and to construct networks of these proteins. Post-mortem tissue samples from the dorsolateral prefrontal cortex (DLPFC) of the subjects were obtained from the University of Pittsburgh Alzheimer Disease Research Center Brain Tissue Bank. It was of interest to examine the synaptic protein networks for the AD subjects and to detect modules of highly interconnected proteins from the network to elucidate the disease physiology.

Protein from gray matter homogenates were assayed in triplicate using targeted mass spectrometry with a stable isotope labeled mammalian brain standard to quantify 283 peptides from 192 synaptically expressed proteins. In the data processing step, for each protein, a protein-level measure was derived by calculating the weighted average of all standardized peptide measures mapped to that protein, where the weights are inversed to the percent of coefficient of variance (CV) of the peptide measures. S1 Table provides the protein-level data for these 59 AD subjects (protein names were coded). We then applied FastGGM to construct a partial-correlation-based protein network for all the 59 AD subjects. For each pair of proteins, we tested whether the partial correlation was zero or not. As a comparison, we also performed marginal-correlation-based analysis to examine the co-expression network of the genes or proteins using the weighted gene co-expression network approach (WGCNA) [27].

We used bootstrap aggregation suggested by Schafer et al. to achieve more reliable and stable estimation [28], since the sample size in this case is small (n = 59). Specifically, we bootstrapped 200 times (with replacement) and computed the graph using FastGGM for each bootstrap. We then averaged over the 200 graphs and obtained a “bagged” partial correlation matrix for the AD group. We then tested whether the partial correlation is zero or not for each pair of proteins. In addition, we performed module detection using the hierarchical clustering approach based on partial-correlation-based network matrix and marginal-correlation-based network matrix, respectively.

Fig 4 presents the heat map of the partial-correlation-based network constructed through FastGGM and the heat map of the marginal-correlation-based network constructed through WGCNA. We selected the same parameters, such as the soft power and minimum module size, in both network construction and module detection. With the marginal correlation approach, most of these synaptic proteins are found strongly connected (reflected by the overall red block in the heat map). There are two modules being detected. However since the entire correlations are so strong, these two modules are not really separable. In contrast, with the partial correlation approach, the overall network is much sparser. There are four modules being detected. Since the partial-correlation-based network quantifies the correlation between each pair of proteins with the effects of other proteins excluded, it only keeps the correlations due to direct causal relationships between the protein pairs, while the correlations originated via intermediate proteins are eliminated. This provides more useful information in elucidating the disease physiology, especially in the case when there are strong marginal correlations among most of the proteins so that the true underlying causal correlations are undistinguished.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Heat maps of synaptic protein network in AD cohort where red indicates stronger correlation and the white indicates weaker correlation.

The left and top color bars indicate the module membership of each protein (grey colored proteins do not belong to any module), with the corresponding hierarchical clustering dendrograms plotted. The left is the heat map based on partial correlations and the right is the heat map based on marginal correlations.

https://doi.org/10.1371/journal.pcbi.1004755.g004

With a FDR threshold of 0.001, 62 pairs of proteins are identified to have significant partial correlations, while 15289 pairs have significant marginal correlations. Table 3 lists the top 10 pairs of proteins that have the most significant FDRs of partial correlations, and the marginal correlations as well as corresponding p-values and FDR values are also listed for comparisons. It is interesting to point out that although most protein pairs with significantly strong partial correlations also have strong marginal correlations, there is a special pair, SLC25A3 ~ VGLUT3, which have strong negative partial correlation but weak marginal correlation. Actually, VGLUT3, which is also known as SLC17A8, belongs to solute carrier transporter superfamily and transports substrates in cells like SLC25A3. And the genomic locations of the two protein genes are very close on chromosome 12. These proteins are worthy of additional verifications for their interaction and functions in the AD physiology. A simple statistical explanation of this interesting phenomenon, that two variables have significant partial correlation but non-significant marginal correlation, could be like this: some confounders affect the correlation between the two variables through other variables, after adjusting for the confounders, i.e. conditioning on other variables, their correlation may be amplified.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Top 10 pairs of proteins with significant partial correlations from the AD study.

https://doi.org/10.1371/journal.pcbi.1004755.t003

The new algorithm finished GGM estimation over all 200 bootstraps with 4 CPUs in about 10 minutes. This example demonstrates that in the situation where the sample size and/or the number of variables are small, our FastGGM is still attractive, especially when resampling techniques are used with a large number of Gaussian graphic model estimations being performed.