1.1 Exponential family random graph models

Consider an order-n graph, G, represented via adjacency matrix Y on support , such that Y_ij corresponds to the state of the edge between vertices i and j; we make no particular assumptions about (e.g., it may consist of directed or undirected graphs, with or without loops, and may be valued), save that all elements of are real and finite. An exponential family random graph model (ERGM) for Y is then given by (1) where is a vector of real-valued sufficient statistics capturing network features of interest (which may implicitly incorporate e.g., nodal or dyadic covariates) and is vector of (curved) model parameters mapped to canonical parameters [19]. The reference measure h determines the baseline behavior of the ERGM distribution when η(θ) = 0, and plays an important role in fixing the shape of the distribution when edges are valued [32]. In general, computation involving (1) is challenging due to the intractable nature of the log-partition function (i.e., normalizing factor), , as is extremely large ( in the binary case), the summand is generally too rough for naive Monte Carlo strategies to converge well, and ψ rarely has a closed-form solution. In the context of iid draws from the same ERGM pmf, we obtain the (homogeneous) pooled ERGM, (2) where Y = (Y¹, …, Y^m) is a vector of random graphs with realizations y = (y¹, …, y^m). Although most work focuses on the single-graph case, our emphasis here is on the case where m > 1 (either because of multiple graph observations, or—in the case of conjugate prior inference—because of “effective” prior observations that are equivalent to an increased m).

As exponential families, the ERGMs have a number of convenient properties of which we will make use [35]. Subject to mild regularity conditions, we may define an invertible function that provides the mean value parameterization of an ERGM on random graph Y. From Eq 2 it is evident that the corresponding function is simply a constant multiple of the base mean value function for a single graph (foreshadowing a property that we employ below). Likewise, the Fisher information matrix of Y is given by , with the pooled equivalent being I_m(η) = m I(η).

2.1 Maximum likelihood inference for pooled ERGMs

We begin with the simple case of maximum likelihood inference. Given IID ERGM observations y^obs = (y¹, y², ⋯, y^m), the joint log-likelihood follows immediately from Eq 2, (3) the maximizer of which () is the maximum likelihood estimator (MLE). As observed, the primary challenge in finding the MLE is in dealing with the log normalizing factor, ψ. Running a Markov chain over the states of each of the m graphs in the set can be used to accomplish this, or equivalently (as is done in e.g., [58]) running a single Markov chain on a combined graph of order nm containing the union of all individual graphs, but the form of Eq 3 shows that this is superfluous in the IID case. Specifically, observe that any maximizer of ℓ is also a maximizer of any positive constant multiple of ℓ, and thus where is the arithmatic mean of the observed statistics, and is the geometric mean of the reference measure over the observed graphs. Since the latter does not depend on θ, we may further simplify the above to (4) which is immediately recognizable as the MLE for a hypothetical single “pseudo-graph” of order n with whose statistics are the means of the observed statistics. It is thus possible to find the MLE for a pooled model on m graphs by fitting a single-graph model (a considerable simplification).

To see the corresponding implications for the sampling distribution of the MLE, we note that inference for θ benefits from standard asymptotics in m (see e.g., [35]), including the consistency and asymptotic normality of the MLE under suitable regularity conditions. In particular, if is the MLE for Y with m observations drawn from a pooled ERGM with parameter θ₀, then it follows from standard exponential family theory [68] that (5) where I(θ) can be obtained from I(η) via the chain rule, i.e., . It thus follows that the asymptotic variance-covariance matrix of the MLE in the m-graph case is equal to that of the single-graph case, divided by m; i.e., (6)

It follows, then, that we may perform maximum-likelihood inference for y^obs with arbitrarily large m at no greater cost than fitting to a single network (and without the use of customized software tools): we simply find the MLE for the a single (imaginary) graph with statistics equal to the mean of the observed statistics using any standard method (e.g., MCMCMLE [30] or stochastic approximation [65]), and then rescale the associated variance-covariance matrix by a factor of m to correct for sample size. This procedure is summarized in Algorithm 1. When m is large, this can result in considerable computational savings; although the trick is quite trivial to implement, it has not to our knowledge been employed for ERGM inference in prior work.

Algorithm 1 Maximum Likelihood Inference for a Pooled ERGM Using Mean Values

Input: Observed data y^obs

1: Compute

2: Find

3: Find

Output:

2.2 Conjugate maximum-a posteriori inference for ERGMs

We now consider the problem of IID pooled ERGM inference in the Bayesian setting. Given prior π(θ), we are interested in the posterior distribution of θ, π(θ|y^obs), (7)

Our focus here is on conjugate priors, in the canonical exponential family context for which η(θ) = θ. In addition to their mathematical convenience, conjugate priors are attractive in the context of exponential families due to their interpretability (being able to be expressed in terms of prior “pseudo-data,” consisting of a prior “mean” and effective “sample size” expressed in the same units as the observed data), the fact that they admit natural non-informative limits, and their status as maximum entropy distributions [69]. To our knowledge, conjugate priors for ERGMs were first examined in the unpublished work of [70], who considered them along with a number of other ERGM prior specifications, but to date they have not been extensively studied. As we show, ERGM conjugate priors allow for extremely computationally efficient inference via their mean value representation. Moreover, there are particularly natural choices of weakly informative conjugate priors that are well-suited to regularization; we consider these in section 2.2.1.

For a canonical ERGM family with η(θ) = θ, conjugate priors take the following form [54]: (8)

Here, are prior expected values of the vector of sufficient statistics, and n₀ is a positive number that measures the confidence in those prior expectations, which can be viewed as the number of pseudo-observations’ worth of information (in units of observed graphs) contained in the prior; denotes the normalizing factor that makes a legitimate probability density function of θ. The existence of such distribution is ensured by [71], who showed (8) is normalizable provided that n₀ > 0 and lies in the interior of convex hull of the support of the measure θ. Substitute (8) for π(θ) in (7), we have (9) where , taking values in [0, 1]. With (9), we note that an analytical form for the prior is not necessary for Bayesian inference, because the prior can be fully characterized by δ (or n₀) and . Standard Bayesian theory tells us that the posterior expectation of ∇ψ(θ) is the Bayes estimate of θ with respect to quadratic loss [72], and is a weighted average of and , with δ controlling the relative weight of contribution of the prior information. For any given prior hyperparameters , as the sample size m becomes large, δ, the relative prior weight, approaches zero, and hence the sample-based information dominates the posterior.

Given a prior specified by δ and , the maximum a posteriori probability (MAP) estimate , is also Bayes estimate under a different choice of loss function (the 0–1 loss; see for example [72]). Note that since MAP is indeed the maximizer of the kernel of posterior density (9), we can employ the same arguments as in the derivation of (4), to obtain (10)

It follows, then, that the pooled ERGM MAP estimator is equal to the MLE that would be obtained for a single pseudo-observation with sufficient statistics .

Under standard regularity conditions, the posterior distribution becomes asymptotically Gaussian as m → ∞, according to the classical Berstein-von Mises theorem [73]. Following the same basic “mean value” procedure used in Algorithm 1 for obtaining the pooled ERGM MLE , we are able to compute the MAP estimate by fitting an ERGM to a single ‘pseudo’-graph whose node set is the same as the observed networks but whose network statistics are taken to be equal to . In addition to the MAP estimate, we can also obtain an estimate of the observed Fisher information , which is approximately the negative Hessian of log-posterior generated by product of the prior and the likelihood of a single ‘pseudo’-graph. However, the Laplace approximation of posterior distribution requires the Hessian of true log-posterior, which should be generated as by the product of prior and the likelihood of all actual observations. Note that the negative Hessian matrix Q_m,δ(θ) of true log-posterior (9) can be approximated by = . Laplace’s approximation of the posterior yields [74] the following result, (11)

We complete the approximation by noting that , which can be obtained by Markov Chain Monte Carlo simulation [30].

Putting the pieces together, Algorithm 2 provides a simple procedure for performing MAP inference for pooled ERGMs under conjugate priors. We begin by specifying the prior parameters and n₀, and computing the mean data vector and relative prior weight δ. The key steps are lines 3–4, which obtain the MAP estimate and associated approximate posterior variance-covariance matrix by performing the same calculations as are required for obtaining a single-graph MLE and its sample variance-covariance matrix: we simply fit to the posterior expectation instead of to an observed data value, and then adjust the information matrix to reflect the total posterior weight (prior pseudo-observations plus m). Not only does this allow us to perform inference for large-m data sets at no additional cost (as we did for the MLE), but it also allows us to perform Bayesian inference using algorithms and/or software implementations that were designed for maximum likelihood inference (or for first-order method-of-moments, which corresponds to maximum likelihood in this case) without additional modification.

Algorithm 2 MAP Inference for a Pooled ERGM Using Mean Values

Input: Observed data y^obs, prior data expectation and sample size n₀

1: Compute

2: Let δ = n₀/(n₀ + m)

3: Find

4: Find

Output:

In addition to its computational convenience, we note that the posterior expected statistic has an intuitive geometric interpretation as a convex combination of the prior information and the observed information, with the respective weight being determined by the relative size of the prior weight n₀ versus m. In particular, note that as δ → 0, we approach the MLE, while the prior becomes unchanged by the observed data in the limit as δ → 1. We examine this behavior in greater detail below. We also observe that so long as lies in the relative interior of the convex hull of , then exists (and is unique). This suggests the use of conjugate MAP to address a common practical problem in ERGM inference, namely the non-existence of the MLE when the observed statistics g(y^obs) lie on the face of the convex hull of possible statistics. In such cases, there is a direction of recession within the parameter space, with respect to which the MLE diverges; often, however, such divergent parameter values arise from very minimal information, as when a small subset of vertices in a sparse graph have no ties to each other (leading to a divergence in the corresponding homophily term). Use of MAP inference with a small δ can improve performance in these cases by acting as a regularizer, shrinking in extreme parameter estimates that have little support from the likelihood without otherwise greatly altering the solution. We examine this further in Section 2.2.1.

3.1 Behavior of the MLE in pooled-likelihood inference

For our first study, we vary the sample size m and examine the observed coverage rates of nominal 95% confidence intervals for model parameters as a function of sample size. Specifically, for each value of m, we generate K = 1000 datasets of size m from , performing pooled likelihood-based inference for each sample as discussed in Section 2.1. Respective burn-in and thinning intervals of 1 × 10⁶ and 2 × 10⁵ were employed for each simulated sample (for both data simulation and MCMC-MLE inference), with MCMC-MLE termination based on the ergm Hotelling criterion (an autocorrelation-adjusted T² test of expected versus target statistics obtaining p > 0.5). Table 1 presents the observed coverage rates of nominal 95% confidence intervals based on the asymptotic distribution of the MLE, as estimated from the size-corrected Fisher information obtained from a pooled (single-graph) estimate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Pooled MLE bias, standard error, and coverage rates of 95% Wald confidence intervals as a function of sample size.

https://doi.org/10.1371/journal.pone.0273039.t001

Table 1 shows the observed bias of the pooled MLE, as well as its standard error and the observed coverage rates of its nominal 95% CIs for all three model parameters under different sample sizes ranging from 1 to 100 graphs. Bias is negligible even for a single graph, declining to the levels of numerical noise once m is greater than 5–10. Likewise, efficiency (as measured by the standard error of the estimator) is high, and scales with in the manner expected from asymptotic theory. It is also evident that CIs based on the asymptotic distribution of Eq 5 perform well, maintaining approximately nominal coverage rates over a wide range of sample sizes. As a practical observation, we note that the construction of such CIs is based on statistical uncertainty, and does not take into account numerical sources of error (arising, e.g., from imperfect optimization, Monte Carlo error, etc.). (This is a slight oversimplification, as the single-graph information matrix estimates here (produced by the ergm library) do incorporate some MCMC error correction. However, it is difficult to account for all sources of numerical error, and in any event the theory of Eq 5 does not address it.) As m → ∞, the statistical error becomes arbitrarily small, thus increasing the proportion of de facto error arising from numerical approximation; put another way, it is possible to enter a regime in which our inferential precision is limited by our ability to compute the MLE (and ) rather than by the limits of our data. To ensure accurate coverage in such extreme-m scenarios, it may be necessary to adopt more stringent MCMC burn-in and thinning settings than are typically necessary for single-graph inference (where statistical uncertainty dominates), or to devise improved error estimates that better account for approximation error. That said, we do find excellent performance for sample sizes considered here, suggesting that the problem may be limited in practice. Further, we observe that the excellent coverage obtained for small m (even m = 1) provides practical validation of the traditional practice of using asymptotic confidence intervals in the m = 1 case; for a review of different types of ERGM asymptotics (and their relationship to classical results) see e.g., [35].

3.2 Prior weights and MAP inference

As discussed in Section 2.2.1, choosing the relative prior weight (δ) is an important aspect of the prior specification; while the choice of can often be made based on either prior data or domain knowledge, the impact of n₀ (hence δ) is less obvious. Here, we examine the impact of δ on the MAP estimate with a particular interest in identifying prior parameter values that are likely to serve as reasonable starting points for use in regularization. Our analysis looks first at the impact of δ on the MAP estimate itself (i.e., the extent of interpolation between the implicit prior natural parameter and the MLE), and then considers the effect of δ on the frequentist properties of the MAP estimate (bias, and the frequentist coverage of the posterior credible intervals).

To specify a prior, we first simulate homogeneous Bernoulli random graphs on the node set of the FMHS network, given expected mean degree fixed at the average degree of all the nodes in three comparable networks (i.e., Goodreau’s Faux Magnolia High School data, Faux Dixon High School data, and Faux Desert High School data [64]). The observed average degree across these data sets is 1.974, leading to an edge coefficient of ; for the Bernoulli family, the parameters for the other two terms are set to 0. (We note in passing that calibration of this kind should generally be done using mean degree rather than density, as mean degree is often close to size-invariant for comparable relations while density is not; see e.g., [26, 87].) We then calculate the average network statistics of 500 draws from this distribution, giving us the prior expected statistics . Since our focus is on δ, we fix our sample size at m = 1 and vary n₀ to obtain the posterior inference under different values of relative prior weights. We perform MAP estimation on 1000 independent realizations of for each choice of δ, comparing the resulting parameter estimates to their true values (θ₀) to assess the bias of MAP estimate and the frequentist coverage probability of the 95% posterior credible intervals arising from the Laplace approximation to the posterior distribution.

We begin by examining the impact of δ on the MAP estimate. As noted above, the MAP estimate must interpolate between the natural parameter equivalent of at δ = 1 and the MLE at δ = 0; equivalently, we may think of the conjugate prior as shrinking the estimate towards the (natural parameter equivalent of the) prior expectation. The detailed pattern of shrinkage is depicted in Fig 1, which shows that parameters change roughly linearly over most of the unit interval, with the most extreme changes occurring near δ = 1 (top panel). Importantly, shrinkage is approximately linear near the non-informative limit (δ → 0, bottom panel), suggesting that small differences in choice of δ do not have a large impact on the posterior mode (a convenient property when selecting minimally informative priors for regularization purposes). A more quantitative picture emerges from Table 2, which shows the mean MAP estimates for each parameter as a function of δ. We observe that choosing δ ≤ 0.02 yields estimates that are extremely close to the MLE (agreeing to 2–3 decimal places), while still placing sufficient weight on the prior to be useful for regularization (i.e., to ensure that the mean value parameter lies in the relative interior of the convex hull of possible statistics).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Mean MAP estimates of model parameters under different relative prior weights when m = 1.

(Data generating parameters θ₀ = (−5.885, 0.532, 1.867); all standard errors for table entries less than 0.0044.).

https://doi.org/10.1371/journal.pone.0273039.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Average MAP estimates for , by δ.

Solutions interpolate between the natural parameters corresponding to the prior (δ = 1) and the MLE (δ = 0); shrinkage is nearly linear in δ near the non-informative limit (bottom panel). Dotted horizontal lines show data generating parameters.

https://doi.org/10.1371/journal.pone.0273039.g001

We now turn to the frequentist properties of the MAP estimate, as a function of δ. Here we compare the MAP estimate (and the 95% posterior intervals arising from the Laplace approximation) to the coefficients of true model , which is θ₀ = (−5.885, 0.532, 1.867); at the other extreme, we have the natural parameter equivalent of the location of the conjugate prior (−4.63, 0, 0). Table 3 shows the estimated bias and frequentist coverage probability for our simulation sample, as a function of δ. As can be seen, bias is minimal until δ ≈ 0.02, becoming substantial for δ > 0.1. Likewise, the 95% posterior intervals maintain good frequentist calibration until roughly δ ≈ 0.02, though coverage degrades rapidly thereafter. For regularizing/minimally informative applications, a choice of n₀ ≈ 0.01 (giving the prior approximately 1% of the weight of a single graph observation) would seem to be a reasonable starting point.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Frequentist properties of MAP estimates and the Laplace approximation for model parameters under different relative prior weights, with sample size fixed at m = 1.

https://doi.org/10.1371/journal.pone.0273039.t003

4.1 Pooled ERGM analysis for brain functional connectivity networks

The study of group-based brain functional connectivity networks has become a topic of increasing interest in neuroscience, due the need to characterize both central tendencies and patterns of variation in interactions among brain regions. Importantly, it is of interest not only to measure specific or mean interactions, but to be able to characterize the distributions of interaction patterns arising under particular conditions, and/or within particular subpopulations. ERGMs have been identified as a promising tool for this purpose, due to their ability to assess how local brain network features give rise to the global structure, and due to their capacity to account for both heterogeneity and dependence among interactions [41, 88].

Brain functional connectivity networks often exhibit both functional segregation and integration [89], where functional segregation in the brain is the ability for specialized processing to occur within densely interconnected groups of brain regions, while functional integration corresponds to the ability to rapidly amalgamate specialized information from scattered brain regions. As an attempt to produce a model with appropriate network sufficient statistics that are able to capture those two concurrent opposing driving forces [42], proposed to first select the “best” metrics from a broader set of potential candidates identified in the literature using model selection techniques for ERGMs, then refit the networks of all subjects with those “best” metrics. They then employed the mean (respectively median) of the resulting individual estimates as estimates of a global, group-level “representative” whole-brain connectivity network model (which they refer to as a “mean” (respectively) “median” ERGM). This method of amalgamating models in the natural parameter space is straightforward and intuitive, but has several disadvantages: as shown in Eq 4, the appropriate pooling for a joint ERGM occurs in the mean value parameter space rather than the natural parameter space; separate estimation of an ERGM for each individual is computationally expensive (and, for the MLE, may encounter problems if some individuals’ networks have statistics that lie on the face of the convex hull of potential statistics); the sampling distributions of the amalgamated model estimates are unclear (especially in the median case); the amalgamated estimator is not in general consistent; and model selection by this approach does not exploit the joint likelihood (which may lead to an inferior pooled model).

By contrast, a pooled-ERGM approach provides a more principled and computationally efficient alternative to the mean/median ERGM approach. For large n, the properties of the pooled estimates and their confidence intervals are ensured by the large sample theory of exponential families, and as shown in Section 3.1 good results can be obtained with even modest numbers of graphs. Moreover, instead of having to fit each observed network separately, as proposed in [42] (with the risk that the MLE will not exist in one or more cases), exactly one ERGM fit is required (and the target statistics for that fit lie on the face of the convex hull only if all input networks do as well). Moreover, the ability to use conjugate-MAP inference for pooled ERGMs provides an inexpensive way of obtaining approximate Bayesian answers where desired, or (when viewing the prior as a regularizer) obtain regularized likelihood estimates. Here, we demonstrate all three approaches in the context of brain functional connectivity networks, building on prior work by [41, 42]. Due to the large number of model fits required for cross-validation, we use ergm’s stochastic approximation method for estimation in this section, with all Markov chains having a thinning interval of 5 × 10⁴ following 2 × 10⁵ burn-in iterations.

4.1.1 Data.

We consider the data reported in [41, 42], which includes brain functional connectivity networks among 10 normal subjects (5 female, average age: 27.7 years old, standard deviation: 4.7 years) who were part (Subject No. 002, 003, 005, 008, 009, 010, 012, 013, 016, 021) of a larger functional MRI study of age-related changes in cross-modal deactivations [90]. Fig 2 depicts the brain connectivity networks of subjects 002 and 003, illustrating both common properties (e.g., clustering, increased probability of ties within brain regions) and heterogeneity across networks; here, we are interested in capturing this distribution via an ERGM form. Note that brain connectivity networks are defined on equivalent sets of nodes, which here correspond to 90 prespecified brain regions (ROIs—Regions of Interest), according to the Automated Anatomical Labeling Atlas (AAL) [91]. Each of these 10 brain connectivity networks is represented by binary adjacency matrix, in which element (i, j) denotes the presence or absence of a functional connection between node i and node j. The establishment of binary functional connections was done by thresholding the temporal correlation coefficient adjusted for motion and physiological noise (see [41, 92] for further details), and hence those brain networks are undirected by construction. The thresholds were selected by the original authors at the subject level to give each network a mean degree of , or equivalently , where n is the total number of nodes (here, n = 90).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Brain connectivity networks of Subjects 002 and 003.

Colors (red, blue) indicate the different hemispheres (left, right); node coordinates are based on an non-metric MDS [93] solution for distances between regions of interest.

https://doi.org/10.1371/journal.pone.0273039.g002

Covariate information associated with these networks includes not only the nodal covariates Hemisphere and Area, but also an edge-level covariate for the spatial distance matrix among the ROIs (Mean: 76.28 mm, SD: 28.93 mm). The 90 regions are divided symmetrically across left and right hemispheres, with each hemisphere consisting of 28 areas as presented in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Number of ROIs in each area of the brain; names follow Simpson et al. (2011).

https://doi.org/10.1371/journal.pone.0273039.t004

4.1.3 Results.

Table 5 presents maximum likelihood estimates of model coefficients and associated standard errors for the group-based brain connectivity network model under pooling, enabling us to infer the extent to which each of the proposed effects shapes the overall distribution of networks across test subjects. (Predictive fit plots in S1 Fig) We see a positive and statistically significant parameter estimate for the GWESP statistic, indicating high levels of triadic closure net of spatial and anatomical features; this is compatible with the theory of functional segregation proposed in prior work. Likewise, we see that bridging is significantly disfavored (i.e., a negative effect for graphlet 1), suggesting that open triads tend not to persist (net of other factors). In estimating mixing effects, we aggregate all areas other than Frontal and Temporal to a single level as “Others” due to the small sizes of these regions, providing a tripartite mixing structure; Frontal-Frontal ties act as the reference category. We see inhibition of ties between different regions, and null or positive tendencies towards formation of within-area ties, which provides additional evidence for functional segregation. We note that these effects persist net of the overall inhibition of ties between more distant regions, with tie probability declining (ceteris paribus) as approximately one over the inverse of the distance between nodes. An important exception is the case of cross-hemispheric interactions, which are actually favored (the negative nodematch indicating that within-hemispheric interactions are disfavored relative to those that cross hemispheres). This can be viewed as an indicator of functional integration, with the need for coordination across hemispheres working against the general tendency against long-range ties. Care is required in the quantitative interpretation of the positive Edges coefficient, given the existence of log.spatial.dist-edgecov. Specifically, note that mean of pairwise distances of the ROIs is 76.28 mm and hence at the mean log(76.28)≈4.334, we have , conditional on the rest of the graph and all other effects held at zero, meaning that the baseline conditional probability of observing an edge (not involved in the creation of other network statistics included in the model) between pairs of regions at the average distance is still very low, as expected for sparse graphs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Model parameter estimates and standard errors for pooled-ERGM analysis of brain functional connectivity networks.

(For nodemix effects, Frontal-Frontal is the reference category).

https://doi.org/10.1371/journal.pone.0273039.t005

4.2 Approximate conjugate Bayesian analysis of brain functional connectivity networks

In this subsection, we demonstrate how one can conduct approximate conjugate Bayesian analysis as introduced in Section 2.2 for the dual purposes of approximating full Bayesian analysis and regularization. The construction of the prior is crucial regardless of the ultimate purpose. We adopt the simulation-based approach of Section 2.2.1 to specify the prior by noting that by construction (i.e., choice of correlation threshold) for all brain functional connectivity networks in this dataset, and thus we set is set to be equal to the mean of network sufficient statistics under a Bernoulli graph with (n = 90). The selection of relative weight δ is subject to vary depending upon the purpose, which is explored and discussed in detail with examples.

4.2.1 MAP estimation for the pooled model.

In the absence of strong a priori information regarding almost all aspects of the brain functional connectivity networks except for the mean degree, it is advisable to incorporate weak prior information; we do this by assigning a small value to the hyper-parameter δ, in this case setting δ = 0.02. Given a specified prior, we conduct Bayesian inference based on Algorithm 2, the resulting parameters being shown in Table 6. (Predictive fit plots in S2 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. MAP estimates and posterior standard deviations, conjugate Bayesian analysis of brain functional connectivity networks.

(For nodemix effects, Frontal-Frontal is the reference category).

https://doi.org/10.1371/journal.pone.0273039.t006

The parameter estimates from the Bayesian analysis are very similar to those of the pooled MLE, supporting the same qualitative conclusions. However, imposing a prior on the parameter vector permits interpretation of the results in terms of Bayesian answers, which may be useful in some settings; we may also use the Laplace approximation to sample from the approximate posterior, enabling us to obtain, for instance, posterior predictive distributions for network properties that take into account uncertainty in the model parameters.

4.2.2 Regularizing ERGMs with MAP.

As noted above, the MLE for the natural parameter of an exponential family distribution does not exist when the observed sufficient statistics lie on the relative boundary of C, the convex hull of the set of possible values of sufficient statistics. A common case of this type in ERGM modeling arises when mixing or differential nodematch parameters are specified for networks containing many small subgroups; if any of the associated statistics are equal to 0 (e.g., there are no observed ties between two groups), then the likelihood has no finite maximizer with respect to the respective directions in the natural parameter space. In the context of the brain connectivity networks, we observe that there are many small areas containing few nodes, potentially leading to such a circumstance. For instance, consider an extension of our previous model intended to quantify the mixing pattern between nodes in the Occipital and Cingulum areas; we may do so by augmenting with nodemix terms involving Occipital and Cingulum, with all other terms in the model unchanged. We denote this model as . It happens, however, that there there are no edges observed between Occipital and Cingulum for any of the networks in the dataset, and hence the vector of mean observed network sufficient statistics is no longer located in rint(C) (as the Occipital.Cingulum-nodemix value of 0 is smallest possible value that can be obtained). From an optimization perspective, we are unable to obtain a finite estimate for model coefficients of this augmented model, because the likelihood can always be further optimized by letting the vector of candidate estimates of model coefficients move towards the direction of recession. Statistically, this reflects the non-existence of the MLE. We now show such issues can be resolved by incorporating an appropriate conjugate prior into the inference to regularize the model and thus avoid extreme inferences on model parameters.

We construct a conjugate prior in the form of (8), where hyper-parameter is determined by calculating the mean of network sufficient statistics observed on 1000 independent random realizations of Bernoulli random graphs with p = 0.056. As our goal here is regularization, we view the prior as a convenient penalty function (rather than as a formal statement of prior knowledge), and treat δ as a hyperparameter subject to optimization. Given our pooled setting, it is natural to evaluate model performance by cross-validation (CV); specifically, we vary δ (or, equivalently, the prior sample size n₀), computing the expected squared Hamming error for each graph under leave-one-out CV based on 1000 draws from each simulated model, and select the value that minimizes the expected loss on the held-out networks. The Hamming error (i.e., the expected number of edge differences between a predicted draw from the model and the observed network) is a straightforward and interpretable metric for models on labeled graphs, where specific connections (as opposed to, e.g., general network properties) are meaningful and where there is sufficient covariate information to make prediction possible; in the case of the brain functional networks, where the vertices have distinct anatomical and functional significance that is conserved across subjects, optimizing the ability of the model to make edgewise predictions is a reasonable goal. We note, however, that other choices are also possible, depending on one’s objectives: for instance, an obvious alternative is the squared error in the predicted sufficient statistics, a quantity more closely related to the MLE. An attractive feature of CV is thus the freedom of the analyst to tune the model based on the needs of the problem at hand.

The results of the hyperparameter tuning process are shown in Table 7. As expected, the unregularized MLE (n₀ = δ = 0) yields suboptimal performance, with improvements obtained until n₀ = 0.004 (δ = 0.0004). Further increases in prior weight (here interpreted as the strength of the penalty function) result in diminished performance, as the fitted model is drawn towards the prior mean. We thus select δ = 0.0004 for subsequent analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Leave-one-out cross validation error for regularized inference on as a function of n₀.

https://doi.org/10.1371/journal.pone.0273039.t007

We may now perform penalized maximum likelihood inference, using the tuned conjugate prior as a regularizer. Table 8 shows the corresponding parameter estimates, standard errors, and significance levels for model . (Predictive fit plots in S3 Fig) As expected, the results for the shared effects (triangulation, spatial interaction, bridging, and hemispheric interaction) after breaking out the additional brain areas remain very similar to what was seen in the unregularized MLE for the collapsed model, though we now have a more complete description of the mixing pattern among localized areas. Importantly, we also observe that the Occipital/Cingulum parameter (for which the MLE does not exist) is now well-characterized. As we would expect from the fact that none of the observed networks had Occipital/Cingulum ties, the estimated coefficient is significantly negative; however, the magnitude is now plausible (and in line with the other observed effects).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Model parameter estimates and standard errors for regularized inference on .

https://doi.org/10.1371/journal.pone.0273039.t008

Finally, we note in passing that, while this final model captures a number of aspects of brain network structure, more improvement seems possible. As seen in S3 Fig, the model is fairly accurate in recovering observed triad census and degree structure (nearly all observed values falling within the 95% simulation intervals), but somewhat underestimates both mean geodesic lengths and the breadth of the ESP distribution. Thus, the model might be satisfactory for investigation of local structure, but less so for larger-scale structure (including heterogeneity in triadic clustering throughout the brain). For applications of this latter sort, further elaboration or alternative parameterizations may be desired. We revisit the question of tradeoffs between model complexity and predictive performance in Section 5.3.

4.3 Analysis of lysozyme structure networks via pooled ERGMs

The functions of proteins and other macromolecules are heavily influenced by their three-dimensional structure. With the increasing sophistication of both experimental technique and molecular modeling, new methods for analyzing the growing body of protein structure data are of increasing interest. Network analytic methods have emerged as particularly useful tools for this purpose, providing a rich representation for topological complexity while still offering substantial coarsening relative to atomistic structure. Among other applications, network representations of protein structure have been used to identify functionally important residues [96], summarize protein dynamics [97], identify functionally significant sub-units [98], distinguish active site conformations [7], and characterize structural differences between protein families [99].

One potential application of ERGMs in the context of protein structure is the characterization of variation within structures of the same protein (either in equilibrium, or in different functional or measurement contexts). ERGMs were first applied to protein structure networks by [95], who used them to control for intrinsic molecular features (e.g., chain membership) while testing hypotheses regarding fold-specific structure. In more recent work, ERGMs have been employed to characterize transient structure in intrinsically disordered proteins [40], and to model protein aggregation [8, 100]. Here, we consider the problem of characterizing variation in measured protein structures obtained via X-ray crystallography (the primary workhorse technique of modern structural biology). While it is common to treat globular proteins as having a native fold associated with a single three-dimensional structure obtained via crystallographic methods (or, more rarely, Nuclear Magnetic Resonance, neutron scattering, or cryo-EM), proteins in solution are extremely dynamic; even in a crystallographic context, repeated crystallization of the same protein will often yield slightly different structures. (In fact, the same crystal frequently contains several distinct conformations within a single asymmetric unit.) Currently, this variation is not well-characterized, and is often ignored (with a single conformation selected as “the” structure of the protein). Statistically, it is natural to think of these observed structures as being drawn from a broader distribution of low-energy conformations [101, 102], and to attempt to model this distribution using the measured conformations.

Here, we apply this notion to observed variation in crystal structures of hen egg-white lysozyme (a widely used reference protein in biophysical research). Lysozyme (N-acetylmuramide glycanhydrolase), is an enzymatic antimicrobial agent produced as part of the innate immune system. A glycoside hydrolase, lysozyme attacks polysaccharides within bacterial cell walls, compromising their integrity and ultimately causing cell lysis; as such, it is produced in large quantities in settings where bacterial growth must be discouraged (e.g., eggs, tears, milk). Our data consist of network representations of 66 independently solved lysozyme structures, each of which is formed from 129 residues (i.e., amino acids) constituting the main chain of wild type hen egg-white lysozyme (residues 19–147 of Uniprot B8YK79). Atomistic protein structures were obtained from the Protein Data Bank (PDB; https://www.rcsb.org/pdb/home/home.do), with the search query limited to X-ray crystallography structures containing only the 129-residue main chain with no modified or substituted residues, missing residues, ligands, or other complexes. Where more than one distinct conformation appeared in the asymmetric unit, each was isolated and treated as a separate conformation for purposes of analysis. Each isolated protein structure was protonated using REDUCE [103], with the resulting coordinates employed to generate a residue-level protein structure network (i.e., an undirected adjacency matrix of size 129 × 129) according to the pairwise distances among residues—any pair of residues is considered to be adjacent if they contain respective atoms that are closer together than 1.2 times the sum of their respective van der Waals radii. Two representative lysozyme structure networks are displayed in Fig 3; while the conformations are very similar, they do show subtle differences (compare e.g., the residues in the top right). A 3D molecular structure of lysozyme is shown in Fig 4, together with the equivalent protein structure network (PSN).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. 2D representation of Lysozyme structure networks (PDB 1AKI [104]; Left), (PDB 1BHZ [105]; Right).

Colors distinguish nonpolar (green) versus polar (blue) residues; node coordinates determined via topology and are not based on physical position.

https://doi.org/10.1371/journal.pone.0273039.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. 3D representations of lysozyme (PDB 1AKI [104]).

(Left) Molecular representation, showing backbone (ribbon), side chains, and surface; residues colored by index. (Right) PSN representation in similar orientation, with vertices positioned by Cα coordinates and colored by index.

https://doi.org/10.1371/journal.pone.0273039.g004

4.3.2 Results.

We perform conjugate MAP inference for the pooled ERGM model on the 66 lysozyme PSNs, using the above-specified model; estimation was performed using ergm under default settings incorporating the backbone-adjacency support constraint. The resulting parameter estimates are provided in Table 9. The model parameters can be interpreted based on the conditional log-odds of an edge between two nodes i and j, bearing in mind that many effects are necessarily simultaneous. For example, while the coefficient for the edges term is positive, it should be interpreted in the context of both mean spatial distances and sequence distances between residues. For example, the log mean distance between the Cαs of residue 4 and residue 9 is 2.143, and their log backbone distance is log(5) = 1.6. Ignoring all other effects, then the conditional probability of Y_4,9 = 1 based solely on these three terms would be [1 + exp[−(34.213 − 19.173(2.143) + 0.293(1.6))]]⁻¹ ≈ 0.002, indicating a low conditional probability of observing an edge; in practice, of course, all terms contribute simultaneously.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Conjugate Bayesian analysis of Lysozyme structure networks.

https://doi.org/10.1371/journal.pone.0273039.t009

As Table 9 shows, all three types of mechanisms play a role in predicting lysozyme network structure. Electrostatic and polar effects act as expected, with complementary charges increasing conditional tie probability and homophily for non-polar residues; although the posterior strongly favors homophily among polar residues, this effect is notably weaker than for the non-polar case.

Aromatic residues at first blush seem to have a lower baseline contact probability (with an additional negative effect for π-stacking), but these “intercept effects” must be weighed against the reduction in the Cα distance penalty for these residues. Let i and j be residues d Å apart, such that i is aromatic and j is not. Then the total effect of the Aromatic, π-stacking, and Aromatic distance effect terms on the conditional log odds of an i, j edge is approximately −1.091 + 0.646 log d; this enhances tie probability for d > 5.4Å, only suppressing it at close range. Likewise, for aromatic-aromatic pairs, the corresponding total effect is approximately −1.091 − 0.752 + 2(0.646) log d = −1.843 + 1.292 log d, which becomes favorable for d > 4.2Å. We would expect mixing among aromatic residues to be favored on physical grounds, and indeed this is true for residue pairs beyond 3.2Å. Overall, we thus see that interactions with aromatic residues are generally favorable (with aromatic-aromatic mixing especially favorable) except at very close range, with these residues particularly likely to interact with other residues over longer distances. Very short-range interactions are somewhat hindered for these residues, however, plausibly due to steric effects.

A similar effect is seen for residue size, with surface area having a negative main effect combined with a greater propensity for longer-range interaction; for residues i and j with respective surface areas s_i Ų and s_j Ų at distance d Å, the total effect of the surface area terms on conditional log odds is −0.04(s_i + s_j) + 0.024(s_i + s_j) log d, which becomes positive for d > 5.3 Å. For reference, the mean non-covalent nearest neighbor distance is approximately 3.8 Å, and the second-nearest is approximately 5.1 Å, so bulk is a positive interaction predictor for the vast majority of potential interactions. The minor inhibitory effect at very small distances, like that of aromaticity, may reflect steric hindrance.

Similar subtlety is seen in the case of the mildly positive effect of backbone distance—net of spatial distance—likely reflecting the tendency of the backbone to fold back on itself (creating strong bridges between parts of the protein that are distinct in sequence space). Note that, marginally, we find that contact probabilities fall off as roughly BBDist^−5/4, so this softening effect should not be confused with a net tendency for tie probability to increase with backbone distance. Rather, we find that when sequence-distant residues happen to be spatially proximate, they are particularly likely to be in contact. Less nuance is needed to interpret the effect of distance from the origin, or of the mean Cα distance between residues: both inhibit contact. The latter effect is, as has been observed, very large, in keeping with the constraints of a folded protein. Finally, we observe that net of everything else, existing contacts have an inhibitory effect of new ones (the negative 2-star parameter), cliques are strongly suppressed (negative triangle parameter), and there is an overall tendency towards triangulation net of clique suppression (positive GWESP parameter). To understand how these latter two effects combine, it is useful to consider the net change in the conditional log odds of adding a first, second, third, etc. shared partner to an adjacent residue pair (holding all other effects constant). The (base or direct) GWESP effect for the kth ESP is here 1.44exp(0.8)(1 − (1 − exp(−0.8))^k) = 3.20(1 − 0.55^k)) for k > 0 (or 0 otherwise); such a configuration also involves k triangles (with a total weight of −0.83k). The net contribution of these terms to the log odds of going from k to k + 1 shared partners is then the difference in the effect sums; for adding the first shared partner, this yields a net contribution of 0.62 (strongly favorable), falling to -0.03 for the second (approximately neutral), -0.39 for the third (unfavorable) and -0.59 for the fourth (strongly unfavorable). These penalties continue to increase, approaching the limiting value of -0.83 (the triangle coefficient). (Note that adding a shared partner also adds at least one 2-star (possibly as many as three, depending on the assumed baseline conformation), reducing the log odds contribution by an additional -0.05 to -0.15. This is, however, a much smaller effect). Although not often used together, a triangle term can usefully combine with GWESP in cases like this where triangle formation must shift from favorable to unfavorable as more shared partners accumulate, a phenomenon that may also manifest in other systems.

As a model adequacy check, we take 1000 draws from the posterior predictive distribution (based on the Laplace approximation), comparing the distribution of several standard structural properties (degree distribution, ESP distribution, geodesic distance distribution, and triad census) with the observed data means. The result is shown in Fig 5. As can be seen, the model is able to recapitulate all of the above features, indicating that it does a reasonable job of capturing the basic structural properties of the lysozyme networks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Model adequacy checks for the pooled lysozyme model; shaded areas/boxes show posterior predictive intervals, while red points indicate observed mean values.

(Note that some intervals in the lower left panel are narrower than box line widths; all intervals are in fact vertical). The lysozyme model successfully recapitulates a range of structural features.

https://doi.org/10.1371/journal.pone.0273039.g005

4.3.3 Reproducing structural variability.

As noted above, one potential use for ERGM analysis of protein structures is to characterize variability, and to identify dimensions of structural variation that may be imperfectly constrained by available data. Here, we simulate draws from the fitted lysozyme model and examine their range of variation with respect to four basic graph-level indices (GLIs) found by [99] to distinguish protein structures. These are:

• Transitivity [108]—a standard measure of triadic closure in network analysis, transitivity reflects the compactness of a PSN in the sense that higher levels of transitivity are associated with the structures that are closely and uniformly packed.
• Standard deviation of degree distribution—a measure of the level of heterogeneity in local packing around chemical groups.
• Standard deviation of the core number [109]—an indicator of the degree of heterogeneity in structural cohesion, which distinguishes between highly organized structures and structures that combine rigidly and loosely bound regions.
• Standard deviation of M-eccentricity—the idea of M-eccentricity stems from eccentricity [110], and was introduced in the context of PSN analysis by [99]. The M-eccentricity of a vertex is the mean distance from that vertex to all other vertices; vertices with low M-eccentricity are more centrally located, while those with high M-eccentricity are peripheral to the graph structure. Thus the standard deviation of M-eccentricity distinguishes between uniformly globular structures and structures with deformations or other elongations.

Fig 6 shows the distribution of the above GLI values for the observed lysozyme networks and for posterior predictive draws from the pooled ERGM; GLIs were calculated using the sna library for R [86]. For each of the GLI distributions, we can see that the posterior predictives cover the observed distributions, while being somewhat wider (reflecting posterior uncertainty). Such distributions have potential uses such as statistical comparison of protein families or variants from pooled crystallographic data, where accounting for uncertainty in the distribution of structural properties is an important consideration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Variation in graph-level indices for lysozyme PSNs, observed versus posterior predictive distributions; simulated networks cover the observed GLI range, but also account for predictive uncertainty.

https://doi.org/10.1371/journal.pone.0273039.g006

5.1 Missing data handling

Our technique depends upon the ability to compute ; when some graphs contain missing (i.e., unobserved) edge variables, their statistics cannot in general be calculated, and neither therefore can . Although we do not treat extension to the incompletely observed case in detail, we here briefly sketch an approach to this problem. Our proposed scheme uses the EM algorithm [111], with a data augmentation scheme related to those of [33, 112]. Given our sample y^obs = (y¹, …, y^m), let us divide the edge variables into observed () and missing () components. We begin with some initial guess at the parameter vector, θ⁽⁰⁾, and then proceed iteratively as follows. At the ith step: (1) draw using ; (2) compute (averaging over multiple imputation draws); and then (3) find via MLE or MAP as described elsewhere in the paper. This is repeated until convergence. This is a standard EM algorithm, with assumptions and solution properties equivalent to those of [112] (who use a nested MCMC strategy to perform Geyer-Thompson based maximum likelihood estimation in the single graph case).

We observe that the computational efficiency of this approach hinges on step (3), but can be undermined by step (1) in the case where missingness is extensive. In the extreme case in which all graphs contain significant amounts of missing data, this algorithm requires running MCMC over all graphs in the sample, and may not be dramatically more efficient than a conventionally pooled Handcock-Gile scheme. On the other hand, when missingness is confined to a small number of graphs, and/or when the number of edge variables to be simulated is small, then the savings from mean value inference in step (3) may still be considerable.

5.2 Pooling versus other approaches

As noted in the introduction, pooling is only one of several approaches for modeling graph sets (lying at one end of continuum that passes from pooling to hierarchical and mixed models, and thence to independent estimation). Pooling is a useful strategy when we either have reason to believe that our sample was drawn independently from a common generating process, or when we wish to use an approximation of this type. For the pooling procedure described here, we note that we are by assumption working with networks of the same size, sharing the same covariates; many common cases where heterogeneity is a serious concern (e.g., classrooms of differing size and composition) are thus beyond the scope of our technique. For graph sets on equivalent vertices, it may be helpful to assess pooling success using analyses like those in Section 4.3.3. If variation in observed network properties greatly exceeds those of the simulated models, then this may suggest that the sample arose from a mixture of generative processes; in this case, latent ERGM mixture models like those of [113] may be helpful in detecting and fitting parameters for the underlying mixture components. Alternately, such models may simply require elaboration. For instance, [114] shows that cross-graph variation in density and/or reciprocity can be captured in pooled models using an appropriate choice of ERGM terms. Excess cross-graph variation does not therefore imply that a pooled model cannot be successful, though it suggests that closer inspection may be in order. As in other matters, adequacy of the pooled approximation depends on the purposes of the associated analysis.

5.3 Pooling and model complexity

Models for complex networks can themselves become quite complex, raising practical, statistical, computational, and interpretational issues. How many terms are “too many,” and should one err on the side of simple models (which are easier to understand and work with, but that may omit important confounders or mechanisms) or higher-dimensional models (which may account for more drivers of network structure, at the expense of computational cost, interpretability, and overfitting risk)? This debate (analogous to the “emulative” versus “intellective” modeling discussion in the agent-based literature [115]) involves inherent tradeoffs between respective modeling strategies, and is in our view ill-posed: models at many different levels of complexity can be useful, and the objective should be to match the capacities and requirements of a model to the uses to which it will be put.

Pooling enters into this discussion in two respects. First, as noted above, models for populations of graphs may require greater complexity to account for cross-graph variation than models for a single graph. For instance, excess variation in density may be accounted for by a change of reference, together with the inclusion of terms for both log edge count and log null count (instead of the usual edge count statistic) [114]; this may account for greater structural variability, but at some cost to parsimony and ease of interpretation. Second, pooled models allow for considerably greater statistical power than single-graph models, particularly where sample sizes are large. This increase in power both makes it possible to reliably fit higher-dimensional models, and to detect very small deviations from the no-effect null hypothesis (i.e., one can very easily reject the hypothesis that a hypothesized term has no effect, even if the parameter has little impact on model behavior). In our view, both considerations strengthen the importance of substantively guided model selection based on a clear sense of modeling objectives. On the one hand, positing a model that will capture all aspects of a large graph set is rarely realistic, and one must thus choose what one will—and will not—seek to capture. And on the other hand, in the large-data regime one cannot count on statistical power or null hypothesis tests to tell one what one should or should not include in a practical model. Particularly as the methods presented here make it computationally feasible to fit models to very large graph sets, the analyst (at least in the equivalent vertex setting) is more often able to shift from asking what models they can fit to what models they should fit. We suggest that this shift motivates further work on substantive adequacy checking for network models in particular settings, particularly including efforts to link predicted network structure with other, non-network data or outcomes. For instance, Friedkin’s ([116]) experimental studies of influence in task groups reliably link equilibrium attitudes to influence networks in ways that suggest that attitudinal configurations could be predicted in part from network models even where the networks themselves cannot be observed; adequacy for this task may be rather different than what is prioritized by conventional metrics. As our ability to efficiently fit network models improves, pursuing such questions becomes an important priority.