Figures
Abstract
When modeling coexpression networks from high-throughput time course data, Pearson Correlation Coefficient (PCC) is one of the most effective and popular similarity functions. However, its reliability is limited since it cannot capture non-linear interactions and time shifts. Here we propose to overcome these two issues by employing a novel similarity function, Dynamic Time Warping Maximal Information Coefficient (DTW-MIC), combining a measure taking care of functional interactions of signals (MIC) and a measure identifying time lag (DTW). By using the Hamming-Ipsen-Mikhailov (HIM) metric to quantify network differences, the effectiveness of the DTW-MIC approach is demonstrated on a set of four synthetic and one transcriptomic datasets, also in comparison to TimeDelay ARACNE and Transfer Entropy.
Citation: Riccadonna S, Jurman G, Visintainer R, Filosi M, Furlanello C (2016) DTW-MIC Coexpression Networks from Time-Course Data. PLoS ONE 11(3): e0152648. https://doi.org/10.1371/journal.pone.0152648
Editor: Alberto de la Fuente, Leibniz-Institute for Farm Animal Biology (FBN), GERMANY
Received: October 30, 2014; Accepted: March 17, 2016; Published: March 31, 2016
Copyright: © 2016 Riccadonna et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: GeneNetWeaver Yeast & E. coli data The datasets (network and time-course) for the four Gene Net Weaver examples can be accessed on figshare with DOI doi:10.6084/m9.figshare.2279628, at the URL https://figshare.com/articles/Gene_Net_Weaver_Dataset/2279628. Metadata are included in the figshare project page at the same URL. Human T-cell data The data can be accessed via the data(tcell) function from the R package longitudinal, available at the URL https://cran.r-project.org/web/packages/longitudinal/index.html. Meta- data are included in the package description.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Inferring a biological graph (e.g., a Gene Regulatory Network) from high-throughput longitudinal measurements of its nodes is one of the critical challenges in computational biology, and several are the proposed solutions to this still unanswered question [1–3]. Although the problem is strongly non-linear, a simple but widespread solution such as the coexpression networks via correlation measures provides a good approximation [4–8], even outperforming more complex approaches [4, 9–11]. This follows from the observation that functionally related genes share similar expression patterns [12], implying that coexpression and functional relationships are correlated [13–15]. Pearson Correlation Coefficient (PCC) is the most used similarity measure [16–18], although alternative correlation functions can be also employed [19–21]. However, PCC lacks sensitivity in case of non-linear relations [22] and time shift between signals [23–25], thus the reliability of a coexpression network would benefit from a measure taking care of these characteristics.
Diverse inference methods have been proposed to include more flexibility and, more generally, to build networks that can capture higher order interactions better than coexpression. Recent reviews [7, 26–34] provide a map of the landscape on network inference to date, while comparison studies and competitions (notably the DREAM challenges series) have tried to detect the optimal solution. However, the widespread range of application of network inference prevents from individuating an elective algorithm, with each novel method claiming advantage in a particular context, often with overoptimistic results [35]. The integration of the predictions from individual methods has thus been proposed to complement advantages while limiting flaws [35–38], however with limited operability, due for instance to the early stage of the ensemble techniques in this area [39, 40] and to the heterogeneity of the challenged tasks [41]. In terms of mathematical techniques, there is a wide choice of approaches alternative to correlation, including mutual information (MI) evaluation, partial correlation estimation, Bayesian network, statistical physics approaches to dynamic systems. In particular, ARACNE [42] for MI and WGCNA [16, 17] for correlation (and the algorithms derived from these, see Methods section) stand as reference methods [4, 9, 28]. Similarity measures specifically targeted to longitudinal data have also entered the game [43], either as variation of existing general purpose methods, or just transferred from other domains such as signal processing, finance, climatology. Recent reviews on similarity functions for time series list a wide diversity of mathematical strategies, including e.g., cross-correlation, Hidden Markov Models, edit distance, Newey–West estimator, spectral distances [44–48].
In particular, two measures proved to be quite effective in tackling the two issues of non-linearity and time-shift: the Maximal Information Coefficient (MIC) [49] and the Dynamic Time Warping (DTW) [50]. MIC is a MI-based association measure aimed at detecting functional (linear and non-linear) dependencies between two variables [49, 51–53], part of the family of Maximal Information-based Nonparametric Exploration statistics. DTW is a classical measure evaluating the distance between two temporal sequences possibly varying in time or speed, applied to temporal sequences of video, audio, graphics and omics data [50, 54–57]. Operatively, DTW works by detecting the optimal mapping between the sequences via Dynamic Programming, to obtain an alignment; by backtracking, DTW provides a natural geometric representation of the time shift between two sequences.
As an alternative to PCC, here we propose to infer coexpression networks from omics time series data by a similarity measure integrating MIC with DTW. We define the DTW-MIC function as the root mean square of MIC and the similarity measure naturally induced by DTW. We quantify the performance of the DTW-MIC approach within a differential network framework based on the Hamming-Ipsen-Mikhailov (HIM) distance [58, 59], thus obtaining a quantification of the difference between the inferred network and the true network, whenever the true reference network is available. In particular, we evaluate DTW-MIC on four synthetic datasets generated by GeneNetWeaver (GNW) [60] following the guidelines of the DREAM Challenge [61]. Further, we apply the method on a gene expression reference dataset on T-cell activation in response to phorbol 12-myristate 13-acetate (PMA) and ionomycin treatment [62], showing a consistent improvement over PCC in network reconstruction. In addition to comparing DTW-MIC with the baseline reference PCC in the WGCNA framework, we evaluate the novel approach with respect to two algorithms differently estimating MI. We consider Transfer Entropy [63, 64], where the MI is integrated by the dynamics of information transport, and TimeDelay ARACNE [65]. The latter assumes that the underlying probabilistic model of the expression profiles is a stationary Random Markov Field and it outperforms the original ARACNE on longitudinal data. On average, the nets inferred by DTW-MIC were closer to the true network than the graphs inferred by both Transfer Entropy and TimeDelay ARACNE.
As a bioinformatics resource, we provide an implementation of the DTW-MIC measure, other association and inference functions, and the HIM distance within ReNette, the Open Source web framework for differential network analysis [66]. ReNette and its companion R package nettools [58] are available on the CRAN archive (http://cran.r-project.org) and on the GitHub repository https://github.com/MPBA/nettools.git.
Methods
Time series similarity measures
Maximal Information Coefficient.
The Maximal Information Coefficient (MIC) measure is a member of the Maximal Information-based Nonparametric Exploration (MINE) family of statistics, introduced for the exploration of two-variable relationships in multidimensional data sets [49, 51, 53]. Operatively, the MIC value is obtained by building several grids at different resolutions on the scatterplot of the two variables, then computing the largest possible MI achievable over all grids and finally normalizing to the [0, 1] range, where larger values correspond to higher similarity. The two distinctive features of MIC are generality, i.e., the ability of capturing variable relationships of different nature, and equitability, that is the property of penalizing similar levels of noise in the same way, regardless of the nature of the relation between variables.
Since its introduction in 2011, a debate arose in the scientific community regarding statistical flaws of MINE [67–75], such as tendency to overestimate MI and to generate false positives: superiority over MIC has also been claimed for alternative measures, such as Brownian distance correlation [76] and biweight midcorrelation [9]. Alternative statistics stemmed from the MIC definition such as Copula Correlation [77], GeneralizedMIC [78], Multivariate Maximal Correlation Analysis [79], Mutual Information based Dependence Index [80] and ImprovedAlgorithmMIC [81]. Altogether they do not match the popularity gained by the original MIC statistic, also in the computational biology community, e.g., in the analysis and inference of various kinds of biological networks. MIC has been coupled to the Context Likelihood of Relatedness (CLR) [82] for network inference from steady state data [83, 84]; MIC has been used for the same purpose in integration with the Interaction Component Model [85]. MIC has been used as an association measure for omics and other data in several systems biology studies, for a partial list, see [86–97]; several studies have specifically considered control of false positive ratio [98–100]. MIC (and the other MINE statistics) can be computed in R [101] by using the minerva package [52].
Example To illustrate the difference between PCC and MIC in detecting non-linear relationships between two variables, we introduce a simple synthetic example
. Consider the following five time series with 100 time points {ti = i : 1 ≤ i ≤ 100}:
where
is the uniform distribution with extremes a < b. While A(i) is just 1/100–th of the identity map, B(i) is a logarithmic map, C(i) is obtained from A(i) by adding a 20% level of uniform noise, D(i) is a more complex non-linear map merging a trigonometric and a logarithmic relation and, finally, E(i) is obtained from D(i) by a vertical offset and then flattening to zero all the values in the time interval [50, 70]. In Fig 1 the plot of the five time series A–E is displayed together with the PCC and MIC values for all pairs of sequences. MIC is able to capture the functional relationship linking all pairs of time series, even in presence of a moderate level of noise: all MIC values are larger than 0.72, and in six cases out of ten MIC attains the upper bound 1. On the other hand, PCC is close to one only when evaluating the pairs (A, B), (A, C), (B, C) and (D, E), while all the remaining six cases display a correlation score smaller than 0.33, confirming that PCC is ineffective as a similarity measure for complex longitudinal data. As a relevant example, note that B(i) has a strong functional dependence from D(i) and E(i) although the shape of the corresponding curves are hugely different: this non-linear behaviour is well captured by MIC, with similarity value 1 to both (B, D) and (B, E), while the corresponding values for PCC are negative.
PCC versus MIC in a synthetic example with five time series A–E on 100 time points (left) and the corresponding PCC values (right panel, top-left triangle) and MIC values (right panel, bottom-left triangle) for all pairs of time series.
Dynamic Time Warping.
Dynamic Time Warping (DTW) [54, 55] is a measure of distance between two sequences that takes care of time shifts. The DTW algorithm finds an optimal alignment between the two series by a non-linear warping of the time axes, also providing a measure of their dissimilarity. By construction, the similarity between curve shapes is a more important factor in DTW rather than the pointwise distance between the time series values. For a comprehensive reference, the reader is referred to [102].
Several variations to the original DTW algorithm have been proposed, first to overcome technical drawbacks and then to target specific data structures. Within the most important alternatives, we list DerivativeDTW [103], IterativeDTW [104], FastDTW [105], WeightedDTW [106], OnlineDTW [107], NearestNeighborDTW [108] and ComplexityInvariantDTW [109], possibily the most promising. Although the DTW alternatives often perform better in specific cases, for the original DTW robustness [106, 107] and effectiveness in a general scenario [110–112] are acknowledged. In particular, alternative measures to DTW specifically tailored to network inference do exist [113], but they have not been as extensively tested as DTW.
To obtain a similarity measure DTWs from the distance DTW we use the function DTWs = 1/(1 + DTWd), where DTWd is the normalized distance between two series, as computed in the R package dtw [114].
Example In what follows, a synthetic example
is used to highlight the difference between DTW and PCC for increasing time shift, with and without a moderate noise level. This example mimics a common situation in omics data, when the activation of a gene induces a delayed activation of an inactive gene, with a similar expression level curve, affected by a certain amount of noise. Consider the following time series with 100 time points {ti = i : 1 ≤ i ≤ 100}:
whose graph is displayed in the top-left panel (yellow background) of Fig 2. Moreover, define the following family of time series originated by r(i), for
:
In this notation,
. Finally, define the two functions
In Fig 2 the plots of the 15 time series
are shown, together with the corresponding values of P(s, k) (italic) and D(s, k) (boldface). Moreover, in the top panel of Fig 3 the curves P(s, k) (squares) and D(s, k) (dots) are displayed for k = 0, 1, 2 (in black, blue and red respectively) versus the time shift s ranging from 0 to 40. The example shows that DTW can model the dependence between
and r(i), even for large time shift s and high noise level k. In particular, as a function of the time shift s, the value for DTW monotonically decreases from 1 to 0.959, 0.804, 0.670 for k = 0, 1, 2 respectively, and D(s, 0) > D(s, 1) > D(s, 2) consistently along the whole range 0 ≤ s ≤ 40. On the other hand, PCC rapidly decreases to very low correlation level even for small time shifts s > 5, with PCC < 0.3 for all values s > 7. Furthermore, the PCC value does not change monotonically on increasing noise: in fact, the curves P(s, k) mutually intersecate. Finally, to assess the significance of the values D(s, k), we compare it against the null distribution
, where the set
consists of 2N random vectors ηj on 100 time points with values randomly and uniformly sampled between two positive real values m < M. In particular, as parameters here we use N = 1000 and, given a noise level k, we set
and
. For all the three cases k = 0, 1, 2, the distribution of the set
is Gaussian shaped, and the 95% Student bootstrap confidence intervals around the mean are quite narrow, namely (0.7429, 0.7441), (0.6570, 0.6584) and (0.5115, 0.5130) for k = 0, 1, 2 respectively. Thus the mean values
, i.e., 0.7435 (k = 0), 0.6577 (k = 1) and 0.5121 (k = 2), can be used as significance thresholds, as shown in the bottom panel of Fig 3: in all the three cases, for the whole range 0 ≤ s ≤ 40, the curve P(s, k) lies above the corresponding significance threshold value.
PCC and DTWs versus the reference series r for the 15 time series with s = 0, 5, 10, 20, 40 and k = 0, 1, 2. Each row corresponds to a different value of S, indicated by the figure in the top right corner of the plot in the first column. Each column corresponds to a different value of k: 0 on the left, with black curves, 1 in the centre, with blue curves and 2 on the right, with red curves. The plot in the top left panel with yellow background is the reference time series
. Under each panel, the corresponding values are reported for P(s, k) (italic) and D(s, k) (boldface).
PCC and DTWs versus the reference series r for the with k = 0, 1, 2 and the time shift s ranging between 0 and 40. Squares correspond to P(s, k), while circles and solid lines indicate D(s, k); the different noise levels k = 0, 1, 2 are denoted by curves in black, blue and red respectively. The dashed lines in the bottom panel indicate the no-information value for DTWs based on the null model described in Example
.
DTW-MIC.
We define DTW-MIC as a novel measure of similarity between two time series by considering the root mean square of MIC and DTWs:
By definition, DTW-MIC joins the contributions of both MIC and DTWs, thus taking care of time shifts and non-linear functional relations. This characteristic makes DTW-MIC more effective than PCC, but also of MIC and DTW considered separately, as demonstrated by the following example
on synthetic data.
Example Consider a set g of three genes g1, g2 and g3 and the corresponding time series of expression
on 100 time points 1 ≤ i ≤ 100 defined as follows:
The graphs of the functions in
are plotted in the top panel of Fig 4, while in the bottom panel we list all values {M(i, j) : M ∈ {PCC, MIC, DTWs, DTW-MIC} and 1 ≤ i < j ≤ 3}, computing for each similarity measure M, the corresponding coexpression network on the gene set g. All the three pairs of series have a very low correlation (PCC ≤ 0.23), but DTW-MIC is still able to capture the existing relation between them (DTW-MIC ≤ 0.5), even when these relations are of different nature. In fact, G2 and G3 have a low DTW similarity, but a high MIC correlation, while the opposite happens for G1 and G3. Finally, the pair (G1, G2) has moderate values for both MIC and DTW. In all three cases the resulting DTW-MIC value is above the significance threshold computed from the null model described in the previous section, which is 0.52 for (G1, G2), 0.29 for (G2, G3) and 0.39 for (G1, G3).
Plots (top) and PCC, MIC, DTWs and DTW-MIC weighted coexpression networks (bottom) for the set of the three time series G1, G2 and G3 (in red, blue and green respectively). Arc width is proportional to edge weight.
Network Analysis
Co-expression networks.
An effective method for simultaneously analysing the mutual relations among a group of interacting agents is provided by graph theory, consisting in (i) building a complex network that has the agents as nodes and (ii) inferring the (weight of the) edges connecting the nodes by applying a similarity measure between the signals of the agents. A typical example in omics science is represented by gene networks: the nodes are the genes and an edge between two genes is weighted by the similarity between their expression levels in a time window as read by microarray or sequencing technologies. In case of a binary network, the edge is declared to exist only if the similarity value lies above a chosen threshold. These graphs are called coexpression networks, having as most popular model the Weighted Gene Co-expression Network Analysis (WGCNA) [16–18], where the adopted similarity is the absolute PCC, soft thresholded by a power function. In detail, given N genes and their expressions g1, …, gn, the resulting WGCNA network is described by the adjacency matrix A whose entries are defined as
(1)
for M = |PCC| and β a positive power, usually tuned according to additional constraints, such as the scale-freeness [115, 116] of the network; the default choice in the WGCNA R/Bioconductor package [17] is β = 6.
Comparison methods.
In the Results section we will use the WGNCA framework with the novel DTW-MIC as the M measure in Eq (1), comparing the obtained networks with those inferred by the classical choice M = |PCC|. Apart from WGCNA, we will use two more algorithms for comparison purposes to DTW-MIC.
Algorithm for the Reconstruction of Accurate Cellular Networks (ARACNE) [42] is the reference method implementing reconstruction of network based on estimation of MI, together with Context Likelihood of Relatedness (CLR) [82], Maximum Relevance/Minimum Redundancy (MRNET) [117] and Relevance Network (RELNET) [118]. Although challenged in performance by novel MI approaches [80, 119–121], ARACNE still remains a valuable baseline when assessing the effectiveness of a novel method, and, for instance, tends to outperform approaches based on partial correlation [4, 95, 122]. However, as declared by the authors, the aim of ARACNE is the detection of transcriptional interactions with high confidence rather than the inference of all transcriptional interactions in a genetic network. Moreover, its potential goes beyond the coexpression assessment, making it suitable to address a wider range of network deconvolution problems. Since ARACNE is not designed to work on longitudinal data, as comparison method we select Time-Delay ARACNE [65], which allows the application of the ARACNE algorithm to time-course expression profiles. In detail, time-delayed dependencies between profiles are assessed by assuming a stationary Markov Random Field as the underlying probabilistic model, and then extracting a MI measure of dependence between the two genes at different time delay. Operatively, we will use Time-Delay ARACNE in its R/Bioconductor implementation provided by the TD-ARACNE package.
The second algorithm, Transfer Entropy [63, 64], although still based on MI estimation, has a different origin: it comes from statistical physics and it is aimed at quantifying the statistical coherence between systems evolving in time. In detail, this alternative information theoretic measure integrates the MI properties with the dynamics of information transport expressed in terms of Kullback entropy. Unlike DTW and TimeDelay-ARACNE, Transfer Entropy does not introduce any time delay in the observation, but rather it generalizes the entropy rate to two signals by measuring the deviation from independence. Hereafter we will test this measure as implemented in the TransferEntropy R package within the WGCNA framework with β = 6. Since Transfer Entropy is not symmetric, we follow the same strategy adopted by the authors of MIC: the weight of an unsigned interaction between the signals of two genes X, Y is the maximum of the intensity of the two directed interactions X → Y, Y → X. The embedding dimension and the neighbor used by the Kraskov estimator are set to 3 and 1, respectively, as shown in the documentation of the R package. In some cases, the considered dataset does not satisfy the assumptions of the Kraskov estimator, thus Transfer Entropy cannot be computed. As suggested by the R package documentation, a small Gaussian noise needs to be added to the data before computing Transfer Entropy.
Hamming-Ipsen-Mikhailov distance.
For the quantitative assessment of the difference between two networks sharing the same nodes a graph distance is required. Among all metrics described in the literature, we choose the Hamming-Ipsen-Mikhailov (HIM) distance for its consistency and robustness [59, 123]. The HIM distance for network comparison is defined as the product metric of the Hamming distance H [124, 125] and the Ipsen-Mikhailov distance IM [126], normalized by the factor to set its upper bound to 1:
for N1, N2 two undirected (possibly weighted) networks. The drawback of edit distances (such as H) is their locality, as they focus only on the network parts that are different in terms of presence or absence of matching links [123]. Spectral distances like IM are global, since they take into account the whole graph structure, but they cannot distinguish isomorphic or isospectral graphs, which can correspond to quite different conditions within the biological context. The HIM distance is a solution tackling both issues: details on HIM and its two components H and IM together with a few application examples are given in [58, 59]. In particular, HIM distance can be computed also for directed networks by using an alternative description of the graph topology. Values of HIM distance range from 0 (when comparing identical networks) to 1, attained only when comparing the full and the empty network.
Example In the example
shown in Fig 5, we selected four non-isospectral networks on four vertices, namely the empty graph E, the full graph F, a network with 1 edge A and a network with 4 edges including a 3-cycle, B. For these 4 graphs, the mutual H, IM and HIM distances are computed and reported as points on the H × IM plane, where each distance HIM(P, Q) between two graphs P and Q is represented by a point of coordinates R = (H(P, Q), IM(P, Q)) and its HIM value is the length of the segment connecting R to the origin (0, 0), divided by
. The visualization in the H × IM plane allows the relative comparison of the values of the two components of the distance: for instance, the Hamming distance between A and E is half the Hamming distance between B and F (1/6 vs. 1/3), but the IM component is much larger for the former pair, yielding two quite similar values for HIM.
Mutual HIM distances in the H×IM space between 4 non-isospectral graphs A, B, E, F on 4 vertices, whose topology is shown below the plot. Distance values are listed in the plot legend.
Results
In this section we apply the novel DTW-MIC similarity measure to two case studies in computational biology.
Each dataset includes a network of connections between n genes, together with the corresponding time series describing, for each gene, the dynamics of the expression level. Our strategy is the same in both applications and it includes two steps: first, the reconstruction of the network in the WGCNA framework in the classical approach via PCC and through DTW-MIC and the two additional benchmark measures TimeDelay ARACNE and Transfer Entropy, and then the evaluation of the HIM distance of the reconstructed networks from the true graph.
In detail, in the first application a suite of three synthetic gene network/time-course datasets is generated, inspired by real biological systems. The second task has the same goal, but expression level measurements come from a publicly available microarray dataset from a human cohort and the true network is experimentally unknown; however, a reasonable approximation of the network has been inferred by GeneNet [127, 128], a dynamical estimator of partial correlation coupled with an ad hoc procedure for the control of the local false discovery rate at a given threshold, an algorithm proven to be well performing in reconstruction [4]. Although the true network is not biologically validated, some landmark publications have used these datasets (e.g., [129, 130]). Indeed, in the few case where an experimental validation is available, either the data are not longitudinal (e.g. [131]) or the time series is too short (e.g. [132]) to guarantee statistical significance to the MIC measure [49, 53], or the network structure is not suitable for being reconstructed by correlation-based methods, as in the cases of causal analysis or encoding directional information [9] (e.g. [131]).
GeneNetWeaver Yeast & E. coli data
The datasets for the synthetic example are generated by GeneNetWeaver (GNW) [60, 133] an open-source tool for in silico benchmark generation, available at the web address http://gnw.sourceforge.net/genenetweaver.html. GNW generates realistic network structures of biologically plausible benchmarks by extracting modules from known gene networks of model organisms like yeast and E. coli [134], endowing them with dynamics using a kinetic thermodynamical model of transcriptional regulation with added internal noise, allowing for different types of customizable perturbations. According to the user prescribed constraints and given a chosen network topology, GNW can also produce steady states and time course datasets with the expression levels of the network nodes. The annual Dialogue for Reverse Engineering Assessments and Methods (DREAM, http://www.the-dream-project.org/) Challenge [2, 35, 61, 135–137] initiative for the quantitative comparison of network inference methods relies on GNW for the synthetic benchmark datasets.
Three synthetic networks are generated by GNW for the first application task, namely Yeast20, Ecoli20, Ecoli50, where the name points to the original reference network and the subscript indicates the number of nodes. In detail, Yeast20 is a subnet of the Yeast transcriptional regulatory network with 4441 nodes and 12873 edges [134, 138], while Ecoli20 and Ecoli50 are subnets of the E. coli transcriptional regulatory network with 1502 nodes and 3587 edges, corresponding to the TF-gene interactions of RegulonDB release 6.7 of May 2010 [134, 139]. In all cases, the selected genes are randomly extracted from the whole set of nodes only requiring that half of the selected nodes be regulators.
For each network, 10 longitudinal datasets {d1, …, d10} of expression levels are generated by a dynamic model mixing ordinary and stochastic differential equations, on 41 time points equally spaced between time 0 and time 1000 {t0 = 0, t1 = 25, …, t40 = 1000}. In each series, the initial time point t0 = 0 corresponds to the wild-type steady-state and, from that moment onwards, a perturbation is applied until time point t20 = 500: at that point, the perturbation is removed, and the gene expression level goes back from the perturbed to the wild-type state [134]. Moreover, a moderate level of noise is added to all the datasets, namely 0.5% for the Yeast data and 1% for the E.coli data; in both cases, the selected model is the microarray noise model described in [140]. Both the noise model and the perturbation scheme are chosen according to the configuration of the DREAM4 challenge [134]. As an example, in Fig 6 we show the plots of the generated time course data of four genes belonging to the selected subnets Yeast20, Ecoli20 and Ecoli50. GNW network and time-course data are publicly available on figshare, at the URL https://figshare.com/articles/Gene_Net_Weaver_Dataset/2279628.
examples of 4 longitudinal expression level data generated by the GNW kinetic model for the synthetic subgraph of Yeast and E. coli regulatory networks. Time course data are defined on 41 time points 0, …, 1000 and they correspond to the genes YFR030W (black, from Yeast20), YNL221C (green, from Yeast20 with dual gene knockout), rhaS (from Ecoli20) and putA (from Ecoli50).
In each of the three cases Yeast20, Ecoli20 and Ecoli50, a network is inferred by PCC, DTW-MIC, Transfer Entropy and TimeDelay ARACNE from each of the time course dataset {d1, …, d10}, and the obtained graph is compared via the HIM distance to the corresponding true network. As an example, in Fig 7 we show the true Yeast20 graph aside the networks reconstructed from the dataset d1. In all experiments, the results for TimeDelay ARACNE are reported for N = 11 normalization bins and likelihood 1.2 as in the R package documentation; worse results (not reported here) were obtained for N = 5 and N = 22.
example of network reconstruction and comparison with the true network. In the top panels, the topology of the synthetic true network Yeast20 (top left) is shown together with the Systematic Name of its 20 genes (top right). In the two bottom panels, the network Yeast20 as inferred from the time course dataset d1 by PCC (middle left), DTW-MIC (middle right), TimeDelay ARACNE (bottom left) and Transfer Entropy (bottom right). For the reconstructed networks, edge width is proportional to arc weight; edges with smaller weights (threshold is 0.001 for PCC, 0.135 for DTW-MIC and 0.005 for Transfer Entropy) are not drawn to avoid cluttering the image. Distance from the true network is 0.57 for the inference by PCC, 0.22 for the reconstruction by DTW-MIC, 0.28 for TimeDelay ARACNE and 0.57 for Transfer Entropy.
The results are reported in Table 1 and summarized in the box and whisker plots of Fig 8. The networks inferred by DTW-MIC are consistently closer to the true network than the graphs created with other inference methods, apart from Ecoli50 with TimeDelay ARACNE, with also smaller standard deviation over the 10 experiments in almost all cases.
box and whisker plot of the HIM distance between the networks inferred from time series and the true graphs, listed in Table 1. For each true network Yeast20, Ecoli20 and Ecoli50, 10 different graphs are reconstructed by PCC, DTW-MIC, TimeDelay ARACNE and Transfer Entropy similarity measures.
For the Yeast20 dataset, four additional time course datasets were generated on the same timepoints, but with a dual gene knockout: the curve of gene YNL221C in Fig 6 is an example of the generated trajectories. The results of the HIM distances from the true network for the networks inferred by the PCC, DTW-MIC, Transfer Entropy and TimeDelay ARACNE on the four datasets d1, …, d4 are reported in Table 2.
Again, the DTW-MIC inferred networks are closer to the true network than the other graphs, in all four experiments, with TimeDelay ARACNE as second best performing algorithm.
Human T-cell data
Rangel and colleagues in [62] investigated the dynamics of the activation of T-lymphocites by analysing the response of the human Jurkat T-cell line subjected to a treatment with phorbol 12-myristate 13-acetate (PMA) and ionomycin. Operatively, they measured the expression of 58 genes across 10 time points (0, 2, 4, 6, 8, 18, 24, 32, 48, and 72 hours after treatment) with two series of respectively 34 and 10 replicates on a custom microarray built by spotting PCR products on amino-modified glass slides using a Microgrid II spotter. The preprocessed array data tcell.34 and tcell.10, log-transformed and quantile normalized, are publicly available in the R package longitudinal. This package was developed by Opgen-Rhein and Strimmer who inferred the corresponding network by shrinkage estimation of the (partial) dynamical correlation [128, 141]. Their result is considered here as the true network, displayed in the top left panel of Fig 9. As an example of the data in the tcell.34 and tcell.10, in the top right panel of the same Fig 9 we show the time course data for the three genes EGR1, CD69 and SCYA2 in the first out of 34 replicates of tcell.34 and in the first out of 10 replicates of tcell.10.
The (true) network as reconstructed by Opgen-Rhein and Strimmer [128] (top left); the time course for three example genes EGR1 (blue), CD69 (red) and SCYA2 (orange), from replicate 1 of the tcell.34 (circles) and of the tcell.10 (squares) dataset. In the second row, the networks inferred by DTW-MIC from the tcell.10 (left) and from the tcell.34 (right) dataset; in these last two graphs, edges with weight smaller than 0.225 are not displayed.
Eight instances of the T-cell network are inferred, by the three similarity measures DTW-MIC, PCC and Transfer Entropy and the reconstruction algorithm TimeDelay ARACNE, starting from the two datasets tcell.34 and tcell.10. In both datasets, the dimension of the longitudinal data for each replicate (10 time points) cannot guarantee robustness in the inference process, since both PCC and MIC are not reliable for datasets of too small sample size [49, 68, 142]. Hence all replicates in the two datasets are consecutively joined so that time point 72h of replicate i is followed by time point 0h for replicate i + 1, thus yielding for each gene a single time course on 340 time points for tcell.34 and on 100 time points for tcell.10. The inferred networks are displayed in Figs 9 and 10, while in Table 3 the H/IM/HIM distances are reported between the true and the inferred T-cell networks.
The Human t-cell network as reconstructed by PCC (top row), TimeDelay ARACNE (middle row) and Transfer Entropy (bottom row), from the tcell.10 (left column) and from the tcell.34 (right column) dataset. Edges with weights smaller than 0.1 for PCC and smaller than 0.0001 for Trasfer Entropy are not displayed.
In Fig 11 we show the plot of the metric multidimensional scaling of all mutual HIM distances.
Planar projection conserving the mutual distances between the true Human t-cell network (TN) and the eight networks inferred from the two datasets tcell.34 (⋅34) and tcell.10 (⋅10) by the four reconstruction algorithms DTW-MIC (D), PCC (P), Transfer Entropy (T) and TimeDelay ARACNE (A).
For both datasets tcell.34 and tcell.10 the HIM distance from the true graph is smaller for the networks inferred by the DTW-MIC. The TimeDelay ARACNE measure reaches the same results, but only after a tuning phase optimizing the parameters N = 11, δ = 3 and likelihood 0.7. Note that, in all cases, the Hamming component of the distance is smaller, while the Ipsen-Mikhailov component is larger. Thus less links are changing between the inferred networks and the true graph, but these changing links induce a strongly different structure between the two nets. Indeed, in this experiment the choice of the similarity measure has a larger impact than the starting dataset, since the nets inferred using the same measure on different datasets are mutually closer than the nets inferred by different methods on the same time courses. Finally, without the power function (with β = 6 as default) applied in the WGCNA for soft thresholding the reconstructed networks are very different from the true graph, regardless of the starting dataset. For instance, the resulting HIM is about 0.47 for PCC and 0.66 for DTW-MIC, with 0.63 the average HIM value for a null model generated by computing the distance from the true graph of 1000 random network with uniform edge weight distribution in (0, 1). This effect does not come unexpected, because of the tendency of MIC to overestimate the MI in a number of situations [9, 22, 67, 68], thus generating false positives. An effective solution to avoid this bias and obtaining a more reliable estimate is the use of a thresholding function, either hard as in [142] or soft as in this case via a power law: these procedures allow discarding completely (hard threshold) or greatly reducing the weight of (soft threshold) the unwanted links wrongly detected by the association measure.
Conclusions
We introduced here DTW-MIC, a novel similarity measure for inferring coexpression networks from longitudinal data as an alternative to the absolute PCC used in the WGCNA approach. By combining Dynamic Time Warping and Maximal Information Coefficient, the DTW-MIC similarity can overcome the well known limitations of PCC when dealing with delayed signals and indirect interactions. Experiments on biologically inspired synthetic data and gene expression time course data demonstrate higher precision on average in the network inference for DTW-MIC with respect to PCC, TimeDelay ARACNE and Transfer Entropy in different conditions, and without the need for a parameter tuning phase. Considering the MIC bias towards false positives and the availability of numerous similarity measures derived from DTW, it is likely to expect as future development the exploration of different alternatives to the DTW-MIC pair. For instance, it has been pointed out that Brownian distance correlation [67, 76] and biweight midcorrelation [9, 143] do not suffer from the issues affecting MIC, and thus they may be adopted as replacements for MIC; on the other hand, ComplexityInvariantDTW [109] appears to be the most effective alternative to DTW, well performing on a wide range of situations, and thus worth exploring as a potential substitute.
Acknowledgments
The authors are grateful to two anonymous referees whose detailed remarks greatly helped in improving the paper.
Author Contributions
Conceived and designed the experiments: SR GJ RV MF CF. Performed the experiments: SR GJ RV MF CF. Analyzed the data: SR GJ RV MF CF. Wrote the paper: SR GJ RV MF CF.
References
- 1. De Smet R, Marchal K. Advantages and limitations of current network inference methods. Nat Rev Microbiol. 2010;8(10):717–729. pmid:20805835
- 2. Marbach D, Prill RJ, Schaffter T, Mattiussi C, Floreano D, Stolovitzky G. Revealing strengths and weaknesses of methods for gene network inference. Proc Natl Acad Sci USA. 2010;107(14):6286–6291. pmid:20308593
- 3. Szederkenyi G, Banga J, Alonso A. Inference of complex biological networks: distinguishability issues and optimization-based solutions. BMC Syst Biol. 2011;5(1):177. pmid:22034917
- 4. Allen JD, Xie Y, Chen M, Girard L, Xiao G. Comparing Statistical Methods for Constructing Large Scale Gene Networks. PLOS ONE. 2012;7(1):e29348. pmid:22272232
- 5. López-Kleine L, Leal L, López C. Biostatistical approaches for the reconstruction of gene co-expression networks based on transcriptomic data. Brief Funct Genomics. 2013;12(5):457–467. pmid:23407269
- 6. Rotival M, Petretto E. Leveraging gene co-expression networks to pinpoint the regulation of complex traits and disease, with a focus on cardiovascular traits. Brief Funct Genomics. 2014;13(1):66–78. pmid:23960099
- 7. Pierson E, the GTEx Consortium, Koller D, Battle A, Mostafavi S. Sharing and Specificity of Co-expression Networks across 35 Human Tissues. PLoS Comput Biol. 2015;11(5):e1004220. pmid:25970446
- 8. Song WM, Zhang B. Multiscale Embedded Gene Co-expression Network Analysis. PLoS Comput Biol. 2015;11(11):e1004574. pmid:26618778
- 9. Song L, Langfelder P, Horvath S. Comparison of co-expression measures: mutual information, correlation, and model based indices. BMC Bioinformatics. 2012;13(1):328. pmid:23217028
- 10. Madhamshettiwar P, Maetschke S, Davis M, Reverter A, Ragan M. Gene regulatory network inference: evaluation and application to ovarian cancer allows the prioritization of drug targets. Genome Med. 2012;4(5):41. pmid:22548828
- 11. Baralla A, Mentzen WI, de la Fuente A. Inferring Gene Networks: Dream or Nightmare? Ann N Y Acad Sci. 2009;1158:246–256. pmid:19348646
- 12. Lee HK, Hsu AK, Sajdak J, Qin J, Pavlidis P. Coexpression Analysis of Human Genes Across Many Microarray Data Sets. Genome Res. 2004;14(6):1085–1094. pmid:15173114
- 13. Lavi O, Dror G, Shamir R. Network-Induced Classification Kernels for Gene Expression Profile Analysis. J Comp Biol. 2012;19(6):694–709.
- 14. Rapaport F, Zinovyev A, Dutreix M, Barillot E, Vert JP. Classification of microarray data using gene networks. BMC Bioinformatics. 2007;8(1):35. pmid:17270037
- 15. Jansen R, Greenbaum D, Gerstein M. Relating Whole-Genome Expression Data with Protein-Protein Interactions. Genome Res. 2002;12(1):376.
- 16. Zhang B, Horvath S. A General Framework for Weighted Gene Co-Expression Network Analysis. Stat Appl Genet Molec Biol. 2005;4(Issue 1):Article 17.
- 17. Langfelder P, Horvath S. WGCNA: an R package for weighted correlation network analysis. BMC Bioinformatics. 2008;9(1):559. pmid:19114008
- 18.
Horvath S. Weighted Network Analysis: Applications in Genomics and Systems Biology. Springer; 2011.
- 19. Kumari S, Nie J, Chen HS, Ma H, Stewart R, Li X, et al. Evaluation of Gene Association Methods for Coexpression Network Construction and Biological Knowledge Discovery. PLOS ONE. 2012;7(11):e50411. pmid:23226279
- 20. Kurt Z, Aydin N, Altay G. A comprehensive comparison of association estimators for gene network inference algorithms. Bioinformatics. 2014;30(15):2142–2149. pmid:24728859
- 21.
Dempsey K, Bonasera S, Bastola D, Ali H. A Novel Correlation Networks Approach for the Identification of Gene Targets. In: Proc HICSS 2011. IEEE; 2011. p. 1–8.
- 22.
Clark M. A comparison of correlation measures; 2013.
- 23. Yule U. Why do we sometimes get nonsense-correlations between time series? A study in sampling and the nature of time series. J R Stat Soc. 1926;89:1–63.
- 24. Granger CWJ, Newbold P. Spurious regressions in econometrics. J Econometrics. 1974;2:111–120.
- 25. Erdem O, Ceyhan E, Varli Y. A new correlation coefficient for bivariate time-series data. Physica A. 2014;414:274–284.
- 26. Oates CJ, Mukherjee S. Network inference and biological dynamics. Ann Appl Stat. 2012;6(3):1209–1235. pmid:23284600
- 27. Emmert-Streib F, Glazko GV, Altay G, de Matos Simoes R. Statistical Inference and Reverse Engineering of Gene Regulatory Networks from Observational Expression Data. Front Genet. 2012;3:8. pmid:22408642
- 28.
Song L. Novel machine learning and correlation network methods for genomic data. UCLA; 2013.
- 29. Ristevski B. A survey of models for inference of gene regulatory networks. Nonlinear Anal Model Control. 2013;18(4):444–465.
- 30. Siegenthaler C, Gunawan R. Assessment of Network Inference Methods: How to Cope with an Underdetermined Problem. PLOS ONE. 2014;9(3):e90481. pmid:24603847
- 31. Omony J. Biological Network Inference: A Review of Methods and Assessment of Tools and Techniques. Annu Res Rev Biol. 2014;4(4):577–601.
- 32. Wang RYX, Huang H. Review on statistical methods for gene network reconstruction using expression data. J Theor Biol. 2014;362:53–61. pmid:24726980
- 33. Ballouz S, Verleyen W, Gillis J. Guidance for RNA-seq co-expression network construction and analysis: safety in numbers. Bioinformatics. 2015;31(13):2123–2130. pmid:25717192
- 34. Hsu CL, Juan HF, Huang HC. Functional Analysis and Characterization of Differential Coexpression Networks. Nat Sci Rep. 2015;5:Article number: 13295.
- 35. Marbach D, Costello JC, Kuffner R, Vega NM, Prill RJ, Camacho DM, et al. Wisdom of crowds for robust gene network inference. Nat Methods. 2012;9(8):796–804. pmid:22796662
- 36.
Slawek J. Inferring Gene Regulatory Networks from Expression Data using Ensemble Methods. Virginia Commonwealth University; 2014.
- 37.
Hill SM, Heiser LM, Cokelaer T, Unger M, Nesser NK, Carlin DE, et al. Inferring causal molecular networks: empirical assessment through a community-based effort. Nat Meth. 2016;Advanced online publication.
- 38. Verleyen W, Ballouz S, Gillis J. Measuring the wisdom of the crowds in network-based gene function inference. Bioinformatics. 2015;31(5):745–752. pmid:25359890
- 39.
Mendoza MR. Exploring ensemble learning techniques to optimize the reverse engineering of gene regulatory networks. Universidade Federal do Rio Grande do Sul; 2014.
- 40.
Mendoza MR. The wisdom of crowds in bioinformatics: what canwe learn (and gain) from ensemble predictions? In: Proc AAAI 2013. AAAI Press; 2013. p. 1678–1679.
- 41. Ud-Dean SMM, Gunawan R. Ensemble Inference and Inferability of Gene Regulatory Networks. PLOS ONE. 2014;9(8):e103812. pmid:25093509
- 42. Margolin AA, Nemenman I, Basso K, Wiggins C, Stolovitzky G, Dalla-Favera R, et al. ARACNE: an algorithm for the reconstruction of gene regulatory networks in a mammalian cellular context. BMC Bioinformatics. 2006;7(7):S7. pmid:16723010
- 43. Kim Y, Han S, Choi S, Hwang D. Inference of dynamic networks using time-course data. Brief Bioinform. 2014;15(2):212–228. pmid:23698724
- 44. Liao TW. Clustering of time series data—a survey. Pattern Recogn. 2005;38(11):1857–1874.
- 45.
Buza K, Nanopoulos A, Schmidt-Thieme L. Fusion of Similarity Measures for Time Series Classification. In: Proc HAIS 2011. vol. 6679 of LNCS. Springer; 2011. p. 253–261.
- 46. Fu T. A review on time series data mining. Eng Appl Artif Intell. 2011;24:164–181.
- 47. Esling P, Agon C. Time-series data mining. ACM Comput Surv. 2012;45(1):1–34.
- 48. Wang X, Mueen A, Ding H, Trajcevski G, Scheuermann P, Keogh E. Experimental comparison of representation methods and distance measures for time series data. Data Min Knowl Discov. 2012;26(2):275–309.
- 49. Reshef DN, Reshef YA, Finucane HK, Grossman S, McVean G, Turnbaugh P, et al. Detecting novel associations in large datasets. Science. 2011;6062(334):1518–1524.
- 50. Sakoe H, Chiba S. Dynamic Programming Algorithm Optimization for Spoken Word Recognition. IEEE Trans Sig Process. 1978;26(1):43–49.
- 51. Speed T. A Correlation for the 21st Century. Science. 2011;6062(334):1502–1503.
- 52. Albanese D, Filosi M, Visintainer R, Riccadonna S, Jurman G, Furlanello C. minerva and minepy: a C engine for the MINE suite and its R, Python and MATLAB wrappers. Bioinformatics. 2013;29(3):407–408. pmid:23242262
- 53. Nature Biotechnology. Finding correlations in big data. Nat Biotechnol. 2012;30(4):334–335. pmid:22491290
- 54.
Keogh E, Pazzani M. An enhanced representation of time series which allows fast and accurate classification, clustering and relevance feedback. In: AAAI, editor. Proc KDD 1998; 1998. p. 239–241.
- 55.
Keogh E, Pazzani M. Scaling up dynamic time warping for datamining applications. In: AAAI, editor. Proc KDD 2000; 2000. p. 285–289.
- 56. Aach J, Church G. Aligning gene expression time series with time warping algorithms. Bioinformatics. 2001;17(6):495–508. pmid:11395426
- 57. Furlanello C, Merler S, Jurman G. Combining feature selection and DTW for time-varying functional genomics. IEEE Trans Sig Process. 2006;54(6):2436–2443.
- 58. Filosi M, Visintainer R, Riccadonna S, Jurman G, Furlanello C. Stability Indicators in Network Reconstruction. PLOS ONE. 2014;9(2):e89815. pmid:24587057
- 59.
Jurman G, Visintainer R, Riccadonna S, Filosi M, Furlanello C. The HIM glocal metric and kernel for network comparison and classification. In: Proc DSAA 2015. vol. 36678. IEEE; 2015. p. 1–10.
- 60. Schaffter T, Marbach D, Floreano D. GeneNetWeaver: In silico benchmark generation and performance profiling of network inference methods. Bioinformatics. 2011;27(16):2263–2270. pmid:21697125
- 61. Prill RJ, Marbach D, Saez-Rodriguez J, Sorger PK, Alexopoulos LG, Xue X, et al. Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges. PLOS ONE. 2010;5(2):e9202. pmid:20186320
- 62. Rangel C, Angus J, Ghahramani Z, Lioumi M, Sotheran E, Gaiba A, et al. Modeling T-cell activation using gene expression profiling and state-space models. Bioinformatics. 2004;20(9):1361–1372. pmid:14962938
- 63. Schreiber T. Measuring Information Transfer. Phys Rev Lett. 2000;85(2):461–464. pmid:10991308
- 64. Kaiser A, Schreiber T. Information transfer in continuous processes. Phys D. 2002;166(1-2):43–62.
- 65. Zoppoli P, Morganella S, Ceccarelli M. TimeDelay-ARACNE: Reverse engineering of gene networks from time-course data by an information theoretic approach. BMC Bioinformatics. 2010;11:154. pmid:20338053
- 66.
Filosi M, Droghetti S, Arbitrio E, Visintainer R, Riccadonna S, Jurman G, et al. ReNette: a web-service for network reproducibility analysis; 2014.
- 67.
Simon N, Tibshirani R. Comment on”Detecting Novel Associations In Large Data Sets” by Reshef Et Al, Science Dec 16, 2011; 2014.
- 68.
Gorfine M, Heller R, Heller Y. Comment on”Detecting Novel Associations In Large Data Sets” by Reshef Et Al, Science Dec 16, 2011; 2012.
- 69.
Posnett D, Devanbu P, Filkov V. MIC Check: A Correlation Tactic for ESE Data. In: Proc MSR 2012. IEEE; 2012. p. 22–31.
- 70. Kinney JB, Atwal GS. Equitability, mutual information, and the maximal information coefficient. Proc Natl Acad Sci USA. 2014;111(9):3354–3359. pmid:24550517
- 71. Kinney JB, Atwal GS. Reply to Reshef et al.: Falsifiability or bust. Proc Natl Acad Sci USA. 2014;111(33):3364.
- 72. Reshef DN, Reshef YA, Mitzenmacher MM, Sabeti PC. Cleaning up the record on the maximal information coefficient and equitability. Proc Natl Acad Sci USA. 2014;111(33):3362–3363.
- 73.
Reshef YA, Reshef DN, Finucane HK, Sabeti PC, Mitzenmacher MM. Measuring dependence powerfully and equitably; 2015.
- 74.
Reshef DN, Reshef YA, Finucane HK, Sabeti PC, Mitzenmacher MM. An Empirical Study of Leading Measures of Dependence; 2015.
- 75. Wang Y, Li Y, Cao H, Xiong M, Shugart YY, Jin L. Efficient test for nonlinear dependence of two continuous variables. BMC Bioinformatics. 2015;16(1):260. pmid:26283601
- 76. Székely GJ, Rizzo ML. Brownian distance covariance. Ann App Stat. 2009;3(4):1236–1265.
- 77.
Ding AA, Li Y. Copula Correlation: An Equitable Dependence Measure and Extension of Pearson’s Correlation; 2015.
- 78.
Luedtke A, Tran L. The Generalized Mean Information Coefficient; 2013.
- 79.
Nguyen HV, Mueller E, Vreeken J, Efros P, Boehm K. Multivariate Maximal Correlation Analysis. In: Proc ICML 2014. IMIS; 2014. p. 775–783.
- 80.
Jain N, Murthy CA. A new estimate of mutual information based measure of dependence between two variables: properties and fast implementation; 2015.
- 81. Wang S, Zhao Y. Analysing Large Biological Data Sets with an Improved Algorithm for MIC. Int J Data Min Bioinformatics. 2015;13(2):158–170.
- 82. Faith JJ, Hayete B, Thaden JT, Mogno I, Wierzbowski J, Cottarel G, et al. Large-Scale Mapping and Validation of Escherichia coli Transcriptional Regulation from a Compendium of Expression Profiles. PLoS Biology. 2007;5(1):e8. pmid:17214507
- 83.
Akhand MAH, Nandi RN, Amran SM, Murase K. Context Likelihood of Relatedness with Maximal Information Coefficient for Gene Regulatory Network Inference. In: Proc ICCIT 2015. IEEE; 2015. p. 1–6.
- 84. Akhand MAH, Nandi RN, Amran SM, Murase K. Gene Regulatory Network Inference Using Maximal Information Coefficient. Int J Biosci Biochem Bioinforma. 2015;5(5):296–310.
- 85. Rau CD, Wisniewski N, Orozco LD, Bennett B, Weiss JN, Lusis AJ. Maximal information component analysis: a novel non-linear network analysis method. Front Genet. 2013;4:28. pmid:23487572
- 86. Das J, Mohammed J, Yu H. Genome-scale analysis of interaction dynamics reveals organization of biological networks. Bioinformatics. 2012;28(14):1873–1878. pmid:22576179
- 87. Zhang Y, Zhang W, Xie Y. Improved heuristic equivalent search algorithm based on Maximal Information Coefficient for Bayesian Network Structure Learning. Neurocomputing. 2013;117:186–195.
- 88. Pernice V, Deger M, Cardanobile S, Rotter S. The relevance of network micro-structure for neural dynamics. Front Comput Neurosci. 2013;7:72. pmid:23761758
- 89.
Li J, Wei H, Zhao PX. DeGNServer: Deciphering Genome-Scale Gene Networks through High Performance Reverse Engineering Analysis. Biomed Res Int. 2013;2013:Article ID 856325.
- 90. de Siqueira Santos S, Takahashi DY, Nakata A, Fujita A. A comparative study of statistical methods used to identify dependencies between gene expression signals. Brief Bioinform. 2014;15(6):906–918. pmid:23962479
- 91. Eiler A, Zaremba-Niedzwiedzka K, Martínez-García M, McMahon KD, Stepanauskas R, Andersson SGE, et al. Productivity and salinity structuring of the microplankton revealed by comparative freshwater metagenomics. Environ Microbiol. 2014;16(9):2682–2698. pmid:24118837
- 92. Zhang Z, Sun S, Yi M, Wu X, Ding Y. MIC as an Appropriate Method to Construct the Brain Functional Network. Biomed Res Int. 2015;2015:Article ID 825136.
- 93. Zhang J, Fan D, Jian Z, Chen GG, Lai PBS. Cancer Specific Long Noncoding RNAs Show Differential Expression Patterns and Competing Endogenous RNA Potential in Hepatocellular Carcinoma. PLOS ONE. 2015;10(10):e0141042. pmid:26492393
- 94. Rodríguez-Ramos T, Marañión E, Cermeño P. Marine nano- and microphytoplankton diversity: redrawing global patterns from sampling-standardized data. Global Ecol Biogeogr. 2015;24(5):527–538.
- 95. Liseron-Monfils C, Ware D. Revealing gene regulation and associations through biological networks. Curr Plant Biol. 2015;3-4:30–39.
- 96. Tan Q, Tepel M, Rasmussen LM, Hjelmborg JB. Generalized Measure of Dependency for Analysis of Omics Data. J Data Mining Genomics Proteomics. 2015;7:183.
- 97. Comte J, Lovejoy C, Crevecoeur S, Vincent WF. Co-occurrence patterns in aquatic bacterial communities across changing permafrost landscapes. Biogeosciences. 2016;13:175–190.
- 98.
Tian Y, Zhang B, Shih IM, Wang Y. Knowledge-guided Differential Dependency Network Learning for Detecting Structural Changes in Biological Networks. In: Proc BCB 2011. ACM; 2011. p. 254–263.
- 99. Zhang B, Tian Y, Zhang Z. Network Biology in Medicine and Beyond. Circ Cardiovasc Genet. 2014;7(4):536–547. pmid:25140061
- 100. Ma C, Xin M, Feldmann KA, Wang X. Machine Learning–Based Differential Network Analysis: A Study of Stress-Responsive Transcriptomes in Arabidopsis. Plant Cell. 2014;26(2):520–537. pmid:24520154
- 101.
R Core Team. R: A Language and Environment for Statistical Computing; 2014. Available from: http://www.R-project.org/.
- 102.
Gusfield D. Algorithms on strings, trees and sequences. Cambridge University Press; 1997.
- 103.
Keogh EJ, Pazzani MJ. Derivative Dynamic Time Warping. In: Kumar V, Grossman R, editors. Proc ICDM 2001; 2001. p. 1–11.
- 104.
Chu S, Keogh E, Hart D, Pazzani M. Iterative Deepening Dynamic Time Warping for Time Series. In: Proc SDM 2002. SIAM; 2002. p. 1–18.
- 105.
Salvador S, Chan P. FastDTW: Toward Accurate Dynamic Time Warping in Linear Time and Space. In: Proc KDD/TDM 2004. ACM; 2004. p. 70–80.
- 106. Jeong YS, Jeong MK, Omitaomu OA. Weighted dynamic time warping for time series classification. Pattern Recognition. 2011;44(9):2231–2240.
- 107. Li H. On-line and dynamic time warping for time series data mining. Int J Mach Learn Cyb. 2014;April:1–9.
- 108.
Petitjean F, Forestier G, Webb GI, Nicholson AE, Chen Y, Keogh E. Dynamic Time Warping Averaging of Time Series Allows Faster and More Accurate Classification. In: Proc ICDM 2014. IEEE; 2014. p. 470–479.
- 109. Batista GEAPA, Keogh EJ, Tatawi OM, de Souza VMA. CID: an efficient complexity-invariant distance for time series. Data Min Knowl Discov. 2014;28:634–669.
- 110.
Rakthanmanon T, Campana B, Mueen A, Westover GBB, Zhu Q, Zakaria J, et al. Searching and mining trillions of time series subsequences under Dynamic Time Warping. In: Q Y, Agarwal D, Pei J, editors. Proc KDD 2012; 2012. p. 262–270.
- 111. Rakthanmanon T, Campana B, Mueen A, Batista G, Westover GBB, Zhu Q, et al. Addressing Big Data Time Series: Mining Trillions of Time Series Subsequences Under Dynamic Time Warping. ACM Trans Knowl Discov Data. 2013;7(3):Article n.10.
- 112. Ding H, Trajcevski G, Scheuermann P, Wang X, Keogh E. Querying and mining of time series data: experimental comparison of representations and distance measures. Proceedings VLDB Endowment. 2008;1(2):1542–1552.
- 113.
ElBakry O, Ahmad MO, Swamy MNS. Inference of gene regulatory networks from time-series microarray data. In: Proc NEWCAS 2010. IEEE; 2010. p. 141–144.
- 114. Giorgino T. Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package. J Stat Softw. 2009;31(7):1–24.
- 115. de Solla Price DJ. Networks of Scientific Papers. Science. 1965;149(3683):510–515.
- 116. Barabasi AL, Albert R. Emergence of scaling in random networks. Science. 1999;286:509–512. pmid:10521342
- 117. Meyer PE, Kontos K, Lafitte F, Bontempi G. Information-Theoretic Inference of Large Transcriptional Regulatory Networks. EURASIP J Bioinform Syst Biol. 2007;2007:79879.
- 118. Butte AJ, Tamayo P, Slonim D, Golub TR, Kohane IS. Discovering functional relationships between RNA expression and chemotherapeutic susceptibility using relevance networks. Proc Natl Acad Sci USA. 2000;97(22):12182–12186. pmid:11027309
- 119. Villaverde AF, Ross J, Morán F, Banga JR. MIDER: Network Inference with Mutual Information Distance and Entropy Reduction. PLOS ONE. 2014;9(5):e96732. pmid:24806471
- 120. Roy S, Bhattacharyya DK, Kalita JK. Reconstruction of gene co-expression network from microarray data using local expression patterns. BMC Bioinformatics. 2014;15(Suppl 7):S10. pmid:25079873
- 121.
Petereit J, Harris FC, Schlauch K. petal: A novel co-expression network modeling system. In: Proc BIBM 2015. IEEE; 2015. p. 234–241.
- 122. Vargha A, Bergman LR, Delaney HD. Interpretation problems of the partial correlation with nonnormally distributed variables. Qual Quant. 2013;47:3391–3402.
- 123. Jurman G, Visintainer R, Furlanello C. An introduction to spectral distances in networks. FAIA. 2011;226:227–234.
- 124. Tun K, Dhar P, Palumbo M, Giuliani A. Metabolic pathways variability and sequence/networks comparisons. BMC Bioinformatics. 2006;7(1):24. pmid:16420696
- 125. Dougherty ER. Validation of gene regulatory networks: scientific and inferential. Brief Bioinform. 2010;12(3):245–252. pmid:21183477
- 126. Ipsen M, Mikhailov AS. Evolutionary reconstruction of networks. Phys Rev E. 2002;66(4):046109.
- 127. Schafer J, Strimmer K. An empirical bayes approach to inferring large-scale gene association networks. Bioinformatics. 2005;21:754–764. pmid:15479708
- 128. Opgen-Rhein R, Strimmer K. Inferring gene dependency networks from genomic longitudinal data: a functional data approach. REVSTAT. 2006;4:53–65.
- 129. Beal MJ, Falciani F, Ghahramani Z, Rangel C, Wild DL. A Bayesian approach to reconstructing genetic regulatory networks with hidden factors. Bioinformatics. 2005;21(3):349–356. pmid:15353451
- 130. Lim N, Şenbabaoğlu Y, Michailidis G, d’Alché-Buc F. OKVAR-Boost: a novel boosting algorithm to infer nonlinear dynamics and interactions in gene regulatory networks. Bioinformatics. 2013;29(11):1416–1423. pmid:23574736
- 131. Sachs K, Perez O, Pe’er D, Lauffenburger DA, Nolan GP. Causal protein-signaling networks derived from multiparameter single-cell data. Science. 2005;308(5721):523–529. pmid:15845847
- 132. Cantone I, Marucci L, Iorio F, Ricci MA, Belcastro V, Bansal M, et al. A yeast synthetic network for in vivo assessment of reverse-engineering and modeling approaches. Cell. 2009;137(1):172–181. pmid:19327819
- 133. Marbach D, Schaffter T, Mattiussi C, Floreano D. Generating Realistic In Silico Gene Networks for Performance Assessment of Reverse Engineering Methods. J Comp Biol. 2009;16(2):229–239.
- 134.
Shaffter T, Marbach D, Roulet G. GeneNetWeaver User Manual, version 3.1; 2012.
- 135. Stolovitzky G, Monroe D, Califano A. Dialogue on Reverse-Engineering Assessment and Methods. Ann N Y Acad Sci. 2007;1115(1):1–22. pmid:17925349
- 136. Stolovitzky G, Prill RJ, Califano A. Lessons from the DREAM2 Challenges. Ann N Y Acad Sci. 2009;1158(1):159–195. pmid:19348640
- 137. Prill RJ, Saez-Rodriguez J, Alexopoulos LG, Sorger PK, Stolovitzky G. Crowdsourcing Network Inference: The DREAM Predictive Signaling Network Challenge. Sci Signal. 2011;4(189):mr7. pmid:21900204
- 138. Balaji S, Babu MM, Iyer LM, Luscombe NM, Aravind L. Comprehensive Analysis of Combinatorial Regulation using the Transcriptional Regulatory Network of Yeast. J Mol Biol. 2006;360(1):213–227. pmid:16762362
- 139. Gama-Castro S, Jiménez-Jacinto V, Peralta-Gil M, Santos-Zavaleta A, Peñaloza-Spinola MI, Contreras-Moreira B, et al. RegulonDB (version 6.0): gene regulation model of Escherichia coli K-12 beyond transcription, active (experimental) annotated promoters and Textpresso navigation. Nucleic Acids Res. 2008;36(suppl 1):D120–D124. pmid:18158297
- 140. Tu Y, Stolovitzky G, Klein U. Quantitative noise analysis for gene expression microarray experiments. Proc Natl Acad Sci USA. 2002;99(22):14031–14036. pmid:12388780
- 141.
Opgen-Rhein R, Strimmer K. Using regularized dynamic correlation to infer gene dependency networks from time-series microarray data. In: Ruusuvuori P, Manninen T, Huttunen H, Linne ML, Yli-Harja O, editors. Proc WCSB 2006; 2006. p. 73–76.
- 142. Gobbi A, Jurman G. A null model for Pearson correlation networks. PLOS ONE. 2015;10(6):e0128115. pmid:26030917
- 143. Langfelder P. Fast R Functions for Robust Correlations and Hierarchical Clustering. J Stat Softw. 2012;46(11):i11. pmid:23050260