Low-dimensional approximation searching strategy for transfer entropy from non-uniform embedding

Jian Zhang

doi:10.1371/journal.pone.0194382

Abstract

Transfer entropy from non-uniform embedding is a popular tool for the inference of causal relationships among dynamical subsystems. In this study we present an approach that makes use of low-dimensional conditional mutual information quantities to decompose the original high-dimensional conditional mutual information in the searching procedure of non-uniform embedding for significant variables at different lags. We perform a series of simulation experiments to assess the sensitivity and specificity of our proposed method to demonstrate its advantage compared to previous algorithms. The results provide concrete evidence that low-dimensional approximations can help to improve the statistical accuracy of transfer entropy in multivariate causality analysis and yield a better performance over other methods. The proposed method is especially efficient as the data length grows.

Citation: Zhang J (2018) Low-dimensional approximation searching strategy for transfer entropy from non-uniform embedding. PLoS ONE 13(3): e0194382. https://doi.org/10.1371/journal.pone.0194382

Editor: Mathias Baumert, University of Adelaide, AUSTRALIA

Received: September 6, 2017; Accepted: March 4, 2018; Published: March 16, 2018

Copyright: © 2018 Jian Zhang. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This study was unfunded. The author received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

In the study of neuroscience, directed information measure has been proven as a critically important tool in a wide range of application scenarios. Over recent years, a variety of time series analysis methods have been introduced to identify the existence and direction of the interactions in nervous system, and there are two most commonly used methods: Granger causality (GC), which is derived from the field of econometrics [1], and its information-theoretic analog, transfer entropy (TE) [2, 3]. Both of them are based on the simple idea that if the prediction of a time series (the effect) could be improved by incorporating the knowledge of past information of a second one (the cause), then the latter is considered to have a causal influence on the former. Moreover, it has been proved [4] that there is a close connection between Granger causality and transfer entropy: these two measures are equivalent for time series under the assumption of Gaussianity. This fact leads to a bridge between econometric and information-theoretic predictive approaches for the evaluation of directed couplings. The result has also been extended to the condition of non-Gaussian probability density distributions [5]. For computationally efficiency, the exact formulation of conditional mutual information (CMI) derived from GC-TE equivalence has been frequently employed in the analysis of time series with high linearity, such as for fMRI data [6], while transfer entropy, as a generally model-free approach which does not require any prior assumptions about the probability distribution, has been mostly adopted to nonlinear complex systems such as in the field of physiology [7–9] or climatology [10–12].

Although transfer entropy has several practical advantages, it still remains some critical drawbacks to overcome. For example, if the dimension of the embedding space that spanned by the variables is relatively large, the estimation of mutual information which is required for the computation of transfer entropy will inevitably suffer from the problem “curse of dimensionality” [10, 13, 14], make it difficult to uncover correctly the underlying causal structure of data. To tackle this problem, it is needed to improve the choice of parameters to get a well fitted model. One promising approach is the shrinking strategy which is based on properly reduction for the overall parameters to be estimated. Several paradigms focus on this objective have been proposed by previous researchers, in econometrics it is called “subset regression” [15–18], a method relies on sequential t-tests and model selection criteria, or by carrying out branch-and-bound strategy to cut subtrees [19], while in physics it is usually under the name “Non-uniform Embedding” [7, 13, 20–22], a method relies on information-theoretic measures and non-uniform state space reconstruction. All of these paradigms are aiming to achieve a parsimonious model by picking up the most informative lags of driver variables that has a significant impact to the target variable, while restricting the state space by reducing the dimensionality.

In this paper we propose a paradigm that makes use of the low-dimensional approximation technique for conditional mutual information, which was originally derived from the study of information-theoretic criterion for feature selection. We employ this method to deal with the problem of searching lagged variables in the computing process of transfer entropy from non-uniform embedding. In the implementation of traditional scheme of non-uniform embedding, the main procedure is to reconstruct a future point of the target variable in the subspace of the joint state space spanned by lagged driven variables. It derives from a sequential selection method which updates the embedding vector progressively, taking all relevant lagged variables into consideration at each step, stopping under certain termination criterion to drop all the vectors on irrelevant lags, and finally identifying the set of components that associated with the most information transfer to the target process. The traditional termination criterion applied by previous researchers [9, 13, 23] is based on a statistical significant test for the conditional mutual information between candidate variables. In this article a new method modified by low-dimensional approximation searching strategy will be introduced to improve the accuracy and efficiency of previous TE algorithms.

The rest of this paper is organized as follows. In the next section, we provide a brief review of transfer entropy from non-uniform embedding and low-dimensional approximation paradigm in feature selection, and then describe in detail the formulation of our approach. In section 3, we present a number of simulation experiments to demonstrate the effectiveness of the proposed algorithm compared to previous methods, In section 4 we test our approach in analyzing intracranial EEG recordings from an epileptic patient to show its practical applicability on real-world data, and the conclusion will be summarized in the final section.

Materials and methods

Transfer entropy from non-uniform embedding

We start by presenting the definition of transfer entropy from non-uniform embedding [13, 20, 23], also known as lag-specific transfer entropy [9, 11], which is a measure to estimate the directional coupling within a dynamical system by using the conditional mutual information for variable selection in the procedure of mixed embedding. Let us firstly consider an overall dynamical system composed of M interacting subsystems, and suppose that we are interested in evaluating the information flow from a driving subsystem X, to the destination subsystem Y, in the presence of the remaining subsystems which are described by the set of processes . Here we discuss our work under the assumption that the states of the system can be described as a multivariate stationary stochastic process, and denoted by X_n, Y_n, and as the observations obtained by sampling the processes at present time n, and accordingly, , and as the respective sets of variables describing the past of the processes. Then, the (multivariate) transfer entropy from process X to Y conditioned on , which quantifies the information provided by the past of X about the present state of Y that has not been contained in the past of Y or any other processes included in , could be defined in the form of conditional mutual information , or equivalently as a difference of two conditional entropies: (1) where H(⋅) stands for Shannon entropy, and I(⋅) stands for mutual information.

The crucial step in the procedure for estimating transfer entropy from non-uniform embedding, is to build a conditioning vector V_n by a sequential selection procedure from the collection of candidate vectors including in the past of X, Y and Z with a fixed searching depth up to a maximum truncation lag L. The candidate vectors of all relevant processes compose the set , and the information transfer to the target can be quantified by means of the sum of all contributions at different time lags. The searching strategy for significant components in Ω can be described as follows:

1. Starting with an empty vector

2. At each step k ≥ 1, perform a searching procedure to select the most informative vector on a specific lag of past variables, , that contribute most significantly to the target variable Y_n, from the available candidate vectors (the subtraction from Ω by ), and construct the optimal conditioning embedding vector by adding to the already chosen (k − 1) vectors, i.e. . Each candidate vector will be tested by a criterion through computing the maximum CMI between and the target vector Y_n conditioned on the already chosen vector : (2)

3. Terminate the loop of selection when there is no can fulfill the significance test. The significance test is designed in line with the statistical method from previous studies [13, 22, 23], in each searching loop 100 surrogate data are generated by a random shift procedure on the time points of causal driver and target variables for zero-setting the significant elements at specific lags, and the CMI value is then compared with the empirical distribution of surrogates according to the null hypothesis of absent causality from the driver to the target, to see if the value is larger than the (1 − α)% percentile of the ensemble of the randomized , before stop searching and return by as the conditioning embedding vector, which is composed of , where , and denote the components of V_n belonging respectively to X, Y, and .

4. Calculate the transfer entropy as: (3)

The motivation to use low-dimensional approximation for transfer entropy

The above construction procedure for embedding vector V_n can be seen as a filter method of feature selection technique that relies on the criterion of CMI to sort and extract the most significant vectors from a high-dimensional state space, in order to establish a subset of history vectors that maximize the amount of information for predicting future state of target variable, and minimize the redundancy within the set of selected variables. To achieve this purpose without any prior assumptions of sample distribution, one needs to directly calculate the value of CMI from the estimated distribution , and rank the candidate vectors by the CMI in each step to search the most informative vector sequentially. (4)

Nevertheless, as the dimension of conditioning embedding vector V_n is increasing with an growing number of candidate vectors W_n are added, the estimation of probability distributions from a higher dimensional V_n becomes less reliable, which unavoidably renders the criterion of CMI more problematic, and may lead to inaccurate judgements for the inclusion/exclusion of candidate vectors. At this point the major barrier of uniform embedding method of TE—the curse of dimensionality which we hope to bypass through the non-uniform embedding scheme—reemerges. This impediment derives from the sparsity of the available data of an increasing volume of state space that makes the estimation of entropy rates progressively decrease towards zero [14]. For the calculation of TE from multivariate time series this problem is always aggravated by the limited length of data, which may commonly happen in physiological systems as in the case of EEG analysis. To overcome this obstacle, in the following sections we will present a low-dimensional approximation paradigm for TE estimation and test its effectiveness.

Low-dimensional approximation strategy for CMI

The problem of estimating high-dimensional CMI from small samples is a long-standing challenge in the field of information theoretic feature selection and several well-accepted techniques have existed in published literature, for instance, MIFS criterion [24], JMI criterion [25], CMIM criterion [26], MRMR criterion [27], CIFE criterion [28], DISR criterion [29], etc. In recent years Brown et al. [30] have shown that the above various feature selection heuristics are all approximate iterative maximisers of the conditional likelihood, which can be interpreted in a unifying framework of conditional likelihood maximisation under certain assumptions of independence. Hence all the methods can be rewritten within a parameterized general criterion: (5) and the differences among them are determined by the scaling factor β/γ, for example, the CIFE criterion can be obtained with β = γ = 1, the MRMR criterion can be obtained with β = 1/|V| and γ = 0, and the JMI criterion is under the case of β = γ = 1/|V|. In [31] the researchers keep on investigating the theoretical underpinnings of high dimensional CMI and provide a principled approach (RelaxMRMR) for modifying feature selection criterion in which takes into account higher order feature interactions by relaxing some assumption of conditional independence. Compared to (5), the main innovation of RelaxMRMR is on the redundancy term that incorporates the second-order interactions between the features, i.e., the three-way feature interaction terms I(W_k;W_i|W_j). (6) where β = γ = 1/|V| and δ = 1/|V| or 1/|V|(|V| − 1) (Form-1 and Form-2 in [31]).

In this paper, we investigate the above two low-dimensional approximation criteria as a substitute measure to the high dimensional CMI in (2), and compare the performance of the methods with the original non-uniform TE algorithm. Suppose the set of conditioning vectors we have already built is V = {W₁, W₂, …W_(k−1)} and Y_n is the target variable. The formulations of low-dimensional approximations (LA) to CMI here we used are: (7) (8) In LA1 (7), we didn’t take into account the second-order interactions between the candidate vectors, but employed a larger factor β = γ = 2/|V| than the JMI criterion, in order to give an outweighed difference between the redundancy I(W_k;W_j) and conditional redundancy I(W_k;W_j|Y_n), compared to the relevance term I(W_k;Y_n). In LA2 (8) we adopted the same formulation and parameters as in Form-2 in [31], in which the second-order interactions (three-way feature interaction terms) were considered with a normalization factor 1/|V|(|V| − 1).

Estimator of entropy

In this paper, the entropy estimators adopted in the implementation of TE are consistent with previous studies in order to compare the algorithms on a fair basis. The first estimator we used is a binning estimator, which divides the observed state space into a set of equal partitions by a fixed number of quantization levels, and the probability distribution is estimated by relative frequencies of occurrence of the quantization values [32]. The second one is nearest neighbor (NN) estimator [33], it adapts the local neighborhood to the dimension of the state space and makes bias compensation in the estimation of entropies of variables in different dimension. The third one is a linear model-based estimator, which works under the assumption that the multivariate process has a joint Gaussian distribution. It has been demonstrated [4] that the conditional entropy terms for the TE under the Gaussian assumption can be quantified by means of linear regressions involving the relevant variables taken from the embedding vector.

All the above three entropy estimators are the same as the ones employed by previous researchers [9, 13, 14, 23], and in the following analysis these estimators are implemented by means of the respective functions from the MATLAB toolbox MuTE [23]. The primary aim of our proposed method is to avoid the high-dimensional CMI as the ranking criterion in the greedy search procedure for TE from non-uniform embedding, but to adopt the method of low-dimensional approximations to tackle the problem of dimensionality curse. From the simulations in the next section we will demonstrate that applying the low-dimensional approximation criteria as a substitute method for selecting the embedding vectors in TE calculation is beneficial for obtaining a better result with higher sensitivity and specificity under various circumstances.

Simulation study

In this section, we present a series of simulation experiments for causality analysis to compare the performance of low-dimensional approximation methods described in the previous section with the traditional non-uniform TE methods, on a number of multivariate linear and nonlinear stochastic models with various coupling strengths at different interaction lags. In all simulations, 100 realizations were generated for each model with data length 256/512/1024 to assess statistically the sensitivity and specificity of the methods. And the threshold of significance test in the selection loop for candidate vectors was set as α = 0.05, the number of surrogates was fixed to 100.

In terms of sensitivity and specificity, we computed the TE results with respect to a varying length of the analysed data. And the ROC (Receiver Operating Characteristic) curves were obtained for all methods from 256 to 1024 time points. The amount of TP (ture positive), TN (true negative), FP (false positive) and FN (false negative) were estimated and classified by grouping all coupled directions (positives) and all uncoupled directions (negatives), to calculate the sensitivity = TP/(TP + FN), specificity = TN/(TN + FP), and F1 score = 2TP/(2TP + FP + FN).

For the first two linear models, we used the linear entropy estimator since it works optimally for the time series with a joint Gaussian distribution. Regarding the other three nonlinear models, binning and nearest neighbor estimators were employed to evaluate the difference of performance.

Model A

Let us start with a linear vector autoregressive (VAR) model which is composed by 4 time series of order 5 (Model 1 in [34]). The equations for this model are: (9) where ϵ_i(t), i = 1, …, 4 are unit Gaussian noise terms with zero mean and unit covariance matrix. We performed the causal analysis by setting the maximum lag according to the largest lag in the generating process, and evaluate the transfer entropy from non-uniform embedding by the traditional algorithm and our low-dimensional approximations searching methods (LA1, LA2). The results from model A for data length of 1024 points are shown in Fig 1, depicted by a causal matrix of interactions from row to column for each method, the ROC curves of 256/512/1024 data points are shown in Fig 2 and the values of sensitivity/specificity/F1 score listed in Table 1. The true causal connections in model A are at the matrix elements (1, 3), (2, 1), (2, 3), and (4, 2). For all three methods the deduced values of TE on these positions are always positive and high, demonstrate a good sensitivity for the true couplings. However we can notice that a number of false positives were given by the traditional method using the high dimensional CMI, especially for the large data of 1024 points. In contrast, although both low-dimensional approximation methods presented a relatively low rejection rate for the data of 256 and 512, for 1024 points they attained a better performance with respect to the sensitivity/specificity/F1 score. And LA1 in this model provides the best possible result compared to LA2.

Download:

Fig 1. Matrix representations of the corresponding causalities for Model A.

Retrieved by traditional TE method (a) and two low-dimensional approximation methods (b) (c) with linear estimator. The results are averaged over 100 simulations of 1024 time points and shown with color revealing the magnitude and transparency indicating the significance. The directions of causal influence are from row to column.

https://doi.org/10.1371/journal.pone.0194382.g001

Download:

Fig 2. ROC curves for Model A.

Sensitivity and specificity are obtained by gradually increasing the time series length from 256 to 1024 points.

https://doi.org/10.1371/journal.pone.0194382.g002

Download:

Table 1. Sensitivity, specificity and F1 score values obtained from Model A.

https://doi.org/10.1371/journal.pone.0194382.t001

Model B

The second model is another linear VAR of order 4 in five variables with unit Gaussian noise: (10) The true causal connections in this model are at the matrix elements (1, 2), (1, 4), (2, 4), (4, 5), (5, 1) and (5, 2). The results from model B for data of 1024 points are shown in Fig 3. ROC curves are shown in Fig 4 and the values of sensitivity/specificity/F1 score listed in Table 2. The scales in Fig 4 are different from Fig 2 but the trends are similar: for the time series of 512/1024 points, both LA1 and LA2 give better performance than the original TE, and the specificity of LA1 is slightly higher than LA2 because the former is a more stringent criterion in the sense of the scaling factor. While for the traditional TE method, a large number of non-relevant connections were identified as significant, and the false positives distributed across the whole matrix, which also gave rise to a certain amount of redundant computation time. The outcome of Model A and B demonstrates that for linear systems the original TE from non-uniform embedding may bring about false positives at a rate higher than the nominal rate of α = 0.05, this fact has been already noticed by previous researchers [13], through introducing the low-dimensional approximation technique for the searching procedure in state space, our proposed method is able to avert this defect and retain the advantage of high sensitivity within the non-uniform embedding scheme.

Download:

Fig 3. Matrix representations of the corresponding causalities for Model B.

Retrieved by traditional TE method (a) and two low-dimensional approximation methods (b) (c) with linear estimator. The results are averaged over 100 simulations of 1024 time points and shown with color revealing the magnitude and transparency indicating the significance. The directions of causal influence are from row to column.

https://doi.org/10.1371/journal.pone.0194382.g003

Download:

Fig 4. ROC curves for Model B.

Sensitivity and specificity are obtained by gradually increasing the time series length from 256 to 1024 points.

https://doi.org/10.1371/journal.pone.0194382.g004

Download:

Table 2. Sensitivity, specificity and F1 score values obtained from Model B.

https://doi.org/10.1371/journal.pone.0194382.t002

Model C

Now we will consider three nonlinear models in which the binning and nearest neighbor estimator will be applied to evaluate the entropy. The first nonlinear model is composed by four time series which has the same coupling strength and causal direction as in the model A: (11) The differences between model A and C are the nonlinear interactions in model C at the X₁ → X₃ and X₄ → X₂. For the calculation method of entropy we set 6 quantization levels for binning estimator and 10 nearest neighbors for NN estimator. The results for data of 1024 are shown in Fig 5, ROC curves shown in Fig 6 and the values of sensitivity/specificity/F1 score listed in Table 3, respectively labeled by the three computation methods (TE/LA1/LA2) and two entropy estimators (b/n). For the binning estimator, the specificity of LA1-b and LA2-b is higher than TE-b, with a comparably similar sensitivity. In the case of the NN estimator, traditional TE-n method failed to detect the causal relationship X₂ → X₃ for any data length, rendered a sensitivity of 75% at most. LA2-n also provided the same sensitivity with a slightly higher specificity, however LA1-n has successfully detected all the correct causal links in this example and attained the best sensitivity (100%) for 1024 points, though the value of TE at the X₁ → X₃ is relatively lower than the other methods.

Download:

Fig 5. Matrix representations of the corresponding causalities for Model C.

Retrieved by traditional TE method and two low-dimensional approximation methods, respectively by using binning entropy estimator (a, b, c) and NN entropy estimator (d, e, f) over 100 simulations of 1024 time points. The results are shown with color revealing the magnitude and transparency indicating the significance. The directions of causal influence are from row to column.

https://doi.org/10.1371/journal.pone.0194382.g005

Download:

Fig 6. ROC curves for Model C.

TE-b/LA1-b/LA2-b are the results by binning entropy estimator and TE-n/LA1-n/LA2-n by NN entropy estimator, sensitivity and specificity are obtained by gradually increasing the time series length from 256 to 1024 points.

https://doi.org/10.1371/journal.pone.0194382.g006

Download:

Table 3. Sensitivity, specificity and F1 score values obtained from Model C by binning(-b) and NN(-n) estimators.

https://doi.org/10.1371/journal.pone.0194382.t003

Model D

The fourth example is another nonlinear VAR process of order 3 in 5 variables (the nonlinear model in [23], Eq (14)): (12) The true causal connections in this model are at the matrix elements (1, 2), (1, 3), (1, 4), (4, 5), and (5, 4). The results for data of 1024 are shown in Fig 7, ROC curves shown in Fig 8 and the values of sensitivity/specificity/F1 score listed in Table 4. For the binning estimator, LA2-b presented a lower specificity compared LA1-b, but it was able to retrieve all the true links for 1024 points. The sensitivity of LA1-b is a bit lower than 1, which is improved by LA1-n. Regarding the TE method, the outcome of TE-n is worse than TE-b, despite the fact that the nearest neighbor estimator as a locally adaptive kernel estimator is more efficient for calculating entropies in high-dimensional spaces for limited data. In contrast, the proposed low-dimensional approximation technique for TE exploits the advantage of nearest neighbor estimator and attains better performance for both methods under the data length of 1024, in which the detection accuracy for the causal drivers is 100%.

Download:

Fig 7. Matrix representations of the corresponding causalities for Model D.

Retrieved by traditional TE method and two low-dimensional approximation methods, respectively by using binning entropy estimator (a, b, c) and NN entropy estimator (d, e, f) over 100 simulations of 1024 time points. The results are shown with color revealing the magnitude and transparency indicating the significance. The directions of causal influence are from row to column.

https://doi.org/10.1371/journal.pone.0194382.g007

Download:

Fig 8. ROC curves for Model D.

TE-b/LA1-b/LA2-b are the results by binning entropy estimator and TE-n/LA1-n/LA2-n by NN entropy estimator, sensitivity and specificity are obtained by gradually increasing the time series length from 256 to 1024 points.

https://doi.org/10.1371/journal.pone.0194382.g008

Download:

Table 4. Sensitivity, specificity and F1 score values obtained from Model D by binning(-b) and NN(-n) estimators.

https://doi.org/10.1371/journal.pone.0194382.t004

Model E

In the fifth example we consider a lattice of five unidirectionally coupled noisy logistic maps, (13) where ρ = 0.2 is the coupling strength and ϵ are unit variance Gaussian noise terms. In this system, the first variable is evolving autonomously, whereas the other variables are influenced by the previous one with coupling ρ, thus forming a cascade of interactions x_i−1 → x_i. The results for data of 1024 are shown in Fig 9, ROC curves shown in Fig 10 and the values of sensitivity/specificity/F1 score listed in Table 5. Since the causal relationships in this model are relatively regular, all the three methods are able to identify the true interaction links and provide a good sensitivity and specificity, and the results from NN entropy estimator are all better than the binning estimator for this model. For the longest data of 1024 points the best result is given by LA2-n, which is nearly the same as LA1-n. It shows that with a proper data size, the F1 score obtained by LA1 and LA2 are always higher than the TE method.

Download:

Fig 9. Matrix representations of the corresponding causalities for Model E.

Retrieved by traditional TE method and two low-dimensional approximation methods, respectively by using binning entropy estimator (a, b, c) and NN entropy estimator (d, e, f) over 100 simulations of 1024 time points. The results are shown with color revealing the magnitude and transparency indicating the significance. The directions of causal influence are from row to column.

https://doi.org/10.1371/journal.pone.0194382.g009

Download:

Fig 10. ROC curves for Model E.

TE-b/LA1-b/LA2-b are the results by binning entropy estimator and TE-n/LA1-n/LA2-n by NN entropy estimator, sensitivity and specificity are obtained by gradually increasing the time series length from 256 to 1024 points.

https://doi.org/10.1371/journal.pone.0194382.g010

Download:

Table 5. Sensitivity, specificity and F1 score values obtained from Model E by binning(-b) and NN(-n) estimators.

https://doi.org/10.1371/journal.pone.0194382.t005

In this example the noise term is defined by a superimposed form with a relatively little strength in accordance with previous researches [35, 36]. When noise strength is larger than 0.04 all the methods failed to detect causal relationships between the variables. To illustrate this, we conducted an experiment varying the noise strength from 0.01 to 0.04 and the coupling strength ρ = 0.1, 0.3, 0.5, and depicted the results in Fig 11 (data length = 512). In the results by using binning estimator, it shows that the both LA methods perform better than original TE method. Except in the example of weak coupling by NN estimator, the method of LA1 gave out a relative low sensitivity, while the LA2 is still superior to the traditional TE.

Download:

Fig 11. ROC curves for Model E with varying noise and coupling strengths.

Sensitivity and specificity are obtained for models with a varying noise from 0.01 to 0.04 (marked in the figure). The first row shows results from the methods applying binning estimator and second row for NN estimator. Column 1 to 3 is with different coupling strength from 0.1, 0.3 to 0.5. Each simulation was performed 100 runs with data length 512.

https://doi.org/10.1371/journal.pone.0194382.g011

Model F

Next we consider three coupled Henon maps (14) Nonlinear couplings x₁ → x₂ and x₂ → x₃ are defined with equal coupling strengths c = 0.1, 0.3, 0.5, under which the system does not become completely synchronized. The results of ROC curves for different data lengths and coupling strengths are shown in Fig 12, and the values of sensitivity/specificity/F1 score listed in Table 6. In the case of weak coupling (c = 0.1), both LA methods could not achieve a good sensitivity compared to traditional TE (except for the 1024 data by NN estimator). But for the data of medium and strong coupling (c = 0.3, 0.5), the sensitivity of LA methods reached 100% together with a high specificity. In results by binning estimator, the LA1 method performs better than the other two, especially for data of longer length (512, 1024), while by NN estimator LA2 is more advantageous. Moreover, the specificity of all three methods by NN estimator tends to decrease with the coupling strength of model.

Download:

Fig 12. ROC curves for Model F with different data lengths and coupling strengths.

Sensitivity and specificity are obtained from three coupled Henon maps with data length from 256 to 1024 and coupling strength (marked in the figure) from 0.1 to 0.5. The first row shows results from the methods applying binning estimator and second row for NN estimator. Column 1 to 3 is with data length from 256 to 1024. Each simulation was performed by 100 runs.

https://doi.org/10.1371/journal.pone.0194382.g012

Download:

Table 6. Sensitivity, specificity and F1 score values obtained from Model F by binning(-b) and NN(-n) estimators with different data lengths and coupling strengths.

https://doi.org/10.1371/journal.pone.0194382.t006

Model G

In the last simulation, a system of three coupled identical Lorenz oscillators is defined as (15) with couplings x₁ → x₂ and x₂ → x₃ of equal strengths c = 1, 2, 3, 4, 5. The first variables x_i of the three interacting subsystems are respectively observed at a sampling time of 0.05 units, and the differential equations is solved using the explicit Runge-Kutta (4, 5) method “ode45” in MATLAB. The results of ROC curves for different data lengths and coupling strengths are shown in Fig 13, and the values of sensitivity/specificity/F1 score listed in Table 7. In most cases of this example, the traditional TE method outperformed LA methods, which indicates that this coupled continuous chaotic system cannot benefit from low-dimensional approximation strategy. The results for LA1 method are especially worse than LA2, partly due to the inclusion/exclusion of the higher order interactions among variables. However in the simulation of larger coupling strength (c = 4, 5) with data length of 1024 by binning method, LA2 achieves higher F1 score than TE, and with longer data the performance of LA2 by NN estimator is also significantly improved. This fact demonstrates that for LA2 method, data size is crucial for the estimation of higher-dimensional influences.

Download:

Fig 13. ROC curves for Model G with different data lengths and coupling strengths.

Sensitivity and specificity are obtained from three coupled Lorenz oscillators with data length from 256 to 1024 and coupling strength (marked in the figure) from 1 to 5. The first row shows results from the methods applying binning estimator and second row for NN estimator. Column 1 to 3 is with data length from 256 to 1024. Each simulation was performed by 100 runs.

https://doi.org/10.1371/journal.pone.0194382.g013

Download:

Table 7. Sensitivity, specificity and F1 score values obtained from Model G by binning(-b) and NN(-n) estimators with different data lengths and coupling strengths.

https://doi.org/10.1371/journal.pone.0194382.t007

Application

In this section we turn to real-world data to show the applicability of our proposed approach for non-uniform embedding. To address this issue we consider a public dataset [37] from an epileptic patient with implanted array of 8-by-8 cortical electrode grid and two depth electrode strips with six contacts each, which amount to 76 time series (More details about the data are given in [38]). For its intrinsic high dimensionality and redundancy, this data is intuitively appropriate to be employed a non-uniform embedding method to disentangle the underlying dynamical interactions. Therefore we applied our low-dimensional approximation method (LA1) to analyze the data corresponding to 8 epileptic seizures and 8 periods just before the seizure onset, respectively averaged the ictal/pre-ictal results and then compared its performance with traditional non-uniform embedding transfer entropy, both by the implementation of nearest neighbor estimator. In this application the data which were recorded at 400 Hz of 10 seconds length were downsampled to 100 Hz, and the maximum embedding order was set to 8. The results are depicted in Fig 14, which shows the matrices of causalities before and during the clinical onset of the seizure, and Fig 15, which shows the difference of total numbers of significant connections between ictal and pre-ictal period, respectively by TE and LA method. From matrix representation by the LA method for the pre-ictal data we note that an obvious causal driver is located at the contact 73 from the second depth electrode strip, this contact can thus be associated to the seizure onset and reflects the fact that the seizure is already active even if it is not yet clinically observable, also in [35, 39], it has been suggested that the last two contacts in the second depth electrode are mostly influencing the cortical activity. The results from ictal data indicates abnormal interactions from the regions corresponding to the lower left corner of the grid (contacts 1–4, 9–11 and 17) which were then resected during an anterior temporal lobectomy for the patient. Moreover, from the ictal period our method has successfully identified a critical node: contact 50, which exhibits the most significant change in the value of betweenness centrality and was considered as a target for therapeutic intervention in [38], for the reason that these nodes with statistically significant increases in betweenness centrality may facilitate seizure activity and their disruption could prevent or abort ictal activity. The traditional TE method, on the contrary, leads to more noisy results with a significant number of false positives, which has extended the computation time to about three folds larger, and it is more sensitive to the confounding effect of volume conduction resulting in the diagonal patterns observed in the matrix compared with our method.

Download:

Fig 14. Results for an epileptic EEG recording.

Matrices of causalities reflect before (top) and during (down) the clinical onset of a seizure from an epileptic patient, the results are respectively averaged from 8 recordings. Contacts 1 to 64 belong to a cortical electrode grid, and contacts 65 to 76 to two depth electrode strips. The values are computed by traditional non-uniform transfer entropy (TE) and low-dimensional approximation approach (LA). The directions of causal influence are from row to column. The brighter colors correspond to more significant values.

https://doi.org/10.1371/journal.pone.0194382.g014

Download:

Fig 15. Results for an epileptic EEG recording.

Matrices depict the difference of total numbers of significant connections between ictal and pre-ictal period (ict—pre). The numbers are respectively summed from 8 recordings. Contacts 1 to 64 belong to a cortical electrode grid, and contacts 65 to 76 to two depth electrode strips. The values are computed by traditional non-uniform transfer entropy (TE) and low-dimensional approximation approach (LA). The directions of causal influence are from row to column. The brighter colors correspond to more significant values.

https://doi.org/10.1371/journal.pone.0194382.g015

Discussion and conclusion

In this article we have presented a novel and effective modification for the well-known causality analysis tool: transfer entropy from non-uniform embedding, which is the state-of-the-art for quantifying causal networks by means of information theoretic measures and has been employed by a large number of researchers from a variety of disciplines over recent years. Besides the widely used transfer entropy as a model-free approach there are also other forms of causality measures that based on different criterion rather than the maximization of CMI and has shown its great statistical power in detecting relationships from a multivariate embedding space [40], e.g. local predictability (LP). For the computation of transfer entropy, the employment of non-uniform embedding strategy has been proved to be suitable for high dimensional embedding spaces because of its ability to reduce the effective dimension of the state space by selecting only the relevant variables at specific time lags that contribute the most to explain the target variable. Thus it is a more flexible procedure for the reconstruction of the multivariate embedding space compared to the uniform embedding and has been proposed by several previous authors [7, 22, 41]. Nevertheless it still encounters the obstacle of estimation for CMI among the candidate vectors and tends to detect false directed couplings. Our suggested approach exploits low-dimensional approximation method that derives from the feature selection framework to improve the termination criterion in the procedure of constructing embedding vectors. The detailed approximation strategies we used are formed in two different ways: one is a stringent version of JMI criterion and the other is RelaxMRMR criterion that takes into account the second order interactions. In a series of simulations we compared the performance of our algorithm with traditional TE implementation under various entropy estimators. We reported the sensitivity and specificity of the results from different data lengths, confirmed the capability of the proposed approach, which could lead to less false positives in the detection of causal flow while retaining the advantage of non-uniform embedding in terms of high sensitivity, especially for the large data sets.

The main development in this paper is that instead of directly applying original CMI as a criterion for identifying significant variables at specific lags as in recent literature, our approach follows a paradigm that makes use of low-dimensional mutual information quantities to approximate higher dependencies between candidate and target variables in the embedding vector reconstruction. It effectively avoids the curse of dimensionality and achieves a more parsimonious model by maximizing relevance and minimizing redundancy between the selected components. To tackle this kind of problem there also exists other propositions, e.g. in [42] the authors introduced a preselection scheme for the subsets of causal predictors to overcome the combinatorial explosion for searching a globally optimal subset and detect the synergetic variables, while for our method it relies on the low-dimensional approximation to alleviate this condition. In the simulation experiments, the approach LA1 with an outweighed factor on the redundancy and conditional redundancy term generally outperformed LA2 which took higher order interactions of embedding vectors at different lags into consideration, and presented the best possible results throughout most of the examples except for the coupled Lorenz system as long as data size is large enough for good statistic. The reason that LA2 did not show much obvious advantage compared to LA1 may partly due to the number of candidate variables in state space is limited to a small number with a small time delay, thus renders the underlying causal structure of the data set not complex enough. In this case a simple criterion such as LA1 may already be sufficiently capable to give a reasonable outcome. Another fact is that the performance of an approximation method is not only affected by the dimensionality of the state space, but also by the data size. Therefore a large data is crucial for the estimation of higher-dimensional influences, which has been shown in the last example.

As with the proposed methodology there are also some limitations. For instance, although in the simulations of linear model our algorithm produced a better result compared to the original non-uniform transfer entropy, it still could not surpass the efficiency of the standard uniform conditioning methods [23], and it is more computationally intensive for the associated randomization significance tests, which is intrinsically required under the non-uniform conditioning framework. For this reason the beneficial effects of applying the low-dimensional approximations for TE may rely mostly upon the nonlinear causal relationships, like in the EEG or MEG data. Also in the experiments of chaotic systems our method did not show obvious advantage compared to TE, and sometimes even failed to detect the underlying causal relationship, which restricted its range of applicability and this issue should be investigated in our further work. In [43], the authors had carried out a simulation study to compare the performance of several causality measures and concluded that the non-uniform TE leading to the best in the case of nonlinear simulation systems and always obtaining the highest specificity. In our simulations this was confirmed that the traditional TE method could already give a 100% sensitivity and nearly 0 false positives and leaves little space to be improved, this may to some extent explain what kind of system is most suitable for the applying of non-uniform scheme. Another issue for the low-dimensional approximation approach is that it is prone to bring about more false negatives, especially for small data sets. A polished version of this method should address this problem theoretically, perhaps using an adaptive balancing factor in which the focus may shift between the relevance and redundancy terms as the unexplained information of the target variable decreases from the earlier to the latter stage of searching and constructing embedding vectors, rather than a fixed parameter pattern for the relevance/redundancy/conditional redundancy in the embedding vector selection process. For different entropy estimators, the effect of low-dimensional approximations differs in the trade-off between the information gain and the nonexistent coupling rejection. A more flexible termination criterion will certainly be helpful to discover the true coupling direction and latency in multivariate causality analysis.

Supporting information

S1 File. Matlab code for LA method by BIN-NN estimator.

https://doi.org/10.1371/journal.pone.0194382.s001

(ZIP)

References

1. Granger CWJ. Investigating causal relations by econometric models and cross-spectral methods. Econometrica. 1969; 37: 424–438.
- View Article
- Google Scholar
2. Marko H. The Bidirectional Communication Theory—A Generalization of Information Theory. IEEE Trans Commun. 1973; 21: 1345–1351.
- View Article
- Google Scholar
3. Schreiber T. Measuring information transfer. Phys Rev Lett. 2000; 85: 461–464. pmid:10991308
- View Article
- PubMed/NCBI
- Google Scholar
4. Barnett L, Barett AB, and Seth AK. Granger causality and transfer entropy are equivalent for Gaussian variables. Phys Rev Lett. 2009; 103: 23.
- View Article
- Google Scholar
5. Hlaváčková-Schindler K. Equivalence of granger causality and transfer entropy: a generalization. Appl Math Sci. 2011; 5: 3637–3648.
- View Article
- Google Scholar
6. Hlinka J, Paluš M, Vejmelka M, Mantini D, Corbetta M. Functional connectivity in resting-state fMRI: Is linear correlation sufficient? Neuroimage. 2011; 54: 2218–2225. pmid:20800096
- View Article
- PubMed/NCBI
- Google Scholar
7. Faes L, Nollo G, Porta A. Information-based detection of nonlinear granger causality in multivariate processes via a nonuniform embedding technique. Phys Rev E. 2011; 83: 051112.
- View Article
- Google Scholar
8. Faes L, Nollo G, Porta A. Non-uniform multivariate embedding to assess the information transfer in cardiovascular and cardiorespiratory variability series. Comput Biol Med. 2012; 42: 290–297. pmid:21419400
- View Article
- PubMed/NCBI
- Google Scholar
9. Faes L, Marinazzo D, Montalto A, Nollo G. Lag-Specific Transfer Entropy as a Tool to Assess Cardiovascular and Cardiorespiratory Information Transfer. IEEE Trans Biomed Eng. 2014; 61: 2556–2568. pmid:24835121
- View Article
- PubMed/NCBI
- Google Scholar
10. Runge J, Heitzig J, Petoukhov V, Kurths J. Escaping the curse of dimensionality in estimating multivariate transfer entropy. Phys Rev Lett. 2012; 108: 258701. pmid:23004667
- View Article
- PubMed/NCBI
- Google Scholar
11. Runge J, Heitzig J, Marwan N, Kurths J. Quantifying causal coupling strength: a lag-specific measure for multivariate time series related to transfer entropy. Phys Rev E. 2012; 86: 061121.
- View Article
- Google Scholar
12. Hlinka J, Hartman D, Vejmelka M, Runge J, Marwan N, Kurths J, et al. Reliability of inference of directed climate networks using conditional mutual information. Entropy. 2013; 15: 2023–2045.
- View Article
- Google Scholar
13. Kugiumtzis D. Direct coupling information measure from non-uniform embedding. Phys Rev E. 2013; 87: 062918.
- View Article
- Google Scholar
14. Wibral M, Vicente R, Lizier JT. Directed Information Measures in Neuroscience. Berlin; Heidelberg: Springer-Verlag. 2014.
15. Penm JHW, Terrell RD. Multivariate subset autoregressive modelling with zero constraints for detecting overall causality. J Econom. 1984; 24: 311–330.
- View Article
- Google Scholar
16. Penm JHW, Terrell RD. The derived moving average model and its role in causality. J Appl Probab. 1986; 23: 99–111.
- View Article
- Google Scholar
17. Brüggemann R, Lütkepohl H. “Lag selection in Subset VAR models with an application to a U.S. monetary system” in Econometric Studies—A Festschrift in Honour of Joachim Frohn, eds R. Friedmann, L. Knüppel, and H. Lütkepohl (LIT: Münster), 2001; 107–128.
18. Brüggemann R, Krolzig HM, Lütkepohl H. Comparison of model reduction methods for VAR processes. Technical Report 2003-W13. Economics Group, Nuffield College. Oxford: University of Oxford. 2003.
19. Gatu C, Kontoghiorghes EJ. Efficient strategies for deriving the subset VAR models. Comput Manage Sci. 2005; 2: 253–278.
- View Article
- Google Scholar
20. Faes L, Kugiumtzis D, Nollo G, Jurysta F, Marinazzo D. Estimating the decomposition of predictive information in multivariate systems. Phys Rev E. 2015; 91: 032904.
- View Article
- Google Scholar
21. Judd K, Mees A. Embedding as a modeling problem. Physica D. 1998; 120: 273–286.
- View Article
- Google Scholar
22. Vlachos I, Kugiumtzis D. Nonuniform state-space reconstruction and coupling detection. Phys Rev E. 2010; 82: 016207.
- View Article
- Google Scholar
23. Montalto A, Faes L, Marinazzo D. MuTE: A MATLAB Toolbox to Compare Established and Novel Estimators of the Multivariate Transfer Entropy. PLoS ONE. 2014; 9: e109462. pmid:25314003
- View Article
- PubMed/NCBI
- Google Scholar
24. Battiti R. Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw. 1994; 5: 537–550. pmid:18267827
- View Article
- PubMed/NCBI
- Google Scholar
25. Yang H, Moody J. Data visualization and feature selection: New algorithms for nongaussian data. Adv Neural Inf Process Syst. 1999; 12: 687–693.
- View Article
- Google Scholar
26. Fleuret F. Fast binary feature selection with conditional mutual information. J Mach Learn Res. 2004; 5: 1531–1555.
- View Article
- Google Scholar
27. Peng H, Long F, Ding C. Feature selection based on mutual information: Criteria of max dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell. 2005; 27: 1226–1238. pmid:16119262
- View Article
- PubMed/NCBI
- Google Scholar
28. Lin D, Tang X. Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion. In: Leonardis A, Bischof H, Pinz A. (eds) Computer Vision—ECCV 2006. ECCV 2006. Lecture Notes in Computer Science, vol 3951. Springer, Berlin, Heidelberg. 2006.
29. Meyer PE, Bontempi G. On the Use of Variable Complementarity for Feature Selection in Cancer Classification. In: Rothlauf F. et al. (eds) Applications of Evolutionary Computing. EvoWorkshops 2006. Lecture Notes in Computer Science, vol 3907. Springer, Berlin, Heidelberg. 2006.
30. Brown G, Pocock A, Zhao MJ, Luján M. Conditional likelihood maximisation: a unifying framework for information theoretic feature selection, J Mach Learn Res. 2012; 13: 27–66.
- View Article
- Google Scholar
31. Vinh NX, Zhou S, Chan J, Bailey J. Can high-order dependencies improve mutual information based feature selection? Pattern Recognit. 2016; 53: 46–58.
- View Article
- Google Scholar
32. Hlaváčková-Schindler K, Paluš M, Vejmelka M. Causality detection based on information-theoretic approaches in time series analysis. Phys Rep. 2007; 441: 1–46.
- View Article
- Google Scholar
33. Kraskov A, Stögbauer H, Grassberger P. Estimation mutual information, Phys Rev E. 2004; 69: 066138.
- View Article
- Google Scholar
34. Winterhalder M, Schelter B, Hesse W, Schwab K, Leistritz L, Klan D, et al. Comparison of linear signal processing techniques to infer directed interactions in multivariate neural systems. Signal Process. 2005; 85: 2137–2160.
- View Article
- Google Scholar
35. Marinazzo D, Pellicoro M, Stramaglia S. Kernel Method for Nonlinear Granger Causality. Phys. Rev. Lett. 100, 144103. pmid:18518037
- View Article
- PubMed/NCBI
- Google Scholar
36. Stramaglia S, Cortes JM, Marinazzo D. Synergy and redundancy in the Granger causal analysis of dynamical networks. New J. Phys. 2014, 16, 105003.
- View Article
- Google Scholar
37. http://math.bu.edu/people/kolaczyk/datasets.html .
38. Kramer MA, Kolaczyk ED, Kirsch HE. Emergent network topology at seizure onset in humans. Epilepsy Res. 2008; 79: 173–186. pmid:18359200
- View Article
- PubMed/NCBI
- Google Scholar
39. Faes L, Marinazzo D, Stramaglia S. Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes. Entropy 2017, 19, 408.
- View Article
- Google Scholar
40. Porta A, Faes L, Bari V, Marchi A, Bassani T, et al. Effect of Age on Complexity and Causality of the Cardiovascular Control: Comparison between Model-Based and Model-Free Approaches. PLoS ONE. 2014; 9: e89463. pmid:24586796
- View Article
- PubMed/NCBI
- Google Scholar
41. Porta A, Faes L. Wiener-Granger causality in network physiology with applications to cardiovascular control and neuroscience. Proceedings of the IEEE. 2016; 104: 282–309.
- View Article
- Google Scholar
42. Runge J, Donner R, Kurths J. Optimal model-free prediction from multivariate time series. Phys Rev E. 2015; 91: 052909.
- View Article
- Google Scholar
43. Papana A, Kyrtsou C, Kugiumtzis D, Djks C. Simulation Study of Direct Causality Measures in Multivariate Time Series. Entropy 2013, 15, 2635–2661.
- View Article
- Google Scholar

[ref1] 1. Granger CWJ. Investigating causal relations by econometric models and cross-spectral methods. Econometrica. 1969; 37: 424–438.
View Article
Google Scholar

[2] View Article

[3] Google Scholar

[ref2] 2. Marko H. The Bidirectional Communication Theory—A Generalization of Information Theory. IEEE Trans Commun. 1973; 21: 1345–1351.
View Article
Google Scholar

[5] View Article

[6] Google Scholar

[ref3] 3. Schreiber T. Measuring information transfer. Phys Rev Lett. 2000; 85: 461–464. pmid:10991308
View Article
PubMed/NCBI
Google Scholar

[8] View Article

[9] PubMed/NCBI

[10] Google Scholar

[ref4] 4. Barnett L, Barett AB, and Seth AK. Granger causality and transfer entropy are equivalent for Gaussian variables. Phys Rev Lett. 2009; 103: 23.
View Article
Google Scholar

[12] View Article

[13] Google Scholar

[ref5] 5. Hlaváčková-Schindler K. Equivalence of granger causality and transfer entropy: a generalization. Appl Math Sci. 2011; 5: 3637–3648.
View Article
Google Scholar

[15] View Article

[16] Google Scholar

[ref6] 6. Hlinka J, Paluš M, Vejmelka M, Mantini D, Corbetta M. Functional connectivity in resting-state fMRI: Is linear correlation sufficient? Neuroimage. 2011; 54: 2218–2225. pmid:20800096
View Article
PubMed/NCBI
Google Scholar

[18] View Article

[19] PubMed/NCBI

[20] Google Scholar

[ref7] 7. Faes L, Nollo G, Porta A. Information-based detection of nonlinear granger causality in multivariate processes via a nonuniform embedding technique. Phys Rev E. 2011; 83: 051112.
View Article
Google Scholar

[22] View Article

[23] Google Scholar

[ref8] 8. Faes L, Nollo G, Porta A. Non-uniform multivariate embedding to assess the information transfer in cardiovascular and cardiorespiratory variability series. Comput Biol Med. 2012; 42: 290–297. pmid:21419400
View Article
PubMed/NCBI
Google Scholar

[25] View Article

[26] PubMed/NCBI

[27] Google Scholar

[ref9] 9. Faes L, Marinazzo D, Montalto A, Nollo G. Lag-Specific Transfer Entropy as a Tool to Assess Cardiovascular and Cardiorespiratory Information Transfer. IEEE Trans Biomed Eng. 2014; 61: 2556–2568. pmid:24835121
View Article
PubMed/NCBI
Google Scholar

[29] View Article

[30] PubMed/NCBI

[31] Google Scholar

[ref10] 10. Runge J, Heitzig J, Petoukhov V, Kurths J. Escaping the curse of dimensionality in estimating multivariate transfer entropy. Phys Rev Lett. 2012; 108: 258701. pmid:23004667
View Article
PubMed/NCBI
Google Scholar

[33] View Article

[34] PubMed/NCBI

[35] Google Scholar

[ref11] 11. Runge J, Heitzig J, Marwan N, Kurths J. Quantifying causal coupling strength: a lag-specific measure for multivariate time series related to transfer entropy. Phys Rev E. 2012; 86: 061121.
View Article
Google Scholar

[37] View Article

[38] Google Scholar

[ref12] 12. Hlinka J, Hartman D, Vejmelka M, Runge J, Marwan N, Kurths J, et al. Reliability of inference of directed climate networks using conditional mutual information. Entropy. 2013; 15: 2023–2045.
View Article
Google Scholar

[40] View Article

[41] Google Scholar

[ref13] 13. Kugiumtzis D. Direct coupling information measure from non-uniform embedding. Phys Rev E. 2013; 87: 062918.
View Article
Google Scholar

[43] View Article

[44] Google Scholar

[ref14] 14. Wibral M, Vicente R, Lizier JT. Directed Information Measures in Neuroscience. Berlin; Heidelberg: Springer-Verlag. 2014.

[ref15] 15. Penm JHW, Terrell RD. Multivariate subset autoregressive modelling with zero constraints for detecting overall causality. J Econom. 1984; 24: 311–330.
View Article
Google Scholar

[47] View Article

[48] Google Scholar

[ref16] 16. Penm JHW, Terrell RD. The derived moving average model and its role in causality. J Appl Probab. 1986; 23: 99–111.
View Article
Google Scholar

[50] View Article

[51] Google Scholar

[ref17] 17. Brüggemann R, Lütkepohl H. “Lag selection in Subset VAR models with an application to a U.S. monetary system” in Econometric Studies—A Festschrift in Honour of Joachim Frohn, eds R. Friedmann, L. Knüppel, and H. Lütkepohl (LIT: Münster), 2001; 107–128.

[ref18] 18. Brüggemann R, Krolzig HM, Lütkepohl H. Comparison of model reduction methods for VAR processes. Technical Report 2003-W13. Economics Group, Nuffield College. Oxford: University of Oxford. 2003.

[ref19] 19. Gatu C, Kontoghiorghes EJ. Efficient strategies for deriving the subset VAR models. Comput Manage Sci. 2005; 2: 253–278.
View Article
Google Scholar

[55] View Article

[56] Google Scholar

[ref20] 20. Faes L, Kugiumtzis D, Nollo G, Jurysta F, Marinazzo D. Estimating the decomposition of predictive information in multivariate systems. Phys Rev E. 2015; 91: 032904.
View Article
Google Scholar

[58] View Article

[59] Google Scholar

[ref21] 21. Judd K, Mees A. Embedding as a modeling problem. Physica D. 1998; 120: 273–286.
View Article
Google Scholar

[61] View Article

[62] Google Scholar

[ref22] 22. Vlachos I, Kugiumtzis D. Nonuniform state-space reconstruction and coupling detection. Phys Rev E. 2010; 82: 016207.
View Article
Google Scholar

[64] View Article

[65] Google Scholar

[ref23] 23. Montalto A, Faes L, Marinazzo D. MuTE: A MATLAB Toolbox to Compare Established and Novel Estimators of the Multivariate Transfer Entropy. PLoS ONE. 2014; 9: e109462. pmid:25314003
View Article
PubMed/NCBI
Google Scholar

[67] View Article

[68] PubMed/NCBI

[69] Google Scholar

[ref24] 24. Battiti R. Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw. 1994; 5: 537–550. pmid:18267827
View Article
PubMed/NCBI
Google Scholar

[71] View Article

[72] PubMed/NCBI

[73] Google Scholar

[ref25] 25. Yang H, Moody J. Data visualization and feature selection: New algorithms for nongaussian data. Adv Neural Inf Process Syst. 1999; 12: 687–693.
View Article
Google Scholar

[75] View Article

[76] Google Scholar

[ref26] 26. Fleuret F. Fast binary feature selection with conditional mutual information. J Mach Learn Res. 2004; 5: 1531–1555.
View Article
Google Scholar

[78] View Article

[79] Google Scholar

[ref27] 27. Peng H, Long F, Ding C. Feature selection based on mutual information: Criteria of max dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell. 2005; 27: 1226–1238. pmid:16119262
View Article
PubMed/NCBI
Google Scholar

[81] View Article

[82] PubMed/NCBI

[83] Google Scholar

[ref28] 28. Lin D, Tang X. Conditional Infomax Learning: An Integrated Framework for Feature Extraction and Fusion. In: Leonardis A, Bischof H, Pinz A. (eds) Computer Vision—ECCV 2006. ECCV 2006. Lecture Notes in Computer Science, vol 3951. Springer, Berlin, Heidelberg. 2006.

[ref29] 29. Meyer PE, Bontempi G. On the Use of Variable Complementarity for Feature Selection in Cancer Classification. In: Rothlauf F. et al. (eds) Applications of Evolutionary Computing. EvoWorkshops 2006. Lecture Notes in Computer Science, vol 3907. Springer, Berlin, Heidelberg. 2006.

[ref30] 30. Brown G, Pocock A, Zhao MJ, Luján M. Conditional likelihood maximisation: a unifying framework for information theoretic feature selection, J Mach Learn Res. 2012; 13: 27–66.
View Article
Google Scholar

[87] View Article

[88] Google Scholar

[ref31] 31. Vinh NX, Zhou S, Chan J, Bailey J. Can high-order dependencies improve mutual information based feature selection? Pattern Recognit. 2016; 53: 46–58.
View Article
Google Scholar

[90] View Article

[91] Google Scholar

[ref32] 32. Hlaváčková-Schindler K, Paluš M, Vejmelka M. Causality detection based on information-theoretic approaches in time series analysis. Phys Rep. 2007; 441: 1–46.
View Article
Google Scholar

[93] View Article

[94] Google Scholar

[ref33] 33. Kraskov A, Stögbauer H, Grassberger P. Estimation mutual information, Phys Rev E. 2004; 69: 066138.
View Article
Google Scholar

[96] View Article

[97] Google Scholar

[ref34] 34. Winterhalder M, Schelter B, Hesse W, Schwab K, Leistritz L, Klan D, et al. Comparison of linear signal processing techniques to infer directed interactions in multivariate neural systems. Signal Process. 2005; 85: 2137–2160.
View Article
Google Scholar

[99] View Article

[100] Google Scholar

[ref35] 35. Marinazzo D, Pellicoro M, Stramaglia S. Kernel Method for Nonlinear Granger Causality. Phys. Rev. Lett. 100, 144103. pmid:18518037
View Article
PubMed/NCBI
Google Scholar

[102] View Article

[103] PubMed/NCBI

[104] Google Scholar

[ref36] 36. Stramaglia S, Cortes JM, Marinazzo D. Synergy and redundancy in the Granger causal analysis of dynamical networks. New J. Phys. 2014, 16, 105003.
View Article
Google Scholar

[106] View Article

[107] Google Scholar

[ref37] 37. http://math.bu.edu/people/kolaczyk/datasets.html .

[ref38] 38. Kramer MA, Kolaczyk ED, Kirsch HE. Emergent network topology at seizure onset in humans. Epilepsy Res. 2008; 79: 173–186. pmid:18359200
View Article
PubMed/NCBI
Google Scholar

[110] View Article

[111] PubMed/NCBI

[112] Google Scholar

[ref39] 39. Faes L, Marinazzo D, Stramaglia S. Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes. Entropy 2017, 19, 408.
View Article
Google Scholar

[114] View Article

[115] Google Scholar

[ref40] 40. Porta A, Faes L, Bari V, Marchi A, Bassani T, et al. Effect of Age on Complexity and Causality of the Cardiovascular Control: Comparison between Model-Based and Model-Free Approaches. PLoS ONE. 2014; 9: e89463. pmid:24586796
View Article
PubMed/NCBI
Google Scholar

[117] View Article

[118] PubMed/NCBI

[119] Google Scholar

[ref41] 41. Porta A, Faes L. Wiener-Granger causality in network physiology with applications to cardiovascular control and neuroscience. Proceedings of the IEEE. 2016; 104: 282–309.
View Article
Google Scholar

[121] View Article

[122] Google Scholar

[ref42] 42. Runge J, Donner R, Kurths J. Optimal model-free prediction from multivariate time series. Phys Rev E. 2015; 91: 052909.
View Article
Google Scholar

[124] View Article

[125] Google Scholar

[ref43] 43. Papana A, Kyrtsou C, Kugiumtzis D, Djks C. Simulation Study of Direct Causality Measures in Multivariate Time Series. Entropy 2013, 15, 2635–2661.
View Article
Google Scholar

[127] View Article

[128] Google Scholar

Figures

Abstract

Introduction

Materials and methods

Transfer entropy from non-uniform embedding

The motivation to use low-dimensional approximation for transfer entropy

Low-dimensional approximation strategy for CMI

Estimator of entropy

Simulation study

Model A

Model B

Model C

Model D

Model E

Model F

Model G

Application

Discussion and conclusion

Supporting information

S1 File. Matlab code for LA method by BIN-NN estimator.

References