Figure 1.
Pooling of data over an ensemble of time series for transfer entropy (TE) estimation.
(A) Schematic account of TE. Two scalar time series and
recorded from the
repetition of processes
and
, coupled with a delay
(indicated by green arrow). Colored boxes indicate delay embedded states
,
for both time series with dimension
samples (colored dots). The star on the
time series indicates the scalar observation
that is obtained at the target time of information transfer
. The red arrow indicates self-information-transfer from the past of the target process to the random variable
at the target time.
is chosen such that
and influences of the state
arrive exactly at the information target variable
. Information in the past state of
is useful to predict the future value of
and we obtain nonzero TE. (B) To estimate probability density functions for
,
and
at a certain point in time
, we collect their realizations from observed repetitions
. (C) Realizations for a single repetition are concatenated into one embedding vector and (D) combined into one ensemble state space. Note, that data are pooled over the ensemble of data instead of time. Nearest neighbor counts within the ensemble state space can then be used to derive TE using the Kraskov-estimator proposed in [57].
Figure 2.
Transfer entropy estimation using the ensemble method in TRENTOOL 3.0.
(A) Data preparation and optimization of embedding parameters in function TEprepare.m; (B) transfer entropy (TE) estimation from prepared data in TEsurrogatestats_ensemble.m (yellow boxes indicate variables being passed between sub-functions). TE is estimated via iterating over all channel combinations provided in the data. For each channel combination: (1) Data is embedded individually per repetition and combined over repetitions into one ensemble state space (chunk), (2) surrogate data sets are created by shuffling the repetitions of the target time series, (3) each surrogate data set is embedded per repetition and combined into one chunk (forming
chunks in total), (4)
chunks of original and surrogate data are passed to the GPU where nearest neighbor searches are conducted in parallel, (5) calculation of TE values from returned neighbor counts for original data and
surrogate data sets using the KSG-estimator [57], (6) statistical testing of original TE value against distribution of surrogate TE values; (C) output of TEsurrogatestats_ensemble.m, an array with dimension [no. channels
5], where rows hold results for all channel combinations: (1) p-value of TE for this channel combination, (2) significance at the designated alpha level (1 - significant, 0 - not significant), (3) significance after correction for multiple comparisons, (4) absolute difference between the TE value for original data and the median of surrogate TE values, (5) presence of volume conduction (this is always set to 0 when using the ensemble method as instantaneous mixing is by default controlled for by conditioning on the current state of the source time series
[119]).
Figure 3.
Creation of surrogate data sets.
(A) Original time series with information transfer (solid arrow) from a source state to a corresponding target time point
, given the time point's history
. Solid arrows indicate the direction of transfer entropy (TE) analysis, while information transfer is present. (B) Shuffled target time series, repetitions are permutes, such that
and
, where
denotes a random permutation. Dashed arrows indicate the direction of TE analysis, while no more information flow is present.
Figure 4.
GPU implementation of the parallelized nearest neighbor search in TRENTOOL 3.0
Chunks of data are prepared on the CPU (embedding and concatenation) and passed to the GPU. Data points are managed in the global memory as Structures of Arrays (SoA). To make maximum use of the memory bandwidth, data is padded to ensure coalesced reading and writing from and to the streaming multiprocessor (SM) units. Each SM handles one chunk in one thread block (dashed box). One block conducts brute force neighbor searches for all data points in the chunk and collects results in its shared memory (red and blue arrows and shaded areas). Results are eventually returned to the CPU.
Figure 5.
Practical performance measures of the ensemble method for GPU compared to CPU.
Combined execution times in s for serial and parallel implementations of k-nearest neighbor and range search as a function of input size (number of data chunks). Execution times were measured for the serial implementation running on a CPU (black) and for our parallel implementation using one of three GPU devices (blue, red, green) of varying computing power. Computation using a GPU was considerably faster than using a CPU (by factors 22, 33 and 50 respectively).
Figure 6.
Transfer entropy reconstruction from non-stationary Lorenz systems.
We used two dynamically coupled Lorenz systems (A) to simulate non-stationarity in data generating processes. A coupling was present during a time interval from 1000 to 2000 ms only (
otherwise). The information transfer delay was set to
. Transfer entropy (TE) values were reconstructed using the ensemble method combined with the scanning approach proposed in [53] to reconstruct information transfer delays. Assumed delays
were scanned from 35 to 55 ms (1 ms resolution). In (B) the maximum TE values for original data over this interval are shown in blue. Red bars indicate the corresponding mean over surrogate TE values (error bars indicate 1 SD). Significant TE was found for the second time window only; here, the delay was reconstructed as
.
Table 1.
Parameter settings for simulated autoregressive processes.
Figure 7.
Transfer entropy reconstruction from coupled autoregressive processes.
We simulated two dynamically coupled autoregressive processes (A) with coupling delays and
, and coupling scenarios: (B) unidirectional coupling
(blue line) with onset around 1 s, coupling
set to 0 (red line); (C) unidirectional coupling
(blue line) with onset around 1 s and an increase in coupling strength at around 2 s, coupling
set to 0 (red line); (D) bidirectional coupling
(blue line) with onset around 1 s and
(red line) with onset around 2 s. (E-G) Time-resolved transfer entropy (TE) for both directions of interaction, blue and red lines indicate raw TE values for
and
respectively. Dashed lines denote significance thresholds at 0.01% (corrected for multiple comparisons over signal combinations). Shaded areas (red and blue) indicate the maximum absolute TE values for significant information transfer (indicated by asterisks in red and blue). (E) TE values for unidirectional coupling; (F) unidirectional coupling with a two-step increase in coupling strength; (G) bidirectional coupling.
Figure 8.
Robustness of transfer entropy estimation with respect to limited amounts of data.
Estimated transfer entropy (TE) values for estimations using varying numbers of data points (color coded) as a function of
. Data was sampled from two Lorenz systems
and
with coupling
. The simulated information transfer delay
is indicated by a vertical dotted line. Sampled data was embedded and varying numbers of embedded data points (500, 2000, 5000, 10000, 30000) were used for TE estimation. For each estimation, the maximum
values for all values of
are indicated by solid dots. Dashed lines indicate significance thresholds (
).
Figure 9.
Transfer entropy reconstruction from electrophysiological data.
Time resolved reconstruction of transfer entropy (TE) from magnetoencephalographic (MEG) source data, recorded during a face recognition task. (A) Face stimulus [88]. (B) Cortical sources after beamforming of MEG data (L, left; R, right: L orbitofrontal cortex (OFC); R middle frontal gyrus (MiFG); L inferior frontal gyrus (IFG left); R inferior frontal gyrus (IFG right); L anterior inferotemporal cortex (aTL left); L cingulate gyrus (cing); R premotor cortex (premotor); R superior temporal gyrus (STG); R anterior inferotemporal cortex (aTL right); L fusiform gyrus (FFA); L angular/supramarginal gyrus (SMG); R superior parietal lobule/precuneus (SPL); L caudal ITG/LOC (cITG); R primary visual cortex (V1)). (C) Reconstructed TE in three single subjects (red box) in three time windows (0−150 ms, 150−300 ms, 300−450 ms). Each link (red arrows) corresponds to significant TE on single subject level (corrected for multiple comparisons). (D) Thresholded TE links over 15 subjects (blue box) in three time windows (0−150 ms, 150−300 ms, 300−450 ms). Each link (black arrows) corresponds to significant TE in eight and more individual subjects (, after correction for multiple comparisons). Blue arrows indicate differences between time windows, i.e. links that occur for the first time in the respective window. (E) Experimental design: stimulus was presented for 200 ms (gray shading), during the inter stimulus interval (ISI, 1800 ms) a fixation cross was displayed.
Table 2.
Reconstructed information transfer delays for magnetoencephalographic data.