Mutual Information Rate and Bounds for It

The amount of information exchanged per unit of time between two nodes in a dynamical network or between two data sets is a powerful concept for analysing complex systems. This quantity, known as the mutual information rate (MIR), is calculated from the mutual information, which is rigorously defined only for random systems. Moreover, the definition of mutual information is based on probabilities of significant events. This work offers a simple alternative way to calculate the MIR in dynamical (deterministic) networks or between two time series (not fully deterministic), and to calculate its upper and lower bounds without having to calculate probabilities, but rather in terms of well known and well defined quantities in dynamical systems. As possible applications of our bounds, we study the relationship between synchronisation and the exchange of information in a system of two coupled maps and in experimental networks of coupled oscillators.


A. Decay of correlation and First Poincaré Returns
As rigorously shown in [1], the decay with time of the correlation, C(t), is proportional to the decay with time of the density of the first Poincaré recurrences, ρ(t, ), which measures the probability with which a trajectory returns to an -interval after t iterations. Therefore, if ρ(t, ) decays with t, for example exponentially fast, C(t) will decay with t exponentially fast, as well. The relationship between C(t) and ρ(t) can be simply understood in chaotic systems with one expanding direction (one positive Lyapunov exponent). As shown in [2], the "local" decay of correlation (measured in the -interval) is given by C(t, ) ≤ µ( )ρ(t, ) − µ( ) 2 , where µ( ) is the probability measure of a chaotic trajectory to visit the -interval. Consider the shift map x n+1 = 2x n , mod 1. For this map, µ( ) = and there are an infinite number of possible intervals that makes C(t, ) = 0, for a finite t. These intervals are the cells of a Markov partition. As recently demonstrated by [P. Pinto, I. Labouriau, M. S. Baptista], in piecewise-linear systems as the shift map, if is a cell in an order-t Markov partition and ρ(t, ) > 0, then ρ(t, ) = 2 −t and by the way a Markov partition is constructed we have that = 2 −t . Since that = µ( ) = 2 −t , we arrive at that C(t, ) ≤ 0, for a special finite time t. Notice that = 2 −t can be rewritten as − ln ( ) = t ln (2). Since for this map, the largest Lyapunov exponent is equal to λ 1 = ln (2), then t = − 1 λ1 ln ( ), which is exactly equal to the quantity T , the time interval responsible to make the system to lose its memory from the initial condition and that can be calculated by the time that makes points inside an initial -interval to spread over the whole phase space, in this case [0, 1].
B. IC , and I l C in larger networks and higher-dimensional subspaces ΣΩ Imagine a network formed by K coupled oscillators. Uncoupled, each oscillator possesses a certain amount of positive Lyapunov exponents, one zero, and the others are negative. Each oscillator has dimension d. Assume that the only information available from the network are two Q dimensional measurements, or a scalar signal that is reconstructed to a Q-dimensional embedding space. So, the subspace Σ Ω has dimension 2Q, and each subspace of a node (or group of nodes) has dimension Q. To be consistent with our previous equations, we assume that we measure M Ω = 2Q positive Lyapunov exponents on the projection Σ Ω . If M Ω = 2Q, then in the following equations 2Q should be replaced by M Ω , naturally assuming that M Ω ≤ 2Q.
In analogy with the derivation of I C and I l C in a bidimensional projection, we assume that if the spreading of initial conditions is uniform in the subspace Ω. Then, P X (i) = 1 N Q represents the probability of finding trajectory points in Q-dimensional space of one node (or a group of nodes) and P XY (i, j) = 1 N C represents the probabilities of finding trajectory points in the 2Q-dimensional composed subspace constructed by two nodes (or two groups of nodes) in the subspace Ω. Additionally, we consider that the hypothetical number of occupied boxes N C will be given by N C (T ) = exp T ( 2Q i=1 λi) . Then, we have that T = 1/λ 1 log (N ), which lead us to Similarly to the way we have derived I l C in a bidimensional projection, if Σ Ω has more than 2 positive Lyapunov exponents, then To write Eq. (1) in terms of the positive Lyapunov exponents, we first extend the calculation of the quantity D to higher-dimensional subspaces that have dimensionality 2Q, where λ 1 ≥ λ 2 ≥ λ 3 . . . ≥ λ 2Q are the Lyapunov exponents measured on the subspace Ω. To derive this equation we only consider that the hypothetical number of occupied boxes N C is given by N C (T ) = exp T ( 2Q i=2 λi) . We then substitute D as a function of these exponents (Eq. (3)) in Eq. (1). We arrive at C. IC as a function of the positive Lyapunov exponents of the network Consider a network whose attractor Σ possesses M positive Lyapunov exponents, denoted byλ i , i = 1, . . . , M , and that can be calculated either directly or via the reconstructed attractor. For a typical subspace Ω, λ 1 measured on Ω is equal to the largest Lyapunov exponent of the network. Just for the sake of simplicity, assume that the nodes in the network are sufficiently well connected so that in a typical measurement with a finite number of observations this property holds, i.e.,λ 1 = λ 1 . But, if measurements provide thatλ 1 >> λ 1 , the next arguments apply as well, if one replacesλ 1 appearing in the further calculations by the smallest Lyapunov exponent, say,λ k , of the network that is still larger than λ 1 , and then, substituteλ 2 byλ k+1 , and so on. As before, consider that M Ω = 2Q.
Then, for an arbitrary subspace Ω, i=2λ i , since a projection cannot make the Lyapunov exponents larger, but only smaller or equal.
i=2λ i , it is easy to see that So, I C , measured on the subspace Σ Ω and a function of the 2Q largest positive Lyapunov exponents measured in Σ Ω , is an upper bound forĨ C , a quantity defined by the 2Q largest positive Lyapunov exponents of the attractor Σ of the network. Therefore, if the Lyapunov exponents of a network are know, the quantityĨ C can be used as a way to estimate how much is the MIR between two measurements of this network, measurements that form the subspace Ω.
Notice that I C depends on the projection chosen (the subspace Ω) and on its dimension, whereasĨ C depends on the dimension of the subspace Σ Ω (the number 2Q of positive Lyapunov exponents). The same happens for the mutual information between random variables that depend on the projection considered.
The Lyapunov exponents of the network, and conse-quentlyĨ C , can be obtained from the reconstructed attractor of a single time-series measure. Hence, the reconstructed attractor can be used to estimate I C .
Equation (5) is important because it allows us to obtain an estimation for the value of I C analytically. As an example, imagine the following network of coupled maps with a constant Jacobian where X ∈ [0, 1] and A represents the connecting adjacent matrix. If node j connects to node i, then A ij = 1, and 0 otherwise. Assume that the nodes are connected all-to-all. Then, the K positive Lyapunov exponents of this network are: λ 1 = log (2) andλ i = log 2[1 + σ], with i = 2, K. Assume also that the subspace Ω has dimension 2Q and that 2Q positive Lyapunov exponents are observed in this space and thatλ 1 = λ 1 . Substituting these Lyapunov exponents in Eq. (5), we arrive at We conclude that there are two ways forĨ C to increase. Either one considers larger measurable subspaces Ω or one increases the coupling between the nodes. This suggests that the larger the coupling strength is the more information is exchanged between groups of nodes.
For arbitrary topologies, one can also derive analytical formulas forĨ C in this network, sinceλ i for i > 2 can be calculated fromλ 2 [3]. One arrives at λ i (ω i σ/2) =λ 2 (σ), (9) where ω i is the ith largest eigenvalue (in absolute value) of the Laplacian matrix L ij = A ij + I j A ij .