Constructing adjoint process spaces

Decidability issues may forever limit our ability to design algorithms for reconstructing optimal adjoint representations from data that are not context specific. On the other hand, for any finite data over finite sample spaces and finite relevant set of parsing rule templates one can in principle find the optimal proxy to an adjoint representation by extensive search algorithms, even though time issues forbid such extensive searches in practical implementations. The general procedure underlying the construction of adjoint representations always remains the same.

Suppose X is a process that emits symbols i drawn from the alphabet , where W is the number of symbols. X generates data streams, x(t) = x_tx_t−1⋯x₁, where every x_t is one of the available symbols in , which contains W elements. Consider now two letters r₁ and s₁ such that the pair r₁s₁ ≡ x_τx_τ−1 for some positions τ in the data x, has been identified to contain relevant information (e.g. because the pair is over-expressed), then we can rewrite the pair r₁s₁ by a new letter m₁ = W + 1, which will become the first letter extending alphabet to , with the parsing rule π₁ = [r₁ s₁ → W + 1].

We can iterate and produce parsing rules π_n = [r_n s_n → W + n], with letter indices r_n < W + n and s_n < W + n. Where π_n maps data over the alphabet to data over . Note that the parsing rules π_n can be uniquely inverted, i.e. we can expand data over to data over using the inverse map . In other words the inverse parsing rules can be thought of being part of a “generative grammar”, [23]. We therefore can construct a sequence of maps π(n) = π_nπ_n−1⋯π₁ such that data x can be mapped to representations y_n = π(n)x. At every parsing level, n, we get a corresponding distribution function of the re-coded data, p_z(y_n), with a letter index z = 1, 2, ⋯, W + n.

The Kraft and McMillan theorems [28, 29] tells us that if all that we know about a process are its marginal relative frequencies, g_i, at which symbols i occur, there exists a shortest reversibly encoding of the data, x, of characteristic length, , the minimal description length (MDL) that gives the theoretically achievable minimal length of x (in units of bits). is a lower bound for the true MDL, L(x), that can only be attained asymptotically. The theorems state that (3) with k = 1/log(b), where b is the basis in which information is measured. For bits we typically have b = 2. kH(g) is the MDL in bits per symbol and is the minimal number of bits required to encode messages of length t.

For data x(t) of length t we find a sequence of representations y_n(t) = π(n)x(t). Suppose now that for every t we find a parsing level n*(t) such that y_n*(t)(t) is a representation of the data x(t) that cannot be distinguished from an i.i.d. process (which we indicate here by a *). It follows that y_n*(t)(t) is entirely determined by its marginal distribution of letters p_z(y_n*(t)(t)) and obtain asymptotically , where ≃ means asymptotically identical. With |.| we indicate the length of the sequence of letters in numbers of letters of the underlying alphabet. For instance we have |x(t)| = t and |y_n+1(t)| ≤ |y_n(t)|. Then, we can asymptotically measure the information production of the process X to be (4)

As discussed above, the “complexity” of a symbol ℓ(z) is the number of letters in the original alphabet it codes for. As a consequence we can compute the average symbol complexity for data y_n*(t)(t) to be given by . As a consequence, we get for the adjoint process Y = lim_{t → ∞}Y_n*(t) that . The adequate unit of information, k_Y, for measuring information production, k_YH(p(Y)), therefore is given by (5)

For simplicity, suppose there is a maximal n* that holds for all t. Since we assumed that y_n* is already indistinguishable from an i.i.d. process, applying another parsing rule would only compress the data without changing its MDL. This means, . If, conversely, we look at a parsing level, n, where the adjoint process is not yet i.i.d., then we can find a parsing rule, π_n+1, such that . Additional knowledge always reduces the attainable information production rate.

In principle, for any finite amount of data x(t) one can construct the optimal map, π(n), for the process, X, by minimizing over all possible sequences of parsing rules, at any fixed parsing level n, if only we know the relevant set of parsing rules to consider and this set is finite. Then we can in principle find n = n*(t) such that no further reduction of the minimal description length is possible by applying any more parsing rules. In practice, an extensive search over all possible sequences of parsing rules is of course not feasible, even if the set of parsing rule templates only consists of the elementary template, and algorithms for inferring adjoint representations of data need to turn to different means of optimization. For theoretical considerations we may, however, assume that for a given finite relevant set of parsing rule templates and any finite t we can find the optimal map π (or one of several possible optimal maps if the map is not unique) or at least a map reasonably close to optimal, since the number of possible maps we would have to evaluate remains finite, too. Intuitively it is clear however, given an unknown process X for which we cannot pre determine the respective relevant set of parsing rule templates, that one typically can no longer decide whether a map π is optimal or not.

However, given an optimal π, the adjoint i.i.d. process, Y, is fully characterized by its marginal distribution, f = p(Y), over symbols in the adjoint alphabet, , and the information production, I(X) = k_YH(f), is given by the Shannon entropy of Y. On the adjoint process space Φ* we can use the measures of Shannon entropy, cross-entropy, and Kullback-Leibler information divergence, given that we use the appropriate unit of information, k_Y of Eq (5). Since Y is i.i.d. over , the adjoint space naturally belongs to the family, Φ*, of all i.i.d. processes over this alphabet. Further, any Y′ in Φ* is fully characterized by its marginal distribution, f′, and the pair (f′, π), determines the process X′ = π⁻¹Y′. Hence, the process class, Φ_π, that naturally generalizes a process, X, with adjoint i.i.d. process, Y = πX, is given by Φ_π = π⁻¹Φ*.

This construction completes the first part of the proof that establishes that we can essentially measure information production of a process as the Shannon entropy of the adjoint process. This entropy, however uses the marginal distributions over the adjoint alphabet as arguments and can therefore not be identified directly with the generalised entropies that use the marginal distributions over the original alphabet as arguments. In the next step we will pull the information measures over the adjoint message space back to to the original message space.

Information measures over extended alphabets

Suppose we have a process X with an adjoint process Y = πX. For data, x, and its adjoint sequence, y = y_n, we obtain two histograms, h_i(x), of symbols and, h_z(y), of symbols , respectively. The associated relative frequency distributions are given by g = p(x) = h(x)/|x| and f = p(y) = h(y)/|y|. Further, every symbol, z, represents a number of ℓ(z) symbols, π⁻¹z, in the original alphabet with π = π(n). We define as the histogram of letters that are parsed together into the symbol . For z ≤ W where , we have, and needs to hold for all . This provides us with constraints, (6) that we need in the next section. As a consequence, we have (7)

We drop the arguments (x) and (y) (or (y_n)) from now on and distinguish histograms by their index i (over ) and z (over the adjoint alphabet ). Let (note that in this notation we identify 〈ℓ〉_f ≡ 〈ℓ〉_Y) and be the expectation values under the distribution f, then, by construction, , , and . This means that we can write the constraints that link distributions g over , Eq (6), with distributions f over in the following way (8)

As mentioned before, the process class, Φ_π, that X belongs to, is completely determined by the map π and the process X, by the pair (f, π), see Fig 1. Therefore, we can identify the entropy of X with (9) with the process-specific Boltzmann factor, k_Y ≡ k/(〈ℓ〉_f). For processes, X and X′, and with f′ = p(πX′), the cross-entropy and the information divergence are (10)

In the special case where X is already an i.i.d. process, no features can be extracted from the data and n = 0, π = π(0) = id, and ℓ(z) = 1, for all z = 1⋯W. Consequently, S_π = kH, , and (Kullback-Leibler divergence), as required.

Pulling back entropies to the original alphabet

In the next step one can construct entropy functionals over the original alphabet, , by lifting a distribution function, g′, on to a distribution function, f′, over by assuming that f = p(πX) is the true distribution function of the process, Y = πX. We proceed by minimizing the information divergence, D_π(f′||f), with respect to f′. More precisely, we minimize the functional ψ(f′, α, η) given by (11) with Lagrange multipliers, α and η_i, that normalize f′ and guarantee the constraints from Eq (8). Solving the variational principle δψ = 0 estimates f′ at the minimum that is compatible with g′. We identify this minimizer as and obtain α = 1/〈ℓ〉_f and (12) which has to be solved self consistently. If f already meets all matching constraints with g′, i.e. if g′ = g with g = p(X), then we have and .

This means that one can lift marginal distributions, g′, on to distributions, , on with respect to a particular process, X. As a consequence one can pull the entropy, cross-entropy, and divergence back from distributions, f, over the adjoint sample space, , to distributions, g, over the initial alphabet, , with respect to a particular process, X ∈ Φ_π. In particular, one can define the projection operator, π_*, through g = p(π⁻¹y) ≡ π_*p(y) = π_*f and the operator, , that lifts distributions, g, over alphabet, , to distributions f over the extended alphabet, , through (13) with respect to the process X, i.e., with respect to the distribution function, p(πX), of the i.i.d. process, πX, over the adjoint alphabet. The lift operator gives us the minimizer . We find that and identify (14) where, typically, g = p(X). We call those measures the pull-back entropy, cross-entropy, and information divergence of the process X. Note, that while S_π, , and D_π are universal on the entire class of processes Φ_π, pulling the information measures back to marginal distributions g over yields information measures that are specific to a particular process X ∈ Φ_π.

Generalised entropies over initial alphabets

The final question is how the pull-back measures S_X, , and D_X, defined in Eq (14), are related to generalised entropy functionals as derived for example in [6, 21]. There, functionals were derived to obtain the most likely histogram, h, observed in a given process after t observations (for large t). Even for non i.i.d. processes, often the probability, P(h|θ), to observe the particular histogram, h, for t = ∑_i∈Ω h_i observations factorizes P(h|θ) = M(h)G(h|θ) into a multiplicity, M(h), and a probability term of the sequences, G(h|θ). θ is a set of parameters that determines the process, X—it defines and parametrizes the process class X belongs to. Whenever such a factorization is possible, one can show that a generalised maximum entropy principle exists. Using the Boltzmann definition of entropy, the logarithm of multiplicity, , and defining , and a generalised information divergence as , where g = h/t, the standard relations remain valid [5].

If we are looking at a family of processes X(θ) rather than a single process X then we can no longer assume a priorly that the same map π⁽¹⁾ that takes some process X(θ⁽¹⁾) to an adjoint representation efficiently is the same map π⁽²⁾ that takes some process X(θ⁽²⁾) from the family efficiently to an adjoint representation. However, there are ways in which we can think of constructing a single map π to some extended alphabet such that all processes X(θ) decorrelate under the same map π. In the weakest version we can start with some extended alphabet and the space of all i.i.d. processes Φ* over that alphabet and a map π and consider the process family Φ_π = π⁻¹Φ*, in which case the natural parameters of the processes X(θ) are given by θ = f, where f are distribution functions over . For a discussion of stronger versions see Text 6 in S1 File.

However, if above assumption is valid, then X(θ) forms a sub-class of processes in Φ_π with distribution functions, g(θ) = p(X(θ)), and adjoint distribution functions, f(θ) = p(πX(θ)). similarly we have , y′ = πx′ and . And in the next step we note that the probability of observing a histogram h′ under process X(θ) is given by (15) with being the respective maximizing argument h*, the histogram over the extended space. The expression are the constraints between the histograms on the original and adjoint alphabet. Since y are sampled i.i.d, in the last line (16) is the multinomial distribution of histograms h*. Also note that in the one but last line of Eq (15) ≃ stands for asymptotically equivalent, meaning, that the relative error we make in log P(h′|θ) by replacing P(h′|θ) with P(h*|f(θ)) vanishes in probability as t → ∞. For more details see Text 7 in S1 File. We note, that this asymptotic equivalence is a consequence of processes that can be mapped into an adjoint representation becoming i.i.d. there and therefore implicitly can be thought of being ergodic on sufficiently large time scales.

Taking logs and multiplying both sides with −k/t and setting g′ = h′/t and f* = k*/|k*|, we then obtain (17) where ≃ again means asymptotically identical for large t; from the second to the third line we used the definition of the lift operator from Eq (13) and D_X from Eq (14). In other words, we have shown that in the limit t → ∞ the generalised information divergence D(g′|θ) is identical to the pull-back divergence D_X(θ) (g′|g(θ)) as a functional. As a consequence we can use that and identify S_X with the generalised entropy and with the generalised cross entropy. We see that the generalised entropy, , for the processes family, X(θ), is given by .

We conclude that for all process classes (at least those that that decorrelate over a common map π) there exist notions of entropy, S_X, cross-entropy, , and divergence, D_X, that behave in the usual way, namely, . This means that for such processes (at least asymptotically) the probability to observe a particular histogram, P ≃ exp(−tD_X), factorizes into a multiplicity term, M ≃ exp(tS_X), associated with entropy, and a sequence probability term, , associated with cross-entropy. In other words, we have shown that Boltzmann’s entropy, the logarithm of multiplicity, remains the correct estimator of information production. For complex systems the multiplicity will differ from a multinomial coefficient in arbitrarily complicated ways that might even depend explicitly on system parameters θ, which would mean a violation of SK1 axiom. In other words, the Boltzmann pull-back entropy functional typically will be more complicated than the Shannon entropy functional and can even violate SK1—yet they are the appropriate generalizations of entropy, in terms of information production. We also learned that beyond such parametric families of generalised entropies, i.e. beyond the pull-back measures, we find the standard notions of entropy, cross-entropy, and divergence present in the adjoint alphabet, where the only thing that is not universal about the information measures is the process-specific Boltzmann constant, k_Y, that needs to be used.

Example: SSR processes

We now demonstrate explicitly that the generalised entropy that –according to the previous section– is identified with the pull-back entropy, S_X, indeed does measure information production. We do that by considering slowly driven sample space reducing (SSR) processes, X, for which the generalised entropy functional is exactly known [21]. SSR processes are models of driven non-equilibrium systems. They are characterized by the fact that as the process unfolds the number of states accessible to the process reduces when no driving is present [30]. In its simplest form, the process relaxes to a ground state from which it has to be restarted. One can think of the process as a ball bouncing down a staircase with random jump sizes. The ball can only jump to steps lower than the last step it visited. Once it reaches the bottom of the staircase one lifts the ball to the top of the staircase (driving), and kicks it down the staircase again. The stairs represent (energy) states, the lowest being 1, the highest is W. The process exhibits path-dependence in the relaxation part, the current through the system breaks detailed balance between states. Such processes exhibit a Zipf law in their distribution function.

The micro-states, x, of the SSR process are sequences of states with elements x_n ∈ {1, 2, ⋯, W} ≡ Ω. The transition probabilities between states j → i are (18) where the first term on the right hand side describes the relaxing part of the SSR process (transitions only happen from higher i to lower states, j, i.e., when j < i) with prior distribution q_i and cumulative distribution, . Θ is the Heavyside function. The second term captures the (slow) driving of the process. Slow here means that the system is only driven once the SSR process reaches its lowest position i = 1. SSR processes are Markovian since transition probabilities depend only on the current position and it is ergodic since after the relaxation process the system is reset to any state with probability, q_i.

To understand the statistics of the process we are interested in the distribution of visits to the individual states. We define the macro-state to be the histogram, h_i, of visits of X to state, i. It is possible to compute , where the multiplicity, M(k), is the number of different sequences, x, of length t with the same histogram h. One finds [21] (19) where p_i = h_i/t are the relative frequencies of observing a state i. Note that this is the Boltzmann entropy of the system, yet it is not of Shannon form since it is derived from a Markov, and not an independent sampling process. Similarly, one finds the cross-entropy (20) and by maximizing , (negative information divergence D), one obtains the characteristic Zipf distribution of the slowly driven SSR process [30] (21)

For the special case of q_i = 1/W for all i, the Zipf distribution, , is obvious, since Q_i = i/W and q_i/Q_i = 1/i. It continues to hold for “well-behaved” q_i, [31]. For instance, if q_i ∝ i^α for α > −1, then Q_i ∝ i^1+α and again q_i/Q_i ∝ 1/i.

In the next step we will use a simple example of a slowly driven SSR process in order to demonstrate how using extended alphabets works and how the respective generalised entropy functional is the adequate measure of information production.