Fig 1.
Information transmission between discrete inputs and response trajectories in biochemical networks.
For fully-observed reaction networks whose dynamics are governed by a known chemical Master equation, information can be approximated to an arbitrary accuracy via Monte Carlo integration for either continuous-time or discrete-time response trajectories (model-based exact Monte Carlo, Section Exact information calculations for fully observed reaction networks). Since full knowledge of the reaction system is used, these approximations are tractable deep in the regimes where model-free estimations break down with uncontrolled errors (Section Model–free information estimators). True information estimates are lower-bounded by model-based maximum a posteriori (MAP) or Bayes optimal decoding (Section Decoding–based information bounds). This decoding gives the lowest average probability of error and the corresponding information lower bound can be used as a benchmark for information estimates derived from other model-free decoding approaches (that have at least the error probability of the MAP decoder); in Section Decoding–based information estimators we compare Support Vector Machine (SVM), Gaussian Decoding (GD) and Neural Network (NN) decoding approaches. Upper bounds like the Feder-Merhav bound [34] and our improvement on it [35] complete the picture by estimating the gap between optimal decoding-derived and exact information values (Section Decoding–based information bounds).
Fig 2.
Example biochemical reaction networks and their behavior.
Three example birth-death processes, specified by the reactions in the top row for each of the two possible inputs (u(1) in blue, u(2) in red), stylize simple behaviors of biochemical signaling networks. (A) Input is encoded in both the transient approach to steady state and the steady state value. (B) Input is encoded in the magnitude of the transient response which is subsequently adapted away. (C) Input is encoded only at the level of temporal correlations of the response trajectory. Bottom row shows example trajectories generated using the Stochastic Simulation Algorithm for the copy number of molecules, t ∈ [0, 2000], for each network and the two possible inputs (light blue, light red); while plotted as a connected line for clarity, each trajectory represents molecular counts and is thus a step-wise function taking on only integer or zero values. Dark blue, red lines show the conditional means over N = 1000 trajectory realizations.
Fig 3.
Information about inputs encoded by complete response trajectories of the example biochemical reaction networks.
Exact Monte Carlo approximation for the information, , is shown for Example 1 (A), Example 2 (B), and Example 3 (C) from Fig 2 in dashed dark gray line; error bars are standard deviations across 20 replicate estimations, each computed over N = 1000 independently generated sample trajectories per input condition. Information is plotted as a function of the trajectory duration, T; yellow vertical line indicates T = 2000 as a representative duration used in further analyses below, at which most of the information about input is in principle available from the response trajectories of our systems.
(dashed black line) is the optimal decoding lower bound, and
(dashed light gray) is the upper bound on the information, computed by applying Eq (29).
Fig 4.
Information loss due to temporal sampling.
(A) Schematic representation of the resampling of a continuous-time response trajectory (left) at d = 14 (middle) or d = 41 (right) equally spaced time points. Resampled response trajectories are represented as d-dimensional real vectors, , for the case of a single output chemical species. (B–D) Exact Monte Carlo information approximations for discrete trajectories, Iexact(X; U) (dark solid gray), optimal decoding lower bound,
(dark solid black), and the upper bound, IUB (light solid gray) are plotted as a function of d. Continuous-time limits from Fig 3 are shown as horizontal lines:
(dashed dark gray),
(dashed black). Error bars as in Fig 3.
Fig 5.
Performance of decoding-based estimators depends on the dimensionality of the response trajectories and on the number of response trajectory samples.
Performance of various model-free decoding estimators (colored lines) for Examples 1 (A, D), 2 (B, E), 3 (C, F), respectively, compared to the MAP bound, IMAP (black line), as a function of input trajectory dimension, d (at fixed N = 1000) in A, B, C; or as a function of the number of samples, N, per input condition (at fixed d = 100) in D, E, F. Error bars are std over 20 replicate estimations. Decoding estimators: linear SVM, ISVM(lin) (orange); radial basis functions SVM, ISVM(rbf) (blue); the Gaussian decoder with diagonal regularization (see S2 Fig for the effects of covariance matrix regularization and signal filtering on Gaussian decoder estimates), IGD (yellow); multi-layer perceptron neural network, INN (green). Dashed vertical orange line marks the d ≤ 100 regime typical of current experiments. Note that while the amount of information must in principle increase monotonically with d, the amount that decoders can actually extract given a limited number of samples, N, has no such guarantee. The decrease, at high d, in Gaussian decoder information estimate in A, B, C and neural network information estimate in C, happens because the number of parameters of the decoder grows with d albeit at fixed number of samples, leading to overfitting that regularization cannot fully compensate, and thus to the consequent loss of performance on the test data.
Fig 6.
Information estimation for multilevel inputs.
(A) Extension of Example 2 from Fig 2B to q = 2, …, 5 discrete inputs. We chose the inputs such that the response for the system at T < 1000 converges towards q equally spaced levels with the same dynamic range as the original example; dynamics at T ≥ 1000 remain unchanged from the original Example 2. (B) Model-based information bounds as a function of the number of input levels for trajectories represented as d = 100 dimensional vectors: exact Monte Carlo calculation (dark gray), MAP decoding bound (black), upper bound (light gray). (C) Performance of model-free estimators, as indicated in the panel, compared to the MAP bound (black). Dashed lines show estimations using N = 103 sample trajectories per condition, solid lines using N = 104 samples per condition; in both cases, we show an average over 20 independent replicates, error bars are suppressed for readability.
Fig 7.
Comparison of decoding-based and knn information estimators.
Information estimates for decoding-based (color bars) and knn (gray bar) algorithms (here we set k = 1, for further details on knn estimation, including varying k, see S5 Fig). Note that knn is not a decoding estimator and thus could exceed (shown as a horizontal black line for each of the three example cases) to approach the exact Iexact(X; U). Here we use trajectories discretized over d = 100 time bins, and N = 104 trajectory samples per input. The performance of knn can be substantially improved by adding a small amount of gaussian noise to the trajectory samples; its resulting performance as a function of N and d is shown in S5 Fig. Red star denotes the failure of knn on Example 3 where substantially negative information values are returned (exact value not plotted).
Fig 8.
Two-level mutual information estimates from single-cell time-series data for nuclear translocation of yeast transcription factors.
(A, B) Data replotted from Ref [27] for Msn2 (top row), Dot6 (middle row), and Sfp1 (bottom row); early transient responses (A) after nutrient shift at t = 0 min from glucose rich (2%, blue traces) to glucose poor (0.1%, red traces) medium are shown in the left column, stationary responses (B) are collected after cells are fully adapted to the new medium. Sampling frequency is 2.5 min, d = 45, and the number of sample trajectories per nutrient condition is N = 100. Thin lines are individual single cell traces, solid lines are population averages. (C, D) Information estimates for the transient (left, C) and stationary (right, D) response periods. Colored bars use model-free decoding-based estimators as indicated in the legend, gray bar is the knn estimate; error bars computed from estimation bootstraps by randomly splitting the data into testing and training sets.
Fig 9.
Multilevel mutual information estimates from single-cell time-series data for mammalian intracellular signaling.
Data replotted from Ref [26] for ERK (top row) and Ca2+ (bottom row). (A) Early transient responses after addition of 5 different levels of EGF for ERK (or 4 different levels of ATP for Ca2+, respectively) at t = 0 min, as indicated in the legend. (B) In the late response most, but not all, of the transients have decayed. (C,D) Information estimation using different methods (legend) in the early (C) and late (D) period, for ERK (left half of the panels) and Ca2+ (right half). Data for ERK: N = 1678 per condition, T = 30 min (d = 30) for early response and T = 30 min (d = 30) for late response. Data for Ca2+: N = 2995 per condition, T = 10 min (d = 200) for early response and T = 5 min (d = 100) for late response. Plotting conventions as in Fig 8.