Network model

The network contains two fundamental circuit motifs (Fig. 1A): excitatory neurons that project both onto themselves and others, and inhibitory neurons which receive excitatory input from the same neurons that they inhibit. Several instances of these motifs together compose the general recurrent circuit that we investigate. In its simplest form, this recurrent network has the following form. There are N neurons, N-1 of which are excitatory and one (index N) is inhibitory (Fig. 1B,C). The excitatory neurons x_{i ≠ N} receive an optional external input I_i, and excitatory feedback α from themself and nearby excitatory neurons (α₁ and α₂, respectively). The single inhibitory neuron x_n sums the β₂ weighted input from all the excitatory neurons, and sends a common β₁ inhibitory signal to each of its excitatory neurons. Each neuron has a resistive constant leak term G_i.

The dynamics of this simple network are: (1) (2) where f(x) is a non-saturating rectification non-linearity. Here, we take f(x) = max(x, 0), making our neurons linear threshold neurons (LTNs).

As long as the key parameters (α_i, β_i) satisfy rather broad constraints [23], this network behaves as a soft winner-take-all by allowing only a small number of active neurons to emerge, and that solution depends on the particular input pattern I(t). After convergence to the solution, the active neurons (winners) will express an amplified version of the input I(t), that is they remain sensitive to the input even if it changes.

After convergence to steady state in response to constant input I_i > 0, the activation of the winner i is: (3) where $g=\frac{1}{1-{\alpha }_1+{\beta }_1{\beta }_2}$ is the gain of the network. Importantly, this gain can be g > > 1 due to excitatory recurrence, which amplifies the signal in a manner controlled by the feedback loop. While above assumes α₂ = 0, similar arguments hold without this assumption [23].

The dynamics of the non-linear system are $\tau \dot{\mathbf{x}}=\mathbf{f}(\mathbf{Wx}+\mathbf{I}(\mathbf{t}\left)\right)-\mathbf{Gx}$ (where f applies the scalar function f(x) component-wise). For example, a minimal WTA with two excitatory and one inhibitory units, x = [x₁, x₂, x₃] (Fig. 1B,C), the weight matrix W is (4) where G = diag(G₁, …, G_n) is a diagonal matrix containing the dissipative leak terms for each unit. The time constant of the system is $\frac{\tau }{G}$. Now we consider the conditions under which the non-linear system is guaranteed to be stable but at the same time powerful, i.e. permitting high gain.

Instability drives computation

Contraction Theory assesses the stability of non-linear systems $\dot{x}=f(x,t)$ using virtual displacements of its state at any point with respect to a chosen uniformly positive definite metric M(x, t), obtained by a transformation Θ(x, t) (see Methods). The key Theorem 2 of [17] asserts that if the temporal evolution of any virtual displacement in this metric space tends to zero, then all other displacement trajectories in this space will also contract (shrink) to the same (common) trajectory.

For our present case, this theorem implies (see Methods) that the trajectories of a system of neurons with Jacobian J are exponentially contracting if (5) The Jacobian J has dimension N and describes the connection matrix of the network and Θ is a transformation matrix that provides a suitable choice of coordinates.

For a WTA with weight matrix W, the Jacobian is (6) where Σ = diag(σ₁, …, σ_n) is a switching matrix, whose diagonal elements σ_i ∈ 0, 1 indicate whether unit i is currently active or not.

Following [16] we will call the subset of N neurons that are currently above threshold the active set. Those that are inactive cannot contribute to the dynamics of the network. Thus, we may distinguish between the static anatomical connectivity W of the network, and the dynamic functional connectivity that involves only the subset of currently active neurons. This active set is described by the switch matrix Σ. The Jacobian expresses which units contribute to the dynamics at any point of time, and is thus a reflection of functional rather than anatomical connectivity. Each possible effective weight matrix has a corresponding effective Jacobian. Thus, ΣW − G is the effective (functional) Jacobian of the network, in which only active units have an influence on the dynamics.

The active sets may be either stable or unstable. In previous work on the hybrid analog and digital nature of processing by symmetric networks, Hahnloser et al. refer to such sets as active and forbidden, respectively [16]. They further show that the eigenvectors associated with positive eigenvalues of forbidden sets are mixed. That is, the eigenvector contains at least one component whose sign is opposite to the remainder of the components. This property ensures that a neuron will finally fall beneath threshold, and that the composition of the active set must change. We now generalize this concept to our asymmetrical networks, and refer to permitted and forbidden subspaces rather than sets, to emphasize the changes in the space in which the computational dynamics play out.

It is the instability (exponential divergence of neighboring trajectories) of the forbidden subspaces rather than stability that drives the computational process. This instability can be approached also through Theorem 2 of [17], which notes that if the minimal eigenvalue of the symmetric part of the Jacobian is strictly positive, then it follows that two neighboring trajectories will diverge exponentially. We will use ’expanding’ to refer to this unstable, exponentially diverging behavior of a set of neurons to avoid confusion with Gaussian divergence, which we will need to invoke in a different context, below.

Thus, we will say that the dynamics of a set of active neurons is expanding if (7) where V is a projection matrix which describes subspaces that are unstable. For example, for a circuit with two excitatory units that cannot both be simultaneously active, V is (8) and Θ is a metric. The constraint (7) asserts that the system escapes the unstable subspaces where Vx is constant. This guarantees that For V as defined above $\mathbf{\text{Vx}}=\left[\begin{array}{c}\hfill \alpha x_1-{\beta }_1{x}_3\hfill \\ \hfill \alpha x_2-{\beta }_1{x}_3\hfill \end{array}\right]$. Each row represents one excitatory unit. Guaranteeing that Vx cannot remain constant for a particular subset implements the requirement that for a subspace to be forbidden, it cannot be a steady state because if it were Vx would remain constant after convergence.

The parameter conditions under which Eqn 7 holds, are given in detail in [24]. Importantly, these conditions guarantee that when the dynamics of our asymmetric network occupies an unstable subspace, all eigenvectors are mixed (see Methods for proof). Consequently, as in the symmetric case of [16, 25], this unstable subspace will be left (it is forbidden ), because one unit will fall beneath its threshold exponentially quickly and so become inactive.

Divergence quenches computational instability

The dynamics of our asymmetric networks can now be explained in terms of the contraction theory framework outlined above. Consider a simple network consisting of N = 5 neurons, one of which is inhibitory and enforces competition through shared inhibition (Fig. 2A). For suitable parameters that satisfy the contraction constraints (see [24]), this network will contract towards a steady state for any input. The steady state will be such that the network amplifies one of the inputs while suppressing all others (for example, Fig. 2B). During this process of output selection and amplification, the network passes through a sequence of transformations, at each of which a different subset of units becomes active whereas the remainder are driven beneath threshold and so are inactive. These transformations continue while the network is in its expansion phase and cease when the network contracts. The network is in the expansion phase while the effective Jacobian is positive definite, and contracts when the Jacobian becomes negative definite (Fig. 2C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Illustration of key concepts to describe biological computation.

(A) Connectivity of the circuit used in this figure. (B–F) Simulation results, with parameters α1 = 1.2, β₁ = 3, β₂ = 0.25, G₅ = 1.5, G_1..4 = 1.1. (B) Activity levels (top) as a function of time for the units shown in (A) and external inputs provided (bottom). After stimulus onset at t = 2000, the network selectively amplifies the unit with the maximal input while suppressing all others. (C) Maximal and minimal eigenvalue of the functional connectivity F_S active at every point of time. At t = 5000 the system enters a permitted subspace, as indicated by the max eigenvalue becoming negative. (D) Divergence as a function of time. The divergence decreases with every subsapce transition. (E) Illustration of all possible subspaces (sets) of the network. Subspaces are ordered by their divergence. For these parameters, only subspaces with at most one unit above threshold are permitted (green) whereas the others are forbidden (red). (F) Divergence as a function of subspace number (red and green circles) as well as trajectory through set space for the simulation illustrated in (B–D). Note how the divergence decreases with each transition, except when the input changes (set 16 is the off state).

https://doi.org/10.1371/journal.pcbi.1004039.g002

The computational process of selecting a solution, conditional on inputs and synaptic constraints, involves testing successive subspaces, expressed as changing patterns of neuronal activation (Fig. 2B). Subspaces do not offer a solution (are forbidden) if for that subspace VJV^T is positive definite. In this case its dynamics are expanding, and because all eigenvectors in this subspace are mixed, the subspace will finally switch. Subspaces that offer solutions (are permitted) will have an effective Jacobian that is negative definite in some metric (they are contracting), and so the system will not leave such a subspace provided that the input remains fixed. Note that there can be subspaces whose Jacobian is neither positive nor negative definite, which are then neither permitted or forbidden. However, by definition, the networks we describe assure that each possible subspace is either permitted or forbidden.

Consider the case in which the computational process begins in a forbidden subspace (Fig. 2F). The process then passes successively through several forbidden subspaces before reaching a permitted subspace. Two properties ensure this remarkable orderly progression towards a permitted subspace. Firstly, the process is driven by the instability that ensures that forbidden subspaces are left. However the trajectory is associated with a progressive reduction in the maximum positive eigenvalue of the active set (Fig. 2C).

Secondly, an orderly progression through forbidden subspaces is ensured by systematic reduction of the state space through Gaussian divergence. Gauss’ theorem (9) asserts that, in the absence of random fluctuations, any volume element δV shrinks exponentially to zero for uniformly negative definite $\text{div}(\frac{\text{d}}{\text{dt}}\delta \text{z})$. This implies convergence to an (n−1) dimensional manifold rather than to a single trajectory. For our system, the Gaussian divergence is the trace of the Jacobian J or equivalently the sum of all of its eigenvalues. Note that this is a much weaker requirement than full contraction. In particular, we are concerned about the trace of the effective Jacobian, which is the Jacobian of only the active elements, because the inactive elements (those below threshold) do not contribute to the dynamics.

The divergence quantifies the rate at which the volume shrinks (exponential) in its n dimensional manifold towards a (n−1) dimensional manifold. The system will enter the (n−1) dimensional and continue its evolution in this new active subset, and so on, until the reduced dimensional system finally enters a permitted subspace (which is contracting). Note that here we refer to the dimensionality of the system dynamics. This dimensionality is not necessarily equal to the number of active units (i.e. at steady state or when one of the units is constant).

Note that negative Gaussian divergence does not follow automatically from expansion. In fact most expanding sets will not have negative Gaussian divergence, and so will not be forbidden

For example consider the linear system $\stackrel{.}{\mathbf{\text{x}}}=\mathbf{\text{Wx}}$ with $W=\left[\begin{array}{ccc}\hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right]$. This system is expanding but it does not have negative divergence.

For orderly computation to proceed, such sets must not exist. We require that all forbidden subspaces are expanding as defined by contraction theory as well as have negative Gaussian divergence. Indeed, for our LTN networks all forbidden subspaces are guaranteed to satisfy both these properties.

Computation is steered by the rotational dynamics induced by asymmetric connections

Because their positive output is unbounded, linear thresholded neurons are essentially insensitive to the range of their positive inputs. Networks of these neurons amplify their inputs in a scale-free manner, according to the slope of their activation functions and the network gain induced by their connections. As explained above, this high gain drives the exploration and selection dynamics of the computational process. The network harnesses its high gain to steer the computation through its asymmetric connections. Indeed, the asymmetric nature of the connectivity in our network is central to its operation, and not a minor deviation from symmetry (approximate symmetry, [26]).

The interplay between high-gain and steering can be appreciated by considering the behavior of the system within one of the subspaces of its computation.

Consider a linear system of the form $\stackrel{.}{\mathbf{\text{x}}}=\mathbf{\text{f}}(\mathbf{\text{x}},t)$, which can be written as $\stackrel{.}{\mathbf{\text{x}}}=\mathbf{\text{M}}x+\mathbf{\text{u}}$ where M is a matrix of connection weights and u a vector of constant inputs. For the example of a WTA with 2 excitatory units and 1 inhibitory unit, (10) Setting M = M₁+M₂ and defining (11) (12) provides a decomposition into a component with negative divergence and zero divergence (see methods for an example).

Any vector field f(x) can be written as the sum of a gradient field ∇V(x) and a rotational vector field ρ(x), i.e. a vector field whose divergence is zero. In analogy, $f(\mathbf{\text{x}})=\stackrel{.}{\mathbf{\text{x}}}={\mathbf{\text{M}}}_{1}x+\mathbf{\text{u}}+{\mathbf{\text{M}}}_{2}x$ where ∇V(x) = M₁x+u and ρ(x) = M₂x.

For our network M₁ is the expansion/contraction component and M₂ is the rotational component. This is an application of the Helmholtz decomposition theorem to our network [27, 28]. These two matrices relate to two functional components in the network architecture: The excitatory recurrence plus input, and the inhibitory recurrence. The first component provides the negative divergence that defines the gradient of computation, while the second steers its direction as follows.

Since M₂ has a divergence of zero, this component is rotational. If, in addition, M₂ is skew-symmetric so that $-{\mathbf{\text{M}}}_2={\mathbf{\text{M}}}_2^T$, the system is rotating, and the eigenvalues of M₂ will be imaginary only. In general, β₂ ≠ −β₁ and M₂ is thus not skew-symmetric. However, note that a transform of the form Φf(x)Φ⁻¹ can be found that makes M₂ skew-symmetric. Because such transform does not change the eigenvalues of M₁, it will not change its divergence either. For above example, (13) which will result in a version of M₂ that is skew-symmetric (14)

The same transformation to a different metric has to be applied to M₁ as well, but as both M₁ and Φ are diagonal this will leave M₁ unchanged, M₁ = ΦM₁Φ⁻¹.

The orderly progression through the forbidden subspaces can be understood in this framework: The negative divergence provided by M₁ enforces an exponential reduction of any volume of the state space while the rotational component M₂ enforces exploration of the state space by directing (steering) the dynamics.

The stability of the permitted subspaces can also be understood in this framework. Permitted subspaces are contracting, despite the strong self-recurrence of the excitatory elements that results in positive on-diagonal elements. This high gain, which is necessary for computation, is kept under control by a fast enough rotational component M₂ which ensures stability. This can be seen directly by considering one of the constraints imposed by contraction analysis on valid parameters affecting an individual neuron: keeping $\alpha <2\sqrt{{\beta }_1{\beta }_2}$ guarantees that the system rotates sufficiently fast to remain stable.

Note that both M₁ and M₂ change dynamically as a function of the currently active subspace. Thus, the direction and strength of the divergence and rotation change continuously as a function of both the currently active set as well as the input.

Changes of entropy during computation

While the network is in the expansion phase, the volume of the state space in which the dynamics of the network evolves is shrinking. This process depends on initial conditions and external inputs. Consequently, the sequence by which dimensions are removed from state space is sensitive to initial conditions. To quantify the behavior of a network for arbitrary initial conditions it is useful to compute its information entropy H(t) as a function of time (see methods). The smaller H(t), the smaller the uncertainty about which subspace the network occupies at time t. Once H(t) ceases to change, the network has converged. To reduce state uncertainty, and so to reduce entropy, is fundamentally what it means to compute [7].

The entropy remains 0 immediately after the application of external inputs to the network, because the same “all on” subspace is reached regardless of initial conditions. Thereafter the network begins to transition through the hierarchy of forbidden subspaces (Fig. 2F). This process initially increases the entropy as the network explores different subspaces. Eventually, after attaining peak entropy, the network reduces entropy as it converges towards one of its few permitted subspaces. Fig. 3 illustrates this process for the network shown in Fig. 2A.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Spontaneous increase followed by reduction of state entropy during the expansion phase.

Random inputs were provided to the 5-node network as shown in Fig. 2. Each input I_i was chosen i.i.d from a normal distribution with μ = 6 and σ = 0.25. (A) Entropy as a function of time and gain of the network. Higher gains are associated with faster increases and reductions of entropy but converge to the same asymptotic entropy. This indicates that each permitted subspace is reached with equal probability. (B) Adding constraints reduces the peak and asymptotic entropy. The more constraints that are added, the larger is the reduction in entropy. (C) Comparison of time-course of average entropy and divergence (trace). (D) Comparison of time-course of average entropy and eigenvalues (min, max). Notice how both the divergence (C) and the eigenvalues (D) reach their maximal values well before the entropy reaches its maximum.

https://doi.org/10.1371/journal.pcbi.1004039.g003

Increasing the gain by increasing the value of the self-recurrence α increases the speed by which entropy is changed but not its asymptotic value. This means that all permitted subspaces are equally likely to be the solution but that solutions are found mode quickly with higher gain. Adding additional constraints through excitatory connections makes some permitted subspaces more likely to be the solution, and so the asymptotic entropy is lower (see Fig. 3B, where a connection α₂ = 0.2 from unit 1 to 2 and α₃ = 0.2 from unit 4 to 2 was added).

Note how the number of constraints and the gain of the network are systematically related to both the increasing and decreasing components of H(t). For example, increasing the gain leads to a more rapid increase of entropy, reaching peak earlier and decaying faster towards the asymptotic value (Fig. 3A). Also, adding constraints results in smaller peak entropy, indicating that the additional constraints limited the overall complexity of the computation throughout (Fig. 3B).

The network reaches maximal entropy when there exists the largest number of forbidden subspaces having the same divergence. This occurs always for an intermediate value of divergence, because then there occurs the largest number of subspaces having an equal number of units active and inactive. This can be seen by considering $\underset{k}{\text{arg max}}\left(\begin{array}{c}N\\ k\end{array}\right)=\frac{N}{2}$ (if N is even), i.e. the network will have maximal entropy when the number of active units is 50%. Numerically, this can be seen by comparing the time-course of the maximal positive eigenvalue or the divergence with that of the time-course of the entropy (Fig. 3C,D).

Overall, H(t) demonstrates the dynamics of the computational process, which begins in the same state (at its extreme, all units on), and then proceeds to explore forbidden subspaces in a systematic fashion by first expanding and than contracting towards a permitted subspace.

Structure and steering of computation

We will now use the concepts introduced in the preceding sections to explain how our circuits compute and how this understanding can be utilized to systematically alter the computation through external inputs and wiring changes in the network.

Provided that the external input remains constant, the network proceeds in an orderly and directed fashion through a sequence of forbidden subspaces. This sequence of steps is guaranteed to not revisit subspaces already explored, because when in a forbidden subspace S₁ with divergence d₁, the next subspace S₂ that the network enters must have more negative divergence d₂ < d₁. It thus follows that when the system has left a subspace with divergence d₁ that it can never return to any subspace with divergence ≥ d₁. It also follows that the network can only ever enter one of the many subspaces S_i with equal divergence d_i = X (Fig. 2F shows an example). Not all subspaces with lower d_i than the current subspace are reachable. This is because once a unit has become inactive by crossing its activation threshold, it will remain inactive. Together, this introduces a hierarchy of forbidden subspaces that the network traverses while in the exploration phase. Fig. 4A shows the hierarchy of the sets imposed by the network used in Fig. 2. This tree-like structure of subspaces constrains computation in such a way that at any point of time, only a limited number of choices can be made. As a consequence, once the network enters a certain forbidden subspace, a subset of other forbidden and permitted subspaces becomes unreachable (Fig. 4B). What those choices are depends on the input whereas the tree-like structure of the subspaces is given by the network connectivity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Hierarchy of subspaces and steering of computation.

(A) 5-node network, using notation from Fig. 2. (B) Simulation of an individual run. Top shows the state and bottom the inputs to the network. The unit with the maximal input wins (unit 4). (C) Trajectory through state space for the simulation shown in (B). Each subspace is numbered and plotted as a function of its divergence. Red and green dots indicate forbidden and permitted subspaces respectively. Numbers inside the dots are the subspace (set) numbers (see Fig. 2E). (C–D) Transition probabilities for the same network, simulated with 1000 different random inputs. Connected subspaces are subsets between which the network can transition, in the direction that reduces divergence (gray lines). The size of dots and lines indicates the likelihood that a subset will be visited or a transition executed, respectively. The subset with most negative divergence is the zero set (all units off), which is not shown. (C) All transitions made by the network. Depending on the value of the input, the network will reach one of the permitted subspaces. Notice the strict hierarchy: all transitions were towards subspaces with lower divergence and only a subset of all possible transitions are possible. (D) All transitions the network made, conditional on that subspace 13 was the solution. Note that this subspace cannot be reached from some subspaces (such as nr 7).

https://doi.org/10.1371/journal.pcbi.1004039.g004

Knowledge of how the network transitions through the hierarchy of forbidden subspaces can be used to systematically introduce biases into the computational process. Such additional steering of the computation can be achieved by adding connections in the network. Additional off-diagonal excitatory connections, for example, will make it more likely that a certain configuration is the eventual winner. An example of this effect is shown in Fig. 5, where adding two additional excitatory connections results in the network being more likely to arrive in a given permitted subspace than others. For identical inputs (compare Figs. 5B and 4B) the resulting permitted subspace can be different through such steering.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Steering of computation.

(A) The same network as shown in Fig. 4A, with two additional excitatory connections α₂ = 0.2 and α₃ = 0.2. (B) Simulation of an individual run. Top shows the state and bottom the inputs to the network. Although the input is identical to Fig. 4B, the computation proceeds differently due to presence of the additional connections. (C) Trajectory through state space for the simulation shown in (C). The state space trajectory is initially identical to the simulation shown in (Fig. 4C), but then diverges. (D) All transitions made by the network. The transition probabilities are now biased and non-equal. Consequently, some permitted subspaces are reached more frequently than others.

https://doi.org/10.1371/journal.pcbi.1004039.g005

The network remains continuously sensitive to changes in the external input. This is important and can be used to steer the computation without changing the structure of the network. In the absence of changes in the external input, the network is unable to make transitions other than those which lead to subspaces with lower divergence. When the input changes, on the other hand, the network can make such changes. For example, if the inputs change as a consequence of the network entering a certain forbidden subspace, the network can selectively avoid making certain transitions (Fig. 6A). This will steer the computation such that some permitted subspaces are reached with higher likelihood. Noisy inputs similarly can lead to transitions which make divergence less negative. Neverthless, the large majority of transitions remains negative as long as noise levels are not too large. For example, repeating the same simulation but adding normally distributed noise with σ = 1 and μ = 0 resulted in 26% of transitions being against the gradient (see Fig. 6B for an illustration).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Steering of computation by external inputs.

The network remains sensitive to changes in its input throughout the computation. Transitions are bi-directional: towards more negative and positive divergence are gray and green, respectively. (A) Example of a selective change in the input. Here, unit 4 receives additional external input only if the network enters forbidden subspace 6 (see inset). This change forces the network to make a transition to a subspace with less negative divergence (green). As a result, the eventual solution is more likely to be subspace 15. (B) Example of continuously changing inputs (input noise). Notice how noise introduces additional transitions towards less negative divergence at all stages. However, the general gradient towards more negative divergence persists: overall, 26% of transitions made the divergence more positive.

https://doi.org/10.1371/journal.pcbi.1004039.g006

So far, we have illustrated how negative divergence and expansion jointly drive the computational process in a simple circuit of multiple excitatory neurons that have common inhibition. While this circuit alone is already capable of performing sophisticated computation,many computations require that several circuits interact with one another [23, 29] The concepts developed in this paper can also be applied to such compound circuits, because the circuit motifs and parameter bounds we describe guarantee that collections of these circuits will also possess forbidden and permitted subspaces and are thus also computing. The compound network is guaranteed to be dynamically bounded, which means that no neuron’s activity can escape towards infinity. This property of the collective system relies on two key aspects: i) collections of individual circuits with negative divergence also have negative divergence, and ii) collective stability [24, 30]. Together, these properties guarantee that collections of the motifs will compute automatically.

Consider a network assembled by randomly placing instances of the two circuit motifs on a 2D plane and connecting them to each other probabilistically (Fig. 7A,B and methods). This random configuration results in some excitatory elements sharing inhibition via only one inhibitory motif, whereas others take part in many inhibitory feedback loops (Fig. 7B). This random circuit will compute spontaneously (Fig. 7C,D). It is not known a priori how many forbidden and permitted subspaces the network has, nor how many possible solutions it can reach. Nevertheless, it is guaranteed that the network will reduce entropy and eventually reach a permitted subspace (Fig. 7E). The more connections (constraints) that are added to the network the smaller the number of permitted subspaces, and generally the harder the computation will become. How long the computation will take to reach a permitted subspace depends on both the network size, and the number of connections (constraints). Generally, the smaller the number of permitted subspaces the harder the computation will be. The important point is that random instances of such circuits will always compute, which means they will always reach a permitted subspace (Fig. 7F).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 7. Large random networks spontaneously compute.

(A) Each site in a 10×10 grid is either empty or occupied by an excitatory (green) or inhibitory (red) neurons (probabilistically). (B) Connectivity matrix. (C) Example run. (D) Entropy and divergence as a function of time for random initial conditions. (E) The number of different subspaces visited reduces as a function of time similarly to the entropy. (F) Time ±s.d. till the network first enters a permitted subspace as a function of network size, specified as grid with. 100 runs of randomly assembled networks were simulated for each size. The larger the network, the longer the network spent going through the forbidden subspaces. Panels A–E show an example of grid size 10. See Methods for simulation details.

https://doi.org/10.1371/journal.pcbi.1004039.g007

Numerical methods

All simulations were performed with Euler integration with δ = 0.01, thus τ is equivalent to 100 integration steps. All times in the figures and text refer to numerical integration steps. Unless noted otherwise, the external inputs I_i(t) were set to 0 at t = 0 and then to a constant non-zero value at t = 2000. The constant value I_i was drawn randomly and i.i.d. from $\mathcal{N}(6,1\sigma )$. All simulations were implemented in MATLAB.

In cases where noise was added, the noise was supplied as an additional external input term $\mathcal{N}(0,\sigma )$ with σ = 1. (15) A new noise sample is drawn from the random variable every τ to avoid numerical integration problems.

Entropy

The information entropy H(t) is (16)

The sum is over all subspaces i and p_i(t) is the probability of the network being in subspace i at time t, over random initial conditions (we used 1000 runs). By definition, we take 0log₂(0) = 0.

Analytical methods

The principal analytical tool used is contraction analysis [17, 59–61]. In this section, we briefly summarize the application of contraction analysis to analyzing asymmetric dynamically bounded networks [24]. Essentially, a nonlinear time-varying dynamic system will be called contracting if arbitrary initial conditions or temporary disturbances are forgotten exponentially fast, i.e., if trajectories of the perturbed system return to their unperturbed behavior with an exponential convergence rate. Relatively simple algebraic conditions can be given for this stability-like property to be verified, and this property is preserved through basic system combinations and aggregations.

A nonlinear contracting system has the following properties [17, 59–61]

• global exponential convergence and stability are guaranteed
• convergence rates can be explicitly computed as eigenvalues of well-defined Hermitian matrices
• combinations and aggregations of contracting systems are also contracting
• robustness to variations in dynamics can be easily quantified

Consider now a general dynamical system in ℝⁿ, (17) with f a smooth non-linear function. The central result of Contraction Analysis, derived in [17] in both real and complex forms, can be stated as:

Theorem Denote by $\frac{\partial \mathbf{\text{f}}}{\partial \mathbf{\text{x}}}$ the Jacobian matrix of f with respect to x. Assume that there exists a complex square matrix Θ(x, t) such that the Hermitian matrix Θ(x, t)^*T Θ(x, t) is uniformly positive definite, and the Hermitian part F_H of the matrix is uniformly negative definite. Then, all system trajectories converge exponentially to a single trajectory, with convergence rate |sup_{x, t}λ_max(F_H)| > 0. The system is said to be contracting, F is called its generalized Jacobian, and Θ(x, t)^*T Θ(x, t) its contraction metric. The contraction rate is the absolute value of the largest eigenvalue (closest to zero, although still negative) λ = ∣λ_max(F_H)∣.

In the linear time-invariant case, a system is globally contracting if and only if it is strictly stable, and F can be chosen as a normal Jordan form of the system, with Θ a real matrix defining the coordinate transformation to that form [17]. Alternatively, if the system is diagonalizable, F can be chosen as the diagonal form of the system, with Θ a complex matrix diagonalizing the system. In that case, F_H is a diagonal matrix composed of the real parts of the eigenvalues of the original system matrix. Here, we choose Θ = Q⁻¹ where Q is defined based on the eigendecomposition J = QΛQ⁻¹.

The methods of Contraction Analysis were crucial for our study for the following reasons: i) Contraction and divergence rates are exponential guarantees rather than asymptotic (note that the more familiar Lyapunov exponents can be viewed as the average over infinite time of the instantaneous contraction rates in an identity metric). ii) No energy function is required. Instead, the analysis depends on a metric Θ that can be identified for a large class of networks using the approach outlined. iii) The analysis is applicable to non-autonomous systems with constantly changing inputs.

The boundary conditions for a WTA-type network to be contracting as well as to move exponentially away from non-permitted configurations were derived in detail in [24]. They are: (18)