Network model, learning paradigm, and quantification of capacity

We are interested in the effect of population sparseness on the optimal strength of plasticity in an associative network. One might hypothesize that sparse representations facilitate a large capacity when there is large synaptic plasticity. This hypothesis is based on the following considerations: In order to establish an association between two neuronal assemblies, the synapses between the neurons that are part of the first assembly and the neurons of the second assembly need to be strengthened. If f is the fraction of the total number of neurons that forms an assembly (activation ratio), the number of potential connections between neurons of the two assemblies scales with f². A low activation ratio is interpreted as high population sparseness, while a large activation ratio means dense representations and hence a low population sparseness (Fig 1A). On the one hand, learning an association between two sparser assemblies involves only a smaller fraction of synapses. In this case, the memory capacity can be larger because each association affects fewer synapses; however, sparseness cannot be arbitrarily low. On the other hand, an association between denser assemblies involves the modification of a larger fraction of synapses, which can lead to catastrophic interference and lower capacity. For example, for sparser assemblies where only 1% of the cells are active (f = 0.01), only a fraction of the synapses is changed, while for denser assemblies with f = 0.5, a quarter (f² = 0.25) of all synapses is updated; however, if synaptic plasticity is lower and not all synapses are updated, denser assemblies might generate higher memory capacity. Therefore, to test the hypothesis, the effect of this interaction needs to be quantified.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Sparse vs dense representations and quantification of capacity.

(A) In a network of neurons with an activation ratio f, which is the number of neurons in an assembly divided by the total number of neurons in the network, the number of connections potentially involved in an association between two assemblies scales with f². The number of relevant connections is thus smaller for sparse representations (left) than for dense representations (right). (B) Illustration of the network model with an input layer (yellow) of size N_in (black circles) with input patterns x (indicated by gray discs and blue circles) of activation ratio f_in and an output layer (green) of size N_out with output patterns y (gray discs and blue circles) of activation ratio f_out that are connected by a binary weight matrix J with a morphological connectivity (not sketched) and a functional connectivity c (lines between input and output layer) (both normalized per output unit). During learning, the connections between active units are turned on with transition probability ; a homeostasis mechanism maintains the functional connectivity c. (C) Schematic illustration of the learning and retrieval processes. After learning the k-th pattern, P additional patterns are learned. In order to determine the memory signal quality of the k-th pattern at time P, input pattern is presented to the network again to calculate an output pattern , which can then be compared to the target output pattern . (D) The quality of the memory signal (signal quality) is defined as (green line). It decays with the number of subsequently stored patterns P. The average Hamming distance between two random f_out-sparse vectors of length N_out (dashed gray line) is an upper bound of the average signal quality. The retrieval threshold is marked as (dashed magenta line). The capacity of the network is defined as the number of patterns P for which the signal quality reaches the retrieval threshold .

https://doi.org/10.1371/journal.pcbi.1013235.g001

We thus investigate the interplay between sparseness and plasticity in a hetero-associative one-layer feed-forward network with N_in input and N_out output units (1B), similar to the architecture of a Willshaw network [29]. This type of hetero-associative network and its storage capacity for both dense and sparse patterns have been of interest for many decades and have been studied extensively [43,44,47,48,52]. The new contribution of our work is an elaboration of the relationship between the sparseness of input and output patterns and the strength of the network’s plasticity as well as their joint effect on the network’s capacity to store memories, especially when this capacity should be as large as possible.

In our setup, a memory is a one-way association between an input pattern x and an output pattern y. Units have two states and can be either active (1) or inactive (0). The input and output activation ratios f_in and f_out are defined as the fraction of active units in the input representation and the output representation, respectively. We denote the number of active input units in an input pattern x as ; similarly, the number of active output units in a target output pattern y is . We typically assume and . Input and output layers of the network are connected by a weight matrix J with binary entries. For the construction of J, we distinguish between morphological and functional connectivity. In biological neural systems, the number of morphological (or structural) synapses per unit is constrained. Therefore, we assume a morphological connectivity . The morphological connectivity is normalized per output unit, and thus each output unit is targeted by exactly morphological connections chosen randomly among all possible connections from input units. Entries of the weight matrix J that do not correspond to a morphological connection are permanently set to zero. We have a total number of morphological connections between the input and the output units, and each connection can be functional (on, value 1) or silent (off, value 0). The weight matrix is initialized such that each output unit is targeted by exactly cN_in connections that are functional (thin gray lines in Fig 1C top), and we call c the functional connectivity. Note that the functional connectivity is bounded by the morphological connectivity: .

The network learns a sequence of uncorrelated input and output pattern pairs , which are pairs of randomly drawn binary vectors that meet the sparseness constraint given by f_in or f_out. Since there is no explicit time component, presenting one pattern pair to the network can be interpreted as one step in time. For each pattern, the weights of the connection matrix J are updated in the following way: First, we consider a connection between an active input unit (filled gray circle in yellow layer in Fig 1C top) and an active output unit (filled gray circle in green layer in Fig 1C top). Such a connection can change only if there is a morphological connection. A connection that was already functional stays on, i.e., it is not changed (thin blue line originating from last input unit in Fig 1C top). A connection that was silent is turned on with some transition probability . These connections are depicted as thick blue lines in Fig 1C top. We interpret the transition probability as the strength of plasticity (or degree of plasticity) of the network. It is this plasticity parameter that determines how plastic the network connections are. For a high transition probability , many connections are updated at once, and the memory is strongly engraved into the network. Note that, even though it is often true that higher plasticity leads to more rapid learning, we do not investigate the speed of learning, e.g., in terms of how many trials are needed for a satisfactory outcome (also see Section ‘Strength of plasticity and related concepts’ in the Discussion). Each pattern pair is presented only once and learned in one shot — regardless of the plasticity parameter .

Furthermore, a homeostasis mechanism maintains the functional connectivity c. Connections that were previously functional are turned off in order to balance the additional connections turned on during a learning step. Otherwise, all morphological connections would eventually become functional. In principle, there are many possible ways to normalize the number of functional connections and provide homeostatic stabilization. Inspired by studies suggesting that neurons monitor the total excitation that they receive and that the summed synaptic surface area per length of dendritic segment is conserved over time [53,54], we apply a post-synaptic homeostasis step after each learning step: The functional connectivity c is maintained per output unit by randomly silencing the necessary number of connections from inactive input units to active output units. To this end, we must assume (see Methods, Section ‘Hebbian and homeostatic update rules from a probabilistic perspective’ for details). Connections targeting inactive output units are not changed. This combination of Hebbian potentiation and homeostatic depression realizes a learning rule that is similar to the ones suggested by [36] and by [37]. In summary, the matrix update is performed in two steps: First, a Hebbian update is performed with success probability ,

(1)

where has entries of 1 for predefined morphologically existing connections, thus , and is a random binary matrix with independent entries. Second, a random homeostatic step is performed which masks away excess connections,

(2)

where the mask is a random binary matrix with N_out constraints .

To assess memory capacity, we quantify changes of the memory trace of the k-th pattern pair after learning P subsequent pattern pairs. With increasing P, the memory trace deteriorates more until the memory is no longer retrievable according to some criterion. We always understand the time point P as relative to the k-th pattern. So after learning the k-th pattern, P additional patterns (illustrated for P = 1 in Fig 1C center; units active in the k + 1st pattern are shown as filled gray circles, units active in the k-th pattern are shown as circles with dark blue stroke) are learned by turning on connections that are relevant for these patterns and turning others off, potentially also connections that were relevant for the k-th pattern. Connections that are turned on in this step are shown as dashed orange lines and connections that are silenced are shown as light blue lines that are crossed out. The network hence realizes a palimpsest memory system [55] where older memories are erased by newer memories due to the homeostasis mechanism.

In order to quantify memory capacity, we follow the line of work started by [46] (see also [45]) but now include not only the plasticity parameter but also the homeostatic update (cf. Eq (2)). As before, we distinguish two types of output units: Output units are either genuine with respect to the k-th pattern, hence active in the original target output or spurious with respect to the k-th pattern, hence inactive in the target output . During retrieval, there are errors because genuine output units might become inactive and spurious units might become active. At any time step P after learning the k-th pattern, the retrieval of the k-th input-output pattern pair can be tested by presenting the original input pattern and calculating the output with the current weight matrix J^[k+P] (Fig 1C bottom). Some of the relevant connections of the k-th pattern are still functional (blue lines) and provide input to the genuine output units (blue output units in Fig 1C, bottom). Other connections from the same input units might be functional due to learning subsequent patterns (dashed orange lines) or due to the initial weights before learning the k-th pattern. These connections provide input to spurious output units (red output unit in Fig 1C bottom).

Output units are modeled as McCulloch-Pitts neurons [56], which sum the weighted inputs and compare them against an activation threshold . Their output is calculated by

(3)

where

(4)

is the Heaviside step function. The activation threshold is adjusted in every time step such that the calculated output pattern has the same activation ratio f_out as the target output. This phenomenologically describes latent competition where winners take all and the rest is deactivated. Note that this choice of threshold is motivated by the comparability of results based on fixed sparseness values. It is not necessarily an optimal activation threshold in that it maximizes capacity or other performance metrics. The difference between the output obtained with the weight matrix J^[k+P] at time (or pattern) P and the original target output of this input pattern is measured by the absolute number of bit flips between and , termed the Hamming distance

(5)

(6)

The larger the number of subsequent patterns P that have been learned since the one that is being tested for retrieval, the more the obtained output will differ from the target output because the summed inputs to the genuine units become too small compared to the summed inputs of the spurious units.

To quantify the strength of the memory after P time steps, we define the signal quality

(7)

where the first term is the average Hamming distance between two random f_out-sparse patterns (Fig 1D, dashed gray line) and the second term is the Hamming distance between the calculated output representation and the target (Fig 1D). For two random f_out-sparse patterns, an error occurs if either the bit of the first pattern is 1 (with probability f_out) and the bit of the second pattern is 0 (with probability ) or the first bit is 0 (with probability ) and the second one is 1 (with probability f_out); the probability for a deviation between the two entries is thus the sum over the probabilities of the two distinct events. Summed over all N_out entries, this gives H_avg.

We define the memory capacity of the network as the lifetime of a memory stored in the network, which is the time that passes until the signal of the memory has decayed to the point that the memory cannot be successfully retrieved anymore. Alternatively, it is the number of patterns that can be stored in addition to the first pattern, without destroying the retrieval of the first pattern. We note that defining capacity in this way has limitations because it does not take into account the information-theoretic efficiency of synapse use [47,52,57] (see ‘Capacity measures’ for a discussion of these limitations). However, a preliminary analysis suggests that the main results of this manuscript are mostly maintained for synaptic (information) capacity (see S8 Appendix).

Practically, the pattern capacity here is determined by introducing a retrieval threshold (Fig 1D, dashed magenta line), which represents the error tolerance between the obtained output and the target. The number of subsequently stored patterns for which the signal quality reaches the retrieval threshold can be interpreted as the lifetime of the memory or the memory capacity of the network, defined by the equation

(8)

We define the threshold

(9)

with , as a fraction of the maximal signal quality, which is the Hamming distance between two random f_out-sparse vectors. A fixed number of wrongly activated units thus has a more severe destructive effect if the total number of active output units is small. Whenever not further specified, we choose and thus .

Theory on distributions of dendritic sums and calculation of capacity

The memory capacity of the network can be calculated numerically in simulations, but it can also be described analytically in a probabilistic sense. We follow an approach that is similar to the one proposed and promoted by an extensive line of research by Willshaw, Buckingham, Palm, Sommer, Knoblauch, and others (see, e.g., [45–50]) and use the distributions of dendritic sums to calculate the retrieval error, which is expressed by the Hamming distance between the target pattern and the calculated output pattern (see also our Eq (5)). The present work extends and complements previous findings by considering an explicit parameter for the strength of plasticity. Furthermore, we include a synaptic homeostasis rule that maintains the number of functional connections per output unit cN_in. Both the strength of plasticity and the homeostasis considerably affect the distributions of dendritic sums. Moreover, assuming a fixed output sparseness f_out affects the placement of the retrieval threshold and makes the quantification of the capacity more involved. Finally, we analyze the role of sparseness for the optimal transition probability and the maximal capacity (see Sections ‘Effect of activation ratio on optimal strength of plasticity and (maximal) capacity’, ‘Role of network size and number of active units’ and ‘Noisy input patterns during retrieval’) and provide an intuitive understanding of the effect of a change in activation ratios on the memory capacity (see Section ‘Activation ratios in the distributions of dendritic sums’).

Let us now first turn to the definition dendritic sums, which is very similar to many previous approaches: Given a particular input pattern , the summed input that an output unit receives is called the dendritic sum of this output unit. Output units can potentially be either genuinely or spuriously active in the calculated output depending on whether they are active or inactive in the target output . For each output unit, the dendritic sum hence follows either the distribution of dendritic sums of spurious output units (red distribution in Fig 2A, will be called spurious distribution) or the distribution of dendritic sums of genuine output units (blue/green distributions in Fig 2A, will be called genuine distribution).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Distributions of dendritic sums and theory on capacity.

(A) The smaller the transition probability , the closer the probability mass function (PMF) of the dendritic sums of genuine output units (, blue/green) is to the PMF of the dendritic sums of spurious output units (, red). (B) The probability of a functional connection to a genuine output unit decays with the output unit usage u. The probability that an output unit is active u times across P patterns is the weight of the corresponding binomial PMF in Eq (11). Different shades of blue/green represent different transition probabilities (same as in (A)). In (A) - (B), P = 20. (C) Spurious units with a dendritic sum larger than the activation threshold are wrongly activated (red area, false positive); genuine units with a dendritic sum smaller than are wrongly not activated (blue area, false negative). (D) shows a histogram of the dendritic sums of output units. The expected Hamming distance is the sum of falsely deactivated genuine units (blue area) and falsely activated spurious units (red area). In (E), the PMFs of dendritic sums of genuine (blue) and spurious (red) units are scaled by f_out and , respectively. To respect the Balance Equation, the mass of the part of the genuine distribution left of the activation threshold (blue area) must correspond to the mass of the part of the spurious distribution right of (red area). (F) At capacity , the activation threshold corresponds to the -quantile of the spurious distribution and to the -quantile of the genuine distribution. In (C) – (F), P = 108. Further parameters: .

https://doi.org/10.1371/journal.pcbi.1013235.g002

The probability mass function (PMF) of the dendritic sum of a spurious unit is

(10)

(see Fig 2A red, derived in Methods, Section ‘Probabilistic description of dendritic sums’ and S1 Appendix, see Eqs (36) and (40)), where denotes a binomial distribution with n trials and success probability p. The spurious distribution is centered at M_inc where . We note that M_inc remains constant when P is changed due to the constant functional connectivity c that ensures a fixed number of functional connections per output unit; when a random input pattern is applied, on average, a fraction f_in of the cN_in connections receives input. The spurious distribution does not change when new uncorrelated patterns are learned, and thus the spurious distribution does not depend on the number of patterns P.

In contrast, the PMF of the distribution of the dendritic sums of genuine units depends on P; it is given by

(11)

(see Fig 2A blue/green) where the probability for a connection to a genuine unit to be functional given that the output unit is active u times across the whole pattern set is

(12)

(see Fig 2B blue/green, derived in Methods, Section ‘Probabilistic description of dendritic sums’ and S1 Appendix, see Eqs (36) and (41)). Immediately after learning a specific pattern, the probability of the relevant connections to a genuine output unit to be functional is larger than the average functional connectivity c, i.e., in Eq (12) we have . Due to the homeostatic plasticity, the probability is reduced every time this output unit is active in subsequent patterns, which is reflected in the term where u is the output unit usage. For , we have . The probability enters Eq (11) in , which is the distribution of the dendritic sums if the output unit was active u times in the other patterns. The distribution of the output unit usage u, hence the number of times that an output unit is active across a set of P patterns, explains the weight of a binomial distribution in Eq (11) (see Fig 2B black). For , the genuine distribution converges to the spurious distribution. The distributions of dendritic sums obtained from numerical simulations match the distributions in Eqs (10) and (11) very well (see the example in S5 Fig).

For an analytical description of the expected Hamming distance between a target output pattern and the corresponding output pattern calculated during retrieval, we need to calculate the expected number of output neurons that are in the wrong state in compared to . We note that the output activation ratio of the calculated output pattern should be the same as the activation ratio f_out of the target output pattern. The activation threshold T_in should be chosen accordingly. This constraint is different from other works (e.g., [47,58,59]) and provides an additional challenge to the derivation of the storage capacity (for example, Eq (15) needs to be respected; see below and Methods, Section ‘Analytical Hamming distance’ for more details). The activation threshold T_in (vertical dashed gray line in Fig 2C) determines which output units are activated and which are not. If the distributions of dendritic sums of genuine and spurious output units are far apart and do not overlap, which can be the case during the first steps after learning a pattern, hence for small P, the activation threshold T_in can perfectly separate the two distributions, and then the calculated output is identical with the target output. If the two distributions are close to each other, a part of the spurious output units is wrongly activated (red area under red distribution in Fig 2C, false positive) and a part of the genuine output units is wrongly inactivated (blue area under blue distribution in Fig 2C, false negative). These two errors both contribute to the Hamming distance between the target output pattern and the calculated output pattern but they are weighted differently, as will be explained in the following (see Methods for a detailed derivation). In order to derive an actual error contribution from these distributions, we have to consider how many units have dendritic sums following either of the two distributions. As mentioned, there are M_out active units in the target output pattern. There are hence M_out output units whose dendritic sum follows the genuine distribution and output units whose dendritic sum follows the spurious distribution. The false negative error and the false positive error thus have to be weighted by M_out and , respectively. It follows that the expected Hamming distance can be expressed as

(13)

where e_fn and e_fp denote the error probabilities due to false negatives (blue area) and false positives (red area), respectively.

Let and denote the cumulative distribution functions (CDF) of the distributions of dendritic sums of genuine units and spurious units, respectively. Then, we can express the expected Hamming distance as

(14)

see Fig 2D (derived in Methods).

As discussed in the previous section, we always adjust the activation threshold such that the output activation ratio of the target output pattern is maintained in the calculated output . This restriction is respected if the Balance Equation

(15)

is fulfilled (see Fig 2E), i.e., if the contribution to the Hamming distance of the wrongly inactive genuine units equals the contribution of the wrongly active spurious units (see derivation in Methods).

With the Balance Equation, we can express in terms of , and the expected Hamming distance simplifies to

(16)

To calculate the capacity, we need to determine the number of patterns P for which the signal quality , as defined in Eq (7), equals the retrieval threshold (see Methods for details), i.e., we have to solve the equation

(17)

which can be simplified to

(18)

Using this relation in the form in the Balance Equation yields

(19)

We now have two equations, Eq (18) and Eq (19), in two variables P and . Solving them gives us the number of patterns P (and the corresponding activation threshold ) for which the signal quality equals the retrieval threshold. This P is the capacity (or memory lifetime) .

Since the dendritic sums follow discrete distributions (see Eqs (10) and (11)), their CDFs and are not invertible over a continuous domain. If we approximate the distributions by appropriate continuous distributions, Eqs (19) and (18) can be reduced to a single equation

(20)

in . The capacity corresponds thus to the number of patterns P for which the -quantile of the spurious distribution equals the -quantile of the genuine distribution (Fig 2F). An analytical solution to Eq (20) is derived in the Methods.

Effect of activation ratio on optimal strength of plasticity and (maximal) capacity

Let us now quantify how the memory capacity depends on the activation ratio of the input pattern f_in, the activation ratio of the output pattern f_out, and on the transition probability . While many earlier works (e.g., [29,38,43,44,46]) have already analyzed how activation ratios impact the capacity of similar networks, we extend these earlier works by adding the strength of plasticity as an additional parameter. We can thus investigate the new triangular relationship between activation ratios, strength of plasticity, and capacity. This allows us to derive and discuss an optimal strength of plasticity and to define a maximal capacity.

Fig 3A shows the signal quality as a function of the number of patterns P for several transition probabilities . The signal quality always decays with time, i.e., with the number P of subsequently stored patterns. The larger the transition probability, the faster the decay because the number of connections that is updated in each time step is larger and the old memories are thus overwritten more quickly. In the long run, the signal quality decays to zero. The initial signal quality (shown at P = 0 in Fig 3A) strongly depends on the transition probability because it determines how many connections are used for the initial memory storage. For a higher transition probability, the memory is more strongly engraved into the system and, initially, can be retrieved better. If the transition probability is too low, too few connections are updated to recall a memory even right after storing it (gray circles in Fig 3A). The transition probability that yields the highest capacity, which we call the optimal transition probability, hence depends on a trade-off between a high initial encoding and a slow decay of the memory signal (Fig 3A).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Effect of activation ratio on optimal strength of plasticity and capacity.

(A) Signal quality decaying with number of patterns P for various transition probabilities . While small cause low initial signal quality, large cause a fast decay. A transition probability of yields the largest memory capacity because it reaches the retrieval threshold (magenta dashed line), which is chosen as half of the maximal signal quality (gray dashed line), for the largest number of patterns P. For approximately , the initial signal quality is lower than the threshold and the capacity is zero (gray circles, see also gray circles in (B) and (C)). Fixed parameters: . (B) For a given transition probability , the memory capacity increases with decreasing input activation ratio f_in until it reaches a maximum and then quickly decays to zero for smaller f_in. The optimal , which is defined as the transition probability that results in the largest memory capacity for a given f_in, increases with decreasing f_in until it reaches and stays at its largest value of 1 (black squares). The solid gray line shows the analytically derived optimal as a function of f_in. The signal-quality traces of the column marked by ‘I’ are shown in (A). White area indicates a capacity of zero. Fixed parameter: f_out = 0.025. (C) Same as (B) for output activation ratio f_out instead of input activation ratio f_in. The memory capacity and the optimal increase with decreasing f_out. Column ‘I’ is shown in (A). Fixed parameter: f_in = 0.025. In (D), the maximal capacity (obtained with the optimal transition probability shown in (E)) for each combination of input activation ratio f_in and output activation ratio f_out is shown. For a fixed f_in, the maximal capacity increases monotonically with decreasing f_out. For a fixed f_out, the maximal capacity has a maximum at a small but nonzero f_in. Columns ‘II’ and ‘III’ are obtained from the results shown in (B) and (C), respectively. (E) Optimal transition probability increases with decreasing f_in and with decreasing f_out. The slope is larger for f_in than f_out. Columns ‘II’ and ‘III’ are shown in (B) and (C), respectively. Further fixed parameters in (A) -(E): , N_avg = 200 (see Methods for details on averaging).

https://doi.org/10.1371/journal.pcbi.1013235.g003

Fig 3B summarizes how the memory capacity depends on the input activation ratio f_in and the transition probability — for fixed f_out. The capacity values of Fig 3A are visible in the marked column (f_in = 0.025) of Fig 3B. In each column of Fig 3B, hence for a fixed f_in, the largest capacity value that thus corresponds to the optimal transition probability is marked by a black square. The corresponding analytical estimate of the optimal transition probability is displayed by the gray line (Eq (103) in Methods, discussed in Section ‘Role of network size and number of active units’). For a fixed transition probability (single row in Fig 3B), the capacity is non-monotonic as a function of f_in, i.e., it increases with increasing input activation ratio f_in and then decreases again. As known from the literature [34,36,38], denser representations (higher f_in) can lead to lower capacity values. However, if the input representations become too sparse (too small f_in), the maximal possible capacity also drops and goes to zero for very sparse input patterns. This non-monotonic dependence of the capacity on the sparseness of the patterns has also been known for decades [37,38,45,52,60] and is confirmed here. The drop in capacity for very small input activation ratios can be explained by the fact that it is not possible for too few active input units to sufficiently excite the respective output units. In this case, a slight change of the weight matrix due to the storage of additional patterns can lead to very different units being activated in the output. In extreme cases — especially if both the transition probability and the input activation ratio f_in are too small — the pattern is not well enough imprinted from the very beginning (cf. Fig 3A, gray circles, and 3B, white region) and the capacity is zero. This issue is particularly severe if the error tolerance is small, hence if the retrieval threshold is high, i.e., is close to its maximum H_avg. In this case, the capacity can be zero even for high transition probabilities if the initial signal quality lies below . In the following section ‘Activation ratios in the distributions of dendritic sums’, we provide an intuitive understanding of the role of the input activation ratio in the distributions of dendritic sums and its impact on the capacity.

Fig 3C summarizes how the memory capacity depends on the output activation ratio f_out and the transition probability — for fixed f_in. The marked column in Fig 3C shows the same capacity values as the marked column in Fig 3B. For a constant transition probability (single rows in Fig 3C), we find that the capacity increases with decreasing output activation ratio f_out. A smaller output activation ratio f_out is beneficial for the capacity because a specific output unit is less likely to be active in subsequent patterns; hence relevant connections are turned off with lower probability. Nevertheless, if the transition probability is too small, as before, the memory is not sufficiently well imprinted from the beginning and the capacity is zero (Fig 3C, white region). Previous work showed that, due to a collapse of the retrieval reliability for high output sparseness as soon as some noise is present in the system, also the output activation ratio can have a non-monotonic effect on the capacity in that capacity decreases again if the activation ratio becomes too small [46,52,60]. However, in the parameter ranges investigated in this study, the capacity always showed a monotonic decrease with the output activation ratio. For an attempt at an intuitive explanation of the dependence of the capacity on the output sparseness via an analysis of the role of the output activation ratio in the distributions of dendritic sums, we refer again to the following section ‘Activation ratios in the distributions of dendritic sums’.

In summary, we find that in our network the sparseness values of the input and the output patterns have differential effects on the capacity of the network. While the smallest output activation ratio achieves the highest capacity values (Fig 3C), there is a non-monotonic dependence of the capacity on the input activation ratio (Fig 3B).

While capacity as a function of sparseness has been extensively investigated in earlier work (e.g., [34,37,38,45,52,60]), we can, in our setup, now extend these dependencies to the maximal capacity obtained with the optimal strength of plasticity: The non-monotonic dependence of the capacity on f_in and its monotonic dependence on f_out are also maintained for the maximal capacity. Fig 3D summarizes the maximal capacity for various pairs of input and output activation ratios f_in and f_out, which is obtained by using the optimal transition probability for each pair. As for the capacity for a fixed in Fig 3B,3C, Fig 3D shows an asymmetric effect of f_in and f_out on the maximal capacity.

Finally, the quantification of the capacity in dependence of the strength of plasticity allows us to derive the optimal strength of plasticity, which is the transition probability that yields the largest capacity — for fixed input and output activation ratios. A central result of our study is that the optimal strength of plasticity depends on the input and output activation ratios. The sparser the patterns (smaller f_in or f_out), the larger the transition probability that allows for the largest capacity (depicted by black squares in Fig 3B,3C). This effect is monotonic and has a larger gradient for the input activation ratio (Fig 3B) than for the output activation ratio (Fig 3C). This behavior is summarized in Fig 3E for a wider range of pairs of f_in and f_out.

In S6 Appendix, we compare our results that were obtained with a homeostasis based post-synaptic normalization to results obtained for a pre-synaptic normalization and for a normalization averaged over all connections. We find that the qualitative dependencies of the optimal transition probability and the maximal capacity on the activation ratios (compare Fig 5D,5E to S6 Appendix) remain unchanged.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Role of the activation ratios in the distributions of dendritic sums.

(A) Distributions of dendritic sums of genuine (blue) and spurious (red) output units. In the bottom panel, the input activation ratio f_in is four times larger than in the top panel. For a larger f_in, the difference between the means of the distributions (black arrows) is larger (note the different scales of the x-axes in the two panels). The standard deviations (red and blue arrows) are also larger for larger f_in but the coefficients of variation are smaller. Also the overall overlap of the two distributions is smaller for larger f_in. (B) The probability of a connection to a genuine output unit to be functional as a function of the output unit usage u. always decays as a function of u and approaches (red) for large u. The decay is faster for larger input activation ratio f_in. (C) The weight distribution moves to the right if f_out is increased (black: f_out four times larger than in orange). The largest weights hence affect binomials (in Eq (11)) with smaller values (blue). Further parameters: .

https://doi.org/10.1371/journal.pcbi.1013235.g004

Activation ratios in the distributions of dendritic sums

While the benefit of sparseness for the capacity for large to moderate activation ratios and the decrease of capacity for small activation ratios (in particular small input activation ratios) were already well-established in the literature (e.g., [37,38,45,52,60]), we provide here a new intuitive analysis of these effects based on the distributions of dendritic sums.

The intuitive understanding of the effects of the input and the output activation ratios on the capacity can be strengthened by having a closer look at the distributions of the dendritic sums, which determine the signal quality of the memory. We again distinguish distributions of dendritic sums of spurious and genuine units (red and blue/green, respectively, in Fig 2A) as given in Eq (10) and Eq (11), respectively. These explicit expressions for dendritic sums of spurious and genuine units allow us to understand why f_in and f_out affect the capacity of the network in different ways.

We first explain the role of f_in, while f_out is assumed to be fixed: A fixed f_out implies that the weighting of the individual binomial mass functions in Eq (11) is fixed for a given P. We can hence discuss the role of f_in in the individual binomials without considering the weighting. The capacity’s non-monotonic dependence on f_in is a combination of two separate effects working together. In the following, we discuss via what dominating mechanism f_in can positively correlate with the capacity, which is the case for very small f_in (left of the gray line in Fig 3B), and what mechanism has to dominate such that f_in is negatively correlated with the capacity, which is the case for larger f_in (right of the gray line in Fig 3B). First, the input activation ratio occurs in the number of trials of the individual binomial distributions in and the number of trials M_in of the spurious distribution . Let us assume for the moment that f_in was increased only in the number of trials of the binomials, and we ignore the dependence of on f_in in Eq (12). Such an increase of f_in in the number of trials pulls the distributions and farther apart since both mean values are proportional to f_in. They are also stretched out more (higher standard deviation with p = c or ) but the overall overlap of the two distributions is reduced because the means grow with M_in and the standard deviations grow only with (also see left hand side of Eq (91) for P = 0). This is illustrated in Fig 4A (top: smaller f_in, bottom: larger f_in). Due to the smaller overlap of the two distributions, the Hamming distance is reduced and this yields a larger capacity. For f_in increasing from zero, this increase of the capacity can be seen for the values left of the gray line in Fig 3B. However, the input activation ratio f_in also occurs in the probabilities (Eq (12)) of the individual binomials in the genuine distributions. If only changed at this place, an increased f_in leads to a faster decay of the probability with increasing u (Fig 4B) and hence with the number of patterns P. For larger f_in, each of the individual binomials in Eq (11) and hence also the total genuine distribution converges faster to the spurious distribution, which implies a deteriorative effect on the capacity. This can be seen for the capacity values right of the gray line in Fig 3B. In summary, these two effects dominate in different ranges of f_in, which can explain the intermediate value of f_in that yields the optimal capacity. For input activation ratios f_in close to zero, the effect of the change of f_in in the number of trials of the distributions (10) and (11) dominates over the effect of the change of f_in in (12). The capacity hence increases with increasing f_in if f_in is small (derived in S5 Appendix). This confirms the detrimental effect of a very small f_in on the network capacity that we discussed in the previous section: If f_in becomes very small, the number of active input units becomes too low to reliably activate the respective output units. For very large input activation ratios f_in, the effect of a change of f_in via the probability (Eq (12)) is stronger than the effect via the number of trials and the capacity decreases with increasing f_in.

The upper limit of f_in is c (see first section on condition ). The capacity for this upper limit of the input activation ratio f_in can be derived for the special case . If f_in takes its maximal value c, the probability (Eq (12)) equals c and the genuine (Eq (11)) and spurious distributions (Eq (10)) become identical. This makes it impossible to differentiate between genuine and spurious units and the capacity is hence zero. In total, the intuition that we have developed here thus matches the numerical results discussed in the previous section.

Let us now turn to the output activation ratio f_out, which affects the capacity also in two ways. We now assume a fixed f_in. First, f_out is the success probability of the binomial distribution, which constitutes the weights in the genuine distribution (Eq (11)). An increased f_out pushes the center of mass of the distribution to larger values of u (Fig 4C). The largest weights thus are given to larger u values, which correspond to binomials with smaller probabilities . This emphasizes the parts of that are more to the left of , which culminates in a shift to the left of the center of mass of the overall genuine distribution. In this sense, increasing f_out increases the overlap of the genuine distribution with the spurious distribution and hence decreases the capacity of the network. At the same time, f_out directly impacts the placement of the activation threshold because we are always enforcing active output units. This is discussed in detail in the Methods, Section ‘The activation threshold and balancing tails of the distributions’. The effect on the capacity of a change of the activation threshold due to an increase of f_out is unknown (also see Methods, Section ‘Deriving the Hamming distance from dendritic sums’). We speculate that this effect is small in the parameter range that we investigated numerically and that the effect of f_out via the weight distribution discussed above dominates the dependence of the capacity on f_out.

Role of network size and number of active units

So far, we have considered a fixed network size. We have seen how the capacity and the optimal transition probability depend on the input and the output activation ratio. Here, we discuss how these results depend on network sizes.

The following results are derived in detail in the Methods, Section ‘Analytical calculation of the capacity of the network’. Assuming that the average number of functional connections per output unit is large () but not too large () and , the capacity of the network can be analytically approximated as

(21)

with A and B being two functions that are defined in Eqs (93) and (94) in the Methods section (see also Eq (98)). Note that both A and B are independent of the input and output network size N_in and N_out. They only depend on the number of active input units M_in and the output activation ratio f_out.

Under the assumption , the optimal transition probability can be approximated by

(22)

(black line in Fig 5A, derived in Methods, Section ‘Optimal transition probability and maximal capacity’, see Eq (103)). It is independent of the input layer size N_in and the output layer size N_out. The maximal capacity for a fixed set of network parameters is obtained by using the optimal transition probability in Eq (21):

(23)

We find analytically and numerically that the size of the input layer strongly impacts the maximal capacity of the network (see Fig 5B): The capacity scales linearly with the input layer size N_in. Since is independent of N_in (5A), the maximal capacity also scales linearly with N_in (Eq (21) and Fig 5B). The maximal capacity depends non-monotonically on the number of active input units M_in. We find that, for any sufficiently large input layer size, the highest capacity is reached for the same number of active input units M_in (Fig 5B, gray line). According to Eqs (21) and (23), the maximum over M_in of the maximal capacity depends on the output activation ratio f_out, the functional and morphological connectivities c and and of course, via , also on the retrieval threshold . Since, for a fixed network size N_in, the input activation ratio is simply a scaled version of M_in, the maximal capacity also depends non-monotonically on f_in. We have seen examples of this non-monotonic dependence on f_in in Fig 3D.

In most figures, we show results for a morphological connectivity and a retrieval threshold . However, our theory also holds for smaller, more realistic morphological connectivities of a few percent and more stringent retrieval criteria, as well as for a combination of the two. The main dependencies between activation ratios, optimal transition probability, and capacity are qualitatively robust to such changes. An increase of the retrieval threshold yields a larger optimal transition probability and, of course, a smaller capacity (S1 Fig). If the morphological connectivity decreases, increases, while decreases (S2 Fig). The minimal number of active input units M_in that is necessary to store patterns and achieve non-zero capacity increases with a decrease in morphological connectivity . An increase in functional connectivity c increases both and , at least for sufficiently large M_in (S3 Fig).

As a probability, the optimal transition probability is bounded between zero and one. It is a monotonically decreasing function of M_in. For small values of M_in, it takes the maximal value one. When M_in becomes large enough, the expression becomes smaller than one and becomes strictly decreasing as a function of M_in (Fig 5A). This monotonicity of as a function of M_in could not be formally shown with our theory but has been confirmed in all evaluations of Eq (22) and all numerical simulations that were performed. In the range where , the maximal capacity is monotonically decreasing as a function of M_in (shown in the Methods, Section ‘Dependence on input parameters’) (compare Fig 5A and 5B). It follows that, if the maximal capacity has a maximum as a function of M_in (for any given network size), it is reached within the range where we have (Fig 5A, the gray line marks the M_in that yields the largest capacity). Note that, depending on the network parameters and the retrieval criterion, it can occur that there is no for which . In this case, the dependence of the maximal capacity on M_in (as well as f_in) is monotonic and the largest capacity is obtained for M_in = 1. The special role of the number of active input units for the optimal transition probability as well as for the maximal capacity is remarkable and we speculate that it is related to the fact that we choose the number c N_in of functional connections per output neuron to scale with the input network size.

As can be seen from Eqs (21) and (23) and as discussed in previous sections, the capacity depends on the output activation ratio f_out. For adequately large N_out, the size of the output layer, on the other hand, does not affect the storage capacity of the network (Eq (21) and Fig 5C, also see Discussion, Section ‘Role of the output size and retrieval criterion’).

Noisy input patterns during retrieval

We previously (Fig 3A) found that the transition probability that leads to the largest capacity depends on a balance between a large initial signal quality and a slow decay of the signal quality with the number of subsequently stored patterns. The top panel of Fig 6A also shows the signal quality as a function of the number of patterns but for a different set of input and output parameters () than Fig 3A. Let us now investigate the effect of noisy input patterns during retrieval on the optimal transition probability and the maximal capacity of the network.

The input pattern that is presented to the network during the retrieval phase does not necessarily have to be the same pattern that was used for training. From a biological perspective, it makes sense to assume that the input during retrieval is not exactly the same but a noisy version of the original input pattern. We follow a similar approach as [46,47]. The noisy input patterns are created by deactivating a particular fraction of the active input units while activating the same number of inactive input units. The input activation ratio f_in is hence maintained. Adding noise to the input pattern during retrieval reduces the memory capacity by deteriorating the signal quality right from the beginning (Fig 6A, top vs bottom). This shift in the initial signal quality is particularly strong for small transition probabilities because the memory relies on few connections to provide a strong input to the output units. If some of the input units that are supposed to be active in a pattern are inactive and others are activated instead, it is likely that many of the genuine output units are not activated anymore. The effect is less drastic for large transition probabilities for which the memory is more reliably imprinted into many connections and does not get lost just because some input units are wrongly activated. However, we find that the change in the signal quality due to noise can neither be explained by a constant shift nor by a constant multiplicative scaling. The change depends on the number of patterns in a more complex way, see S3 Appendix. There we derived an analytical description of the factor that has to be applied to the signal quality calculated without noise during retrieval to obtain an estimate of the signal quality with noise.

Despite the complex relationship between and , the qualitative dependence of the optimal transition probability on the input and output activation ratios remains the same as without noise (Fig 6B, compare to Fig 3E) — it increases with decreasing activation ratios. Further, adding noise to the input pattern during retrieval maintains the non-monotonic effect of the input activation ratio and the monotonic effect of the output activation ratio on the maximal capacity (Fig 6C, compare to Fig 3D).

We extend our theory to retrieval with noisy input patterns and find that the optimal transition probability can again be described as

(24)

where and (Eqs (112) and (113)) now additionally depend on the noise level (derived in Methods, Section ‘Extension to noisy input patterns during retrieval’). For a fixed set of parameters, noise during retrieval increases the optimal transition probability:

(25)

(Fig 6D, compare solid line to dashed line). It can be approximated as a scaled version of the optimal transition probability without noise:

(26)

(Fig 6D, dash-dotted lines, also see S4 Fig. As discussed previously, the optimal transition probability corresponds to a sweet spot that realizes both a good initial encoding of the memory and a slow decay of the memory trace. The additional noise impacts both aspects but since we find that , the noise has a more detrimental effect on the initial signal quality than on the decay of the signal quality.

The maximal capacity can again be theoretically described as

(27)

(Fig 6E, solid lines, derived in Methods, Section ‘Extension to noisy input patterns during retrieval’, see also S4 Fig). It can further be approximated by

(28)

where is the capacity without noise (Eq (21)) evaluated for and the last approximation step assumes and (Fig 6E, dash-dotted lines, see also S4 Fig; derived in S4 Appendix). Hence, the maximal capacity with noise during retrieval is the maximal capacity without noise (evaluated at ) shifted down by a fixed amount that scales linearly with . The M_in that yields the highest capacity is again independent of N_in but it increases with the noise level (Fig 6E, vertical dashed lines, see also S4 Fig).

We can conclude that in a system with noisy input patterns, the capacity is lower than in a system without noise. However, with noise, it is beneficial for the capacity to learn with a higher transition probability.

Palimpsest memory and homeostatic plasticity

The architecture of the network used in this manuscript is similar to the Willshaw network [29] in that it is a feedforward one-layer network that learns associations between binary input and output patterns in a Hebbian way (see Section ‘Network model, learning paradigm, and quantification of capacity’). In contrast to a classical Willshaw network in which all memories disappear when the capacity is reached, our network realizes a palimpsest memory system in which older memories are erased by newer memories due to a homeostasis mechanism (see, e.g., Fig 3A). This degradation of older memories allows for an online learning regime where an in principle infinite sequence of patterns can be learned — new patterns always gradually replacing older ones [36,38]. The signal quality of a memory pattern hence does not only passively decay over time due to the interference between patterns, but also decays actively due to the deactivation or depression of synaptic connections. The underlying heterosynaptic plasticity, which is characterized by an input-unspecific change of synaptic efficacies [62,63], has been found, e.g., in the hippocampus and the neocortex [64,65]. Heterosynaptic plasticity can provide an effective homeostatic constraint on Hebbian plasticity [66,67]. In the presented work, a combination of Hebbian homosynaptic and homeostatic heterosynaptic learning was used (see Section ‘Network model, learning paradigm, and quantification of capacity’). Hebbian plasticity increases connection weights with a predefined probability if both the pre- and the postsynaptic units are active, and homeostatic plasticity decreases weights with another predefined probability if only the postsynaptic unit is active (details in Methods, Section ‘Hebbian and homeostatic update rules from a probabilistic perspective’). The update probabilities, which represent the strength of plasticity, are balanced such that the total number of functional connections per output unit remains constant. This special rule of heterosynaptic plasticity is inspired by experimental evidence of a constant total summed synaptic weight on a dendritic branch [54,68].

In general, variations in the implemented homeostatic plasticity are possible. For example, in contrast to our approach, the total number of functional connections per input unit could be conserved to achieve presynaptic homeostasis by depressing connections from active input neurons to inactive output neurons (as suggested by [36], e.g., reported by [69]). Alternatively, the number of functional connections could be conserved only throughout the network, i.e., not normalized per input or per output unit; such a normalization could be achieved by randomly silencing connections that are not involved in a given association of patterns [37]. Numerical simulations based on either of these two variations suggest that the qualitative dependence of the optimal strength of plasticity on the sparseness of the patterns is the same as for the learning rule analyzed in detail in this paper (see S6 Appendix). Finally, a change in synaptic efficacies could occur due to synaptic aging where connections that have not been involved in a pre- and postsynaptic co-activity for a long time might be more likely to be depressed than those that have been strengthened more recently [70]. Such an age-dependent homeostasis rule requires synapses with metaplasticity, which could have a beneficial effect on the storage capacity of the network [38,40,71] even though not necessarily for sparse codes [38]. Furthermore, models that incorporate structural plasticity focus not only on changes in connection strength but also on the formation and elimination of synapses and the remodeling of dendritic spines [72,73]. Such models have been shown to be capable of increasing storage capacity, in particular for sparse patterns in sparsely connected networks [74,75]. It is beyond the scope of this manuscript to analyze in detail the impact of metaplasticity or structural plasticity.

Input sparseness vs output sparseness and morphological connectivity

In our heteroassociative network, both a change in the input and the output sparseness value affect the optimal strength of plasticity and the corresponding maximum memory capacity (Figs 3, 5, and 6, Eqs (21)–23)). The optimal transition probability monotonically decreases with an increase in the input activation ratio as well as with an increase in the output activation ratio, but the gradient is steeper for the input activation ratio (see Figs 3E, 6B, 13). In order to learn with high plasticity, sparseness in the input representations is thus more beneficial than sparseness in the output representations.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Role of network size and number of active units.

Results from numerical simulations are depicted by dots, and theoretical results are depicted by solid lines. (A) The optimal transition probability as a function of M_in monotonically decreases from a plateau at one to zero. does not depend on the input layer size N_in. The overall highest capacity for a fixed network size is obtained for an M_in which yields an optimal of one (vertical gray dashed line). (B) shows the maximal capacity as a function of the number of active input units for N_in = 1000, 2000, 4000, 8000 and 16000. The size of the output layer is fixed to N_out = 1000 in the simulations. The maximal capacity as a function of M_in scales linearly with N_in. The vertical gray dashed line marks the number of active input units M_in that yields the highest capacity. The highest capacity for a fixed network size is obtained at the same M_in for any network size. In (C), the output layer size N_out (=1000, 2000, 4000, 8000) is varied instead of the input layer size N_in. The number of input units N_in is fixed to 1000. The size of the output layer N_out does not have an impact on the maximal memory capacity of the network. Further parameter values in (A)–(C): .

https://doi.org/10.1371/journal.pcbi.1013235.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. With noisy input patterns during retrieval, maximal capacity decreases and optimal transition probability increases.

The noise level is set to in the entire figure. (A) The memory signal quality decays with the number P of patterns learned by the network. For smaller transition probabilities , the signal quality starts lower but decays more slowly than for larger . Top: No noise during retrieval; the largest capacity is attained for . Bottom: With noise on the input patterns during retrieval; the signal quality is lower than without noise, particularly for small transition probabilities . All capacity values with noise are slightly smaller than the corresponding ones without noise. The largest capacity is now attained for . In (A) the solid curves show results from simulations. The dotted curves are an approximation of the signal quality with noise where the signal quality without noise is multiplied by a factor obtained from theory (see S3 Appendix). Fixed parameters in (A): . (B),(C) The optimal transition probability and the maximal memory capacity obtained with noise during retrieval as functions of input and output activation ratios f_in and f_out. and behave qualitatively similar to the maximal memory capacity and the optimal transition probability obtained without noise (cf. Fig 3D,3E). In (A)–(C): . (D) The optimal transition probability with noise (solid curve, Eq (24)) is slightly higher than without noise (dashed curve, Eq (22)). The dash-dotted curve (which aligns closely with the solid curve) shows an approximation of as a multiple of (Eq (26)). The vertical dark gray and light gray dashed lines mark the M_in that yields the highest capacity with and without noise, respectively. With noise, the optimal M_in is larger than without noise. All black curves are derived from theoretical considerations, the colored dots show results from simulations (N_out fixed to 1000). (E) Noise on the input patterns during retrieval reduces the maximal capacity . Furthermore, scales with the input layer size N_in. Dashed curves show the maximal capacity without noise (Eq (23)), solid curves show the maximal capacity with noise (Eq (27)), dash-dotted curves (which align closely with the solid curves) show an approximation of the maximal capacity with noise for small (Eq (28)), and dots show numerical results (N_out fixed to 1000). Further parameter values in (A)–(E): .

https://doi.org/10.1371/journal.pcbi.1013235.g006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Distributions of dendritic sums.

Analytical probability mass functions of dendritic sums after storing P = 0, 10, 20, 30, 40, 50 patterns (Eqs (44) and (46)). Spurious units in red; genuine units in blue shades, for several transition probabilities . With an increasing number of subsequent patterns P, the distributions of the dendritic sums of genuine units approach the distribution of the dendritic sums of spurious units. The genuine distributions are multimodal, in particular for large transition probabilities and small numbers of patterns P > 0. Other parameters: .

https://doi.org/10.1371/journal.pcbi.1013235.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Distributions of dendritic sums with noise.

Analytical probability mass functions of dendritic sums with noise on the input patterns during retrieval after storing P = 0, 10, 20, 30, 40, 50 patterns (Eq (56) with (41) and (53); Eq (36) with (40)). Spurious units in red; genuine units in blue shades, for several transition probabilities . With an increasing number of subsequent patterns P, the distributions of the dendritic sums of genuine units approach the distribution of the dendritic sums of spurious units. Compared to the noise-less case (see Fig 7), the genuine distributions are closer to the spurious distribution. Other parameters: . The same figure but with can be found in S6 Fig.

https://doi.org/10.1371/journal.pcbi.1013235.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Hamming distance and signal quality.

The Hamming distance between the target and the calculated output pattern (salmon line) increases with an increasing number of patterns P and approaches H_avg (gray line) for . The signal quality (green line) decreases with increasing P and approaches 0.

https://doi.org/10.1371/journal.pcbi.1013235.g009

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. The activation threshold and balancing tails of the distributions.

(A) Distributions of dendritic sums of genuine (blue) and spurious (red) output units. Output units with a dendritic sum larger than the activation threshold are activated. (B) Distributions of dendritic sums scaled by the number of units whose dendritic sums follow the respective distributions. The sum of the number of genuine (blue) and the number of spurious (red) units that have a dendritic sum larger than the threshold corresponds to the predefined number of active output units M_out. (C) Distributions of dendritic sums of genuine (blue) and spurious (red) units scaled by f_out and , respectively. The mass of the part of the genuine distribution left of the activation threshold (blue area) must correspond to the mass of the part of the spurious distribution right of (red area). (D) Same as (C) but for larger number of patterns P = 114 instead of P = 68. The threshold () moves to the left in order to maintain the balance of the tails, which ensures the right amount of active output units.

https://doi.org/10.1371/journal.pcbi.1013235.g010

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Deriving the Hamming distance from dendritic sums.

(A) Hamming distance is the sum of falsely deactivated genuine units (blue area) and falsely activated spurious units (red area). (B) If f_out is changed, the threshold needs to be adapted such that the ratio of the red area to the blue area corresponds to the ratio . Upper: , lower: and hence the red area is much smaller compared to the blue area in the lower panel than in the upper panel. This is achieved by a larger , which can also affect the Hamming distance.

https://doi.org/10.1371/journal.pcbi.1013235.g011

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. Signal Quality Equation.

The activation threshold corresponds to the -quantile of the spurious distribution and to the -quantile of the genuine distribution.

https://doi.org/10.1371/journal.pcbi.1013235.g012

As known from the literature [29,33,34,36], the capacity of an associative network benefits from sparse representations. We find that the same is true for the maximum capacity, which is obtained by learning with the optimal transition probability for each combination of parameters. While enforcing sparser output patterns decreases the overlap in the output space and hence decreases the likelihood of retrieval errors up to extreme sparseness values of only one active output unit, using too sparse input patterns can be detrimental to the capacity. If the input representations contain a too small number of active units M_in, it is difficult for the signal, which scales as M_in, to overcome the noise, which scales as (Fig 3A,3B and Fig 5B). In other words, a too small number of active input units cannot reliably drive the respective output units. If M_in is smaller than some optimal value, the initial signal quality approaches the retrieval threshold and the capacity decreases, and if M_in becomes even smaller, the initial signal quality can be below the retrieval threshold and therefore the capacity is zero [37,38,46,52]. This collapse of the maximal capacity for too sparse input patterns occurs only in the range of M_in-values where the optimal transition probability has already reached its maximal value 1 (Fig 5A; details in Methods, Section ‘Dependence on input parameters’). Therefore, for too sparse input patterns, a decrease in M_in cannot be compensated by an increase in the transition probability.

A similar limitation may arise in networks with strongly diluted morphological connectivity . Such connectivity values are substantially more biologically realistic than the fully connected case (used in many figures of this manuscript), since anatomical connectivity in cortical networks is typically sparse [76]. In terms of the approximation of the distributions of dendritic sums by a binomial distribution, is the worst-case scenario (see S4 Appendix for details). The analytical approximation of the capacity derived in this manuscript hence improves significantly for more realistic values of . However, in the case , particularly for very sparse input representations, the number of active feedforward inputs contributing to the signal can become too small for reliable retrieval. As discussed in previous work on sparse associative memories [74,75], very sparse morphological connectivity can therefore further reduce the robustness of sparse representations and lead to vanishing capacities. To address this point, we evaluated the model for a wide range of small morphological connectivity values . As expected, reduced connectivity decreases the achievable capacity and increases the optimal transition probability to compensate for the reduced number of available synapses. However, even for very small morphological connectivities and strict retrieval criteria ( close to 1), many patterns can be learned and successfully retrieved if the number of active input units M_in is sufficiently large — and the qualitative dependencies of the capacity and strength of plasticity on sparseness are maintained (S1 Fig and S2 Fig).

Choice of the activation threshold and optimized capacity

The central purpose of this work is to establish a relationship between the population sparseness of the patterns and the optimal strength of plasticity of the network to maximize its capacity. This requires quantifying the capacity, i.e., the lifetime of a pattern; therefore, the setup must allow for a meaningful comparison of the target and calculated output patterns. As a measure, we used the Hamming distance between the target and the calculated output patterns. This measure is most robust if the number of active output units M_out is fixed, which is implemented by choosing the activation threshold of the McCulloch-Pitts output neurons accordingly (Eq (3)). The maximum capacity calculated in this framework (Eq (23)) represents the highest capacity obtained under the given constraints using the optimal transition probability. In other approaches, the beneficial effect of sparseness on capacity has been extensively studied in different settings. For example, [34] could increase the capacity by using an optimized covariance-based learning rule. Moreover, [46] found that the capacity can be optimized by individually adjusting the threshold of each neuron, and [77] suggested the use of a numerically optimized activation threshold in a setup that is very similar to ours. A general model that unifies earlier Willshaw- and Hopfield-type networks provides optimal synaptic weights as well as an optimal firing threshold for each neuron [78]. In our manuscript, such aspects have been excluded in an attempt to keep the model as simple as possible to allow for a detailed analysis of the effect of sparseness on the strength of plasticity.

Capacity measures

A limitation of the present work is the use of the performance measure “pattern capacity”, which is defined as the number of subsequent associations that can be stored before losing retrievability of a pattern. As an alternative, network capacity is commonly also defined through information-theoretic measures that optimize resource use, such as synaptic capacity (see, e.g., [29,43,52,57]). For sparse patterns, it has been shown that parameters that yield optimal pattern capacity do not necessarily also optimize synaptic (information) capacity [47,52,57], implying an under-utilization of synaptic resources. In addition, outside the very sparse regime, when the number of active neurons per pattern becomes sufficiently large, optimizing pattern capacity may in principle overestimate the amount of synaptic memory required to store a given number of associations. In this case, the information content of the stored sparse patterns exceeds the minimal information necessary to distinguish between P memories, and alternative memory models could therefore encode lighter-weight representations using fewer synaptic resources. In our work, pattern capacity is employed as a functional measure of memory performance, focusing on the ability of the network to reliably retrieve stored patterns. It is not intended to quantify the information-theoretic optimality of synaptic storage. The results are thus best understood as characterizing the retrieval performance for different activation ratios and transition probabilities under the given homeostatic constraints of the network—and not as statements about how efficiently synapses are used in terms of information storage.

In S8 Appendix, we provide a translation of our main results into the information-theoretic measure of synaptic capacity. These preliminary results suggest that the decrease of the optimal transition probability with increasing activation ratios, which is the core result of the present work, is maintained for synaptic capacity. The synaptic capacity obtained with the optimal transition probability decreases with increasing input and output activation ratios. A detailed information-theoretic analysis of these results is beyond the scope of this study, but provides an interesting direction for future work.

Role of the output size and retrieval criterion

The analytical description of memory capacity relies on the distributions of dendritic sums, which are based on the synaptic connections of single output units (Eqs (10) and (11)). For the Hamming distance (Eq (14)) and thus also for the signal quality (Eq (7), also see Methods, Section ‘Signal quality after P subsequent patterns’), the number of output units N_out has to be taken into account because the probabilities of output units being in the wrong state are scaled by the respective number of units. We defined memory capacity as the number of subsequent patterns that can be learned until the signal quality reaches the retrieval threshold (Eq (8) and Eq (17)). The retrieval threshold , with a fixed , is chosen as a fraction of , which is the average Hamming distance between two random f_out-sparse patterns of size N_out (Eq (9)). For most figures in this manuscript, we chose as an exemplary retrieval ratio, where up to half of the units of a pattern are allowed to be wrong. However, network simulations as well as the analytical theory show that the main results, such as the monotonic decrease of the optimal transition probability as a function of the activation ratios and the non-monotonic dependence of the maximal capacity on the number of active input units, are robust with respect to changes in the retrieval threshold (see S1 Fig). The qualitative results hold also for much less permissive retrieval criteria with high values of that allow only for a small percentage of wrong units.

Due to the choice of the retrieval threshold as a multiple of H_avg, the number of wrong output units that are accepted for the pattern to be classified as retrievable depends on N_out and f_out. This dependence accounts for the fact that a fixed number of wrongly activated units has a more severe destructive effect if the total number of active output units is small. For example, if in a pattern with M_out = 100 active output units nine are wrong, the pattern is still very similar to the original one; if M_out = 10, nine wrong units are detrimental. The capacity is obtained by comparing the signal quality to the retrieval threshold or, equivalently, by comparing the normalized signal quality to the retrieval ratio . Since the normalized signal quality does not depend on N_out (derived in Methods, Section ‘Signal quality after P subsequent patterns’) but only on the distributions of the dendritic sums and f_out (see Eqs (59) and (61) as well as S3 Appendix), the capacity does not depend on N_out either.

As an alternative, which has not been explored in this manuscript, the retrieval threshold could be defined independently of H_avg and thus independently of N_out, e.g., implementing the retrieval criterion that a maximum of two units may be wrongly activated or that at least ten units must be correctly activated. In such alternative cases, the capacity could depend on the output network size N_out.

Furthermore, the output size as well as the input size in combination with the output and input activation ratios, respectively, pose an upper limit to the representational capacity, i.e., the maximal number of distinct patterns. If every input pattern and every output pattern should occur maximally once, the number of distinct input-output pattern pairs is bounded by .

In general, however, even if the size of the output layer is negligible for the storage capacity under specific circumstances, it plays an important role in a potential additional readout layer, e.g., a classification of the output representations. In [79], it has been shown that an increase in the number of output units increases the performance of a readout unit that performs a binary classification.

Strength of plasticity and related concepts

In the context of strength of plasticity, several distinct but related notions are used. Here, it seems useful to explain and compare the concepts of transition probability, weight step size, and learning rate, which could all be used to model the strength of synaptic plasticity. Apart from that, we make the distinction between the strength of plasticity and the speed of learning. The discussion of the speed of learning introduces the additional notions of number of trials and convergence time.

In this manuscript, we model the strength of plasticity by the transition probability, which refers to the likelihood that a synaptic weight increases during learning. Instead, a change in the strength of plasticity could also be realized by a change in the magnitude of synaptic weight updates (here called the weight step size). Weights could, for instance, be modeled to take on a number of discrete values. A high degree of plasticity could then be realized by a larger jump from a small weight value to a large weight value, while a low degree of plasticity would allow for updates from a small to an intermediate weight value. The transition probability and the weight step size are closely related as both affect changes in weights, and the average weight change in one learning step is proportional to the product of the two parameters. The qualitative results for the effect of sparseness on the optimal strength of plasticity of the present study are likely to also hold for models based on the weight step size or on a combination of both the transition probability and the weight step size. Note that while the assumption of only binary synaptic weights combined with a transition probability is a simplification of the state transitions that occur in real synapses, assuming a bounded number of discrete states is more realistic [80,81]. It therefore makes sense to assume that the strength of plasticity — e.g., in terms of the amount of potentiation experienced by synapses combined with a probability of potentiation — is bound to have an upper limit.

In this manuscript, we consider a time-discrete model of plasticity. In contrast, in time-continuous models, the strength of plasticity could be represented by the learning rate, which controls how fast the weights change. Again, we believe that a time-continuous model would yield qualitatively similar results to our time-discrete model relying on the transition probability if the number of connections in the network is large. It would generally be interesting to analyze more complex and more realistic synapse models (e.g., [38,40,71]) in this context, but all this is beyond the scope of the present work.

Furthermore, the strength of plasticity should be distinguished from the speed of learning. In our learning paradigm, the number of learning trials is kept constant and equal to one. Thus, each pattern pair is learned in one shot by changing a number of connections, which depends on the strength of plasticity. Although high plasticity and more rapid learning can often go hand in hand (also see next section ‘Sparseness and plasticity in the brain’), the speed of learning (one trial) is here fixed and thus not a parameter of investigation in this study.

Finally, we relate the so-called “convergence time” to the strength of plasticity, although the term “convergence” and its magnitude do not refer to how many or how much weights are altered, but it is a system-level outcome. Convergence refers to how quickly a neural network stabilizes or reaches its learning objective, e.g., a minimum in a gradient-descent learning algorithm. While very plastic synapses (e.g., in terms of transition probability, weight step size, or learning rate) can speed up appropriate weight updates, fast convergence is achieved by balancing these factors to ensure that the network efficiently settles into an optimal or stable solution. A high learning rate can accelerate updates towards a solution, but it does not necessarily relate to fast convergence. If the update steps are too large, they risk overshooting or instability. In [82] it is argued that sparse representations yield faster convergence because the learning rate can be increased without risking instabilities, but how this relates to the capacity of the network was not investigated. Furthermore, it is not clear how this translates to biologically more plausible local learning paradigms. In our model, convergence is not evaluated because every pattern is presented only once, and thus there is only one weight update step. It would be interesting to investigate the effect of sparseness on transition probability in combination with convergence speed in a network model that allows for both. Building on the results of this manuscript and the reasoning in [82], we believe that high sparseness would yield a high optimal transition probability (or learning rate) while converging fast because sparseness allows for large update steps without instability. A concrete possibility of introducing the concept of convergence into our model — when allowing repeated encoding of a pattern pair — is to analyze how many trials are necessary to perfectly imprint, in the sense that all connections from active input to active output units in the pattern are functional, the association into the network. For a transition probability of one, this is the case after one shot. For lower transition probabilities, it could take many more trials, i.e., the speed of learning would be slower.

Sparseness and plasticity in the brain

As already outlined in the Introduction, high population sparseness is prevalent throughout the brain [1–3,5,6,8,9,20]. Our work suggests that brain areas with higher sparseness could make use of more plastic synapses than brain areas with lower sparseness. The results of several experimental studies support this relation between increased sparseness and higher plasticity, which often goes together with more rapid learning.

The hippocampus, for example, is well known for its highly sparse representations and is also well known to be involved in rapid learning with high plasticity of synapses, serving episodic and spatial memory. A high turnover rate of dendritic spines as an indicator of high plasticity has been reported for the mouse hippocampus [83]. Studies using single-trial learning paradigms showed that the hippocampus can encode and store information after only a single exposure to an experience [84,85]. This is often called one-shot learning. Research with rodents [86–88] and humans [89–92] demonstrates that concept cells can develop very quickly (within seconds to minutes) and that hippocampal neurons can rapidly form and reorganize place fields or episodic representations during single learning episodes.

Compared to the hippocampus, the neocortex seems to have a lower population sparseness. According to the theory of complementary learning systems in the context of systems memory consolidation, the neocortex is assumed to be a slow learning system, which needs repetitive exposure to acquire a new memory, for example, in the form of repeated memory replay by the hippocampus during slow-wave sleep [93–95]. This feature is in line with the observation that plasticity in the neocortex is lower than in the hippocampus [96,97]. An artificial increase in plasticity in the neocortex has been reported to disrupt memory processes [98]. However, engrams can also form rapidly in the neocortex, either in parallel with engrams in the hippocampus or even without hippocampal support [84,99,100]. Fast memory formation was observed in the neocortex in rodents, for example, in combination with unusually high plasticity in neocortical regions for memories related to prior knowledge [101,102] and in humans during the learning of new words [103]. A recent analysis of a large set of fMRI data suggests that under specific conditions the human parahippocampal cortex (PHC), a part of the neocortex, can exhibit even higher plasticity than the hippocampus [104].

In light of these studies and our theoretical considerations, a deeper experimental investigation of the causal or functional relationship between sparseness and the strength of plasticity would be of great interest for the neuroscience community.

Optimal sparseness values

In terms of the capacity obtained with an optimized transition probability, our study predicts an optimal small but not too small input activation ratio f_in and an optimal output activation ratio (Fig 3). The optimal f_in depends on the input size N_in while the optimal number of active input units M_in does not (Fig 5). Nevertheless, the optimal M_in cannot be interpreted as a quantitative prediction of the optimal number of active units because it depends on the retrieval threshold (see Methods, Section ‘Dependence on input parameters’). If is increased, i.e., if the retrieval criterion becomes more strict, the optimal M_in increases (and the corresponding maximum capacity decreases).

In addition, the quantification of the optimal strength of plasticity in our model is focused solely on optimizing the pattern capacity of the network. In reality, the brain might favor a trade-off of various properties instead of optimization for one purpose [105]. Although sparseness can be advantageous, e.g., in terms of energy efficiency [106], storage capacity [107], and downstream classification [79], it might be beneficial to avoid extreme sparseness because it can reduce the representational capacity [105,108], the information capacity per pattern [109,110], the generalization capacity [105,111]; and extreme sparseness is also not robust to noise or damage of neuronal structures [108, 105]. Moreover, sparse connectivity, which is also ubiquitous in the brain, seems to yield similar favorable effects as population sparseness and could be optimized in combination with sparse activity [105,112].

In general, it is therefore not reasonable to predict one optimal activation ratio or one optimal number of active neurons. Optimal (and actual) values are probably different for different brain regions, cell types, tasks, etc. First steps towards more concrete optimal sparseness values could combine experimentally measured values, obtained with modern sophisticated techniques such as Neuropixels recordings, and computational models that take into account the specific properties and presumed role(s) of particular brain regions.

Probabilistic description of dendritic sums

To better understand the learning dynamics of the network and the decay of the quality of the memory signal, which is used to quantify the capacity, we derive a probabilistic description of the dendritic sums of output units (see also, e.g., [45,46,49–51]). The dendritic sum of the j-th output unit is its net input

(33)

In accordance with [45], we introduce the following nomenclature: In general, a calculated output pattern differs from the target output pattern . We call output units that are active (1) in the target output genuine output units and output units that are inactive (0) in the target output spurious units. In the calculated output, on the one hand, genuine units can be correctly active or incorrectly inactive. On the other hand, spurious units can be correctly inactive or incorrectly active. The terms genuine and spurious do not indicate whether the calculated activity of an output unit is 1 or 0, respectively, but whether it should be 1 or 0, respectively, to match the target output. In this section, we derive the distributions of the dendritic sums of genuine and spurious output units separately, and these dendritic sums are denoted by symbols (spurious) and (genuine). The shapes of these two distributions (especially their overlap) is related to the Hamming distance between target and calculated output and consequently also to the signal quality. These relationships will become clear later in the Methods, i.e., in section ‘Analytical calculation of the capacity of the network’.

With respect to a particular pattern pair, we further distinguish between four types of connections:

• genuine-genuine (g-g) connections connect active input to active output units,
• genuine-spurious (g-s) connections connect active input to inactive output units,
• spurious-genuine (s-g) connections connect inactive input to active output units, and
• spurious-spurious (s-s) connections connect inactive input to inactive output units.

Note that the neuronal activity that determines the connection type is here understood as the activity in the original pattern pair and not in the calculated output pattern .

Distributions of dendritic sums

In what follows, we follow the work of [29,43,46] and calculate the distribution of the net input to an output unit j where denotes a spurious (s) or a genuine (g) output unit. This input is also called the dendritic sum of output unit j. The term is the j-th line of the weight matrix obtained for P additional patterns after learning pattern k. To understand how the P subsequent patterns change the weight matrix, we have to take into account how often the output unit j is active across all P patterns, which is also called the output unit usage. As suggested by [45], we assume that the number r of times an output unit is active in P statistically independent patterns follows a binomial distribution , thus

(34)

This describes the output unit usage more accurately than the assumption that each output unit is active exactly times across a pattern set of size P, as assumed in the original paper on the Willshaw network [29] (also see [61]).

The dendritic sum of the output unit j gets contributions only from connections that originate at one of the M_in input units that are active in the k-th pattern. Independently of the state of a connection (functional or silent), connections from inactive input units do not contribute to the net input to an output unit. We define as the probability that a genuine-spurious connection to the output unit j is functional given that the output unit j is active u times in the pattern set. Similarly, the probability that a genuine-genuine connection is functional is called . The functionality of a connection implicitly takes into account that the connection is available morphologically. We emphasize that , for , depends on whether the respective output unit is genuine (), i.e., active in the target output, or spurious (), i.e., inactive in the target output. The derivation of and is discussed in the S1 Appendix. Moreover, the probability that a spurious-spurious or a spurious-genuine connection is functional is less interesting because these connections do not contribute to dendritic sums due to lack of input. However, these probabilities must be considered implicitly in S1 Appendix to be able to determine and . Further, the probability of a functional spurious-genuine connection, called , is important if the input pattern used for retrieval is noisy. This case will be discussed in Section ‘Noisy input patterns during retrieval’.

For a particular output unit usage u and statistically independent output units, the dendritic sums are binomially distributed with

(35)

for . In the extreme case where all connections from input units that are active in the k-th pattern to an output unit are functional, its dendritic sum equals the number of active input units M_in. Combining Eq (34) and Eq (35), we obtain

(36)

for , as the distribution of the dendritic sums . This probability is described here for the sake of completeness, but it is the same as derived in [46]. However, the probabilities of functional connections in our work differ from those considered in [46] (see S1 Appendix).

In what follows, we abbreviate

(37)

and

(38)

Before including the probability of a single genuine-genuine or a single genuine-spurious connection being functional after storing a set of additional patterns in which the output unit was active u times (derived in S1 Appendix), we review the learning rule from a probabilistic perspective and determine important fractions of connections.

Hebbian and homeostatic update rules from a probabilistic perspective

We assume a morphological connectivity of connections per output unit and start from a random choice of functional connections targeting each output unit. In each learning step k, only connections targeting output units that are active in the current pattern, i.e., genuine-genuine and spurious-genuine connections, are updated. We further can distinguish between four types of connections: They can be either silent or functional before learning the k-th pattern and they can either originate from an active or an inactive input unit in the k-th pattern. Two types of connections are not updated: If the connection is already functional and the corresponding input unit is active (in addition to the corresponding output unit being active), the connection is protected and remains functional. If the connection is previously silent and the input unit is inactive, the connection always remains silent. The other two types can transition between the two states (functional/silent): Genuine-genuine connections (a fraction f_in of all connections to one output unit) of the k-th pattern that are previously silent (a fraction ) are made functional with a transition probability , i.e., the fraction of all connections becomes functional in one learning step.

For each genuine output unit in the k-th pattern, the same number of connections that were made functional is also silenced such that there is a constant fraction c of functional connections per output unit after Hebbian learning and homeostatic normalization. Only spurious-genuine connections of the k-th pattern (fraction ) that are previously functional (fraction c) could be silenced due to the homeostasis mechanism. This is a total fraction of all connections. The probability that such a connection is silenced depends on the number of additional connections that became functional due to the Hebbian mechanism in the same learning step. As discussed above, a fraction becomes functional in each step. The probability for a previously functional connection from an inactive input unit to be silenced due to homeostasis is

(39)

Normalization is possible in this way if this probability is less than or equal to one, which is always fulfilled if we assume that (regardless of the values of and ).

In summary, the numerator in Eq 39 is the fraction of connections that are made functional in this step and the denominator is the fraction of connections available to be silenced; together, this is the probability to be newly silenced. Since the fraction of connections made functional, , equals the fraction of connections newly silenced, , the overall fraction c of functional connections is maintained. Since patterns are uncorrelated, these probabilities remain the same for every pattern that is learned, and they will be used several times in the derivations in the following subsections.

Probabilities of functional connections

As derived in detail in S1 Appendix, the probability for a connection to a spurious unit to be functional is

(40)

It does not change with the number of times u the output unit was active in other patterns. The probability that a connection to a genuine unit is functional, however, does strongly depend on u. It is given by

(41)

Thus, we can fully characterize the distributions of dendritic sums for spurious and genuine output units. We insert the probability of a functional connection to a spurious unit into Eq (36), define , and obtain the probability mass function (PMF) of the dendritic sum of a spurious output unit:

(42)

(43)

(44)

Since is constant as a function of the output unit usage u, the distribution of dendritic sums for spurious output units simplifies to a single binomial distribution with parameters n = M_in and p = c, and does not depend on the number of subsequently learned patterns P.

Analogously, we insert into Eq (36) and obtain the PMF of the dendritic sum of a genuine output unit

(45)

(46)

Thus, depends on the number of subsequent patterns P (in contrast to ). It is not a single binomial but a linear combination of binomial PMFs , for , see Fig 7. This multimodality is analyzed in more detail in S2 Appendix (also see [61] for a comparison to a binomial distribution). The relation between and and their dependence on the activation ratios f_in and f_out is discussed in the Results, Section ‘Activation ratios in the distributions of dendritic sums’. S5 Fig shows a comparison of Eqs (44) and (46) to distributions of dendritic sums obtained from numerical simulations.

Let us finally discuss some limiting cases. We have

(47)

because , , 0 < f_in, and . Note that the constraint stems from the normalization condition as discussed previously in Section ‘Hebbian and homeostatic update rules from a probabilistic perspective’. For the term in Eq (41), it thus follows that we either have

(48)

which implies

(49)

or, for , , and f_in = c,

(50)

which implies

(51)

(52)

In the second case, the part of the memory stored in the connections targeting one output unit is completely lost as soon as this output unit is active one additional time for another pattern.

Noisy input patterns during retrieval

So far we always assumed that during recall the original input pattern is shown to the network. Due to the inherent presence of noise in biological systems, it makes sense to assume that the input pattern presented to the network during retrieval is not exactly the same as the input pattern used for training. In the following, we discuss the consequences on the previous derivations when a noisy cue is presented to the network by putting noise onto the input pattern (cf. [46,47]). To be able to compare the results in a fair way, the input activation ratio f_in should not be changed, hence a fraction of the input units that are active in the original pattern (genuine input units) is deactivated while the same absolute number of input units that are inactive in the original pattern (spurious input units) is activated. Thus there is a number of original input units (genuine input units) and a number of wrong input units (spurious input units) active, in total we have active units (see [45,46]). The parameter is restricted to values that yield . Note that, in the numerical simulations, we set each of the genuine input units to zero with probability , count how many changed and flip the same amount of spurious input units to one. This means and are not fixed exact numbers but follow distributions. Only on average, genuine units are deactivated. The additional variability coming from deactivating each genuine input unit with probability would make the analytical description more involved while hardly affecting the expected result for large numbers. It is therefore neglected in the theory derived in this section. Instead, we assume that, for generating a noisy cue, exactly of the genuine input units are deactivated and of the spurious input units are activated. Note that, nevertheless, in both numerical simulations and theory, it is always ensured that the absolute number of active units M_in in the noisy input pattern presented to the network does not change.

Noisy cues do not alter the distribution of dendritic sums of spurious output units, but they do alter the distribution of dendritic sums of genuine output units. Regarding the latter, the probability of a functional genuine-genuine connection is the same as before, that is, . But we now need to consider, in addition, the probability of a functional spurious-genuine connection, which is derived in S1 Appendix as

(53)

Similarly to , also depends on the number of times u an output unit has been active across the pattern set. The probability of a functional spurious-genuine connection is lower than the probability of a functional genuine-genuine connection (and even lower than the functional connectivity c, see Figures in S1 Appendix).

For a particular output unit usage u, the dendritic sums are distributed following

(54)

where . In total, the probability of the dendritic sum of a genuine output unit to have a particular value x can be described by

(55)

(56)

as derived in [45] but, as an extension, with the probabilities of functional connections and derived in S1 Appendix that include the transition probability and the homeostatic constraint of a fixed in-degree. Compared to the distribution of the dendritic sums for the noise-less input pattern, this distribution is shifted to the left. Thus, it is closer to the distribution of the dendritic sums of the spurious units, the Hamming distance between the target output and the calculated output pattern is larger, and thus the capacity decreases (compare Fig 8 and S6 Fig to Fig 7). The distribution of dendritic sums in Eq (56) will be used in Section ‘Extension to noisy input patterns during retrieval’ to derive the capacity for retrieval with noisy cues.

Analytical calculation of the capacity of the network

In what follows, our aim is to analytically quantify the capacity of the network. The capacity is defined as the number of subsequent patterns P for which the signal quality has declined to the retrieval threshold. Therefore, we first discuss the signal quality as a function of P. Before we enter the details, we briefly recap basic quantities: The difference between a calculated output pattern given a particular input pattern after learning P subsequent pattern pairs and the corresponding target output pattern is measured in terms of the Hamming distance between the two, denoted by . This Hamming distance is always averaged across a large number of patterns, and it lies between 0 and the average Hamming distance between two random patterns of length N_out and activation ratio f_out, which is

(57)

We define the memory signal quality at P patterns as the difference between and the Hamming distance between the target output and the calculated output :

(58)

see Fig 9, green curve. The signal quality can be expressed as a fraction of H_avg

(59)

We call the normalized signal quality. With , the Hamming distance reads as

(60)

(see Fig 9, salmon line), where we introduce the additional symbol

(61)

out of convenience for the derivations in this section.

Analytical Hamming distance

In this section, we derive the expected Hamming distance between the target output and the output calculated after learning P additional patterns. We follow a similar approach as several earlier works (e.g., [45–47,51,52,61]): The distributions of dendritic sums allow us to describe the error in retrieval (expressed by the Hamming distance) via the probabilities of wrongly active and wrongly inactive units. However, the constraint of a fixed output sparseness adds a layer of complexity to our derivations (see the following Subsection ‘The activation threshold and balancing tails of the distributions’).

The activation threshold and balancing tails of the distributions.

Since the Hamming distance depends on the activation threshold , we first discuss the choice of , which is self-consistently defined such that the correct number of output units, M_out, is activated:

(62)

The activation of output unit j is calculated as

(63)

where is the Heaviside step function. The number of active output units is hence

(64)

Since the weight matrix J^[k+P] is updated in every learning step and hence depends on the number of subsequently learned patterns P, the activation threshold also depends on P. It thus has to be newly determined in every update step in order to obtain the given number of active output units M_out.

The active output units can either be genuine, hence active in the target output pattern, or spurious, hence inactive in the target output pattern. The dendritic sum

(65)

of a genuine output unit j follows the genuine distribution (Eq (46)). A genuine output unit is activated if its dendritic sum is at least as large as the activation threshold , hence if

(66)

(blue area in Fig 10A). There is a total amount of M_out genuine output units. Thus, if we assume , the number of genuine output units that are activated for a given threshold can be approximated by using the law of large numbers as

(67)(68)

where G is the set of indices of genuine output units. Analogously, assuming , the number of spurious units that are activated can be approximated as

(69)(70)

where denotes the dendritic sum of the j-th output unit, which is spurious, and S is the set of indices of spurious output units (cf. red area in Fig 10A). In order to activate the right amount of output units, the activation threshold thus has to be chosen such that

(71)

see Fig 10B. By basic computations, we have that

(72)(73)

where and are the cumulative distribution functions (CDF) of and , respectively (Fig 10C). We call the last equation the Balance Equation because it balances the false positive and the false negative errors.

With an increasing number of patterns P, the center of mass of the genuine distribution moves to the left, while the spurious distribution stays constant. For a fixed T_in, the left-hand side of Eq (73) would hence grow while the right-hand side would not, which would lead to an output activation ratio that is smaller than f_out. In order to keep the Balance Equation fulfilled, the activation threshold has to move to the left for increasing P; see Fig 10C for P = 68 and Fig 10D for P = 114.

Deriving the Hamming distance from dendritic sums.

The Hamming distance

(74)

is the sum of the number of genuine units that are wrongly set to 0 and the number of spurious units that are wrongly set to 1 in the calculated output :

(75)

In terms of the distributions and , the expected Hamming distance reads as

(76)(77)

see Fig 11A. The Hamming distance is hence the sum of the part of the genuine distribution that lies below the activation threshold and the part of the spurious distribution that lies above — weighted by the corresponding number of units. Using the Balance Equation (73), we can express in terms of , and the expected Hamming distance simplifies to

(78)

The dependence on the number of patterns P is now solely contained in the activation threshold because is constant in P. The Hamming distance increases with decreasing and, as discussed in the previous Section ‘The activation threshold and balancing tails of the distributions’, decreases with increasing P. Thus, the larger the number of subsequently learned patterns P, the larger the Hamming distance between target output and calculated output and, according to Eq (58), the smaller the signal quality .

Note that the dependence of the Hamming distance on the output activation ratio f_out is less straightforward. On the one hand, if f_out is increased, the factor in Eq (78) yields a decrease in the Hamming distance. On the other hand, an increase of f_out also decreases the activation threshold and thus increases the factor in Eq (78), which yields an increase in the Hamming distance (see Fig 11B). It is not easy to determine analytically which of these opposing effects is stronger. As mentioned in the Results in Section ‘Activation ratios in the distributions of dendritic sums’, we speculate that the combined effect on the Hamming distance is small in the parameter range that we investigated numerically.

Signal quality after P subsequent patterns

In order to derive the signal quality (see Eqs (58) and (59)) after P additional learning steps, we combine Eqs (60) and (78) and obtain

(79)

Inserting yields

(80)

(81)

Using Eq (80), we find that the Balance Equation (73) is fulfilled if and only if

(82)

(83)

Fig 12 illustrates the two equations. We can equate the right-hand sides of Eqs (81) and (83) to get rid of the explicit value of and obtain a single equation

(84)

which we call the Signal Quality Equation. If this equation is solved for , inserting its solution into Eq (59) with determines the signal quality . There is an explicit solution if and are easy to calculate.

Notably, since the CDF of a discrete probability distribution is a piecewise constant function, the CDFs in Eqs (81) and (83) are not generally invertible. A strict inverse such that does not always exist because the same CDF value F(x) can correspond to multiple arguments x. Being aware of this fact, we use the notation and here anyway. Later in this section, the binomial distributions (see Eqs (44) and (46)) are approximated by normal distributions (also see S4 Appendix), and their approximated CDFs become strictly increasing and continuous, and thus invertible.

Details regarding an approximate analytical solution and a numerical solution of the Signal Quality Equation (84) are provided in S3 Appendix.

Memory capacity of the network

Instead of calculating the signal quality at any P subsequent patterns (see S3 Appendix), which can then be used for finding the P for which the signal quality reaches the retrieval threshold, the network capacity can be determined by directly equating the signal quality to the retrieval threshold (expressed in terms of ) and solving the resulting equation for P.

We are searching for the number of patterns P for which the signal quality as defined in Eq (58) equals the retrieval threshold , which can be expressed as a fraction of H_avg:

(85)

with . In analogy to , we define for convenience. Instead of calculating for a given P, we now use the Signal Quality Equation (84) to find the number of patterns P for which (which is equivalent to ) and is given. We thus want to solve the Capacity Equation

(86)

for P, which is the only unknown in this equation for a fixed set of network parameters. Notably, the Signal Quality Equation (84) and the Capacity Equation (86) are of the same form but, in the first, the number of patterns P is given and the fraction is unknown, which determines the signal quality while, in the second, the retrieval ratio is given and the number of patterns P is unknown, which corresponds to the capacity.

In order to solve the Capacity Equation (86), we first approximate the linear combination of binomial PMFs in by a single binomial PMF and we then approximate this binomial distribution as well as by normal distributions (see S4 Appendix for details). Regarding the first approximation step, [61] provides conditions for when this type of distribution converges to a binomial distribution and finds that a binomial distribution is a good approximation for a wide range of parameters, even though it can slightly overestimate storage capacity due to an underestimation of retrieval errors. In S4 Appendix, we present an empirical analysis of the quality of this approximation in dependence of the parameters of the model used here. For the second approximation step, the two assumptions

(87)

need to be respected (more details in S4 Appendix). We define the function

(88)

and introduce the abbreviations

(89)

and

(90)

Then the mentioned approximation steps allow us to express Eq (86) as

(91)

Solving Eq (91) for P (for details see S4 Appendix) yields the capacity

(92)

where

(93)

(94)

If and are known, this expression allows us to directly calculate the capacity of the network. The approximation of the error function used for calculating and can be found in S4 Appendix.

Optimal transition probability and maximal capacity

From Eq (92), we can derive the optimal transition probability and therewith the maximal capacity for a fixed set of network parameters.

We assume and use the Taylor series of at x = 0 to approximate

(95)

The capacity as a function of then reads as

(96)

with A and B as defined in Eqs (93) and (94) and

(97)

In the following, we will ignore the rounding of to an integer value in (96) and allow the capacity to take any (non-negative) real value. We then have

(98)

as also stated in the Results in Eq (21). Next, we calculate the optimal transition probability by differentiating with respect to

(99)

and solving

(100)

(101)

This is a global maximum of because

(102)

The optimal transition probability must be bounded to (0,1]. We can assume that (otherwise there is no real solution for (92)). If , we have . If , the capacity as a function of the transition probability is monotonically increasing in the range that we are interested in because we have shown that has a global maximum at . We thus have

(103)

as also displayed in Eq (22) in the Results. Fig 13 shows as a function of the input and output activation ratios f_in and f_out.

The maximal capacity that can be obtained for a given set of network parameters by optimizing the transition probability is then defined as

(104)

Comparisons of the analytical approximation of the optimal transition probability (Eq (103)) and the maximal capacity (Eq (104)) with results obtained from numerical network simulations (cf. Section ‘Network simulations’) are presented in Figs 3B, 3C, 5.

Dependence on input parameters.

Since in Eq (104) only depends on N_in while A and B do not (see also Eqs (93)–(97)), the capacity is proportional to the input layer size N_in (Figs 5B, 14 bottom row). Further, we note that the optimal transition probability in Eq (103) does not depend on the input layer size N_in, but it does depend on the number of active input units (Figs 5A, 14 top row).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 13. Analytical optimal transition probability.

Optimal transition probability (Eq (103)), color coded, as a function of f_in and f_out. The probability decreases both as a function of f_in and as a function of f_out but the slope is larger for f_in. For comparison, in Fig 3E in the Results, the corresponding optimal transition probability obtained from numerical simulations is shown. (Other parameters: .).

https://doi.org/10.1371/journal.pcbi.1013235.g013

Download:

• PNG
  larger image
• TIFF
  original image

Fig 14. Optimal transition probability and maximal capacity for various N_in.

Dependence of optimal transition probability (top) and maximal capacity (bottom) on number of active input units () for various input layer sizes (). Dots: numerical simulations; solid lines: optimal obtained from Eq (103) and maximal capacity calculated with Eq (98); dashed lines: maximal capacity calculated with Eq (92), i.e., without the Taylor approximation (95). In (A), f_out = 0.1, in (B), f_out = 0.025 and in (C), f_out = 0.006. Other parameters: .

https://doi.org/10.1371/journal.pcbi.1013235.g014

We further observe that the maximal capacity as a function of the number of active input units M_in has a maximum that is independent of N_in. In the range where , the maximal capacity as a function of M_in

(105)

is strictly monotonically decreasing: For , we have

(106)(107)(108)

where we used that because A is monotonically increasing as a function of M_in (due to for f_out<0.5) and that A has to be positive in order to yield a real valued number for the capacity in Eq (92). Thus, if there is a maximum of , it must occur within the range of M_in values that have an optimal transition probability of 1 (see Fig 14A,14B,14C). In order to find this maximum, we set in Eq (98), and then we set the derivative (with respect to M_in) of the maximal capacity

(109)

to zero:

(110)

This equation can be solved numerically for fixed values of , and . For example, for and , we obtain an optimal number of active input units M_in = 13, which yields a maximal capacity (see Fig 14C).

The fit between the theoretical approximation and the actual capacity obtained in numerical simulations improves for increasing input layer size N_in, increasing input activation ratio f_in, and increasing output activation ratio f_out (see comparison in Table 2). As discussed before, we need and in order to approximate the distributions of the dendritic sums by Gaussian distributions. A large f_out reduces the multimodality of the genuine distribution and hence also allows for a better approximation (see Discussion in S4 Appendix).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Deviation between theory and simulations: Maximal capacity obtained from theory divided by maximal capacity obtained from simulations for various values of N_in and f_out averaged across all M_in values shown in Fig 14 except the two smallest M_in = 3 and M_in = 6 (which does not fulfill the condition and for which the fit is thus not accurate enough). Deviation calculated based on Eq (92) (corresponding to dashed lines in Fig 14). In brackets: Deviation calculated based on (98), i.e., with the Taylor approximation (95) (corresponding to solid lines in Fig 14).

https://doi.org/10.1371/journal.pcbi.1013235.t002

Extension to noisy input patterns during retrieval

An extension of the derivation of the memory capacity to the case of noisy input patterns during retrieval with noise level (see the earlier section ‘Noisy input patterns during retrieval’ for details on the distributions of dendritic sums with noisy cues) yields the following capacity (see S4 Appendix for details on the derivation):

(111)

with

(112)

and

(113)

The optimal transition probability and hence the maximal capacity of the network can be calculated in the same way as without noise (compare to Section ‘Optimal transition probability and maximal capacity’). The only difference lies in the expressions and that were previously called A and B. Expression remains the same as in Eq (97). We have

(114)

and

(115)

as also stated in Eqs (24) and (27) in the Results (Section ‘Noisy input patterns during retrieval’). In the range where , we thus can express the maximal capacity as

(116)

We can assume that , where is the maximal capacity for defined in Eq (98), because the additional noise on the input patterns during retrieval naturally has a detrimental effect on the capacity. In the range of parameters where (see Eq (105)) and , we thus have

(117)

since C > 0 and we can assume A/B > 0 and in order to obtain a real-valued capacity.

The optimal transition probability with noise during retrieval can be further approximated (details in S4 Appendix) as a multiple of the optimal transition probability without noise by

(118)

and the maximal capacity by

(119)

For and , we thus obtain the approximation

(120)

A comparison of and and its approximation given in Eq (118) as well as of and and its approximation given in Eq (120) is shown in Fig 6D,6E (also see S4 Fig).

Network simulations

In this section, we describe the network simulations performed to obtain the numerical results presented in this paper. First, we report how the main parameters were sampled. Then, we explain the simulation paradigm in detail. Last, we clarify how we average across results for different patterns before briefly broaching the issue of enforcing the output activation ratio.

All simulations were performed in python 3.9. All code written in support of this publication will be made publicly available at https://itbgit.biologie.hu-berlin.de/auer/sparseness_plasticity_lifetime before publication.

Parameter sampling

Unless stated otherwise, we perform all simulations for transition probabilities

(121)

and define the value for which the capacity is the largest as the optimal transition probability. For the reference network size of , the activation ratios are typically sampled as

(122)

For larger network sizes, smaller values f_in and f_out are added to achieve minimal M_in and M_out values comparable to those of the small reference network.

The simulation paradigm

We now describe how patterns are sampled and how the weight matrix is initialized as well as how a simulation step is executed and how many steps are carried out in total. As also explained in the Results, Section ‘Network model, learning paradigm, and quantification of capacity’, we aim at quantifying the memory capacity of the network by tracking the signal quality, which depends on the Hamming distance between a calculated output vector and the corresponding target output vector. The capacity is defined as the number of subsequently learned patterns for which the signal quality reaches a specific value. In each simulation step the association between one pattern pair is learned, and in order to calculate the capacity we need a number of simulation steps that is larger than the capacity itself.

One simulation step.

Each simulation step k corresponds to one pattern pair being learned. In one simulation step k, the weights of the matrix in the previous step, , are first updated according to Hebbian learning and then updated according to the homeostatic mechanism (described in the Results, Section ‘Network model, learning paradigm, and quantification of capacity’). This corresponds to transforming into J^[k].

Then, for a fixed J^[k], we evaluate the output of the network for all input patterns of pairs that have been learned up to this point; the input patterns are presented to the network again one after the other with plasticity off, and the corresponding output patterns are calculated. By we denote the output of the i-th input pattern calculated after learning j subsequent pattern pairs. To summarize, the outputs

(123)

for each of the input patterns

(124)

are calculated and compared to the original target outputs

(125)

respectively. The k + 1 Hamming distances

(126)

between target and calculated output are stored. Note that these Hamming distances that are stored after one particular simulation step belong to different numbers of additional pattern pairs that are stored after the pattern that is tested. With respect to pattern 0, the maximum number k of additional pattern pairs have been stored, while with respect to pattern k, no additional pattern pair has been stored.

Number of simulation steps.

The average Hamming distance between two random f_out-sparse vectors of length N_out is

(127)

To define a threshold for the successful retrieval of an output pattern, we use a certain fraction of ,

(128)

where . If not stated otherwise, the retrieval threshold is chosen as

(129)

and hence, .

The degradation of the calculated output vector is quantified by means of the signal quality S, which is defined as the difference between the average Hamming distance of two random vectors, , and :

(130)

For a successful retrieval of the calculated output, we then require . This translates to the assumption that an output pattern can be recovered if the Hamming distance between the target output y and the calculated output is smaller than the threshold . In this case, the output, which is typically degraded by subsequent storage of other input-output pairs, is still close enough to the original output pattern. To be more specific, let us consider the example of input-pattern pair 0 and k subsequent learning steps. The signal quality is then

(131)

which is based on the Hamming distance of the target output of the very first pattern pair and its output calculated with weight matrix J^[k].

The maximum number of simulation steps for a particular parameter combination was determined by the observed degradation of calculated output patterns. Once the signal quality for the first time is less than (or equivalently, when the Hamming distance is above ), 200 additional patterns are learned before the simulation is stopped.

We denote the total number of patterns that have been learned by the network at the end of a simulation by K.

Averaging

The signal quality that is shown in various figures in this manuscript is always averaged across many patterns. Recall that the number of subsequently learned patterns P is understood as relative to the pattern number . For each pattern that the network learns, we compute the Hamming distance of the target output and the calculated output after patterns additionally learned after the k-th pattern, hence with weight matrix J^[k+P]. For a fixed P (always relative to a particular pattern), we average across all patterns for which we have learned at least P subsequent patterns:

(132)

All simulation data shown in this manuscript are obtained by averaging across at least N_avg = 200 patterns. This means, if the largest P shown in a figure is , simulations ran for at least steps.

Whenever we investigate the distributions of dendritic sums for a particular number of subsequent patterns P, two kinds of averaging are done: First, we pool the dendritic sums of all genuine output units and of all spurious output units of one pattern after learning P subsequent patterns, respectively. In addition, we pool a certain number of patterns. A comparison of distributions of dendritic sums obtained from numerical simulations and from theory (Eqs (44) and (46)) is shown in S5 Fig.

A note on the output sparseness

During retrieval, the activation of the output units is calculated as

(133)

where is the Heaviside step function. Note that, due to the discrete nature of the dendritic sums of the output units, it is unlikely that an activation threshold exists that allows to activate exactly M_out output units. Most of the times, there is a group of units with identical dendritic sums that are exactly at the critical value. If the threshold is chosen larger than these dendritic sums, too few units are activated, if it is chosen smaller, too many units are activated. Since we want to keep M_out fixed, we have to find a way to enforce the exact output sparseness. Hence, we typically randomly choose the right number of these units to be activated and the rest to be inactive. On average, this has a slightly deteriorative effect on the capacity of the network because spurious units might be chosen to be activated even though some genuine units have the same dendritic sum. In S7 Appendix, we discuss in more detail for which parameters this might have significant effects on the capacity of the network and how the enforcement of the output sparseness level could be handled differently in order to avoid this problem.