Fig 1.
Exemplary development of the response (C), synaptic weights (D), and annealing characteristic (D, red) for a neuron with two inputs (A,B) with mean amplitudes of 1 and 1.2, respectively, and same average occurrence frequency; dashed lines show input coincidences. Inset shows the finally resulting distribution of neural responses between an activation of zero (left) and one (right).
Fig 2.
Functions used in model equations.
A: Neural activation, see Eq (2); we use a saturating function, where in case of small membrane potential (y) the activation v is zero, which is based on empirical observations, e.g., see. [20, 21]; B: Annealing function, which renders a close to zero annealing rate until threshold value va is reached, and afterwards increases abruptly, see Eqs (4) and (5); note that this function is additionally scaled by the annealing rate ρ in Eq (4).
Fig 3.
Schematic diagram of the recurrent network with neurons responsive to different input combinations indicated.
“x” means input can be 0 or 1.
Fig 4.
A: Separation properties calculated analytically B: Histograms of numerical results for ALL- and AMH-rules in case of Gaussian distribution of input amplitudes. In all cases, input presentation frequencies are equal. In B input coincidence is 10% everywhere; input amplitudes: mean for the blue distributions was normalized to 1.0 and for the orange ones to ϕ; standard deviations indicated above the plots. Annealing parameters are va = 0.7, ρ = 0.2. Initial weights are ω(0) = [0.001, 0.001]T and initial learning rate μ0 = 0.0005; Euler integration with step dt = 1. Disks in A mark the points with the corresponding plots in B. Tilted lines are truncation marks for the blue histograms.
Fig 5.
Histograms of neuron inputs (first column) and outputs v for the ALL-rule.
A: Equal presentation frequency; B: Different presentation frequency. Parameters: va = 0.7, ρ = 0.1, std = 0.1. Mean amplitudes of the inputs are indicated in the first column. Initial weights are ω(0) = [0.001, 0.001]T and initial learning rate is μ0 = 0.0005; Euler integration with step dt = 1. For other parameters: see plots. Response histograms (blue or yellow) in case of amplitude or presentation frequency difference are grouping very close to zero, where we truncate the zero bin to optimize for visibility (see truncation marks).
Fig 6.
Classification error (coincidence vs. not coincidence) of the ALL-rule in respect to parameter variations.
Parameters are annealing onset threshold and annealing rate. Decision threshold is 0.5. Panels (A-L) variable amplitude, coincidence and presentation frequency; panels (M-P) extreme cases: bigger variance, smaller coincidence, bigger amplitude difference, bigger frequency difference. Averages over 20 trials are shown. Initial weights are ω(0) = [0.001, 0.001]T and initial learning rate μ0 = 0.0005; Euler integration with step dt = 1.
Fig 7.
Comparison to reference methods: Results for BCM, Oja and Synaptic Scaling.
Two inputs with coincidence 30% everywhere. Amplitudes and standard deviation (std) are shown above each column. Presentation frequency is equal, except in the last column where it is 2:1. Parameters: μ = 0.001. For Oja and Syn.Scaling: ω(0) = [0.001, 0.001]T, for BCM: ω(0) = [0.2, 0.2]T, ΘM(0) = 0.2, γ = 10 and v0 = 0.2; Synaptic Scaling: y0 = −200, ξ = 0.01.
Fig 8.
Three input coincidence sorting for ALL and BCM rules.
A: Output histograms. Note that 3 examples for BCM are shown using the same intrinsic parameters but different stimulus sequencing. B: Weight development. Note the different x-axis scales. C: Parameter space analysis: Errors for classification “one active input”, “two active inputs”, “three active inputs” are based on response thresholds 0.25 and 0.75, averages over 20 trials are shown. Light color corresponds to good coincidence sorting. Circles in the error plots show parameter combinations for which histograms are shown in panel (A). Parameters: mean amplitude is 1 in case the input is active, STD = 0.1, ω(0) = [0.2, 0.2, 0.2]T, μ = 0.001, pair-coincidence 30% for every possible combination (12, 13, 23) in respect to that pair, triple co-incidence for 123: 6%; for BCM: ΘM(0) = 0.1; Euler integration with dt = 1 in all cases.
Fig 9.
Input coincidence sorting properties under more variable conditions.
A: Results for the ALL rule for the three input case with amplitude variation. Errors for classification “one active input”, “two active inputs”, “three active inputs” are based on thresholds 0.25 and 0.75. Light color corresponds to good coincidence sorting. Parameters: average amplitude provided above the plots, STD = 0.1, ω(0) = [0.2, 0.2, 0.2]T, μ(0) = 0.001, pair-coincidence 30% for every possible combination (12, 13, 23) in respect to that pair, triple co-incidence for 123: 6%; plots show averages over 20 trials. Histograms of individual runs below correspond to the two circles in parameter plots above. B: Results on input coincidence sorting for ALL and BCM (Intrator-Cooper) rule for a five input case. For 5 inputs there are 31 possible combinations of neurons driven by n ≥ 1 inputs: 5 × 1, 10 × 2, 10 × 3, 5 × 4 and 1 × 5 inputs as indicated beneath the abscissa. Parameters: ω(0) = [0.1, 0.1, 0.1, 0.1, 0.1]Tμ = 0.001, binary subsets of five presented in equal probability, random order, Euler integration with dt = 1 in both cases, for ALL: va = 0.7, ρ = 0.1, for BCM: ΘM(0) = 0.2, v0 = 0.4, γ = 10.
Fig 10.
Box plots for the number of neurons representing different combinations for the ALL-rule.
A: Input number N = 3. B: Input number N = 5. Combinations are aligned in ascending order of active inputs, with color code indicating the number of inputs, see legend at the bottom. Combinations are indicated by decimal numbers corresponding to binary set notation (e.g. “3” means the combination: 00011, where only the two last inputs are active). “o” means other, where this denotes occurrences of cells signaling several different combinations. The size of the neural network is M = 200, average connectivity c = 2, annealing parameters are: annealing rate ρ = 0.3, where the annealing threshold va for each neuron individually is drawn from a uniform distribution [0.75,0.95]. Decision threshold is 0.7. Initial weights are chosen from Gaussian distribution with mean = 0.001 and std = 0.0002. Initial learning rate μ(0) = 0.0005. Euler integration with dt = 1. Median, mean and standard deviation are shown on the basis of 100 trials.
Fig 11.
Different combination distribution based on decision threshold and neural network architecture.
A and B: three input case; C and D: five input case. Numbers 1 to 5 indicate combinations responsive to corresponding number of inputs; “Other” represent cells signaling more than one combination (see text for explanation), “Sub.” denotes sub-threshold cases, while “Sust.” denotes sustained activity, which does not subside after switching off the inputs, which does not happen here (but in the baseline, see Fig 12). Neural network (NN) architecture notation: “No of neurons”- “connectivity” (M-c). Numbers above column groups denote percentage of combination-selective neurons (vs. “Other” and “Sub.” neurons). Initial settings and learning parameters as in Fig 10. Note, Fig 10 corresponds to the results indicated by ovals.
Fig 12.
A: Three input case, B: Five input case. The column group “learned” shows performance of the ALL-rule, M = 200, c = 2; copied from Fig 11; “permuted” is for the case with learned weights randomly permuted; “permuted x 1.5” and “permuted x 2” for cases with permuted weights multiplied by 1.5 and 2, respectively. Decision threshold kept at 0.7 everywhere. Abbreviations: “Sub.” = sub-threshold, “Sust.” = sustained activity. Green numbers denote percentage of “Other”.
Fig 13.
Box plot for different combinations of inputs for the cases N = 5 inputs for the ALL-rule with 20% inhibitory cells.
Combinations are aligned in ascending order of active inputs, with color code indicating the number of inputs, see legend at the bottom. Combinations are indicated by decimal numbers corresponding to binary set notation (e.g. “3” means the combination: 00011, where only the two last inputs are active). “o” means other, where this denotes occurrences of cells signaling several different combinations. The size of the neural network is M = 200 (excitatory cells) with 40 inhibitory cells added. Average connectivity of excitatory cells onto excitatory cells is c = 2; connectivity onto inhibitory cells c = 20. Each excitatory cell, in addition, is given 10 inhibitory connections, with a fixed weights of 0.01. Annealing parameters are: annealing rate ρ = 0.3, where the annealing threshold va for each neuron individually is drawn from a uniform distribution [0.75,0.95]. Decision threshold is 0.7. Initial weights for excitatory inputs are chosen from Gaussian distribution with mean = 0.001 and std = 0.0002. Initial learning rate μ(0) = 0.0005. Euler integration with dt = 1. Median, mean and standard deviation are shown on the basis of 100 trials.