Random feature networks.

We start by introducing the main learning algorithm and the neuronal model of our work, the RFN. Consider a two-layer, feedforward ANN. Traditionally, all the weights are initialized randomly and learned through backpropagation by minimizing some loss objective. In sharp contrast, RFNs have their hidden layer weights sampled randomly from some distribution and fixed. Each hidden unit computes a random feature of the input, and only the output layer weights are trained (Fig 1). Note that the weights are randomly drawn but the neuron’s response is a deterministic function of the input given the weights.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Random feature networks with structured weights.

We study random feature networks as models for learning in sensory regions. In these networks, each neuron’s weight w is fixed as a random sample from some specified distribution. Only the readout weights β are trained. In particular, we specify distributions to be Gaussian Processes (GPs) whose covariances are inspired by biological neurons; thus, each realization of the GP resembles a biological receptive field. We build GP models of two sensory areas that specialize in processing timeseries and image inputs. We initialize w from these structured GPs and compare them against initialization from unstructured white-noise distribution.

https://doi.org/10.1371/journal.pcbi.1010484.g001

Mathematically, we have the hidden layer activations and output given by (1) where is the stimulus, are the hidden neuron responses, and is the predicted output. We use a rectified linear (ReLU) nonlinearity, σ(x) = max(0, x) applied entrywise in Eq (1). The hidden layer weights are drawn randomly and fixed. Only the readout weights β₀ and β are trained in RFNs.

In our RFN experiments, we train the readout weights and offset using a support vector machine (SVM) classifier with squared hinge loss and ℓ² penalty with regularization strength of tuned in the range [10⁻³, 10³] by 5-fold cross-validation. Our RFNs do not include a threshold for the hidden neurons, although this could help in certain contexts [36].

In the vast majority of studies with RFNs, each neuron’s weights are initialized i.i.d. from a spherical Gaussian distribution . We will call networks built this way classical unstructured RFNs (Fig 1). We propose a variation where hidden weights are initialized , where is a positive semidefinite covariance matrix. We call such networks structured RFNs (Fig 1), to mean that the weights are random with a specified covariance. To compare unstructured and structured weights on equal footing, we normalize the covariance matrices so that Tr(C) = Tr(I_d) = d, which ensures that the mean square amplitude of the weights .

Receptive fields modeled by linear weights.

Sensory neurons respond preferentially to specific features of their inputs. This stimulus selectivity is often summarized as a neuron’s receptive field, which describes how features of how the sensory space elicits responses when stimulated [5]. Mathematically, receptive fields are modeled as a linear filter in the stimulus space. Linear filters are also an integral component of the widely used LN model of sensory processing [6]. According to this model, the firing rate of a neuron is a nonlinear function applied to the projection of the stimulus onto the low-dimensional subspace of the linear filter.

A linear filter model of receptive fields can explain responses of individual neurons to diverse stimuli. It has been used to describe disparate sensory systems like visual, auditory, and somatosensory systems of diverse species including birds, mammals, and insects [7, 10–12, 15]. If the stimuli are uncorrelated, the filters can be estimated by computing the spike triggered average (STA), the average stimulus that elicited a spike for the neuron. When the stimuli are correlated, the STA filter is whitened by the inverse of the stimulus covariance matrix [38]. Often these STAs are denoised by fitting a parametric function to the STA [6], such as Gabor wavelets for simple cells in V1 [9].

We model the receptive field of a neuron i as its weight vector w_i and its nonlinear function as σ. Instead of fitting a parametric function, we construct covariance functions so that each realization of the resulting Gaussian process resembles a biological receptive field (Fig 1).

Structured weights project and filter input into the covariance eigenbasis.

We generate network weights from Gaussian processes (GP) whose covariance functions are inspired by the receptive fields of sensory neurons in the brain. By definition, a GP is a stochastic process where finite observations follow a Gaussian distribution [39]. We find that networks with such weights project inputs into a new basis and filter out irrelevant components. We will see that this adds an inductive bias to classical RFNs for tasks with naturalistic inputs and improves learning.

We view our weight vector w as the finite-dimensional discretization of a continuous function w(t) which is a sample from a GP. The continuous function has domain T, a compact subset of , and we assume that T is discretized using a grid of d equally spaced points {t₁, …, t_d} ⊂ T, so that w_i = w(t_i). Let the input be a real-valued function x(t) over the same domain T, which could represent a finite timeseries (D = 1), an image of luminance on the retina (D = 2), or more complicated spatiotemporal sets like a movie (D = 3). In the continuous setting, the d-dimensional ℓ² inner product gets replaced by the L²(T) inner product 〈w, x〉 = ∫_t∈Tw(t)x(t)dt.

Every GP is fully specified by its mean and covariance function C(t, t′). We will always assume that the mean is zero and study different covariance functions. By the Kosambi-Karhunen–Loève theorem [40], each realization of a zero-mean GP has a random series representation (2) in terms of standard Gaussian random variables , functions ϕ_i(t), and weights λ_i ≥ 0. The pairs are eigenvalue, eigenfunction pairs of the covariance operator , which is the continuous analog of the covariance matrix C. If C(t, t′) is positive definite, as opposed to just semidefinite, all and these eigenfunctions ϕ_i form a complete basis for L²(T). Using Eq (2), the inner product between a stimulus and a neuron’s weights is (3)

Eq (3) shows that the structured weights compute a projection of the input x onto each eigenfunction 〈ϕ_i, x〉 and reweight or filter by the eigenvalue λ_i before taking the ℓ² inner product with the random Gaussian weights z_i.

It is illuminating to see what these continuous equations look like in the d-dimensional discrete setting. Samples from the finite-dimensional GP are used as the hidden weights in RFNs, . First, the GP series representation Eq (2) becomes w = ΦΛz, where Λ and Φ are matrices of eigenvalues and eigenvectors, and is a Gaussian random vector. By the definition of the covariance matrix, , which is equal to ΦΛ² Φ^T after a few steps of linear algebra. Finally, Eq (3) is analogous to w^T x = z^T ΛΦ^T x. Since Φ is an orthogonal matrix, Φ^T x is equivalent to a change of basis, and the diagonal matrix Λ shrinks or expands certain directions to perform filtering. This can be summarized in the following theorem:

Theorem 1 (Basis change formula) Assume with C = ΦΛ² Φ^T its eigenvalue decomposition. For , define (4)

Then for .

Theorem 1 says that projecting an input onto a structured weight vector is the same as first filtering that input in the GP eigenbasis and doing a random projection onto a spherical random Gaussian. The form of the GP eigenbasis is determined by the choice of the covariance function. If the covariance function is compatible with the input structure, the hidden weights filter out any irrelevant features or noise in the stimuli while amplifying the descriptive features. This inductive bias facilitates inference on the stimuli by any downstream predictor. Because the spherical Gaussian distribution is the canonical choice for unstructured RFNs, there is a simple way to evaluate the effective kernel of structured RFNs as (see S1 Appendix).

Our expression for the structured kernel provides a concrete connection to the kernel theory of learning using nonlinear neural networks. For readers interested in such kernel theories, a full example and simulation results of how these work is given in S1 Appendix. There we show that there can be an exponential reduction in the number of samples needed to learn frequency detection using a structured versus unstructured basis (Fig A in S1 Appendix).

Warm-up: Frequency selectivity from stationary covariance.

To illustrate some results from our theoretical analysis, we start with a toy example of temporal receptive fields that are selective to particular frequencies. This example may be familiar to readers comfortable with Fourier series and basic signal processing. Let the input be a finite continuous timeseries x(t) over the interval T = [0, L]. We use the covariance function (5) where ω_k = 2πk/L is the kth natural frequency and are the weight coefficients. The covariance function Eq (5) is stationary, which means that it only depends on the difference between the timepoints t − t^′. Applying the compound angle formula, we get (6)

Since the sinusoidal functions cos(ω_kt) and sin(ω_kt) form an orthonormal basis for L²(T), Eq (6) is the eigendecomposition of the covariance, where the eigenfunctions are sines and cosines with eigenvalues . From Eq (2), we know that structured weights with this covariance form a random series: (7) where each . Thus, the receptive fields are made up of sinusoids weighted by λ_k and the Gaussian variables .

Suppose we want receptive fields that only retain specific frequency information of the signal and filter out the rest. Take λ_k = 0 for any k where ω_k < f_lo or ω_k > f_hi. We call this a bandlimited spectrum with passband [f_lo, f_hi] and bandwidth f_hi − f_lo. As the bandwidth increases, the receptive fields become less smooth since they are made up of a wider range of frequencies. If the λ_k are all nonzero but decay at a certain rate, this rate controls the smoothness of the resulting GP [41].

When these receptive fields act on input signals x(t), they implicitly transform the inputs into the Fourier basis and filter frequencies based on the magnitude of λ_k. In a bandlimited setting, any frequencies outside the passband are filtered out, which makes the receptive fields selective to a particular range of frequencies and ignore others. On the other hand, classical random features weight all frequencies equally, even though in natural settings high frequency signals are the most corrupted by noise.

Insect mechanosensors.

We next consider a particular biological sensor that is sensitive to the time-history of forces. Campaniform sensilla (CS) are dome-shaped mechanoreceptors that detect local stress and strain on the insect exoskeleton [42]. They are embedded in the cuticle and deformation of the cuticle through bending or torsion induces depolarizing currents in the CS by opening mechanosensitive ion channels. The CS encode proprioceptive information useful for body state estimation and movement control during diverse tasks like walking, kicking, jumping, and flying [42].

We will model the receptive fields of CS that are believed to be critical for flight control, namely the ones found at the base of the halteres [43] and on the wings [14] (Fig 2A). Halteres and wings flap rhythmically during flight, and rotations of the insect’s body induce torsional forces that can be felt on these active sensory structures. The CS detect these small strain forces, thereby encoding angular velocity of the insect body [43]. Experimental results show haltere and wing CS are selective to a broad range of oscillatory frequencies [14, 44], with STAs that are smooth, oscillatory, selective to frequency, and decay over time [13] (Fig 2B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Random receptive field model of insect mechanosensors.

(A) Diagram of the the cranefly, Tipula hespera. Locations of the mechanosensors, campaniform sensilla, are marked in blue on the wings and halteres. (B) Two receptive fields of campaniform sensilla are shown in blue. They are smooth, oscillatory, and decay over time. We model them as random samples from distributions parameterized by frequency and decay parameters. Data are from the hawkmoth [14]; cranefly sensilla have similar responses [13]. (C) Two random samples from the model distribution are shown in red. (D) The smoothness of the receptive fields is controlled by the frequency parameter. The decay parameter controls the rate of decay from the origin (not shown).

https://doi.org/10.1371/journal.pcbi.1010484.g002

We model these temporal receptive fields with a locally stationary GP [45] with bandlimited spectrum. Examples of receptive fields generated from this GP are shown in Fig 2C. The inputs to the CS are modeled as a finite continuous timeseries x(t) over the finite interval T = [0, L]. The neuron weights are generated from a covariance function (8) where ω_k = 2πk/L is the kth natural frequency. As in the warmup, the frequency selectivity of their weights is accounted for by the parameters f_lo and f_hi. As the bandwidth f_hi − f_lo increases, the receptive fields are built out of a wider selection of frequencies. This makes the receptive fields less smooth (Fig 2D). Each field is localized to near t = 0, and its decay with t is determined by the parameter γ. As γ increases, the receptive field is selective to larger time windows.

The eigenbasis of the covariance function Eq (8) is similar to a Fourier eigenbasis modulated by a decaying exponential. The eigenbasis is an orthonormal basis for the span of λ_ke^−t/γ cos(ω_kt) and λ_ke^−t/γ sin(ω_kt), which are a non-orthogonal set of functions in L²(T). The hidden weights transform timeseries inputs into this eigenbasis and discard components outside the passband frequencies [f_lo, f_hi].

We fit the covariance model to receptive field data from 95 CS neurons from wings of the hawkmoth Manduca sexta (data from [14]). Briefly, CS receptive fields were estimated as the spike-triggered average (STA) of experimental mechanical stimuli of the wings, where the stimuli were generated as bandpassed white noise (2–300 Hz).

To characterize the receptive fields of this population of CS neurons, we compute the data covariance matrix C_data by taking the inner product between the receptive fields. We normalize the trace to be the dimension of each receptive field (number of samples), which in this case is 40 kHz × 40 ms = 1600 samples. This normalization sets the overall scale of the covariance matrix. The data covariance matrix shows a tridiagonal structure (Fig 3A). The main diagonal is positive while the off diagonals are negative. All diagonals decay away from the top left of the matrix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Spectral properties of mechanosensory RFs and our model are similar.

We compare the covariance matrices generated from (A) receptive fields of 95 mechanosensors from [14], (B) the model Eq (8), and (C) 95 random samples from the same model. All covariance matrices show a tri-diagonal structure that decays away from the origin. (D) The first five principal components of all three covariance matrices are similar and explain 90% of the variance in the RF data. (E) The leading eigenvalue spectra of the data and models show similar behavior.

https://doi.org/10.1371/journal.pcbi.1010484.g003

To fit the covariance model to the data, we optimize the parameters f_lo, f_hi, and γ, finding f_lo = 75 Hz, f_hi = 200 Hz, and γ = 12.17 ms best fit the sensilla data. We do so by minimizing the Frobenius norm of the difference between C_data and the model (see S1 Appendix). The resulting model covariance matrix (Fig 3B) matches the data covariance matrix (Fig 3A) remarkably well qualitatively. The normalized Frobenius norm of the difference between C_data and the model is 0.4386. Examples of biological receptive fields and random samples from this fitted covariance model are shown in Fig B in S1 Appendix. To simulate the effect of a finite number of neurons, we generate 95 weight vectors (equal to the number of neurons recorded) and recompute the model covariance matrix (Fig 3C). We call this the finite neuron model covariance matrix C_finite, and it shows the bump and blob-like structures evident in C_data but not in C_model. This result suggests that these bumpy structures can be attributed to having a small number of recorded neurons. We hypothesize that these effects would disappear with a larger dataset and C_data would more closely resemble C_model.

For comparison, we also calculate the Frobenius difference for null models, the unstructured covariance model and the Fourier model (5). For the unstructured model, the Frobenius norm difference is 0.9986 while that of the Fourier model is 0.9123. The sensilla covariance model has a much lower difference (0.4386) compared to the null models, fitting the data more accurately. We show the covariance matrices and sampled receptive fields from the null models in Fig C to E in S1 Appendix.

Comparing the eigenvectors and eigenvalues of the data and model covariance matrices, we find that the spectral properties of both C_model and C_finite are similar to that of C_data. The eigenvalue curves of the models match that of the data quite well (Fig 3E); these curves are directly comparable because each covariance is normalized by its trace, which makes the sum of the eigenvalues unity. Further, all of the data and the model covariance matrices are low-dimensional. The first 10 data eigenvectors explain 97% of the variance, and the top 5 explain 90%. The top 5 eigenvectors of the model and its finite sample match that of the data quite well (Fig 3D).

Primary visual cortex.

We now turn to visually driven neurons from the mammalian primary cortex. Primary visual cortex (V1) is the earliest cortical area for processing visual information (Fig 4A). The neurons in V1 can detect small changes in visual features like orientations, spatial frequencies, contrast, and size.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Random receptive field model of Primary Visual Cortex (V1).

(A) Diagram of the mouse brain with V1 shown in blue. (B) Receptive fields of two mouse V1 neurons calculated from their response to white noise stimuli. The fields are localized to a region in a visual field and show “on” and “off” regions. (C) Random samples from the model Eq (9) distribution. (D) Increasing the receptive field size parameter in our model leads to larger fields. (E) Increasing the model spatial frequency parameter leads to more variable fields.

https://doi.org/10.1371/journal.pcbi.1010484.g004

Here, we model the receptive fields of simple cells in V1, which have clear excitatory and inhibitory regions such that light shone on the excitatory regions increase the cell’s response and vice-versa (Fig 4B). The shape of the regions determines the orientation selectivity, while their widths determine the frequency selectivity. The receptive fields are centered to a location in the visual field and decay away from it. They integrate visual stimuli within a small region of this center [46]. Gabor functions are widely used as a mathematical model of the receptive fields of simple cells [9].

We model these receptive fields using another locally stationary GP [45] and show examples of generated receptive fields in Fig 4C. Consider the inputs to the cortical cells to be a continuous two-dimensional image x(t), where the domain T = [0, L] × [0, L′] and . Since the image is real-valued, x(t) is the grayscale contrast or single color channel pixel values. The neuron weights are then generated from a covariance function of the following form: (9)

The receptive field center is defined by c, and the size of the receptive field is determined by the parameter s. As s increases, the receptive field extends farther from the center c (Fig 4D). Spatial frequency selectivity is accounted for by the bandwidth parameter f. As f decreases, the spatial frequency of the receptive field goes up, making the weights less smooth (Fig 4E).

The eigendecomposition of the covariance function Eq (9) leads to an orthonormal basis of single scale Hermite wavelets [47, 48]. When c = 0, the wavelet eigenfunctions are Hermite polynomials modulated by a decaying Gaussian: (10) where H_k is the kth (physicist’s) Hermite polynomial; eigenfunctions for nonzero centers c are just shifted versions of Eq (10). The detailed derivation and values of the constants c₁, c₂, c₃ and normalization are in S1 Appendix.

We use Eq (9) to model receptive field data from 8,358 V1 neurons recorded with calcium imaging from transgenic mice expressing GCaMP6s; the mice were headfixed and running on an air-floating ball. We presented 24,357 unique white noise images of 14 × 36 pixels using the Psychtoolbox [49], where the pixels were white or black with equal probability. Images were upsampled to the resolution of the screens via bilinear interpolation. The stimulus was corrected for eye-movements online using custom code. The responses of 45,026 cells were collected using a two-photon mesoscope [50] and preprocessed using Suite2p [51]. Receptive fields were calculated from the white noise images and the deconvolved calcium responses of the cells using the STA. For the covariance analysis, we picked cells above the signal-to-noise (SNR) threshold of 0.4; this gave us 8,358 cells. The SNR was calculated from a smaller set of 2,435 images that were presented twice using the method from [4]. As a preprocessing step, we moved the center of mass of every receptive field to the center of the visual field.

We compute the data covariance matrix C_data by taking the inner product between the receptive fields. We normalize the trace to be the dimension of each receptive field, which in this case is (14 × 36) pixels = 504 pixels. The data covariance matrix resembles a tridiagonal matrix. However, the diagonals are non-zero only at equally spaced segments. Additionally, their values decay away from the center of the matrix. We show C_data zoomed in at the non-zero region around the center of the matrix (Fig 5A); this corresponds to the 180 × 180 pixel region around the center of the full 504 × 504 pixel matrix. The full covariance matrix is shown in Fig F in S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Spectral properties of V1 RFs and our model are similar.

We compare the covariance matrices generated from the (A) receptive fields of 8,358 mouse V1 neurons, (B) the GP model Eq (9), and (C) 8,358 random samples from the model. These resemble a tri-diagonal matrix whose diagonals are non-zero at equally-spaced segments. (D) The leading 10 eigenvectors of the data and model covariance matrices show similar structure and explain 68% of the variance in the data. Analytical Hermite wavelet eigenfunctions are in the last row and differ from the model due to discretization (both cases) and finite sampling (8,358 neurons only). (E) The eigenspectrum of the model matches well with the data. The staircase pattern in the model comes from repeated eigenvalues at each frequency. The model curve with infinite neurons (black) is obscured by the model curve with 8,358 neurons (red).

https://doi.org/10.1371/journal.pcbi.1010484.g005

In the covariance model, the number of off-diagonals, the center, the rate of their decay away from the center are determined by the parameters f, s and c respectively. The covariance between pixels decays as a function of their distance from c. This leads to the the equally-spaced non-zero segments. On the other hand, the covariance also decays as a function of the distance between pixels. This brings the the diagonal structure to the model. When the frequency parameter f increases, the number of off-diagonals increases. Pixels in the generated weights become more correlated and the weights become spatially smoother. When the size parameter s increases, the diagonals decay slower from the center c, increasing correlations with the center pixel and leading the significant weights to occupy more of the visual field.

We again optimize the parameters to fit the data, finding s = 1.87 and f = 0.70 pixels, by minimizing the Frobenius norm of the difference between C_data and the model. We do not need to optimize over the center parameter c, since we preprocess the data so that all receptive fields are centered at c = (7, 18), the center of the 14 × 36 grid. The resulting model covariance matrix (Fig 5B) and the data covariance matrix (Fig 5A) match remarkably well qualitatively. The normalized Frobenius norm of the difference between C_data and the model is 0.2993. Examples of biological receptive fields and random samples from the fitted covariance model are shown in Fig G in S1 Appendix. To simulate the effect of a finite number of neurons, we generate 8,358 weights, equal to the number of neurons in our data, to compute C_finite shown in Fig 5C. This finite matrix C_finite looks even more like C_data, and it shows that some of the negative covariances far from center result from finite sample size but not all.

For comparison, we also calculate the normalized Frobenius difference for null models, the unstructured covariance model and a translation invariant V1 model. In the translation invariant model, we remove the spatially localizing exponential in Eq (9) and only fit the spatial frequency parameter, f. For the unstructured model, the Frobenius norm difference is 0.9835 while that of the translation invariant model is 0.9727. The V1 covariance model has a much lower difference (0.2993) and is a better fit to the data. We show the covariance matrices and sampled receptive fields from these null models in Fig H to J in S1 Appendix.

Similar spectral properties are evident in the eigenvectors and eigenvalues of C_model, C_finite, C_data, and the analytical forms derived in Eq (10) (Fig 5D and 5E). The covariances are again normalized to have unit trace. Note that the analytical eigenfunctions are shown on a finer grid than the model and data because the analysis was performed in continuous space. The differences between the eigenfunctions and eigenvalues of the analytical and model results are due to discretization. Examining the eigenvectors (Fig 5D), we also see a good match, although there are some rotations and differences in ordering. These 10 eigenvectors explain 68% of the variance in the receptive field data. For reference, the top 80 eigenvectors explain 86% of the variance in the data and all of the variance in the model. The eigenvalue curves of both the models and the analytical forms match that of the data (Fig 5E) reasonably well, although not as well as for the mechanosensors. In S1 Appendix, we repeat this analysis for receptive fields measured with different stimulus sets in the mouse and different experimental dataset from non-human primate V1. Our findings are consistent with the results shown above (Fig K to P in S1 Appendix).

Frequency detection.

CS naturally encode the time-history of strain forces acting on the insect body and sensors inspired by their temporal filtering properties have been shown to accurately classify spatiotemporal data [52]. Inspired by this result, we test sensilla-inspired mechanosensory receptive fields on a timeseries classification task (Fig 6A, top). Each example presented to the network is a 100 ms timeseries sampled at 2 kHz so that d = 200, and the goal is to detect whether or not each example contains a sinusoidal signal. The positive examples are sinusoidal signals with f₁ = 50 Hz and corrupted by noise so that their SNR = 1.76 (2.46 dB). The negative examples are Gaussian white noise with matched amplitude to the positive examples. Note that this frequency detection task is not linearly separable because of the random phases in positive and negative examples. See S1 Appendix for additional details including the definition of SNR and how cross-validation was used to find the optimal parameters f_lo = 10 Hz, f_hi = 60 Hz, and γ = 50 ms.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Random mechanosensory weights enable learning with fewer neurons in time-series classification tasks.

We show the test error of random feature networks with both mechanosensory and classical white-noise weights against the number of neurons in their hidden layer. For every hidden layer width, we generate five random networks and average their test error. In the error curves, the solid lines show the average test error while the shaded regions represent the standard error across five generations of the random network. The top row shows the timeseries tasks that the networks are tested on. (A, top) In the frequency detection task, a f₁ = 50 Hz frequency signal (purple) is separated from white noise (black). (B, top) In the frequency XOR task, f₁ = 50 Hz (purple) and f₂ = 80 Hz (light purple) signals are separated from white noise (black) and mixtures of 50 Hz and 80 Hz (gray). When their covariance parameters are tuned properly, mechanosensor-inspired networks achieve lower error using fewer hidden neurons on both frequency detection (A, bottom) and frequency XOR (B, bottom) tasks. However, the performance of bio-inspired networks suffer if their weights are incompatible with the task.

https://doi.org/10.1371/journal.pcbi.1010484.g006

For the same number of hidden neurons, the structured RFN significantly outperforms a classical RFN. We show test performance using these tuned parameters in Fig 6A. Even in this noisy task, it achieves 0.5% test error using only 25 hidden neurons. Meanwhile, the classical network takes 300 neurons to achieve similar error.

Predictably, the performance suffers when the weights are incompatible with the task. We show results when f_lo = 10 Hz and f_hi = 40 Hz and the same γ (Fig 6A). The incompatible RFN performs better than chance (50% error) but much worse than the classical RFN. It takes 300 neurons just to achieve 16.3% test error. The test error does not decrease below this level even with additional hidden neurons.

Frequency XOR task.

To challenge the mechanosensor-inspired networks on a more difficult task, we build a frequency Exclusive-OR (XOR) problem (Fig 6B, top). XOR is a binary function which returns true if and only if the both inputs are different, otherwise it returns false. XOR is a classical example of a function that is not linearly separable and thus harder to learn. Our inputs are again 100 ms timeseries sampled at 2 kHz. The inputs either contain a pure frequency of f₁ = 50 Hz or f₂ = 80 Hz, mixed frequency signals with both f₁ and f₂, or white noise. In both the pure and mixed frequency cases, we add noise so that the SNR = 1.76. See S1 Appendix for details. The goal of the task is to output true if the input contains either pure tone and false if the input contains mixed frequencies or is white noise.

We tune the GP covariance parameters f_lo, f_hi, and γ from Eq (8) using cross-validation. The cross validation procedure and algorithmic details are identical to that of the frequency detection task. Using cross validation, we find the optimal parameters to be f_lo = 50 Hz, f_hi = 90 Hz, and γ = 40 ms. For incompatible weights, we take f_lo = 10 Hz, f_hi = 60 Hz, and the same γ.

The structured RFN significantly outperform classical RFN for the same number of hidden neurons. We show network performance using these parameters in Fig 6B. Classification error of 1% can be achieved with 25 hidden neurons. In sharp contrast, the classical RFN requires 300 hidden neurons just to achieve 6% error. With incompatible weights, the network needs 300 neurons to achieve just 15.1% test error and does not improve with larger network sizes. Out of the four input subclasses, it consistently fails to classify pure 80 Hz sinusoidal signals which are outside its passband.

Image classification.

We next test the V1-inspired receptive fields on two standard digit classification tasks, MNIST [53] and KMNIST [54]. The MNIST and KMNIST datasets each contain 70,000 images of handwritten digits. In MNIST, these are the Arabic numerals 0–9, whereas KMNIST has 10 Japanese hiragana phonetic characters. Both datasets come split into 60,000 training and 10,000 test examples. With 10 classes, there are 6,000 training examples per class. Every example is a 28 × 28 grayscale image with centered characters.

Each hidden weight has its center c chosen uniformly at random from all pixels. This ensures that the network’s weights uniformly cover the image space and in fact means that the network can represent any sum of locally-smooth functions (see S1 Appendix). We use a network with 1,000 hidden neurons and tune the GP covariance parameters s and f from Eq (9) using 3-fold cross validation on the MNIST training set. Each parameter ranges from 1 to 20 pixels, and the optimal parameters are found with a grid search. We find the optimal parameters to be s = 5 pixels and f = 2 pixels. We then refit the optimal model using the entire training set. The parameters from MNIST were used on the KMNIST task without additional tuning.

The V1-inspired achieves much lower average classification error as compared to the classical RFN for the same number of hidden neurons. We show learning performance using these parameters on the MNIST task in Fig 7A. To achieve 6% error on the MNIST task requires 100 neurons versus 1,000 neurons for the classical RFN, and the structured RFN achieves 2.5% error with 1,000 neurons. Qualitatively similar results hold for the KMNIST task (Fig 7B), although the overall errors are larger, reflecting the harder task. To achieve 28% error on KMNIST requires 100 neurons versus 1,000 neurons for the classical RFN, and the structured RFN achieves 13% error with 1,000 neurons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Random V1 weights enable learning with fewer neurons and fewer examples on digit classification tasks.

We show the average test error of random feature networks with both V1 and classical white-noise weights against the number of neurons in their hidden layer. For every hidden layer width, we generate five random networks and average their test error. The solid lines show the average test error while the shaded regions represent the standard error across five generations of the random network. The top row shows the network’s test error on (A) MNIST and (B) KMNIST tasks. When their covariance parameters are tuned properly, V1-inspired networks achieve lower error using fewer hidden neurons on both tasks. The network performance deteriorates when the weights are incompatible to the task. (C) MNIST and (D) KMNIST with 5 samples per class. The V1 network still achieves lower error on these fewshot tasks when the parameters are tuned properly.

https://doi.org/10.1371/journal.pcbi.1010484.g007

Again, network performance suffers when GP covariance parameters do not match the task. This happens if the size parameter s is smaller than the stroke width or spatial scale f doesn’t match the stroke variations in the character. Taking the incompatible parameters s = 0.5 and f = 0.5 (Fig 7A and 7B), the structured weights performs worse than the classical RFN in both tasks. With 1,000 hidden neurons, it achieves the relatively poor test errors of 8% on MNIST (Fig 7A) and 33% on KMNIST (Fig 7B).

Structured weights improve generalization with limited data.

Alongside learning with fewer hidden neurons, V1 structured RFNs also learn more accurately from fewer examples. We test few-shot learning using the image classification datasets. The training examples are reduced from 60,000 to 50, or only 5 training examples per class. The test set and GP parameters remain the same.

Structured encodings allow learning with fewer samples than unstructured encodings. We show these few-shot learning results in Fig 7C and 7D. The networks’ performance saturate past a few hundred hidden neurons. For MNIST, the lowest error achieved by V1 structured RFN is 27% versus 33% for the classical RFN and 37% using incompatible weights (Fig 7C). The structured network acheives 61% error using structured features on the KMNIST task, as opposed to 66% for the classical RFN and 67% using incompatible weights (Fig 7D).

Networks train faster when initialized with structured weights.

Now we study the effect of structured weights as an initialization strategy for fully-trained neural networks where all weights in the network vary. We hypothesized that structured initialization allows networks to learn faster, i.e. that the training loss and test error would decrease faster than with unstructured weights. We have shown that the performance of RFNs improves with biologically inspired weight sampling. However, in RFNs Eq (1) only the readout weights β are modified with training, and the hidden weights W are frozen at their initial value.

We compare the biologically-motivated initialization with a classical initialization where the variance is inversely proportional to the number of hidden neurons, . This initialization is widely known as the “Kaiming He normal” scheme and is thought to stabilize training dynamics by controlling the magnitude of the gradients [55]. The classical approach ensures that , so for fair comparison we scale our structured weight covariance matrix to have Tr(C) = 2. In our studies with RFNs the trace is equal to d, but this weight scale can be absorbed into the readout weights β due to the homogeneity of the ReLU.

We again compare structured and unstructured weights on MNIST and KMNIST tasks, common benchmarks for fully-trained networks. The architecture is a single hidden layer feedforward neural network (Fig 1) with 1,000 hidden neurons. The cross-entropy loss over the training sets are minimized using simple gradient descent (GD) for 3,000 epochs. For a fair comparison, the learning rate is optimized for each network separately. We define the area under the training loss curve as a metric for the speed of learning. Then, we perform a grid search in the range of (1e⁻⁴, 1e⁰) for the learning rate that minimizes this metric, resulting in the parameters 0.23, 0.14, 0.14 for structured, unstructured and incompatible networks respectively. All other parameters are the same as for image classification.

In both the MNIST and KMNIST tasks, the V1-initialized network minimizes the loss function faster than the classically initialized network. For the MNIST task, the V1 network achieves a loss value of 0.05 after 3,000 epochs compared to 0.09 for the other network (Fig 8A). We see qualitatively similar results for the KMNIST task. At the end of training, the V1-inspired network’s loss is 0.08, while the classically initialized network only reaches 0.12 (Fig 8B). We find that the the V1-initialized network performs no better than classical initialization when the covariance parameters do not match the task. With incompatible parameters, the V1-initialized network achieves a loss value of 0.11 on MNIST and 0.15 on KMNIST.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. V1 weight initialization for fully-trained networks enables faster training on digit classification tasks.

We show the average test error and the train loss of fully-trained neural networks against the number of training epochs. The hidden layer of each network contains 1,000 neurons. We generate five random networks and average their performance. The solid lines show the average performance metric across five random networks while the shaded regions represent the standard error. The top row shows the network’s training loss on (A) MNIST and (B) KMNIST tasks. The bottom row shows the corresponding test error on (C) MNIST and (D) KMNIST tasks. When their covariance parameters are tuned properly, V1-initialized networks achieve lower training loss and test error under fewer epochs on both MNIST and KMNIST tasks. The network performance is no better than unstructured initialization when the weights are incompatible with the task.

https://doi.org/10.1371/journal.pcbi.1010484.g008

Not only does it minimize the training loss faster, the V1-initialized network also generalizes well and achieves a lower test error at the end of training. For MNIST, it achieves 1.7% test error compared to 3.3% error for the classically initialized network, and 3.6% using incompatible weights (Fig 8C). For KMNIST, we see 9% error compared to 13% error with classical initalization and 15% using incompatible weights (Fig 8D).

We see similar results across diverse hidden layer widths and learning rates (Fig Q to T in S1 Appendix), with the benefits most evident for wider networks and smaller learning rates. Furthermore, the structured weights show similar results when trained for 10,000 epochs (rate 0.1; 1,000 neurons; not shown) and with other optimizers like minibatch Stochastic Gradient Descent (SGD) and ADAM (batch size 256, rate 0.1; 1,000 neurons; not shown). Structured initialization facilitates learning across a wide range of networks.

However, the improvement is not universal: no significant benefit was found by initializing the early convolutional layers of the deep network AlexNet [56] and applying it to the ImageNet dataset [57], as shown in S1 Appendix and Fig U in S1 Appendix. The large amounts of training data and the fact that only a small fraction of the network was initialized with structured weights could explain this null result. Also, in many of these scenarios the incompatible structured weights get to performance on par with the compatible ones by the end of training, when the poor inductive bias is overcome.

Improving representation with structured random weights.

We have shown how structured receptive field weights can improve the performance of RFNs and fully-trained networks on a number of supervised learning tasks. As long as the receptive fields are compatible with the task itself, then performance gains over unstructured features are possible. If they are incompatible, then the networks performs no better or even worse than using classical unstructured weights.

These results can be understood with our theoretical framework. Structured weights effectively cause the input x to undergo a linear transformation into a new representation following Theorem 1. In all of our examples, this new representation is bandlimited due to how we design the covariance function. (The V1 weights have all eigenvalues nonzero, but the spectrum decays exponentially, so it acts as a lowpass filter.) By moving to a bandlimited representation, we both filter out noise—high-frequency components—and reduce dimensionality—coordinates in outside the passband are zero. In general, noise and dimensionality both make learning harder.

It is easiest to understand these effects in the frequency detection task. For simplicity, assume we are using the stationary features of our warm-up to do frequency detection. In this task, all of the signal power is contained in the f₁ = 50 Hz frequency, and everything else is due to noise. If the weights are compatible with the task, this means that w is a sum of sines and cosines of frequencies ω_k in some passband which includes f₁. The narrower we make this bandwidth while still retaining the signal, the higher the SNR of becomes since more noise is filtered out (see S1 Appendix).