Ethics statement

All animal procedures were approved by the Animal Care Committee of the Research Institute, McGill University Health Centre, and are in accordance with the guidelines of the Canadian Council on Animal Care.

Animal preparation

Detailed descriptions of the procedures for animal surgery, recording, and maintenance have been published previously [35] and will only be summarized here. Adult cats of either sex were anesthetized with isofluorane/oxygen (3–5%), followed by intravenous (i.v.) cannulation. A bolus i.v. injection (5 mg·kg^-1) followed by a continuous infusion (5.3 mg·kg^-1·h^-1) of propofol was provided for surgical anesthesia. The animal was secured in a stereotaxic apparatus, and a craniotomy and small durotomy were performed over the cortical region of interest (Areas 17/18). The cortical surface was protected by 2% agarose and petroleum jelly. Body temperature was maintained using a thermistor-controlled heating pad. The corneas were protected during the initial surgery using topical ophthalmic carboxymethylcellulose (1%), and subsequently, neutral contact lenses were inserted for long-term corneal protection. The eyes were brought into focus at the distance at which visual stimuli would be presented (57 cm) using spectacle lenses, with artificial pupils (2.5mm) to further improve optical quality. Nictitating membranes were retracted, and pupils were dilated using topical phenylephrine hydrochloride (2.5%) and a mydriatic (atropine sulfate, 1%, or cyclopentolate, 1.0%), which were re-applied daily.

The animal was connected to a ventilator supplying oxygen/nitrous oxide (70:30) after the completion of all surgical procedures. Gallamine triethiodide (i.v. bolus, to effect, followed by infusion, 10 mg·kg^-1h^-1) was used to induce and maintain paralysis. The propofol infusion rate was reduced (5.3 mg·kg^-1h^-1) and additional anaesthesia was provided with i.v. remifentanil (bolus injection, 1.25 μg·kg^-1, followed by infusion, 3.7 μg·kg^-1h^-1). Vital signals (temperature, heart rate, expired CO2 and EEG activity) were monitored and kept at normal levels during recording experiments conducted over a period of 3–4 days.

Electrophysiology and visual stimuli

Extracellular recordings were obtained from 32-channel multielectrode probes (NeuroNexus, linear array: A1x32-6mm-100–177, linear edge: A1x32-Edge-5mm-100–177, or polytrode: A1x32-Poly2–5mm-50–177), using a Plexon Recorder (f_s = 31.25kHz) or OpenEphys (f_s = 30kHz) data acquisition system. Spiking responses (300Hz-5kHz) from a channel of interest were monitored “on-line”. All 32 channels of broadband neural data, as well as CRT photodiode signals (see below), were streamed to a hard disk for later off-line spike sorting using Kilosort 2.0 software [36], MUA and LFP signal extraction, and temporal registration between stimuli and neural signals. The sorted neural spikes, MUA and LFP signals were then analyzed in detail as described below.

Probes were inserted in approximately vertical, near-columnar penetrations, so that the recorded neurons generally had similar receptive field location, preferred spatial frequency, and preferred orientation [37,38,39]. An initial estimate of the neurons’ receptive field location and dominant eye were identified using an interactively controlled bar stimulus. The non-dominant eye was occluded, and the CRT monitor centered approximately on the receptive field.

Visual stimuli were generated on a CRT monitor (NEC FP1350, 20”, 640 x 480 pixels, 150 Hz, 36 cd/m²) using custom software written in MATLAB (MathWorks, USA) with Psychophysics Toolbox [40–42] running on a Macintosh computer (MacPro, 2.66 GHz Quad Core Intel Xeon, 6 GB, NVIDIA GeForce GT 120, MacOSX 10.6.8). The monitor’s gamma nonlinearity was measured with a photometer (United Detector Technology) and corrected using inverse lookup tables. Stimulus onset/offset timing and electrophysiological recordings were temporally registered using a photodiode (TAOS, TSL12S) in a corner of the monitor screen. Photodiode signals were also used to verify the absence of dropped frames.

Detailed descriptions of the production of visual stimuli have been published [43] and will only be summarized here. Natural image stimuli were constructed from monochrome 480x480 pixel subsets of photographs from the McGill Calibrated Color Image Database [44]. Images having low pixel standard deviations were discarded, since these images were nearly blank. We subtracted the mean luminance in remaining images and clipped to 8 bits. These natural images were the same as used previously [34,43], except that in some experiments the image stimuli had higher RMS contrast, with the RMS contrast normalized or not. The observed final results were independent of these variations in the stimuli (S1 Fig) and will be considered jointly. Ensembles of 375 such images were presented in 5 sec trials. The CRT refresh rate was 150 Hz, with each stimulus frame presented twice, for an effective rate of 75 Hz. Stimulus image ensembles were divided into 3 sets, designed to be used for training, regularization, and testing (see below). The training dataset was used to fit multiple models explained in this work, the hold-back regularization dataset for cross-validation, and the hold-back test dataset for evaluating the accuracies of model predictions. Data collection was carried out in 5 trial blocks (henceforth referred to as “sweeps”) each consisting of 20 training ensembles, and 5 ensembles each for regularization and test sets. Image ensemble sets were presented in pseudorandom order over a period of ca. 45 minutes.

Neural data processing

Neural data was processed in MATLAB (MathWorks, USA). The recorded broadband neural signals (sampling rate f_s = 31.25kHz with Plexon or f_s = 30kHz with OpenEphys) were filtered using Matlab’s designfilt and filtfilt functions. Neural signals were first low-pass filtered (<100Hz: passband frequency: 100, stopband frequency: 120, design method: ‘kaiserwin’, impulse response: ‘fir’, passband ripple: 0.1, stop band attenuation: 60), and also notch filtered to remove mains noise (60 Hz: half power frequencies: 59 and 61, design method: ‘butter’, impulse response: ‘iir’) to obtain local field potentials (LFPs), and high-pass filtered (>600Hz: passband frequency: 600, stopband frequency: 550, design method: ‘kaiserwin’, impulse response: ‘fir’, passband ripple: 0.1, stop band attenuation: 60) to obtain multiunit spiking activity.

To identify the recording channels in grey matter, current source density (CSD) analysis was employed on LFP signals extracted from responses to drifting sinewave gratings at varying spatial frequencies or orientations, which had been collected for conventional tuning curve measurements. These LFPs were analysed across the channels using the CSD plotter software [45] to identify the grey matter boundaries (S2 Fig). Channels with a prominent current sink in the CSD profile were specified as the granular layer, and sources above and below this sink were identified as the supragranular and infragranular layers respectively. Further analyses were performed only for grey matter channels.

Spike waveforms were extracted and classified using Kilosort 2.0 software [36] to obtain single neuron spike time series, and to associate each neuron to a principal recording channel (Kilosort parameters include Th: [10,4], minFR: 1, spkTh: -5, AUCsplit: 0.9). Spike waveform clusters identified from Kilosort were manually accepted if the spike time correlograms showed a clear minimum near zero, spike amplitudes showed a normal distribution unique from the multi-unit spike amplitude distribution, and the shape of the spike waveform showed a clear distinction from noise. The acceptance, rejection and merging of spike clusters were carried out manually in Phy-GUI which is integrated with Kilosort. Spike-sorted neurons having an average frequency <1.0 spikes/sec in response to natural image ensembles were considered to be unresponsive and discarded from further analysis.

Trial-to-trial response variability

As illustrated in Fig 1, a neuron’s response to repeated presentation of a natural image movie typically varied across trials. For each neuron we estimated a measure of its trial-by-trial variability as follows. For each of the n repetitions of the regularization set, the Pearson’s correlation (R) between the neuron’s response to that repetition and the average response for the (n-1) other repetitions of the same stimulus were calculated. provided a reliability measure of each neuron’s response for a given trial i (i = 1, …, n).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Trial-wise variability.

Each natural image movie is presented across multiple (n) trials and the corresponding single neuron responses are compared to calculate a variability ratio, VR (Eq 1-3).

https://doi.org/10.1371/journal.pcbi.1013661.g001

(1)

where refers to the neuron’s spiking response for the i-th trial, and denotes Pearson’s correlation. Since we have used 20 repetitions for our regularization datasets, n = 20. The average of these correlation coefficients was calculated:

(2)

This value is similar in principle to the “reliability ratio” of a neuron [46,47]. The variability ratio for each neuron was then quantified as:

(3)

VR values could range from zero (perfect reproducibility) to unity (maximal trial-wise variability).

Spike-LFP phase locking

We analysed the relationship of each neuron’s spiking activity to different LFP frequency bands, i.e., δ (0.5-4Hz), θ (4-7.5 Hz), α (7.5-12.5Hz), β (12.5-30Hz) and γ (30–100Hz) (Fig 2A and 2B). These LFP frequency bands were extracted in MATLAB by designing finite impulse response (FIR) bandpass filters using a Kaiser window applied to downsampled (x20) raw signals (i.e., for Plexon recorded data: from 31.25kHz downsampled to 1562.5Hz, OpenEphys recorded data: from 30kHz downsampled to 1500Hz). We then calculated a phase locking strength (PLS_f) of each neuron’s firing to each filtered LFP band [8,48]:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Spiking activity vs LFP frequency bands.

A) Spiking activity of an example neuron 1 along with bandpass-filtered LFP signals recorded simultaneously. B) Similar to A, for example neuron 2. C) Phase locking strength (PLS) values (comparing spiking activity in relation to the phase of LFP bands) calculated for neuron 1. D) Similar to C, for example neuron 2.

https://doi.org/10.1371/journal.pcbi.1013661.g002

(4)

where f indexes the LFP frequency band, N is the total number of spikes for the neuron, and is the phase of the LFP frequency band f at a given time t_k of the k-th spike. was extracted using MATLAB’s hilbert function. PLS values could range from zero (no consistent phase relation) to unity (perfect phase locking).

Cortical state dynamics

Local field potentials (LFPs) and multi-unit activity (MUA) show strong correlations with cortical state dynamics [7]. To identify the fluctuations of cortical state reflected by the population spiking activity across discrete time points, we extracted MUA (level crossings at ±5 SD of the > 600 Hz high-passed signal) from each of the channels in grey matter. To avoid confusion with responses to the visual stimuli, MUA-based cortical state analysis was performed during the 4 blank screen periods between the 5 sweeps. These MUA signals were binned (100 ms) to examine the up/down phases in network firing using a global fluctuation index (GFI, Eq 5) [26]. A comparatively higher GFI indicates prominent up/down phases characteristic of a synchronized cortical state. For each blank screen period, the binned firing rates were averaged across the channels to obtain channel-averaged firing rates (). The GFI was taken as the ratio between the standard deviation and the mean of :

(5)

To analyse cortical state fluctuations in a continuous time scale throughout the data recording session, we calculated a synchrony index (SI, Eq 6) [49] based on the LFP signals from the middle channel in layer IV identified from CSD analysis. The power spectral density (PSD) was first computed using Matlab’s pspectrum function (type: ’spectrogram’, FrequencyLimits: [0 100], ‘TimeResolution’: 30 sec and ‘OverlapPercent’: 83.33) spanning throughout the data recording session. Then the Synchrony Index was calculated:

(6)

where P_low refers to power in the frequency range of [0.5-15Hz] and P_high refers to power in the frequency range of [15–100Hz] obtained from the PSD analysis. SI could vary from zero to unity, with higher SI values indicating a more synchronized brain state and lower SI values indicating a more desynchronized brain state.

Model architecture and parameter estimation

We introduce a convolutional neural network model including parallel pathways: a stimulus-driven pathway to capture the neuron’s stimulus-dependent response (r_stim), and cortical state pathways driven by two population signals, LFP and MUA, to capture the cortical state fluctuations (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Model architecture for system identification.

Visual stimuli (S) are convolved with a spatiotemporal filter (c), then passed through a PReLU (G) nonlinearity and a gaussian map (w) layer to provide the stimulus-driven response (r_stim). Concurrent LFP and MUA signals provide inputs to the cortical state pathway, passing through LFP (f_LFP) and MUA (f_MUA) temporal filters to produce cortical state modulatory responses (). These signals summate with r_stim and then are subjected to a final output nonlinearity (N) to produce the estimated neural response r_est.

https://doi.org/10.1371/journal.pcbi.1013661.g003

Stimulus-driven pathway.

Details of the model architecture for the stimulus-driven pathway (Eq 7) have been described previously [34], so here will only be briefly summarized.

(7)

A cropped and downsampled stimulus S (see below) was first spatiotemporally convolved with the filter c. This was similar to the initial linear component of a linear-nonlinear model for a small region of the stimulus. The two-dimensional image output of this convolution at a given time t was then passed through a static nonlinearity G, modelled using a Parametric Rectified Linear Unit (PReLU) with a single parameter PReLU_α. The PReLU parameter, α, can take a value in a continuum from -1 to +1 indicating simple or complex type neural responses as well as intermediate types. The “map” layer w was modelled as a parameterized two-dimensional spatial Gaussian to identify the regions in visual space where the subunit filter c is operative.

Model architecture.

LFPs and MUA, recorded simultaneously with spiking activity, both showed variations in SI and GFI (explained above) in our recorded data. We used these brain signals as indicators of cortical states to capture the variability in neural responses. To implement this, frames of natural image movies and the corresponding LFP and MUA segments were fed in parallel to the stimulus-driven pathway and LFP- and MUA-driven pathways of the model, respectively.

Overall activity from the cortical state-related signals was combined additively with responses from the stimulus-driven pathway, and subjected to a final output nonlinearity (N):

(8)

The output nonlinearity was modelled as a rectified power law, with gain (g) and exponent (exp):

(9)

We also evaluated models in which the cortical state signals acted multiplicatively [28,50], or with divisive normalization [51], but additive modulation of stimulus-driven neural activity by the cortical state-driven activity provided higher predictive accuracy (S4 Fig).

Model parameter estimation.

Optimization of the model parameters was implemented in Python using tools from the TensorFlow 2.0.0 [52] and Keras 1.1.2 [53] packages. To avoid overfitting, we applied L2 regularization to the model coefficients from both LFP and MUA temporal filters and incorporated early stopping in the complete model with a patience value of 50 epochs. Preliminary analysis showed that the use of L2 regularization in the stimulus-driven pathway did not bring additional improvements in prediction accuracy. Model parameters were optimized as explained below.

First, a three-pass cropping procedure [34] was employed to identify regions of the stimulus that encompass the receptive field. We initially estimated the parameters of the model containing only the stimulus-driven pathway using full 480x480 stimulus images downsampled to 30x30 in the first pass. During the second pass, a square-shaped cropping window enclosing an area slightly larger than the apparent receptive field was identified and downsampled to 30x30 to re-train the model. In the final pass, we adjusted the cropping window based on the model estimated from the second pass to identify more accurate boundaries of the receptive field. We imposed a minimum cropping window size of 120 pixels to handle very small receptive fields. The final cropped stimulus from the third pass was downsampled to 30x30 to be used in further analyses as input to the stimulus-driven pathway. The cropped images in each pass are downsampled to 30x30, to avoid an excessive number of parameters to be estimated, which could cause serious overfitting as well as requiring increased computational time.

The two parameters of the final output nonlinearity (Eq 9) were estimated in two stages, as previously described [34]. In the first stage we set N to be a simple ReLU activation function and obtained a preliminary estimate of the receptive field filter and the filters for the LFP and MUA signals. For this optimization, we initialized the spatiotemporal receptive field filter with a Glorot normal initializer, PReLU α at 0.5 and the Gaussian map layer at the center of the image with standard deviation equal to the length of the image. For the second stage, we used the pre-trained receptive field weights from the first stage as initial conditions of the spatiotemporal receptive field (with other model parameters initialized similar to the first stage), and re-trained the model with N set to be an adaptive exponent output nonlinearity defined in a custom Keras layer with two trainable parameters, gain (g) and power law exponent (exp) as in Eq 9.

The models’ predictive accuracy for the hold-back test dataset was measured by the variance accounted for (VAF), computed as the square of the Pearson correlation between the neuron’s actual measured response () and the response of the estimated model () expressed as a percentage:

(10)

The VAF measures the overall model performance in explaining the neural response (i.e., the sum of stimulus-driven response, cortical state-driven response as well as other ongoing intrinsic noise). However, to explicitly quantify the trial-wise response variability explained by our cortical state-driven pathway, we calculated a “”. For this, we calculated the variable portion of the trial-wise response, which is considered as the “ongoing activity” [26]. The recorded ongoing activity () for the test dataset was derived by taking the difference between each trial’s response () and the trial-averaged response for 20 trials (), which is considered as an estimate of the visual evoked response without brain state effects:

(11)

From the system identification model, we calculated the difference between the predicted responses from the full model containing the stimulus and cortical state-driven pathway () and from the base model containing only the stimulus-driven pathway (), which we term the ongoing activity estimated by the model ().

(12)

By comparing these two, we obtained the :

(13)

The can vary between 0%, where the cortical state-driven pathway does not account for any of the observed ongoing activity, to 100%, where the cortical state-driven pathway completely accounts for the observed ongoing activity.

Quality of receptive field estimates

We estimated the effective structures of the receptive fields by convolving the filter function (c) with the map function (w), which we had introduced as “receptive field restorations” in our previous work [34]. For simplicity these restorations will be termed as “estimated receptive fields”. To quantify the statistical quality of estimated receptive fields, with or without incorporating brain state fluctuations, we calculated the z-score at the peak time lag of the receptive field using:

(14)

(15)

Here, is the i,j-th filter weight for the peak time lag of the receptive field, is the mean value of all the filter weights in the peak time lag, and is the standard deviation of the filter weights in the zero-th time lag of the spatiotemporal receptive field. Here z is a 2D array with values z_i,j corresponding to each filter weight in the peak time lag. The variance of these values (Eq 15) was utilized to comparatively evaluate the qualitative improvements in receptive fields at the peak time lags between models.

In the calculation above we used from the zeroth time lag as a measure of background noise of the estimated receptive fields (Eq 14). Since this lag is only nominally zero, but actually represents a time bin of 13.3 msec, it seemed prudent to check if there might be significant filter weights present at this lag depending on whether we incorporate brain state effects. To do this we confirmed the similarity of in the zeroth time lag of the estimated receptive fields before and after incorporating cortical state effects by using the Wilcoxon rank-sum test (S5 Fig).

Dimensionality reduction of the filtered LFP power spectra

To examine the LFP frequencies of importance for different neurons, we further analysed the power spectra of recorded LFPs filtered by the estimated temporal filters for neurons which showed a VAF improvement >5% (S6 Fig). We then used a UMAP python package [54] for nonlinear dimensionality reduction of the normalized power spectra (magnitude varying from 0 to 1), with parameters n_neighbors = 20, min_dist = 0.1, to obtain a high-dimensional graph. The number of dimensions of this graph is the length of the normalized power spectrum (i.e., 100 in this study). This graph was then used for the Louvain community detection algorithm [55] to identify distinct clusters (resolution = 1.0). The resolution parameter was selected to maximize the modularity score which is a measure of the quality of clusters (S7 Fig). For visualization purposes, we projected the high-dimensional graph into two dimensions using UMAP. We compared the variability ratios and the VAF improvements of neurons in different clusters provided by Louvain clustering, to assess differential effects of different LFP power spectra.

Trial-to-trial response variability

We often observed single neuron spiking responses which differed across repeated presentations of the same stimulus. We quantified the extent of these trial-wise differences by a “variability ratio (VR)” (see Methods) which compares the similarity of individual trial responses to the trial-averaged response. Fig 4A shows spikes from an example neuron with a relatively low variability in response to repeated presentation of a 5 sec natural image movie. For this neuron the trial-averaged response and individual trial-wise responses were comparatively similar, yielding a low VR of 0.31. However, another example neuron as shown in Fig 4B exhibited much greater variability (VR = 0.87).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Trial-wise response variability of neurons.

A) Spiking activity of a neuron with a low trial-to-trial variability (VR = 0.31, the same neuron as in Fig 2A). B) Spiking activity of a neuron with a higher trial-to-trial variability (VR = 0.87, same neuron as in Fig 2B) recorded simultaneously with the neuron in panel A. C) Scatter plot for the population of neurons (n = 154) showing firing rate (FR) plotted against VR. Correlation coefficient between FR and VR is -0.21 (p = 9.9e-03). D) Distribution of variability ratio across the population of neurons (n = 154). E) Distribution of variability ratios of neurons across different datasets containing at least 2 simultaneously recorded neurons. In E, each data point on a column shows VR of a simultaneously recorded neuron in that dataset. In C, D, and E, data points from example neuron 1 is shown in purple and neuron 2 in red.

https://doi.org/10.1371/journal.pcbi.1013661.g004

Comparing the two example neurons in Fig 4A and 4B, one might suppose that higher VRs are found in neurons with lower firing rates. To illustrate the relationship between VRs and mean FRs across the sample population, a scatter plot is presented in Fig 4C for the 154 neurons. While there is little evident relationship, statistical analysis shows a slight though significant negative correlation between the mean FRs and VRs (n = 154, r = -0.18; p = 0.03), suggesting that neurons with a low mean FR have a (slight) tendency to exhibit a higher VR and vice versa.

The VRs across the sampled population of neurons (n = 154) are displayed in Fig 4D, showing VRs ranging from 0.31 to 0.98 with a mean of 0.73 (median 0.79) and standard deviation of 0.14 (inter-quartile 0.21). The high mean value of VRs suggest that the neural spiking activities are poorly predictable or explainable from the visual stimulus inputs alone. Traditional system identification models have usually ignored such variabilities and sought only to predict the trial-averaged firing rates [34].

It would seem conceivable that neurons with diverse VR values were those recorded at different times of higher or lower variability - in that case, one might expect that simultaneously recorded neurons would have similar VRs. Fig 4E shows a scatter bar plot of neuronal VRs for 32 datasets containing at least 2 simultaneously recorded neurons. In many instances, even the simultaneously recorded neurons showed very different amounts of trial-to-trial response variability, ranging from a maximum difference of 0.56 (dataset number 4 of Fig 4E) to a minimum of 0.01 (dataset number 3 of Fig 4E). This result shows that the sources causing the response variabilities can affect simultaneously recorded neurons differentially, and therefore neurons should be treated at an individual level to account for the observed response variabilities.

Cortical state dynamics

The fluctuations of cortical network state might give rise to trial-to-trial response variability [7]. We thus analysed MUA and deep layer LFPs, as indicators of cortical network state fluctuations, using indices based on the firing patterns of simultaneously recorded populations of neurons and the LFP power in different frequency bands.

During a synchronized brain state, MUA shows prominent “up phases” when neurons’ membrane potentials are depolarized and a majority of neurons in the local network are spiking, and “down phases” when neurons’ membranes are hyperpolarized with little spiking activity [7]. In contrast, during a desynchronized state, these coordinated fluctuations in local neuronal network activities are not seen. For example in the context of sleep, synchronized states are mainly observed during slow wave sleep, whereas desynchronized states are seen during rapid eye movement sleep [7,57,58]. We analysed MUA signals during intersweep periods of spontaneous activity when the neuronal firing patterns are not affected by visual stimuli. LFP signals show high power in low frequency ranges and low power in high frequency rages during synchronized states and vice versa during desynchronized states [7]. Since LFP signals (<100Hz) are less affected by high frequency firing due to visual stimuli, we analysed LFPs throughout a recording session to continuously measure brain states, complementary to MUA that provided a measure of cortical states only during the intersweep intervals.

We computed MUA firing rate (100ms bins) for each of the channels in the grey matter during the four inter-sweep intervals. In some intervals (e.g., Fig 5A, intervals 1, 3, and 4), firing activity showed clear synchronization across channels, responding mainly during “up phases” and being completely silent during “down phases”. However, firing patterns in interval 2 were much less synchronized to the repeated stimuli, indicative of a relatively desynchronized state. The average firing rates across the channels (Fig 5B) confirmed the above impression that intervals 1, 3 and 4 show more pronounced fluctuations, with higher peaks for the “up phases”, than interval 2. The synchronized vs. desynchronized phases are further confirmed by quantifying a global fluctuation index, GFI, [26], defined as the ratio of the standard deviation (SD) to the mean of channel-averaged firing rate (Methods, Eq 5), in Fig 5C. For additional examples, see also S8B, S8D and S8F Fig. Interval 2 showed a much lower GFI value than intervals 1, 3, and 4. Although these GFI values are calculated from responses when no visual stimuli were presented, previous studies show that the ongoing cortical state fluctuations can outlast visual stimulus presentations [26].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Cortical state fluctuations within data recordings.

A) Binned firing rate of MUA across 18 channels in the grey matter (100 ms bins) during inter-sweep intervals while presenting a uniform grey screen. B) MUA firing rate averaged across channels in panel A. C) Global fluctuation index (GFI, Eq 5) using channel-averaged MUA firing rates in panel B. D) Power spectral density (PSD) of a deep layer LFP (middle channel of layer IV) across a data recording session (from same dataset as A-D). E) Synchrony index (SI, Eq 6) using PSD of panel D. Red data points indicate inter-sweep intervals.

https://doi.org/10.1371/journal.pcbi.1013661.g005

Cortical state dynamics observed in terms of GFI could be obtained only at discrete times (i.e., during inter-sweep periods without any visual stimulation) during a data recording session. To identify network fluctuations in a more continuous manner, we analysed LFP signals spanning throughout a data recording session. Time frequency analysis of a deep layer LFP signal showed that the distribution of its power spectral density across frequencies varied greatly over time (Fig 5D, results from same recording session as for Fig 5A-C). We defined a synchrony index (SI) as the ratio of LFP power in low frequency ranges (0.5-15Hz) divided by the total power in low and high (0.5-100Hz) frequency ranges [49]. Overall, SI values in this example varied in a range approaching unity (range of 0.88 and 0.96) which could be due to the global cortical state under anesthesia. However, we observed continuous fluctuations of SI across time during the recording session (Fig 5E, also see S8 Fig for other examples) showing small rapid variations of network state dynamics which might result in the observed trial-to-trial spiking response variability. We observed a similar pattern of changes in the cortical state fluctuations from MUA and LFP when we overlaid them (Fig 5C and 5E) in all the data recordings. However, the statistical significance of correlation between the two measures varied in different data recordings (S8 Fig).

These fluctuations in GFI and SI show variations of the ongoing cortical state during a data recording session which might result in the observed neural response variabilities.

System identification model

To examine how cortical state variations might influence neuronal activity, we incorporated candidate brain state indicators in a system identification model optimized to give best predictive performance of the neuronal spiking responses to visual stimuli. We developed the model (Fig 3) in stages incorporating successively more indicators of brain state.

Stimulus-driven (base) model.

First, we consider the performance of the model for the stimulus-driven pathway alone (upper part of Fig 3), without incorporating brain state effects, which we term the “base model”. With the recorded neural data and the corresponding stimulus input, the model parameters estimated for the example neuron 1 are presented in Fig 6A. The estimated model explained well the responses of this neuron (VAF 48.9%). The estimated filter weights contained a central inhibitory region surrounded by an excitatory region peaking at short time lags, which is reversed at larger time lags. The map layer, parameterized as a Gaussian, localized the activity of the filter. This example neuron had a PReLU alpha value of 0.62, corresponding to a nearly linear relationship indicative of a simple-type cell. The receptive field (i.e., the result of convolving the spatiotemporal filter weights with the Gaussian map) appeared less noisy and had a more clearly delineated excitatory surround. The convolutional output from the map layer passed through the final output nonlinearity (exponent 0.96) to generate the final neural response.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Estimated parameters for the stimulus-driven pathway for two example neurons.

A) Estimated parameters from the stimulus-driven pathway for example neuron 1 (same neuron as in Fig 2B); spatiotemporal filter weights (top row), PReLU alpha, Gaussian map, Output nonlinearity and the final estimated receptive field (bottom row). B) Same as A, for example neuron 2 (same neuron as in Fig 2C).

https://doi.org/10.1371/journal.pcbi.1013661.g006

The base model estimates for the second example neuron provided a VAF of 5.23%, much lower than for neuron 1. Fig 6B shows the spatiotemporal filter estimate (top row), PReLU alpha of 0.05 (indicative of a complex-like cell), estimated gaussian map, final output nonlinearity with exponent 1.50, and the receptive field (bottom row). The estimated receptive field for this neuron was spatially Gabor-like, i.e., with elongated, alternating excitatory and inhibitory regions, which reversed polarity at larger time lags.

Neurons with VAF < 1% using the model consisting of stimulus-driven pathway only, were discarded from further analysis to help ensure reliability of the estimated model parameters.

We reasoned that the lower VAF for the second neuron could be a result of its higher trial-to-trial response variability (Fig 4A and 4B). Thus, we next incorporated the influence of cortical state indicators to see whether there are improvements in the model performance, i.e., predictive VAF accuracy.

Performance improvement with cortical states estimated using LFPs.

Since cortical state fluctuation patterns can be detected with LFPs recorded simultaneously with spiking activity [9,59], we incorporated LFP signals in a cortical state-driven pathway, parallel to the stimulus-driven pathway in our model architecture (Fig 3) to predict trial-to-trial response variability. An LFP signal itself can carry information related to the stimulus [60,61]. However, on a subset of neurons we find higher VAF accuracy in a model with both stimulus and LFP pathways compared to a model with a LFP pathway alone (S4E Fig), implying that the LFP carries additional information beyond that related to the stimulus alone.

For each of the neurons we re-trained the base model with the cortical state-driven pathway, and the optimization algorithm estimated a temporal filter to enhance LFP frequencies from a nearby channel related to the trial-wise variable response. Here we will first examine results in detail for the two example neurons, and then consider results for the sample population. The estimated temporal filter for the first example neuron (Fig 7A) had a relatively low gain (± 0.001) with little evident temporal structure. However, the LFP temporal filter for the second neuron (Fig 7B) was ~ 20 times larger in gain (± 0.02), with clear positive weighting of shorter time lags, and a negative peak around 25 ms. The filter weights for the first example neuron were relatively small for all the time lags and therefore the filter would dampen the LFP signals passing through it. For the second example neuron, the measured filter weights were relatively larger in magnitude, with an evident temporal structure, such that the filter could retain relevant frequencies for the neuron and dampen less important frequencies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Estimated parameters for the cortical state-driven pathways, for two example neurons.

A) Estimated temporal filter from LFP-driven pathway for example neuron 1 when using LFP only from the channel above. B) Similar to A, for example neuron 2. Note ordinate scales are same for A and B, to facilitate comparison. C) Frequency spectrum of LFP before (black) and after (grey) applying the temporal filter shown in panel A. D) Similar to C, for neuron 2. (These power spectra deviate from the conventional 1/f shape at low frequencies due to the high-pass filtering properties of the recording system) E) Estimated temporal filters from LFP-driven pathway for neuron 1 when using LFPs from all the grey matter channels. Dashed red lines indicate recording channel of the neuron. F) Similar to E for neuron 2. G) Estimated temporal filters from MUA-driven pathway for neuron 1 when using MUA from the above and below channels. H) Similar to G for neuron 2. Note ordinate scales are same for G and H. (See also S6 and S9 Figs for additional examples).

https://doi.org/10.1371/journal.pcbi.1013661.g007

To identify the relevant frequency bands of importance for each neuron we calculated the frequency spectrum of the LFP signal filtered through the estimated temporal filter. For the first example neuron, the filtered LFP had a similar power spectrum as the raw LFP, but with a broadband reduction in power (Fig 7C). The power spectrum for this filtered LFP had higher power in the delta frequency range (0.5-4Hz), comparable with the LFP-spike phase locking values for the neuron, showing an abundance of spikes phase-locked to the delta band (0.5-4Hz) (Fig 2C). For the second example neuron, the filtered LFP spectrum’s shape was markedly different compared to the raw LFP power spectrum, with greater power in the higher frequency ranges including beta (15–30Hz) and gamma (30–100Hz) compared to the lower frequencies (Fig 7D). This behavior was comparable with the LFP-spike phase-locking values for the neuron, showing spikes more phase-locked to the higher frequencies (Fig 2D). Overall, phase-locking strengths for the second example neuron had relatively larger values (0-0.6), and showed higher spike-LFP coherence (Fig 2B and 2C), compared to the first neuron (0-0.4),

Consistent with these observations, the predictive VAF accuracy for the second example neuron improved from 5.23% to 8.14% when the LFP pathway was included in the model. This improvement was higher than for the first neuron, which only improved from 48.9% to 49.1%. These observations suggested that the second neuron shows higher cortical state effects compared to the first neuron, as evident from its larger trial-to-trial variability ratio and higher VAF improvement with the incorporation of LFP signals in the system identification model.

To further examine how neurons varied in terms of the LFP frequencies filtered by the estimated temporal filters, and whether this was related to the VAF improvements, we examined the power spectra of filtered LFPs of all the sampled neurons which showed VAF improvements >5%. We excluded neurons with smaller VAF improvements since their temporal filter values were close to zero (e.g., Fig 7A) - the filtered LFP spectra for a representative subset of the remaining n = 61 neurons are shown in S6 Fig. We performed nonlinear dimensionality reduction of the filtered LFP power spectra using UMAP clustering (see Methods) to identify whether there are specific categories of spectra, and if so, their relationships to response variabilities and VAF improvements. We identified three distinct clusters (Fig 8A); one cluster showed a low-pass filtered LFP power spectrum (“class 1”: n = 22, Fig 8B, purple), while a second group showed a broader band-pass filtered spectrum (“class 2”: n = 15, Fig 8C, yellow) and a third cluster showed a narrow band-pass filtered spectrum (“class 3”: n = 24, Fig 8D, cyan). However, the VAF improvements and the variability ratios of the neurons in these three classes of filtered LFP power spectra were not significantly different (Fig 8E and 8F, p > 0.05). These observations suggest that different neurons can show a range of response variabilities and VAF improvements independent from the LFP frequency ranges that influence them.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Diversity of filtered LFP power spectra.

A) Scatter plot of filtered LFP power spectra in UMAP space, colored by Louvain cluster membership. Neurons with a VAF improvement > 5% (n = 61) were used in this analysis. B) Two example power spectra for class 1 (purple cluster in panel A). C) Similar to B, for class 2 (yellow cluster in panel A). D) Similar to B and C, for class 3 (cyan cluster in panel A). E) VAF improvements of neurons which belong to the 3 classes. F) Similar to C, for the variability ratios of neurons.

https://doi.org/10.1371/journal.pcbi.1013661.g008

In the above analysis, only the LFP from one nearby channel was used in the model. We evaluated further improvements in VAF by including LFPs from all the channels in grey matter (except the primary channel for the recorded neuron). Re-training of this model including the cortical state-driven pathway with all LFP channels provided separate temporal filters for each channel. Estimated filter weights for the first example neuron (Fig 7E) had lower magnitudes (± 0.008) for the first neuron compared to the second neuron (Fig 7F, ± 0.03). The filter weights showed clear spatial distribution for the second neuron, i.e., having higher magnitude weights at recording sites distributed around the primary channel for the neuron (red dashed line in Fig 7F). This spatial distribution of estimated weights was not observed for the first neuron (Fig 7E). The model prediction accuracy (VAF) improved by 2-fold for the second neuron (VAF 5.23% to 11.55%) while negligible improvement was observed for the first neuron (VAF 48.85% to 49.90%). Thus, these results support the idea that the variable portion of a neuron’s response might be caused by cortical state dynamics which are reflected in the LFP. Further, the second neuron which had a higher VR benefited more from the LFP-driven pathway, while showing strong coupling of the spiking response to ongoing network activity across nearby channels.

Contribution of MUA-based cortical state to model performance

Previous work has shown that average firing rate of a population of simultaneously recorded nearby neurons can be predictive of the observed trial-to-trial response variability of individual neurons [22]. Additionally, compared to LFP which carries information related to synaptic activity in the local network [62], MUA has been shown to carry specific information related to nearby spiking activity [22,63]. Hence, we evaluated whether the predictive performance for a single neuron’s spiking activity improves by incorporating MUA-based cortical state dynamics measured from two nearby channels (i.e., 100 μm above and below) in an additional parallel pathway in our model. In a preliminary analysis with a subset of neurons, we observed improvements after incorporating an MUA pathway - however, consistent with a previous study [8], this improvement was lower than the improvements with the LFP pathway (S4D Fig). MUA from the same channel was not used, to avoid potential contamination from the recorded neuron itself. Further signal “cleaning” was carried out by removing co-occurring spike signals in adjacent channels to avoid the effect of spikes from the same neuron appearing in multiple nearby channels (see Methods; S3 Fig). Alternatively, MUA signals from the same channel could have been used after removing the spikes from the target neuron. However, to keep the processing steps comparable to LFP, we removed MUA signals from the same channel. Preliminary analysis showed the sufficiency of using only two MUA channels - the predictive accuracy with just two channels was higher (or similar) compared to that using additional channels. The MUA filter weights located further from two channels were close to zero in magnitude (S3 Fig), so including them in the model can cause the predictive accuracy to drop due to the increase of the number model parameters. Therefore, we used MUA signals from only two nearby channels separated by 100 μm from the neuron’s recorded channel and re-trained the model with the stimulus-driven pathway along with the pathway containing LFP signals from all the channels.

The pair of MUA temporal filters from two channels for the first example neuron (Fig 7G) were very small (+0.006/ -0.004) compared to those for the second neuron (Fig 7H) (+0.04/ -0.02). The temporal filters showed prominent peaks at latencies around 20–30 ms for the second neuron, but this effect was not observed for the first neuron. The VAF accuracy further improved for the second neuron from 11.55% to 17.41%, while negligible improvement was observed for the first neuron (VAF 49.90% to 49.45%). Thus, the second example neuron benefited comparatively more from inclusion of the MUA-driven pathway, similar to the effect from incorporating the LFP-driven pathway. Estimated LFP and MUA temporal filters for additional example neurons are shown in S9 Fig.

Improvement in estimated spatiotemporal receptive fields

Given the observed VAF improvement for the second example neuron compared to that for the first, we examined whether we could also observe parallel improvements in the estimated receptive fields. We evaluated the estimated spatiotemporal receptive fields, while progressively incorporating the cortical state effects from both LFP and MUA signals.

Fig 9 shows, for the two example neurons, estimated receptive fields when using models with the stimulus-driven pathway alone (base model) in the first row, with LFPs from a nearby channel (Model 1) in the second row, with LFP from all the channels (Model 2) in the third row, and with the full model incorporating LFP from all the channels and MUA from two nearby channels in the fourth row (Model 3). For the first example neuron, we did not observe qualitative improvements in the receptive field (Fig 9A), which was not surprising in view of the absence of significant improvements in predictive performance (VAF = 48.85% with the stimulus-driven pathway alone and only VAF = 49.45% with the full model). However, for the second example neuron, we observed clear qualitative improvements in the receptive field after incorporating the cortical state effects (Fig 9B). This is evident from both the clear localization of the inhibitory and excitatory regions in the peak time lag at -26.7 ms, and better delineation of excitatory and inhibitory regions at the larger time lags (e.g., appearance of temporal reversal of the receptive field at time lag of -66.7 ms), after incorporating LFP- and MUA-driven pathways rather than simply using the stimulus-driven pathway (Fig 9B). This is also evident from the z-score values for the receptive fields of the two example neurons at the peak time lag. Neuron 1 had a z-score of 1.13 from the base model and 1.17 from model 3 (z-score improvement = 0.04), while neuron 2 had a z-score of 1.55 from the base model and 1.79 from model 3 (z-score improvement = 0.24). These examples suggest that for neurons with higher variability, accounting for the cortical state dynamics can not only improve the model predictive accuracy (VAF), but provide better visualizations of the spatial receptive fields and their temporal dynamics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Improvements in receptive field estimates with cortical state.

A) Improvement of spatiotemporal receptive fields with the incorporation of the brain state effects for example neuron 1. The base model is the system identification model with only the stimulus-driven pathway, Model 1 incorporates cortical state effects from one LFP channel located 100 μm away, Model 2 incorporates all the LFP channels and Model 3 additionally incorporates two nearby MUA channels. B) Similar to A, for example neuron 2.

https://doi.org/10.1371/journal.pcbi.1013661.g009

VAF and receptive field improvements across the population of neurons

The initial VAF of the n = 154 neurons with only the stimulus-driven pathway (base model) ranged from 0% to nearly 50%. The improvements in model predictive power for this sample population of neurons are shown in Fig 10. With the stimulus-driven pathway alone (base model), neurons exhibiting a lower variability ratio (VR) yielded higher predictive performance, whereas those with a higher variability exhibited lower predictive performance (Fig 10A). This inverse relationship between VR and the VAF predictive performance was confirmed statistically (Fig 10A, r = -0.89, p = 2.29e-54).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Improvements in predictive performance for the models with cortical state indicators, across the population of recorded neurons.

A) Scatter plot for the population of neurons (n = 154) showing the VAF from the stimulus-driven pathway against response variability ratio (VR). B) Scatter plot for the population of neurons showing VAF from model with LFP from the above channel, against VAF from only the stimulus-driven pathway. Line indicates 1:1 ratio. Data points for each neuron are color coded according to their variability ratio. C) Similar to B, with VAF from the model with LFPs from all channels. D) Similar to B, but showing VAF from model with LFPs from all channels and MUA from above and below channels. E) Improvement in VAF after incorporating cortical state effects in Model 3, plotted against response variability ratio for each neuron. F) Improvement in predictive VAF plotted against the for each neuron. captures the model’s capability in explaining the spontaneous ongoing activity (see Methods) G) Distribution of VAF improvements of neurons across 32 datasets, each containing at least 2 simultaneously recorded neurons. Each data point on a column shows a simultaneously recorded neuron in that dataset.

https://doi.org/10.1371/journal.pcbi.1013661.g010

Given the very different observed VAF improvements in two example neurons (Fig 9), we examined the improvements of VAF across the population of 154 neurons. We analyzed the gradual improvements in the VAF predictive accuracy when we incorporate LFP signals from a channel located 100μm away (Model 1, Fig 10B), LFPs from all the channels (Model 2, Fig 10C), LFPs from all the channels and also MUA signals from two nearby channels (Model 3, Fig 10D), compared to the base model (stimulus-driven pathway alone, abscissa of plots in Fig 10B, 10C and 10D). The data points of these scatter plots represent single neurons, color-coded according to variability ratio. These neurons showed statistically significantly higher VAFs compared to the base model using LFPs from a nearby channel (Fig 10B; VAF from Model 1 compared to base model: p = 9.53e-05, n = 154), LFPs from all the channels (Fig 10C; VAF from Model 2 compared to base model: p = 8.34e-07, n = 154) and with the complete model including both LFP and MUA parallel pathways (Fig 10D; VAF from Model 3 compared to base model: p = 9.95e-09, n = 154). In addition, we observed successive improvements in VAF from model 1 to model 3 (median VAFs for base model = 12.44%; model 1 = 17.17%; model 2 = 18.67%; model 3 = 19.82%; n = 154).

These improvements were greater for neurons with higher response variability, but relatively minor for neurons with a lower response variability (Fig 10E, Pearson’s r = 0.49, p = 1.52e-10). Overall, we observed improvements of the predictive performance (VAF) in the population of neurons improving from 0% to 40% (Fig 10E). Given the range of observed VAF improvements, we explored whether they are related to the amount of spontaneous ongoing activity explained by addition of the parallel cortical state pathway in our model, i.e., “”. For this measure we compare, for each neuron, the variable part of the response to the additional response predicted by Model 3 compared to the base model (i.e., with the stimulus pathway alone, see Methods). Higher indicates that the additional responses predicted by Model 3 compared to the base model are accounting for most of the variable part of the neuronal response. The observed overall VAF improvements correlated well with the values, i.e., the neuronal response variability was predicted well by our cortical state-driven pathways (Fig 10F: r = 0.85, p = 1.65e-44). Thus, these results illustrate the capability of the model to account for the observed trial-to-trial response variability by incorporating cortical state dynamics along with the stimulus-driven pathway.

Simultaneously recorded neurons in the population often exhibited quite different amounts of response variability (Fig 4E). Therefore, we explored whether these simultaneously recorded neurons also show different amounts of VAF improvements after accounting for the cortical state effects using LFP and MUA signals. Across the sets of simultaneously recorded neurons, we observed different amounts of VAF improvements ranging from a maximum difference of 27.2% to a minimum of 0.07% with a median of 4.39% (Fig 10G).

Given the observed apparent improvements in the receptive fields of the second example neuron (Fig 9B), we examined the extent of this improvement across our population of neurons. To quantify the apparent qualitative improvements in estimated receptive fields for the population of neurons, we calculated z-score values for the receptive fields at the peak time lag (i.e., -26.7ms; see Methods). The z-score values after incorporating cortical state dynamics (median 1.08, inter-quartile range 0.86) were slightly larger than those from the base model with the stimulus pathway alone (median 0.91, inter-quartile range 0.72), and the difference was statistically significant (Fig 11A, p = 1.38e-02, n = 154). Fig 11B plots the z-scores for the two models plotted against one another. Among the 154 neurons, 100 neurons (64.94%) have higher z-scores for the full model (dots above the diagonal line), while 54 neurons (35.06%) have lower z-scores for the full model (dots below the diagonal line). However, Fig 11C shows that the improvements in the receptive fields were not statistically correlated with the observed VAF improvements (r = 0.13, p = 9.6e-02).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Receptive field improvements with cortical state across the sample population of neurons.

A) Box plot of z-scores for the receptive fields estimated without and with cortical state-driven pathways. B) Similar to A, shown as a scatter plot. C) Scatter plot of z-score improvement and VAF improvement.

https://doi.org/10.1371/journal.pcbi.1013661.g011

Inference of cortical state fluctuations

Our evaluation of the effects of cortical states on response variability is based on the prevalence of ongoing cortical state dynamics within a data recording session. We confirm this by deriving a global fluctuation index (GFI, adapted from [26]) from binned MUA during the blank inter-sweep periods, and a synchrony index (SI, adapted from [64] from LFPs across each data recording session. These fluctuations showed a similar pattern when compared at the same time points (Fig 4, red dots in panel E vs. panel C data points, S8 Fig). Thus, the cortical state of the animals under anesthesia fluctuated continuously during the recording session, consistent with the finding that brain states vary across a continuum [7].

The estimates of cortical state could constrain the performance of our proposed model. We used simultaneously recorded MUA and LFP brain signals to quantify cortical state dynamics. Since MUA and LFP signals have been shown to carry information related to input to the local network and local spiking activities, respectively, [62,65,66], we expected that they might carry complementary or independent information about cortical states. Cortical state characteristics have often been regarded as slow fluctuations on a time scale of seconds or more [7] - these characteristics should be reflected in low frequency LFP signals. However, our estimated temporal filters for both LFP and MUA contain an optimal time window in the millisecond scale (Fig 7). Preliminary analysis showed that increasing the temporal window length by including more time points or including an additional parallel pathway for ultra-low LFP signals (<0.5Hz) did not bring additional improvements in predictive accuracies. Therefore, our millisecond scale time window may be primarily capturing rapid up-down phases [7,67] embedded within the slow fluctuations. These relatively fast dynamics could also reflect millisecond-scale resolution of brain state embeddings, including substates and rapid events shown in recent studies [68]. In spite of similar time scales of the estimated filters, incorporating the MUA pathway does yield improved predictive performance, suggesting that it provides some additional higher temporal resolution information related to cortical states.

Variation in LFP frequencies of importance

Different frequency ranges of LFP correspond to distinct behavioral states - for example, LFPs exhibit higher power in the low frequency bands during synchronized states present in slow wave sleep, and relatively more power in higher frequency ranges in desynchronized states of rapid eye movement sleep or wakefulness [7,69]. Across our sample of neurons, we observed a wide diversity in the estimated LFP filters, and consequently of the LFP frequencies effective for the cortical state-driven pathway (Figs 7A-D, S6 and S9). This result suggests that cortical state dynamics can affect individual neurons differentially.

The filtered LFP frequency bands from the system identification model aligned well and showed substantial similarities to values of phase locking strength computed using LFP signals and spiking responses (i.e., similar frequency ranges in Figs 2B, 2C vs 7C, 7D and in S9B vs S9C). This validated the capability of our model in estimating temporal filters to extract the significant LFP frequency bands when the whole LFP signal provided as an input to the model. Therefore, our approach works flexibly without requiring the use of LFP signals segmented into pre-determined frequency bands.

Previous studies in area MT of awake behaving macaque monkeys showed strong links of spiking activity to specific frequency bands of LFPs, including high frequency gamma (30–70 Hz) and low frequency delta (1–4 Hz) bands [8]. This difference from our results, in which we saw a range of relevant frequencies, could be due to various factors, including the difference of species, brain areas, and our approach of estimating filters applied to raw LFPs rather than wavelet coefficients.

The aforementioned discrepancy could also be related to differences in anesthetized vs. awake cortex, which might be relevant to many aspects of our findings. Previous work has demonstrated the prevalence of strong low frequency fluctuations in anesthetized cortex compared to an awake cortex where high frequency fluctuations are more prevalent [7]. Furthermore, neurons in anesthetized cortex usually have decreased firing rates, increased burstiness, and greater synchrony [70], while neurons in awake cortex usually exhibit irregular, desynchronized spiking, and have higher variability across trials [71]. Our findings from anesthetized animals may not be entirely generalizable to awake cortex, however, our modeling approach could also be applied to identify the effects of brain states on neural variability during wakefulness.

Cortical state-driven pathways

Our model architecture for cortical state pathways utilizes simple linear filters that are unique for individual neurons, and to infer cortical states from LFP and MUA signals. We evaluated several alternative approaches including filter-rectify-filter (FRF) cascades to capture envelopes of high frequency LFPs, using dimensionality-reduced LFP signals, and allowing the brain state modulation to act at intermediate stages along the visual processing pathway. However, the predictive accuracy of each of these was comparatively lower. Nevertheless it is conceivable that some more complex model architecture might have better incorporated these population signals, though at the expense of additional model parameters.

Another limitation is that we measured LFPs and MUA only for sites along our linear array, but signals from other columns and brain areas might carry relevant and distinct cortical state-related information. Including these additional signals, e.g., in parallel pathways in a model like ours, might further improve the prediction accuracies.

Other studies have used measures of cortical states derived from other brain signals including scalp EEG [72] and hippocampal LFPs [73]. Cortical state can also be reflected in behavioural signals such as whisker movements [74], locomotion velocity and pupil diameter [75]. Including such signals in parallel cortical state-driven pathways could enrich a model’s representation of brain state fluctuations. The extent to which these more elaborate models could boost the neural response predictive accuracy is uncertain since many of these brain state-indicative signals may correlate with one another [76]. Nevertheless, it would be interesting to examine whether using these behavioural signals in additional parallel pathways in a model like ours could further improve the predictive accuracies.

Spike contamination in LFP and MUA signals

A potentially important concern is that LFPs obtained from an electrode recording site could be contaminated by spikes from nearby neurons. Therefore, if we use the LFP from a given channel to predict the spiking activity of a neuron recorded from the same channel, that LFP signal could contain a significant contribution from the spike itself and yield misleading predictive accuracy. While previous work showed that although spikes can be removed from LFP using a complex spike removal algorithm [77], residual spike contamination effects may still remain [78]. Hence one remedy is to use spikes and LFPs from nearby but distinct recording sites [78]. Consistent with this, [8] found similar system identification model weights estimated using spike-removed LFPs versus LFPs 100 μm away from the spike recording electrode. Consequently, we excluded LFPs from the same channel as that of the recorded neuron, to mitigate effects of residual spike signals from that neuron.

The spike waveform of a neuron can be recorded several tens/hundreds of micrometers distant [79,80] and therefore, across several recording channels (S3A Fig). Hence, as described in the Methods, to avoid contamination of the modelled single neuron’s spikes on MUA signals, we removed the spikes of the single neuron from the MUA signals recorded on the nearby channels.

Issues in accounting for response variability

The present model did not completely account for the observed response variability (Fig 10F). Even with the incorporation of brain state-related signals in our model, the predictive performance remains well below 100% (Fig 10D). An elementary reason for predictive error could in part due to intrinsic neuronal noise, including cellular, electrical, and synaptic noise that arise at different stages of sensory processing [81]. Most sources of intrinsic noise would be relatively uncorrelated across neurons, and thereby contribute less to neuronal population signals. Hence, our model might not capture intrinsic noise as compared to brain state fluctuations that can exert significant effects on neuronal spiking variability.

Previous studies have found that neurons’ response variability increases across the visual hierarchy from retina to LGN to V1 and other higher cortical areas, and is also inversely correlated with the average firing rates across these three brain regions [50,82]. Consistent with this, we did observe a negative correlation between the neurons’ time-averaged firing rates and their response variability, which was relatively weak though statistically significant (Fig 4C). The observed weak correlation arises because our population contains only area 17/18 neurons, but not recordings across the visual hierarchy. However, if this correlation were greater, addition of a firing rate dependency into the model architecture might be useful.

Stimulus-driven pathway

The stages in our stimulus-driven pathway are easily interpretable and applicable to both simple and complex cells in the primary visual cortex [34]. The two nonlinearities, PRELU and the output power law of our model only captures a neuron’s simple/complex nature and final spiking response nonlinearities, respectively. However, important improvements in predictive performance might be gained by developing more elaborate model architectures to handle other known aspects of visual processing, particularly neural response properties such as gain control [83], surround suppression, adaptation, or second-order processing [84]. For example, a popular model for gain control involves divisive normalization, in which responses of a neuron are divided by a common factor that typically includes the summed activity of a pool of nearby neurons [51], and some form of a filter-rectify-filter model architecture is often proposed to underlie second order processing [85]. Some studies have reported that deep neural networks, e.g., with more filters and/or filter layers, can provide improved predictive power for cortical neurons’ visual responses [86–88]. While such approaches might be desirable and interesting, they would entail greater challenges in optimizing the additional model parameters while avoiding overfitting, and in the case of deep neural networks, would entail a substantial cost in interpretability.

Previous studies reported that the structures of V1 receptive fields could substantially change when measured in different contexts, e.g., with receptive fields becoming wider in synchronized cortical states and narrower during desynchronized states [89]. Preliminary analysis with a subset of our neurons revealed that in some cases the receptive field size increased with synchronized states, consistent with the [89] report, while others exhibited an opposite result - in any case, these effects were relatively small (S10 Fig). Therefore, in our system identification approach we treated the receptive fields as having a common structure and size throughout a given data acquisition session, regardless of cortical state variations. If such changes were shown to be a more serious effect, then this limitation could conceivably be addressed in future approaches by using a model architecture that incorportates cortical state effects inside the stimulus-driven pathway.

Improvements in receptive field estimation

Incorporating brain state fluctuations often led to improvements in predictions of neurons’ responses, and also to better estimates of RFs in terms of z-scores – however, across the sample population these two kinds of improvement were not statistically correlated (Fig 11). This discrepancy could be due to multiple factors. The training of our model is based on optimizing the accuracy in predicting neural responses - in doing so, the optimization process might focus less on fitting the most accurate RF. Conversely, inadequacies of our visual pathway model, which would degrade the overall predictive accuracy, could vary substantially across neurons, in a manner not necessarily related to the z-scores. Thus for example, a poor visual response model applied to a neuron with little brain state dependence, might give a “clean” (high z-score) RF estimate which is quite biased, resulting in poor predictive performance.

In addition, our z-score measure might not be an ideal measure of the quality of an estimated RF. For example, collapsing the spatial z-score values into a single number (Eq 15) in deriving our z-score measure would not be completely appropriate for some neurons. As a result, comparing quantitative improvements in VAF to qualitative improvements in RFs in terms of z-score might lead to the observed discrepancies.

Future directions

Our results have demonstrated varying amounts of response variability, and of improvement in predictive performance, for different neurons - most interestingly, even for simultaneously recorded neurons. This heterogeneity of brain state effects across neurons has not often been well explored at the single neuronal level in previous studies which mainly focused on global level brain states analysing LPFs or scalp-recorded EEG. Hence, it would be interesting to further explore this heterogeneity, by examining whether there are systematic differences between neurons affected differentially by brain state, e.g., receptive field properties or laminar location. This approach could provide insights to better understand the effects of brain states on different categories of neurons in the primary visual cortex.