Fig 1.
Sampling noise confounds estimation of the correlation between model prediction and neuronal tuning curve.
The expected (true) spike counts in response to a set of 10 stimuli (solid green points) is perfectly correlated with a model (red points), yet owing to sampling error (neural trial-to-trial variability) the estimated tuning curve (green open circles) has correlation less than one with the model ().
Fig 2.
Simulation of the naive and unbiased
estimators for model-to-neuron fits at varying levels of
where m = 362, n = 4, and σ2 = 0.25.
(A) For true r2 = 1, at a moderately low SNR = 0.5, (blue) is on average 0.67 whereas
(orange) is on average 1.00. The bias of
(see Methods, “Bias of
”) is small relative to its variability (90% quantile = [0.93, 1.07] vertical bars) and to the bias of
. (B) Same simulation as A but at five levels of
(0, 0.25, 0.5, 0.75, 1). Lines show mean values of
(blue) and
(orange). Black line (beneath orange) shows true
; error bars show 90% quantile.
Fig 3.
Comparison of and
for estimating model-to-neuron fit across broad, relevant ranges of SNR, n, and m.
(A) Average performance of naive (blue) and corrected
(orange) as a function of SNR for a simulation where true
(horizontal black line), m = 362, n = 4, and σ2 = 0.25. Error bars indicate 90% quantiles. (B) Performance of estimators as a function of n, the number of repeats of each stimulus. Simulation like (A), except SNR = 0.5 and n is varied. (C) Performance as a function of m, the number of unique stimuli, for a low number of repeats (n = 4). Like (A), except SNR = 0.5 and m is varied.
Fig 4.
Comparison of with published estimators of
on the basis of simulated and real data.
(A) Low SNR (0.25) simulation where estimators on vertical axis are sorted from top to bottom by smallest MSE with respect to estimating . Traces show mean and SD of each estimator. (B) Same simulation at higher SNR (1.0) but same m, n. (C) Estimated fit of DNN to V4 data by
and published estimators. Each trace is the estimated fit of the model for one neural recording.
Fig 5.
Validation of confidence interval (CI) methods by simulation—example CIs for three methods.
Simulation parameters: n = 4, m = 40, true , dynamic range d2 = 0.25, trial-to-trial variability σ2 = 0.25, and target confidence level α = 0.8. Of 2000 independent simulations, CIs for the first 100 are plotted here for three different methods. CIs for all methods were calculated using the same set of randomly generated responses. (A) For the non-parametric bootstrap method, the upper end (orange) and lower end (blue) of the CI were almost always both below the true correlation value (0.91, green line), indicating an overwhelming failure to achieve 80% containment of the true value. (B) The parametric bootstrap method and (C) our ECCI method perform substantially better. Performance of all three methods over the full range of true
is plotted in Fig 6.
Fig 6.
Comparison of four methods for computing confidence intervals for spanning the full range of true correlation.
The fraction of times the CI contained the true value is plotted for each method (see line style inset) as a function of the true correlation value, , at 100 values linearly spaced between 0 and 1. The target α-level was 0.8. Open circles indicate that the fraction deviated from 0.8 significantly (p < 0.01, Bonferroni corrected).
Fig 7.
Applying our unbiased estimator with CIs to fit four example MT neuronal direction tuning curves to a sinusoidal model.
(A) Example neuron tuning curve (orange trace with SEM bars) with excellent fit to sinusoidal model (blue trace, ), high SNR and tight CI (parameters specified above plot panel). (B) Example neuron with poor fit to sinusoidal model but with a reasonable SNR and narrow CI that provide confidence that the neuronal tuning systematically deviates from the model. (C) Example neuron with poor SNR and wild estimate of
, which is reflected in large CI = [0.3, 1], suggesting that no conclusion can be made about how well the model describes any actual tuning here. (D) Example neuron with a seemingly reasonable
, but the low SNR and CI covering the entire interval [0, 1] reveals that this fit cannot be trusted.
Fig 8.
Confidence intervals (α = 90%, vertical lines) and point estimates (red dashes) for across all MT neuron direction tuning curves fit to sinusoidal model.
Data points are grouped into two intervals on the basis of (of the direction tuning curves) being less than or equal to or higher than the median value (3.5), revealing that lower SNR (left interval) is associated with much longer CIs.
Fig 9.
Relationship of naive and corrected
between fits of sinusoidal model to MT data.
Units with greater than the median across the population (
) are plotted in red and those less than or equal to in black.
Fig 10.
Applying to analyze performance of a deep neural network (DNN) in predicting V4 responses to natural images.
(A) For single-unit (orange) and multi-unit (blue) recordings, is plotted against the naive
. The relatively short α = 0.1 CIs (vertical bars) suggest that most of these correlation values are trustworthy. (B) The mean
value across multi-unit recordings (horizontal blue line) is significantly higher than that for the set of single-unit recordings (orange horizontal line; Welch’s t-test t = 3.7, p = 0.005). Because individual estimates are asymptotically unbiased, the group average inherits this lack of bias.
Fig 11.
Relationship of naive and corrected
with n, the number of repeats for V4 data.
Different colors indicate different recordings. Solid lines show the average estimate across random shuffling of trials (with replacement); vertical bars indicate SD. Dashed lines show average
.
Fig 12.
A comparison of our data quality metric, the signal-to-noise ratio estimator (Eq 16), across several datasets.
(A) The cumulative distribution of under the original experimental protocols. Traces with the same line thickness have similar numbers of n and m. Thick line (blue): MT data has n ≈ 10, m = 8. Medium lines (green, orange, red): V4 data has n ≈ 5, m ≈ 350. Thin lines: Allen Inst. data has n ≈ 50, m ≈ 120. The Allen Inst. data has two recording modalities: extracellular action potentials (spikes) on Neuropixel probes (NP) and two-photon calcium imaging (Ca). Both were recorded for the same stimuli: natural scenes and gratings (see Methods, “Electrophysiological data”). (B) Distribution of
after normalization with respect to the duration of the spike counting window (traces for calcium signal are not included). The normalization assumes that the original average spike rate can be applied to a 1 s counting window. But, if firing rates tend to decay over time, this will produce overestimates for recordings shorter than 1 s and underestimates for recordings longer than 1 s.
Fig 13.
The minimal SNR needed to reliably detect tuning as a function of m, the number of unique stimuli, and n, the number of repeats of each stimulus.
White arrows indicate the approximate location in (m, n) corresponding to the datasets used in Fig 12. Gray diagonal lines indicate constant number of total trials (n × m).
Fig 14.
Illustrative schematic of confidence interval estimation.
Given an observed estimate x* (green dashed vertical) from the distribution of the estimator X with CDF T(x) (solid black curve) associated with the parameter being estimated μ (black dashed vertical), the upper limit of the α-level confidence interval is the μU (purple vertical dashed) corresponding to the cumulative distribution of XU, U(x) (solid purple curve) that would generate values less than x* with probability α/2 (purple horizontal dashed). Thus U(x) is defined by U(x*) = α/2. Under the assumption the family of CDFs of X are stochastically increasing in μ, the event that T(x) ≥ α/2 corresponds to the event that μ < μU, thus the upper limit of the confidence interval contains the true value of μ. In graphical terms, if the black horizontal dashed line is above the purple, then it is guaranteed that the purple vertical dashed is to the right of the black. Thus these two events have the same probability: Pr(μ ≤ μU) = Pr(α/2 ≤ T(X)) = 1 − α/2. Here we have used generic symbols for illustrative purposes, but for reference to the proof (see Methods, “Proof of α-level confidence intervals”), the notation used here correspond as follows: ,
,
,
,
, and
.