Motion map dataset

The motion map dataset was collected de nuevo for this project. It was designed to capture a range of real-world motion patterns in typical scenes that contain both animate and inanimate figures.

Motion estimation.

The recordings were used to estimate the speed and direction of motion at each pixel in selected epochs of activity (Fig 1A). Motion estimation was performed using the MATLAB implementation of the deformable image registration algorithm described in Vishnevsky et al. (imregdeform) [33]. This method was selected due to its robustness to changes in scene illumination, ability to capture sharp motion edges, and capacity to simultaneously capture a wide range of speeds within a scene. As a sanity check, motion estimation was also performed using a standard algorithm for local optical flow based on differential motion estimation (the Farneback algorithm) and the global speed and direction results were overall similar but noisier so here we report the results from the more robust algorithm. To reduce errors due to pixel intensity noise, we smoothed the data slightly in both space and time prior to motion estimation. First, each frame was spatially median filtered with a 3x3 pixel filter. Next, we selected a target frame (f), computed the displacement field from the previous frame (f-1) and to the next frame (f + 1), and averaged the results together. Each frame pair was histogram equalized and then histogram-matched to optimize performance. Component motion estimates in the horizontal and vertical directions were converted to speed and motion direction. Motion direction is reported in degrees with 0 corresponding to rightward motion, 90 upward, 180 leftward, and 270 downward. While this method generally produced high-quality flow fields, manual inspection revealed consistent and significant motion noise in large featureless regions where the optimization is poorly constrained, most notably in regions of the sky. Sky regions were therefore excluded from analysis. Lastly, we measured the noise floor of the motion estimation algorithm by applying the same algorithm to sets of identical frames (i.e., frames for which the ground truth motion was zero). Across a set of samples from both recording sites, the maximum spurious motion speed was 0.14 pixels/frame (0.67 deg/s) and the 99th percentile was 0.02 pixels/frame (0.096 deg/s). We therefore conservatively limit our analysis to speeds that exceed 0.5 deg/s. The motion above this threshold likely still includes some spurious motion (see regions of detected motion around the background buildings in Fig 1A, upper row), however, higher thresholds appeared to remove legitimate slow speed motion.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Motion maps and distance maps with figure-ground annotations were generated for analysis.

A) A dataset of motion maps was created by applying motion estimation to grayscale movies (left), yielding estimates of motion speed (left) and direction (right). The red outline indicates the central frame for which motion was estimated. B) Epochs of visual motion for analysis were sampled from each 4-minute-long movie by selecting two local maxima in the mean speed and then sampling the frames associated with the peak, onset, and offset of motion. C) RGB images (left) and distance maps (right) were obtained from the dataset described in [32]. D) Figure region (red) and ground region (blue) annotations of selected images from the distance and motion map datasets were performed by professional human annotators. Black pixels indicate excluded sky regions and figure-ground boundaries.

https://doi.org/10.1371/journal.pcbi.1013573.g001

Distance map dataset

The distance map dataset was derived from a publicly available natural scenes database, for which the methods are described in detail elsewhere [32]. This database constitutes color images and distance maps captured at 98 locations around the campus of the University of Texas at Austin in Austin, Texas. Relevant information is briefly summarized here.

Figure ground annotation

A professional image annotation company (BUNCH) was employed to generate figure labels for both the motion map and distance map datasets. This approach was selected to ensure high-quality annotation borders and segmentations that reflect human interpretations of the scenes. For annotation of the motion map dataset, human annotators were instructed to view a 30-s long movie clip encompassing each epoch before beginning the annotation. The movie clip served to provide context to the still image to be annotated. Annotations were performed on the still images alone (i.e., the annotators were not provided with distance maps or motion maps to annotate). All images were gamma-corrected for viewing on a conventional display prior to annotation. In each image, annotators were instructed to identify 1–3 primary figure regions, which were defined as the “part(s) you would consider to be the main object(s) you’re looking at in the scene.” Aggregates of objects (e.g., a crowd of people, a cluster of leaves) were allowed to be considered as one figure as long as they were sufficiently close together. Once figures were identified, annotators used software tools to label all pixels within their selected figures. When possible, regions designated as figures were also specified in terms of a predetermined list of subcategories (e.g., person, bike, building, lawn). Annotators were instructed to label all other pixels in the images as either the remaining “ground regions” (i.e., surrounding area) or sky. A separate label was designated for sky so that sky pixels could be excluded from analysis. In the distance map dataset, the excluded sky regions constituted 0.6% of all pixels and in the motion map dataset it was 6.5%. However, in practice the number of actual sky pixels was higher because they were challenging to individually annotate. Example annotations are shown in Fig 1D.

Data quality screening

Each motion map, distance map and annotation was manually reviewed for quality. For the motion map dataset, of the 98 motion epochs identified, 12 were excluded due to data quality issues (e.g., motion blur, registration errors, poor annotation alignment), resulting in 86 motion epochs for analysis (totaling 258 motion maps because each epoch included 3 clips representing the onset, peak and offset of motion). For the distance map dataset, of the original 98 recording sites 6 were excluded due to similar data quality issues. Fourteen additional exclusions were made based on a pilot study in which a cohort of human observers performed manual annotations in the lab, based on the observation that observers could not reliably identify specific figure/ground regions. Most of these excluded images contained heavy vegetation (e.g., bushes, plants, etc.) that was tightly packed together, making the boundaries unclear or ambiguous. These exclusions resulted in 78 scenes for our distance map analysis. In the final datasets, the overall alignment between the figure labels and the scene content was high quality. However, to avoid cross contamination when computing visual statistics of figure and ground regions, we excluded any points within a 5-pixel buffer around all annotation borders from analysis.

Receptive field simulation

Spatial sampling.

To examine local statistics, circular regions on par with the size of MT receptive fields were sampled from both the motion map and distance map datasets. These simulated receptive fields (sRFs) were of four diameters (2.5, 5, 10, and 15 deg), which correspond to the receptive field sizes of MT neurons associated with eccentricities in the parafoveal and peripheral visual field, as the receptive field sizes of MT neurons roughly match their eccentricities [34,35]. SRFs smaller than 2.5 deg were impractical to simulate due to the limited spatial resolution of the datasets, and larger ones were impractical due to the field of view of the cameras used to create each dataset. We generated 800 sRFs from the motion map dataset and 800 from the distance map dataset (200 of each sRF size). For each sRF, a random scene was first selected, and then a random position in the scene was selected for the center of the sRF (in Fig 2, gray circles illustrate sRF placements). SRFs were discarded if more than 50% of the pixels contained missing measurements (e.g., sky, motion below threshold). We aimed to examine the statistics of visual features at figure-ground borders, so samples were also discarded if they did not contain at least 25% pixels designated as figure and 25% designated as ground. No restrictions were applied to the number of distinct figure regions within the sRF. Finally, for each sRF we selected a putative fixation point in order to simulate the effects of gaze (Fig 2 green dots), and sRFs without a valid fixation point at the appropriate eccentricity were also discarded. This process was repeated until the desired number of samples was obtained for each sRF size.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Simulated receptive fields (sRFs) and fixation points were used to sample from both datasets.

This figure illustrates these sRFs and how the associated fixation point can affect retinal speed and disparity (it is not a real scene from our datasets). After an sRF (gray circles) was identified, we assigned a random fixation point based on the scene’s saliency map (green circles) at the appropriate eccentricity (white dashed lines). A,B) For the motion maps, the calculated motion vector at the point of fixation was subtracted from the motion at all other points to simulate retinal motion (inset). In this example, the annotated figure region is moving leftward and the rest of the scene is stationary. Panel A illustrates the average speeds in the resulting figure and ground regions (s_f and s_g) for a stationary fixation point (such that the figure region speed is faster) and panel B illustrates the same for a moving fixation point on the figure (such that the ground region speed is faster). C,D) For the distance map dataset, the calculated distance at the point of fixation was used to determine the binocular retinal disparity. In this example, the annotated figure region is closer than the rest of the scene. Panel C illustrates average disparities in the resulting figure and ground regions (d_f and d_g) for a far fixation point and panel D illustrates the same for a near fixation point on the figure. In both cases, the figure region has a more positive disparity than the ground. Images are modified from artwork obtained on Pixabay.

https://doi.org/10.1371/journal.pcbi.1013573.g002

Experimental design and statistical analyses

Globally, figures tend to move faster, move more horizontally, and be nearer than their surroundings

Before examining statistical features within simulated receptive fields, we start by characterizing global frequency statistics of world motion and distance in our datasets. With respect to motion, we found that speed in the scenes was skewed towards slower speeds. Across all scenes, 69.7% of the estimated motion was at or below 0.5 deg/s – that is, the majority of visible content was stationary or slow-moving. Of the remaining estimates, the median speed was 1.2 deg/s (Fig 3A gray shading). This pattern is consistent with previous studies of natural motion statistics [44,45]. However, the speed distribution in our dataset was also slightly bimodal, with a second smaller peak at relatively fast speeds (~45 deg/s). The peak at fast speeds was caused by moving figures, whereas the slower peak was dominated by the ground regions but also contained stationary figures (Fig 3A red and blue lines).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Globally, figures tend to move faster, move more horizontally, and be nearer than their surroundings.

(A) The overall frequency distribution of all speeds (gray shading) is shown for the motion map data set, excluding motion below 0.5 deg/s. Red and blue lines show the distributions separately for pixels labeled as figure and ground, respectively. (B, C) The overall frequency distribution of motion directions and scene distances, as well as the figure and ground distributions, are plotted in the same manner as in A.

https://doi.org/10.1371/journal.pcbi.1013573.g003

Figures in the motion map dataset tended to be people, cars, small infrastructure, and animals (representing 15.7% of the valid pixel labels). The mixture of animate and inanimate figure classes is likely the cause of the broad distribution of figure speeds. The area surrounding the figures, however, was not completely stationary: inspection of the movies suggested that ground motion tended to be caused by wind-blown vegetation and animate objects that were not labeled as figures (e.g., people far from the camera), as well as moving shadows. On average, for the above threshold motions we found that the figures tended to move faster (median = 21.2 deg/s) than the ground regions (median = 0.8 deg/s), and this difference was statistically significant (z = 5219, r = 0.33, p<<0.001). However, we also observed notable overlap in the figure and ground speed distributions.

Motion direction had a tendency towards horizontal motion (Fig 3B gray shading) and this horizontal dominance was most visually pronounced in the figures (Fig 3B red line) as compared to the ground regions (Fig 3B blue line). This difference likely reflects the tendency of animate figures to move orthogonal to gravity along the ground plane, which will favor horizontal retinal motion for an observer also standing on the ground. Consistent with these observations, both the figure (z = 123900, p<<0.001) and ground direction distributions (z = 86063, p<<0.001) differed significantly from being uniform and the distributions were also significantly different from each other (k = 7.00x10¹³, p = 0.001).

The distribution of distances was also slightly bimodal, with a median of 12.9 m (Fig 3C gray shading). This pattern is generally consistent with previous measurements of natural scene geometry, and the relatively far median reflects the fact that this dataset was focused on views without much content near the camera (i.e., no objects closer than 3 m) [46,39,47]. Figures in the distance map dataset represented 23% of the valid pixel labels, and tended to be buildings and small infrastructure. On average, we found that the figures (Fig 3C red line, median = 4.9 m) tended to be nearer than the ground regions (Fig 3C blue line, median = 16.8 m), and this difference was statistically significant (z = -7725, r = 0.62, p<<0.001). However, it was not always the case that a figure was nearer than the surroundings. For example, regions of lawns and pathways were visible in front of figures like buildings. Indeed, as for the motion distributions there was substantial overlap in the figure and ground distance distributions.

In summary, the global statistics suggest that there is a tendency for visual figures to move faster and in a more dominantly horizontal direction than the ground regions, as well as to be nearer in distance. Given the strength of these patterns, it seems likely that visual input at the level of neuronal receptive fields will exhibit robust statistical regularities. However, these global patterns may be disrupted or altered by local figure-ground relationships as well as the effects of gaze direction and eye movements. We explore our simulation of the resulting receptive field statistics in the next sections.

Within simulated receptive fields, figure regions tend to move faster than the ground regions

We start by examining how the world speed statistics translate to local regions comparable to MT receptive fields (the sRFs). Consistent with the global statistics, we found that points associated with the figure within an sRF tended to have faster retinal speeds than the ground within that sRF. The mean figure-ground retinal speed difference (s_f - s_g) within an sRF was 0.28 log[deg/s] (5.7 deg/s) which was significantly greater than zero (t(799)=14.5, p<<0.001, D = 0.51) (Fig 4A). Across all sRFs, the average figure region speed was faster than the average ground region speed 68% of the time (CI: 65%-71%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Within sRFs, figure regions tend to move faster than the ground regions.

(A) The frequency distribution of speed differences between figure (f) and ground (g) regions in log deg/s across all sRFs is plotted. Positive values indicate sRFs in which the average figure moves faster than the average speed in the ground. (B) The average and 95% confidence intervals for these figure-ground speed differences are plotted separately for each sRF eccentricity. (C) For relative speed, we show the average frequency across all points in the sRFs (gray shading), and separately for the figure regions and ground regions (red and blue lines, respectively). (D) The mean figure and ground results from panel C are plotted as a probability ratio as a function of relative speed. Values greater than 1 (red background) indicate that points are more likely to be in figure regions and values less than 1 (blue background) indicate that points are more likely to be in ground regions. In C and D, dashed lines indicate 95% confidence intervals, but in some panels the intervals are barely wider than the line thickness.

https://doi.org/10.1371/journal.pcbi.1013573.g004

There is one notable situation in which it seems unlikely for figure regions to consistently cause faster retinal motion than ground regions: when the eyes are tracking the same visual figure that falls within the receptive field (see Fig 2B). In this case, the eye movement would stabilize the figure motion on the retina, resulting in a bias towards slower speeds in figure regions. We therefore hypothesized that a slower figure bias might be likely to occur in sRFs nearer to the fovea, because the small eccentricity of the sRF would increase the likelihood that the nearby salient features were part of the same object that comprised the figure region in the sRF. Indeed, we found that the average figure-ground speed difference varied significantly as a function of sRF eccentricity (Fig 4B, F(3,796) = 28.2, p << 0.001, η² = 0.1). We did not observe a slow speed figure bias at any eccentricity, however, the figure-ground speed difference was near zero for the lowest eccentricity sRFs and increased significantly with increasing eccentricity. All pairwise comparisons were statistically significant (Table 1). For the lowest eccentricity sRFs, the average figure speed was faster than the average ground speed only 52% of the time (CI: 44%-59%), whereas for the highest eccentricity sRFs, the average figure speed was faster than the average ground speed 82% of the time (CI: 76%-87%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Pairwise t-tests comparing figure-ground speed differences at each sRF eccentricity.

https://doi.org/10.1371/journal.pcbi.1013573.t001

In addition to these average speed differences, we also examined how the probability that a given point corresponded to the figure versus the ground varied from the slowest point to the fastest point within the sRF. First, we examined the frequency distribution of relative speeds in the figure regions and ground regions (Fig 4C, red and blue lines). This analysis is analogous to the global speed analysis (Fig 3A) but it focuses on the relative speeds with the sRFs, incorporating the effects of spatial sampling and gaze dynamics. Similar to the global speed distributions, the distribution of figure speeds was biased towards faster speeds than the ground speed distribution (the gray shading shows the frequency distribution of all points within an sRF). Accordingly, the ratio of the figure/ground probabilities varied: slow speeds were more likely to correspond to the ground regions and faster speeds to figure regions (Fig 4D). A slight reduction in the figure/ground probability ratio is observed at the highest speeds, which occurs because the frequency distribution of figure speeds peaks slightly below the maximum relative speed in the sRFs on average.

Within simulated receptive fields, figure regions tend to move slightly more coherently than the ground regions

We also considered that the horizontal motion direction bias observed in the global scene analysis (Fig 3B) might cause figure regions to have more coherent motion directions than ground regions in sRFs. Thus, we compared the circular variance of the motion directions within figure regions (v_f) and ground regions (v_g). Consistent with our expectation, we found that the circular variance in figure regions tended to be slightly lower than the ground regions, suggesting more coherent figure motion (Fig 5A). The mean figure-ground difference in circular variance was -0.03 deg² and significantly less than zero (t(799)=-3.2, p = 0.002, D = 0.11) – although the effect size was notably smaller than the one associated with figure-ground speed difference. Across all sRFs, the average figure region variance was lower than the average ground region variance 58% (CI: 54%-61%) of the time. We hypothesize that the lower coherence in ground regions results from a combination of complex motion patterns in background foliage as well as animate features that were not identified as figures, such as multiple people in the background walking in different directions. This coherence difference was again eccentricity dependent, with a larger difference associated with higher eccentricity sRFs (F(3,796) = 3.8, p = 0.010, η² = 0.01), although the effect size was again smaller (Fig 5B). Pairwise comparisons between the highest eccentricity (15 deg) and the two lowest (2.5 and 5 deg) were both statistically significant (Table 2). For the lowest eccentricity sRFs, the average figure region variance was lower than the average ground variance only 49% of the time (CI: 42%-56%), whereas for the highest eccentricity sRFs, it was lower 71% of the time (CI: 64%-77%). This pattern, however, was not clearly reflected in the normalized direction histogram. As a reminder, normalized directions were computed by first determining the dominant (most frequent) direction in the sRF and then rotating the histogram to align the dominant direction at 0 deg. While there was a strong tendency for all motion directions to align with the dominant motion (Fig 5C, 0 deg), the figure-ground probability ratio was relatively isotropic across motion directions (Fig 5D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Within sRFs, figure regions tend to move more coherently than the ground regions.

(A) The frequency distribution of differences in circular variance between figure and ground is plotted as a measure of coherence difference. Positive values indicate lower coherence in the figure region and negative values indicate higher coherence in the figure region. Note that the x axis is flipped. (B) The average and 95% confidence intervals for these figure-ground variance differences are plotted separately for each sRF eccentricity. (C) For relative direction, we show the frequency across all points in the sRFs (gray shading), and separately for the figure regions and ground regions (red and blue lines, respectively). 0 deg corresponds to the dominant motion in each sRF. (D) The mean figure and ground results from panel C are plotted as a probability ratio as a function of relative motion direction. Values greater than 1 (red background) indicate that points are more likely to be in figure regions and values less than 1 (blue background) indicate that points are more likely to be in ground regions. In C and D, dashed lines indicate 95% confidence intervals, but in some panels the intervals are barely wider than the line thickness.

https://doi.org/10.1371/journal.pcbi.1013573.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Pairwise t-tests comparing figure-ground circular variance differences at each sRF eccentricity.

https://doi.org/10.1371/journal.pcbi.1013573.t002

Taken together, the speed and coherence results suggest distinct statistical regularities in natural motion of figure regions at the level of neuronal receptive fields, particularly in the peripheral visual field. The specific prediction of this simulation is that retinal speeds falling in receptive fields that are faster are more likely to be figures (although clearly figures may also be stationary or slow). When figures are moving, their motion is slightly more likely to move in a coherent direction, providing an additional motion-based cue for figure-ground segregation within local receptive fields.

Within simulated receptive fields, figure regions tend to be nearer than the ground regions and have more positive binocular disparity

We next examined how the scene-wide distance statistics translate to binocular disparities in the sRFs. Consistent with the global statistics, the figure regions tended to be nearer than the surrounding ground region within an sRF, and therefore have more near (positive) binocular disparity. The mean relative disparity between figure and ground regions was 0.49 deg, reflecting the fact that figure regions were significantly and consistently closer than ground regions (t(799)=56.7, p<<0.001, D = 2.00) (Fig 6A). As with the motion statistics, this disparity difference was significantly eccentricity-dependent, with larger eccentricities associated with larger disparity differences (F(3,796)=6.0, p<<0.001, η² = 0.02) (Fig 6B). However, the effect size associated with eccentricity was quite small and the disparity difference was notably positive across all eccentricities. This difference increased slightly but significantly with increasing eccentricity (from 0.44 to 0.54 deg from the smallest to the largest eccentricity), with most pairwise comparisons statistically significant (Table 3). This makes sense – if a given point is closer than another point in space, the relative distance relationship, and therefore relative disparity, is not affected by the vergence state of the eyes. At each of the sRF eccentricities, the average figure disparity was nearer than the average ground disparity 98–100% of the time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Within sRFs, figure regions tend to be nearer than the ground regions and have more positive (nearer) binocular disparity.

(A) The frequency distribution of relative disparity between figure and ground across all sRFs is plotted. Positive values indicate sRFs in which the average figure distance is nearer than the average distance in the ground. (B) The average and 95% confidence intervals for these figure-ground disparity differences are plotted separately for each sRF eccentricity. (C) The frequency of binocular disparities is presented for figure regions and ground regions (red and blue lines, respectively). Gray shading indicates the probability across all points in the sRFs. (D) The mean figure and ground results from panel C are plotted as a probability ratio as a function of binocular disparity. Values greater than 1 (red background) indicate that points are more likely to be in figure regions and values less than 1 (blue background) indicate that points are more likely to be in ground regions. In C and D, dashed lines indicate 95% confidence intervals, but in some panels the intervals are barely wider than the line thickness.

https://doi.org/10.1371/journal.pcbi.1013573.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Pairwise t-tests comparing figure-ground disparity differences at each sRF eccentricity.

https://doi.org/10.1371/journal.pcbi.1013573.t003

The probability that a given point corresponded to the figure versus ground also varied from crossed to uncrossed disparities. Fig 6C shows the average frequency distribution of figure regions (red line) as compared to the ground regions (blue line) across disparity (the gray shading again shows the distribution of all points within an sRF). The figures were notably biased towards crossed, positive disparities (nearer distances), while the ground regions were biased towards uncrossed, negative disparities (farther distances). This pattern is consistent with the global distance statistics, which showed that figures tend to be nearer (Fig 3C). As such, the ratio of the figure/ground probability was highest for near points and reversed for far points (Fig 6D).

Overall, these results suggest a robust statistical regularity in natural retinal binocular disparities at the level of neuronal receptive fields that complements the natural motion statistics. When individual receptive fields are stimulated by surfaces at multiple distances at a figure-ground border, the closer surfaces are more likely to be figures, which translates to a strong disparity-based figure-ground cue: points with nearer disparity are more likely to correspond to visual figures. But unlike the speed statistics, this pattern was present and similar in magnitude across all eccentricities.

Relationship with neurophysiological recordings: Motion

When presented with stimuli moving at two different speeds within their receptive fields, individual MT neurons have response biases toward the faster component across a broad range of stimulus speeds [27,26]. Specifically, neuronal responses to these “bi-speed” stimuli are well-explained as a weighted sum of the responses to the component speeds, with a higher weight given to the faster speed component. Because our results suggest that motion associated with figures in receptive fields tends to be faster than motion associated with ground regions, a neural representation that favors stimulus elements with faster speeds may help to prioritize the representation of visual figures.

However, figure-ground segregation relies on populations of neurons, not individual ones. We thus performed a population-level simulation to explore how the interaction between individual neuronal response biases and figure motion statistics might play out at the level of a neural population. We start with a neural population with Gaussian tuning curves over log speed (Fig 7A). We assume that the receptive fields of the neurons in this population are spatially overlapping, such that together they encode the stimulus speed at a particular visual field location. We next consider a stimulus at this location that contains a figure region and a ground region (Fig 7B). The average speed in the figure region (s_f) is faster than the average speed in the ground (s_g) by some variable amount. Although there is also variation within each region, for simplicity we will just consider the averages. An individual neuron in the population (Fig 7C) will receive input consistent with both speeds (s_f and s_g), which individually would elicit response rates of r_f and r_g, respectively. We considered three possible strategies to determine the response of each individual neuron to the bi-speed input: prioritizing the faster speed, response averaging, and prioritizing the slower speed (Fig 7D). We implemented a simple population decoder (response-weighted average of all tuning preferences) for each strategy (Fig 7E). Fig 7F shows the decoded speed as a function of the speed of the figure region. The ‘prioritize faster’ strategy (green) – inspired by the empirical measurements from MT described above – reliably decodes the speed of the figure, whereas the averaging (yellow) and ‘prioritize slower’ (purple) strategies decode speeds that are slower than the figure. This example illustrates how natural statistics in figure-ground speed differences could be leveraged to prioritize the representation of the figure speed, although more sophisticated decoders may be able to also leverage these differences to recover both the figure and ground speeds (see [27]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Toy example of neural population responses to figure-ground borders with faster figure speeds.

A) Each line represents a neuron’s tuning curve within a population of neurons broadly tuned for stimulus speed. One example tuning curve is indicated with a thicker line. We assume the receptive fields of these neurons are spatially overlapping. Our model included 50 neurons with Gaussian tuning curves of standard deviation equal to 1 log[deg/s] and with means uniformly spaced in log speed. B) We consider a stimulus within this spatial location that contains a figure region (f) and a ground region (g). An illustrative histogram indicates that the average speed in the figure region (s_f) is faster than the average speed in the ground (s_g). C) An individual neuron in the neural population is shown – when stimulated by a figure-ground border this neuron will receive input consistent with both speeds (s_f and s_g). In isolation, these speeds would elicit response rates of r_f and r_g, respectively. D) We considered 3 possible strategies to determine the neural response to bi-speed input (R): prioritizing the faster speed such that the bi-speed response matches the response to the faster stimulus, averaging, and prioritizing the slower speed such that the bi-speed response matches the response to the slower stimulus. E) A simple decoding strategy was used to recover the stimulus speed from the population responses associated with each of these strategies: we computed the response-weighted average of each neuron’s preferred single stimulus speed. F) The decoded speed as a function of the speed of the figure region is plotted for eight example stimuli. For each stimulus, s_f was first selected and then s_g was set to be slower by a random scale factor ranging from 0.3 to 0.8. Line/marker colors correspond to the three strategies in panel D.

https://doi.org/10.1371/journal.pcbi.1013573.g007

While this fast speed bias has been observed in MT neurons when the stimulus moved with a broad range of speeds from 1.25 - 40 deg/s, the MT responses to even faster stimulus speeds were observed to be closer to averaging or a slow speed bias [26,27]. We hypothesize that the fast speed regime might represent situations in which retinal motion is likely dominated by self-motion and the resulting motion parallax on the retinas, such that faster retinal speed is no longer a strong discriminating factor for figures. Indeed, our current motion statistics reflect situations in which retinal motion is overall slow (less than 20 of our 800 samples contained retinal speeds >20 deg/s in both the figure and the ground). In this regime, motion-based cues might particularly support figure-ground segregation, for example, to plan the next eye movement that tracks a new target of interest. Indeed, the majority of MT neurons prefer speeds less than 20 deg/s [43].

A similar bias has been observed on the basis of motion coherence: MT responses to two overlapping stimuli moving in different directions are biased to the component direction that has a higher motion coherence [28]. We also found that figure regions tend to contain more coherent retinal motion, but this pattern was weaker than the pattern observed for speed. We speculate that since both figure region motion (e.g., locomotion of animals and people) and ground motion (e.g., wind-blown vegetation) may contain good directional coherence in the world, the ultimate amount of coherence difference on the retinas is limited. Nonetheless, our results suggest that neuronal biases towards higher motion coherence may also support the representation of figural motion.

Relationship with neurophysiological recordings: Disparity

A recent neurophysiological study suggests a similar connection between figure-ground distance statistics and MT responses to disparity, although the relationship is less direct. Chakrala et al. [29] found that MT neurons show a bias for one disparity over the other when presented with overlapping stimuli at two different disparities (“bi-distance” stimuli); however, the bias differed between animals. In one animal, the bias was always toward the nearer disparity regardless of the neuron’s disparity preference for single surfaces. In two animals, MT neurons were biased toward the surface that was most similar to the neuron’s preferred disparity [29]. Combined with previous findings that more neurons in MT prefer near disparities when stimulated by single surfaces (DeAngelis and Uka, 2003), a population level bias toward the preferred disparity could support the representation of near visual figures. However, a meta-analysis of disparity tuning in a different brain area (V1) suggested that population-level near-biases may be spurious [39]. Nonetheless, the figure-ground probability ratio for near disparities in natural scenes documented here is quite strong, so even a weak neural bias might be sufficient to support prioritization of figures. Additional investigation into both disparity tuning biases and visual statistics is needed to understand this potential connection more deeply.

Effect of eye movements

In this study, we simulated gaze and assumed that the foveal stimulus was being tracked perfectly with a smooth pursuit eye movement. To the extent that these simulations deviate from real gaze patterns, they may result in biased estimates of visual statistics. For example, it has been shown that the speed of eye movements can lag substantially behind the speed of stimulus motion [48,49]. In some scenarios, the point falling on the fovea may not be pursued at all. To examine how robust our findings are to different gaze assumptions, we re-ran the sRF motion analyses assuming that the observer does not pursue the object at fixation and instead keeps the eyes stationary. That is, we simply removed the pursuit simulation. The key motion statistics were largely robust to this manipulation. With a stationary eye, figure motion on the retina was faster than the ground motion 95% of the time and more coherent 81% of the time (as compared to 68% and 57% with pursuit). This analysis suggests that the main effect of pursuit eye movements was to reduce the motion differences between figure and ground derived from world motion, rather than to amplify these differences. Interestingly, both the figure-ground speed and coherence differences tended to increase with eccentricity even without pursuit (speed: F(3,796) = 91.4, p < 0.001, η² = 0.26, coherence: F(3,796) = 25.9, p < 0.001, η² = 0.09). We hypothesize that receptive field size differences, on their own, may be sufficient to drive some of the eccentricity-dependent effects in Figs 4B and 5B, which may then be further enhanced via eye movements.

Testable predictions for eccentricity-dependent neural differences

Our findings make new predictions about how neural responses to bi-speed and bi-distance stimuli should vary as a function of retinal eccentricity. We found that the figure-ground motion differences increased systematically with retinal eccentricity, but the magnitude of figure-ground disparity differences was relatively constant over eccentricity (Figs 4B, 5B, and 6B). If our working hypothesis that neural response biases are connected to figure-ground statistics is correct, then neurons should be more likely to prioritize fast speeds and coherent motion in peripheral and perifoveal vision as compared to foveal vision. Consistent with this hypothesis, the neurons that were observed to have a faster speed bias in Huang et al. [27] had receptive fields with an average eccentricity around 11 deg, which falls within the range where figure regions consistently tended to move faster. However, limited variability in the eccentricities from this dataset precluded a post hoc analysis to test for eccentricity-dependence. On the other hand, we would not expect to see a strong eccentricity-dependent effect for disparity-driven responses to bi-distance stimuli. Receptive fields tend to be smaller near the fovea, so it is also possible that neurons with foveal receptive fields are more likely to be stimulated by only figure regions or only ground regions, rather than a combination. If that is the case, we might expect foveal receptive fields to differ fundamentally in their coding of bi-speed and bi-distance stimuli. While neural recordings across a broad range of eccentricities and tuning properties can be challenging to obtain, these predictions highlight visual field eccentricities as a key element in the interaction between neural responses and figure-ground segregation during natural experience.

Limitations

There is always a concern when measuring natural statistics that findings are influenced by the measurement devices, processing algorithms, and sampling biases. While we used calibrated equipment, avoided image compression and nonlinearities, and confirmed that motion findings were robust to different algorithms, caveats still remain. For example, some amount of the motion in our data may be due to sensor noise and spurious measurements from the sky. Also in both datasets, the choices of samples were made by experimenters: these samples therefore may not reflect overall statistics of natural experience. One way to address the sampling concern is to use first-person wearable capture methods [50,51,39]. Such methods would also address the limitations of our saliency model, which does not capture potential differences in salience for fast moving or nearby objects, and the lack of self motion in our data set. Notably, statistics captured in this way would include effects of motion parallax, enriching our understanding of how motion cues support figure-ground segregations in a broader set of behavioral contexts. Given that different animals likely experience different visual statistics, it would also be desirable to have data derived from the visual experience, behavior, and neural responses of the same animal. However, here we wanted to focus on high-quality measurements of motion and distance in the world, which required the use of equipment that was not amenable to wearable systems. In addition, the nature of this study required semantic segmentations with very precise boundaries between figure and ground regions; we thus decided to use a professional annotation company to generate the figure-ground labels. This constrains the number of annotators that were used for each individual frame. Because figure-ground judgments can vary amongst individuals (Shishikura et al., 2023; Yamane et al., 2020), this makes the annotations subject to the biases of the annotator. Ideally, several annotators would draw figure-ground designations for the same frame, and the results would be averaged across individuals.