Fig 1.
(A) shows the subject walking in the Woodchips terrain wearing the Pupil Labs binocular eye tracker and Motion Shadow motion capture system. Optometrist roll-up sunglasses were used to shade the eyes to improve eye tracker performance. (B) shows a sample of the data record, presented as a sample frame for S1 Video. On the right is the view of the scene from the head camera, with gaze location indicated by the crosshair. Below that are the horizontal and vertical eye-in-head records, with blinks/tracker losses denoted by vertical gray bars. The high velocity regions (steep upwards slope) show the saccades to the next fixation point, and the lower velocity segments (shallow downwards slope) show the component that stabilizes gaze on a particular location in the scene as the subject moves towards it, resulting a characteristic saw-tooth appearance for the eye signal (without self-motion and the associated stabilizing mechanisms these saccades would exhibit a more square-wave like structure). On the left, the stick figure shows the skeleton figure reconstructed form the Motion Shadow data. This is integrated with the eye signal which is shown by the blue and pink lines. The representation of binocular gaze here shows the gaze vector from each eye converging on a single point (the mean of the two eyes). The original ground intersection of the right and left eye is shown as a magenta or cyan dot (respectively, more easily visible in S1 Video). The blue and red dots show the foot plants recorded by the motion capture system. The top left figure shows the scene image centered on the point of gaze reconstructed from the head camera as described in the Methods and Materials section.
Table 1.
List of videos in this manuscript.
Fig 2.
Retinal vs head-centered optic flow.
Optic flow patterns (down-sampled) for a sequence of 5 video frames from S3 Video, altered for visibility in print. Head centered flow shows optic flow in the reference frame of the head mounted “world” camera, and represents optic flow free from the effects of eye movements. Retinal flow shows optic flow in the references frame of a spherical pinhole camera stabilized on the subject’s fixation point. Purple and white lines show the integral curves of the measured flow fields, determined by using the streamlines2 function in Matlab to make a grid of particles drift along the negation of the flow fields measured by the DeepFlow optical flow detection algorithm in OpenCV. The red trace shows the movement of the head-centered FoE moving down and to the right across the 5 frames. In contrast, note the stability of the retina-centered flow over the same period.
Fig 3.
Focus of Expansion velocity in head-centered coordinates.
Focus of Expansion velocity across all conditions (black histogram), as well as split by condition (colored insets). The thick line shows the mean across subjects, and shaded regions show +/-1 standard error.
Fig 4.
Head velocity and acceleration during locomotion.
Head velocity and head/chest/hips acceleration patterns of a single representative subject walking in the “Distant Fixation” condition (where they walk while maintaining fixation on a point far down their path). Each blue trace is the velocity or acceleration trace for a single step, normalized for the time period from heel-strike to the subsequent heel-strike. Acceleration traces derive from the triaxal accelerometers of the subjects’ IMU sensors, while velocity traces are derived from numerical integration of these signals. The white line shows the mean and the shaded region is +/- 1 standard deviation. Right and Left steps were computed separately for the Medial/Lateral data, and are shown in red and cyan respectively. The vertical solid lines show mean toe-off time, with the vertical dashed lines showing +/- 1 standard deviation.
Fig 5.
Spherical pinhole camera model of the eye.
The spherical pinhole camera model of the eye used to estimate retinal optic flow experienced during natural locomotion. A, B show a sagittal plane slice of the 3D eye model. A shows the eye fixating on a point on the ground (pink line shows gaze vector, black circle shows fixation point) as points in the upper (orange) and lower (green) visual fields project on the back of the eye after passing through a pinhole pupil. B shows a closer view of the sagittal slice of the eye model. C, D show the full 3D spherical pinhole eye model. C shows the 3D eye fixating a point on the ground (black crosshairs), with field of view (60 degree radius) represented by the black outline. Note that the circular field of view of the eye is elongated due to its projection onto the ground plane. Red and blue dots represent points in the right and left visual field, respectively. D shows the retinal projection of the ground points from C on the spherical eye model. Ground dot location in retinal projection is defined in polar coordinates (ϑ, ρ) relative to the fovea at (0,0), with ϑ defined by the angle between that dot’s position on the spherical eye and the ‘equator’ on the transverse plane of the eye (blue circle) and ρ defined as the great-circle (orthodromic) distance between that dot and the fovea of the spherical eye. The retinal projection has been rotated by 180 degrees so that the upper visual field is at the top of the image.
Fig 6.
Retinal optic flow during natural locomotion.
Retinal flow simulation based on integrated eye and full-body motion of an individual walking over real world terrain. Panels A-C are based on frames taken from S2 Video, which shows data derived from a subject walking in the Woodchips condition under the Ground Looking instructions (S10 Video). (A) shows a case where the fixation point (pink line) is aligned with ground projection of the eye’s velocity vector (green arrow). Middle circular panel shows simulated optic flow based on fixation location and body movement. Left and right upper circular panels show the results of applying the curl and divergence operators to the retinal flow field in the middle panel. Left and right bottom panels show the projection of the curl (left) and divergence (right) onto a flat ground plane. The green arrow shows the walker’s instantaneous velocity vector (scaled for visibility), which always passes through the maximum of the retinal divergence field(which always lies within the foveal isoline (blue circle)) (see Fig 7). (B) and (C) show cases where the fixation point is to the left or right of the eye’s velocity vector (respectively). Fixation to the left of the eye’s velocity vector (B) results in a global counter-clockwise rotation of the retinal flow field and positive retinal curl at the fovea, while fixation to the right of the eye’s velocity vector results in clockwise flow and negative curl at the fovea (Fig 7).
Fig 7.
The relationship between curl, divergence, and the angle between the eye’s velocity vector and the fixation point.
The x-axis in A & B shows the angle between the vertical projection of the eye’s velocity vector and the fixation point, with zero denoting that the eye’s instantaneous velocity vector passes directly through the point of fixation on the ground, negative values mean the eye will pass to the left of the fixation point, and positive values mean the eye will pass to the right. The Y-axis of (A) represents the curl of the retinal flow field at the fovea (foveal curl). Each dot represents a single recorded frame from S10 Video, with the different colors denoting data from the three subjects. The Y-axis of (B) shows the angle between the point of maximum divergence in the retinal flow field and the point of fixation, measured similarly to the angle between the eye vector and the fixation point on the x-axis.
Fig 8.
Deviation from initial fixation location over the course of fixation.
Initial fixation location is computed and tracked over the course of each fixation, and compared to current fixation for duration of each fixation. Median deviation value is calculated for each fixation. The histogram captures the extent of variability of initially fixated locations relative to the fovea over the course of fixations, with most initially fixated locations never deviating more than 2 degrees of visual angle within the fixation. Long tail of distribution likely arises from erroneously labeled fixations where there are multiple small saccades. Fig 1B. Histogram of directions of retinal slip during a fixation. Most of the values are downward, as expected by a gain of less than 1.0.
Fig 9.
A representative flow field during a perfect 250ms fixation (stabilized) and when undergoing 1 degree of slip.
An eye translation direction downwards and slightly to the left of fixation is examined. The top row shows the stabilized case (i.e. perfect fixation) and the second row shows the unstabilized case (i.e. retinal slip). The bottom row shows the difference between the two, calculated by simple subtraction. The two plots on the left show the flow fields for the stabilized and unstabilized cases, and are very similar qualitatively, with the focus of expansion slightly shifted in the opposite of motion slippage. The plots on the right are centered on the fovea, and show velocity fields, curl, and divergence values. During stabilization, velocities are low with 0 deg/s at the fovea due to simulated perfect fixation. For stabilization with slip, velocity at the fovea is 4.17 deg/s and velocities at other retinal locations are shifted either higher or lower depending on location, however the magnitude of this shift does not exceed the foveal slip. There are small changes in the curl and divergence fields but the patterns are very similar.
Fig 10.
Comparison of representative flow field during stabilization versus during a saccade (195 deg/s up and to the left).
The top row shows retinal motion while the eye is fixating and the second row shows motion during a saccade. The bottom row shows the difference for velocity, curl, and divergence plots between the motion during a fixation and motion during a saccade (that is, between the top and middle rows. The flow fields on the left show that when gaze is stable, motion is outwards, away from fixated location. During the saccade, flow is strongly influenced by the direction of the saccade (motion in the opposite direction is added). On the right side of the Figure, velocities are low with 0 deg/s at the fovea when gaze is stabilized. By contrast the saccade results in speed at the fovea equal to the speed of saccade (195 deg/s), with velocities at all other retinal locations increased by more than an order of magnitude. The saccade also results in increase in curl values by more than order of magnitude and alters the spatial structure of curl pattern. Similar to effect on curl values, divergence values change by order of magnitude, and spatial structure is altered. Note that the scale of the difference plot in the bottom row differs by a factor of over 10 in Fig 9.
Fig 11.
Divergence, curl, and spiral flow patterns.
Canonical examples of Divergence (expansion), Curl (rotation), and the combination of divergence and curl (which produces spiral patterns. In the case of flowfield showing pure expansion (A—positive divergence), flow velocity is proportional to position on the grid—e.g. a step in the positive X direction results in an increase in X velocity (and similar for the Y direction). In the case of pure rotation (B—Curl) flow velocity increases in the orthogonal direction of the step—so a step in the positive X direction yields an increase velocity in the positive Y direction (for clockwise rotation/positive curl) or the negative Y direction (for counter clockwise rotation/negative curl). Summing the velocities of the divergence and curl flow fields results in spiral motion (C).