Experimental design.

Fish swimming measurements were carried out on 6–9 days-old larvae a few millimeters in length in a cubic glass tank of dimensions 7 × 7 × 7 cm (see S3A Fig and Materials and Methods: (a) Instrumentation). Movies of the zebrafish were obtained using 3 synchronized high-speed cameras taking orthogonal images of the fish tank at 500 frames per second (fps) (S1A Fig). While two orthogonal axes are in principle sufficient to reconstruct the position of simple rod-like shapes, three axes provide redundancy to correct for small errors due to water refraction, and additional information for more complex shapes, such as a highly bent backbone that obscures parts of itself from certain view angles. We calibrated the cameras taking into account refraction at the water-glass and glass-air interfaces using a dot grid. We also estimated a projection function, modelled as a cubic polynomial, that maps 3-D coordinates in the tank to 2-D pixel coordinates for each camera (see Materials and Methods: (a) Instrumentation: Camera calibration: Mapping from lab to camera coordinates and S4 Fig). A 3 × 3 × 3 cm cubic region in the center of the tank overlapped the field of view of all the three cameras. For all our experiments, we exclusively recorded larvae swimming in this central cubic region of the tank to avoid edge effects from interactions between the fish and the walls. We used three orthogonally placed and diffused high-power (50 W) near-infrared LEDs (850 nm) for illumination outside the larvae’s wavelength sensitivity [33]. This illumination did not heat the water significantly yet compensated for the high frame rate necessary to capture maneuvers reliably and the small optical aperture (large f-ratio) necessary in order to achieve a large depth of field. For dark flash experiments, a white LED was used in addition to the infrared illumination, to provide the dark flash stimulus (see Materials and Methods: (a) Instrumentation: Experimental Setup and S3B Fig).

Zebrafish larvae were placed in the tank and allowed to swim in free swimming, acoustic startle, and dark flash contexts. We collected a total of 630 movies of larvae swimming in 3-D, with at least 100 movies from each type of experiment (free swimming– 303, acoustic startle– 162, dark flash– 165). Free swimming videos were recorded for a duration of 20 seconds each at an interval of 3 minutes. For acoustic startle experiments, an acoustic stimulus was generated by dropping a weight from a fixed height onto the platform mounting the glass tank (S3C Fig). A dark flash stimulus was generated by turning off a white LED placed near the tank for 10 s (S3B Fig). To increase measurement throughput, both types of stimuli were triggered automatically whenever the three cameras detected at least one larva in the 3 × 3 × 3 cm imaging region. Consecutive stimuli were separated by at least 4 minutes in time to prevent habituation of the larvae to the stimulus. This inter-stimulus interval (ISI) is significantly larger than an ISI of 15 seconds reported in Ref. [34] to induce habituation to acoustic startle stimuli. We found in our experiments that an ISI of 4 minutes was sufficient to obtain consistent larval responses to dark flashes in the form of O-bends [34]. On average, a video for acoustic startle and dark flash experiments was recorded every 10 minutes for an interval of 4 seconds. The stimulus was provided 1 second after beginning the recording, providing sufficient time (3 seconds) to capture the response. The setup is described in detail in Materials and Methods: (a) Instrumentation and S3 Fig.

The 3-D physical model of a larval zebrafish.

We represented a larva using a physical model. The physical model requires 12 3-D coordinates to define a larval pose: 10 equally spaced coordinates along its backbone, and 2 coordinates defining the centroid of its eyes. The physical model is composed of 9 segments connected by flexible hinges, and includes a larval anterior composed of head, eyes, and ventral region located by the first two segments and a posterior composed of seven tail segments (S1C Fig). The coordinates of the physical model are then used to render realistic 2-D larval projections (see Materials and Methods: (b) Template-based pose estimation: Rendering larval projections, Physical model of the larva for more details). We refer to these projections as “physical model images”. We previously found that 9 backbone segments were sufficient to converge behavioral analysis in 2-D [23].

A definition of the physical model in terms of the angles between its consecutive segments is more suitable for direct quantitative analysis of behavior. Thus, we also formulated an equivalent description of the physical model in 3-D by specifying a set of adjustable parameters encoded in the 22-parameter vector p = (x₀, y₀, z₀, θ₀, Δθ₁,… Δθ₈, φ₀, Δφ₁,… Δφ₈, γ₀) and a fixed parameter, l (fish length). x₀, y₀, z₀ determine the centroid of the head relative to the lab reference frame x_lab, y_lab, z_lab, and the Euler angles θ₀, φ₀, γ₀ determine the head orientation (yaw, inclination, and roll, respectively) in the lab reference frame. ϴ = (Δθ₁,… Δθ₈, Δφ₁,… Δφ₈) is the set of angles that give the orientation of each body segment and determines the fish shape. Here, Δθ_i and Δφ_i are the bending angles for the (i+1)^th segment in the lateral and dorsal planes, respectively, as measured in the fish reference frame X, Y, Z, defined by the head orientation θ₀, φ₀, γ₀ (see S1C Fig for angle nomenclature).

A larval pose can thus be uniquely identified using either of the following equivalent representations:

1. 3-D pose coordinates: 12 3-D coordinates in the cartesian lab reference frame x_lab, y_lab, z_lab that define the physical model;
2. 2-D projection pose coordinates: 2-D projections of the 3-D pose coordinates obtained using the projection function;
3. 22-parameter vector p and l.

(a) Instrumentation.

Animals. Adult AB genotype zebrafish larvae were obtained from the Zebrafish International Resource Center (ZIRC, Oregon). Larvae used in the experiments were obtained by breeding the adult fish in the lab and raised at 28°C in Danieau’s solution [68] until 6 dpf (days post fertilization).

Experimental setup and data collection. The experimental setups used for the 3-D experiments are shown in S3 Fig. Fish swimming measurements in 3-D were carried out in a custom-built cubic glass tank of dimensions 7 × 7 × 7 cm. Zebrafish larvae at 6–9 dpf and measuring a few mm in length were placed in the tank and were allowed to swim freely without external stimulus. Movies of the larvae were obtained at 500 fps using 3 synchronized high-speed cameras (Ximea xiQ MQ003MG-CM) aligned with their principal axes orthogonal to each other and intersecting at a unique point in the center of the tank. The cameras were synchronized with a waveform generator (33250A, Agilent), which generated a 500-Hz square wave to trigger the three cameras at 2-ms intervals with 0.1-μs timing tolerance. The cameras were aligned as follows. Briefly, a mirror was placed facing the camera to be aligned, with its surface along the nearest face of the cubic tank. The center of the camera lens seen in its reflected image was detected using image analysis. In order to align the principal axis perpendicular to the mirror (and hence to the face of the tank), the (detected) center of the lens was made to coincide with the center of the image by rotating the camera on its mount. Similarly, the other two cameras were also rotated. To ensure that their principal axes intersected at the center of the tank, we translated the cameras using the detected centers as reference. Infrared illumination at wavelengths >700 nm enabled the same imaging modality for all experiments, including dark flashes. We used 3 orthogonally placed and diffused infrared lamps (Chanzon, λ = 850 nm) with a high-power output (50 W) to compensate for the high frame rate and small optical aperture (large f-ratio) of the cameras, which was necessary in order to achieve a large depth of field.

Zebrafish larvae were habituated in the 3-D arena for 10 minutes after being transferred from the incubator. Movies of larval swimming were then collected in three experimental contexts: free swimming, acoustic startle, and dark flash stimulus. Acoustic startles were generated by releasing a small cylindrical weight from an electromagnet on the platform supporting the glass cube (S3C Fig). Vibrations generated in the tank from the impact of the weight on the platform were minimal (<70 μm, estimated from imaging of the dot grid target; see Materials and Methods: (a) Instrumentation: Camera calibration: Mapping from lab frame to camera coordinates). A dark flash stimulus was provided by initially habituating the larva to a white light LED for at least 4 minutes before instantaneously turning the light off for 10 s (S3B Fig). Automated stimuli were generated using an Arduino UNO board interfaced with MATLAB and the stimulus source (electromagnet/white LED) triggered using an NPN transistor.

It was rare for a larva to be in the field of view of all three cameras at the same time. Thus, a search for such a larva was conducted once every second and a stimulus was triggered only when one was detected simultaneously by all the three cameras. Since the camera parameters that project a real 3-D point in the 2-D pixel space are non-linear, triangulating the position of the larva using the center of mass detected by the cameras required non-linear optimization. This process is computationally expensive in MATLAB and prevents monitoring the larvae in real time, once a second. Thus, we used the following algorithm to perform this real-time search:

1. Detect the centroids {C_⍺ⁱ}, where C_⍺ⁱ ∈ ℝ² is the centroid of the i^th larval projection on camera α, where α ∈ {1,2,3} and i ∈ {1,2,3,..,N} and N is the total number of larval projections detected in camera α’s field of view.
2. Determine R_⍺ⁱ, the ray of light incident on pixel C_⍺ⁱ of the camera sensor array, using Lookup Table R (see Materials and Methods: (a) Instrumentation: Camera calibration: Reverse mapping from camera coordinates to lab frame via Lookup Table R). By construction of the lookup table, this ray of light should have originated from a point coinciding with a larva in the tank.
3. Determine all the tuples (R₁ⁱ, R₂^j, R₃^k) iterating over all i, j, k in the three cameras.
4. For each tuple (R₁ⁱ, R₂^j, R₃^k), compute δ = max[d(R₁ⁱ,R₂^j), d(R₂ⁱ,R₃^k), d(R₁^j,R₃^k)], where d(R_α^m,R_βⁿ) is the minimum distance between the rays projecting on the pixel C_⍺^m of camera α and C_βⁿ of camera β.
5. If δ < ε, the rays (R₁ⁱ, R₂^j, R₃^k) originate from the same larva (ε is a small positive number determined heuristically).

We also performed 2-D behavioral experiments to develop a 2-D pose estimation workflow (see Materials and Methods(d) Modeling external datasets: 2-D pose estimation: Model training). We collected free swimming, acoustic startle and dark flash videos using the imaging setup described in Ref. [23]. Acoustic startles were generated using a speaker (40 Hz monotonic signal designed using MATLAB) and dark flash experiments were performed by habituating the larvae in white light for 4 minutes and suddenly turning the white LED off, using an Arduino UNO board. We collected a total of 289 videos (free swimming: 88 acoustic startle: 82, dark flash: 119), each consisting of multiple swim bouts (see Materials and Methods: (b) Template-based pose estimation: Optimization).

Camera calibration: Mapping from lab frame to camera coordinates. In the absence of the water tank, the mapping from a 3-D lab frame coordinate (x_lab, y_lab, z_lab) to a 2-D point on the camera frame (x_i, y_i) is a straightforward linear operation given the camera matrix , where i = {1,2,3} is the index of the camera [69]. The camera matrix P_i is a 4 × 3 matrix modeled as a product M_i K_i of a 4 × 3 ‘extrinsic’ matrix M_i and a 3 × 3 ‘intrinsic matrix’ K_i. M_i is determined by extrinsic parameters characterizing the rotation (quantified by the Euler rotation matrix, R_3x3) and translation (characterized by the vector T_3x1) of each camera relative to the lab coordinate system. The ‘intrinsic’ matrix K_i is determined by intrinsic parameters of the camera such as the focal length (in pixels), f_x, f_y; principal point coordinates (in pixels), μ₀ and v₀; and the skew coefficient of the pixels γ. A dot grid target (R2L2S3P4, Thorlabs) was used in the camera calibration process to determine all the camera parameters listed in the matrices M_i and K_i (see S1 Table). Several sets of images of the target at different orientations to the camera were taken (shown in S4A Fig) for camera calibration. The positions of the dots were detected using a custom-written code. Camera calibration was performed with the Computer Vision System Toolbox in MATLAB.

In fish swimming measurements, the projection from a 3-D lab coordinate of a point on the fish to its 2-D coordinate on the camera is no longer a linear operation because of the refraction caused by water and glass. There still exists a function f_i for camera i such that (x_i, y_i) = f_i(x_lab, y_lab, z_lab). However, an accurate analytical expression for this function is cumbersome and slow to compute. Rather than seeking the exact solution, we obtained an empirical function f_i by fitting images of the dot grid target. First, we collected a set of pictures of the dot grid target without the cubic tank (S4A Fig). Then, we placed the filled tank around the target while keeping the target at the same position as before and collected another set of pictures (S4B Fig). Replacing the tank surrounding the grid required removing the grid from its position. A kinematic base (KB50/M, Thorlabs) allowed us to position the grid in the same position and orientation after placing the tank. We obtained two sets of 2-D projection coordinates of the dot patterns, and for each camera, corresponding to the images taken with and without refraction due to the tank and water, respectively. Here, = (x_ni, y_ni) are the 2-D projection coordinates of dot n ∈ {1,2,3,..,N} on the camera i = {1,2,3} in the presence of refraction due to the tank and water, and are the 2-D projection coordinates of the same dot n on the same camera i in the absence of refraction (S4C Fig). The 3-D lab coordinates of the N dots were then reconstructed from , by minimizing the function . Because the lab coordinate system x_lab, y_lab, z_lab in S1C Fig is based on one of the cameras, with three cameras perpendicular to each other, we found that it was possible to simplify the projection function by calculating the two camera coordinates (x_i, y_i) independently. For instance, for camera 1, in the relation (x₁, y₁) = f₁(x_lab, y_lab, z_lab), with x₁ being perpendicular to y_lab, the value of x₁ was independent of y_lab. Thus, the projection function could be separated into two independent functions x₁ = f_1x(y_lab, z_lab) and y₁ = f_1y(x_lab, z_lab), each of which was fitted to a cubic function for interpolation to achieve satisfactory accuracy. As shown in S4D Fig, the fit function f_1x matched the experimental data (plotted as gray dots) well. With the empirical projection functions, we are able to reduce the mean triangulation error from 1.07 mm to 30 μm (S4E Fig), a small fraction of the body diameter of the animal.

Camera calibration: Reverse mapping from camera coordinates to lab frame via Lookup Table R. Having found the mapping from 3-D lab frame coordinate (x_lab, y_lab, z_lab) to 2-D point on the camera frame (x_i, y_i), it is necessary for analyzing larval swimming movies to construct the reverse mapping from (x_i, y_i) to the set of 3-D coordinates that project onto (x_i, y_i). Given that the laws of ray optics hold true for our setup, we expect a set of coordinates on a unique ray, a + Λr, to be projected, after refraction through the water-glass-air interface, onto a given pixel (x_i, y_i) of the sensor array of camera i (S5 Fig). Here, Λ parametrizes the ray, a ∈ R³ defines the origin of the ray in the lab reference frame at Λ = 0 and r ∈ R³ is the directional vector of the ray, defining the slopes along the 3 lab coordinate axes. The origin of the ray is defined as the point of intersection of the ray with the plane perpendicular to the principal axis of the camera, passing through the center of the tank. Since determining the set of 3-D points whose projection falls on the pixel (x_i, y_i) involves nonlinear optimization, it is computationally inefficient and thus motivates the use of a lookup table. For each camera, an independent ‘Lookup Table R’ for aiding in reverse mapping stores the vectors a and r for every pixel (x_i, y_i) of the camera sensor array. The values of a and r are determined by finding three random 3-D points in the tank that are projected onto the given pixel (x_i, y_i), using the non-linear mapping functions f_i(x_lab, y_lab, z_lab) above, and fitting a line through these three points. Lookup Table R is used in triangulating the center of mass of larvae in the tank (see pseudo code in Materials and Methods: (a) Instrumentation: Experimental Setup and Data Collection above) and in fitting the three camera images to a physical model of the larva (see Materials and Methods: (b) Template-based pose estimation: Physical Model of the Larva).

(b) Template-based pose estimation.

Preprocessing. The videos of larval swimming were preprocessed to improve the speed and accuracy of larval pose estimates. The complete preprocessing pipeline is described in (S7 Fig). The raw images were background-subtracted to increase the contrast between the foreground and the background and to reduce noise. The background image was created by computing the 90^th percentile intensity of every pixel over the entire video. This image was then passed through a Gaussian filter (window size: 5 × 5 pixels, standard deviation: σ = 1) to reduce high frequency noise further. The images of the larva were then segmented into binary images using a heuristically defined intensity threshold, and a bounding box was defined around the larva. The size of the bounding box was determined so that the largest 2-D projection of the larva (seen when the larva is closest to the camera) could be contained within the box. Since multiple projections of larvae were often detected in several frames, the larva of interest needed to be tracked. Tracking over multiple frames was performed by triangulating the centroid of the larva in 3-D coordinates in the current frame and detecting the nearest neighbor centroid in the following frame. For all the analysis downstream, we used this background subtracted and cropped image. Note that all images illustrated in the figures are these preprocessed images.

Physical model of the larva. We defined a physical model of the larva using 22 adjustable parameters (Fig 1C) and 20 fixed parameters (see below). We showed previously that 10 segments are sufficient to characterize the variations in the shape of the larval backbone [23]. Using 10 segments also allows for direct comparison of our results with the previous 2-D data [23]. The model is based on a scaffold comprising a chain of 9 rigid segments, allowing for 2 degrees of freedom for every joint connecting successive pairs of segments. Thus, the larval pose, representing the shape of its backbone, is defined in our framework by 16 parameters Δθ₁,… Δθ₈, Δφ₁,… Δφ₈ that represent the radial and azimuthal angles of each segment measured relative to the larval head orientation. Our choice of the number of segments defining the physical model is constrained only by computational power and the imaging resolution. In principle, the entire analysis can be reproduced for an arbitrarily large number of segments. The remaining six degrees of freedom—coordinates of the center of mass (x₀, y₀, z₀) and the yaw θ₀, pitch φ₀, and roll γ₀ of the larval head—establish the position and orientation of the larva in the lab reference frame, which is defined with respect to the bottom camera.

The anterior of the larva (eyes, head, and belly) is modeled as a set of ellipsoids, with the first rigid segment used as a scaffold (S6A Fig). We use the following 19 fixed parameters to define these parts of the larval anterior: brightness of the eyes, head, and belly (3 parameters); distance between the eyes (1 parameter); width, length, and height of the eyes, head, and belly (9 parameters); position of the eyes, head, and belly with respect to the scaffold segment (6 parameters). The position of the belly and head with respect to the scaffold segment can be defined uniquely using only two parameters each, since the belly and head have a constraint to be laterally symmetric about the scaffold segment. The position of the eyes is also defined using 2 parameters, because the centroids of the eyes are constrained on a segment perpendicular to the scaffold segment, with each centroid located at the end of the segment. The length of this segment is twice the distance between the eyes. For a given roll angle of the larva, this segment can be uniquely determined using two parameters: its perpendicular distance from the scaffold segment and its position along the scaffold’s length. The length of the larva L brings the count of fixed parameter to 20. L also serves to scale the absolute size of the eyes, head, and belly of the physical model of the larva during the second round of optimization (see Materials and Methods: (b) Template-based pose estimation: Optimization). A scaling parameter for the length of the larval anterior, l_A, which depends on L and the position of the larva in the tank, is used for rendering 2-D projections of the anterior in the first round of optimization (see Materials and Methods: (b) Template-based pose estimation: Optimization).

The larval posterior (tail) segments are constructed on the scaffold of the remaining 8 rigid segments, with a single fixed parameter scaling the width of the tail segments. This parameter was determined by averaging images of multiple larvae. Since our results were found to be robust to small changes to the width of the tail segments, the scaling parameter for the width is determined uniquely for all swim bouts and kept fixed for the entire analysis. In addition, the length of each tail segment is set to 1/9^th times the length of the whole larva L, allowing it to vary between larvae.

The larval anterior (head, eyes, and belly) and the posterior (tail segments) are rendered independently. For the anterior, we used a voxel-based 3-D model to generate 2-D projections that recreate the preprocessed images (S1F and S6A Figs). The intensity profile V(x_lab, y_lab, z_lab) of the voxels is given by max(N(μ_eye1,Σ_eye1), N(μ_eye2,Σ_eye2), N(μ_head,Σ_head), N(μ_belly,Σ_belly)), where N(μ,Σ) is the normal distribution with mean μ and covariance matrix Σ, uniquely defined by the fixed parameters. μ_k ∈ R³ is the center of mass of organ ‘k’ and Σ_k is a 3 × 3 covariance matrix that encodes the dimensions of k. To obtain 2-D projections of the voxels, we either perform an orthographic projection or a nonlinear projection using the camera matrix, depending on the round of optimization (see Image Analysis: Optimization and S1D–S1F Fig). Pixel intensities of the projections are obtained by summing over the intensity of all the voxels that are projected on the respective pixel. For example, for the orthographic projection with respect to the bottom camera (camera 1 in S1 Fig), pixel intensities P_zlab(x_lab,y_lab) are given by ∑_zlabV(x_lab, y_lab, z_lab).

For the larval posterior, the 2-D projections of the tail segments are rendered using a voxel-equivalent model to make the optimization more efficient in time and memory. In this model, the projections of the tail are constructed on the scaffold of the 8 rigid segments using a series of trapeziums connected by disks (S6B Fig). The intensity of each of these segments has a Gaussian profile along the width and a linear profile along the length, decreasing in the belly-tail direction. This construction is equivalent to the projection of a voxel-based model consisting of tapered cylinders connected by spheres.

Rendering larval projections. We generate realistic larval projections using either of two approaches differing in computational complexity: (i) Using Lookup Table P to render larval projections completely, (ii) Using Lookup Table P to render the larval posterior (tail segments), while the larval anterior is rendered by projecting individual voxels of the voxel-based model (see Materials and Methods: (b) Template-based pose estimation: Physical model of the larva) using the projection function (see Materials and Methods: (a) Instrumentation: Camera calibration: Mapping from lab to camera coordinates). We employ (i) during the first round of optimization (see Materials and Methods: (b) Template-based pose estimation: Optimization), where it is critical to render larval projections rapidly. On the other hand, (ii) is used during the second round of optimization and while rendering physical model images used to train the neural network (see Fig 1E), where generating accurate projections is critical while sacrificing on the computational speed.

In order to construct larval projections from Lookup Table P for a given parameter vector p ∈ R²², along with L, we render the larval anterior and posterior independently. The procedure for generating digitized larval projections during optimization using Lookup Table P involves determining the appropriate lookup table indices corresponding to the scaling parameter and orientation of the larval anterior and tail segments. The procedure to render larval anterior using Lookup Table P is described in detail in SI: Rendering larval anterior using Lookup Table P).

For rendering the tail, this procedure is not necessary because the indices of the tail segments, by construction of Lookup Table P for tail segments, are defined with respect to the orientation, length, and offset of the 2-D projections explicitly. We first compute the 2-D projections of the larval backbone (the chain of 8 rigid segments forming the posterior) using the non-linear projection functions f_i(x_lab, y_lab, z_lab) described above (see Materials and Methods: (a) Instrumentation: Camera calibration: Mapping from lab to camera coordinates) and determine their projected length. This gives the index corresponding to the scaling parameter l_T. The entry of Lookup Table P determining the orientation for the tail segments can be evaluated directly by computing the orientation of the 2-D projection of the individual tail segments forming the larval backbone. Lastly, the subpixel offsets can be determined by computing the fractional part of their x and y coordinates.

We also render the larval anterior directly using the voxel-based model in some cases, as described above in this section in (ii). Rendering every voxel individually using the non-linear projection function (see Materials and Methods: (a) Instrumentation: Camera calibration: Mapping from lab to camera coordinates) assures higher accuracy but is computationally expensive. While rendering the larval anterior in this way, a voxel-based model of the 3-D larval anterior is first generated (see Materials and Methods: (b) Template based pose estimation: Physical model of the larva). Every voxel is mapped from the 3-D lab-coordinates to the 2-D camera coordinates and the voxel intensities projected at every camera coordinate are summed, resulting in a 2-D image. Further, the intensity of the 2-D image is scaled such that the brightest pixel has a value of 255.

Optimization. The physical model of each larva is constructed from the unique chain defined by the parameter vector p ∈ R²²: p = (x₀, y₀, z₀, θ₀, Δθ₁,… Δθ₈, φ₀, Δφ₁,… Δφ₈, γ₀), in addition to the total length L of the larva. In order to compute the larval pose in any frame of a swim bout movie, we perform a search over the space of 22 adjustable parameters such that digitally generated larval projection images of the physical model best match the three views recorded by the cameras. A cost function, defined as the sum of the 2-norms of the three difference images (of the background subtracted image and the digitally generated projection) for the three cameras, is minimized globally using the patternsearch algorithm in MATLAB’s Global Optimization Toolbox. Each frame of the bout is analyzed in parallel on a node of an Intel Xeon Gold 6150 @2.7 GHz cluster.

The entire pipeline for the analysis of swimming movies is described in S1 Fig. We performed the optimization of the physical model against the collected data in two rounds. In the first round, a coarse estimate of the model’s parameter vector p is generated using a lookup table (see Materials and Methods: (b) Template-based pose estimation: Lookup Table P) that stores a large array of larval projections (S1E Fig). In the second round, fine optimization improves the estimate in a small neighborhood of p obtained in the first round (S1F Fig). Here, the larval projections are obtained by rendering the voxels defining the physical model at every iteration (see Materials and Methods: (b) Template-based pose estimation: Rendering larval projections).

Before initializing the optimization, we estimate the fixed parameter L from the length of the binary, segmented image of the larva from the bottom camera in the initial and final 5 frames of the swim bout, during which the larval backbone is expected to be straight. Specifically, we first find the best orientation angles of the larva (θ₀, φ₀) that minimize the cost function, assuming a physical model with a straight backbone and a larval length of 4.5 mm. L is averaged over all 10 frames.

In the first iteration of the coarse optimization, the adjustable parameters are initialized by an approximate guess of the centroid of the larva (x₀, y₀, z₀), triangulated using the centroids of the segmented larval images captured in the three cameras. This triangulation is performed using the nonlinear projection function by nonlinear optimization (fmincon function in MATLAB). Since recording is started before the stimulus is generated, we expect the larval backbone to be a straight line in the first frame and find the best orientation (θ₀, φ₀, γ₀) that minimizes the cost function. For this step, we use a brute force approach by iterating over a large number of orientations (θ₀, φ₀, γ₀) and render the corresponding digital projection of the larva (see Materials and Methods: (b) Template-based pose estimation: Rendering larval projections). The constraint on the shape of the backbone significantly reduces the volume of the search space and makes the brute force approach tractable. For the first frame, where the larval backbone is straight, we use the parameter vector obtained from the brute force approach as an initial guess and compute the parameter vector p₀ that minimizes the cost function, using the patternsearch algorithm in MATLAB. For all the succeeding frames, the initial guess is the optimized parameter vector from the preceding frame.

In the succeeding iterations of coarse optimization, Lookup Table P is used to render the larval anterior and the larval posterior, given the parameter vector p (see Materials and Methods: (b) Template-based pose estimation: Rendering larval projections). During rendering, for pixels shared by the projection of overlapping larval anterior and posterior segments, the higher pixel intensity value is selected. The estimate p₀ is computed by optimization of the cost function described previously. Since the Lookup Table P entries are rendered for discrete orientations of the larva, the estimate of p₀ is coarse grained, and we find that the estimates of roll angle γ₀ are often inaccurate. We thus pass p₀ through a round of fine optimization in the local neighborhood of p₀.

During fine optimization, we use p₀ as the initial guess and search for local minima in a small neighborhood around this guess. Here, an image of the larval anterior is rendered by performing a refraction-based non-linear projection of the voxel-based model for every iteration (see Materials and Methods: (b) Template-based pose estimation: Rendering larval projections). Moreover, the optimization is performed for the anterior and the posterior independently to reduce the volume of the search space. In order to check for convergence, we then perform a final iteration of optimization over the 22-dimensional space and optimize the anterior and the posterior together. Since fine optimization is extremely slow (10–12 minutes /CPU/frame), we manually selected ~150 swim bouts each from free swimming, escape response and dark flash experiments for this task. The fits obtained from coarse optimization were visually inspected to select the 450 swim bouts.

(c) Neural network pose estimation.

Generation of ensemble of physical model poses. A total of 35714 real larval poses were selected to generate the ensemble of physical model poses (Fig 1C), later used to generate the training dataset. These real larval poses are a subset of the poses estimated using the template-based approach (Materials and Methods: (b) Template-based pose estimation) that have a pose prediction accuracy greater than or equal to 0.9. We found that the set of larval poses was highly skewed, favoring poses with a nearly straight backbone (Fig 1B Biased data). Thus, before we could use these larval poses to generate the training dataset, we sought to resample the larval poses uniformly.

First, a kernel density estimate was performed using a Gaussian kernel to approximately infer the probability distribution of larval poses in the (<|Δθ_i|>, <|Δφ_i|>) space. In order to evaluate a reasonable bandwidth for the Gaussian kernel, we performed a search over 50 bandwidths uniformly selected in the range [0.01 rad, 0.1 rad]. The data were iteratively split into a test set and a training set in a 1:9 ratio. A kernel density estimate was evaluated using the training set and the iterable bandwidth. For each iteration, the log likelihood of the test set was calculated. The bandwidth that maximized the log likelihood of the test set was selected. The search for bandwidth was implemented using the GridSearchCV function of python’s scikit-learn (version 0.24.2) library. A 20-fold cross-validation was used to iteratively split the ensemble of real poses (Fig 1A) into test set and training set. The probability distribution computed using the kernel density estimate was used to sample 2500 poses that are uniformly distributed in the (<|Δθ_i|>, <|Δφ_i|>) space (Fig 1B Uniform data). To achieve this uniform sampling, we choose 2500 poses, each with a probability inversely proportional to .

Next, we generated an ensemble of 500,000 physical model poses, which are modelled on real larval poses. Using the 2500 poses in the uniform data (Fig 1B Uniform data), we computed their probability distribution ℘_angle in the 18-dimensional space defined by Δθ₁… Δθ₈, Δφ₁… Δφ₈, φ₀, γ₀. The kernel bandwidth used to compute ℘_angle was inferred using the same procedure as described in the previous paragraph. We sampled 500,000 vectors conditional on ℘_angle and to each vector, assigned x₀, y₀, z₀ and θ₀ sampled from a uniform distribution. The range of (x₀, y₀, z₀) was constrained by the dimensions of the imaging volume in the fish tank, while the range of θ₀ was chosen as (−π, π]. The estimate of the kernel bandwidth obtained using our maximum likelihood approach described in the previous paragraph results in a continuous probability distribution that is a good approximation of the underlying data. This estimate serves as a guideline to the user for a suitable choice of the kernel bandwidth. However, a larger bandwidth can offer a more diverse dataset, possibly improving the network’s generalizability. (S2 Table shows the effect of a larger kernel bandwidth on the network’s pose prediction performance.)

Preprocessing training dataset. Using the ensemble of physical model poses, we generated a training dataset of 500,000 physical model images to train the neural network. Generation of all the physical model images was performed in 7 hours using MATLAB on 25 Xenon Gold 6150 CPUs. For every physical model pose, a set of three physical model images was rendered, as seen by the three cameras in our setup (Fig 1D and 1E), using the voxel based model (see S1F Fig and Materials and Methods: (b) Template-based pose estimation: Physical model of the larva) and the projection function (S4 Fig). The 20 fixed parameters of the physical model (see Materials and Methods: (c) Neural network pose estimation: Physical model of the larva) were estimated using template-based optimization. While rendering physical model images, the intrinsic parameters of the voxel-based model were varied uniformly randomly by ±5% around their estimated values. All the physical model images were then pre-processed before passing to the neural network.

More specifically, the three 2-D larval projections corresponding to each of the 500,000 physical model poses were rendered onto three 141x141 images with pixel values scaled between 0 and 1. The larval projections were displaced at random by 0–20 pixels, both horizontally and vertically. To each image, we added Gaussian noise, whose mean and standard deviation were sampled randomly in order to mimic the background noise in the real images. The brightness at each background pixel of the projections of the bottom camera and side cameras is sampled from Gaussian distributions N(m_bottom, s_bottom) and N(m_side, s_side), respectively. For each projection of the bottom camera, the mean m_bottom of this Gaussian distribution is a product of two random variables—X_μ,bottom, drawn from a uniform distribution between 0 and 1/255 and Y_μ,bottom, drawn from a Gaussian distribution with mean 50 and standard deviation 10, while the variance, σ²_bottom of the Gaussian distribution is a uniform random variable between 20/(255)² and 70/(255)². For each projection of the side camera, the mean m_side of the Gaussian distribution is a product of two random variables—X_μ,side, drawn from a uniform distribution between 0 and 1/255 and Y_μ,side, drawn from a Gaussian distribution with mean 20 and standard deviation 10, while the variance, σ²_side of the Gaussian distribution is a uniform random variable between 10/(255)² and 60/(255)². Next, we rescaled the pixel values between 0 and 255, to mimic the format of the recorded images–unsigned 8-bit.

The sampling of physical model poses in the ensemble of physical model poses, varying of the 20 fixed parameters of the physical model and the downstream preprocessing of the physical model images makes our training dataset diverse, generalizing it beyond the features of our imaging data. It is better to have an overly diverse training set, even if not all its poses are weighted heavily in neural network outputs, than to have an under-representative training set.

Preprocessing real data. The real images were preprocessed before passing as inputs to the neural network. We performed background subtraction and cropping on every frame (see S7 Fig and Materials and Methods: (b) Template-based pose estimation: Preprocessing) obtained from the recorded videos to validate the trained neural network. The neural network accepts three images each of dimensions 141×141. Since the cropped 2-D larval images were of varying size, they were also padded appropriately with zeros to dimensions 141×141. In order to simulate background noise in the padded region, we first generated a background vector of pixel values along the pre-processed image’s border. This border was defined by the region with 5 pixels of the image boundary. The padded region was assigned pixel values from this vector, selected uniformly at random. This procedure of simulating the background noise works in most cases because the background distribution is uniform, i.e. the distribution of the intensity in the border is the same as that in any other region of the image not occupied by the larva. In some frames, there is a possibility of an irrelevant larva (from the context of pose estimation) occurring in the 5-pixel border. In order to avoid including pixels corresponding to a larva in the background vector, we excluded all pixels that have a value greater than 3σ with respect to the background vector’s mean. Notably, this procedure is not ensured a 100% success in simulating the background. We find rare cases where the padded region’s noise distribution does not represent the background noise distribution of the pre-processed image. The network predictions are inaccurate for such frames. One could overcome this issue by originally cropping 141×141 windows around the larva before performing pre-processing. However, such cropping dimensions may not be practical to achieve in cases where the larva occupies a region near the frame’s border. For our network evaluation, we challenge the network model by performing the above padding procedure for all frames, essentially assuming the worst-case scenario of the larva always being in the border region.

The comparison of template-based pose estimations and neural network pose estimations (see S8 Fig) was performed over 424 swim bouts (41756 frames), for which the computationally expensive template-based pose estimates were evaluated. For a fair comparison between the template-based pose estimations and the neural network pose estimations (see S8 Fig), we manually discarded all frames that had more than one larva in the cropped preprocessed images (see Materials and Methods: (b) Template-based pose estimation: Preprocessing) passed as input to the convolutional neural network. Since the network model is only trained with a training dataset involving single larvae, we observed that the network predictions are inaccurate when there are multiple larvae in the cropped preprocessed input image.

While analyzing our entire dataset of 630 movies (68181 frames) to assess the network’s performance (see Fig 2) and larval swimming kinematics (see Figs 4 and 5), the issue of having multiple larvae in the cropped preprocessed input image was resolved by first using a binary mask of the larva of interest and assigning all the background pixels an intensity of 0. Realistic background noise was then artificially rendered by the procedure described above. The larval mask was obtained by using the first round of optimization of the template-based pose estimation (see Materials and Methods: (b) Template-based pose estimation: Optimization), and rendering a physical model image based on the estimated pose. Pose estimation using the first round of optimization takes 2 minutes per frame with reasonably accurate pose estimates. This means that some of the inaccurate larval masks used to remove any irrelevant larvae also remove parts of the larva of interest. Neural network pose predictions corresponding to such cases are inaccurate and are automatically filtered out by setting a threshold on the accepted prediction score for analysis. We choose a threshold of 0.85 for our analysis in Figs 4 and 5.

Here we emphasize that we do not expect the user to manually discard frames having multiple larvae in the cropped preprocessed input image to generate pose estimations. We also do not propose generation of larval masks to remove irrelevant larvae from the cropped preprocessed input image. Both these procedures are performed in these cases for a fair assessment of the network’s performance. In order to estimate the larval pose directly from video recordings and only using the neural network, a user may use one or both of the two approaches below:

1. perform experiments with low larval density, such that individual larvae can be cropped easily;
2. automatically discard network outputs for cropped preprocessed input image containing multiple larvae, based on a cut-off on the pose prediction score.

Convolutional neural network model. Our network consists of an encoder of four bottleneck residual blocks as implemented in Ref. [35] with 32, 64, 128, and 256 channels, respectively. The convolutional operations involved kernel size = 3×3, stride = 1 and padding = 3. The encoder is followed by a decoder, consisting of three fully connected layers of dimensions 1×288, 1×144, and 1×72, respectively. The output of the decoder is passed through a sigmoid layer and scaled between 0 and 141. This scaled output (a 1×72 vector) is reshaped into three 2×12 matrices, corresponding to the 2-D projection of the larval pose on the three camera views. Every convolutional and fully connected layer is preceded by a batch normalization [70] layer and succeeded by a leaky rectified linear unit activation function [71] (negative slope = 0.01). The network was trained for 120 epochs using the Adam optimizer with a learning rate of 0.001 and a batch size of 300. The convolutional neural network was implemented using the PyTorch library (version 1.10.1) and trained in parallel on 2 NVIDIA V100 GPUs [72]. The network is completely trained in 5–6 hours. Network validation was performed on a single NVIDIA A100 GPU.

The loss function for training the network consists of two terms: ‘backbone loss’ and ‘eye loss’. The backbone loss is simply the root mean squared error between the 10 2-D projection pose coordinate predictions for the three camera views (3×2×10 vector) and the corresponding ground truth. The eye loss is similarly defined by the root mean squared error between the 2 2-D eye coordinate predictions for the three camera views (3×2×2 vector) and the ground truth 2-D eye coordinates, with a small modification: the eye loss is made symmetric with respect to the eye indices. More specifically, the eye loss is defined as: where subscripts 1 and 2 for the eye correspond to the 11^th and the 12^th coordinate of the 2x12 dimensional 2-D projection pose coordinates, subscripts p and g for the eye correspond to the network prediction and the ground truth and the ℒ_α corresponds to the root mean squared error for camera α. The total loss function is defined as (backbone loss) + λ*(eye loss), where λ captures the relative importance of the eye loss with respect to the backbone loss. For all our neural network experiments, we set λ = 5.

Triangulation of 3-D pose coordinates. The ten 3-D backbone coordinates are triangulated using the projection function and the 2-D projection pose coordinates obtained as the neural network’s output. Each of the ten points is triangulated independently by minimizing the least square error between the 2-D coordinate obtained from the neural network and the 2-D projection of the triangulated coordinate obtained using the projection function. We use the center of the tank as the initial guess for this coordinate and implement the optimization using the least_squares function of the scipy.optimize package (version 1.7.3) in Python 3.9.9.

The coordinates of the eyes cannot be triangulated directly because of the symmetry in the loss function with respect to the eye coordinates. The symmetry in the loss function causes an ambiguity in the indices of the eyes in the network’s predicted 2-D projection pose coordinates. For example, the network may predict the right-eye as the 11^th 2-D coordinate in its 2x12 output vector for the bottom camera, but the 11^th coordinate in the output vector for one of the side cameras may be the left-eye. In such a case, one cannot directly triangulate the 11^th coordinates in the three views to obtain the 3-D coordinate of one of the eyes. This ambiguity of eye indices is tackled by first permuting over all 8 combinations of eye indices and triangulating the 3-D coordinates of the two eyes in each case. The indices that result in the least triangulation error are accepted. This ambiguity of eye indices is tackled by first permuting over all 8 combinations of eye indices and triangulating the 3-D coordinates of the two eyes in each case. The indices that result in the least triangulation error are accepted.

The ambiguity in the indices of the eyes caused by the symmetry in the loss function with respect to the eye coordinates also means that there is no notion of left and right eye. This makes the assignment of roll angle γ₀ impossible. To assign the roll angle γ₀ in all frames, we assume that the larva never swims belly up. This reintroduces asymmetry in the two eyes, allowing for a unique γ₀ to be assigned for the given prediction of the eye centroid. The assumption could be relaxed to allow belly-up orientation if one examines belly vs. eye height.

(d) Modeling external datasets.

To test the robustness of our method, prediction was performed on 2-D datasets and 2-camera 3-D datasets that were collected by us and obtained from other groups [37–39]. Our 2-camera 3-D dataset was created by simply using our original 3-D dataset and discaring all images obtained from camera 2.

The workflow for pose estimation of the 2-D and 2-camera 3-D datasets was essentially the same as that used in 3-D pose estimation, with slight differences in the training dataset generation and the convolutional neural network model architecture (see S1 Text. Supporting Methods: Modeling external datasets for a detailed description of these workflows). Briefly, for 2-D pose estimation, the training dataset was generated using a new ensemble of physical model poses that was inferred using template-based pose estimation on our 2-D dataset (please refer S2D Fig). For 2-camera 3-D pose estimation, we used the same ensemble of physical model poses used for 3-D pose estimation (Fig 1). For each physical model pose, a single physical model image was rendered for the bottom camera. For the 2-camera 3-D external dataset [37], two images were rendered using the appropriate mapping from the lab frame to the camera frame for each of the two cameras (see S2C-S2F Fig, [37] and S1 Text. Supporting Methods: Modeling external datasets). Three training datasets were generated for pose estimation on (1) all 2-D datasets, (2) 2-camera 3-D external dataset, (3) our 2-camera 3-D dataset. A convolutional neural network model was trained for each of the datasets. The model’s architecture was similar to the one used for our 3-D pose estimation workflow (see Neural network pose estimation: Convolutional neural network model), with small differences to account for the change in the size of the network’s input and output vectors. The size of the fully connected layers in the decoder module were modified to obtain the required vector outputs representing 2-D pose projection coordinates (see S1 Text. Supporting Methods: Modeling external datasets). All network models were trained for 100 epochs.

The input images were preprocessed (see S2B and S7 Figs) and passed through the trained convolutional neural network (see S2H Fig) to obtain 2-D pose projection coordinates. For 2-camera 3-D pose prediction, the 3-D pose was reconstructed using the mapping from camera frame to lab frame (see S1 Text. Modeling external datasets and Materials and Methods(c) Neural network pose estimation: Triangulation of 3-D coordinates). The predicted poses were evaluated using Pearson’s correlation coefficient as described previously in our 3-D pose estimation workflow (see Fig 2C and Results: Convolutional neural network model trained on physical model images performs fast and accurate pose estimation on real images: Model evaluation).

(e) Statistical analysis.

Statistical significance of the observed difference between two distributions. We performed statistical tests to check the significance of the differences between two given distributions (see Convolutional neural network model trained on physical model images performs fast and accurate pose estimation on real images: Model evaluation, Figs 4, 5 and S8, and 3-D kinematic parameters reveal unique features of larval swims across the three experiments). To calculate the statistical significance that the mean of one distribution (with n samples) is greater than that of the other (also with n samples), we posed the null hypothesis that the samples belonged to the same underlying distribution. The p-value of the test was calculated as the probability of obtaining the observed or a larger difference of means μ₁ and μ₂, given the above null hypothesis. (Here, μ₁>μ₂, without loss of generality.) Combining the two samples into a single distribution, we used bootstrapping to generate a distribution of means using P. We sampled two distributions of size n uniformly at random and calculate the difference of their means. We repeated this computation of difference of means of two sampled distributions N = 10000 times, resulting in a distribution of difference of means, D_μ. The p-value of the test was computed as the cumulative probability p = Pr(D_μ≥μ₁−μ₂). We used the same approach to test the statistical significance of differences in the lower and upper quartiles of all distributions in Figs 4 and 5.

Statistical significance of asymmetry. To test the statistical significance of asymmetry in a given distribution between its positive and negative samples (distributions in 3-D kinematic parameters reveal unique features of larval swims across the three experiments and distributions in Fig 4), we calculated the probability of obtaining the observed fraction of positive and negative samples or a higher fraction of the same, under the null hypothesis that the data comes from a binomial distribution with p = 0.5. A similar test was used to test the predominance of dataponts in the second quadrant in Fig 5D with p = 0.25.