Fig 1.
FlyLimbTracker uses active contour models to annotate the Drosophila body and legs.
(A) The body model is a closed snake consisting of 18 control points (c[0] to c[17]). Control points c[0] and c[9] correspond, respectively, to the posterior-most position on the abdomen and the anterior-most position on the head. All other control points are symmetric along the anteroposterior axis of the body (e.g., control points c[3] and c[15]). (B) Six leg anchor positions (yellow) between the coxa and thorax are defined empirically based on a linear combination of distances from the head-thorax boundary, the thorax-abdomen boundary, and a distance from the thoracic midline. These positions are then shifted depending on how the body model is optimally deformed to fit the contours of a specific animal. (C) The leg model consists of four control points including a thorax-coxa attachment l[0], the femur-tibia joint l[1], the tibia-tarsus joint l[2], and the pretarsus/claw l[3]. For simplicity, control points for only a single leg are shown. (D) In sum, 27 positions are calculated for each fly per frame: a centroid (0), anterior point (A), posterior point (P), as well as the body anchor, first intermediate, second intermediate and tip for each of the six legs. Our data labeling convention is as follows. Right and left legs are numbered 1 to 3 (front to rear) and 4 to 6 (front to rear), respectively. Each leg has four control points labeled 1 to 4 in the units digit that correspond the body anchor (1), leg joints (2 and 3), and claw (4). In each label, the leg number is shown in the tenths digit and the control point in the units digit. For example, the label “11” refers to the body anchor of the right prothoracic leg 1. For simplicity, only the control points for leg 3 are shown. (E) An example raw image of the ventral surface of a fly used for segmentation. (F) This image is first segmented using the parametric body snake consisting of 18 control points (red and blue crosses). (G) Subsequently, leg segmentation is initialized through automatic tracing from body anchor points to user-defined leg tips. From this initialization, an annotation is performed using open snakes consisting of four control points (yellow crosses). (H) Body and (I) leg segment tracking annotation for flies during a 455-frame (1.93 s) sequence. Annotation results (red) and the centroid in H or leg tip positions in I (blue) for each frame are overlaid.
Fig 2.
(A) The user manually indicates the approximate location of the fly’s body in an arbitrarily chosen video frame (t1). FlyLimbTracker then optimizes a closed active contour model that encapsulates the fly’s body in the correct orientation. The user then manually indicates the location of each leg’s tip. FlyLimbTracker then optimizes an open active contour model that runs across the entirety of each leg. (B) The user then runs FlyLimbTracker’s automatic tracking algorithm to propagate body and leg models to subsequent video frames (or prior frames if run in reverse). (C) Either during or after automated tracking, the user can look for tracking errors. After manually correcting these errors, the user can re-run automatic tracking. In each image, the frame number is indicated.
Fig 3.
Sensitivity of leg tracking to changes in spatial or temporal video resolution.
(A) Sample video image (top-left) after 2x (top-right), 4x (bottom-left), or 8x (bottom-right) spatial down-sampling. Adult female flies imaged are approximately 375, 187, 93, and 46 pixels in length in the 1x, 2x, 4x, and 8x spatial down-sampled videos, respectively. (B) Representations of the difference between successive images (t1 and t2 overlaid in magenta and green, respectively) for different frame rate videos after temporal down-sampling. (C-D) The number of corrections required per frame as a function of spatial resolution (C), or temporal resolution (D). (E-F) The average time required to semi-automatically annotate a single frame as a function of spatial resolution (E), or temporal resolution (F). In C-F, data for videos depicting a fly walking straight, turning, grooming its forelegs, head, or abdomen are shown in orange, purple, green, cyan, and red, respectively.
Fig 4.
Analysis and visualization of FlyLimbTracker leg tracking data.
Visualizations of leg segment annotation results for videos of a fly (A) walking straight, (B) turning, (C) grooming its forelegs, (D) grooming its head, or (E) grooming its abdomen. (A1-E1) Leg segmentation results (red) and joint positions (color-coded by frame number) are overlaid on the final frame of the image sequence. (A2-E2) Leg segment trajectories are rotated and color-coded by frame number. This permits alignment and comparison of leg movements across different datasets. (A3-E3) Joint and claw movements are represented in isolation. (A4-E4) The instantaneous speeds of each leg tip (claw) are color-coded.