< Back to Article

Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration

Fig 2

Overview of ZephIR algorithm.

A: Examples of input datasets, created with BioRender.com. ZephIR can track keypoints in various biological systems, including fluorescent cellular nuclei in a tissue and body parts that summarize a posture. Input dimensions can range from 3D (time, XY) to 5D (time, channel, XYZ). Colored dots indicate example keypoints to be tracked. B: Frame sorting schemes. A branch defines an ordered queue of frames to be tracked. Each branch begins at a manually annotated reference frame (orange), but subsequent parent (blue) and child (green) frames in a single branch can be sorted either by chronology (top) or by minimizing the similarity distance between each parent-child pair (bottom). C: Overview of tracking loss. Tracking loss is comprised of four terms: 1) (top left), overlap of local image features around each keypoint, sampled from the current frame and its nearest reference frame, 2) (top right), elastic connections between neighboring keypoints with varying stiffnesses based on covariance of the connected keypoints, 3) (bottom left), proximity to features detected by a shallow model selector network that takes in a number of existing feature detection software as input channels, 4) (bottom right), smoothness of temporal dynamics at each keypoint position. D: Overview of steps for manual verification and additional supervision. Users can verify tracking results as correct or identify incorrect results. After fixing a few key incorrect results, ZephIR can use those new annotations as well as the verified correct tracking results to improve tracking results for all other keypoints in that frame (and all its child frames).

doi: https://doi.org/10.1371/journal.pcbi.1012075.g002