FIMTrack: An open source tracking and locomotion analysis software for small animals

Imaging and analyzing the locomotion behavior of small animals such as Drosophila larvae or C. elegans worms has become an integral subject of biological research. In the past we have introduced FIM, a novel imaging system feasible to extract high contrast images. This system in combination with the associated tracking software FIMTrack is already used by many groups all over the world. However, so far there has not been an in-depth discussion of the technical aspects. Here we elaborate on the implementation details of FIMTrack and give an in-depth explanation of the used algorithms. Among others, the software offers several tracking strategies to cover a wide range of different model organisms, locomotion types, and camera properties. Furthermore, the software facilitates stimuli-based analysis in combination with built-in manual tracking and correction functionalities. All features are integrated in an easy-to-use graphical user interface. To demonstrate the potential of FIMTrack we provide an evaluation of its accuracy using manually labeled data. The source code is available under the GNU GPLv3 at https://github.com/i-git/FIMTrack and pre-compiled binaries for Windows and Mac are available at http://fim.uni-muenster.de.


Introduction
FIMTrack is a tracking program primarily designed for acquiring locomotion trajectories of Drosophila melanogaster larvae but it is also possible to analyze the behavior of other small model organism such as C. elegans. The software is optimized for FIM images. FIM is an acronym for FTIR-based Imaging Method, whereby FTIR is the short form for Frustrated Total Internal Reflection. Details regarding the imaging technique can be found in [1,2]. Furthermore, a video tutorial demonstrating the biological usability of the FIM system and FIMTrack can been found in [3].
The main focus in this document is on the how-to use the software and on providing an overview of all calculated features and parameters.

Installation Instructions
FIMTrack is available for the operating systems Windows, Mac, and Linux. The source code is available under the GNU GPLv3 at https://github.com/i-git/FIMTrack and pre-compiled binaries for Windows and Mac are available at http://fim.uni-muenster.de.

How To Track
After starting FIMTrack the main graphical user interface (GUI) appears ( Figure 1). To track larvae which were imaged using FIM imaging, first click the Load Job button, navigate to the folder containing the TIFF or PNG image files, and select the image files which you would like to analyze ( Figure 2). IMPORTANT: FIMTrack can only handle TIFF or PNG images. If the images are not available in the required format, conversion can be done using Fiji 1 , ffmpeg 2 , or other third party software.
You can get a preview of an image by clicking on the image name inside the loaded jobs view on the left side of the GUI (Figure 3). Furthermore, you can zoom in and out by using cmd /Ctrl + mouse wheel or for faster zooming cmd /Ctrl + Shift + mouse wheel while our mouse courser is located over the preview image. If zooming is not possible, please click with the left mouse button in the preview image.  Navigation inside the preview image can be done by using the scrollbars around the image. The preview itself is useful to adjust the following parameters (at the bottom of the main GUI): • Gray Threshold specifies the minimal brightness of the foreground objects (i.e., larvae). If no larval foreground object is highlighted in the preview image, the Gray Threshold parameter has to be decreased. If artifact which are darker compared to the foreground object are highlighted this parameter has to be increase.
• Min Larvae Area specifies the minimal amount of pixels covered by the larval contour. All objects which consist of less pixels are neglected in further process.
• Max Larvae Area specifies the maximal amount of pixels covered by the larval contour. All objects which consist of more pixels are neglected in further process.
A foreground object is considered to represent a larva if the associated contour is highlighted in blue in the preview image. Red, yellow, and green circles serve as additional markers in order to indicate a correct detection and segmentation of an animal. The head is marked in red, the tail is marked in green, and the thickness of a larva is approximated by yellow circles. The blue number next to a larva indicates the covered area in pixels ( Figure 4a) in order to provide an initial estimation for the Min and Max Larvae Area parameters. Since currently FIMTrack is not able to resolve colliding animals, it is important that the Max Larvae Area parameter is smaller than the area covered by two colliding animals. If the area covered by larvae is smaller than the Min Larvae Area parameter value, reduce this parameter and select an image for preview from the loaded jobs view on the left side of the GUI or click on the Preview button at the bottom right of the GUI in order to refresh the preview. Detected objects highlighted in red occupies a larger area as allowed by the Max Larvae Area parameter value. (Figure 4b). Furthermore, coiled larvae (i.e., the head is touching the tail) are highlighted in the preview too. These animals are marked by purple circles inside the contour (Figure 4c).

How To Adjust The Preferences
Preferences can be adjusted by selecting Edit → Preferences from the menu bar. There are two types of preference panels: • Standard preferences includes camera related parameters (i.e., the frames per second (fps) used for tracking) (Figure 5a).
• Advanced preferences include all tracking and feature extraction related parameters ( Figure 5b).

Standard Preferences
Before starting the tracking, adjust the frames per second rate which is given by the image acquisition rate of your camera. This parameter will be used later on for calculating parameters such as velocity or movement direction.

Advanced Preferences
The advanced preferences are structured in groups. Each group can be edit by un-checking the Use Default or Use Dynamic ... Calculation check box.
• Background Subtraction Parameters to calculate a static background image where no animals are visible but which includes almost all immovable artifact present on the tracking surface.
-From Image specifies the first image from the image sequence which should be used for calculating the background image.
-Offset specifies the step size which should be used for background image calculation. A step size of two means that only every second image will be used.
-To Image specifies the last image from the image sequence which should be used for calculating the background image.
• Larvae Extraction Parameters for larval feature calculation.
-Number of Spine Points specifies the number of points equidistantly placed along the spine. This value has to be odd in order to guarantee a real mid-spine point.
-Coiled Recognition Parameters to detect coiled structures.  . * Perimeter To Spinelength Threshold is used to identify coiled structures. A larva is in a coiled state, if the head is touching the tail. Given such situation the program cannot estimate distinct head and tail points ( Figure 6b). Furthermore, in such cases, the ratio between the perimeter (P in Figure 6b) and the spine length (L in Figure 6b) converges to π (Figure 6a). Thus this ratio has to be above this threshold ( P L ≥ τ ).  Figure 6c) and the overall perimeter (P in Figure 6c) is above a given threshold K the animal is assumed to be in a coiled state (Figure 6c).
-Body Bending Calculation for head sweep detection. * Angle Threshold specifies the aberration of the main body bending (in both directions) to be detected as a head-sweep / bending (in degree).
-Movement Direction Calculation * Frames For Direction Calculation: given high frame rates (e.g. 10 frames per second), the movement detection cannot be calculated between consecutive frames since the traveled distance between consecutive frames is to short. Thus, the direction is calculated for every frame by looking at the larva in the current frame i and at the position of the same animal in the frame at i+offset, whereas this parameters specifies offset.
-Stop & Go Phase Calculation to classify stop and go phases. * Frames For Speed Calculation: given high frame rates (e.g. 10 frames per second), the correct speed cannot be calculated between consecutive frames since the traveled distance between two consecutive frames is to short. Thus, an offset is used to calculate the speed of an animal between frame i and i+offset, whereas this parameter specifies the offset. * Speed Threshold: given the Frames For Speed Calculation offset from above, this parameter specifies the minimal velocity a larva has to have during a go phase. Otherwise the larva is considered to be in a stop phase. * Angle Threshold: assuming that strongly bended animals cannot be in a go phase, this threshold specifies the maximal body bending angle which is acceptable during a go phase.
-IPAN Contour Curvature Parameters for calculating the spine of the animal. The IPAN algorithm proposed in [4] is used to calculate the curvature along the contour (Φ in Figure 6d). Therefore, a triangle is generate at each contour point. * Minimal Triangle Side Length specifies the minimal length of a triangle edge (in pixels; d min in Figure 6d). * Maximal Triangle Side Length specifies the maximal length of the triangle edge (in pixels; d max in Figure 6d). * Maximal Curvature Points Distance specifies the minimal distance the head and the tail point have to be apart (in pixels; distance in Figure 6d). * Curvature Window Size is used to smooth the calculated curvatures. The window size n is used to calculate the mean curvature at the anchor point (Figure 6d).
-Tracking Assignment: is used to extends the range of organisms which can be analyzed even at various spatial and temporal resolutions. For standard tracking, the Hungarian algorithm [5,6] in combination with overlap-based costs should be used to associate the animals over time. In contrast, an unsuitable temporal resolution or a relatively fast lateral locomotion of the animals lead to false assignments using the Hungarian algorithm with overlapping contour costs since no overlaps can be detected within contours of consecutive frames. Similarly, C. elegans moves in a snake like motion so that contour-overlap based assignments may fail. * Assignment Method: you can choose between the Hungarian algorithm and an algorithm which follows the greedy pattern. * Cost Measure: the following three cost measurements are available: 1. The intersecting area in pixels given two consecutive contours (Figure 7a). 2. Euclidean distance between the center of masses (momentum in Figure 7b). 3. Euclidean distance between the mid-spine points (Figure 7c). * Threshold for Greedy Assignment: specifying the maximal distance between two consecutive points if distance based costs are used or the minimal amount of overlap in case of contour-based costs. This parameter is only available if the greedy assignment method should be used.

Set Region Of Interest
You can mark an arbitrary number of image parts which should be included into tracking by defining so-called region of interests (Figure 8). Everything outside these regions will be neglected. A region of interest is set by right clicking in the preview window. Select the shape of your preferred region (rectangular or ellipsoid) and assign a unique name for the region. You can move a region around by left clicking into the region and moving the mouse while holding the left mouse button down. Furthermore, the region can be resized by selecting the lower right corner of the region and by clicking and holding the left mouse button while moving the mouse (use cmd /Ctrl to highlight the corner). You can change the color of the regions by double-clicking at the colored outside of a region.

Start Tracking
If all relevant parameters are set up, click on the Track button in the main GUI ( Figure 1) to start tracking. You will get intermediate results of the tracking process if the Show Tracking Progress check box is selected. It is normal that after clicking Track nothing happens for some time. This is due to the background image calculation, which is performed prior to the tracking (see message in the statusbar at the lower left part of the main GUI). If tracking is done, a short message will appear in the statusbar at the lower left part of the main GUI.

Batch Processing
If image capturing was done using the same settings (e.g camera and/or illumination settings), the resultant image sequences can be analyzed in a batch processing manner. Just add as many jobs as necessary using the Load Job button (Figure 9). Single jobs can be removed from the list using the Delete Job button. Reset All Jobs removes all jobs from the list.

Output Of FIMTrack
After tracking is done, FIMTrack creates a folder in the respective image folder and stores several files. The naming of the created folder is output date time where date is the current date and time is the current time. FIMTrack outputs: • table.cvs: comma-separated values file containing all calculated features.
• output.yml: YAML files containing same features as the table.cvs file and some additional informations which will be used in the FIMTrack Results viewer (see below).
• tracks.tif : an image showing all computed trajectories which are color-coded. The spine is plotted for all animals and for all time points thus head sweeping can bee seen in these images.

View Results using FIMTrack
FIMTrack offers the possibility to review the tracking results and thus the calculated features. Furthermore, it is also possible to correct the tracking results or to manually track several animals if they could not be tracked by the software. Select Edit → Results viewer... to open the Results Viewer ( Figure 10). Click on Load Results... to select both, images and tracking results. Please make sure to select the same images which were used for tracking (Figure 11a). After selecting the right images you will be ask to select the calculated tracking results, i.e., the output.yml file (Figure 11b). It is important that you have selected the same images which were used for tracking. You can see the absolute image path and the image names which were used for tracking by opening the output.yml file with a text editor and check the first entries in this file.
Using the Results Viewer The Results Viewer module has an Image, a Table, and a Larva tab to view the images and calculated features. Most of the calculated features are plotted color-coded onto the raw images at the image tab (Figure 12a). You can zoom in and out by using cmd/Ctrl+mouse-wheel or cmd/Ctrl+shift+mouse-wheel. By using the table tab you can inspect all calculated features of the larvae which were tracked at a specific time point (Figure 12b). The animal view can be used to inspect  the results for a single animal in more detail. A cropped region of a single animal is provided at the upper left part in this view. Furthermore, you have the opportunity to select some calculate features which should be plot from a dropdown menu in order to provide a first detailed view of the behavior of a single animal (Figure 12c). Push the play button or use the slider at the bottom of the Results Viewer GUI to step through time. All currently available animal trajectories are drawn in the image tab. You can also use the left and right button to step from frame to frame.

Manual Tracking
At the larva tap you have the possibility to adjust the tracking results.
The Manual Tracking Panel The left panel in the image view offers the following options (Figure 13a): 1. Tabs for selecting the larvae.
2. Left and right button to scroll through the larva tabs.
3. Change the color of the current selected larva.  9. Invert the model (swap head and tail). The inversion dialog will appear (see below).
10. Attach the selected larva to another animal in order to unite the trajectories. The attach to dialog will appear (see below).
11. Insert a duplicate of this larva before or after the current frame.
12. Delete model in the current frame (i.e., the larva at this specific time point).
13. By clicking on Remove Larva From Results you will delete all measurements of the selected larva.
14. Remove Short Tracks can be used to remove larval trajectories which are shorter then a minimal number of frames (i.e., time points). The minimal length in frames can be set at the remove larva dialog (see below).
Invert Larval Model For swapping the head and tail please use the invert larval model button (9 in Figure 13a). After clicking on this button the invert larva dialog will appear (Figure 13b). It is possible to invert the model at the current frame or in all frames Before or After the frame selectable in the dropdown menu below.
Attach Another Model For uniting two trajectories use the Attach Another Model button. Select the model which should be attached to the current larva model (Figure 13c).  Remove Short Tracks Trajectories which are to short can be removed from the results by using the Remove Short Tracks button. In the following dialog, the minimal number of frames for a valid trajectory has to be entered (Figure 13d).
Adjust Single Larva You can also adjust the posture for each animal (Figure 14a and Figure 14b). First select an animal which posture you would like to adjust by selecting the larva tab of the animal (see 1 in Figure 13a). Subsequently, select the spine point you would like to change in the result image using the left mouse button. Move the mouse while holding the left mouse button to change the location of the selected spine point. You can also change the radius which is drawn around each spine point (not of the head and the tail) and which approximates the thickness of the animal by selecting the points (left mouse button) and pressing the cmd /Ctrl key while moving the mouse. All changes have to be confirmed in the edit panel to avoid unintentional changes of the measured trajectories and postures (Figure 14c).

Insert Stimuli
You can add different stimuli marker at the Results Viewer (Figure 15). A right click inside the results viewer image (at the image tab) gives you the opportunity to select one of the following stimulus marker: • Point which is (x, y) position • Line which is a straight line • Rectangle which is an arbitrary sized axis-aligned 2D rectangle • Ellipsoid which is an arbitrary sized axis-aligned 2D ellipsoid For all markers the additional features distance to stimulus, bearing angle to stimulus, and is in stimulus region are calculated for each time point and animal. The size of all stimulus marker can be adjusted equivalent to the procedure described for region of interests (see Section 2.3).

Save Adjusted Results And Stimuli
To save your adjustments permanently, click on the Save Results... button at the top of the Results Viewer. At the following dialog you have to enter the (new) file names for the YAML, the CSV, and the TIFF image file.

Features Calculated By FIMTrack
Here is a list of all features calculated by FIMTrack. These parameters are stored in the table.csv and the output.yml file and can be separated into primary, secondary, and tertiary parameters.

Primary Features
The primary features are directly calculated from the contour of an animal.

Secondary Features
Most of the secondary features are calculated based on the primary features.
• bodyBending specifies the body bending of an animal. Bending of 180 • means no bending, values > 180 • indicate bending to the left and values < 180 • specify bending to the right (Figure 17c; in degree).
• spineLength specifies the length of the spine (in pixels) • isLeftBended and isRightBended is 1 (otherwise 0) if the larva is bended to the left or to the right. This value primarily depends on the threshold which was set at the preference option Angle Threshold. (binary indicator).
• isCoiled is 1 (otherwise 0) if the larva is in a coiled state (i.e., head is touching the tail; binary indicator).
• Distance to stimulus is the distance to a given stimulus (in pixels).
• In Stimulus Indicator is 1 if the animal is inside a stimulus region, otherwise 0 (binary indicator).

Tertiary Features
The tertiary features are calculated based on the primary features and time informations.
• movDirection specifies the movement direction measured to the y-axes in degree. A value of 90 • represents a movement to the right and a value of 270 • a movement to the left (Figure 17b; in degree).
• momDist specifies the distance between two consecutive center of mass positions (in pixels).
• Velocity is the velocity of the larva (in pixels fps ) • Acceleration is given in pixels fps 2 .
• accDist specifies the accumulated distance over time (in pixels).
• dstToOrigin specifies the direct linear distance to the origin of the trajectories (in pixels).
• The bearing direction specifies movement direction towards or away from a stimulus: an angle of 0 • indicates that an animal moves straight towards a stimulus and a value of 180 • indicates that an animals moves in the opposite direction away from the stimulus (in degree).
• The peristalsis is indirectly given by the area over time (in pixels).
• isGoPhase is 1 if the larva is in a go phase and 0 indicates that a larva is in a stop phases (binary indicator).