Ethoscopes: An open platform for high-throughput ethomics

Here, we present the use of ethoscopes, which are machines for high-throughput analysis of behavior in Drosophila and other animals. Ethoscopes provide a software and hardware solution that is reproducible and easily scalable. They perform, in real-time, tracking and profiling of behavior by using a supervised machine learning algorithm, are able to deliver behaviorally triggered stimuli to flies in a feedback-loop mode, and are highly customizable and open source. Ethoscopes can be built easily by using 3D printing technology and rely on Raspberry Pi microcomputers and Arduino boards to provide affordable and flexible hardware. All software and construction specifications are available at http://lab.gilest.ro/ethoscope.

: Flowchart of the video tracking algorithm. In each consecutive frame, ROIs are automatically defined and used to extract as many sub-images. They are then all processed separately. Firstly, simple denoising is applied. A background model is then used to extract foreground pixels. The preprocessed image is also used to update the background model. Objects (connected components) are detected in the foreground. When no object was detected, tracking is aborted. If more than one object is found, the foreground model is used to keep only the 'most valid' (i.e. likely) object. If only one object is found, it is considered to be valid. Features of the remaining valid object are computed and used to update the foreground model. The learning rate of the background model is decreased only when exactly one object was found (i.e. unambiguous match), and increased otherwise. The background is not updated in the neighbourhood of the foreground object (not shown in this figure).

Background model
Background model is based on a classical weighted running average with a mask: It was observed that the frame rate of devices was heterogeneous, depending on system load.
Therefore, instead of implicitly assuming constant frame rate (i.e. α is constant), our background model accounts explicitly for the delay δt between two consecutive frames: t 1/2 corresponds to the 'half-life' of the information in a pixel. The larger t 1/2 , the longer it takes for the model to learn. Importantly, the learning rate is dynamic t 1/2 ∈ [1, 100]s (see sub-section ).

Foreground features
At each frame, the preprocessed image (src) is subtracted from the background model (bg) and thresholded with thr = 20: Then, connected components are located. If none are present, tracking is simply aborted. If more than one object is found, the foreground model (see next sub-section) is used to select only the most likely object. When exactly one object is detected, it is considered as 'unambiguous' foreground and used immediately.
Several primary features such as XY position, orientation, width, height, area and average pixel intensity are computed on the single resulting foreground object. In order to obtain an accurate measurement of velocity, subpixel XY position was computed using greyscale image moments as opposed to simply using binary moments.

Foreground model
A vector of three features -log 10 (area), height and average pixel intensity -describing the detected foreground object is stored in a feature matrix M . The likelihood L of candidate foreground objects, with a vector of feature X, is then computed based only on marginals, assuming a Gaussian distribution for the three features. For every column j, we have, where µ j and σ j are the average and the standard deviation of the feature j, respectively.
Then, we have Importantly, the foreground model is shared between all ROIs, which assumes relative feature homogeneity between foreground objects. When tracking visually similar animals (e.g. fruit flies of approximately the same size), this assumption is expected to hold.

Updating background and foreground models
The background model is systematically updated for each frame, but its half-life is set dynamically such that it decreases when foreground identification was ambiguous (several objects), or aborted (no objects).
In addition, the background is selectively updated using the last valid foreground object as a mask [2,3].
The foreground model is updated with the features from any detected foreground object. Features are stored, in a rolling buffer, up to a maximum of 10 3 observations.

FPS-dependent velocity correction
It was noticed that resulting velocity computation depended on the frame rate of the processed video. It is difficult to predict analytically the relationship between measured velocity and Frame Per Second (FPS) since it depends both on the structure of the background noise and the types of foreground movements. Therefore, we decided to proceed empirically (Fig. 2).
We downsampled the original validation video (144h at 25 FPS, see algorithm described above on each of them independently and computed all instantaneous velocities ( Fig. 2A). The maximal velocity (variable chosen for behavioural scoring, see figure 4B of the main article) in each consecutive 10s epoch was computed in all cases. Ground truth data was used to perform ROC curves and, for each FPS, define the threshold that lead to a False Positive Rate (FPR) of movement equals to 0.5% (T F P S=0.005 ). The relationship between FPS and max velocity was then modeled with linear regression: Fitting gave a = 0.003 and b was not statistically significantly different from zero, so we used b = 0 Therefore, velocity (V corr ) calculation were corrected with: This way, max(V corr ) > 1 indicates the foreground is moving, with with a specificity of 99.5% To further ensure effectiveness of this method, the density of all instantaneous velocity was re-computed after correction (Fig. 2B). Visual inspection confirmed the performance of our empirical correction.

Software implementation and R package
Tracking algorithm was implemented using OpenCV [4] (python bindings) in combination with numpy [5] and scipy [6] libraries.
Acquisition of several days of behavioural data from multiple animals generates a fairly large amount of data (e.g.. 25M B · week −1 · animal −1 ). In order to simplify the behavioural data visualisation and analysis rethomics, a novel package for R statistical software [7], was developed as part of this project (https://github.com/gilestrolab/rethomics).