^{1}

^{2}

^{3}

^{1}

^{3}

^{‡}

^{1}

^{2}

^{‡}

The authors have declared that no competing interests exist.

‡ These authors also contributed equally to this work.

Mathematical modeling of behavioral sequences yields insight into the rules and mechanisms underlying sequence generation. Grooming in

Analysis of temporally rich behavioral sequences provides a quantitative description of the rules underlying their generation.

Sequential animal behaviors are often composed of repetitions of simple subroutines. In driving motor action execution, nervous systems must integrate sensory information with internal priorities and dynamics. Both external and internal conditions contribute to sequence organization, but their respective weights are unknown. At one extreme, purely sensory-driven, reflexive processes produce actions, such as larval escape sequences, solely based on recent sensory input [

Grooming, a common behavior across species, confers social and survival benefits [

On long time scales,

At an intermediate temporal scale, grooming is organized in units that we refer to as motifs. We identify two classes of motifs (anterior and posterior), which are named for the set of legs used to execute them (front and back, respectively). Motifs consist of consecutive alternations between body-directed grooming bouts and leg rubbing bouts. Bouts are defined as sustained periods of a single grooming action (e.g. head cleaning or wing cleaning). Grooming bouts occur at the shortest time scale we consider here, as individual bouts typically last somewhere between 150 ms and 2 s. Flies use the same pair of legs to execute bouts within a motif, allowing them to transition between within-motif actions easily. Specifically, an anterior grooming motif consists of consecutive alternations between bouts of head cleaning and front leg rubbing. A posterior motif consists of alternating bouts of abdomen cleaning, wing cleaning, and back leg rubbing.

Analysis of freely-behaving flies reveals behavioral structure at multiple time scales, with coarse-grained descriptions being sufficient to describe long-term trends and higher resolution descriptions providing increased predictive power at shorter time scales [

Several classes of Markov models, in which the probability of an event occurring is contingent upon previous events, have been used to describe factors involved in non-deterministic decision-making in various animals. Different Markov models vary in their structure and number of parameters but they each require a well-defined state space. Navigation and foraging have been described using Markov models, as these sequential behaviors can be decomposed into subroutines which can be categorized using easily observable dimensions such as direction and velocity [

It should be noted that non-Markovian dynamics have also been identified in animal vocal sequence production, suggesting that other classes of models may be useful for describing behavioral generation [

Here, we use our in-house Automatic Behavior Recognition System (ABRS,

A: The ABRS classifier is trained to annotate video data with labels corresponding to one of five grooming actions observed in

First, this data set allows us to quantify behavioral trends at several temporal scales on a larger data set than has been previously described.

Grooming ethograms possess at least three distinct time scales. The grooming progression (top), which takes many minutes to observe in its entirety, occurs at the longest time scale we consider here. We refer to this phenomenon as a progression because we observe that, on average, flies begin grooming with anterior-oriented actions and end with posterior-oriented actions and more walking. At an intermediate time scale, ethograms consist of grooming motifs (middle). Motifs are composed of consecutive grooming bouts which use the same pair of legs (either anterior or posterior). At the shortest time scale, we observe individual grooming bouts (bottom), which consist of sustained grooming movements that range from several hundreds of milliseconds to a few seconds in duration.

Next, we use a set of Markov models to characterize probabilistic rules governing grooming action transitions. We use a Markovian framework as a first approximation in order to discover relevant features of grooming, since Markov models are simple and carry few assumptions. As such, these models serve as a tool for grooming sequence feature identification rather than as a full explication of grooming decision-making factors. Previous work has indicated that such analysis can reveal temporal relationships in sequential behavioral data, indicating that, even if the behavior being analyzed is not fully Markovian in reality, Markov models can still be useful as a descriptive exploratory tool [

The simplest of our models, a first order discrete time Markov chain, highlights the presence of grooming motif structure. Subsequent models incorporate temporal information by considering grooming bout durations. By comparing these models to statistical null model hypotheses, which shuffle bout order or bout duration, we discover the contribution of bout duration to grooming sequence structure at short time scales.

Though we rely on a Markovian framework to identify features of grooming syntax, our analysis does not preclude the possibility of non-Markovian dynamics in fruit fly grooming. Indeed, our data exhibits two notable nonstationarities. First, we observe an overall trend in grooming proportions, as has been reported previously [

We find that a time-varying Markov renewal process (MRP) which incorporates bout duration dependence recapitulates the observed grooming structure at long, intermediate, and short time scales. This model is partially Markovian, as transition probability matrices dictate state transition dynamics. However, the model also includes renewal process dynamics, in which the duration of the ensuing bout is determined using a random draw from the empirically observed bout duration distribution. This model is nonstationary due to the fact that Markov transition probabilities change over time. This model recapitulates both the observed bulk grooming progression statistics and the fine-grain temporal structure of grooming motifs, suggesting that sensory drive can modulate internal programs in a simple manner to prioritize different grooming strategies in different contexts.

Our models clarify the syntax of grooming, demonstrating that, at the scale of individual bouts, transitions are dictated by both the identity and duration of the preceding bout. While previous studies have established the non-random nature of fruit fly grooming [

The discovery of duration dependence in grooming sequence structure suggests the existence of neural circuits in

Briefly, the classifier is trained on grooming data using a combination of supervised and unsupervised protocols in several steps. First, a Fourier transform is performed on pixel intensity data across 17-frame sliding windows (∼0.5 s), resulting in two-dimensional spectra referred to as space-time images (ST-images). ST-images are subsequently modified by Radon transformation, which can be used to represent 2D shapes in a rotation and translation-invariant manner. Fast Fourier transformation is then applied to the Radon transformed ST-images, resulting in position and orientation-invariant ST-images [

ST-images are then decomposed using singular value decomposition (SVD) and clustered using linear discrimination analysis (LDA) with human labeled behavioral categories. Clusters identified using LDA correspond to five predefined grooming behaviors (f = front leg rubbing, h = head cleaning, a = abdomen cleaning, b = back leg rubbing, w = wing cleaning) and two non-grooming behaviors (wk = walking, s = standing). After classification, each video is represented by a 50,000-element vector, with each element containing a label denoting the behavior identified in the corresponding frame. Classifier performance and validation are shown in

To generate discrete bout ethograms, each 50,000-element vector is transformed into a two-dimensional vector in a series of steps. The first row of this discrete bout ethogram contains the bout identity, or the label corresponding to one of the seven behaviors described above. The second row contains the bout durations, or the amount of time spent in the corresponding behavior before transitioning to another behavior.

Consecutive frames containing identical actions are consolidated into one bout identity entry. Then, the corresponding bout duration entry is calculated by counting the number of consecutive identical frames and converted into seconds by dividing by the video frame rate (30 Hz). As a result, the number of entries in discrete bout ethograms corresponds to the number of observed bouts, which varies between flies. However, the sum of bout durations for each ethogram is identical (27.8 minutes).

After ethogram generation, behavioral bouts persisting for less than 167 ms are deemed artifacts and deleted. This threshold is chosen because manual inspection of video data suggests that individual leg sweeps occur at approximately this time scale. This is also the approximate average transition time between behavioral states described by Berman et al. [

In order to include bout duration as a data dimension, we introduce a binning scheme. To implement this change, the continuous-valued entries in the second row of discrete bout ethograms are collapsed into duration category labels. Grooming bouts are classified into three categories (short, medium, and long) based on the duration of the associated bout. Although our results are consistent even when using more than three duration categories (

Bin edges for these categories are determined independently for each grooming action (f, h, a, b, w) and walking (wk), as each action possesses a unique duration distribution. Bin edges are chosen such that each bin is populated by an equal number of samples (i.e. the total number of actions labeled “short” is equal to the number of “long” actions) and are depicted in

Ethograms are modified by introducing a binning scheme based on grooming bout duration. Here, we show the binning strategy used to create three duration categories for grooming actions. Grooming actions are classified as short, medium, or long bouts, indicated by the red, green, and blue regions of the probability density functions, respectively. Region boundaries are chosen such that each duration category contains an equivalent number of samples. Notably, anterior motif actions (f, h) possess strikingly similar duration distributions, suggesting coupling between these actions.

Markov processes are a class of probabilistic models that describe transitions between states of a system in terms of previous states. The simplest version obeys the Markov property, which declares that the next transition between states depends only on the current state. As such, long-term history dependence can be neglected when considering Markov processes, greatly simplifying the models. In discrete time, the dynamics of a first order Markov chain are described by the equation
_{t} denotes the state probability vector at time _{t} represent the probability of the system being in the state corresponding to that entry at time _{ij} is the probability of transitioning from state _{t} will have dimensionality corresponding to the number of possible states in the system. In this case, there are either six, twelve, or eighteen grooming states, depending on whether grooming bout duration is incorporated into the model. Given sufficient data about state transitions in a real system, it is possible to generate a maximum likelihood estimate (MLE) transition probability matrix, denoted by _{ij} is the total number of observed transitions from state _{iu} is a normalization factor which counts the total number of transitions from state

First order Markov chain maximum likelihood transition probabilities are determined for each ethogram as described in

Statistical null models, which randomize isolated features of data while maintaining overall statistical features, provide a basis for hypothesis testing to determine the significance of results obtained from experimental data. To assess the contributions of bout identity and bout duration to sequence structure, we generate null hypotheses which independently randomize these features of the ethogram data while preserving bout duration frequency distributions within grooming action types. Null hypothesis transition probabilities are analytically tractable in the limit of infinite permutations, so we use exact formulas to calculate matrix entries (

Duration permuted: This null hypothesis preserves action order information, but transition matrix entries are independent of duration.

Order permuted: This null hypothesis preserves action duration information, but transition matrix entries are independent of action order.

By shuffling bout duration or bout order and comparing the resultant null hypothesis transition matrices to the maximum likelihood estimate, we identify which features are statistically significant in the data and contribute to sequence structure. Shown here are schematics of the permutations we perform on the observed ethograms to generate null hypothesis transition matrices. In the duration permuted hypothesis, bout order is preserved but duration categories are randomly permuted (h = head, wk = walking, f = front leg, s = short, m = medium, l = long). In the order permuted hypothesis, bout order is permuted but the durations associated with each bout remain the same.

The resulting model fit is determined using the Bayesian Information Criterion (BIC), defined as

The BIC provides a metric for evaluating model goodness-of-fit by penalizing models which use a large number of free parameters (the first term in

To evaluate the ability of our models to generate sequences that are similar to our observed data, we use them to create synthetic ethograms. Synthetic sequences are generated using Monte Carlo methods within a Markov renewal process (MRP) framework. In an MRP, state transitions are governed in a manner identical to Markov chains (i.e. using a transition probability matrix). However, the amount of time spent in each state is determined in parallel by a renewal process, in which action durations are drawn from a known distribution. Here, we use the empirically observed duration distributions shown in

After exposure to an irritant,

A: Average grooming progression of

We find that grooming bouts possess action-specific duration distributions.

Interestingly, bouts durations are not normally distributed, suggesting a mechanism other than random generation of durations around a mean value. Instead, anterior motif duration distributions possess “shoulders” on each side of the central peak, suggesting that dividing actions into bout duration categories is not an unnatural distinction. Moreover, these “shoulders” are relatively broad due to the fact that grooming bouts consist of many quanta of individual leg sweeps (in the case of body-directed actions) or rubs (in the case of leg-directed actions), which are cyclical in nature and last approximately 150 ms per cycle. Due to video frame rate resolution limitations, we do not consider this extremely fast behavioral scale here. In this analysis, we highlight results which divide grooming actions into three duration categories in order to provide an intuitive demarcation between “short”, “medium”, and “long” actions. However, it is important to note that our conclusions are not strictly dependent on this choice of categorization scheme, as our results hold for higher numbers of duration categories as well

We find that the vast majority of transitions occur within grooming motifs. Approximately 88% of transitions from anterior motif grooming bouts are to another anterior motif behavior, while about 70% of transitions from posterior motif bouts are to another posterior motif behavior. This tight intra-motif coupling is illustrated in

We find that this strong coupling emerges from the structure of individual grooming motifs.

A: Schematic of an anterior motif. Anterior motifs are defined as continuous consecutive bouts of head grooming (h) or front leg rubbing (f) flanked by non-anterior motif actions on either side. Posterior motifs are defined similarly, but consist of abdomen grooming, back leg rubbing, and wing grooming (a, b, and w). Both motifs contain body-directed actions (h, a, and w), which clear irritant from the body. Leg-directed actions (f and b) clear irritant from the legs, which collect irritant during body-directed grooming actions. In this example, the anterior motif consists of six grooming bouts of varying duration. B: Anterior motifs exhibit a strong linear relationship between the total amount of time spent performing body and leg-directed actions (left). Each point corresponds to an observed motif consisting of four or more actions. Posterior motifs (right) display a weaker linear trend with greater amounts of leg-directed grooming.

This tight intra-motif coupling is also reflected in the structure of Markov transition matrices fit to our behavioral data (ethograms). We fit a first order discrete time Markov chain to our data and the resulting matrix, denoted by

A: The maximum likelihood transition matrix,

In

A: Probabilities from the maximum likelihood transition matrix (top) differ significantly from null hypothesis transition matrices (bottom) to varying degrees, indicating that both grooming action order and bout duration contribute to grooming syntax. Shown on the top left is the network visualization of anterior motif transitions, with edge withs proportional to transition probabilities. Notice that transitions between actions belonging to the same duration category possess relatively high probabilities and appear nearly symmetric, suggesting a coupling mechanism between anterior motif actions. BIC values (right of matrices) provide validation that the maximum likelihood model captures statistically significant features of the data. The maximum likelihood transition matrix has a lower BIC value than the null hypotheses used for comparison, indicating that both bout order and duration contribute to sequence syntax. B: The residual values, or the difference between the maximum likelihood matrix and the null model matrices, illustrate that specific transitions differ from what the null hypotheses would predict. The duration permuted null model matrix exhibits block structure but fails to capture the temporal relationship between bouts, as illustrated by the red values in several “long-to-long” transitions, for example. The order permuted null model differs even more severely, indicating that, while duration dependence plays a role in sequence structure, action order is still the primary determinant.

Additionally, we observe that, regardless of the number of duration bins, the anterior motif transition probability structure is nearly mirror symmetrical along the diagonal (i.e. transitions between front leg rubbing and head cleaning bouts have the same probabilities). In contrast, posterior motif transitions are less strongly symmetrical, with transitions between abdomen cleaning and back leg rubbing exhibiting the highest probabilities (

We find that the probabilistic rules describing transitions between consecutive bouts depend on two features of immediate behavioral history: previous bout identity and previous bout duration. To validate this observation, we evaluate the quality of the maximum likelihood model using the Bayesian Information Criterion (BIC). This metric, which is described mathematically by

Inspection of BIC values indicates that bout order is the strongest individual determinant of grooming syntax and that bout duration provides an additional contribution.

By construction, each null hypothesis transition matrix contains identical row structure within sets of rows corresponding to the same grooming action. However, only the duration permuted transition matrix possesses block structure similar to that found in the maximum likelihood matrix, as seen in

Shown here are the population average transition probabilities for the early phase (left) and late phase (right) of grooming. The border between these phases is indicated by the dashed gray line in

Qualitatively, these matrices exhibit strong similarities, despite the differences in grooming proportions, as shown in

Markov models are steady state models and, as such, only exhibit equilibrium dynamics when used in a generative manner to create synthetic data. The first order maximum likelihood Markov model presented above is best suited for describing the late stages of grooming, which resembles steady state behavior. This is an obvious shortcoming of using a Markovian framework for analyzing the ethograms presented here, as flies display a dynamic progression (

In order to more explicitly model these changing sensory dynamics, we utilize a nonstationary, or time-varying, Markov renewal process (MRP). Here, we introduce a nonstationarity in the form of a time-varying transition probability matrix. In order to maintain as parsimonious a model as possible, we define the transition probability matrix as a time-dependent convex combination of an early phase matrix and a late phase matrix (

The time-varying transition probability matrix

Surprisingly, using this simple linear interpolation as a proxy for changing sensory conditions yields a model which produces ethograms that closely resemble our observed data (

A: Illustration of the time-varying transition matrix, _{early} and _{late}, are fit to the first and last 200 actions in the discrete time ethogram with 3 duration categories, respectively. As time evolves in the synthetic ethogram simulation, _{early} and after 13 minutes, it is identical to _{late}. Between those times, it is a linear combination of the two matrices. B: Synthetic ethograms display the characteristic progression from anterior to posterior grooming, as seen from comparison with

After exposure to an irritant,

Here, we isolate short time scale grooming syntax by using a binning scheme to represent grooming bouts as semi-discrete actions (e.g. short, medium, or long bouts). Across different binning schemes, within-motif transitions (i.e. anterior-to-anterior or posterior-to-posterior) dominate syntax at the scale of consecutive bouts, as indicated by the block-like structure of transition matrices in

Moreover, the amount of front leg rubbing over the course of the grooming progression corresponds nearly exactly to the amount of head cleaning performed by the flies. The amount of back leg rubbing is also directly proportional to the amount of abdomen and wing cleaning (

Anterior motif transitions display symmetric transition rules, as shown in

These observations support the hypothesis that alternations between anterior grooming bouts are regulated by a common source or common dynamics that dictate intra-motif transition frequency. If this is the case, future investigation of neural circuitry could focus on identifying neural activity that oscillates in phase with grooming bout alternations. Under this model, flies would then need to combine discrete decisions about which motif to perform with continuous decisions about how long to maintain a motif. This provides other targets for future experiments, as we could look for neural activity corresponding to transitions between motifs, explore manipulations that induce transitions between motifs, and search for factors that affect the duration of motifs.

Although sensory input is sufficient to initiate individual grooming bouts, it has been difficult to assess the role of sensory input in grooming on longer time scales, since direct quantification of irritant relies on invasive protocols [

Here, we abstract the contribution of sensory input, incorporating it as a parameter in a time-varying Markov renewal process. Specifically, we approximate changing sensory conditions as a linear phenomenon which dictates the degree to which flies favor early versus late grooming rules (

Since our MRP reliably reproduces true ethogram statistics without explicitly modeling sensory input, we propose that, rather than acting in a purely reflexive manner,

From a computational perspective, sensing is costly when it requires high-frequency updates and can provide readouts at a high resolution, as specialized neural circuits must be formed, maintained, and integrated in order to provide accurate, real-time readouts. This can lead to increased metabolic costs on several fronts, as increased neural activity and behaviors intended to protect and maintain sensory machinery can also require energetic resources [

During the execution of a grooming motif, frequent, high-resolution sensory updates may not be necessary. Instead, the moment-to-moment decisions about execution of grooming bouts can be automated by utilizing patterned internal dynamics. Several models of behavioral generation in which internal triggers drive sequence generation have been proposed [

In discussing temporal correlations between neural activity across spatial scales, Berman et al. [

By combining sensory input and internal dynamics,

In larval and mature

Since CPGs can modulate motor programs independently of sensory input, they are a natural candidate to execute grooming behavior on the scale of individual leg sweeps and rubs within a grooming bout. We do not explicitly model behavior at this temporal scale, but the ABRS classifier applied to video at the frame rate used here identifies actions nearly at the temporal resolution of individual leg movements. We plan to extend analysis to this time scale in future work.

Additionally, robotic systems that utilize multi-layer CPG architectures can carry out locomotor tasks, indicating that hierarchical CPG structures can generate sequential behaviors from smaller elements [

When describing complex behavioral phenomena, formal mathematical models can bridge the gap between phenomenology and biological mechanisms by providing parameters that may correspond to underlying neuronal activity. In many cases, statistical model parameters do not directly describe to underlying biological structures, but data analysis can suggest hypotheses for future exploration.

Here, we analyze large-scale automatically annotated data sets to reveal the syntax of

As the tools for computational analysis of behavioral data continue to develop, interdisciplinary approaches that use mathematical tools to illuminate biological mechanisms will yield further insight into the generation of sequential behavior. Future experiments involving the manipulation of sensory experiences using optogenetic stimulation or other methods will also allow researchers to test hypotheses with unprecedented precision and scope. Together, these advances promise to improve our understanding of sequence generation by connecting mathematical descriptions of behavior with the underlying neural circuitry, as we are now motivated to search for circuits controlling long, intermediate, and short time scale grooming rules and determine how they interact with sensory circuits. Finally, our work suggests that temporal dynamics can and should be included when using statistical models to assess complex behaviors.

A: Here, grooming behaviors are visualized in low-dimensional space using t-SNE, which preserves local distances between nearby points. Color indicates behavior type. Behaviors are well-separable by type. B: Example classifications of ethograms of grooming behaviors are shown. Human-algorithm agreement is about 75% (in terms of number of matching frames).

(TIF)

When transition probabilities are calculated from ethograms with 4 (left) or 5 (right) duration categories, fine-grain temporal structure is still present (compare the matrices shown here to those shown in

(TIF)

Two examples of individual fly transition probabilities illustrate that, although syntax varies between individuals and over time, individuals are relatively stable and resemble the group average (compare to

(TIF)

Plotted here is the relationship between consecutive anterior grooming action durations. Linear regression produces a poor fit, indicating that duration dependence is not linear.

(TIF)

To ensure that our choice of boundary between early and late phases does not alter our results, we partition our data in a stricter manner so that the transition period which occurs during the middle of the grooming progression is excluded. Partitioning the data into thirds produces transition matrices with structure and stationarity similar to those shown in

(TIF)

We find that 253 of the 266 (95.1%) non-zero entries change by less than 5% and 168 (63.2%) entries change by less than 1%. This was a very surprising finding and suggests that, although some non-stationarity does exist, it is much smaller than we had expected given that the average time spent performing different behaviors varies between the early and late phases.

(TIF)

Probability density plots of bout duration distributions illustrate that bout durations remain consistent between early and late phases of grooming. Early phase data is plotted with a thick line. Late phase data is plotted with a thin line.

(TIF)

Given in S1 Table are the analytic formulas for transition matrix entries, _{ij}, for each null hypothesis transition matrix. Here, _{action,j}|_{action,i}) is the absolute probability of transitioning from the grooming action represented by row _{duration,j}) is the absolute probability of the grooming action represented by row _{j}) is the absolute probability of state

(TIF)

Thanks to Tianyi Qin for drawings of individual grooming actions. Behavioral data collection assisted by Phuong Chung.