The authors have declared that no competing interests exist.
A central task in the analysis of human movement behavior is to determine systematic patterns and differences across experimental conditions, participants and repetitions. This is possible because human movement is highly regular, being constrained by invariance principles. Movement timing and movement path, in particular, are linked through scaling laws. Separating variations of movement timing from the spatial variations of movements is a well-known challenge that is addressed in current approaches only through forms of preprocessing that bias analysis. Here we propose a novel nonlinear mixed-effects model for analyzing temporally continuous signals that contain systematic effects in both timing and path. Identifiability issues of path relative to timing are overcome by using maximum likelihood estimation in which the most likely separation of space and time is chosen given the variation found in data. The model is applied to analyze experimental data of human arm movements in which participants move a hand-held object to a target location while avoiding an obstacle. The model is used to classify movement data according to participant. Comparison to alternative approaches establishes nonlinear mixed-effects models as viable alternatives to conventional analysis frameworks. The model is then combined with a novel factor-analysis model that estimates the low-dimensional subspace within which movements vary when the task demands vary. Our framework enables us to visualize different dimensions of movement variation and to test hypotheses about the effect of obstacle placement and height on the movement path. We demonstrate that the approach can be used to uncover new properties of human movement.
When you move a cup to a new location on a table, the movement of lifting, transporting, and setting down the cup appears to be completely automatic. Although the hand could take continuously many different paths and move on any temporal trajectory, real movements are highly regular and reproducible. From repetition to repetition movements vary, and the pattern of variance reflects movement conditions and movement timing. If another person performs the same task, the movement will be similar. When we look more closely, however, there are systematic individual differences. Some people will overcompensate when avoiding an obstacle and some people will systematically move slower than others. When we want to understand human movement, all these aspects are important. We want to know which parts of a movement are common across people and we want to quantify the different types of variability. Thus, the models we use to analyze movement data should contain all the mentioned effects. In this work, we developed a framework for statistical analysis of movement data that respects these structures of movements. We showed how this framework modeled the individual characteristics of participants better than other state-of-the-art modeling approaches. We combined the timing-and-path-separating model with a novel factor analysis model for analyzing the effect of obstacles on spatial movement paths. This combination allowed for an unprecedented ability to quantify and display different sources of variation in the data. We analyzed data from a designed experiment of arm movements under various obstacle avoidance conditions. Using the proposed statistical models, we documented three findings: a linearly amplified deviation in mean path related to increase in obstacle height; a consistent asymmetric pattern of variation along the movement path related to obstacle placement; and the existence of obstacle-distance invariant focal points where mean trajectories intersect in the frontal and vertical planes.
When humans move and manipulate objects, their hand paths are smooth, but also highly flexible. Humans do not move in a jerky, robot-like way that is sometimes humorously invoked to illustrated “unnatural” movement behavior. In fact, humans have a hard time making “arbitrary” movements. Even when they scribble freely in three dimensions, their hand moves in a regular way that is typically piecewise planar [
These invariances imply that movements as a whole have a reproducible temporal form, which can be characterized by movement parameters. Their values are specified before a movement begins, so that one may predict the movement’s time course and path based on just an initial portion of the trajectory [
A key source of variance of kinematic variables is, of course, the time course of the movement itself. The invariance principles suggest that this source of variance can be disentangled from the variation induced when the movement task varies. In this paper, we will first address time as a source of variance, focussing on a fixed movement task, and then use the methods developed to address how movements vary when the task is varied.
Given a fixed movement task, movement trajectories also vary across individuals. Individual differences in movement, a personal movement style, are reproducible and stable over time, as witnessed, for instance, by the possibility to identify individuals or individual characteristics such as gender by movement information alone [
A third source of variation are fluctuations in how movements are performed from trial to trial or across movement cycles in rhythmic movements. Such fluctuations are of particular interest to movement scientists, because they reflect not only sources of random variability such as neural or muscular noise, but also the extent to which the mechanisms of movement generation stabilize movement against such noise. Instabilities in patterns of coordination have been detected by an increase of fluctuations [
A systematic method to disentangle these three sources of movement variation, time, individual differences, and fluctuations, would be a very helpful research tool. Such a method would decompose sets of observed kinematic time series into a common trajectory (that may be specific to the task), participant-specific movement traits, and random effects. Given the observed decoupling of space and time, such a decomposition would also separate the rescaling of time across these three factors from the variation of the spatial characteristics of movement.
The statistical subfield that deals with analysis of temporal trajectories is the field of functional data analysis. In the literature on functional data, the typical approach for handling continuous signals with time-warping effects is to pre-align samples under an oversimplified noise model in the hope of eliminating the effects of movement timing [
Decomposition of time series into a common effect (the time course of the movement given a fixed task), an individual effect, and random variation naturally leads to a mixed-effects formulation [
We use as of yet unpublished data from a study of naturalistic movement that extends published work [
Software for performing the described types of simultaneous analyses of timing and movement effects are publicly available through the
Ten participants performed a series of simple, naturalistic motor acts in which they moved a wooden cylinder from a starting to a target position while avoiding a cylindric obstacle. The obstacle’s height and positition along the movement path were varied across experiments (
Participants move the cylindrical object
The movements were recorded with the Visualeyez (Phoenix Technologies Inc.) motion capture system VZ 4000. Two trackers, each equipped with three cameras, were mounted on the wall 1.5 m above the working surface, so that both systems had an excellent view of the table. A wireless infrared light-emitting diode (IRED) was attached to the wooden cylinder. The trajectories of markers were recorded in three Cartesian dimensions at a sampling rate of 110 Hz based on a reference frame anchored on the table. The starting position projected to the table was taken as the origin of each trajectory in three-dimensional Cartesian space. Recorded movement paths for two experimental conditions are shown in
The figures display (a) raw acceleration in recorded time, (b) raw acceleration in percentual time, (c) smoothed acceleration in recorded time, and (d) smoothed acceleration in percentual time. The plots allow visualization of the variation across participants and repetitions.
Obstacle avoidance was performed in 15 different conditions that combined three obstacle heights
The present data set is described in detail in [
Not every movement has the exact same duration. Comparisons across movement conditions, participants, and repetitions are hampered by the resulting lack of alignment of the movement trajectories. For a single condition, this is illustrated in
To handle nonlinear variation of timing, the signal must be time warped based on an estimated, continuous, and monotonically increasing function that maps percentual time to warped percentual time, such that the functional profiles of the signals are best aligned with each other. Such warping has traditionally been achieved by using the dynamic-time warping (DTW) algorithm [
To illustrate the difference between DTW and the proposed method, consider the example displayed in
(a)-(b)
Two questions naturally arise. Firstly, are the warping functions unique? Other warping functions could perhaps have produced similarly well-aligned data. Secondly, do we want perfectly aligned trajectories? There is still considerable variation of the amplitude in the warped
Using the time-warping functions that were determined by aligning the z-coordinates of the movements to now warp the trajectories for the
(a) displays the warped
In the following, we describe inference for a single experimental condition. For a given experimental condition—an object that needs to be moved to a target and an obstacle that needs to be avoided—we assume there is a common underlying pattern in all acceleration profiles; all
Time was implicit, up to this point, and the observed acceleration profile was decomposed into additive, linear contributions. We now assume, in addition, that each participant,
Altogether, we have described the following statistical model of the observed acceleration profiles across participants:
Compared to conventional methods for achieving time warping, the proposed
The estimation/prediction procedure is inspired by the algorithmic framework proposed in [
Let
We note that all random effects are scaled by the noise standard deviation
We model the underlying profile,
For fixed warping functions
Similarly to the linear mixed-effects setting [
We can write the local linearization of
Altogether, twice the negative profile log likelihood function for the linearized
So far, the
To model the amplitude effects, we use a cubic B-spline basis
We require that the fixed warping function
We assume that the sample paths of the serially correlated effects
Finally, in order to consistently penalize the participant-specific spline across experimental conditions with varying variance parameters, we will use penalization weights that are normalized with the variance of the amplitude effects,
The algorithm for doing inference in
1:
2: Compute
3:
4:
5: Estimate and predict warping functions by minimizing the posterior
6:
7:
8: Recompute
9:
10: Estimate variance parameters by minimizing the linearized likelihood
11:
12:
13:
In the following we consider two approaches, with (1) samples parametrized by recorded time (
The number of basis functions
The dashed trajectory shows the estimate for
A simulation study that validates the method and implementation on data simulated using the maximum likelihood estimates of the central experimental condition (
A first assessment of the strength of the statistical
The classification of the movement data is based on the characteristic acceleration profile computed for each participant. For this to work it is important that individual movement differences are not smoothed away. The hyperparameters of the model were chosen with this requirement in mind. In the following, we describe alternative methods we considered. For all approaches, the stated parameters have been chosen by 5-fold cross-validation on the experimental conditions with obstacle distance
We evaluate classification accuracy using 5-fold cross-validation, which means that eight samples are available in the training set for every participant. The folds of the cross-validation are chosen chronologically, such that the first fold contains replications 1 and 2, the second contains 3 and 4 and so on. The results are available in
obstacle | NP | NP_{p} | MBM | MBM_{p} | RME | RME_{p} | DTW | DTW_{p} | FR | FR_{p} | FR_{E} | FR_{Ep} | TMS | TMS_{p} | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
15.0 cm | 0.36 | 0.48 | 0.53 | 0.43 | 0.55 | 0.57 | 0.52 | 0.52 | 0.47 | 0.54 | 0.62 | 0.51 | 0.70 | ||
0.36 | 0.46 | 0.38 | 0.45 | 0.41 | 0.43 | 0.49 | 0.56 | 0.36 | 0.49 | 0.47 | 0.46 | 0.66 | |||
0.41 | 0.47 | 0.41 | 0.46 | 0.49 | 0.50 | 0.43 | 0.43 | 0.32 | 0.56 | 0.49 | 0.49 | 0.62 | |||
22.5 cm | 0.36 | 0.49 | 0.34 | 0.46 | 0.37 | 0.50 | 0.45 | 0.44 | 0.44 | 0.51 | 0.50 | 0.51 | 0.68 | ||
0.38 | 0.44 | 0.42 | 0.53 | 0.46 | 0.55 | 0.38 | 0.42 | 0.32 | 0.45 | 0.42 | 0.55 | 0.62 | |||
0.36 | 0.49 | 0.45 | 0.54 | 0.46 | 0.57 | 0.40 | 0.53 | 0.48 | 0.57 | 0.53 | 0.61 | ||||
30.0 cm | 0.27 | 0.29 | 0.41 | 0.45 | |||||||||||
0.30 | 0.42 | 0.40 | 0.45 | ||||||||||||
0.37 | 0.44 | 0.50 | 0.45 | ||||||||||||
37.5 cm | 0.28 | 0.45 | 0.41 | 0.49 | 0.42 | 0.51 | 0.45 | 0.50 | 0.36 | 0.51 | 0.39 | 0.56 | 0.69 | ||
0.26 | 0.33 | 0.33 | 0.37 | 0.35 | 0.41 | 0.40 | 0.49 | 0.35 | 0.37 | 0.32 | 0.53 | 0.57 | |||
0.31 | 0.43 | 0.38 | 0.43 | 0.40 | 0.46 | 0.37 | 0.29 | 0.50 | 0.48 | 0.49 | 0.55 | 0.63 | |||
45.0 cm | 0.25 | 0.38 | 0.33 | 0.45 | 0.32 | 0.42 | 0.34 | 0.51 | 0.32 | 0.45 | 0.37 | 0.41 | 0.65 | ||
0.29 | 0.31 | 0.29 | 0.38 | 0.38 | 0.39 | 0.43 | 0.43 | 0.36 | 0.48 | 0.38 | 0.45 | 0.53 | |||
0.29 | 0.39 | 0.45 | 0.48 | 0.48 | 0.57 | 0.38 | 0.45 | 0.39 | 0.44 | 0.44 | 0.47 | 0.58 | |||
average | 0.323 | 0.418 | 0.393 | 0.460 | 0.427 | 0.482 | 0.417 | 0.463 | 0.387 | 0.484 | 0.449 | 0.487 | 0.649 |
In the previous section, the proposed modeling framework was shown to give unequalled accuracy of modeling the time series data for acceleration. In this section we use the warping functions obtained to analyze the spatial movement paths and their dependence on task conditions. Temporal alignment of the spatial positions along the path for different repetitions and participants is necessary to avoid spurious spatial variance of the paths. The natural alignment of two movement paths is the one that matches their acceleration signatures. In other words, spatial positions along the paths at which similar accelerations are experienced should correspond to the same times. Thus, each individual spatial trajectory was aligned using the time warping predicted from the TMS_{p}-results of the previous section. Every sample path was represented by 30 equidistant sample points in time at which values were obtained by fitting a three-dimensional B-splines with 10 equidistantly spaced knots to each trajectory.
As an exemplary study, we analyze how spatial paths depend on obstacle height. We do this separately for each distance, so that we perform five separate analyses, one for each obstacle placement. In each analysis we have 10 participants with 10 repetitions for each of 3 obstacle heights. The three-dimensional spatial positions along the movement path,
We are interested in understanding how the space-time structure of movement captured in the 30 by 3 dimensions of the trajectories,
We identify the low-dimensional subspace based on a novel factor analysis model. In analogy to principal component analysis (PCA),
The idea is to use the mean movement trajectory,
The loading matrix
The models were fitted using maximum likelihood estimation by using an ECM algorithm [
The fitted mean trajectories for the three different obstacle heights at distance
We fitted both models for every obstacle distance and performed likelihood-ratio tests. The p-values can be found in
Obstacle distance | 15.0 cm | 22.5 cm | 30.0 cm | 37.5 cm | 45.0 cm |
---|---|---|---|---|---|
p-value | 0.478 | 0.573 | 0.093 | 0.764 | 0.362 |
A strength of our approach to time warping is that we can estimate variability more reliably. In addition to observation noise, we model three sources of variation in the observed movement trajectories: individual differences in the trajectory, individual differences caused by changing obstacle height, and variation from repetition to repetition. The variances described from these three sources are independent of obstacle height. In Figs
Along the trajectory eight equidistant points (in percentual warped time) are marked, and at each point 95% prediction ellipsoids are drawn. The three columns represent the random effects tied to participant, subjective reaction to height change and repetition, respectively.
Along the trajectory eight equidistant points (in percentual warped time) are marked, and at each point 95% prediction ellipsoids are drawn. The ordering is the same as in
Note the asymmetry of the movements with respect to obstacle position, both in terms of path and variation. This asymmetry reflects the direction of the movement. Generally, variability is higher in the middle of the movement than early and late in the movement. Individual differences caused by change in obstacle height (middle column) are small and lie primarily along the path. That is, individuals adapt the timing of the movement differently as height is varied. Individual differences in the movement path itself (left column) are largely differences in movement parameters: individuals differ in the maximal elevation and in the lateral positioning of their paths, not as much in the time structure of the movements. Variance from trial to trial (right column) is more evenly distributed, but is largest along the path reflecting variation in timing.
These descriptions are corroborated by the comparisons of the amounts of variance explained by the three effects in
A final demonstration of the strength of the method of analysis is illustrated in
Green lines correspond to low obstacles, yellow to medium obstacles and orange to tall obstacles.
We have proposed a statistical framework for the modeling of human movement data. The hierarchical nonlinear mixed-effects model systematically decomposes movements into a common effect that reflects the variation of movement variables with time during the movement, individual effects, that reflect individual differences, variation from trial to trial, as well as measurement noise. The model amounts to a nonlinear time-warping approach that treats all sources of variances simultaneously.
We have outlined a method for performing maximum likelihood estimation of the model parameters, and demonstrated the approach by analyzing a set of human movement data on the basis of acceleration profiles in arm movements with obstacle avoidance. The quality of the estimates was evaluated in a classification task, in which our model was better able to determine if a sample movement came from a particular participant compared to state-of-the-art template-based curve classification methods. These results indicate that the templates that emerge from our nonlinear warping procedure are both more consistent and richer in detail.
We used the nonlinear time warping obtained from the acceleration profiles to analyze the spatial movement trajectories and their dependence on task conditions, here the dependence on obstacle height and obstacle placement along the path. We discovered that the warped movement path scales linearly with increasing obstacle height. Furthermore, we separated the variation around the mean paths into three levels: individual differences of movement trajectory, individual differences caused by change in obstacle height, and trial to trial variability. This combination of models uncovered clear and coherent patterns in the structure of variance. Individual differences in trajectory and variance from trial to trial were the largest sources of variance, with individual differences being primarily at the level of movement parameters such as elevation and lateral extent of the movement while variance from trial to trial contained a larger amount of timing variance. We documented a remarkable property of the movement paths when obstacle distance along the path is varied at fixed obstacle height: all paths intersect at a single point in space.
We believe that the approach we describe enhances the power of time series analysis as demonstrated in human movement data. The nonlinear time warping procedure makes it possible to obtain reliable estimates of variance along the movement trajectories and is strong in extracting individual differences. This advantage can be leveraged by combining the nonlinear time warping with factor analysis to extract systematic dependencies of movements on task conditions at the same time as tracking individual differences, both base-line and with respect to the dependence on task conditions, as well as variance across repetitions of the movement.
Recent theoretical accounts have used the analysis of variance across repetitions of movements to uncover coordination among the many degrees of freedom of human movement systems [
(TEX)
(TIFF)
(TIFF)
The left panel shows results for ordinary least square (OLS) estimation and the right panel shows the results for the proposed model and estimation algorithm. Both models were fitted using the correctly specified spline model for the mean. Note that the density is displayed on squareroot scale.
(TIFF)
(TIFF)
Dashed red lines indicate the true values of the parameters.
(TIFF)