Decomposing spontaneous sign language into elementary movements: A principal component analysis-based approach

Félix Bigand; Elise Prigent; Bastien Berret; Annelies Braffort

doi:10.1371/journal.pone.0259464

Abstract

Sign Language (SL) is a continuous and complex stream of multiple body movement features. That raises the challenging issue of providing efficient computational models for the description and analysis of these movements. In the present paper, we used Principal Component Analysis (PCA) to decompose SL motion into elementary movements called principal movements (PMs). PCA was applied to the upper-body motion capture data of six different signers freely producing discourses in French Sign Language. Common PMs were extracted from the whole dataset containing all signers, while individual PMs were extracted separately from the data of individual signers. This study provides three main findings: (1) although the data were not synchronized in time across signers and discourses, the first eight common PMs contained 94.6% of the variance of the movements; (2) the number of PMs that represented 94.6% of the variance was nearly the same for individual as for common PMs; (3) the PM subspaces were highly similar across signers. These results suggest that upper-body motion in unconstrained continuous SL discourses can be described through the dynamic combination of a reduced number of elementary movements. This opens up promising perspectives toward providing efficient automatic SL processing tools based on heavy mocap datasets, in particular for automatic recognition and generation.

Citation: Bigand F, Prigent E, Berret B, Braffort A (2021) Decomposing spontaneous sign language into elementary movements: A principal component analysis-based approach. PLoS ONE 16(10): e0259464. https://doi.org/10.1371/journal.pone.0259464

Editor: Peter Andreas Federolf, University of Innsbruck, AUSTRIA

Received: June 18, 2021; Accepted: October 19, 2021; Published: October 29, 2021

Copyright: © 2021 Bigand et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting information files. All raw motion capture data are available from the MOCAP1 corpus on the ORTOLANG website (https://www.ortolang.fr/market/corpora/mocap1/).

Funding: This work has been funded by the Bpifrance (https://www.bpifrance.fr/) investment project “Grands defis du numerique”, as part of the ROSETTA project (RObot for Subtitling and intElligent adapTed TranslAtion). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Sign Languages (SLs) are the first languages of 70 million deaf people in the world [1]. Yet, deaf SL users face many communication barriers. In particular, the vast majority of automatic communication tools are not compatible with SL content, but only with spoken or written one. Developing successful tools for automatic SL processing (i.e., SL automatic recognition, generation and translation) would allow breaking down these barriers. For that aim, further insights must be gained into multiple disciplines, in particular motion science. Indeed, beyond the sparsity of research and developments conducted in SL compared to spoken languages, the automatic processing of SL is challenging because of the intrinsic complexity of SL movements. For instance, SL involves multiple motion features from various body parts, such as movements of the torso, arms, hands and fingers as well as facial expressions. These movements involve coordinating many biomechanical degrees of freedom (DOFs) [2, 3]. Furthermore, the spatial and temporal coordination of SL gestures is driven by a complex linguistic system, whose modeling does not yet meet with a broad consensus among linguists. This raises the question of determining how the motor system actually controls such complex movements and how computational models of SL motion could be improved by exploiting these control strategies.

Indeed, despite the complexity of SL motion, SL users have no difficulty engaging in various SL conversations throughout the day. More generally, the ease with which humans fluidly execute movements has questioned how the motor system could coordinate the many DOFs and master such a high-dimensional space of postures [4]. One hypothesis is that the multiple DOFs are controlled within a subspace of lower dimension than the available number of DOFs [4–6]. Instead of processing the multiple DOFs individually, the motor system controls a reduced set of compositional elements called synergies (i.e., patterns of muscle activations) [7–9]. These synergies are combined and, depending on their respective weights, are used to generate different movements.

With the progress of motion capture (mocap) systems, converging evidence has been provided across various movement contexts that motion datasets could be properly defined using a reduced number of synergies. For instance, Principal Component Analysis (PCA) has been shown to be effective in extracting synergies, by decomposing motion data into uncorrelated principal movements (PMs). Troje [10] first used PCA to disentangle the motion patterns of human gait. Similarly to “eigenfaces” [11] or “eigenvoices” [12], the whole movement of the walker was decomposed into simpler one-directional PMs (i.e., time series of “eigenpostures”), which maximized the variance in the original motion. The first PMs accounted for most of the variance in the movements and can be interpreted as the kinematic elements (i.e., synergies) recruited by the motor system to organize the movement [13]. PCA has then been successfully applied to various common movements, such as walking or running [14–17], but also in more complex sport contexts, such as skiing [18], karate [19] or diving [20].

In most of the studies mentioned above, PM decomposition has allowed gaining insights into the coordinative structure of complex movements and into the underlying mechanisms of motor control. This data-driven technique has also allowed shedding light on specific motor mechanisms related to gesture expertise. In Federolf et al. [18], PMs were projected back onto the original 3D space and were visualized. This allowed interpreting and comparing the skiing movements of athletes in terms of distinct PMs, such as lateral body inclination, flexion-extension of the legs or rotation of the skis. The same method has been successfully applied to other sports, such as karate [19] or diving [20]. In juggling, individual differences due to experience have also been found in specific PMs [21]. Beyond expertise, such differences in PMs have allowed determining specific biomechanical mechanisms due to impairments, such as knee osteoarthritis [14] or cerebral palsy [15, 22]. Several studies have also used this technique to investigate human posture control [23–27], notably to detect mechanisms due to perturbations of the postural control system, such as head shaking [28]. Additionally, Haid et al. [26] have reported age effects in postural control characterized by control differences in specific PMs. In the artistic domain, Tits et al. [29] have showed that the finger gestures of pianists can be decomposed into eight PMs and that the complexity of the decomposition was a function of the expertise of the pianists.

To the authors’ knowledge, such a holistic evaluation of the upper-body movements in spontaneous SL discourses has not been proposed yet. Some studies investigating hand synergies have provided insights into the movements of the dominant hand of signers during the production of highly constrained isolated ASL signs (i.e., shaping the hand into static letters of the alphabet, or numbers) [30–33]. PCA application was successful, which suggests that hand control of ASL letters occupies a space of low dimension. These results support the potential contribution of PM decomposition to automatic SL processing, as it allows models to use a reduced number of dimensions and thus simplifies automatic tasks such as recognition, as previously shown for ASL letters [30]. However, producing ASL letters or numbers with the hand is only a reduced part of particular ASL signs. Moreover, SL productions made in isolation hardly provide complete descriptions of how SL is used by signers in real-life conditions, even for more complex signs than letters or numbers. For instance, it has been demonstrated that spontaneous SL mocap recordings could reveal faster movements than isolated signs [34–36]. Furthermore, SL motion involves the coordination of far more body parts than the dominant hand, including the other hand, but also torso, head, shoulders and arms.

Investigating PM decomposition for the description of spontaneous upper-body SL movements thus presents a two-fold interest. First, it may provide unexpected fundamental insights into how the complex movements of SL are structured and into the motor control strategies used to produce such complex motion. These analyses could interestingly complement prior studies made on a linguistically limited set of hand gestures [30–33]. Furthermore, it could be used to improve technological tools dedicated to SL, notably as it allows for substantial dimensionality reduction. Indeed, PCA has been successfully applied to human full-body movements for machine learning purposes, such as automatic prediction of gender [10], identity [37], mental state [38] or expertise level [19, 20], which calls for further investigations using PMs as relevant inputs of automatic SL models. For instance, dense mocap datasets could be decomposed using PCA before more complex machine learning procedures, notably considering the crucial role of proper pre-processing steps in improving the performance of deep neural networks for automatic SL recognition [39]. Again, such recognition models could complement the prior work made on the automatic classification of ASL alphabet letters using PCA [30]. Furthermore, taking advantage of the inter-segment coordination with PMs could open up promising perspectives for automatic SL generation via virtual signers (or signing avatars), as previously shown for robot control problems [5], such as control of artificial hands [7, 40, 41]. For instance, animation models could use dense mocap datasets to produce realistic movements by processing only a reduced subset of PMs while keeping most of the information about the original movements.

In the present study, we used PCA in order to determine the extent to which upper-body motion of French Sign Language (LSF) could be described within a low number of PMs. Moreover, we aimed to quantify whether these PMs were shared across various signers in spontaneous discourses. For that aim, (1) common PMs were extracted from a mocap dataset containing spontaneous LSF motion of six signers; (2) individual PMs were extracted from the separate data of each signer; (3) the consistency of the principal movements across signers was assessed.

Materials and methods

Motion capture corpus

The data used in the present analyses were taken from a previously reported study [42]. In brief, each of six deaf native and fluent signers (3/3 males/females) had freely described the content of 25 pictures using French Sign Language (LSF). The signers had been selected so that they were all deaf, fluent in LSF and working in professions that require them to be comfortable expressing themselves in front of a camera (e.g., teacher, journalist, story-teller, translator). They all gave informed written consent (translated into LSF for better accessibility) before the experiment, in accordance with the ethical standards of the Declaration of Helsinki. Using a motion capture system equipped with 10 cameras (Optitrack S250e), the data consisted of the upper-body movements recorded at 250 fps, in three dimensions. Further details (e.g., picture content, type of SL discourse, mocap equipment) are available from the original mocap corpus [43, 44]. From the 25 mocap recordings, only 24 were taken into account in the present study, as one of them was not available for one signer. Moreover, from the 23 original body markers available for all signers, we used 21 markers that optimally describe the major joints of the body. We chose to represent each elbow using one marker instead of two, as the second marker did not add substantial information and removing it eased the visualization of stick figures. As shown in Fig 1, the markers were (L = left, R = right, F = front, B = back): (1) pelvis, (2) stomach, (3) T10 kidneys, (4) sternum, (5) C7 thyroic gland, (6) LB head, (7) LF head, (8) RB head, (9) RF head, (10) L shoulder, (11) L elbow, (12) LB wrist, (13) LF wrist, (14) LB hand, (15) LF hand, (16) R shoulder, (17) R elbow, (18) RB wrist, (19) RF wrist, (20) RB hand, (21) RF hand.

Download:

Fig 1. The 21 upper-body markers taken from the motion capture corpus.

https://doi.org/10.1371/journal.pone.0259464.g001

Data pre-processing

All the data processing was conducted in Python (Python Software Foundation https://www.python.org/) using custom code, which is publicly available at the following GitHub repository: https://github.com/felixbgd/SL-PMs_Bigandetal_2021.

The 3D positions of all markers were defined in reference to the pelvis (used as the origin). From each of the 24 original recordings, one mocap recording unit with the duration of 5-second was extracted from the beginning of the utterance, irrespective of the semantic content. Each mocap recording unit was thus related to a different SL utterance. This resulted in 24 mocap examples per signer, of 5-second duration each.

The movements of each individual signer (i.e., the concatenation of their 24 mocap examples) were described in a matrix containing 30,000 posture vectors (rows) defined by the 3D coordinates of the 21 markers (columns) at each time frame t: (1)

A two-step normalization was then applied to these data, in order to allow extracting common PMs across different signers while reducing the effects of anthropometric differences, as defined in prior work [23]. First, the mean posture of each signer was computed over the 24 mocap examples (Eq 2). The mean posture was then subtracted from the posture vectors for each signer (Eq 3), in order to capture the variance caused by postural movements (i.e., deviations from the mean) rather than by differences in the mean postures: (2) where . (3) where is the centered posture vector.

Furthermore, the centered posture vectors were normalized to their mean Euclidean norm, in order to ensure an equal contribution by each signer to the variance of the combined matrix. The Euclidean norm was defined as follows: (4)

Then, centered posture vectors were divided by the mean of the Euclidean norm. The normalized vectors thus had the following form: (5)

In order to investigate common PMs across individuals, the normalized 30, 000 × 57-posture matrices of the six signers were concatenated into a 180, 000 × 57-matrix.

Extraction of principal movements

PMs were extracted by applying PCA to the normalized posture matrix. PCA was performed using singular value decomposition and produced principal components (PCs) (or eigenvectors) and their respective eigenvalues. The normalized eigenvalues indicated the percentage of variance explained by the related PCs. Each posture could then be reconstructed using a linear combination of the PCs. The PMs were then characterized by projecting separately each specific PC back onto the original 3D space (Eq 6). PMs were resynthesized using stick figures. This allowed visualizing the PMs and comparing the motion patterns they described. (6) where is the vector that describes the movements of the i^th PM. w_i(t) is the projection of the normalized posture vector onto the i^th PC-vector, .

In order to ensure that the extracted PMs reflect actual, volitional, movements rather than noise, their frequency content was estimated. For that aim, the Power Spectral Density (PSD) of the w_i(t) was computed using the Welch method [45]. This analysis revealed that the highest power resided in frequencies below 3 Hz, but visible power was still found in the frequency range between 3 to 6 Hz (see S1 Fig). Therefore, the w_i(t) were low-pass filtered using a 4th-order Butterworth Filter with a cut-off frequency of 6 Hz. The effect of noise was thus deemed to have been sufficiently reduced.

Statistical analyses

This study aimed to assess the extent to which upper-body motion in spontaneous SL could be described within low-dimensional subspaces shared by different signers. Therefore, a leave-one-out cross-validation was conducted to evaluate the vulnerability of the PMs to signer changes in the input (i.e., a signer was added or left out before applying the PCA). On average, the first eight PMs were found to be robust to signer changes (i.e., the PC-vector did not change its orientation in posture space by more than 15° when a signer was left out) [26] and explained around 94.6% of the total variance. They were therefore included in the further analyses. Yet, note that PM5 and PM6 were slightly less robust when Signer 5 was left out (i.e., for this specific signer change, the 5^th and 6^th PC-vectors changed their orientations by 21.9° and 20.6°, respectively), which motivated further statistical analyses assessing the extent to which the PMs could be common across different signers.

To address this problem, we further extracted individual PMs (i.e., from the 30, 000 × 57-matrices of each signer separately) and assessed their similarity to the common PMs (i.e., extracted from the 180, 000 × 57-matrix combining the six signers). This analysis notably aimed to investigate whether the PMs could be similar across signers although ranked in a different order by the PCA, notably due to differences in the discourse, which was not constrained and thus could activate PMs differently across signers. The similarities between the common and individual PMs were assessed using cosine similarity [46, 47]. This analysis was conducted on the normalized PMs, in order to compare postural movements beyond anthropometric differences. The 30, 000 × 57-posture matrix of each normalized PM was reshaped into a vector of length 1, 710, 000. Then, the cosine similarity (sim) between two PMs was computed as follows: (7) where and are the 1, 710, 000-vectors of the two PMs to be compared and N is the length of both vectors (N = 1, 710, 000).

Finally, beyond comparing individual and common PMs one-by-one as in the analysis above, we further aimed to compare the PM subspaces specific to each signer. Therefore, we evaluated the extent to which the individual PM subspace of one signer could account for the variance in the movements of the other signers, by computing the cross-projection similarity [31, 48]. For that aim, the movements of one signer were projected onto their first N PC-vectors and the cumulative amount of variance of these projections was computed (V₁). Then, the movements of this first signer were projected onto the first N PC-vectors of a second signer and, similarly, the cumulative amount of variance was computed (V₂). Finally, we calculated the ratio V₂/V₁, which quantified the extent to which the second subspace was similar to the first. The closer this ratio is to 1, the higher the similarity is. As this measure is not symmetric (i.e., its value would not necessarily be equivalent if we projected the movements of the second signer onto the N PC-vectors of the first one), the cross-projection similarity between the subspaces of two signers was then obtained as the mean of the ratios computed in both directions.

Results

Structure of common principal movements

Common PMs were computed from the mocap dataset containing the 24 examples of the six signers. As shown in Fig 2, most of the overall variance was explained by the first eight common PMs. Combined, the first eight common PMs explained 94.6% of the cumulative variance.

Download:

Fig 2. Variance explained by the first 15 common PMs.

https://doi.org/10.1371/journal.pone.0259464.g002

The first eight common PMs (Videos 1 to 8 in S1 Dataset) are shown in Fig 3 and are described in details in Table 1. In summary, the first eight PMs were mainly defined as motion patterns visible in the frontal and sagittal planes. PM1 to PM4 quantified movements of the two hands along the vertical, anteroposterior and mediolateral axes, as well as upper-body rotation around the vertical axis. PM5 and PM6 quantified specific joint movements, such as arm internal rotation or elbow flexion. Higher-order PMs (PM7 and PM8) extracted finer movements, such as flexion of the wrists or shoulder abduction. Moreover, PM8 reported covarying movements between the head, the torso and the arms of the signers. For instance, a low negative weighting of PM8 was related to high flexion of the head, anteroposterior inclination of the torso, and high abduction of both shoulders.

Download:

Fig 3. The first eight common PMs.

Stick figures represent the PM at the time instants corresponding to the minimum (gray) and the maximum (black) PM weighting, across signers and examples. PMs are displayed in their main plane of motion (e.g., frontal or sagittal).

https://doi.org/10.1371/journal.pone.0259464.g003

Download:

Table 1. Characterization of the first eight common PMs.

EV is the Explained Variance in original movements.

https://doi.org/10.1371/journal.pone.0259464.t001

Similarities in principal movements across signers

Individual PMs (i.e., specific to each signer) were computed from the mocap data of individual signers. As shown in Fig 4, most of the overall variance was explained by the first eight individual PMs. Combined, the first eight individual PMs explained 95.7% (SD = 0.6%) of the cumulative variance. By comparison, the first eight common PMs explained 94.6%, and the first seven individual PMs explained 94.3% (SD = 0.9%). Therefore, although the common dataset contained a wide variety of unsynchronized movements performed by six different signers, the common PMs explained a similar amount of the movements variance, compared with PMs computed separately for each signer.

Download:

Fig 4. Variance explained by the first 15 individual PMs.

Mean was computed across the six signers, error bars indicate standard errors across signers.

https://doi.org/10.1371/journal.pone.0259464.g004

Visual animations of the first eight individual PMs seemed highly similar to the common PMs for most signers, although they may sometimes be ranked in a different order. This observation was further confirmed by the cosine similarity measures, as show in Fig 5. For instance for Signer 4 (Videos 9 to 16 in S1 Dataset), all the first seven PMs reported high levels of similarity (sim > 0.7 [47]) and PM8 reported a level of similarity near the latter threshold (sim = 0.69). All the PMs of this signer were similar to common PMs of the same order (i.e., individual i^th PM is similar to common i^th PM), except for PM3 and PM4, which were ranked in reverse order. Of the total number of 48 individual PMs across signers, 25 PMs reported a similarity level above 0.7, and 36 reported a similarity level above 0.5. The amount of similarity could vary depending on the signer. For instance, although some signers also presented consequent levels of similarity (e.g., Signer 3, where sim > 0.7 for five PMs, and sim > 0.5 for the three others), others reported lower similarity between their PMs and the common ones (e.g., Signer 6, where sim > 0.7 for only three PMs, and sim < 0.5 for four PMs).

Download:

Fig 5. Cosine similarity between individual and common PMs.

The similarity measures are specified when sim > 0.2, for sake of visibility.

https://doi.org/10.1371/journal.pone.0259464.g005

However, most of the PMs that reported low levels of similarity when compared one-by-one displayed non-negligible similarities to multiple common PMs (e.g., PM2 to PM5 of Signer 6 or PM5 and PM6 of Signer 1). In other words, these PMs may be combinations of common PMs. This outcome suggests that, although some PMs were not perfectly similar when compared individually with common ones, the PM subspace of the signer may still account for the movements of all signers and thus reflect similar low-dimensional dimensions used to control upper-body SL motion. A demonstration of these potential combinations appears notably in PM1 of Signer 1 (Video 17 in S1 Dataset), which can be characterized as vertical movement of the hands (sim = 0.71 with common PM1), jointly with anteroposterior movement of the hands (sim = 0.45 with common PM3).

In summary, most of the individual PMs were similar to the common ones, although they were sometimes ranked in a different order or were combinations of several common PMs. A few PMs could eventually be interpreted as signer-specific, when the cosine similarity was significantly low and not reflecting a combination of common PMs (e.g., PM7 of Signer 5, or PM6 of Signer 6). Still, the reported one-by-one similarities called for further analysis investigating the extent to which the PM subspaces, rather than the PMs individually, were similar across signers. This was achieved using cross-projection similarity (i.e., for each signer, we computed the amount of variance in the movements that could be accounted for by the first N PMs of another signer). This measure revealed that the PM subspaces specific to each signer were highly similar: on average, the first eight PMs of each signer explained 96.9% of the variance in the movements of the other signers (see Fig 6).

Download:

Fig 6. Mean cross-projection similarity between PM subspaces across signers.

Mean was computed across the six signers, error bars indicate standard errors.

https://doi.org/10.1371/journal.pone.0259464.g006

Discussion

In the present paper, we used PCA to decompose the upper-body movements of spontaneous LSF into a reduced set of simpler, elementary, movements. The original motion data were transformed into a new space spanned by principal components, called principal movements (PMs). The first eight common PMs (i.e., computed on the whole dataset containing all signers) accounted for 94.6% of the variance in the movements, which suggests that the control of spontaneous SL motion can be limited to a low number of dimensions—far fewer than the multiple dozens of DOFs of the upper body—, as previously demonstrated for full-body movements, such as gait [10], diving [20], skiing [18] or juggling [21]. This outcome is also in line with prior work on the control of hand gestures, such as grasping [31, 48], piano playing [29, 49] and in particular the production of SL letters [30–33]. However, our analysis differs from these prior SL-related studies in one crucial respect. The LSF productions used in the present study are termed spontaneous as signers freely described pictures without any constraints in time, signs or structure. Signers could thus express themselves freely, which elicited a wider variety of SL linguistic forms (e.g., lexical signs, but also depicting signs that describe size and shapes of entities), by contrast with the highly constrained movements (i.e., ASL letters) assessed in previous studies on hand gestures. Furthermore, the present SL movements were recorded in context, within continuous discourses, which is known to give rise to further motion features and coordination properties, by contrast with isolated productions like ASL letters [34–36]. The PMs extracted from our motion dataset are thus more likely to reflect potential synergies used by signers in real-life conditions, which is crucial notably to conceive efficient real-life communication tools.

Although it is now well known that PCA is efficient in decomposing human motion into low-dimensional subspaces, the fact that it is confirmed for unconstrained continuous SL is still an intriguing result, considering the variety of analyzed movements in the discourses (e.g., gestures were neither necessarily consistent in structure nor synchronized in time across signers and examples). By contrast, all studies investigating the synergies of SL in hand gestures [30–33] were limited to a specific, well-defined, set of linguistic forms: letters of the alphabet. As precisely outlined by one of the latter studies [30], the common synergies extracted across signers producing ASL letters may be expected because of the hand postures that were targeted to conform to the same forms defined by the alphabet. Although the hand has by definition a high number of biomechanical DOFs, the production of ASL letters imposes a strong linguistic constraint on the motion, which therefore reduces the number of potentiel hand configurations (i.e., could be referred to as linguistic DOFs, as compared to biomechanical ones). Note that in this respect, our results also differ from other evidence of synergistic control strategies in full-body motion made on movements that shared similar temporal structures (e.g., gaits [10], karate’s kata [19], dives [20] or juggling patterns [21]). Using spontaneous discourses, the present study further demonstrated that, despite a high number of biomechanical DOFs and potential linguistic gestures, upper-body movements of SL can be limited to low dimensions.

The second outcome of this study is that the low-dimensional PM subspaces were consistent across signers. First, the number of PMs needed to account for most of the variance in the movements was nearly the same for individual PMs as for common ones. Although the number of PMs retained is a simple metric, it was not trivial that it would be the same across the common and individual datasets. For instance in prior work on full-body synergies, the PMs common to all participants did not necessarily represent the original movements as well as the PMs computed for each participant [18]. For a comparison with prior SL-related work on hand gestures, a similar synergistic structure common across signers has been reported but it was more likely to occur, given that all signers were constrained by the common forms of an alphabet [30], by contrast with the free discourses used here. Given prior outcomes outlined on postural mechanisms related to gesture expertise [21, 29], we may also note that the similar number of PMs retained across the six signers of the present study could reflect that they had a similar level of expertise in SL gestures, all being fluent signers. Furthermore, although PMs of lower variance (i.e., PMs > 8) can be deemed to be directions of noise [19, 20], intriguing results including recent work on ASL hand gestures [31, 50] have shown that high-order PMs can be highly structured, which calls for further research investigating whether the high-order PMs (e.g., accounting for less than 1%-variance) of upper-body motion in spontaneous SL could be under volitional control rather than being related to noise.

Beyond the similar number of synergies, PM subspaces of the present study quantified consistent motion patterns across signers. Indeed, we showed that a high number of individual PMs (i.e. computed separately from the mocap data of each signer) were similar to common ones, although they were sometimes ranked in a different order or expressed as combinations of the common PMs. Furthermore, a cross-projection similarity analysis revealed that the first eight individual PMs specific to each signer accounted for most of the variance in the movements of all other signers. The extent to which full-body synergies are utilized across various locomotion tasks has been assessed in a few studies [46], which showed that similar PMs were recruited across tasks but that they could be prioritized differently depending on the task. As regards SL hand gestures, different signers were also reported to yield similar PM subspaces when producing ASL letters [31]. To the authors’ knowledge, the present study is the first to demonstrate the similarity between synergies used across signers and across an unconstrained range of linguistic gestures.

The motion patterns described by common PMs were mostly visible in the frontal and sagittal planes, in line with the vast majority of prior studies on full-body synergies [10, 14, 15, 18, 19, 23, 24, 51], except for some PMs of juggling reported in the transverse plane [21]. In the present study, common PM4 (i.e., trunk rotation) can be described in the transverse plane but it was clearly visible from a frontal view. PM3 was best visible from a sagittal view. Still, the patterns quantified by most PMs were occurring in the frontal plane. These findings interestingly complement prior SL-related work on the synergies of hand gestures [30–33]. Indeed, the mocap data used in the present study include upper-body trajectories by contrast with its predecessors, which have investigated the kinematics of the dominant hand only. Inversely, these prior studies have applied PCA to precise recordings of finger gestures, while our mocap data were recorded on various upper limbs but only included global motion data of the wrists and hands. Both approaches provide fundamental insights into the complex structure of SL motion. Taken together, these findings could be of particular interest for improving automatic SL processing models, which requires relevant representations of both upper-body movements and finger gestures, in particular for automatic recognition [39, 52] and generation [53–56]. Further work examinating larger mocap datasets and across a higher number of signers could be of interest in order to refine the definition of common PMs, which here were common to six signers only.

The successful application of PCA to the complex movements of spontaneous SL provides potential contributions to SL research for both fundamental and application purposes. First, PCA allows resynthesizing PMs in the original 3D space, which enables researchers to visualize these directions of high movement variability. For these reasons, PM decomposition has been widely used to better understand the coordinative structure of complex movements and could shed light on the motor control of SL movements, as previously investigated through common laws of motion [43, 57]. Furthermore, it allows for dimensionality reduction. This has a significant potential impact on machine learning procedures used in automatic SL tasks, in the same way as some past studies aimed to drastically lower the frame rate [58] or the number of markers [59] to reduce the bandwidth in SL telecommunication. Following the present results, the dimensionality of dense mocap datasets could be considerably reduced using only a subset of PMs while keeping most of the information. For instance, it could significantly reduce the dimensionality of the input of deep learning models used for the automatic recognition of SL and, more importantly, exploit knowledge from the structure of SL motion to improve their performance. Furthermore, both the potential to resynthesize movements from the PMs and the potential to reduce dimensionality make PM decomposition very promising for the improvement of other SL automatic tools: generation models. By shedding light on SL synergies, PCA could allow improving the existing synthesis models [53, 54]. It could also ease the incorporation of high-dimensional mocap recordings, which are known to be efficient in solving the problem of naturality and comprehensibility of virtual signers [56, 60, 61]. These potential applications to SL generation call for further work evaluating the observers comprehension of SL messages when resynthesized from a reduced set of PMs.

Supporting information

S1 Fig. Frequency content of the first eight common PMs.

Power Spectral Density was estimated using the Welch method.

https://doi.org/10.1371/journal.pone.0259464.s001

(TIF)

S1 Dataset. Principal movements.

Video examples of PMs shown as Point-Light Displays in frontal and sagittal planes (Total 17 files included). It includes examples of both common and individual PMs.

https://doi.org/10.1371/journal.pone.0259464.s002

(ZIP)

References

1. of the Deaf WF. Our work; 2016. Available from: http://wfdeaf.org/our-work/.
2. Bruton M, O’Dwyer N. Synergies in coordination: A comprehensive overview of neural, computational, and behavioral approaches. Journal of Neurophysiology. 2018;120(6):2761–2774. pmid:30281388
- View Article
- PubMed/NCBI
- Google Scholar
3. ElKoura G, Singh K. Handrix: animating the human hand. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation; 2003. p. 110–119.
4. Bernstein N. The co-ordination and regulation of movements. The co-ordination and regulation of movements. 1966.
5. Flash T, Hochner B. Motor primitives in vertebrates and invertebrates. Current opinion in neurobiology. 2005;15(6):660–666. pmid:16275056
- View Article
- PubMed/NCBI
- Google Scholar
6. Tresch MC, Jarc A. The case for and against muscle synergies. Current opinion in neurobiology. 2009;19(6):601–607. pmid:19828310
- View Article
- PubMed/NCBI
- Google Scholar
7. Santello M, Bianchi M, Gabiccini M, Ricciardi E, Salvietti G, Prattichizzo D, et al. Hand synergies: integration of robotics and neuroscience for understanding the control of biological and artificial hands. Physics of life reviews. 2016;17:1–23. pmid:26923030
- View Article
- PubMed/NCBI
- Google Scholar
8. d’Avella A. Modularity for motor control and motor learning. Progress in Motor Control. 2016; p. 3–19. pmid:28035557
- View Article
- PubMed/NCBI
- Google Scholar
9. Giszter SF. Motor primitives—new data and future questions. Current opinion in neurobiology. 2015;33:156–165. pmid:25912883
- View Article
- PubMed/NCBI
- Google Scholar
10. Troje NF. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of vision. 2002;2(5):2–2. pmid:12678652
- View Article
- PubMed/NCBI
- Google Scholar
11. O’Toole AJ, Abdi H, Deffenbacher KA, Valentin D. Low-dimensional representation of faces in higher dimensions of the face space. JOSA A. 1993;10(3):405–411.
- View Article
- Google Scholar
12. Kuhn R, Junqua JC, Nguyen P, Niedzielski N. Rapid speaker adaptation in eigenvoice space. IEEE Transactions on Speech and Audio Processing. 2000;8(6):695–707.
- View Article
- Google Scholar
13. Wang X, O’Dwyer N, Halaki M. A review on the coordinative structure of human walking and the application of principal component analysis. Neural regeneration research. 2013;8(7):662. pmid:25206712
- View Article
- PubMed/NCBI
- Google Scholar
14. Federolf P, Boyer K, Andriacchi T. Application of principal component analysis in clinical gait research: identification of systematic differences between healthy and medial knee-osteoarthritic gait. Journal of biomechanics. 2013;46(13):2173–2178. pmid:23910389
- View Article
- PubMed/NCBI
- Google Scholar
15. Zago M, Sforza C, Bona A, Cimolin V, Costici PF, Condoluci C, et al. How multi segmental patterns deviate in spastic diplegia from typical developed. Clinical Biomechanics. 2017;48:103–109. pmid:28806590
- View Article
- PubMed/NCBI
- Google Scholar
16. Daffertshofer A, Lamoth CJ, Meijer OG, Beek PJ. PCA in studying coordination and variability: a tutorial. Clinical biomechanics. 2004;19(4):415–428. pmid:15109763
- View Article
- PubMed/NCBI
- Google Scholar
17. Lamoth C, Daffertshofer A, Huys R, Beek P. Steady and transient coordination structures of walking and running. Human movement science. 2009;28(3):371–386. pmid:19027972
- View Article
- PubMed/NCBI
- Google Scholar
18. Federolf P, Reid R, Gilgien M, Haugen P, Smith G. The application of principal component analysis to quantify technique in sports. Scandinavian journal of medicine & science in sports. 2014;24(3):491–499. pmid:22436088
- View Article
- PubMed/NCBI
- Google Scholar
19. Zago M, Codari M, Iaia FM, Sforza C. Multi-segmental movements as a function of experience in karate. Journal of sports sciences. 2017;35(15):1515–1522. pmid:27560105
- View Article
- PubMed/NCBI
- Google Scholar
20. Young C, Reinkensmeyer DJ. Judging complex movement performances for excellence: a principal components analysis-based technique applied to competitive diving. Human Movement Science. 2014;36:107–122. pmid:24968369
- View Article
- PubMed/NCBI
- Google Scholar
21. Zago M, Pacifici I, Lovecchio N, Galli M, Federolf PA, Sforza C. Multi-segmental movement patterns reflect juggling complexity and skill level. Human movement science. 2017;54:144–153. pmid:28499158
- View Article
- PubMed/NCBI
- Google Scholar
22. Rethwilm R, Böhm H, Dussa CU, Federolf P. Excessive lateral trunk lean in patients with cerebral palsy: is it based on a kinematic compensatory mechanism? Frontiers in bioengineering and biotechnology. 2019;7:345. pmid:31803737
- View Article
- PubMed/NCBI
- Google Scholar
23. Federolf P, Roos L, Nigg BM. Analysis of the multi-segmental postural movement strategies utilized in bipedal, tandem and one-leg stance as quantified by a principal component decomposition of marker coordinates. Journal of biomechanics. 2013;46(15):2626–2633. pmid:24021753
- View Article
- PubMed/NCBI
- Google Scholar
24. Federolf PA. A novel approach to study human posture control: “Principal movements” obtained from a principal component analysis of kinematic marker data. Journal of biomechanics. 2016;49(3):364–370. pmid:26768228
- View Article
- PubMed/NCBI
- Google Scholar
25. Longo A, Haid T, Meulenbroek R, Federolf P. Biomechanics in posture space: Properties and relevance of principal accelerations for characterizing movement control. Journal of biomechanics. 2019;82:397–403. pmid:30527635
- View Article
- PubMed/NCBI
- Google Scholar
26. Haid TH, Doix ACM, Nigg BM, Federolf PA. Age effects in postural control analyzed via a principal component analysis of kinematic data and interpreted in relation to predictions of the optimal feedback control theory. Frontiers in aging neuroscience. 2018;10:22. pmid:29459826
- View Article
- PubMed/NCBI
- Google Scholar
27. Berret B, Bonnetblanc F, Papaxanthis C, Pozzo T. Modular control of pointing beyond arm’s length. Journal of Neuroscience. 2009;29(1):191–205. pmid:19129397
- View Article
- PubMed/NCBI
- Google Scholar
28. Wachholz F, Kockum T, Haid T, Federolf P. Changed temporal structure of neuromuscular control, rather than changed intersegment coordination, explains altered stabilographic regularity after a moderate perturbation of the postural control system. Entropy. 2019;21(6):614. pmid:33267328
- View Article
- PubMed/NCBI
- Google Scholar
29. Tits M, Tilmanne J, D’Alessandro N, Wanderley MM. Feature extraction and expertise analysis of pianists’ Motion-Captured Finger Gestures. In: ICMC; 2015.
30. Jerde TE, Soechting JF, Flanders M. Biological constraints simplify the recognition of hand shapes. IEEE transactions on biomedical engineering. 2003;50(2):265–269. pmid:12665044
- View Article
- PubMed/NCBI
- Google Scholar
31. Yan Y, Goodman JM, Moore DD, Solla SA, Bensmaia SJ. Unexpected complexity of everyday manual behaviors. Nature communications. 2020;11(1):1–8. pmid:32678102
- View Article
- PubMed/NCBI
- Google Scholar
32. Ajiboye AB, Weir R. Muscle synergies as a predictive framework for the EMG patterns of new hand postures. Journal of neural engineering. 2009;6(3):036004. pmid:19436081
- View Article
- PubMed/NCBI
- Google Scholar
33. Vinjamuri R, Patel V, Powell M, Mao ZH, Crone N. Candidates for synergies: linear discriminants versus principal components. Computational intelligence and neuroscience. 2014;2014. pmid:25143763
- View Article
- PubMed/NCBI
- Google Scholar
34. Bigand F, Prigent E, Berret B, Braffort A. How fast is Sign Language? A reevaluation of the kinematic bandwidth using motion capture. To be published in: Proceedings of the 29th European Signal Processing Conference. 2021.
35. Braffort A, Bolot L, Segouat J. Virtual signer coarticulation in Octopus, a Sign Language generation platform. In: GW 2011: The 9th International Gesture Workshop; 2011.
36. Koech CC. A kinematic analysis of sign language. 2006.
37. Zhang Z, Troje NF. View-independent person identification from human gait. Neurocomputing. 2005;69(1-3):250–256.
- View Article
- Google Scholar
38. Sigal L, Fleet DJ, Troje NF, Livne M. Human attributes from 3d pose tracking. In: European conference on computer vision. Springer; 2010. p. 243–257.
39. Belissen V, Braffort A, Gouiffès M. Dicta-Sign-LSF-v2: remake of a continuous French sign language dialogue corpus and a first baseline for automatic sign language processing. In: LREC 2020, 12th Conference on Language Resources and Evaluation; 2020.
40. Brown CY, Asada HH. Inter-finger coordination and postural synergies in robot hands via mechanical implementation of principal components analysis. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2007. p. 2877–2882.
41. Ciocarlie M, Goldfeder C, Allen P. Dimensionality reduction for hand-independent dexterous robotic grasping. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2007. p. 3270–3275.
42. Bigand F, Prigent E, Berret B, Braffort A. Machine Learning of Motion Statistics Reveals the Kinematic Signature of the Identity of a Person in Sign Language. Frontiers in Bioengineering and Biotechnology. 2021; p. 603. pmid:34368103
- View Article
- PubMed/NCBI
- Google Scholar
43. Benchiheub MeF, Berret B, Braffort A. Collecting and Analysing a Motion-Capture Corpus of French Sign Language. In: Workshop on the Representation and Processing of Sign Languages. Portoroz, Slovenia; 2016. Available from: https://hal.archives-ouvertes.fr/hal-01633625.
44. Benchiheub MEF, Braffort A, Berret B, Verrecchia C. MOCAP1; 2016. https://www.ortolang.fr/market/item/mocap1.
45. Welch P. The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacoustics. 1967;15(2):70–73.
- View Article
- Google Scholar
46. Stetter BJ, Herzog M, Möhler F, Sell S, Stein T. Modularity in motor control: similarities in kinematic synergies across varying locomotion tasks. Frontiers in sports and active living. 2020;2. pmid:33345175
- View Article
- PubMed/NCBI
- Google Scholar
47. Singh RE, Iqbal K, White G. Muscle synergy adaptation during a complex postural stabilization task. In: 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE; 2018. p. 1–4.
48. Todorov E, Ghahramani Z. Analysis of the synergies underlying complex hand manipulation. In: The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. vol. 2. IEEE; 2004. p. 4637–4640.
49. Furuya S, Flanders M, Soechting JF. Hand kinematics of piano playing. Journal of neurophysiology. 2011;106(6):2849–2864. pmid:21880938
- View Article
- PubMed/NCBI
- Google Scholar
50. Santello M, Flanders M, Soechting JF. Postural hand synergies for tool use. Journal of neuroscience. 1998;18(23):10105–10115. pmid:9822764
- View Article
- PubMed/NCBI
- Google Scholar
51. Promsri A, Federolf P. Analysis of postural control using principal component analysis: The relevance of postural accelerations and of their frequency dependency for selecting the number of movement components. Frontiers in Bioengineering and Biotechnology. 2020;8:480. pmid:32509755
- View Article
- PubMed/NCBI
- Google Scholar
52. Koller O, Zargaran O, Ney H, Bowden R. Deep sign: Hybrid CNN-HMM for continuous sign language recognition. In: Proceedings of the British Machine Vision Conference 2016; 2016.
53. Filhol M, Mcdonald J, Wolfe R. Synthesizing sign language by connecting linguistically structured descriptions to a multi-track animation system. In: International Conference on Universal Access in Human-Computer Interaction. Springer; 2017. p. 27–40.
54. Filhol M, Mcdonald J. The Synthesis of Complex Shape Deployments in Sign Language. In: Workshop on the Representation and Processing of Sign Languages; 2020.
55. Wolfe R, McDonald J, Schnepp JC. Avatar to depict sign language: Building from reusable hand animation. 2011.
56. Kipp M, Heloir A, Nguyen Q. Sign language avatars: Animation and comprehensibility. In: International Workshop on Intelligent Virtual Agents. Springer; 2011. p. 113–126.
57. Benchiheub MEF. Contribution à l’analyse des mouvements 3D de la Langue des Signes Française (LSF) en Action et en Perception. Université Paris-Saclay (ComUE); 2017.
58. Foulds RA. Biomechanical and perceptual constraints on the bandwidth requirements of sign language. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2004;12(1):65–72. pmid:15068189
- View Article
- PubMed/NCBI
- Google Scholar
59. Tartter VC, Knowlton KC. Perception of sign language from an array of 27 moving spots. Nature. 1981;289(5799):676–678. pmid:7464933
- View Article
- PubMed/NCBI
- Google Scholar
60. Gibet S. Building french sign language motion capture corpora for signing avatars. In: Workshop on the Representation and Processing of Sign Languages: Involving the Language Community, LREC 2018; 2018.
61. Lu P, Huenerfauth M. Collecting a motion-capture corpus of American Sign Language for data-driven generation research. In: Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies; 2010. p. 89–97.

[ref1] 1. of the Deaf WF. Our work; 2016. Available from: http://wfdeaf.org/our-work/.

[ref2] 2. Bruton M, O’Dwyer N. Synergies in coordination: A comprehensive overview of neural, computational, and behavioral approaches. Journal of Neurophysiology. 2018;120(6):2761–2774. pmid:30281388
View Article
PubMed/NCBI
Google Scholar

[3] View Article

[4] PubMed/NCBI

[5] Google Scholar

[ref3] 3. ElKoura G, Singh K. Handrix: animating the human hand. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation; 2003. p. 110–119.

[ref4] 4. Bernstein N. The co-ordination and regulation of movements. The co-ordination and regulation of movements. 1966.

[ref5] 5. Flash T, Hochner B. Motor primitives in vertebrates and invertebrates. Current opinion in neurobiology. 2005;15(6):660–666. pmid:16275056
View Article
PubMed/NCBI
Google Scholar

[9] View Article

[10] PubMed/NCBI

[11] Google Scholar

[ref6] 6. Tresch MC, Jarc A. The case for and against muscle synergies. Current opinion in neurobiology. 2009;19(6):601–607. pmid:19828310
View Article
PubMed/NCBI
Google Scholar

[13] View Article

[14] PubMed/NCBI

[15] Google Scholar

[ref7] 7. Santello M, Bianchi M, Gabiccini M, Ricciardi E, Salvietti G, Prattichizzo D, et al. Hand synergies: integration of robotics and neuroscience for understanding the control of biological and artificial hands. Physics of life reviews. 2016;17:1–23. pmid:26923030
View Article
PubMed/NCBI
Google Scholar

[17] View Article

[18] PubMed/NCBI

[19] Google Scholar

[ref8] 8. d’Avella A. Modularity for motor control and motor learning. Progress in Motor Control. 2016; p. 3–19. pmid:28035557
View Article
PubMed/NCBI
Google Scholar

[21] View Article

[22] PubMed/NCBI

[23] Google Scholar

[ref9] 9. Giszter SF. Motor primitives—new data and future questions. Current opinion in neurobiology. 2015;33:156–165. pmid:25912883
View Article
PubMed/NCBI
Google Scholar

[25] View Article

[26] PubMed/NCBI

[27] Google Scholar

[ref10] 10. Troje NF. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of vision. 2002;2(5):2–2. pmid:12678652
View Article
PubMed/NCBI
Google Scholar

[29] View Article

[30] PubMed/NCBI

[31] Google Scholar

[ref11] 11. O’Toole AJ, Abdi H, Deffenbacher KA, Valentin D. Low-dimensional representation of faces in higher dimensions of the face space. JOSA A. 1993;10(3):405–411.
View Article
Google Scholar

[33] View Article

[34] Google Scholar

[ref12] 12. Kuhn R, Junqua JC, Nguyen P, Niedzielski N. Rapid speaker adaptation in eigenvoice space. IEEE Transactions on Speech and Audio Processing. 2000;8(6):695–707.
View Article
Google Scholar

[36] View Article

[37] Google Scholar

[ref13] 13. Wang X, O’Dwyer N, Halaki M. A review on the coordinative structure of human walking and the application of principal component analysis. Neural regeneration research. 2013;8(7):662. pmid:25206712
View Article
PubMed/NCBI
Google Scholar

[39] View Article

[40] PubMed/NCBI

[41] Google Scholar

[ref14] 14. Federolf P, Boyer K, Andriacchi T. Application of principal component analysis in clinical gait research: identification of systematic differences between healthy and medial knee-osteoarthritic gait. Journal of biomechanics. 2013;46(13):2173–2178. pmid:23910389
View Article
PubMed/NCBI
Google Scholar

[43] View Article

[44] PubMed/NCBI

[45] Google Scholar

[ref15] 15. Zago M, Sforza C, Bona A, Cimolin V, Costici PF, Condoluci C, et al. How multi segmental patterns deviate in spastic diplegia from typical developed. Clinical Biomechanics. 2017;48:103–109. pmid:28806590
View Article
PubMed/NCBI
Google Scholar

[47] View Article

[48] PubMed/NCBI

[49] Google Scholar

[ref16] 16. Daffertshofer A, Lamoth CJ, Meijer OG, Beek PJ. PCA in studying coordination and variability: a tutorial. Clinical biomechanics. 2004;19(4):415–428. pmid:15109763
View Article
PubMed/NCBI
Google Scholar

[51] View Article

[52] PubMed/NCBI

[53] Google Scholar

[ref17] 17. Lamoth C, Daffertshofer A, Huys R, Beek P. Steady and transient coordination structures of walking and running. Human movement science. 2009;28(3):371–386. pmid:19027972
View Article
PubMed/NCBI
Google Scholar

[55] View Article

[56] PubMed/NCBI

[57] Google Scholar

[ref18] 18. Federolf P, Reid R, Gilgien M, Haugen P, Smith G. The application of principal component analysis to quantify technique in sports. Scandinavian journal of medicine & science in sports. 2014;24(3):491–499. pmid:22436088
View Article
PubMed/NCBI
Google Scholar

[59] View Article

[60] PubMed/NCBI

[61] Google Scholar

[ref19] 19. Zago M, Codari M, Iaia FM, Sforza C. Multi-segmental movements as a function of experience in karate. Journal of sports sciences. 2017;35(15):1515–1522. pmid:27560105
View Article
PubMed/NCBI
Google Scholar

[63] View Article

[64] PubMed/NCBI

[65] Google Scholar

[ref20] 20. Young C, Reinkensmeyer DJ. Judging complex movement performances for excellence: a principal components analysis-based technique applied to competitive diving. Human Movement Science. 2014;36:107–122. pmid:24968369
View Article
PubMed/NCBI
Google Scholar

[67] View Article

[68] PubMed/NCBI

[69] Google Scholar

[ref21] 21. Zago M, Pacifici I, Lovecchio N, Galli M, Federolf PA, Sforza C. Multi-segmental movement patterns reflect juggling complexity and skill level. Human movement science. 2017;54:144–153. pmid:28499158
View Article
PubMed/NCBI
Google Scholar

[71] View Article

[72] PubMed/NCBI

[73] Google Scholar

[ref22] 22. Rethwilm R, Böhm H, Dussa CU, Federolf P. Excessive lateral trunk lean in patients with cerebral palsy: is it based on a kinematic compensatory mechanism? Frontiers in bioengineering and biotechnology. 2019;7:345. pmid:31803737
View Article
PubMed/NCBI
Google Scholar

[75] View Article

[76] PubMed/NCBI

[77] Google Scholar

[ref23] 23. Federolf P, Roos L, Nigg BM. Analysis of the multi-segmental postural movement strategies utilized in bipedal, tandem and one-leg stance as quantified by a principal component decomposition of marker coordinates. Journal of biomechanics. 2013;46(15):2626–2633. pmid:24021753
View Article
PubMed/NCBI
Google Scholar

[79] View Article

[80] PubMed/NCBI

[81] Google Scholar

[ref24] 24. Federolf PA. A novel approach to study human posture control: “Principal movements” obtained from a principal component analysis of kinematic marker data. Journal of biomechanics. 2016;49(3):364–370. pmid:26768228
View Article
PubMed/NCBI
Google Scholar

[83] View Article

[84] PubMed/NCBI

[85] Google Scholar

[ref25] 25. Longo A, Haid T, Meulenbroek R, Federolf P. Biomechanics in posture space: Properties and relevance of principal accelerations for characterizing movement control. Journal of biomechanics. 2019;82:397–403. pmid:30527635
View Article
PubMed/NCBI
Google Scholar

[87] View Article

[88] PubMed/NCBI

[89] Google Scholar

[ref26] 26. Haid TH, Doix ACM, Nigg BM, Federolf PA. Age effects in postural control analyzed via a principal component analysis of kinematic data and interpreted in relation to predictions of the optimal feedback control theory. Frontiers in aging neuroscience. 2018;10:22. pmid:29459826
View Article
PubMed/NCBI
Google Scholar

[91] View Article

[92] PubMed/NCBI

[93] Google Scholar

[ref27] 27. Berret B, Bonnetblanc F, Papaxanthis C, Pozzo T. Modular control of pointing beyond arm’s length. Journal of Neuroscience. 2009;29(1):191–205. pmid:19129397
View Article
PubMed/NCBI
Google Scholar

[95] View Article

[96] PubMed/NCBI

[97] Google Scholar

[ref28] 28. Wachholz F, Kockum T, Haid T, Federolf P. Changed temporal structure of neuromuscular control, rather than changed intersegment coordination, explains altered stabilographic regularity after a moderate perturbation of the postural control system. Entropy. 2019;21(6):614. pmid:33267328
View Article
PubMed/NCBI
Google Scholar

[99] View Article

[100] PubMed/NCBI

[101] Google Scholar

[ref29] 29. Tits M, Tilmanne J, D’Alessandro N, Wanderley MM. Feature extraction and expertise analysis of pianists’ Motion-Captured Finger Gestures. In: ICMC; 2015.

[ref30] 30. Jerde TE, Soechting JF, Flanders M. Biological constraints simplify the recognition of hand shapes. IEEE transactions on biomedical engineering. 2003;50(2):265–269. pmid:12665044
View Article
PubMed/NCBI
Google Scholar

[104] View Article

[105] PubMed/NCBI

[106] Google Scholar

[ref31] 31. Yan Y, Goodman JM, Moore DD, Solla SA, Bensmaia SJ. Unexpected complexity of everyday manual behaviors. Nature communications. 2020;11(1):1–8. pmid:32678102
View Article
PubMed/NCBI
Google Scholar

[108] View Article

[109] PubMed/NCBI

[110] Google Scholar

[ref32] 32. Ajiboye AB, Weir R. Muscle synergies as a predictive framework for the EMG patterns of new hand postures. Journal of neural engineering. 2009;6(3):036004. pmid:19436081
View Article
PubMed/NCBI
Google Scholar

[112] View Article

[113] PubMed/NCBI

[114] Google Scholar

[ref33] 33. Vinjamuri R, Patel V, Powell M, Mao ZH, Crone N. Candidates for synergies: linear discriminants versus principal components. Computational intelligence and neuroscience. 2014;2014. pmid:25143763
View Article
PubMed/NCBI
Google Scholar

[116] View Article

[117] PubMed/NCBI

[118] Google Scholar

[ref34] 34. Bigand F, Prigent E, Berret B, Braffort A. How fast is Sign Language? A reevaluation of the kinematic bandwidth using motion capture. To be published in: Proceedings of the 29th European Signal Processing Conference. 2021.

[ref35] 35. Braffort A, Bolot L, Segouat J. Virtual signer coarticulation in Octopus, a Sign Language generation platform. In: GW 2011: The 9th International Gesture Workshop; 2011.

[ref36] 36. Koech CC. A kinematic analysis of sign language. 2006.

[ref37] 37. Zhang Z, Troje NF. View-independent person identification from human gait. Neurocomputing. 2005;69(1-3):250–256.
View Article
Google Scholar

[123] View Article

[124] Google Scholar

[ref38] 38. Sigal L, Fleet DJ, Troje NF, Livne M. Human attributes from 3d pose tracking. In: European conference on computer vision. Springer; 2010. p. 243–257.

[ref39] 39. Belissen V, Braffort A, Gouiffès M. Dicta-Sign-LSF-v2: remake of a continuous French sign language dialogue corpus and a first baseline for automatic sign language processing. In: LREC 2020, 12th Conference on Language Resources and Evaluation; 2020.

[ref40] 40. Brown CY, Asada HH. Inter-finger coordination and postural synergies in robot hands via mechanical implementation of principal components analysis. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2007. p. 2877–2882.

[ref41] 41. Ciocarlie M, Goldfeder C, Allen P. Dimensionality reduction for hand-independent dexterous robotic grasping. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2007. p. 3270–3275.

[ref42] 42. Bigand F, Prigent E, Berret B, Braffort A. Machine Learning of Motion Statistics Reveals the Kinematic Signature of the Identity of a Person in Sign Language. Frontiers in Bioengineering and Biotechnology. 2021; p. 603. pmid:34368103
View Article
PubMed/NCBI
Google Scholar

[130] View Article

[131] PubMed/NCBI

[132] Google Scholar

[ref43] 43. Benchiheub MeF, Berret B, Braffort A. Collecting and Analysing a Motion-Capture Corpus of French Sign Language. In: Workshop on the Representation and Processing of Sign Languages. Portoroz, Slovenia; 2016. Available from: https://hal.archives-ouvertes.fr/hal-01633625.

[ref44] 44. Benchiheub MEF, Braffort A, Berret B, Verrecchia C. MOCAP1; 2016. https://www.ortolang.fr/market/item/mocap1.

[ref45] 45. Welch P. The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on audio and electroacoustics. 1967;15(2):70–73.
View Article
Google Scholar

[136] View Article

[137] Google Scholar

[ref46] 46. Stetter BJ, Herzog M, Möhler F, Sell S, Stein T. Modularity in motor control: similarities in kinematic synergies across varying locomotion tasks. Frontiers in sports and active living. 2020;2. pmid:33345175
View Article
PubMed/NCBI
Google Scholar

[139] View Article

[140] PubMed/NCBI

[141] Google Scholar

[ref47] 47. Singh RE, Iqbal K, White G. Muscle synergy adaptation during a complex postural stabilization task. In: 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE; 2018. p. 1–4.

[ref48] 48. Todorov E, Ghahramani Z. Analysis of the synergies underlying complex hand manipulation. In: The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. vol. 2. IEEE; 2004. p. 4637–4640.

[ref49] 49. Furuya S, Flanders M, Soechting JF. Hand kinematics of piano playing. Journal of neurophysiology. 2011;106(6):2849–2864. pmid:21880938
View Article
PubMed/NCBI
Google Scholar

[145] View Article

[146] PubMed/NCBI

[147] Google Scholar

[ref50] 50. Santello M, Flanders M, Soechting JF. Postural hand synergies for tool use. Journal of neuroscience. 1998;18(23):10105–10115. pmid:9822764
View Article
PubMed/NCBI
Google Scholar

[149] View Article

[150] PubMed/NCBI

[151] Google Scholar

[ref51] 51. Promsri A, Federolf P. Analysis of postural control using principal component analysis: The relevance of postural accelerations and of their frequency dependency for selecting the number of movement components. Frontiers in Bioengineering and Biotechnology. 2020;8:480. pmid:32509755
View Article
PubMed/NCBI
Google Scholar

[153] View Article

[154] PubMed/NCBI

[155] Google Scholar

[ref52] 52. Koller O, Zargaran O, Ney H, Bowden R. Deep sign: Hybrid CNN-HMM for continuous sign language recognition. In: Proceedings of the British Machine Vision Conference 2016; 2016.

[ref53] 53. Filhol M, Mcdonald J, Wolfe R. Synthesizing sign language by connecting linguistically structured descriptions to a multi-track animation system. In: International Conference on Universal Access in Human-Computer Interaction. Springer; 2017. p. 27–40.

[ref54] 54. Filhol M, Mcdonald J. The Synthesis of Complex Shape Deployments in Sign Language. In: Workshop on the Representation and Processing of Sign Languages; 2020.

[ref55] 55. Wolfe R, McDonald J, Schnepp JC. Avatar to depict sign language: Building from reusable hand animation. 2011.

[ref56] 56. Kipp M, Heloir A, Nguyen Q. Sign language avatars: Animation and comprehensibility. In: International Workshop on Intelligent Virtual Agents. Springer; 2011. p. 113–126.

[ref57] 57. Benchiheub MEF. Contribution à l’analyse des mouvements 3D de la Langue des Signes Française (LSF) en Action et en Perception. Université Paris-Saclay (ComUE); 2017.

[ref58] 58. Foulds RA. Biomechanical and perceptual constraints on the bandwidth requirements of sign language. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2004;12(1):65–72. pmid:15068189
View Article
PubMed/NCBI
Google Scholar

[163] View Article

[164] PubMed/NCBI

[165] Google Scholar

[ref59] 59. Tartter VC, Knowlton KC. Perception of sign language from an array of 27 moving spots. Nature. 1981;289(5799):676–678. pmid:7464933
View Article
PubMed/NCBI
Google Scholar

[167] View Article

[168] PubMed/NCBI

[169] Google Scholar

[ref60] 60. Gibet S. Building french sign language motion capture corpora for signing avatars. In: Workshop on the Representation and Processing of Sign Languages: Involving the Language Community, LREC 2018; 2018.

[ref61] 61. Lu P, Huenerfauth M. Collecting a motion-capture corpus of American Sign Language for data-driven generation research. In: Proceedings of the NAACL HLT 2010 Workshop on Speech and Language Processing for Assistive Technologies; 2010. p. 89–97.

Figures

Abstract

Introduction

Materials and methods

Motion capture corpus

Data pre-processing

Extraction of principal movements

Statistical analyses

Results

Structure of common principal movements

Similarities in principal movements across signers

Discussion

Supporting information

S1 Fig. Frequency content of the first eight common PMs.

S1 Dataset. Principal movements.

References