Human development, Motor Babbling, and Goal Babbling

The paradigmatic example of motor acquisition is infant motor development. Humans are born without a complete innate knowledge of how their bodies are configured. They lack motor coordination and have to learn how to control their hands, heads, feet, etc., by exploratory activity. It has been observed that infants undergo a phase in their development during which they seem to perform random movements [10,11]. Piaget’s view was that motor development is organized in distinct phases and that this presumed random movement phase might serve to learn an inverse kinematic model of the body. That is, he suggested that during this random movement learning phase, infants do not perform purposeful actions. According to this hypothesis, only in a second phase, when the inverse model has been learned, infants start performing goal-directed action [10].

This idea has inspired learning algorithms in Cognitive and Developmental Robotics. From a computational perspective, the problem of inverse motor model learning can be seen as a search problem, i.e., the search for a control policy or inverse model that links the behavioural goals (states in goal space) to available actions (states in motor space). So-called Motor Babbling algorithms [e.g., 12] solve this problem in a very straightforward, brute-force manner inspired by the traditional view on infant development described above: In an initial ‘infant’ stage, robots perform random actions (Motor Babbling) to uniformly and exhaustively sample the motor space. They observe and memorize the consequences of the action to derive a generalized rule for action-outcome mappings (forward model). This model can then, in a second phase, be inverted (at least locally) for control. The resulting inverse model is used to perform goal-directed actions in the ‘adult’ robot.

Motor Babbling algorithms have also been proposed as generative models of infant motor acquisition [12,13], in line with Piaget’s stage-wise view, which furthermore is compatible with the traditional view on computational inverse model learning, i.e., that a feedforward model has to exist first to acquire an inverse model second [14,15]. However, in human development, Piaget’s view that motor learning is prevalently driven by random exploration has been criticized [11] in view of empirical evidence that infants perform goal-directed action right from the outset of motor learning [11,16], which results in complicated learning dynamics, as it has been shown in elaborate experiments [17]. Furthermore, it has been observed that the first successful reaching actions in infants appear to be executed fully feedforward [18,19], which would require the existence of an inverse model to start with. These results speak against the traditional view of a stage-wise motor acquisition process with a random exploration phase (forward model learning) followed by a phase of inverse model generation and purposeful action, and in favour of a direct acquisition of the inverse model while behaving.

Goal Babbling is a computational approach inspired by this latter idea that infant motor acquisition might be goal-directed right from the start. It has recently been proposed by Rolf, Steil & Gienger [8,9] as a model for learning of an inverse model while behaving (direct inverse model learning). Thereby, the exhaustive random sampling phase characteristic of Motor Babbling can be avoided and there is no need to distinguish explicit data collection and exploitation phases anymore. Related goal directed exploration schemes for motor learning that share the idea to guide exploration by goal-related feedback have also been proposed by others [20].

In Goal Babbling, a robot’s exploration of the motor space is not just random. Instead, it is constrained and guided by goals. A starting position (home posture) seeds the search for a successful policy. The expansion of the motor repertoire around the home posture occurs through random perturbations that are reinforced in the direction that is most associated with behavioural success: Goal-directed feedback (reinforcement) favours the future use of actions that are associated with a reduction in a goal-related error signal.

A robot thereby locally expands its action repertoire around an initially minimal inverse model control policy that maps all goals onto a single motor command, which keeps the robot resting at the home posture. Goal-irrelevant parts of the motor space or parts that contain solutions that are redundant to the initially learned one are thus never explored (see Method Sect Goal Babbling Simulations and [8,9] for details of the Goal Babbling algorithm).

In a reinforcement learning context, Goal Babbling for inverse model learning can be seen as a one-step reinforcement learning problem, where distance to the goal can be reinterpreted as a reward [21], but for a very large number of goals, which distinguishes inverse model learning from standard reinforcement learning problems. Standard episodic reinforcement learning algorithms, for instance, aim at learning a temporal control policy to solve a particular defined task or task sequence with a potentially distant reward. Many successful computational algorithms for this type of problem have been proposed in recent years [22]. They are heavily applied in robotics and have been linked to human reinforcement learning. However, these reinforcement learning algorithms, though related to Goal Babbling, are not easily applicable to the problem of direct inverse model learning from scratch discussed here. There is ample evidence [5] that (adult) human motor control relies on both forward and inverse models and the question how inverse models are learned is an open research question.

Other computational models for learning inverse models online have been proposed recently, but again require the prior existence of a working feedback controller (see the review in [23]). In all these approaches, additional measures have to be taken to deal with redundancy in the action-goal mapping. In summary, neither standard theories of model learning nor (episodic) reinforcement tackles the problem of acquiring an inverse model for a novel sensorimotor mapping in a redundant domain that we focus on here.

In the robotics domain, Goal Babbling is particularly useful for platforms that, like humans, have many and redundant Degrees of Freedom (DoFs) with non-linear dependencies between them [24]. Other inverse model learning approaches like Motor Babbling struggle to learn the inverse models for such platforms. Goal Babbling as a model for infant motor skill acquisition [25] has already been shown to reproduce the dynamics of the so-called U-shaped sequence of disappearance and reappearance of a skill for pre-reaching to balls presented to infants of early age [17]. Here, we study Goal Babbling as a computational model for adult human motor learning. The experiment we present here investigates whether learning dynamics similar to Goal Babbling might occur when humans learn an unfamiliar and redundant sensorimotor mapping from scratch: It tests for Goal Babbling-like motor exploration (i.e., local goal-directed expansion of the action repertoire, no distinction between data collection and exploitation phases) in a motor skill acquisition task with adult volunteers (see Introduction Sects Motor Skill Acquisition in Human Adults and Do adult humans use error feedback to guide motor skill acquisition?).

Motor skill acquisition in human adults

Human adults who have already mastered control over their bodies can also learn new high-dimensional and redundant motor tasks, when interacting with objects such as musical instruments, tools, toys, or sports equipment. This kind of motor skill acquisition differs from motor development in the sense that it involves adding a new layer on top of the already familiar control of one’s body. However, the requirements posed on the neural mechanisms of learning are similar: The interaction with the object brings about sensory effects in response to motor actions, such as sounds when playing instruments, forces when operating tools, or visual effects when playing computer games. The rules that connect sensory effects with movements are initially unfamiliar. To master control of the interaction, humans have to explore the oftentimes high-dimensional and non-linear mapping from states in motor space to states in goal space and to constrain and memorize the non-unique inverse mapping from goal states to motor states, much like during infant motor development. We here test whether Goal Babbling simulations can qualitatively predict the dynamics and outcome of motor skill acquisition in adult humans, which would suggest that humans might use similar principles for motor exploration and inverse model learning.

It has been suggested that motor skill acquisition involves the formation of Motor Synergies, i.e., of patterns of DoF co-activation that can serve as motor primitives and reduce the dimensionality of a motor control problem [7,26]. In many tasks that involve redundant DoFs, motor behaviour has been shown to be reducible to the linear combination of just two or three spatio-temporal movement components (synergies), for instance, kicking behaviour in frogs [27], human object grasping [28] or even full body walking and reaching [29]. The question of redundancy resolution can therefore also be understood as a question of synergy formation. We here use this approach, that is, we measure motor organization and dimensionality reduction by synergy formation, which we define as principal components in the motor space (see Method Sect Data Analysis). This approach has also been previously used to describe redundancy resolution in robotic Goal Babbling [30].

Do adult humans use error feedback to guide motor skill acquisition?

The focus of our work is on human adult learning of a new, redundant inverse sensorimotor mapping from goals to actions. A number of principles have been shown to influence motor learning of redundant motor control problems. For instance, participants adapt behaviour optimally according to simple, goal-related cost functions [2, 31] as well as to achieve dynamic stability of control [32, 33]. Here, we were interested primarily in principles of motor exploration and whether they follow more the traditional view (which we here call “Motor Babbling”), where a random exploration phase is followed by a goal-directed action phase, or the integrated process view that underlies the Goal Babbling approach. Do humans use goal-related feedback right from the start to guide the exploration of a motor space and the continuous formation of synergies as in Goal Babbling? In particular, we were interested in testing the following predictions from Goal Babbling:

1. P1. Synergies are learned gradually during the course of the learning dynamics (motor organization).
2. We here define synergies as a subset of motor Principle Components that can explain most of the variance in motor space (dimensionality reduction) and that represent DoF co-activation patterns [30] (Results Sect Motor Synergy Formation, Methods Sect Postures in Motor Space (q₁, q₂, q₃)). Goal Babbling predicts the gradual formation of synergies, Motor Babbling predicts a phase of random exploration (no synergies) followed by a phase of goal-directed action (possible transition to using synergies).
3. P2.The inverse model gradually unfolds from the home posture (starting condition) in motor space and is biased towards its location.
4. Comparing two conditions with different home postures, Goal Babbling predicts that the learned solutions will expand away from the home posture and might remain biased towards their location (Results Sect Location of solutions). Motor Babbling suggests exhaustive sampling of the motor space and thus predicts no such bias.
5. P3. If local movement of a particular DoF is associated with partial task success, this DoF will be used more for exploration.
6. We manipulate the local effectivity of the DoFs by choice of the home posture in the two conditions. Goal Babbling predicts that a DoF is used more for action in the condition where it is more locally effective (we measure this by its contribution to the first synergy PC1, Results Sect Relative use of DoFs in synergies). Motor Babbling predicts no differences in use, as exploration is deemed random.

In summary, Motor Babbling predicts distinguishable phases of exploration and goal-directed behaviour as well as learning outcomes that do not depend on the condition (home posture). Goal Babbling predicts an integrated exploration, learning and goal-directed action process, where task-related feedback for action guides and biases the learning process and outcome for inverse model learning.

To test these predictions, we trained human participants in an unfamiliar and redundant sensorimotor control task (Fig 1 and Method Sects Task and Experimental Setup and Procedure). We also qualitatively compared the results to those of a population of simulated agents that were trained on the same task using the Goal Babbling learning approach (Method Sects Goal Babbling Simulations), to illustrate the Goal Babbling predictions more concretely. For these agents, the task was to set the joint angles of a three DoF planar arm (Fig 1A) to reach to targets in a quarter segment of the circular reach space (Fig 1B, bottom, and Method Sect Task). Human participants were trained in the same task but the input variables were presented in an unfamiliar transformation to avoid that participants might rely on prior knowledge, e.g., about physics or joint kinematics, which can facilitate motor learning [34]. Participants therefore controlled the joint angles of the task by the elevation of the left index, right index, and right middle fingers, unaware that this was internally interpreted as joint angle settings (Method Sect Experimental Setup and Procedure).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The three DoF planar arm reaching task.

A: Both humans and simulated agents learned to move a three DoF planar arm with joints q₁, q₂ and q₃ (three-dimensional motor space) to reach to targets in an x and y plane (two-dimensional goal space). B: Top: The home postures H₁ and H₂ are different but reach to the same point in goal space. Note that in home posture H₁, the q₂ joint is folded backwards (q₂ = π) so the reach-endpoint is located on top of the q₂ joint. Bottom: The colour map depicts the redundancy, that is the relative number of joint configurations (q₁, q₂, q₃) that reach to a given point in the goal space (lighter colours imply more joint configurations to reach this point). The task is redundant, especially for targets close to the origin of the goal space. Participants were trained and tested on targets in the top quadrant of the goal space (green disks).

https://doi.org/10.1371/journal.pcbi.1006676.g001

Moreover, the positions of targets and reach-endpoints in goal space were presented as static, deforming ellipses (Method Sect Experimental Setup and Procedure) to avoid that the solution learned is constrained by principles of spatial processing, as it is the case in human motor skill acquisition of spatial control tasks. For instance, Mosier et al. [35] have shown that participants tend to learn solutions that generate rectilinear movements of a cursor in visual space. Sailer et al. [36] have investigated hand-eye coordination in a similar task and have found that with increasing mastery of the task, participants first learned to perform predictive eye movements and later saccades to a target, which demonstrates remapping not only between hand movements and cursor movements but also alignment with oculomotor control. By presenting the targets and reach-endpoints as static, deforming ellipses, we aimed at reducing such biases inherent to spatial tasks. In summary, the mapping from finger movements to elliptic shapes was, mathematically speaking, that of reaching with a three DoF planar arm (Fig 1). However, participants experienced this as deforming ellipses with up-down finger movements (Method Sect Experimental Setup and Procedure).

The task we chose is redundant: There are infinitely many DoF configurations to reach each target. Especially targets that are close to the origin of the goal space can be reached from many parts of the motor space (Fig 1B, bottom). Note that the degree of redundancy, here measured as number of different configurations to reach a particular target in a discretized version of the task is non-homogeneous and strongly varying across the target space (Fig 1B, bottom, colour coding). We analysed redundancy resolution by performing Principal Component Analysis (PCA) on the reach-endpoints in motor space in a test phase with no feedback. Thereby, we could identify DoF co-activation patterns (synergies) in the learned inverse mappings from goal states to motor states that can reduce the dimensionality of a complex, redundant control problem and are a sign of motor organization (Method Sect Data Analysis).

We compared the learning dynamics and outcome starting from two different initial positions (home postures H₁ and H₂; Fig 1B, top). These home postures reach to the same target in goal space but are located in different parts of the motor space (Fig 1B, top). In particular, in H₁ the reach-endpoint is located on top of the q₂ DoF, which modulates the effectiveness of this joint. The different home postures predict substantial differences in learning dynamics and outcome for predictions P2 and P3. If human learning is sensitive to the home posture in the ways predicted, this would indicate that also humans use local, goal-related feedback to structure exploration behaviour right from the start. The Goal Babbling simulations results are presented alongside the results from the human experiment in Sect Results for illustrative purposes.

Performance

Participants learned to perform the task in both conditions within the time allocated (70 minutes). They acquired an inverse model of the trained mapping over the course of the 24 training blocks, which allowed them to move the reach-endpoints on average closer to the target than at the beginning of training and above chance in the test blocks without feedback (Fig 2B). A two-way repeated-measures ANOVA (rmANOVA) on the average distance to the targets (error) with factors Time (block number) and Condition (H₁, H₂) shows a main effect of the factor Time (F(23,940) = 10.9, p≪0.001), confirming that participants learned across blocks. There are neither a significant main effect of the factor Condition (F(1,959) = 0.01, p = 0.944) nor an interaction between the factors Condition and Time (F(23,959) = 0.5, p = 0.507; full ANOVA table in S1 Table). We had hypothesised that participants might learn faster starting in home posture H₂, as it was the case for the Artificial agents (Fig 2A). For the agents, the extended q₃ joint in H₂ (Fig 1B) made it easier to expand the goal space (cf. Results Sect Relative use of DoFs in synergies), so even after one block of learning, the reach error was much lower than in condition H₁. However, there was no such condition-dependent difference in learning speed in the human population (Fig 2B and 2C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Average performance across blocks.

Mean and standard error of the mean in Condition H₂ (red) and in Condition H₁ (blue). The black line indicates baseline performance if the hand is not moved at all (slightly lower during training as targets are spaced differently). A: Artificial Agents, test B: Human participants, test. C: Human participants, training.

https://doi.org/10.1371/journal.pcbi.1006676.g002

Moreover, the human participants’ performance error remained at a higher level than in the Goal Babbling simulations (Fig 2A). Also, the test performance error (Fig 2B) converged at a higher value earlier than the training error (Fig 2C). This was likely due to proprioceptive and motor noise when executing the intended actions. More detailed results about reach performance during test can be found in S1 Fig (individual learning curves for participants), S2 Fig and S3 Fig (Distribution of reach-endpoints for individual test targets in agents (S2 Fig) and participants (S3 Fig)) and in S4 Fig (performance across time for different targets).

Redundancy resolution

Motor synergy formation.

To identify Motor Synergies, i.e., patterns of DoF co-activation that characterize a solution to the task and reduce dimensionality (prediction P1), we performed a PCA on the set of postures used to reach the targets in each three consecutive blocks (Method Sect Postures in Motor Space (q₁, q₂, q₃)). Even though we were interested in inverse model learning, i.e., motor behaviour in the absence of goal-related feedback, we present results from both the training blocks (with feedback) and the test blocks (without feedback), as the training results are less noisy (cf. Fig 2B and 2C) but show the same qualitative behaviour.

Fig 3A depicts the amount of variance explained by the first, second, and third principal components (motor synergies PC1, PC2 and PC3) across the artificial agent populations for both conditions. Fig 3B and 3C depicts the same measure for the human participants in the test phase (Fig 3B) and the training phase (Fig 3C). For both populations, the first two synergies PC1 and PC2 can together explain >90% of the total variance at the end of training, which is evidence for motor organization and dimensionality reduction by synergies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Synergy formation during motor skill acquisition.

The percentage of total variance explained by each PC in artificial agents (A) and in human participants during the test blocks (B) and the training blocks (C). Both agents and human participants need two PCs to explain >90% of the total variance in reach postures (q₁, q₂, q₃) at the end of training. In human participants, this involves a dimensionality reduction, i.e., the amount of variance explained by PC1 goes up with training (sign-test difference beginning vs. end: p = 0.041 in test; training p = 0.263), whereas the amount of variance explained by PC3 goes down (sign test difference beginning vs. end: p<0.001 in test, p = 0.041 in training). The lines in panels A, B and C depict the across-participant median, the error bars the interquartile range.

https://doi.org/10.1371/journal.pcbi.1006676.g003

For the human participants, motor organization is already evident after one block, but also increases during training: The median variance explained by PC1 increases with training blocks, whereas the variance explained by PC3 decreases (see caption Fig 3 for statistical test results; the increase in variance explained by PC1 is only significant for the test results, not the training results). In the Goal Babbling simulations, by contrast, motor behaviour is most organized right at the start: agents started off just one synergy (PC1) and only add a second movement component (PC2) over the course of learning (Fig 3A). This difference between artificial agents and human participants will be addressed again in Results Sect Relative use of DoFs in synergies and in Sect Discussion. In both populations, the formation of synergies proceeded gradually, as it would be expected from a Goal Babbling-like integrated process of learning, exploration and behaviour (P1). A transition from an exploration phase to a goal-oriented action phase, as it would be predicted by Motor Babbling, is not evident.

Examples of redundancy resolution.

Fig 4 illustrates the motor redundancy in the task and its solutions with two different learning outcomes for example participants tested in Condition H₁ (Fig 4A and 4B) and Condition H₂ (Fig 4C and 4D). These examples are not chosen because they are particularly representative, but to illustrate the across-participant variation and how they are captured by the measures investigated in Results Sects Location of solutions, Relative use of DoFs in synergies and Absolute variability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Examples of synergies learned.

Two example solutions from different participants in the test phase for Condition H₁ (A+B) and Condition H₂ (C+D) respectively (first two synergies PC1 and PC2). Panels A and C depict the PC1/PC2 solution plane in the three-dimensional motor space. The plane extends 2σ (standard deviations) away from the central posture in the directions of PC1 and PC2 respectively. The pale ‘shadows’ are projections of the results onto the planes defined by the coordinate axes for a better impression of 3D shape. The empty disks depict the two home postures H₁ and H₂. Panels B and D depict the same result in the goal space (colour code is identical to A and C). The thick lines depict the arm configurations that correspond to the points of the same colour in motor space. The thin lines depict the reach-endpoints along the grid that connects the dots. The inlays depict the respective home postures H₁ and H₂.

https://doi.org/10.1371/journal.pcbi.1006676.g004

Fig 4A and 4C show the plane defined by the first two synergies PC1 and PC2 in motor space; Fig 4B and 4D show the corresponding postures and reach-endpoints in goal space. The two solutions differ in many aspects. They occupy different regions in motor space, which means they make use of different postures to reach to the same targets, in this case postures that are more similar to their respective training home posture (cf. Fig 1B). The solutions also have different orientations in motor space: For instance, the solution learned in Condition H₂ (Fig 4C) makes use of all three DoFs q₁, q₂ and q₃, whereas the solution learned in Condition H₁ (Fig 4A) nearly exclusively uses the q₁ and q₃ DoFs (different slants of the two planes). Lastly, the PC1/PC2-planes for the two solutions differ in area size: The range of joint angles used is larger in Condition H₁ (Fig 4A) than in Condition H₂ (Fig 4C). Despite all these differences, the two example solutions are equally viable solutions to the reaching task trained. In both cases, the reach actions associated with the solution planes cover most but not all of the area in the goal space where the test targets are located (Fig 4B and 4D).

The following sections take a closer look at the location of solutions in motor space in the light of prediction P2 (Results Sect Location of solutions), the relative use of DoFs in synergies in the light of prediction P3 (Results Sect Relative use of DoFs in synergies), and at the overall variability of postures used (Results Sect Absolute variability).

Location of solutions.

Fig 5A and 5D show the postures at the centre of the learned solution manifolds (central postures) for artificial agents (Fig 5A) and for human participants (Fig 5D) at the end of training. In the agent population, the distribution of learned central postures follows an arch in motor space that connects H₁ and H₂ (Fig 5A). This arch corresponds to a manifold of postures that reach to the central target in goal space (Fig 5B). The central postures remain very close to the training home posture in motor space (Fig 5C). This is owed to the Goal Babbling learning procedure, which introduces small variations to gradually expand the motor repertoire from the home posture in the direction of behavioural success (prediction P2). In Motor Babbling, no such influence of the home posture is expected, as the random initial exploration is exhaustive, not local, which should lead to identical learning outcomes (forward and inverse models) in both conditions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Location of learned solutions in motor and goal space at the end of training.

A-C: Artificial Agents. D-G: Human Participants. Panels A and D depict the central postures (mid-point of learned solutions) in motor space for all 100 agents and 20 participants (Cond H₁: blue, Cond H₂: red). Panels B and E depict the corresponding arm configurations in goal space, which shows that the reach-endpoints of the central postures cluster around the mid-point of the array of test targets. Panels C, F and G depict the average location of solutions relative to H₁ and H₂ across time (population mean and standard error of the projection on the line connecting H₁ and H₂). Both agents (C) and humans (test F, training G) are biased towards the starting home posture, but this bias is weaker in humans.

https://doi.org/10.1371/journal.pcbi.1006676.g005

The human test population showed the same pattern as the agents who learn the task by Goal Babbling, albeit with more noise and a weaker bias towards the home posture (Fig 5D–5G): A two-way rmANOVA with the factors Condition and Time reveals a significant main effect of the factor Condition (F(1,300) = 4.6, p = 0.046), i.e., the learned solutions are biased towards the home posture. There is no main effect of the factor Time (F(7,300) = 0.4, p = 0.875) and no interaction between the factors Time and Condition (F(7,300) = 0.6, p = 0.725; full ANOVA table in S2 Table).

Relative use of DoFs in synergies.

To assess the relative use of DoFs, we analysed the absolute q₁, q₂ and q₃ components in the first two synergies, and in particular the use of q₂ in the first synergy (PC1). In Goal Babbling, motor learning favours DoFs that cause maximum variability in the goal space, as the exploratory noise has the biggest task-error-reducing effect when using these DoFs (prediction P3; [cf.30]). We selected the training home postures with this constraint in mind. In H₁, the q₃ DoF is fully contracted (cf. Fig 6B), which renders movement of the q₂ DoF ineffective in goal space, as the reach-endpoint rests on top of the q₂ joint. Therefore, the q₂ DoF did not play a prominent role in the solution learned by the agents in Condition H₁ (Fig 6A top). By contrast, in H₂ the q₃ joint is fully extended. Both configurations are usually singular in actual robotics but this is not important for the purposes of the experiment presented here. This makes movement of the q₂ DoF maximally effective in the goal space, so the agents incorporated q₂ movements into the learned solution (Fig 6A bottom). This difference increased with training (Fig 6A and 6D top). In agreement with previous work [30], variance in the DoFs that have the smallest effect on reach-endpoints (q₂ in Condition H₁, q₃ in Condition H₂) is mostly explained by PC3, which contributes least to solving the task.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The relative use of DoFs in learned synergies.

A+C: Population median of the absolute values of q₁, q₂ and q₃ in PC1, PC2 and PC3 (unit length vectors) in artificial agents (A) and human participants during training (C). Brighter colours indicate a higher absolute value of the respective DoF. B: Illustration of the home postures H₁ (blue) and H₂ (red) in goal space. Note that the q₃ joint is contracted in H₁ and extended in H₂. D: Median and interquartile range of absolute q₂ values in PC1 across agents (top) and participants during training (bottom).

https://doi.org/10.1371/journal.pcbi.1006676.g006

The results from human participants differed substantially from the Goal Babbling simulation results and are overall noisier. However, it almost appears as if PC1 and PC3 were inverted between the two populations (cf. Fig 6A and 6C). In both conditions, q₁ contributed least to PC1 in the human participants. In both conditions, the variance of q₁ is largely explained by PC3, that is the least relevant movement component. Also, the influence of home posture on the use of q₂ in PC1 follows the opposite trend for the human participants: The q₂ DoF was used more in Condition H₁, where it was ineffective, than in Condition H₂ (Fig 6C and 6D bottom). However, this trend is not significant (Wilcoxon signed-rank test of the median difference between conditions across time: z = 1.38, p = 0.167). The initial exploratory activity was not as strongly modulated by the local effectiveness of DoFs as in the simulated agents, and any possible trend has the opposite polarity: Increased local effectiveness of a DoF tended to diminish the use of DoFs instead of boosting it.

This trend to invert prediction P3 is neither agreeable with the Goal Babbling simulation nor with a Motor Babbling explanation, which would predict an equal use of all DoFs during the exploratory phase, irrespective of how effective they are. This pattern suggests that goal-relevant feedback might have been used in a different way by the human participants than by the Goal Babbling agents (see Discussion). However, as this trend is not significant, further experiment with less motor noise would be required to draw conclusions from this.

Absolute variability.

In Goal Babbling, local noise around a home posture leads to a gradual, feedback-guided expansion of the region in motor space that is used to solve a task (Sect Goal Babbling Simulations). This can be seen in the summed variance in motor space (Fig 7 left), which starts at zero in the artificial agent population, grows during learning and converges when a functional solution is found. In humans, overall variability started high and decreased across time (a two-way ANOVA with the factors Time and Condition shows a main effect of the factor Time: F(7,300) = 2.4, p = 0.024), which again is an inversion of the behaviour observed for agents. The overall variability in postures used was not modulated by the home posture (no main effect of factor: Condition; F(1,300) = 0.1, p = 0.762 and no interaction between the factors Time and Condition; F(7,300) = 1.0, p = 0.457; full ANOVA table in S3 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Absolute variability and influence of finger mapping on motor organization.

The summed variance of all three synergies for artificial agents (left) and human participants (right) across time (median and interquartile range).

https://doi.org/10.1371/journal.pcbi.1006676.g007

This difference between agents and humans might explain some of the previously reported differences between the two populations. It appears that humans initially explored large parts of the motor space (high variability, Fig 7 right) to sample it with comparably little direction or structure (more synergies are necessary to explain initial behaviour, Fig 3B and 3C). In this sense, human exploration can be seen as more random and less goal-directed than in the population of Goal Babbling agents. However, the evidence for synergy formation (P1, Sect Motor Synergy Formation) and a location bias (P2, Results Sect Location of solutions) starting from the first block of experimentation onwards and throughout the experiment still is evidence for a Goal Babbling-like integrated exploration, learning and goal-oriented action process (with a large learning rate or a lot of exploratory noise) in humans and cannot be explained by Motor Babbling, which predicts a random exploration phase followed by a goal-directed action phase.

Still, this higher randomness could explain why humans have avoided the use of DoFs with extreme consequences in the goal space, as random and large exploratory movements can lead to unstable effects in the task. This could then also explain why the first two synergies in the human population do not involve the q₁ DoF, which has a high local effectiveness (inversion of prediction P3, Results Sect Relative use of DoFs in synergies). The artificial agents by contrast expanded their motor repertoire only in very small steps (Fig 7 left), so they had no behavioural instability and thus initially favoured DoFs with a strong local effectiveness that allowed introducing more variance in reach-endpoints (Fig 6A and 6D). The near exclusive use of the q₁ DoF, which is the most effective DoF, can initially explain nearly all of the variance in postures in the simulated agents (Fig 3A).

In this interpretation, both humans and artificial agents used goal-related feedback to guide motor exploration and redundancy resolution, which is why systematic differences in the use of DoFs occurred in both populations (Fig 6). However, the motor exploratory strategies differed dramatically between the two populations (Fig 7), which could explain why the differential use of DoFs follows the opposite pattern in agents and humans (Fig 6A vs. Fig 6C) and why the effect of home posture is weaker in the human population (Fig 5F and 5G, Fig 6C and 6D). There are a number of possible reasons for why humans might have used a different exploratory strategy than the agents in the given task (cf. Sect Discussion).

Human morphology and task motor space

Motor skill acquisition in human adults involves adding a new layer of motor control on top of the already existing control of the body in space. In the current experiment, existing motor synergies for finger movements and biomechanical constraints of the hand likely had an effect on motor skill acquisition. To decrease these effects, we chose to randomize the assignment of fingers to task DoFs and counter-balanced this assignment across conditions.

To get an idea of the strength of such biasing effects of human morphology, we performed two additional PCAs on the reach-endpoints pooled across participants, to compare the amount of motor organization in the biological morphology motor space with the amount of motor organization in the task motor space. That is, in a first PCA, we looked for motor synergies for physical finger control (human morphology) and in a second PCA, we looked for motor synergies for task DoF control (q₁, q₂ and q₃). More variance explained by the first synergies (PCs) indicates more organization of motor control. Fig 8 shows the variance explained by PC1, PC2 and PC3 in both analyses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Percentage of variance in the across-participant reach-endpoint space that is explained by the first, second and third PC across time.

More variance explained by the first synergies indicates more organization of motor control. Left: PCA on human morphology (left index, right index, right middle fingers). Right: PCA on the task motor space (q₁, q₂, q₃). Top: test blocks, bottom: training blocks.

https://doi.org/10.1371/journal.pcbi.1006676.g008

In all test blocks, the first two synergies of finger movement (PC1 and PC2, human morphology) can explain approximately 80% of the overall variance throughout the experiment (Fig 8, left). This confirms that there was above-chance motor organization in finger control, i.e., participants used patterns of finger co-activation. Thus, human morphology influenced motor organization throughout the experiment.

In the task motor space, the variance explained by the first two synergies (PC1 and PC2) started at a similar level but then increased across the learning phase (Fig 8, right), to the point that the first two synergies explain 90% of the variance at the end of the experiment. The learned motor organization in task motor space is stronger than the organization in the previously existing morphological motor space.

Ethics statement

The experiment was approved by the ethics committee of the Bielefeld University and was conducted according to the principles expressed in the Declaration of Helsinki. Informed written consent was obtained from the participants.

Task

Both humans and simulated agents were trained in the same planar reaching task with a three DoF planar arm (Fig 1A). The three segments l₁, l₂ and l₃ of the arm are of length l₁ = 0.55, l₂ = 0.225 and l₃ = 0.225 (length in arbitrary spatial units). No joint limits were implemented: The joint angles that define a posture of the arm (q₁, q₂, q₃) could be set to any value in [0, 2π [(angles in radians).

Two different home postures H₁ and H₂ were defined as starting conditions for learning (Fig 1B; H₁ = (¾ π, 1.99, π), blue; H₂ = (1.51, 1.99, 0), red). These home postures reach to the same point (-0.39, 0.39) in goal space but are located in different parts of the motor space (Fig 1B).

The home postures are located at the edge of an array of 16 training targets that are distributed in a regular manner (in polar coordinates) in a quarter segment of the arm’s reach space (Fig 1B; distances from origin: [0.25, 0.45, 0.65, 0.85]; angles: [¼ π, 5/12 π, 7/12 π, ¾ π]). Nine additional targets were used as test targets (Fig 1B); five within the training array (polar coordinates: (0.35, 2/3 π), (0.75, 2/3 π), (0.55, ½ π), (0.35, 4/3 π), (0.75, 4/3 π)) and four just outside the training array (polar coordinates: (0.55, 1/6 π), (0.15, ½ π), (0.95, ½ π), (0.55, 5/6 π)).

Goal babbling simulations

A population of 100 simulated agents was trained on the task using the Goal Babbling procedure, which we here describe only briefly. For technical details of the procedure, please refer to [24] or to pages 65–70 of [42].

The goal space and motor space for the simulated agents were identical to those used in the experiment with human participants (Method Sect Task). That is, the forward kinematics are given by (1) where are the joint angles. Goal Babbling aims at learning the inverse model: (2) which maps desired outcomes (hand positions) to actions (joint angle configurations) such that (x,y) ≈ f(g(x,y)) for a set of targets X,Y* in the goal space. Training and test targets were identical to those used in the experiment with human participants (see visualization in Fig 1).

A local linear map with learning rate η = 0.2 is used [42,43] as learning model for the inverse function. Training starts at the home posture H₁ or H₂ (cf. Method Sect Task), depending on condition. The agent is trained using the training targets (see Fig 9C). Training involved moving the hand from a given training target x,y*(t_n) to the next training target x,y*(t_n+1) in 25 steps. That is, 24 sub-targets were generated to transition from the current training target x,y*(t_n) to the next training target x,y*(t_n+1) by linear interpolation in the goal space. The agent tried to reach these targets by utilizing the respective current estimate of the inverse function . After each step in each trial the estimate of the inverse function is immediately updated (online learning).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Setup and Procedure for the Experiment with Human Participants.

A: Participants experience the task as deforming a black ellipse to track the shape of a white ellipse by moving their left index (LI), right index (RI) and right middle (RM) fingers up and down (left). Internally, finger positions are mapped to the joint angles q₁, q₂ and q₃ of the 3 DoF planar arm (bottom right, Method Sect Task). The size and elongation of the ellipses represents the position of target and reach-endpoint in goal space (top right). B: When a participant keeps the three fingers LI, RI and MI level (bottom left), this corresponds to taking the Baseline Posture q* (top left), which is in the middle of the two home postures (top middle and top right) in motor space. To take one of the home postures, the fingers have to be moved away from the midline in a manner that depends on the random mapping between fingers and joints (example mapping: (-2, 3, -1)). C: Each session consisted of 24 training and test blocks (right). A training block (top left) was started by taking the home posture. The deforming white target ellipse had to be tracked with the black ellipse (reach-endpoint) continuously for the next 80 s. In test blocks, only the white target ellipse was displayed and participants moved their finger till they reached a configuration they thought corresponds to the target and then submitted their response by pressing a foot pedal.

https://doi.org/10.1371/journal.pcbi.1006676.g009

In each trial, exploratory noise e was added to the outcome of the local linear map estimate of the inverse function to generate novel behaviour and gradually expand the explored region of the motor space.

(3)

The noise e follows a slow and bounded random walk of linear perturbations as defined in [42], p. 68 (Equations 6.8–6.10 with parameter values σ = 0.5,σ_Δ = 0.005). The amount of noise was equal for all joints q_i. Implementing exploratory noise as a random walk ensures both continuity of behaviour (to avoid instability) and ensures that, over time, previously unknown parts of the motor space are explored.

A training block involved the presentation of 40 training targets that were selected at random from the set defined in Method Sect Task. The home posture was chosen as a target with probability 0.1. Considering the 24 sub-targets on the way to the next training target, a total of 1000 parameter updates were thus performed during learning in one block.

The agents were trained in ten subsequent blocks. Note that these blocks have no special meaning in the learning algorithm. They are solely distinguished for evaluation purposes, i.e., to assess the current estimate at regular intervals using the nine test targets that are also shown in Fig 9C. During these test blocks, the learning dynamics were disabled.

All simulation results show the average and variance over a population of 100 agents, which differ according to the random exploration scheme (exploratory noise) described above. These results are presented merely to qualitatively illustrate the kind of behaviour to be expected from Goal Babbling-like learning in the participants, as it is known from the literature and reflected in the experimental hypotheses. We therefore do not phrase or test any hypotheses about the behaviour of the simulated agents or how it quantitatively compares to the behaviour exhibited by human participants.

Experimental setup and procedure

To avoid that participants use prior knowledge about arm kinematics, space or physics to learn the task, we applied an unintuitive and unfamiliar transformation to the DoFs in motor space (joint angles q₁, q₂, q₃) and the feedback about target and reach-endpoint position (x and y coordinates). Participants controlled joint angles q_i by elevation of the left index, right index and right middle fingers (Fig 9A). They received visual feedback about reach-endpoint and target as deformation of two static ellipses (Fig 9A). That is, formally speaking, participants were controlling movements of a three DoF planar arm but their experience of the experiment was that of moving the fingers up and down to deform an ellipse. The assignment of fingers to DoFs of the simulated planar arm was randomized across participants and conditions (see Procedure). S1 Video shows a participant performing the task for illustration.

Data analysis

The following variables were analysed to assess differences in test performance between the two conditions and across time. In 1.4% of the cases, the motion sensor did not register the position of all three fingers. These data points (175 in total) were discarded and treated as missing data in all the analyses below.

For most of the variables, we tested for differences using two-way rmANOVAs to test for the effects of the factors Time (block number or bin-of-block number) and Condition (H₁, H₂). For the factor Condition, we decided to use the finger mapping as grouping variable, which was counter-balanced across participants (cf. Procedure) rather than the participant identity. Especially for synergy learning, finger mapping had a strong biasing effect (Fig 8 left), which seemed more important to consider in the analysis than participant identity (both was not possible). The ANOVA tables are reported as supporting tables S1 Table, S2 Table, S3 Table and S4 Table. Partial eta-squared () was used as a measure of statistical effect size as it is established in the literature, even if generalized eta-squared () is regarded by some as preferable [46].