Design of a Virtual Player for Joint Improvisation with Humans in the Mirror Game

Chao Zhai; Francesco Alderisio; Piotr Słowiński; Krasimira Tsaneva-Atanasova; Mario di Bernardo

doi:10.1371/journal.pone.0154361

Abstract

Joint improvisation is often observed among humans performing joint action tasks. Exploring the underlying cognitive and neural mechanisms behind the emergence of joint improvisation is an open research challenge. This paper investigates jointly improvised movements between two participants in the mirror game, a paradigmatic joint task example. First, experiments involving movement coordination of different dyads of human players are performed in order to build a human benchmark. No designation of leader and follower is given beforehand. We find that joint improvisation is characterized by the lack of a leader and high levels of movement synchronization. Then, a theoretical model is proposed to capture some features of their interaction, and a set of experiments is carried out to test and validate the model ability to reproduce the experimental observations. Furthermore, the model is used to drive a computer avatar able to successfully improvise joint motion with a human participant in real time. Finally, a convergence analysis of the proposed model is carried out to confirm its ability to reproduce joint movements between the participants.

Citation: Zhai C, Alderisio F, Słowiński P, Tsaneva-Atanasova K, di Bernardo M (2016) Design of a Virtual Player for Joint Improvisation with Humans in the Mirror Game. PLoS ONE 11(4): e0154361. https://doi.org/10.1371/journal.pone.0154361

Editor: Gonzalo G. de Polavieja, Champalimaud Foundation, PORTUGAL

Received: January 20, 2016; Accepted: April 12, 2016; Published: April 28, 2016

Copyright: © 2016 Zhai et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This work was supported by European Project AlterEgo FP7 ICT 2.9 - Cognitive 321 Sciences and Robotics, Grant Number 600610 (MdB, http://www.euromov.eu/alterego/project).

Competing interests: The authors have declared that no competing interests exist.

Introduction

Human social interactions give rise to a variety of self-organized and emergent motor behaviors [1–4]. A typical example is joint improvisation between two humans performing some task together, as engaging in a conversation or public performance [5–7]. Experimental results suggest that coordination in joint actions unconsciously fosters social rapport and promotes a sense of affinity between two individuals [8, 9].

To investigate the mechanisms behind the emergence of social interaction between two individuals, the Human Dynamic Clamp paradigm has been recently proposed in [4, 10, 11] where a model-driven avatar (or virtual player) replaces one of the two humans. In so doing, the features of the motion of the virtual player (VP) can be manipulated in order to understand whether and how the interaction with the human subject is affected.

Here we present a mathematical framework for the study of joint improvisation between two participants, to whom no roles of leader and follower are assigned beforehand. In particular, we first propose a set of metrics to quantify some features of joint improvisation. Then, taking a top-down approach in which we make some hypotheses on the key factors governing joint improvisation as defined by such metrics, we propose a mathematical model able to reproduce in-silico the results observed experimentally from the interaction between two humans. We also use such mathematical description, based on optimal control theory, to control a model-driven virtual player and enable it not only to interact with a human subject, but also to generate jointly improvised movements with him/her.

We focus on the mirror game, a paradigm of joint human interaction which was recently proposed in [12, 13] (see [14] for more details on the link between mirror game and psychological constructs of attachment). Contrary to previous approaches where models were typically used to generate simple oscillatory motion or to reproduce the motion of a human subject [4], we present a model able to capture the complex movements generated by human subjects playing the game and generate new motion. Furthermore, we show that the model can capture and reproduce the essential kinematic features of the movement of a reference human subject as encoded by the Individual Motor Signature recently introduced in [15, 16], opening the possibility of testing in-silico the interaction between different individuals in a number of different configurations.

The ability of the model to reproduce the experimental results and its ability to drive a computer avatar in real-time are tested and validated via an extensive numerical investigation, accompanied by a mathematical analysis of its convergence.

Materials and Methods

Mirror game

As a simple yet effective paradigm to study interpersonal coordination between two individuals we use the mirror game [12]. Specifically, as shown in Fig 1, two players facing each other are asked to coordinate the motion of two balls mounted on two respective parallel strings. The players can be asked to play in a leader-follower condition (LF), where one is instructed to follow the motion of the other, or in a joint improvisation condition (JI), where they are instructed to imitate each other, create synchronized and interesting motions and enjoy playing together, without any designation of leader and follower. Only the latter experimental condition will be considered in this paper.

Download:

Fig 1. Mirror game between two human players at University of Montpellier, France.

Two participants face each other and are asked to perform synchronized motion by moving two balls along a string to which they are respectively attached.

https://doi.org/10.1371/journal.pone.0154361.g001

All the study discussed in this work was carried out according to the principles expressed in the Declaration of Helsinki and approved by the local ethical committee (University of Montpellier, France). The participants provided verbal informed consent to participate in the study, and such consent was approved by the ethics committee. Written consent was not necessary given the small pre-test nature of the experiments and full anonymization of the data.

Individual motor signature

As shown in [15, 16], the motion of each individual in the mirror game is characterized by different kinematic features that can be accounted for by examining the velocity profile of the player’s motion during the game. The velocity profile comprises a velocity frequency distribution, termed in [16] as Individual Motor Signature (IMS), and can be used to classify and distinguish the movement generated by different human subjects. To acquire the IMS of a human subject, we asked him/her to play the mirror game in a Solo condition, i.e. in the absence of the other participant. In this condition, the human subject was asked to generate interesting complex motion for 60 seconds. The position time series were recorded during the experiment, and were next used to estimate the velocity PDF of the player’s motion.

Experimental set-ups

We use two experimental set-ups to carry out experiments.

Set-up 1 (shown in Fig 1) consists of two parallel strings (180 cm long), with a ball that can slide on each. Two human players are asked to move their own ball back and forth along the strings, respectively, while seated. The position of the balls are detected by cameras disposed around the two participants. Details of the set-up can be found in [15, 16].
Set-up 2 (schematically shown in Fig 2) employs a cheap leap motion controller, whose spatial accuracy is below 0.2mm [17], in order to detect the fingertip position of the human player (HP).
Both the leap motion controller and a laptop computer (employed to implement and run the theoretical model driving the computer avatar) are placed on a table. The HP is asked to wave his/her own index finger horizontally over the leap motion controller with a horizontal range of around 60cm. The fingertip position of the HP is mapped into the interval [−0.5, 0.5] and visualized as a blue solid circle on the computer screen. A green solid circle, whose movement is computed from the model presented in this paper, is also visualized to represent the position of the virtual player. The advantage of using this simple set-up, consisting of cheap off-the-shelf elements, is its accessibility and ease of implementation.

Download:

Fig 2. Experimental setup of the mirror game between a HP and a VP.

The position of the human fingertip r_p(t) is detected by a leap motion controller, and the sampled position r_p(kT) is sent to the computer, while the position x(t) of the VP is generated by implementing the numerical algorithm of the single model. Two circles are shown on the computer screen, which correspond to the end effectors’ position of the HP (blue circle) and the VP (green circle), respectively.

https://doi.org/10.1371/journal.pone.0154361.g002

It is worth pointing out that, due to the longer range of movement in experimental set-up 1, the generated position trace is affected mostly by the motion of arm, forearm and palm, while in experimental set-up 2 the movement generation involves mostly forearm and palm. As a consequence, the motion captured by means of experimental set-up 2 exhibits generally a simpler trajectory as the end-effector system has less degrees of freedom. Nevertheless, as shown in [16], both set-ups can be used to capture individual characteristics of motion of different participants, and hence be treated as equivalent.

Data and evaluation metrics

In order to assess the level of coordination between the players and define some of the typical features of joint improvisation in the mirror game, we use the following metrics.

Position temporal correspondence. The root mean square (RMS) of the normalized position error between the two players defined as is used to describe their movement synchronization during the game. Here L refers to the range of admissible positions (e.g. the length of the strings in the set-up shown in Fig 1 or the range of motion detected by the leap motion controller), n is the number of sampling steps in the simulation, and x_1,k and x_2,k denote the positions of two participants at the k-th sampling step, respectively.
Distance between IMSs. The earth mover’s distance (EMD) is used to quantify differences between the velocity PDFs of the two players’ motion (e.g. to assess how similar/dissimilar their IMSs are). It is a proper metric in the space of PDFs [16, 18] and is computed as follows: where Z denotes the integration domain, and CDF_{p_i}(z) denotes the cumulative distribution function of the distribution p_i, i ∈ {1, 2}. Furthermore, we normalize the EMDs with the maximal η_max given by the length of the integration domain η_max = |Z|.
Relative phase distribution. The PDF of the relative phase ϕ₁₂ between the players motion (estimated by means of wavelet coherence [19]) is used to check the directionality of the interaction during the game (i.e. if one player is leading or following the other).

Human benchmark

To establish a benchmark dataset to compare with the model simulations, we obtained data from 8 different human dyads. Data from each dyad contains 3 solo trials for each human participant and 3 joint trials between them in JI condition. Description of all the available data (Matlab structure in S1 Dataset) can be found in Section B of S1 File. A representative example of the data collected in the mirror game between two HPs (Dyad 1, HPs JI trial 3) is shown in Fig 3. Here below we list the main observations that guided us towards a definition of joint improvisation.

Fig 3A shows the trajectories of the first solo trial of HP₁ (light red) and the third solo trial of HP₂ (dark red), while Fig 3D shows the trajectories of the two HPs interacting in JI condition (light and dark blue). Additionally, we show the velocity PDFs estimated from each of the respective position time series: panels B-C from Solo and panels E-F from JI between the two HPs. The low value of RMS position error observed experimentally (e_p ≃ 0.08) shows that the two players managed to reach a good level of movement synchronization when interacting together.
In order to visualize the relations between the distributions, we used multidimensional scaling (MDS), a data mining and visualization technique [16, 20] that uses distances/ dissimilarities between objects to represent them as points in a geometric space. An example of such a geometric representation of the relations between the PDFs of the players’ velocities is shown in Fig 4A. Different velocity PDFs are denoted with different markers: σ_i (red dots) indicates the motor signature of the i-th human player when playing solo; μ_i (blue dots) indicates the velocity PDFs of the i-th human player during runs of the mirror game with the other player.
In agreement with previous studies [13], we found that the kinematic characteristics of motion change with respect to solo conditions when the participants are improvising together. Namely, velocity PDFs of the two HPs (Fig 4A, light and dark blue dots) move away from their respective motor signatures (Fig 4A, light and dark red dots), because of mutual imitation, adaptation and synchronization, which results in their velocity PDFs moving towards each other during the game. In other words, the players exhibit behavioral plasticity as defined in [16]. The values of the distances between the velocity PDFs depicted in Fig 4A were computed to be η(σ₁, μ₁) = 0.102, η(σ₂, μ₂) = 0.052 and η(μ₁, μ₂) = 0.030.
Finally, in Fig 4B, we plot the distribution of the relative phase between the two players. We found that it is quite broad and centered around 0, indicating that neither of the two players was clearly leading the interaction during the game.
Similar results were obtained for all the trials of every dyad (Fig 5). Each panel of Fig 5 corresponds to a single dyad. For each dyad, three PDFs of relative phase from the three respective JI trials are shown with different scales of blue. It is possible to appreciate how all the PDFs are quite broad and centered around 0. The only exception is Dyad 7 where the maximum of the relative phase PDF is shifted on the right for all the three trials indicating that, despite the instruction given to the two participants, one player consistently ended up leading the game (Fig 5G).

Download:

Fig 3. Example of position time series and velocity PDFs from experimental results for Dyad 1.

A: position time series for the solo trials of the HPs. B-C: PDFs of velocity corresponding to the two players. D: position time-series of the HPs from the JI trial. E-F: PDFs of velocity of the two HPs in JI condition.

https://doi.org/10.1371/journal.pone.0154361.g003

Download:

Fig 4. Example of data analysis for Dyad 1.

A: visualization of the relations between kinematics of two human players obtained by means of MDS. Red dots σ_1,2 indicate the signatures of the two respective HPs. Blue dots labelled as μ_1,2 indicate the velocity profile of the motion of the HPs in the JI trial. B: PDF of relative phase ϕ₁₂ between the two HPs.

https://doi.org/10.1371/journal.pone.0154361.g004

Download:

Fig 5. PDFs of relative phase between HPs in all the dyads.

Each panel corresponds to a single dyad. For each dyad, three HP₁-HP₂ trials are shown with different scales of blue. The relative phase ϕ₁₂ between players is estimated by means of wavelet coherence (with 1Hz cut-off frequency). PDFs are estimated from histograms with a kernel density estimation.

https://doi.org/10.1371/journal.pone.0154361.g005

Results

Theoretical model of joint improvisation

Our first result is a mathematical model of Joint Improvisation. Our experimental observations of two HPs playing the mirror game suggest that their interaction in a JI condition is driven by three key factors: (i) their will to synchronize each other’s movement; (ii) the tendency of each player to exhibit some individual preferred movement features (or IMS); and (iii) the attempt each player makes to imitate the way the other moves (or mutual imitation). As shown in Fig 6, we proposed to map each of these three factors onto a specific behavioral goal. In particular, synchronization of joint movements can be translated into the goal of minimizing the position mismatch between the balls moved by the two participants, which is related to the temporal correspondence (TC) between their positions. Spontaneous motion preferences arise from the tendency of each participant to move according to his/her own IMS. Finally, mutual imitation can be achieved by the participants minimizing their velocity mismatch (velocity TC) during the mirror game. We captured each of these properties into a new mathematical model of interaction during JI formulated as the following optimal control problem.

Download:

Fig 6. Theoretical model of joint improvisation.

Three key factors governing JI (from left to right): 1. movement synchronization, corresponding to temporal correspondence (TC) between positions of the players’ end effectors; 2. preferred movement, captured by their respective IMS; 3. mutual imitation, modeled by temporal correspondence between their velocities.

https://doi.org/10.1371/journal.pone.0154361.g006

Specifically, following the approach of [21–24], we modeled the motion of each of the players using a nonlinear Haken-Kelso-Bunz (HKB) oscillator [25] of the form (1) where x_i and denote position and velocity of player i, u_i is the coupling function through which player i modulates its motion according to that of the other player, while α_i, β_i, γ_i and ω_i are intrinsic parameters determining the intrinsic properties of the player’s motion, such as speed of reaction and settling time. We represented the coupling function u_i as a nonliner control input that each player computes by minimizing the following cost function over each sampling period T = t_k+1 − t_k the whole trial duration is being split into. Namely, the problem is that of finding the inputs u_i such that (2) where (3) with θ_p,i, θ_σ,i, θ_v,i, η_i > 0 being tunable control parameters satisfying the constraint θ_p,i + θ_σ,i + θ_v,i = 1. Here, σ_i encodes the IMS of player i as his/her velocity time series during solo trials.

The cost function described in Eq (3), which is a more general form of that proposed in [22–24] as it includes the leader-follower model, contains four terms. The first three terms correspond to each of the three factors characterizing JI shown in Fig 6, while the fourth aims at minimizing the control effort over each sampling period. Indeed, the three tunable weights θ_p,i, θ_σ,i and θ_v,i allow for movement synchronization (position TC), preferred movement (motor signature) and mutual imitation (velocity TC), respectively, while η_i allows to regulate the control energy, i.e. the amplitude of the control signal u_i. Different set values for θ_p,i, θ_σ,i and θ_v,i can be used to change the balance between the terms described above. In the most trivial cases:

setting θ_p = 1, θ_σ = 0 and θ_v = 0 would make the VP behave as a perfect follower, as it would simply reproduce the movement of the other player;
setting θ_p = 0, θ_σ = 1 and θ_v = 0 would make the VP behave as a blind leader, as it would simply replay a pre-recorded trajectory without taking into account the behavior of the other player;
setting θ_p = 0, θ_σ = 0 and θ_v = 1 would make the VP copy the velocity of the other player without considering the position mismatch (a finite sampling rate might lead to a large shift between the positions of the two players).

In other words, the optimal control framework allows to incorporate into the same cost function all the key factors identified to govern joint improvisation. In what follows we will refer to each of the players modelled by Eqs from (1) to (3) as a virtual player. We will denote VPs as VP_i where i denotes that the model receives as an input the pre-recorded velocity profile σ_i of the i-th HP playing solo.

Model testing and validation

To test the effectiveness of the model, we compared numerical results with the human benchmark described in Section Human benchmark. A total of 9 different interactions in each of the 8 dyads were considered, since there were 3 solo trials available for each participant (corresponding to the reference velocity profiles σ_i used in the model). Indeed, if we refer to the IMS of the i-th HP recorded in the j-th Solo trial of each dyad as σ_i,j, the 9 different interactions were obtained by feeding VP₁ with σ_1,h and VP₂ with σ_2,k, where h,k = 1,2,3 (Fig 7). Description of all the available data (Matlab structure in S2 Dataset) and more information on the composition of all dyads can be found in Section B of S1 File.

Download:

Fig 7. Schematic diagram of VP-VP interaction.

VP₁ is fed with the motor signature of HP₁, while VP₂ with that of HP₂. We refer to the IMS of the i-th HP recorded in the j-th Solo trial of each dyad as σ_i,j. In this case h,k = 1,2,3 give rise to 9 different combinations for each dyad. The JI session played by the virtual players resembles the one performed by the two HPs, whose respective motor signatures are fed to the VPs.

https://doi.org/10.1371/journal.pone.0154361.g007

In so doing, for each dyad, each model equation was used to describe the kinematic behavior of a corresponding HP. The parameters of the model were set heuristically to the following values: α₁ = α₂ = 1, β₁ = β₂ = 1, γ₁ = γ₂ = 1, ω₁ = ω₂ = 1, T = t_k+1−t_k = 0.016s. The weights θ_p,i, θ_σ,i and θ_v,i were also set heuristically by trial-and-error in order to best match the experimental results (see Table A in Section A of S1 File for further details on the values of the weights and how to interpret them).

Single JI trial comparison.

For the sake of clarity, we begin by showing a quantitative comparison of experimental data from a single HP-HP dyad with the corresponding data obtained by simulation of the model equations. In particular, we considered the third JI trial of Dyad 1. We denote by ν_i the velocity PDF of VP_i and by μ_i the velocity PDF of its corresponding HP_i from a simulated and experimental JI trial, respectively.

The values of the EMDs between velocity PDFs are given in Table 1. We found that the distance η(σ_i, ν_i) between the velocity PDFs of each virtual player (evaluated from the JI trial) and its reference motor signature σ_i matches closely the distance η(σ_i, μ_i) between the corresponding HPs and their own signatures. Moreover, the distance η(ν_i, ν_j) between two VPs interacting with each other is quite close to that observed when the two HPs they model interact together in the mirror game, η(μ_i, μ_j). This shows how, just like in the case of two humans (previously analyzed in Section Human benchmark), the velocity PDFs of the two VPs move away from their respective signatures and get close to each other (while remaining close to the velocity PDFs of their human counterparts in the JI trial).

Download:

Table 1. Evaluation of the model via EMDs between velocity PDFs for a single trial.

https://doi.org/10.1371/journal.pone.0154361.t001

Computation of the relative phases PDFs in HP-HP interactions and VP-VP interactions confirmed that they are close to each other, thus leading to the conclusion that, just like in the human scenario, neither of the two VPs turned out to be a leader during the JI trial. Indeed, if we denote with ϕ_VP and ϕ_HP the PDFs of the relative phase between the two VPs and the two HPs they model, respectively, the EMD between them was computed to be η(ϕ_VP, ϕ_HP) = 0.024.

The previous findings show how the proposed model succeeds in capturing the main characteristics of the interaction between two human players improvising together, thus demonstrating its ability to reproduce in-silico the mirror game between two human subjects in a JI condition.

Matching results for the 8 dyads.

Next, we present results for all the 8 experimental data-sets and show how our model is able to capture the experimental observations in terms of: 1) RMS position error, 2) changes in EMD between velocity PDFs and 3) EMD between relative phase PDFs.

Fig 8 shows good agreement between RMS position errors observed in the experiments (blue crosses, three trials) and those obtained from corresponding VP dyads (green boxes, nine trials). In particular, it is worth pointing out that good levels of movement synchronization are achieved for both HP-HP and VP-VP dyads (e_p is always below 14%), and how the higher level of synchronization between HPs in Dyad 5 (lower value of e_p) was also captured in the VP-VP simulations.
Fig 9 confirms the model ability to capture the behavioral plasticity of the IMS of the players during a typical game. Data from JI trials between HPs is shown in blue, while data corresponding to the VPs simulations is shown in green. We found that the EMD between velocity PDFs of two HPs was similar to that of the two corresponding VPs for all the dyads (Fig 9A). Moreover, such similarity was found also in the relations between each player’s motion and their signatures (Fig 9B and 9C). It is worth pointing out how the higher value of EMD between HP₁ and his/her corresponding signature for Dyad 7 and Dyad 8 was well captured by the model simulations (Fig 9B).
Fig 10 shows a quantitative comparison of the PDFs of the relative phase computed from trials between HPs and VPs dyads. It is possible to appreciate how the EMD between relative phases in VP-VP trials, ϕ_VP, and HP-HP trials, ϕ_HP, is low for all the eight dyads. This means that the absence of an emerging leader observed in a JI session between humans was also replicated in simulations between corresponding virtual players.

Download:

Fig 8. Matching in terms of position temporal correspondence between HPs and between VPs.

Blue crosses (x) show the RMS position error from three JI trials of HPs, whilst box-plots depict the distributions of RMS position error from nine simulations of JI interaction between VPs (corresponding to nine combinations of the HPs’ individual signatures). In particular: thick horizontal green lines indicate median of the distribution; central light green boxes show central 50% of the data with lower and upper boundary lines being at the 25% and 75% quantiles of the data; two vertical whiskers estimate 99% of the range of the data; black crosses (+) show outliers.

https://doi.org/10.1371/journal.pone.0154361.g008

Download:

Fig 9. Matching in terms of relations between kinematics of HPs and VPs.

ρ_i denotes the velocity PDF of the i-th player from a JI trial, that is μ_i for HP_i and ν_i for VP_i. A: degree of similarity between PDFs of velocities recorded during JI trials. Blue crosses show 3 trials for HPs and the green box-plots depict distributions of EMDs between velocities from 9 JI trials between VPs. B and C show how far the movements of the players in JI condition were shifted away from their motor signatures (blue for HPs, green for VPs). Notches on the box-plots indicate confidence intervals of the medians. Two medians are significantly different at the p = 0.05 level if their notches do not overlap. Notches extending beyond the box indicate that the confidence intervals extend beyond central 50% of the data points.

https://doi.org/10.1371/journal.pone.0154361.g009

Download:

Fig 10. Matching in terms of relative phase between HPs and between VPs.

Box-plots illustrate distributions of EMDs between PDFs of relative phase from JI trials of HPs and VPs. Each box-plot corresponds to a single dyad and is constructed from 27 EMDs between three HP-HP relative phase PDFs and nine for the coupled VPs; black crosses (+) show outliers.

https://doi.org/10.1371/journal.pone.0154361.g010

A model-driven avatar

We used our theoretical model to drive a computer avatar able to play the mirror game with a human player in a JI condition, such that the features previously analyzed for human dyads could be reproduced in HP-VP interactions as well. We employed the low-cost experimental set-up 2 described in Section Experimental set-ups where the human player moves one of the two solid circles on the screen via a leap motion controller while the other is moved by the computer avatar. Description of all the available data (Matlab structure in S3 Dataset) can be found in Section B of S1 File.

The avatar computes the position of the circle it is moving by solving just one of Eq (1), say for i = 1, with u₁ obtained by solving the optimal control problem described in Eq (3). Now, x₂ and indicate position and velocity of the HP (whose signature is denoted with σ_HP) interacting with the VP. In particular, velocity and position of such human player are estimated over each interval according to and Moreover, σ₁ in Eq (3) indicates the IMS of a different HP (which is fed to the VP) and is denoted with σ_VP.

All the other model parameters were selected heuristically as follows: α₁ = 1, β₁ = 1, γ₁ = 1, ω₁ = 1, η₁ = 10⁻⁴, T = t_k+1−t_k = 0.04s, θ_p,1 = 0.2, θ_σ,1 = 0.4 and θ_v,1 = 0.4. The initial position and velocity of the avatar were set to 0.

The position time series recorded in the experiment for both players are shown in Fig 11A. We remind the reader that the different visual appearance of the position traces in Figs 3D and 11A results from the differences in the experimental set-ups. However, we would like to emphasize that both illustrated interactions satisfy our description of JI. More specifically, we found that the main features of JI observed in the case of HP-HP interaction were replicated when replacing one of the two HPs with a VP. Indeed:

the value of the RMS of the normalized position error between HP and VP (e_p ≃ 0.16) is comparable with that obtained in HP-HP interactions (Fig 8), showing that in both cases a good level of movement synchronization is achieved;
both HP and VP move away from their own signatures and converge towards each other (Fig 11B). In particular, η(ν, σ_VP) = 0.048, η(μ, σ_HP) = 0.074 and η(μ, ν) = 0.042, with μ and ν being the velocity PDFs obtained from the JI interaction between HP and VP, respectively;
the wide PDF of the relative phase between HP and VP indicates that there is no effective leader during the interaction (Fig 11C).

Download:

Fig 11. Interaction between a VP and a HP.

A: positions of HP (blue) and VP (green). B: relations between kinematics of motion of the players visualized by means of MDS. C: PDF of the relative phase between the two time-series of A.

https://doi.org/10.1371/journal.pone.0154361.g011

Both results confirm that a computer avatar driven by our theoretical model is able to jointly improvise its motion in real-time with a human subject providing a new powerful tool for discovery and investigation of social interaction and movement coordination in the mirror game.

Convergence analysis

Finally, we confirmed via a theoretical analysis that the model we propose guarantees convergence between the players when either two coupled VPs are considered or when the model-driven avatar interacts with a human subject. Our main stability results can be listed as follows (see S1 Appendix for a proof of the findings and further details):

the solution to the minimization problem described from Eqs (1) to (3) ensures bounded position error between two VPs when in-silico experiments are considered;
the solution to the minimization problem described from Eqs (1) to (3) ensures bounded position error between HP and VP when the model-driven avatar interacts with a human subject;
if the nonlinear HKB dynamics of the VP end-effector described in Eq (1) is substituted with a simpler linear model, achievement of the optimal solution to the minimization problem described in Eqs (2) and (3) is guaranteed over each subinterval.

Discussion

In this work we presented a new mathematical model, based on the use of a nonlinear oscillator and an optimal control theoretic framework, to explain and reproduce joint improvisation in human dyads as defined by the introduction of appropriate metrics. Using both model simulations and experiments, we demonstrated the applicability of our modelling approach to capture the features of joint improvisation between two human players in the mirror game, and its capability to drive a computer avatar to produce jointly improvised movements with a human player, respectively. Indeed, both VP-VP and HP-VP interactions exhibited the main characteristics of JI as defined within the context of HP-HP interactions in the mirror game. Specifically: 1) high levels of movement synchronization, by means of a low value for the RMS of the normalized position error between the agents; 2) behavioral plasticity of the players, measured by means of changes in EMD between velocity profiles; 3) absence of a clear leader, by means of a relative phase distribution of the players centered around 0.

The availability of such an enhanced model-driven avatar provides a new fundamental tool to explore the important tenet in Social Psychology that behavioral similarity between people facilitates their interaction [26, 27]. In particular, as recently proposed in [16], the similarity or dissimilarity between the IMSs of two individuals playing the mirror game can be an important factor affecting the level of their mutual interaction and coordination. Our proposed model can be used to test this hypothesis both in-silico and via real-time experiments.

Future work will include finding a strategy that would help to appropriately choose the weights in the cost function. One approach could be to implement adaptive laws to make the weights vary over time during the game session, according to the performance evaluated in real time. Another possible extension of our work includes the possibility of carrying out Turing-test experiments, where human participants are asked to perform joint improvisation with another agent and then guess whether it was a human or a virtual player. Future work will also include looking into the joint improvisation among multiple human participants [28].

Finally we wish to highlight that the work presented in this paper opens the exciting possibility of performing in-silico experiments to assess how two human players (whose IMS have been recorded during solo trials) would interact when playing the mirror game in a JI condition. This can be useful for rehabilitation purposes such as those being explored as part of the EU project AlterEgo [29].

Supporting Information

S1 File. Parameter setting of the coupling weights for all the dyads and description of the available data.

We show and interpret the values of θ_p,i, θ_σ,i and θ_v,i for al the dyads of VPs used to test matching performance of the proposed mathematical model with the human benchmark. Moreover, we give a detailed description of all the available data and present tables showing the composition of the dyads mentioned in our analysis.

https://doi.org/10.1371/journal.pone.0154361.s001

(PDF)

S1 Appendix. Theoretical analysis of the proposed mathematical model of joint improvisation.

We show that the solution to the minimization problem guarantees bounded position error between the two players involved in the game.

https://doi.org/10.1371/journal.pone.0154361.s002

(PDF)

S1 Dataset. S1_Dataset.mat.

Matlab structure containing data from the interaction between two human players (HP-HP) recorded using experimental set-up 1.

https://doi.org/10.1371/journal.pone.0154361.s003

(MAT)

S2 Dataset. S2_Dataset.mat.

Matlab structure containing data from simulated interactions between two virtual players (VP-VP).

https://doi.org/10.1371/journal.pone.0154361.s004

(MAT)

S3 Dataset. S3_Dataset.mat.

Matlab structure containing data from the interaction between human and virtual players (HP-VP) recorded using experimental set-up 2.

https://doi.org/10.1371/journal.pone.0154361.s005

(MAT)

Acknowledgments

This work was funded by the European Project AlterEgo FP7 ICT 2.9—Cognitive Sciences and Robotics, Grant Number 600610. The authors wish to thank Prof. Benoit Bardy, Dr. Ludovic Marin and Dr. Robin Salesse at EUROMOV, University of Montpellier, France for all the insightful discussions and Ed Rooke at University of Bristol for collecting some of the experimental data that was used to validate the approach presented in this paper.

Author Contributions

Conceived and designed the experiments: CZ FA PS KTA MdB. Performed the experiments: CZ FA. Analyzed the data: CZ FA PS. Contributed reagents/materials/analysis tools: CZ FA PS. Wrote the paper: CZ FA PS KTA MdB.

References

1. Kelso JS. Dynamic patterns: The self-organization of brain and behavior. MIT press; 1997.
2. Schmidt RC, Richardson MJ. Dynamics of interpersonal coordination. In: Coordination: Neural, behavioral and social dynamics. Springer; 2008. p. 281–308.
3. Schmidt R, Nie L, Franco A, Richardson MJ. Bodily synchronization underlying joke telling. Frontiers in human neuroscience. 2014;8(633):10–3389.
- View Article
- Google Scholar
4. Dumas G, de Guzman GC, Tognoli E, Kelso JS. The human dynamic clamp as a paradigm for social interaction. Proceedings of the National Academy of Sciences. 2014;111(35):E3726–E3734.
- View Article
- Google Scholar
5. Johnstone K. Impro: Improvisation and the theatre. Routledge; 2012.
6. Sawyer RK. Creating conversations: Improvisation in everyday discourse. Hampton Press (NJ); 2001.
7. Noy L, Levit-Binun N, Golland Y. Being in the zone: physiological markers of togetherness in joint improvisation. Frontiers in human neuroscience. 2015;9. pmid:25999832
- View Article
- PubMed/NCBI
- Google Scholar
8. Lakens D, Stel M. If they move in sync, they must feel in sync: Movement synchrony leads to attributions of rapport and entitativity. Social Cognition. 2011;29(1):1.
- View Article
- Google Scholar
9. Wiltermuth SS, Heath C. Synchrony and cooperation. Psychological science. 2009;20(1):1–5. pmid:19152536
- View Article
- PubMed/NCBI
- Google Scholar
10. Kelso JS, de Guzman GC, Reveley C, Tognoli E. Virtual partner interaction (VPI): exploring novel behaviors via coordination dynamics. PloS one. 2009;4(6):e5749. pmid:19492044
- View Article
- PubMed/NCBI
- Google Scholar
11. Kostrubiec V, Dumas G, Zanone PG, Kelso JS. The virtual teacher (VT) paradigm: learning new patterns of interpersonal coordination using the human dynamic clamp. PloS one. 2015;10(11):e0142029. pmid:26569608
- View Article
- PubMed/NCBI
- Google Scholar
12. Noy L, Dekel E, Alon U. The mirror game as a paradigm for studying the dynamics of two people improvising motion together. Proceedings of the National Academy of Sciences. 2011;108(52):20947–20952.
- View Article
- Google Scholar
13. Hart Y, Noy L, Feniger-Schaal R, Mayo AE, Alon U. Individuality and togetherness in joint improvised motion. PloS one. 2014;9(2):e87213. pmid:24533054
- View Article
- PubMed/NCBI
- Google Scholar
14. Feniger-Schaal R, Noy L, Hart Y, Koren-Karie N, Mayo AE, Alon U. Would you like to play together? Adults’ attachment and the mirror game. Attachment & human development. 2016;18(1):33–45.
- View Article
- Google Scholar
15. Słowiński P, Rooke E, di Bernardo M, Tanaseva-Atanasova K. Kinematic characteristics of motion in the mirror game. In: 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). IEEE; 2014. p. 748–753.
16. Słowiński P, Zhai C, Alderisio F, Salesse R, Gueugnon M, Marin L, et al. Dynamic similarity promotes interpersonal coordination in joint action. Journal of The Royal Society Interface. 2016;13(116):20151093.
- View Article
- Google Scholar
17. Guna J, Jakus G, Pogačnik M, Tomažič S, Sodnik J. An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors. 2014;14(2):3702–3720. pmid:24566635
- View Article
- PubMed/NCBI
- Google Scholar
18. Levina E, Bickel P. The earth mover’s distance is the Mallows distance: some insights from statistics. In: 2001 IEEE International Conference on Computer Vision (ICCV). vol. 2. IEEE; 2001. p. 251–256.
19. Grinsted A, Moore JC, Jevrejeva S. Application of the cross wavelet transform and wavelet coherence to geophysical time series. Nonlinear processes in geophysics. 2004;11(5/6):561–566.
- View Article
- Google Scholar
20. Borg I, Groenen PJ. Modern multidimensional scaling: Theory and applications. Springer Science & Business Media; 2005.
21. Zhai C, Alderisio F, Tsaneva-Atanasova K, di Bernardo M. A novel cognitive architecture for a human-like virtual player in the mirror game. In: 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). IEEE; 2014. p. 754–759.
22. Zhai C, Alderisio F, Tsaneva-Atanasova K, di Bernardo M. Adaptive tracking control of a virtual player in the mirror game. In: 2014 IEEE 53rd Annual Conference on Decision and Control (CDC). IEEE; 2014. p. 7005–7010.
23. Zhai C, Alderisio F, Tsaneva-Atanasova K, di Bernardo M. A model predictive approach to control the motion of a virtual player in the mirror game. In: 2015 IEEE 54th Annual Conference on Decision and Control (CDC); 2015. p. 3175–3180.
24. Alderisio F, Antonacci D, Zhai C, di Bernardo M. Comparing different control approaches to implement a human-like virtual player in the mirror game. In: 15th European Control Conference (ECC); 2016. In press.
25. Haken H, Kelso JS, Bunz H. A theoretical model of phase transitions in human hand movements. Biological cybernetics. 1985;51(5):347–356. pmid:3978150
- View Article
- PubMed/NCBI
- Google Scholar
26. Folkes VS. Forming relationships and the matching hypothesis. Personality and Social Psychology Bulletin. 1982;8(4):631–636.
- View Article
- Google Scholar
27. Marsh KL, Richardson MJ, Schmidt RC. Social connection through joint action and interpersonal coordination. Topics in Cognitive Science. 2009;1(2):320–339. pmid:25164936
- View Article
- PubMed/NCBI
- Google Scholar
28. Alderisio F, Bardy BG, di Bernardo M. Entrainment and synchronization in networks of Rayleigh—van der Pol oscillators with diffusive and Haken—Kelso—Bunz couplings. Biological cybernetics. 2016.
- View Article
- Google Scholar
29. AletrEgo Project. Enhancing socio-motor interactions using information technology. Available: http://www.euromov.eu/alterego/.

[ref1] 1. Kelso JS. Dynamic patterns: The self-organization of brain and behavior. MIT press; 1997.

[ref2] 2. Schmidt RC, Richardson MJ. Dynamics of interpersonal coordination. In: Coordination: Neural, behavioral and social dynamics. Springer; 2008. p. 281–308.

[ref3] 3. Schmidt R, Nie L, Franco A, Richardson MJ. Bodily synchronization underlying joke telling. Frontiers in human neuroscience. 2014;8(633):10–3389.
View Article
Google Scholar

[4] View Article

[5] Google Scholar

[ref4] 4. Dumas G, de Guzman GC, Tognoli E, Kelso JS. The human dynamic clamp as a paradigm for social interaction. Proceedings of the National Academy of Sciences. 2014;111(35):E3726–E3734.
View Article
Google Scholar

[7] View Article

[8] Google Scholar

[ref5] 5. Johnstone K. Impro: Improvisation and the theatre. Routledge; 2012.

[ref6] 6. Sawyer RK. Creating conversations: Improvisation in everyday discourse. Hampton Press (NJ); 2001.

[ref7] 7. Noy L, Levit-Binun N, Golland Y. Being in the zone: physiological markers of togetherness in joint improvisation. Frontiers in human neuroscience. 2015;9. pmid:25999832
View Article
PubMed/NCBI
Google Scholar

[12] View Article

[13] PubMed/NCBI

[14] Google Scholar

[ref8] 8. Lakens D, Stel M. If they move in sync, they must feel in sync: Movement synchrony leads to attributions of rapport and entitativity. Social Cognition. 2011;29(1):1.
View Article
Google Scholar

[16] View Article

[17] Google Scholar

[ref9] 9. Wiltermuth SS, Heath C. Synchrony and cooperation. Psychological science. 2009;20(1):1–5. pmid:19152536
View Article
PubMed/NCBI
Google Scholar

[19] View Article

[20] PubMed/NCBI

[21] Google Scholar

[ref10] 10. Kelso JS, de Guzman GC, Reveley C, Tognoli E. Virtual partner interaction (VPI): exploring novel behaviors via coordination dynamics. PloS one. 2009;4(6):e5749. pmid:19492044
View Article
PubMed/NCBI
Google Scholar

[23] View Article

[24] PubMed/NCBI

[25] Google Scholar

[ref11] 11. Kostrubiec V, Dumas G, Zanone PG, Kelso JS. The virtual teacher (VT) paradigm: learning new patterns of interpersonal coordination using the human dynamic clamp. PloS one. 2015;10(11):e0142029. pmid:26569608
View Article
PubMed/NCBI
Google Scholar

[27] View Article

[28] PubMed/NCBI

[29] Google Scholar

[ref12] 12. Noy L, Dekel E, Alon U. The mirror game as a paradigm for studying the dynamics of two people improvising motion together. Proceedings of the National Academy of Sciences. 2011;108(52):20947–20952.
View Article
Google Scholar

[31] View Article

[32] Google Scholar

[ref13] 13. Hart Y, Noy L, Feniger-Schaal R, Mayo AE, Alon U. Individuality and togetherness in joint improvised motion. PloS one. 2014;9(2):e87213. pmid:24533054
View Article
PubMed/NCBI
Google Scholar

[34] View Article

[35] PubMed/NCBI

[36] Google Scholar

[ref14] 14. Feniger-Schaal R, Noy L, Hart Y, Koren-Karie N, Mayo AE, Alon U. Would you like to play together? Adults’ attachment and the mirror game. Attachment & human development. 2016;18(1):33–45.
View Article
Google Scholar

[38] View Article

[39] Google Scholar

[ref15] 15. Słowiński P, Rooke E, di Bernardo M, Tanaseva-Atanasova K. Kinematic characteristics of motion in the mirror game. In: 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). IEEE; 2014. p. 748–753.

[ref16] 16. Słowiński P, Zhai C, Alderisio F, Salesse R, Gueugnon M, Marin L, et al. Dynamic similarity promotes interpersonal coordination in joint action. Journal of The Royal Society Interface. 2016;13(116):20151093.
View Article
Google Scholar

[42] View Article

[43] Google Scholar

[ref17] 17. Guna J, Jakus G, Pogačnik M, Tomažič S, Sodnik J. An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors. 2014;14(2):3702–3720. pmid:24566635
View Article
PubMed/NCBI
Google Scholar

[45] View Article

[46] PubMed/NCBI

[47] Google Scholar

[ref18] 18. Levina E, Bickel P. The earth mover’s distance is the Mallows distance: some insights from statistics. In: 2001 IEEE International Conference on Computer Vision (ICCV). vol. 2. IEEE; 2001. p. 251–256.

[ref19] 19. Grinsted A, Moore JC, Jevrejeva S. Application of the cross wavelet transform and wavelet coherence to geophysical time series. Nonlinear processes in geophysics. 2004;11(5/6):561–566.
View Article
Google Scholar

[50] View Article

[51] Google Scholar

[ref20] 20. Borg I, Groenen PJ. Modern multidimensional scaling: Theory and applications. Springer Science & Business Media; 2005.

[ref21] 21. Zhai C, Alderisio F, Tsaneva-Atanasova K, di Bernardo M. A novel cognitive architecture for a human-like virtual player in the mirror game. In: 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). IEEE; 2014. p. 754–759.

[ref22] 22. Zhai C, Alderisio F, Tsaneva-Atanasova K, di Bernardo M. Adaptive tracking control of a virtual player in the mirror game. In: 2014 IEEE 53rd Annual Conference on Decision and Control (CDC). IEEE; 2014. p. 7005–7010.

[ref23] 23. Zhai C, Alderisio F, Tsaneva-Atanasova K, di Bernardo M. A model predictive approach to control the motion of a virtual player in the mirror game. In: 2015 IEEE 54th Annual Conference on Decision and Control (CDC); 2015. p. 3175–3180.

[ref24] 24. Alderisio F, Antonacci D, Zhai C, di Bernardo M. Comparing different control approaches to implement a human-like virtual player in the mirror game. In: 15th European Control Conference (ECC); 2016. In press.

[ref25] 25. Haken H, Kelso JS, Bunz H. A theoretical model of phase transitions in human hand movements. Biological cybernetics. 1985;51(5):347–356. pmid:3978150
View Article
PubMed/NCBI
Google Scholar

[58] View Article

[59] PubMed/NCBI

[60] Google Scholar

[ref26] 26. Folkes VS. Forming relationships and the matching hypothesis. Personality and Social Psychology Bulletin. 1982;8(4):631–636.
View Article
Google Scholar

[62] View Article

[63] Google Scholar

[ref27] 27. Marsh KL, Richardson MJ, Schmidt RC. Social connection through joint action and interpersonal coordination. Topics in Cognitive Science. 2009;1(2):320–339. pmid:25164936
View Article
PubMed/NCBI
Google Scholar

[65] View Article

[66] PubMed/NCBI

[67] Google Scholar

[ref28] 28. Alderisio F, Bardy BG, di Bernardo M. Entrainment and synchronization in networks of Rayleigh—van der Pol oscillators with diffusive and Haken—Kelso—Bunz couplings. Biological cybernetics. 2016.
View Article
Google Scholar

[69] View Article

[70] Google Scholar

[ref29] 29. AletrEgo Project. Enhancing socio-motor interactions using information technology. Available: http://www.euromov.eu/alterego/.

Figures

Abstract

Introduction

Materials and Methods

Mirror game

Individual motor signature

Experimental set-ups

Data and evaluation metrics

Human benchmark

Results

Theoretical model of joint improvisation

Model testing and validation

Single JI trial comparison.

Matching results for the 8 dyads.

A model-driven avatar

Convergence analysis

Discussion

Supporting Information

S1 File. Parameter setting of the coupling weights for all the dyads and description of the available data.

S1 Appendix. Theoretical analysis of the proposed mathematical model of joint improvisation.

S1 Dataset. S1_Dataset.mat.

S2 Dataset. S2_Dataset.mat.

S3 Dataset. S3_Dataset.mat.

Acknowledgments

Author Contributions

References