Design of a Virtual Player for Joint Improvisation with Humans in the Mirror Game

Joint improvisation is often observed among humans performing joint action tasks. Exploring the underlying cognitive and neural mechanisms behind the emergence of joint improvisation is an open research challenge. This paper investigates jointly improvised movements between two participants in the mirror game, a paradigmatic joint task example. First, experiments involving movement coordination of different dyads of human players are performed in order to build a human benchmark. No designation of leader and follower is given beforehand. We find that joint improvisation is characterized by the lack of a leader and high levels of movement synchronization. Then, a theoretical model is proposed to capture some features of their interaction, and a set of experiments is carried out to test and validate the model ability to reproduce the experimental observations. Furthermore, the model is used to drive a computer avatar able to successfully improvise joint motion with a human participant in real time. Finally, a convergence analysis of the proposed model is carried out to confirm its ability to reproduce joint movements between the participants.


Model-driven avatar
Let us recall that, over each sampling period T = t k+1 − t k , the control input u is obtained by solving the optimal control problem: where with θ p , θ σ , θ v , η > 0 being tunable control parameters satisfying the constraint Here, x and ẋ refer to position and velocity of the VP whose dynamics is modelled by: Moreover, σ refers to the IMS of a given HP in solo trials, while rp and rv represent the estimated position and velocity of the HP the VP is interacting with, respectively.In particular: Theorem 1.The solution to the optimization problem described in Eq (1) ensures bounded position error between VP and HP.
Proof.Let J 0 denote the value of the cost function described in Eq (2) when u ≡ 0. In addition, let J * and x * correspond to the optimal value of the cost function and the optimal position of the VP, respectively.It is clear that J * ≤ J 0 since J * is the minimum value of the cost function.According to Theorem 5.1 in [1], there exists a limit cycle in the HKB oscillator, and thus x and ẋ are bounded in J 0 .Considering that rp , σ and rv are all bounded, we conclude that J 0 is bounded.It follows from the inequality that the position error between VP and HP is bounded as well.
Corollary 1.If the nonlinear HKB dynamics of the VP end-effector is substituted with a simpler linear dynamics of the form achievement of the optimal solution to the minimization problem described in Eq (1) is guaranteed over each subinterval.
Proof.According to the fundamental theorem of the calculus of variations, we need to examine the second variation of the given cost function in order to establish the optimum.From existing conclusions in [2], the second variation of the cost function described in Eq (2) in the Hamiltonian formalism is given by where δX = [δx δ ẋ] T and H is the Hamiltonian T .Rewriting the linear system in matrix form we obtain Ẋ = AX + Bu where Let X = X * + δX and u = u * + δu, where X * and u * denote the optimal state and optimal control, respectively.Since Ẋ * = AX * + Bu * , we get Thus, it follows from H Xu = H uX = [0 0] T , H uu = η > 0 and Moreover, δ 2 J = 0 is equivalent to δx(t k+1 ) = 0, δ ẋ(t) = 0 and δu(t) = 0 for all t ∈ [t k , t k+1 ], which yields δx(t) = δx(t k ) = 0 from Eq (4).This corresponds to the optimal solution X * and the optimal control u * .Therefore, we conclude that the optimal control ensures achievement of the minimum value of the cost function described in Eq (2) over each sampling period.

Two coupled VPs
Let us recall that the model of two interacting VPs we propose consists of two coupled HKB oscillators: where x 1 and x 2 refer to the positions of the two virtual players VP 1 and VP 2 , respectively.Analogously to the previous case, the control input for each HKB oscillator can be derived by making each VP solve the following optimal control problem min where and In order to perform theoretical analysis for the nonlinearly coupled system described in Eq (5), we formulate the Hamiltonian for each of the two previous optimization problems as follows where X i = [x i ẋi ] T and λ i = [λ i1 λ i2 ] T , i ∈ {1, 2}.Applying the minimum principle [2], we get the optimal open loop control inputs given by and the corresponding optimal state equations with initial condition X i (t k ) = [x i (t k ) ẋi (t k )] T .Also, the optimal costate equations can be written as or equivalently as or equivalently )) 0 and λ 2 (t k+1 ) = θ p,2 (x 2 (t k+1 ) − x 1 (t k+1 )) 0 Hence, the solution of the coupled model can be obtained by solving the above boundary value problem described by Eq (6), Eq (7) and Eq (8).Following the same proof strategy as in Theorem 1, it is guaranteed that the position error between two VPs is bounded, and the proof is thus omitted to avoid redundancies.In particular, the achievement of joint improvisation movement is available by properly tuning the parameters θ p,i , θ σ,i and θ v,i .