Fig 1.
Experimental paradigm and setup.
Experimental paradigm featuring a trajectory-following task with two interaction scenarios. Panel A and B: Experimental setup showing a subject sitting in front of the industrial robot UR10. The workspace is displayed on a computer screen located between the human and the robot. The robot end-effector determines the current location in the trajectory. The subject indicates commands to the robot via key-presses. Panel C: sequential Collaboration (sC) scenario, in which human and robot are each responsible for a certain area in the workspace (left), and panel D: intermittent Collaboration (iC) scenario in which both human and robot are responsible in the entire space, but each having control over two out of four directions (right).
Table 1.
Overview of human-robot takeover situations.
Variants of takeover (HR, RH) and non-takeover situations (HH, RR) focusing on the action before (tr − 1) and after (tr).
Fig 2.
ERP responses to human-robot takeover situations.
ERP time courses over channel Cz time-locked to the onset of robot movement before (tr − 1) and after (tr) an anticipated takeover situation average across all subject. Anticipated in this context refers to the user’s mental preparation to the taking over action produced by the specific task allocation in the experimental protocol. All plots depict the grand average aggregating events of both collaboration scenarios (sC, iC). Plots in color (blue, magenta) depict ERP responses in takeover situations (HR, RH) and plots in grey depict ERP responses in non-takeover situations (HH, RR). Grey-shaded areas show significant time-points. The difference grand average is furthermore depicted as topographic plots at relevant time points above each plot, whereby white markers show significant electrodes in the respective time-point. The grand average difference ERSPs masked with the minimum (in blue) and maximum (in yellow-red) significant bins, are depicted below each plot (in Panel A, no ERSP results are reported because no significant bins were found). Panel A and B show the results of contrasting responses to a robot takeover from the human (HR) versus the human continuation (HH), before and after the anticipated takeover situation. Panel C and D show the results of contrasting responses to a human takeover from the robot (RH) versus robot continuation (RR), before and after the alleged takeover situation. Subject specific ERP responses are detailed in S3 Appendix, Fig 1.
Fig 3.
ERP responses before an anticipated human-to-robot takeover comparing sequential and intermittent collaboration.
ERP time courses over channel Cz time-locked to the onset of robot movement comparing the anticipated human takeover from robot versus robot continuation across the two collaboration scenarios average across all subject. Panel A depicts the grand average responses during the sequential collaboration (sC), whereas panel B depicts the grand average responses during the intermittent collaboration (iC). The difference grand average is furthermore depicted as topographic plots at relevant time points above each plot. White-markings show significant electrodes in the respective time-point. In addition, the grand average difference ERSPs masked with the minimum (in blue) and maximum (in yellow-red) significant bins, are depicted below each plot. Subject specific ERP responses are detailed in S3 Appendix, Fig 1.
Table 2.
Results of single-trial decoding performance.
Average across subjects (n = 11) single-trial classification performance (M±SD [%]) for decoding anticipation of variants of takeover (HR, RH) against non-takeover situations (HH, RR) for sequential collaboration (sC) and intermittent collaboration (iC). Bolded numbers show results that are significantly above the sample-size adjusted chance-level pchance = 54.3%. Detailed per subject results of decoding performance are reported in S3 Appendix.
Fig 4.
Results of simulation-based HRI feasibility study.
Panel A: robot learning curve for decoding performance in line with the empirically determined (emp. ACC) results of the experimental study. Panel B: various robot learning curves for the empirical decoding performance (emp. ACC) and hypothetical variations of ERP decoding performance of anticipated takeover (whereby ACCxx=TNRHH=TPRRH=TNRRR=TPRRH). Panel C: learning curves in case of a dynamic task re-assignment after every 400 trials / 40 episodes. Panel D: variations of the original experiment with increased numbers of subtasks illustrating the scalability of the approach towards more complex tasks as well as robustness towards imbalanced subtask assignment (6 subtasks with balanced assignement, 10 subtasks with balanced assignment, and 10 subtasks with 2 human and 8 robot assigned subtasks). In the example chosen for the simulation, a single episode corresponds to 10 trials.