Figure 1.
Experimental setup for the teleoperation scenario – a holistic feedback control of semi-autonomous robots.
In the teleoperation scenario an operator is wearing an exoskeleton and, with the support of a virtual scenario, is tele-manipulating a robotic arm. A: three kinds of virtual response cubes (different responses are required for different types of warnings); B: different kinds of stimuli: unimportant stimulus (STATE OK – no response required), warning (first target – response required), repeated and enhanced warning (second target – response required), third warning (response is critical, e.g., exoskeleton control is disabled); C: labyrinth that the robot has to be moved through; D: virtual hand.
Figure 2.
Adaptation of an operator monitoring system by BR.
The currently implemented message scheduling procedure which is controlled by the operator monitoring system (OMS) is shown. The OMS considers the cognitive state that is detected by BR and allows to infer the behavior of the human. The general procedure is described in the following: After a warning the operator's EEG is analyzed by BR. Detection of successes versus no success in the recognition of important information by BR allows to infer future behavior (response or no response) by eBR. As a consequence, the behavior of the OMS is adapted, i.e., the tolerated response time is extended or a second warning is presented right away by the OMS. In case the operator does not respond to the second warning, a third warning follows. Approximate time required for predictions made by BR and predefined response times are given in the arrows.
Figure 3.
Adaptation of the exoskeleton's control by BR.
It is shown how BR adapts the exoskeletons control. The exoskeleton is supporting the user while moving (free mode: FM). In case the user stops moving, the exoskeleton will lock in to support the arm at a chosen position (position control mode: PC). For release the user has to press against sensors that are integrated into the exoskeleton. To ease the release BR detects movement intention. The movement prediction score is then used to modulate the exoskeleton's control by eBR: the higher the prediction score (i.e., the more certain the classifier is) the stronger is the adaptation of the exoskeleton's control and the lower is the effort for the user to transfer the exoskeleton from PC to FM mode. Pressure against the sensors is always required for release, which is minimizing the risk of false lock out in case of possible false detection of movement intention by BR. Adapted from [52].
Figure 4.
Experimental setup Labyrinth Oddball.
In the Labyrinth Oddball setup subjects perform a dual-task, i.e., they play a virtualized labyrinth game and react to less frequent first and second target stimuli by pressing a buzzer. A second target is presented in case that the first target was missed. Brain activity recorded after the different stimuli was averaged over all subjects, sessions, and runs (total number of trials after artifact removal: target 1 (red ERP curve, right side): ; missed target 1 (blue ERP curve, right side):
; standards (black ERP curve):
).
Figure 5.
Transfer of classifier between classes is visualized. Classifier transfer is applied between the class standard and missed targets. Hence, for training the class standard was used, in test the class missed targets was used instead.
Figure 6.
Average EMG activity of subject that was averaged based on two different events is displayed. A: Averaged activity based on buzzer press event is shown. B: Average activity based on EMG onset is shown.
Figure 7.
The mean and median of response time for each subject across two sessions based on EMG and buzzer press events are displayed. A: Mean of response time. B: Median of response time.
Figure 8.
Averaged ERPs in the Labyrinth Oddball scenario.
Different averaged ERP patterns evoked by standards, targets, and missed targets are shown for two subjects. A: Subject : No significant difference in ERP amplitude between targets and missed targets but significant difference in ERP amplitude between standards and missed targets for the late window was found. B: Subject
: A higher P300 effect on targets compared to both standards and missed targets and no significant difference in ERP amplitude between standards and missed targets for the late window was found.
Figure 9.
Classification performance obtained in the Labyrinth Oddball scenario for different windows of EEG data.
The dependency between classification performance and window size as well as start point of window are displayed for the classification of missed targets versus recognized targets. The start position (y-axis) is given relative to stimulus onset. The inset on the right indicates the optimized performance using the window from to
ms. The different windows are compared using the AUC, while the optimized performance is given as BA.
Figure 10.
Classification performance for different time windows.
The mean classification performance is shown for each time window and each subject.
Figure 11.
Classification performance in the Labyrinth Oddball scenario.
For each subject for a window from to
ms the evaluated classification performance and statistics are shown. The red lines in the main diagram mark the median values of obtained classification performances for each subject. The inserted diagram shows that highest classification performance was obtained for subject
and
(mean classification performance and standard error of mean (SEM) are depicted).
Figure 12.
The experimental setting Armrest is illustrated and most relevant ERP activity evoked by brain processes involved in target recognition and failure in target recognition as well as motor preparation are shown. A: Experimental setup is displayed. B: Three types of virtual response cubes (B-1) and the virtual target ball (B-2) are shown. C: Averaged difference curve between electrodes C3 and C4 (number of trials for movement events: ) shows differences recorded over the primary motor cortex ipsi- and contralateral to the side of movement (movement onset marker at dashed line). D: Averaged ERP patterns at electrode Pz on different stimulus types (number of standards:
, number of targets:
, number of missed targets:
) are depicted.
Figure 13.
Classifier evaluation for sliding windows.
It is illustrated how evaluation was performed in the sliding window approach. Evaluation depends on the end time of a sliding window: (i) less than ms: true label “no movement preparation”, (ii) between
to
ms: true label “movement preparation”, (iii) in gray shaded area: left out for evaluation due to unknown true label or already started movement. The black line illustrates the average ERP difference curve for channels C3/C4 over all subjects.
Figure 14.
Classification performance in the Armrest scenario.
Results for the performance of the classifier trained in the dual BR scenario for the classification of missed target vs. target instances after classifier transfer are shown for all subjects individually. The red lines in the main diagram mark the median values of obtained classification performance for each subject. The inserted diagram illustrates mean classification performance values and standard error of mean (SEM). Highest classification performance is observed for subject .
Figure 15.
Method illustration and performance for different training windows.
The diagram illustrates the combination of training time of two windows using the previously found clusters (see methods description for details), classification performance and statistics. Classification performance of a -fold cross validation for four subjects quantified with mean BA and standard error is presented by the dots in the diagram. The x-axis shows different training settings: A, B, C – one training window per movement marker ending at different times with respect to the movement marker; A+A, B+B, C+C, A+B, B+C, C+A – two training windows per movement marker, combined within the same cluster or with other clusters. All – all
training windows were used to train a classifier.