Conceived and designed the experiments: YW. Performed the experiments: YW. Analyzed the data: YW T-PJ. Contributed reagents/materials/analysis tools: YW T-PJ. Wrote the paper: YW T-PJ.
I have read the journal's policy and have the following conflicts: Funding was received from Abraxis Bioscience Inc., but is not accompanied by any other relevant declarations relating to employment, consultancy, patents, products in development or marketed products and does not alter the authors' adherence to all the PLoS ONE policies on sharing data and materials.
Electroencephalogram (EEG) based brain-computer interfaces (BCI) have been studied since the 1970s. Currently, the main focus of BCI research lies on the clinical use, which aims to provide a new communication channel to patients with motor disabilities to improve their quality of life. However, the BCI technology can also be used to improve human performance for normal healthy users. Although this application has been proposed for a long time, little progress has been made in real-world practices due to technical limits of EEG. To overcome the bottleneck of low single-user BCI performance, this study proposes a collaborative paradigm to improve overall BCI performance by integrating information from multiple users. To test the feasibility of a collaborative BCI, this study quantitatively compares the classification accuracies of collaborative and single-user BCI applied to the EEG data collected from 20 subjects in a movement-planning experiment. This study also explores three different methods for fusing and analyzing EEG data from multiple subjects: (1) Event-related potentials (ERP) averaging, (2) Feature concatenating, and (3) Voting. In a demonstration system using the Voting method, the classification accuracy of predicting movement directions (reaching
Electroencephalogram (EEG) based brain-computer interfaces (BCI) in human studies have been demonstrated as a new tool to people with severe motor disabilities to communicate with their environments
Besides successes in the clinical research and practices in the past, researchers have been interested in applying the BCI technology to improving human performance
Motor action paradigm: Because a BCI establishes a direct link between the brain and the output device, the conventional pathway of peripheral nerves and muscles in motor control can be bypassed. Furthermore, delays between early stages of sensory information processing in other brain cortices (e.g., the visual cortex and the parietal cortex) and the stage of motor control in the motor cortex could be eliminated as well. Using this paradigm, motor behaviors can be predicted more rapidly than the actual motor reaction time (RT). For example, in the motor cortex, spatial patterns of movement-related cortical potentials (MRCPs) could be extracted to identify the body side of an upcoming movement (e.g., left hand or right hand)
Mental-state monitoring paradigm: Many studies have shown that signal changes related to alertness, arousal, and cognition are presented in EEG
Visual target detection paradigm: Generally, target detection tasks need manual responses to confirm detections of targets. In practice, a target can also be indicated by brain activities such as a P300 event-related potential (ERP), which is elicited by a rare target event
Additional input paradigms: Combined with traditional input devices, a BCI can provide an additional method to improve the speed of inputting. For example, when playing a video game, BCI-based controls combined with traditional input modalities (e.g., joystick, mouse, keyboard, etc.) could make operations faster
The concept of using BCIs to augment human performance is quite straightforward; however, this topic has not attracted much attention in real-life applications for ‘normal’ healthy populations due to technical limits of the EEG measurement and processing in real-world environments. One of the most challenging problems comes from the poor BCI performance caused by low signal-to-noise ratio (SNR) of EEG signals, making a BCI a frustrating alternative to other input modalities. For instance, the accuracies of a single-trial EEG classification using a binary finger-tapping task (left hand vs. right hand) in BCI Competition II ranged from 51% to 84% from 15 research groups
This study presents and discusses many issues in hardware/software designs and data processing for a collaborative BCI. First, it proposes a centralized paradigm and a distributed paradigm for system designs. Next, implementation of a collaborative BCI using a motor action paradigm is introduced. After that, three different data processing methods: (1) ERP averaging, (2) Feature concatenating, and (3) Voting, are proposed and compared in detail. Finally, this study demonstrates the efficacy of the collaborative BCI method in improving human performance.
The study was approved by the Human Research Protections Program of the University of California San Diego. All participants were asked to read and sign an informed consent form before participating in the study.
A collaborative BCI and a conventional BCI differ in many respects. A conventional BCI mainly aims to help the individual with motor disability to communication with the environment, whereas a collaborative BCI is specifically designed for improving human performance of healthy users. The basic design and operation of a collaborative BCI is shown in
To implement a collaborative BCI, there are several specific requirements for hardware and software designs due to the employment of multiple users. First, multiple EEG recording systems need to work independently and simultaneously. Second, multiple-subject data need to be received and synchronized with respect to the common environmental events. Third, multiple-subject data recording and data processing procedures have to be performed in (near) real time. Ideally, the system can be implemented using a centralized paradigm similar to a conventional BCI (
(A) a centralized paradigm; (B) a distributed paradigm.
To find a remedy to these problems associated with the centralized paradigm, we propose a distributed paradigm to facilitate the implementation of a collaborative BCI. As shown in
This BCI study adopted a motor action paradigm reported in
An EEG and a behavior experiments were run separately on two groups of subjects. Twenty right-handed participants (12 males and 8 females, mean age 25 years) with normal or corrected-to-normal vision participated in the EEG experiment. Another group of 18 subjects participated in the behavior experiment (12 males and 6 females, mean age 23 years).
A delayed saccade-or-reach task was used in the EEG study, allowing us to look for direction information in the EEG during the phase of movement planning. The experiment was comprised of nine conditions differing by movement types (saccade to target, reach without eye movement, or visually guided reach) and movement directions (left, center, or right). Each task was indicated to the subject by, first, giving an effector cue telling the type of action to be performed, followed by a direction cue and, finally, by an imperative action cue. Subjects were seated comfortably in an armchair at a distance of 40 cm from a 19-inch touch screen. A chin rest was used to help them maintain head position.
Subjects used their right hands to perform the reaching tasks. At the beginning of each trial, the subject's forearm rested on the table with an index finger holding down a key on a keypad placed 30 cm in front of screen center. The sequence of visual cues in each trial is shown in
(A) an EEG trial; (B) a behavior trial; (C) visual cues used to indicate effector and direction of a task.
The behavior experiment was designed to measure the actual RT of a reaching movement using the same paradigm except that there was no delay after the direction cue, i.e., the direction cue was also used as the Go cue (see
In the EEG experiments, EEG data were recorded using Ag/AgCl electrodes from 128 scalp positions distributed over the entire scalp using a BioSemi ActiveTwo EEG system (Biosemi, Inc.) referenced to the CMS-DRL ground. Eye movements were monitored by additional bipolar horizontal and vertical EOG electrodes. All signals were amplified and digitized at a sample rate of 256 Hz. Three cue presentation events and two manual response events (“button release” and “screen touch”) were recorded on an event channel synchronized to the EEG data by DataRiver software (A. Vankov). EEG and behavioral data were recorded from 20 subjects on different days using the exactly same target presentation sequences. Some practice blocks were run before starting the EEG recording. For each subject, the experiment consisted of four blocks (with breaks in between) each including five runs of 45 trials. Within each block, there was a 20-second rest between runs. A total of 900 trials were equally distributed between the nine tasks, which were presented to the subject in a pseudorandom sequence.
In the behavior experiment, only the events were recorded for obtaining the actual RT for a reaching response. For each subject, the experiment consisted of three blocks with a total of 675 trials equally distributed among the nine tasks.
This study focused on the estimations of planned direction of movement. For simplicity, we only used “left” and “right” conditions for “hand” tasks for further analysis. The same analysis could be applied to data under “eye” and “both” conditions. Epochs from the response delay period, 0 to 700 ms following the onsets of direction cues, were extracted from the continuous data, and labeled by movement directions. The period [−100 ms 0 ms] was used as the baseline for each trial. Electrodes with poor skin contact were identified by their abnormal activity patterns and then removed from the data.
We used independent component analysis (ICA) as an unsupervised spatial filtering technique to remove artifacts arising from eye and muscle movements. For each subject, all trials were band-pass filtered (1–30 Hz), concatenated, and then decomposed using the EEGLAB toolbox
(A) Grand average 128-channel scalp maps of ERPs and difference waves (
The goal of this study is to demonstrate the efficacy of a collaborative BCI, rather than the EEG dynamics associated with all different task conditions. Therefore, the analysis below focuses only on the classification performance of predicting the intended movements based on the directional EEG information generated in the parietal cortex. To this end, two lateral electrodes over the PPC areas were selected for feature extraction based on the significance of ERP difference between left and right conditions.
Solid lines indicate the reaching
For classification, a support vector machine (SVM) classifier with an RBF kernel was implemented in the MATLAB® Bioinformatics Toolbox. The RBF kernel was optimized according to average classification performance across all subjects. To facilitate the training procedure, the scaling factor in the RBF kernel was fixed at 10 for all SVM classifiers. In this study, 10×10-fold cross validation was run to estimate classification performance for all classification tasks.
For each subject, classification of “left” versus “right” trials was performed using a standard machine-learning paradigm. For a collaborative classification based on data from multiple subjects, we propose three approaches to fuse the information from multiple subjects: (1) ERP averaging across subjects, (2) feature combination (e.g., concatenating features from multiple subjects), and (3) voting using an ensemble classifier. All these approaches can be implemented in the centralized paradigm, but for the distributed paradigm, only the voting approach is practical because data from each subject are processed separately in each of the BCI subsystems.
A widely used method for analyzing ERP has been to average EEG measurements over an ensemble of trials within a subject or across subjects
where
where
According to ERP studies, the model in equation (2) is not true when considering a more complicated ERP model, which involves multiple components
where
In the machine learning theory, feature combination can improve overall classification accuracy due to independence between features. Recently, following the wide employment of machine learning techniques in BCI studies, feature combination methods have been introduced in EEG classification
where the combined feature vector is a concatenation of feature vectors from
Theoretically, feature combination is optimal for a collaborative BCI. However, considering the fact of a BCI system that training data is always limited and feature combination will significantly increase the dimensionality of feature space, the feature combination method might encounter an overfitting problem. For example, the dimension of features from a single subject is 50 in equation (1) when using the time window of [0 ms 500 ms], which will be increased to 1000 for 20 subjects. However, the number of the training samples remains the same as in the single subject condition (100 trials per condition). Therefore, the performance gain of feature combination will be weakened due to a small training-set size.
Ensemble classifiers have been widely used in the area of machine learning
where
As mentioned before, the voting method is the only solution for a collaborative BCI using the distributed paradigm. Ideally, if there is no interaction between subjects, the voting method is supposed not to lose useful information for classification.
Realization of training and testing procedures of a collaborative BCI depends on the method used in feature extraction. The ERP averaging and Feature concatenating methods have to be realized on a centralized computer infrastructure where original EEG data from different subjects can be collected and processed. The Voting method can be realized either on a centralized or a distributed system.
In the collaborative BCI regime, a ‘single-trial’ actually comprised multiple epochs from multiple subjects following the same task stimulus. A 10×10-fold cross validation was used to assess classification performance. For the ERP averaging method, features of each trial were obtained by averaging feature vectors (Equation (1)) across subjects. An SVM classifier was then trained with the training set and applied to classification of the testing set. The Feature concatenating method used a similar way except that features were obtained by concatenating feature vectors from individuals (Equation (5)). In the Voting method, an SVM sub-classifier was used for training and testing for each subject separately. The collaborative classification was then performed using Equation (6).
The number of subjects is an important parameter for a collaborative BCI. In general, more subjects can provide more information for improving classification. Generally, when average performance is poor, any subject who has classification accuracy higher than the chance level can improve the overall performance of a collaborative BCI. However, the system costs (including hardware, software, and human resources) will also increase when more subjects are involved. Therefore, a tradeoff between the system performance and system cost should be made according to the specificity of the application.
To answer the question of how many subjects are needed to implement a satisfactory collaborative BCI, we evaluated system performance with respect to the number of subjects. For each number
In an application regime such as a target detection task, response time is always a critical parameter for evaluating human behavioral performance. In the motor action paradigm used in this study, we aimed to improve human performance through accelerating a motor decision-making, compared to RT. Therefore, it would be interesting to find out how fast a collaborative BCI can predict the direction of an upcoming reaching movement.
The actual mean RT for the hand reaching tasks measured in the behavior experiment was 464±62 ms across 18 subjects. As discussed before, response direction can be determined through extracting brain activities related to the visuomotor transmission procedure. According to prediction time, the system improves the overall performance when response direction can be accurately predicted at any time point earlier than the RT. To explore the system's capability of accelerating motor decision-making, we evaluated the system performance at different time durations used for feature extraction. Time windows with zero onset and different offsets starting from 100 ms and ending at 500 ms, incrementing with an interval of 10 ms, (i.e., 0–100 ms, 0–110 ms, …, 0–500 ms) were used to calculate accuracy-time curves. To show interaction between the prediction time and the number of subjects, different numbers of subjects (1, 5, 10, 15, and 20) were included for comparison.
Thin solid lines indicate classification results for 20 subjects, and the thick solid line shows the averaged accuracy. The dash line indicates the mean response time (RT) measured in the behavior experiment (464 ms).
Theoretically, the collaborative classification is expected to achieve a significant gain in the overall system performance.
Solid lines indicate mean accuracy, and dashed lines indicate mean accuracy ± standard deviation.
Classification performance for all three collaborative methods had been significantly improved when data from multiple subjects were combined and integrated. The ANOVA showed a highly significant effect of ‘number of subjects' on classification performance (Voting: F(19, 9980) = 3061.83, p = 0.00; ERP averaging: F(19, 9980) = 1634.06, p = 0.00; Feature concatenating: F(19, 9980) = 809.35, p = 0.00). When data of two subjects were combined, the T-test showed a significant difference between the individual performance and the collaborative performance when using the Voting method (p<0.01) and the Feature concatenating method (p<0.001) respectively. For the ERP averaging method, at least three subjects were required to reach a significant level (two subjects: p>0.05, three subjects: p<0.0001). A more prominent significance was obtained when the number of subjects increased. Although classification accuracy for single subject was low (mean across subjects: 66%), the collaborative method could still reach a high classification performance. For example, when using all 20 subjects, all three methods showed significantly improved accuracy (95% for the Voting method, 92% for the Averaging method, and 84% for the Feature Concatenating method).
The classification accuracy was enhanced substantially as well as the standard deviation decreased when the number of subjects increased. For example, using the Voting approach, the accuracy increased from 66% to 80%, 88%, 93%, and 95% as the number of subjects increased from 1 to 5, 10, 15, and 20, respectively, meanwhile, the standard deviation reduced from 7.0% to 1.0% when the number of subjects increased from 1 to 20. These results proved the existence of independence between subjects, which made all subjects contribute to the improvement of system performance and robustness.
The Voting method is optimal for collaborative EEG classification. The Voting method always outperformed the ERP averaging method when multiple subjects were involved. Accuracy of the Feature concatenating method was obviously affected by the overfitting problem. As shown in
As mentioned before in the method section, time required to make a prediction is a very important parameter to evaluate the performance of a BCI system in a motor action paradigm.
The dash line indicates the mean response time (RT) measured in the behavior experiment (464 ms).
This study demonstrated that a collaborative BCI could significantly improve system performance through integrating useful information from a group of users. Obviously, system performance can be further improved if more subjects are involved in the system, however the system cost and complexity also increase accordingly.
This study also explored three methods for fusing and analyzing collaborative EEG from multiple subjects. The results of this study showed that the Voting method was optimal for collaborative EEG classification, while all three collaborative BCI outperformed the single-subject BCI.
Currently, there are several challenges that have to be resolved before an online collaborative BCI system can become a reality. First, a collaborative BCI needs multiple BCI hardware platforms, which consist of an EEG recording system and a real-time signal-processing platform. Because commercial EEG products used for EEG research are still expensive, the total cost for building a collaborative BCI will be high. In practice, low-cost and customized EEG recording devices and signal-processing platforms can be used for implementing a collaborative BCI. Second, a collaborative system requires specific software development. As mentioned in the method section, the system needs seamless communication between EEG systems and signal-processing platforms, and between the BCI subsystems and the data server. Furthermore, data processing in BCI subsystems and the data server has to be implemented in (near) real time. Third, the complexity of a collaborative BCI needs to be further reduced to reality. When using conventional wet electrodes, capping and user training will be very labor intensive and time consuming. In a collaborative BCI regime, these procedures need to be simplified considerably.
Although the system demonstrated in the current study was implemented in an offline manner, it can be directly transferred to an online system if the hardware and software requirements can be met. With advances in biomedical electronics and telecommunication technology, it will soon be possible to implement an online collaborative BCI system. Recently, Wang et al.
This study demonstrated an application of the collaborative BCI to accelerate motor decision-making of a reaching movement. Moreover, a collaborative BCI can be applied to many other applications in which the overall human-system performance is critical. It will be especially useful for real-time situations where classification accuracy is critical, but performance of single-user BCI is poor.
A collaborative BCI system can also be used as a platform for studying the human brain in naturalistic environments. For example, using a collaborative BCI system, social interaction involving a group of people can be studied with real-time monitoring of brain activities to explore the underlying brain mechanisms.
In addition, other emerging applications of BCI's such as classroom education, neuroeconomics, and video gaming might also benefit from a collaborative BCI. A collaborative BCI might enhance the effectiveness of training and educational programs through monitoring either the student's attention/concentration or ability to participate effectively. Similarly, it can be applied to the field of neuroeconomics to evaluate the effects of designs of advertisements on the brain activities. Recently, the BCI technology has also been introduced into video gaming
Future directions to improve single-subject performance include:
Using more electrodes: Despite that all 128 channels were used in the procedure of ICA-based artifact removal, two electrodes placed near the PPC areas were selected for feature extraction. The system performance might be improved if more electrodes are employed. Firstly, for more electrodes over the PPC area, spatial filtering techniques can be applied to improve the SNR through removing task irrelevant activities
Using subject-specific parameters: For simplicity, this study used the same time-frequency parameters for different subjects, which might not be optimal due to individual variability. It has been shown in previous studies that subject-specific parameters can significantly improve classification accuracy of a BCI
Adding additional EEG features: This study only used ERP amplitudes as the features for classification. Other features such as spectral modulation might be a complementary feature for improving the classification performance. For example, Thut et al.
Future directions for improving the overall collaborative BCI system include:
To improve the ERP averaging method, weighted averaging methods might be helpful for enhancing the SNR of EEG signals
Improvement of the Feature concatenating method might be achieved from several directions, aiming to reduce overfitting. First, the dimension of the features can be reduced using feature selection methods. Second, generalization ability can be improved through increasing the size of training data. Third, classifiers specifically designed for high-dimensional data with a small training set might be helpful.
To improve the Voting method, the effort should be put on using other ensemble classifiers with better performance. Besides, some other ensemble learning techniques, such as the boosting and the bagging methods
This study proposed a collaborative BCI paradigm, which fused single-trial EEGs from multiple subjects to improve the overall BCI system performance. By comparing system designs and data analysis methods, this study showed that a distributed paradigm combined with a Voting classifier is a practical solution for implementing a collaborative BCI system. The feasibility and efficacy of the proposed BCI system was demonstrated through a collaborative BCI that could accelerate motor decision-making of a cue-guided reaching movement. The classification accuracy of the system showed a significant improvement over that of the single-subject BCI. Furthermore, the collaborative BCI allowed the subject's reaching direction to be estimated much earlier than his/her actual motor response. In summary, this study designed and demonstrated the use of the collaborative BCI technology to improve human performance in natural environments.
Melody Jung is appreciated for editorial assistance.