Fig 1.
Experiment Software Architecture.
The figure illustrates a conceptual overview of the software architecture used in the experiments. The BioSemi module provided a data stream containing the EEG data from the participant. The Tobii X2-30 module provided a data stream containing gaze and pupil diameter data of the participant. The SNAP module with custom scripts provided a data stream containing annotated stimulus events and markers. The Lab Streaming Layer collected and time synchronised all experimental data streams. The LabRecorder module recorded all data to file using the XDF format. Data analysis on the user study generated XDF files was carried out using the BCILAB and EEGLAB software libraries.
Fig 2.
Timing of the Experimental Task Within One Trial.
The auditory go cue consisted of the utterance of the words “Left” or “Nothing” according to the class that the trial belongs to. The auditory stop cue consisted of the utterance of the word “Stop”. A micro-break of 6 seconds was given to participants within each trial.
Fig 3.
The Figure illustrates a set of common spatial patterns (CSPs) filters of a single participant in the study. The CSPs are optimized for the discrimination of left hand motor imagery from a control rest condition.
Table 1.
Different Metrics of Pupil Diameter.
We investigated several pupil diameter derived metrics to try to find the feature that obtains better classification accuracy. The average value for the statistics for all participants in the user study for both experimental conditions was calculated. The middle point between the previous two values was used as a threshold for classification. The classification accuracy is shown in the rightest column with the standard deviation. In the last row, FBA stands for mean Frequency Band Amplitude.
Fig 4.
Illustration of the Different Classification Approaches.
A legend of the symbols used in this figure is contained in the right side of the figure. The dashed red box encapsulates the method used to classify motor imagery using only the pupil signal. A moving average of the pupil diameter is continuously updated and compared to a reference baseline period right before trial start. The dotted blue box encapsulated the method used to classify motor imagery using the EEG derived signals and the CSP algorithm. The alternately dashed and dotted purple box illustrates our proposed method by which the average pupil diameter is added as an extra feature to the EEG derived feature vector used for classification. After extension of this vector, classification is carried out.
Table 2.
Average classification accuracy, Standard deviation, Kappa Coefficient and Information Transfer Rate (ITR) in bits/trial and bits/min for all classification methods split by the training and test blocks experimental approach and the 10-fold cross-validation within block experimental training approach.
Fig 5.
Average classification accuracy for the different methods; error bars show Confidence Interval. The figure is color-coded to match the bounding boxes in Fig. 4 that identify the different paradigms employed in data analysis.
Table 3.
Correlations of classification accuracy between pupil diameter and EEG, pupil and EEG+pupil as well as EEG+pupil and EEG; The table is split by the training and test blocks experimental approach and the 10-fold cross-validation within block experimental approach.
Fig 6.
Scatter Plots of Correlation Analysis Between Different Classification Feature Sets.
The Figure illustrates in the top row with black dots the classification accuracy with the separate training and test set approach and in the bottom row with blue dots the classification accuracy with the cross validation training approach.
Fig 7.
Pupillary Diameter Time Course for a Single User During an Entire 5 Minutes Block.
The Figure illustrates the inherent noise of the signal and the difficulty to discern patterns of motor imagery by pupil dilation with the naked eye. Trial classes (left hand imaginary movement and nothing condition) are marked in the Figure as pink and cyan vertical bars respectively. The beginning of the experiment is marked with a red vertical line. The end of the experiment is marked with a vertical green line. Notice that the second half of the interval between vertical bars corresponds to the micro-break given after the auditory stop cue within each trial and hence it was not used in the calculation of the average pupil diameter.
Fig 8.
Pupil Diameter Time Course Averages within Experimental Epochs for All Participants.
The figure illustrates the average time course of pupil diameter within all trials for both experimental conditions. The red color denotes the left hand motor imagery class and the blue line the no task condition. Pupillary diameter enlarges during the the motor imagery condition epochs and its average is higher than the average pupil diameter during the control condition for almost all participants.
Fig 9.
Pupil Diameter Frequency Response within Experimental Epochs for All Participants.
The figure illustrates the average frequency domain response of pupil diameter within all trials for both experimental conditions for all participants. The red color denotes the left hand motor imagery class and the blue line the no task condition. Pupillary oscillations in the frequency band between 0 and 0.3Hz seem larger in the left motor imagery trials during the motor imagery condition in most but not all participants.
Fig 10.
Event Related Spectral Perturbation for all Participants During Left Hand Motor Imagery.
The figure illustrates the frequency spectrum time course of EEG activity within motor imagery epochs for each participant. Event Related Desychronization in α and/or β frequency bands is apparent in some but not all participants. This is to be expected in a large pool of subjects due to the anatomical differences of cortical foldi and the resulting effects of volume conduction.