Brain response to luminance-based and motion-based stimulation using inter-modulation frequencies

Steady state visual evoked potential (SSVEP)-based brain computer interface (BCI) has advantages of high information transfer rate (ITR), less electrodes and little training. So it has been widely investigated. However, the available stimulus frequencies are limited by brain responses. Simultaneous modulation of stimulus luminance is a novel method to resolve this problem. In this study, three experiments were devised to gain a deeper understanding of the brain response to the stimulation using inter-modulation frequencies. First, luminance-based stimulation using one to five inter-modulation frequencies was analyzed for the first time. The characteristics of the brain responses to the proposed stimulation were reported. Second, the motion-based stimulation with equal luminance using inter-modulation frequencies was also proposed for the first time. The response of the brain under these conditions were similar to that of luminance-based stimulation which can induce combination frequencies. And an elementary analysis was conducted to explain the reason of the occurrence of combination frequencies. Finally, the online test demonstrated the efficacy of our proposed two stimulation methods for BCI. The average ITRs reached 34.7836 bits/min and 39.2856 bits/min for luminance-based and motion-based stimulation respectively. This study demonstrated that the simultaneous modulation of stimulus luminance could extend to at least five frequencies to induce SSVEP and the brain response to the stimulus still maintained a certain positive correlation with luminance. And not only luminance-based stimulation, but also motion-based stimulation with equal luminance can elicit inter-modulation frequencies to effectively increase the number of targets for multi-class SSVEP.


Ethics statement
Subjects were studied after giving informed written consent in accordance with a protocol approved by the institutional review board of Xi'an Jiaotong University.

Design of luminance-based stimulation
In this study, a green straw hat LED is chosen as the visual stimulation. To avoid eye discomfort caused by the bright light from the middle of the LED, a hemispherical lampshade made of light diffusing material with 85% light transmittance is used. The light transmittance through the lampshade from the fluorescent lamp ranges from 80% to 90%. And 85% light transmittance is a balance between the participants' visual comfort and the intensity of the stimulus. What's more, the diameter of the hemispherical lampshade is 56mm which is just in the commonly used standard size. And the viewing distance is 70 cm in commonly studies. So the visual angle is 4.6 degrees which is a suitable visual angle for subjects. The stm32F103RB (STMicroelectronics company) is chosen as the microcontroller to produce multiple frequencies. By using the timer of STM32, the pulse width modulation (PWM) signal can be produced.
Changing the duty ratio of the PWM signal changes the luminance of the LED. If the duty ratio of the PWM changes as DR (see formula (1)), the multi-frequency stimulation is possible. Besides, the timer is also used to guarantee the accuracy of the cycle. The time interval is 1 millisecond (ms).
where f 1 ,f 2 ,. . .,f n are frequencies of the stimulation, A 1 ,A 2 ,. . .,A n are amplitudes, n is the number of the multiple frequencies and t is time interval.
To guarantee the proposed method could produce the desired simultaneous flickering frequencies, the following test is performed prior to the experiments. Photistor (3DU5C) is chosen as the sensor to measure the change of light intensity of LED flickering. A resistor was connected to the photistor and a 3.3V power supply was used. The change in the resistance voltage was acquired by the data acquisition card NI9234 (National Instruments, USA). LED and photistor were placed face to face. The distance between them was 6 centimeter (cm). The simultaneous flickering frequencies were 16Hz, 15Hz, 14Hz, 13Hz and 12Hz. Fig 1 shows the result of the test. Fig 1A shows the simulation of duty ratio of PWM. Fig 1C shows the signal acquired by NI9234. Fig 1A and 1C are almost same. Their spectrums also have the same peak frequencies (Fig 1B and 1D respectively).

Design of motion-based stimulation
The ring chessboard is chosen as the basic pattern of the proposed motion-based stimulation. The ring chessboard paradigm is a development of Newton's rings paradigm [17] which can provide a comparable performance with low-adaptation characteristic and less visual discomfort for BCI applications. In the ring chessboard shown in Fig 2, each ring is divided into black and white lattices of equal numbers and sizes, so the area of the bright and dark areas in each ring are always equal. The brightness value of the central part of the stimulus unit is always set to the background brightness, and the area of the central part remains unchanged. As such, the luminance of the designed motion-based stimulation remains unchanged. The S1 File in the Supporting Information shows the main coding algorithms for producing the stimulation with dual frequencies (the style of the paradigm needs users to create it by themselves). The method to produce the chessboard paradigm proposed in this study is described below.
The stimulus program is developed under MATLAB using the Psychophysics Toolbox. Formula for generating pattern of ring checkerboard stimulating unit is as follows. : where φ(t) is the phase value function in contraction expansion, N c is the number of white and black squares contained in a single ring, L 0 is background brightness, W is the width of the ring and A is the amplitude of the motion. The motion reversal process of the motion-based stimulation is controlled by phase functions φ(t). where f 1 ,f 2 ,. . .,f n are frequencies of the stimulation. The ring chessboard will contract as the phase changes from 0 to pi and it will expand as the phase changes from pi to 0 (The specific forms of movement can refer to the S1 Video). Motion direction changes twice in one cycle. The motion inversion frequency is defined as the frequency of motion direction changes. And the motion inversion frequency is twice of motion frequency. The annular expansion and contraction of the ring chessboard forms visual stimuli with inter-modulation frequencies and the luminance remains unchanged.

Experimental design
Ten healthy male subjects (ages 23-25) participated in the experiments. They were all volunteers from Xi'an Jiaotong University. EEG signals were recorded from six EEG electrodes (G. USBamp, G.tec Guger Technologies, Austria). Based on the international 10-20 system, the six EEG electrodes were placed at PO3, PO4, POz, O1, O2, Oz. The unilateral (left or right) earlobe was used as the recording reference and Fpz was used as ground. All electrodes' impedances were kept below 5 kOhm. The sampling frequency was 1200 Hz. Band-passed filter between 2 and 100 Hz and notch filter from 48Hz to 52Hz were used to remove artifacts and power line interface when acquiring EEG signals. The subjects were asked to sit on a comfortable chair in a quiet and ordinarily lit office room. All of the subjects had undergone BCIs before and had normal eyesight. Experiments were designed as described in Table 1. E1: Offline experiment of brain response under luminance-based stimulation using inter-modulation frequencies. There were two tasks during this experiment: equal amplitude stimuli using multiple inter-modulation frequencies and unequal amplitude stimuli using fixed dual inter-modulation frequencies. The green LED with a hemispherical lampshade was placed 70 cm in front of the subject. The LED would flicker for 5 s and light out for 5 s automatically. All the subjects were required to stare at the flickering LED for 5 s per trial, 5 trials each inter-modulation frequency, with an interval of 5 s. Then the inter-modulation frequencies of the flickering LED would change as described below.

E2: Offline experiment of comparisons between luminance-based and motion-based stimulation using dual inter-modulation frequencies.
Luminance-based stimulation and motion-based stimulation described before were all used in this experiment. The inter-modulation frequencies were 5+8 Hz. All the subjects were required to stare at the two stimulations for 10 s per trial, with five trials for each stimulation, with an interval of 5 s respectively. As to luminance-based stimulation, the green LED with a hemispherical lampshade was placed 70 cm in front of the subject. And the LED would flicker for 10 s and then turned off for 5 s automatically. As for motion-based stimulation, it was presented in the middle of a LCD monitor. The diameter of the ring chessboard was 60 mm. The ring chessboard would for 10 s and then disappeared for 5 s automatically too. EEG signals were recorded in the computer hard disk automatically.
E3: Online experiment of luminance-based and motion-based stimulation using intermodulation frequencies. In regard to luminance-based stimulation, there were ten targets with dual inter-modulation frequencies including 16+15 Hz, 16+14 Hz, 16+13 Hz, 16+17 Hz,  13.3+20 Hz, 12+12 Hz, 12+20 Hz, 20+20 Hz. The fifteen targets were presented on the monitor simultaneously. There were three rows and five targets in each row. The stimulation duration per trial was divided into six levels, from 1 s to 6 s with an interval of 1 s. Subjects were required to stare on the targets one by one. The luminance-based experiments were carried out for 5 runs and the motion-based experiments were carried out for 6 runs. And the identification results would be shown to subjects automatically during the interval.

Data analysis
Canonical correlation analysis. Canonical Correlation Analysis (CCA) is widely used in SSVEP target recognition. It is carried out to calculate the correlations between reference signals and multi-channel EEG data [18]. The formula to calculate correlation coefficient ρ is as follows.
where X is the EEG data, Y is the reference signals.
In this study, the reference signal Y 1 in the luminance-based online experiment and the reference signal Y 2 in the motion-based online experiment were chosen as follows: where Fs is the sampling frequency and f 1 ,f 2 were the inter-modulation frequencies. Then the maximum of canonical coefficients was considered as the focused target. Besides, due to the latency delay in the visual system based on prior research [19], the date epochs began to be processed at 0.14 s. CCA-based spatial filter. To enhance the signal-to-noise ratio, the CCA-based spatial filter was used to analyze the offline experimental data. When the correlations were calculated as in formula (4), the weight vectors w x could be obtained at the same time. The multi channel EEG data can be converted into one-dimensional signals by multiplying the weight vectors w x as follows.X Power spectrum density. The peak values at the stimulation frequencies in the power spectrum density (PSD) reflect the effect of inducing SSVEP by the designed stimulus. Hence the power spectrum density was chosen as the method to analyze the characteristics of the EEG response to multiple inter-modulation frequencies visual stimulus.
In this study, the Welch power spectrum density was used to estimate signals by dividing a datum with a length of N into M segments. The formula is as follows.
Statistical analysis. Statistical analysis is conducted using repeated measures analysis of variance with Bonferroni posthoc analysis. Statistical significance is defined as p < 0.05.

Results
Offline experimental results of brain response under luminance-based stimulation using inter-modulation frequencies Fig 3 shows the time domain waveforms, PSD and time-frequency maps acquired while a participant was staring at the equal amplitude stimulation using multiple inter-modulation frequencies as described before. It was clearly observed from the PSD that the peaks were evoked at the frequencies of the stimuli. However, previous studies [20][21][22] showed that it would evoke the peaks at m × f 1 + n × f h (m,n = 0,±1,±2. . .) where f 1 and f h were the dual inter-modulation frequencies. So as for the stimulus with 16+15+14 Hz, the peak at 14 Hz in the PSD may have been evoked by 14 Hz in the stimulus or in the combination frequency (14 = 2 × 15-16). So ANOVA on the peak values in the PSD was conducted to distinguish the difference. Table 2 shows the P values for all the subjects. Take subject 1 as an example, there was a significant difference on the peak values at 15 Hz in the PSD between the subject staring at the stimulation with 16 Hz and the stimulation with 16+15 Hz (P = 0.005<0.05). Similarly, there was significant difference on the peak values at 14 Hz in the PSD between the subject staring at the stimulation with 16+15 Hz and the stimulation with 16+15+14 Hz (P = 0.004<0.05). And there was a significant difference on the peak values at 13 Hz in the PSD between the subject staring at the stimulation with 16+15+14 Hz and the stimulation with 16+15+14+13 Hz (P = 0.029<0.05). Furthermore, there was a significant difference on the peak values at 12 Hz in the PSD between the subject staring at the stimulation with 16+15+14+13 Hz and the stimulation with 16+15+14+13+12 Hz (P = 0.04<0.05). The above mentioned results demonstrated that the stimulation would induce the corresponding frequencies in SSVEP when increasing frequencies in the inter-modulation frequencies stimulation and that the newly added induced frequencies in SSVEP were mainly caused by the added frequencies in the stimulation and not by the combination frequencies.
Even though the amplitudes of the stimulus at each frequency were equal, the peak values at the corresponding frequencies in PSD evoked by the stimulus were not the same. There was an  The results showed that there were no significant difference between the different sums of PSD for each subject (P>0.05). That meant the sum of the peak values at the multiple inter-modulation frequencies in the PSD demonstrated no significant difference. Fig 5 shows the mean of the peak values at 15 Hz in the PSD among all the participants and the mean of the peak values at 16 Hz in the PSD among all the participants while they were staring at the unequal amplitude stimulation using dual-modulation frequencies as described before. As the increase of one amplitude of the stimulation using dual inter-modulation frequencies, the peak values at corresponding frequency in the PSD increased. However the sum of the peak values at 15 Hz and 16 Hz in the PSD had no significant difference (P = 0.999 >0.05) as shown in Fig 6. As such, it was concluded that the sum of the peak values at the two frequencies in the PSD had no significant difference as to unequal amplitude stimuli using fixed dual inter-modulation frequencies. That's to say the brain response had a positive correlation with the light intensity of the stimulation using inter-modulation frequencies.

Offline experimental results of brain response under luminance-based stimulation and motion-based stimulation using inter-modulation frequencies
The Oz site experiences the main neuronal activities in the visual area because it is closest to the striate cortex [23]. Fig 7 shows the frequency spectrums of EEG data at the Oz site while one participant was staring at the luminance-based stimulation and motion-based stimulation respectively. The EEG data were dealt with using time domain averaging first. Several combination frequencies can be clearly observed in both spectrum. As for luminance-based stimulus, the peaks in the spectrum occurred at f 1 ,f 2 ,2 × f 1 ,2 × f 2 ,f 2 ±f 1 ,3 × f 2 ± f 1 ,3 × f 1 + f 2 where f 1 = 5Hz, f 2 = 8Hz. As for motion-based stimulus, the peaks in the spectrum occurred at where f 1 = 5Hz, f 2 = 8Hz. So both the luminance-based and motion-based stimulation using dual inter-modulation frequencies could induce SSVEP at combination frequencies. But there were some differences between them. The induced frequencies by motion-based stimulation were combined by the motion frequencies ð f 1 2 ; f 2 2 Þ, not the motion inversion frequencies. And the peaks occurred at the motion inversion frequencies, not the motion frequencies.  https://doi.org/10.1371/journal.pone.0188073.g007 Brain response to L&M stimulation using inter-modulation frequencies Experimental results of online recognition accuracy  Fig 8(A). When the dual frequencies were 17.1+15 Hz, 17.1+13.3 Hz, 17.1+12 Hz, 17.1+20 Hz, 15+13.3 Hz, 15+12 Hz, 15+20 Hz, 13.3+12 Hz, 13.3 +20 Hz, and 12+20 Hz in proper order, as to motion-based stimulation, the main induced frequencies were 16.05 Hz, 15.4 Hz, 14.5 Hz, 18.5 Hz, 14.1 Hz, 13.5 Hz, 17.5 Hz, 12.6 Hz, 16.6 Hz, 16 Hz successively as shown in Fig 8(B).
In total, as to luminance-based stimulation, the main peaks in the spectrum occurred at f 1 and f 2 . The amplitudes at the combination frequencies in the luminance-based stimulation were not outstanding. So Y 1 in forum (5) was chosen as the reference signals to perform CCA. https://doi.org/10.1371/journal.pone.0188073.g008 Brain response to L&M stimulation using inter-modulation frequencies As to motion-based stimulation, the main peaks in the spectrum occurred at f 1 þf 2 2 . So Y 2 in forum (5) was chosen as the reference signals to perform CCA.
CCA was used to calculate the recognition accuracy of luminance-based stimulation and motion-based stimulation for each subjects. The recognition accuracies of luminance-based stimulation and motion-based stimulation of all the subjects according to stimulation duration are shown in Tables 3 and 4 respectively. Noticeably, the recognition accuracy of S10 reached 98% with only 2 s of stimulation. Regarding the average accuracies, the luminance-based stimulation had higher recognition accuracy than the motion-based stimulation did when the duration was lower than 4 s. And both the recognition accuracies of the two stimulations were decrease sharply as the duration decreased. When the stimulation duration exceeded 4 s, the change of recognition accuracy tended to be stable and the motion-based stimulation had higher recognition accuracy than the luminance-based stimulation did. Fig 9 shows the mean information transfer rate (ITR) of subjects with different stimulation duration. The ITR reached the highest value at 3 s for 34.7836 bits/min and 39.2856 bits/min with luminance-based and motion-based stimulation respectively. In total, the ITRs of the motion-based stimulation were higher than that of the luminance-based stimulation for the different durations except 1 s. resonances [29], we supposed that the input signal IS was changed into NS when the visual stimulus was transferred to the primary visual cortex. NS contained the abundant intermodulation frequencies and so on.
Due to different sensitivities of the brain to different frequencies, some combination frequencies did not occur. And the amplitudes of the combination frequencies were different. So induced combination frequencies reported by other studies were different.
Besides, simultaneous modulation of stimulus was used to increase the number of targets using limited frequencies. Actually, n frequencies could totally generate C 1 n þ C 2 n þ C 3 n þ . . . þ C n n selections in our study. And our stimulation only contained one flickering LED or one target with multiple frequencies which did not cause the 'attention-shift' problem noted in previous studies [10,11,30]. What's more, Hakvoort [31] mentioned that more than 16% of the selections were misclassified because of the multiple harmonics using conventional CCA. In this study, Y 1 and Y 2 in formula (5) were chosen as the reference signals for the luminance-based and motion-based stimulations respectively. And online test showed the average ITRs reached 34.7836 bits/min and 39.2856 bits/min respectively. This meant that reference signals using CCA should be chosen based on the stimulus. In regard to ITR, the motion-based stimulation was better than the luminance-based stimulation. But the motion-based stimulation was difficult to extend to multiple intermodulation frequencies and only had a regular rule with dual inter-modulation frequencies. This was not the case with luminance-based stimulation. For the further research, the classification algorithms, stimulus frequencies, and induced frequencies still need to be optimized to increase the ITR and recognition accuracy.

Conclusion
The present study presented a thorough investigation on stimulation for SSVEP using intermodulation frequencies. Importantly, luminance-based stimulation using one to five intermodulation frequencies could induce the corresponding stimulus frequencies in the primary visual cortex. And the response maintained a certain positive correlation with luminance. Additionally, the brain response to motion-based stimulation with equal luminance had similar characteristics with that of luminance-based stimulation. And a hypothesis was presented to explain the reason for the occurrence of combination frequencies. Furthermore, the ITRs in the online test indicated that both of the proposed stimulations were feasible for multi-class SSVEP-BCI systems.