The relevant resting-state brain activity of ecological microexpression recognition test (EMERT)

Zhang, et al. (2017) established the ecological microexpression recognition test (EMERT), but it only used white models’ expressions as microexpressions and backgrounds, and there was no research detecting its relevant brain activity. The current study used white, black and yellow models’ expressions as microexpressions and backgrounds to improve the materials ecological validity of EMERT, and it used eyes-closed and eyes-open resting-state fMRI to detect relevant brain activity of EMERT for the first time. The results showed: (1) Two new recapitulative indexes of EMERT were adopted, such as microexpression M and microexpression SD. The participants could effectively identify almost all the microexpressions, and each microexpression type had a significantly background effect. The EMERT had good retest reliability and calibration validity. (2) ALFFs (Amplitude of Low-Frequency Fluctuations) in both eyes-closed and eyes-open resting-states and ALFFs-difference could predict microexpression M. The relevant brain areas of microexpression M were some frontal lobes, insula, cingulate cortex, hippocampus, parietal lobe, caudate nucleus, thalamus, amygdala, occipital lobe, fusiform, temporal lobe, cerebellum and vermis. (3) ALFFs in both eyes-closed and eyes-open resting-states and ALFFs-difference could predict microexpression SD, and the ALFFs-difference was more predictive. The relevant brain areas of microexpression SD were some frontal lobes, insula, cingulate cortex, cuneus, amygdala, fusiform, occipital lobe, parietal lobe, precuneus, caudate lobe, putamen lobe, thalamus, temporal lobe, cerebellum and vermis. (4) There were many similarities and some differences in the relevant brain areas between microexpression M and SD. All these brain areas can be trained to enhance ecological microexpression recognition ability.


The ecological microexpression recognition test (EMERT)
Microexpressions are very transitory expressions lasting about 1/25-1/2 s, which can reveal people's true emotions they try to hide or suppress [1,2]. Matsumoto  Japanese and Caucasian Brief Affect Recognition Test (JACBART, classical microexpressions recognition) to measure microexpression recognition. The participants would see a microexpression presented for a little time between two neutral expression backgrounds for 2000ms before or after it. Participants needed to check out the microexpression type. The neutral expression backgrounds could eliminate the visual aftereffects of the microexpressions. But it did not examine the influence of backgrounds with emotional expressions. Therefore, Zhang, Fu, Chen and Fu [4] explored the background effect on microexpressions and found that all microexpressions (anger, disgust, fear, surprise and happiness) recognition accuracies under negative (sadness) backgrounds were significantly lower than those under positive (happiness) or neutral backgrounds; when the backgrounds and the microexpressions were consistent in the property (negative or positive), microexpression recognition accuracies were significantly lower than those when they were inconsistent in the property. The research has broken through the JACBART paradigm. But it did not explore all backgrounds or all microexpressions and needed to be further developed. Yin, Zhang, Shi, and Liu [5] for the first time proposed that all basic expression kinds for both backgrounds and microexpressions needed to be detected to set up the ecological microexpression recognition test. Therefore, Zhang et al. [6] examined the recognition characteristics of six basic expression kinds of microexpressions (sadness, fear, anger, disgust, surprise, happiness) under seven basic expression kinds of backgrounds (the six basic expressions and neutral) to establish an ecological microexpression recognition test-EMERT, and found that EMERT had good retest reliability, criterion validity and ecological validity: (1) EMERT was generally significantly related to JACBART. (2) The backgrounds main effect of sadness, fear, anger and disgust microexpressions were significant; the backgrounds main effect of surprise and happiness microexpressions were not significant, but there was a wide difference between them with the common expressions. (3) The ecological microexpression recognition had stable fluctuation. Zhu et al. [7] used the simplified edition of EMERT to find microexpression recognition difference between depressive patients and normal people. Yin, Tian, Hua, Zhang, and Liu [8] extended EMERT to WEMERT (weak ecological microexpression recognition test).
But EMERT by Zhang et al. [6] only used white models' expressions as microexpressions and backgrounds, so white, black and yellow models' expressions need to be used as microexpressions and backgrounds to improve the ecological validity of materials.

Brain activation of ecological microexpression recognition
There were few published types of research detecting brain activation of ecological microexpression recognition. Shen [9] in Xiaolan Fu's team used fNIRS to find that the brain area responsible for JACBART microexpressions recognition was in the left frontal lobe, while the brain area responsible for common expressions recognition was in the right frontal lobe. Zhang [10] in Xiaolan Fu's team used fMRI to find that for anger and neutral microexpressions, the inferior parietal lobule was activated more in the negative expression backgrounds than in the neutral expression backgrounds, while the right precuneus was activated more in the positive expression backgrounds than in the neutral expression backgrounds. For happiness microexpressions, the parahippocampal gyrus was activated more in positive backgrounds. These studies revealed the brain mechanisms of classical microexpressions and three ecological microexpression recognition, but more ecological microexpression recognition needs further research.
As there were 36 ecological microexpressions in EMERT [6], it is neither feasible nor economical to adopt task-state fMRI. Resting-state fMRI is a viable and economical option. Resting-state fMRI investigates spontaneous activity or functional connections within the brain at rest. If a certain cognitive task is associated with certain brain areas that are active in a resting state, then these brain areas are associated with the cognitive task. If brain areas whose activity in resting state related to two cognitive tasks differ, then the brain mechanisms underlying execution of these two cognitive tasks are different [11][12][13]. Brain spontaneous activity in restingstate is a stable index to measure the individual cognitive characteristics [14]. One of the classic indexes is ALFFs value (the Amplitude of Low-Frequency Fluctuations, 0.01~0.1 HZ), including most of the psychological, cognitive process. The higher and lower amplitudes are background noise such as physiological activity. There were eyes-closed and eyes-open restingstates. Nakano, etc. [15,16] found that in eyes-closed, subjects focused on internal feeling and self-consciousness, while in eyes-open, subjects turned to external stimulus processing, and the transition from eyes-closed to eyes-open was from internal feeling and self-consciousness to external stimulus processing. However, there was no microexpression research using resting-state fMRI.

Improvements made in the current study
The current study would use white, black and yellow models' expressions as microexpressions and backgrounds to improve the materials ecological validity of EMERT. It would use eyesclosed and eyes-open resting-state fMRI to detect relevant resting-state brain activity of EMERT for the first time.

Participants
Sixty-five college students were selected to participate in the study. Males and females were 32 and 33. The age M ± SD = 21.71 ± 2.58. They were all right-handed with normal or correctedto-normal eyesight and without colour blindness. They all volunteered and could quit at any time. Each participant completed an informed consent form before the experiments. They got corresponding rewards after completing the experiments. The experiments were by the ethical guidelines of the Declaration of Helsinki and were approved by the Scientific Review Committee of Faculty of Psychology, Southwest University, China.

Experimental apparatus and materials
Seven kinds of basic expression opened mouth pictures of eight models (four male and four female, including white, black and yellow people) from the NimStim face expression database [17] were used as the backgrounds, namely, neutral, sadness, fear, anger, disgust, surprise, and happiness. Except for neutral expression, the other six kinds of expressions were used as microexpressions. The pixels of all images were modified to be 338 × 434 with grey background (GRB: 127, 127, 127) [6]. A custom experimental program ran under E-prime 2.0 on a PC (Lenovo LX-GJ556D), with a 17-inch colour display (resolution 1024 × 768, refresh rate 60 Hz).

Experimental design and procedures
The experiment was 7 (expression backgrounds: neutral vs sadness vs. fear vs anger vs disgust vs surprise vs happiness) × 6 (microexpressions: sadness vs fear vs. anger vs disgust vs surprise vs happiness) × 2(test times: first EMERT vs the second EMERT) within-subject design. As there were seven kinds of expression backgrounds, to balance the sequential effect, the Latin square design was used to set up seven groups with about nine participants (four or five females and males) in each group. Each dependent variable in seven groups was averaged in the result analysis [6].
Participants were 70 cm away from the screen. On the computer keyboard, six keys of SDF-JKL corresponded with 'anger', 'disgust', 'fear', 'sadness', 'surprise' and 'happiness'. Before the experiment, the participants were asked to put the ring finger, middle finger, index finger of their left hands on the SDF keys respectively while the index finger, middle finger, ring finger of their right hands on JKL keys. And then they did key pressing practices. First, one of the six kinds of expressions (except neutral) was presented 1000 ms; then six labels "anger, disgust, fear, sadness, surprise, happiness" appeared on the screen, and the participants needed to recognise it and press the right key as accurately as possible. There were 30 trials, and six kinds of expressions were pseudo-randomly presented for 5 times.
After the key pressing practice was completed, the instructor informed the participants of the procedure. First, the centre of the screen would show the "+" for 400 ms; second, the empty screen lasted 200 ms; then the front background expression image was presented for 800 ms, after which the microexpression image would appear for 133 ms, followed by 800 ms of back background expression image [3,6]. The front and back backgrounds and microexpressions were of the same model's face, and the front and back backgrounds were the same. Participants needed to try to identify the briefly-presented microexpression between front and back backgrounds. Later, six labels "anger, disgust, fear, sadness, surprise, happiness" appeared on the screen. The participants were asked to press a key according to the microexpression they saw as accurately as possible instead of as soon as possible (no time limit). After the participants pressed the key, an empty screen would show for 1000 ms. Then the fixation point "+" was presented for 400 ms, and the next trial started. The experiment procedure is shown in Fig 1. The participants practised the experimental procedure after understanding the instructions. There was a total of 14 trials, of which 7 kinds of backgrounds appeared 2 times, and 6 kinds of microexpressions each appeared 2 to 3 times. The participants were asked to determine the type of microexpressions. After the experimental procedure practice was completed, they started a formal trial. To allow the participants to get enough rest, the experiment was divided into seven blocks. Rest between every two blocks was 1 minute. The experiment had 7 (backgrounds) × 6 (microexpressions) × 8 (models) = 336 trails.
The participants needed to do two EMERT measurements, of which interval time was at least one day.
Pretreatment and analysis of resting-state data used DPARSF 3.0 Advanced Edition Calculate [18] in Original Space (Warp by DARTEL), following standard procedures: (1) Conversion of raw DICOM-format data to NIFTI format. To allow for signal stabilisation of the image, the first 10 TR images were removed, after which time layer correction (slice timing) and head motion correction (realign) were conducted. If head motion greater than 2 mm occurred during resting-state, the data were deleted. (2) The new segment + DARTEL was used to split the structural T1 data without standardisation, and register the T1 split data directly to the resting-state functional images. Before registration of structural and functional data, the AC-PC line of each participant's T1 image and the resting-state function was registered, and then automatic registration was applied. Therefore, the resting-state analysis took place in the original T1 space. (3) Head motion (adopting Friston 24), linear drift, white matter, and cerebrospinal fluid via regression were adjusted for. (4) Low frequency fluctuations ALFFs (filter range: 0.01 to 0.1 Hz) were calculated. (5) The resting-state function was registered to the standard MNI space (normalisation), using a 3 × 3 × 3 mm 3 voxel size, with 4 × 4 × 4 mm 3 full widths at half maximum (FWHM) smoothing. REST1.8 [19] was first used to extract the amplitude of low-frequency fluctuations (ALFFs) during resting-states in 116 Anatomical Automatic Labeling (AAL) brain areas. Second, SPSS19.0 was used to implement correlation analyses between ALFFs in 116 AAL brain areas and the scores of EMERT. The ALFFs-difference of eyes-open minus eyes-closed was used as an index of transition from internal feeling and self-consciousness to external stimulus processing. We detected its psychological significance by correlation analyses between it and the scores of EMERT.

SPSS 19
.0 was used for statistics. There were sixty-five valid participants in EMERT, fifty-eight valid participants in eyes-closed resting-state, and sixty two valid participants in eyes-open resting-state because seven participants' head movement were greater than 2 mm in eyesclosed resting-state, and three participants in eyes-open resting-state.

Behavioral data
The scores of EMERT were showed in Table 1. Because the accuracy of microexpression recognition in the second EMERT might contain training effect, the accuracy of that in the first EMERT was taken as the microexpression recognition ability. Since the participants have 6 keys to choose for each trial, the random level is 1/6. A single sample t test was made for each microexpression recognition accuracy with random level 1/6, and it was found that almost all the microexpression recognition accuracies in the first EMERT were significantly higher than random (ps<0.001), except that fear under surprise, was not significantly higher than random (p>0.05).
It can be seen that the EMERT indexes were too many to be recapitulative enough for both participants and researchers. Then the mean of accuracy rates of a microexpression type under six backgrounds (except the same expression grounds as the microexpression, because in that case, it was a normal expression rather than a microexpression) was used as the index of this

PLOS ONE
The relevant resting-state brain activity of EMERT microexpression type recognition. It was abbreviated as microexpression M. The standard deviation of accuracy rates of this microexpression type under six backgrounds (except the same expression grounds as the microexpression) was used as the background effect index of this microexpression type recognition, which was called the fluctuations of the microexpression type recognition [6,8], and it was abbreviated as microexpression SD. Therefore we got two new recapitulative indexes of EMERT. A single sample t test was made for each microexpression M with random level 1/6, and it was found that all were significantly higher than random (ps<0.001). A single sample t test was made for each microexpression SD with random level 0, and it was found that all were significantly higher than random (ps<0.001).
Pearson correlation was made between the two EMERT. It was found that each microexpression M in the first EMERT was significantly positively related to the corresponding one in the second EMERT and the rs (the plural of r, the same below) were high; and that each microexpression SD except surprise SD in the first EMERT was significantly positively related to the corresponding one in the second EMERT.
Pearson correlation was made between the first EMERT and the first JACBART (microexpressions under neutral backgrounds belong to the first EMERT), and it was found that each microexpression M in the first EMERT was significantly positively related to the corresponding microexpression in the first JACBART. The new indexes of the two EMERT and their statistical results were shown in Table 2.

Brain imaging data
Pearson correlation analysis was made between ALFFs of resting-state and microexpression M (see Table 3 and Fig 2). (1) In the eyes-closed resting state, ALFFs in the frontal lobe, insula, cingulate cortex, hippocampal, caudate nucleus, thalamus and vermis were significantly correlated with some microexpression M. (2) In the eyes-open resting state, ALFFs in the frontal lobe, insula, cingulate cortex, hippocampus, parietal lobe, caudate nucleus, thalamus, temporal lobe, cerebellum and vermis were significantly correlated with some microexpression M. (3) In the ALFFs-difference of eyes-open minus eyes-closed resting-states, ALFFs-difference in the frontal lobe, insula, amygdala, occipital lobe, fusiform, temporal lobe, cerebellum and vermis were significantly correlated with some microexpression M.

The EMERT had good reliability and validity
In the current study, we used white, black and yellow models' expressions as microexpressions and backgrounds to improve the materials ecological validity of EMERT, and used two new recapitulative indexes such as microexpression M and microexpression SD.
Almost all the microexpression recognition accuracies and all the microexpression Ms were significantly higher than random, which means that the participants could effectively identify almost all the microexpressions. Only fear under surprise was not significantly higher than random, which might because the fear microexpression and the surprise backgrounds had similar face muscle status. All the microexpression SDs were significantly higher than random, which means that each microexpression type had significantly background effect [6,8].
Each microexpression M in the first EMERT was significantly positively related to the corresponding one in the second EMERT, and the rs were high. Each microexpression SD except surprise SD in the first EMERT was significantly positively related to the corresponding one in the second EMERT, which showed that the EMERT had good retest reliability. Each microexpression M in the first EMERT was significantly positively related to the corresponding microexpression in the first JACBART, which showed that the EMERT had good criterion validity [6,8].

The relevant brain areas of microexpression M in EMERT
In the eyes-closed resting state, ALFFs in the frontal lobe, insula, cingulate cortex, hippocampal, caudate nucleus, thalamus and vermis were significantly correlated with some microexpression M, of which the insula, cingulate cortex, hippocampal and thalamus were common brain areas of expression recognition [21], the frontal lobe, insula, cingulate cortex, hippocampal and thalamus might be responsible for microexpressions consciousness and attention [22,23], and the caudate nucleus and vermis might be responsible for the change from expression backgrounds to microexpression [10], which of course need further research to determine, the same as below.
In the eyes-open resting state, ALFFs in the frontal lobe, insula, cingulate cortex, hippocampus, parietal lobe, caudate nucleus, thalamus, temporal lobe, cerebellum and vermis were  significantly correlated with some microexpression M, of which the insula, cingulate cortex, hippocampus, thalamus and temporal lobe were common brain areas of expression recognition, the frontal lobe, parietal lobe, hippocampus, insula, cingulate, hippocampus, thalamus and temporal lobe might be responsible for microexpressions consciousness and attention. The caudate nucleus, cerebellum and vermis might be responsible for the change from expression backgrounds to microexpression [24]. It can be seen that microexpression M was significantly correlated with similar brain areas in both eyes-closed and eyes-open restingstates. Still, in the eyes-open resting state, there were more relevant brain areas such as parietal lobe and temporal lobe.
In difference of eyes-open minus eyes-closed resting-states, ALFFs-difference in the frontal lobe, insula, amygdala, occipital lobe, fusiform, temporal lobe, cerebellum and vermis were significantly correlated with some microexpression M, of which the insula, amygdala, occipital lobe, fusiform and temporal lobe were common expression recognition brain areas, the frontal lobe, insula and temporal lobe might be responsible for microexpressions consciousness and attention. The cerebellum and vermis might be responsible for the change from expression backgrounds to microexpression. It can be seen that there were some similar relevant brain areas in the ALFFs-difference as in eyes-closed and eyes-open resting-states. Still, in the ALFFs-difference, there were new relevan brain areas such as the amygdala, occipital lobe and fusiform.
It was found that ALFFs in both eyes-closed and eyes-open resting-states and ALFFs-difference could predict microexpression M of EMERT. Their predictability was similar, but there were also differences. According to the relevant brain areas and logic, there might be three cognitive processes in ecological microexpression recognition, such as the expression recognition, microexpressions consciousness and attention, and the change from expression background to microexpression. We need to explore whether and when each of them occurs and whether some other cognitive processes exist by developing new behavioural measurement methods to separate them and by task-state fMRI and ERP in the future. Nakano et al. [15,16] found that transition from eyes-closed to eyes-open was from internal feeling to external stimulus processing. However, no study has taken the ALFFs-difference as a quantitative sensitivity index

PLOS ONE
The relevant resting-state brain activity of EMERT from internal feeling to external stimulus, and no study has investigated its psychological significance. In the current study, we defined the ALFFs-difference as the quantitative sensitivity index from internal feeling to external stimulus, and it was found that the ALFFs-difference could predict EMERT, indicating psychological significance.
Shen [9] found that the brain area responsible for classical microexpression recognition was in the left frontal lobe, while the brain area responsible for expression recognition was in the right frontal lobe. In the current study, it was found that for EMERT, both the left and right frontal lobes and more brain areas were involved. Zhang [10] found that for anger and neutral microexpressions, activation of the inferior parietal lobule was induced more in the negative expression backgrounds than in the neutral expression backgrounds, while activation of the right precuneus was induced more in the positive expression backgrounds than in the neutral expression backgrounds. For happiness microexpressions, activation of the parahippocampal gyrus was induced more in the positive backgrounds. The current study also found that these brain areas except the right precuneus were involved in EMERT, and more brain areas were involved. There might be three reasons for this difference: (1) The EMERT in the current study was more comprehensive and ecological, and there was more background effect.
(2) The correlation analysis of resting-state was adopted in the current study. Still, the comparative analysis of task-states has been used in previous studies either between microexpressions and expressions or among different microexpressions, therefore many common brain areas either of microexpressions and expressions or of different microexpressions might be ignored by statistics. (3) We detected the relevant brain areas of new recapitulative index of EMERT such as microexpression M and did not pay attention to details. Zhang et al. [6] established EMERT but did not investigate the relevant brain areas. In the current study, the relevant brain areas of EMERT were comprehensively investigated. Of course, further researches are needed to determine which function brain areas are responsible for.

The relevant brain areas of microexpression SD in EMERT
In the eyes-closed resting-state, ALFFs in the frontal lobe, insula, cingulate cortex, occipital lobe, parietal lobe, precuneus, caudate lobe, putamen lobe, thalamus, temporal lobe, cerebellum and vermis were significantly correlated with some microexpression SD. In the eyes-open restingstate, ALFFs in the frontal lobe, insula, cingulate cortex, cuneus, occipital lobe, parietal lobe, precuneus, caudate lobe, putamen lobe, thalamus, temporal lobe, cerebellum and vermis were significantly correlated with some microexpression SD. It can be seen that microexpression SD was significantly associated with similar brain areas in both eyes-closed and eyes-open resting-states.
In the ALFFs-difference of eyes-open minus eyes-closed resting-states, ALFFs-difference in the frontal lobe, insula, cingulate cortex, amygdala, fusiform, occipital lobe, parietal lobe, temporal lobe, cerebellum and vermis were significantly correlated with some microexpression SD. It can be seen that there were many similar relevant brain areas in the ALFFs-difference as in eyes-closed and eyes-open resting-states. Still, in the ALFFs-difference, there were new relevant brain areas such as amygdala and fusiform.
It was found that ALFFs in both eyes-closed and eyes-open resting-states and ALFFs-difference could predict microexpression SD. Their predictability was similar, but there were also differences. In EMERT, Zhang et al. [6] and Yin, Tian, Hua, Zhang, & Liu [8] defined the microexpression SD as the fluctuation of the ecological micro expression to quantify the background effect. Still, they did not investigate the relevant brain areas. The current study comprehensively investigated the relevant brain areas involved in the quantification of the background effect. Of course, further researches are needed to determine which function brain areas are responsible for.

The similarities and differences of the relevant brain areas of microexpression M and SD
The microexpression M is the index of a microexpression type recognition. The microexpression SD is the background effect index of this microexpression type recognition [6,8]. The former is a kind of ability, but the latter is the degree that this ability changes in different contexts, which, in turn, can be thought of as the stability of this ability. Therefore, there should be similarities and differences in brain mechanisms between them.
In the eyes-closed resting state, ALFFs in the frontal lobe, insula, cingulate cortex, caudate nucleus, thalamus and vermis were significantly correlated with both some microexpression M and some microexpression SD, which indicates they need emotional perception and feeling. But ALFFs in hippocampal were only significantly correlated with both some microexpression M, which indicates that the microexpression type recognition ability need memory more; and ALFFs in the occipital lobe, parietal lobe, precuneus, putamen lobe, temporal lobe and cerebellum were only significantly correlated with some microexpression SD, which indicates that the stability of the microexpression type recognition ability need cognitive control, consciousness and motion more.
In the eyes-open resting state, ALFFs in the frontal lobe, insula, cingulate cortex, parietal lobe, caudate nucleus, thalamus, temporal lobe, cerebellum and vermis were significantly correlated with both some microexpression M and some microexpression SD, which indicates they need emotional perception and feeling. But ALFFs in hippocampal were only significantly correlated with both some microexpression M, which indicates that the microexpression type recognition ability need memory more; and ALFFs cuneus, occipital lobe, precuneus and putamen lobe were significantly correlated with some microexpression SD, which indicates that the stability of the microexpression type recognition ability need visual, consciousness and motion more.
In difference of eyes-open minus eyes-closed resting-states, ALFFs-difference in the frontal lobe, insula, amygdala, occipital lobe, fusiform, temporal lobe, cerebellum and vermis were significantly correlated with both some microexpression M and some microexpression SD, which indicates they need emotional perception and feeling. But ALFFs-difference in the cingulate cortex and parietal lobe was only significantly correlated with some microexpression SD, which indicates that the stability of the microexpression type recognition ability needs cognitive control more.
Taken together, both microexpression M and microexpression SD need emotional perception and feeling, but the former need memory more, and the latter need cognitive control and consciousness more. Of course, a certain ability requires its related brain areas and memory, but in addition to the brain areas associated with this ability, the stability of this ability requires cognitive control and consciousness. The similarities and differences in brain mechanisms of microexpression M and SD are logical. All these relevant brain areas can be trained to enhance ecological microexpression recognition ability.

Conclusion
The current study used white, black and yellow models' expressions as microexpressions and backgrounds to improve the materials ecological validity of EMERT. It used eyes-closed and eyes-open resting-state fMRI to detect relevant resting-state brain activity of EMERT. The result showed: 1. Two new recapitulative indexes of EMERT were adopted, such as microexpression M and microexpression SD. The participants could effectively identify almost all the microexpressions, and each microexpression type had a significantly background effect. The EMERT had good retest reliability and calibration validity.