Semantic processing of sentences in gaze behaviour and in brain activity

The semantic processing of sentences involves not only the processing of the individual words; the relationships between words also need to be considered to grasp the meaning contained in the sentences. It has been proposed that forming a representation of a sentence proceeds through three stages: 1) building a local phrase structure, 2) assigning syntactic and thematic and semantic relations and 3) integration to semantic representation [3]. Of particular interest here are stages 2 and 3.

Several factors affect eye movements during semantic processing. Predictability, for example, is a major determinant of fixation duration during reading [4]. In addition, the thematic relationships between words modulate the gaze behaviour. For example, when a word clearly violates the sentence context, it will result in a longer first-pass gaze duration (sum of fixation durations during the first pass of the target word) [5,6]. In earlier research merely implausible or improbable word in relation to thematic rules was not reflected in the first-pass measures but rather in the total fixation duration (sum of all fixation durations during the trial allocated to the target word), which indicates that implausibility is processed at a later stage than outright theme violation in the form of a semantic anomaly [5]. However more recent research with larger sample sizes and more suitable statistical approaches has shown that plausibility is capable of modulating first pass measures[7,8] and this includes influences from parafoveal processing[9,10].

A systematic way of inducing ERP effects reflecting semantic processing is to set up a semantic context with a sentence and then disrupt it by inserting a word that is anomalous with the established context. Consequently, this anomalous word violates the set of thematic rules established by the other words and gives rise to N400, a centroparietal brain response with negative polarity typically reaching its maximum amplitude in RSVP experiments at about 400ms post onset of the anomalous word [11]. N400 is thought to result from difficulty in integrating the meaning of the violating word with the context of the preceding sentence [11] or to reflect indexing of the goodness of fit of the currently processed word with the context of the sentence [12]. Both of these suggested functions are similar to stage 2 of the model described above [3]. Other factors besides congruency have also been shown to influence N400. For example word frequency affects the amplitude of N400 in low-context situations where prior context has not been defined, for example, at the beginning of the sentence. Thus, in order to avoid confounds from word frequency and to maximise semantic incongruity, the manipulation should be placed at the end of the sentence. Further, the presentation parameters of RSVP have direct consequences to N400. Faster presentation rates diminish amplitudes and shorten the latency of N400 [13], which emphasises the uncertainty regarding the ecological validity of RSVP experiments with slow presentation rates.

Generally, N400 experiments have manipulated only the semantic aspects without paying attention to low-level features, such as word length or visuo-spatial similarities of anomalous and plausible words. A fairly recent experiment [14] tested whether the deviation of a single letter from a probable word was sufficient to cause N400 –i.e. does the anomalous orthographic word neighbour of the predicted word cause a similar N400 as an anomalous word without a similar word body? N400 was found to be diminished for anomalous word neighbours compared to anomalous words without orthographic resemblance to the correct word, which was interpreted to indicate facilitated integration of words with similar word bodies. This effect was also present for pseudoword neighbours containing no semantic information, thus providing strong evidence that a set of orthographic features was pre-activated by the preceding context. As these findings were found in a fairly slow-paced (250ms stimulus duration + 250ms blank screen) RSVP design, the question remains as to whether a one-letter deviation from a plausible word would elicit similar effects during free reading.

Recently P600, a parietal positive ERP reaching maximal amplitude at 600ms that has been classically defined as a response to syntax violations [15], has also been shown to be elicited by semantic anomalies [3,16,17]. Similarly to N400, the P600 effects seem to diminish with faster presentation rates [18]. Curiously, P600 has been found to be related to regressive eye movements in an FRP study investigating the effects of free reading on brain activity during reading comprehension [1]. It was also found to be absent in the RSVP version of the experiment. This led the authors to conclude that coupling between P600 and regressions indicates attempts to recover or re-organise confusing content. It has also been argued that rather than encountering anomalies in a specific aspect of language, such as semantics, syntax or grammar, P600 reflects an integration phase or combinatory reprocessing of the elements of the sentence [3,19–21] or even a conscious-level perception of the whole sentence-level semantic anomaly [12].

The neural generators of N400 seem to be rather widespread [22,23]. Studies of individuals with brain lesions, studies with intracranial recordings and studies with magnetoencephalographic data have implicated contributions of the left and right temporal lobe, with a left hemispheric predominance [24]. The larger left than right hemispheric response can also be observed in a study comparing typically reading adults with dyslexic adults [25]. A recent study using a beamformer analysis of the magnetic equivalents of N400 (N400m) and P600 (P600m) found left superior temporal and posterior frontal regions to underlie N400m and distributed activation of bilateral frontal, posterior temporal and parietal regions P600m [26]. Further, recordings with functional magnetic resonance imaging (fMRI) suggest that the processing of semantic anomalies occurs in mid portions of the superior temporal region and insular cortex, both bilaterally, whereas syntactic processing is associated with the anterior portion of the left superior temporal sulcus and left posterior frontal operculum [27]. In positron emission tomography (PET) studies, the role of the angular gyrus has also been implicated in semantic processing [28,29]. It needs to be noted here that different measuring and analysis methods as well as different experimental designs have different sensitivities and can thus lead to quite distinct location results. Different results do not render conclusions right or wrong but rather reflect different aspects of the process, which are reflected in different signals.

In our current study, we investigated semantic processing and how orthographic similarity between anomalous and plausible words affects semantic processing during free reading in 12–13-year-old children using FRPs (co-registered eye tracking and EEG time-locked to fixation onset). We employed a free sentence reading paradigm to examine the effects of a semantic anomaly in two conditions: unrelated anomalous words and anomalous orthographic word neighbours of plausible words as the target words appearing at the end of the sentence. We were especially interested in whether a deviation of a single letter from a plausible word would elicit complementary effects on gaze behaviour in relation to brain activation during free reading. Based on previous studies [14], the anomalous orthographic word neighbours were expected to produce reduced N400 compared to the unrelated anomalous words. As the fixation duration and N400 effects are associated with each other [2] and react to the same manipulations, it was expected that the fixation duration effect of the anomalous word neighbour would be similarly attenuated (similar to the N400 effect) when compared to unrelated anomalous words. Further, since P600 is thought to reflect an integration phase in sentence comprehension [3], we expected that unrelated anomalous words would generate stronger P600 responses than anomalous neighbours of a plausible word. We also considered the possibility that the semantic responses would be delayed in the case of the anomalous word neighbours of the plausible words.

Methodological considerations when studying brain activity during reading

RSVP (rapid serial visual presentation) is most common stimulus presentation procedure for studying brain function during reading. In RSVP, each word in a sentence (or other text material) is usually presented on the screen individually one word at a time in a sequence to the participant’s fovea, and the brain activity is time-locked to each word (screen) onset. Arguably, the elicitation of ERPs by individual words makes RSVP an attractive approach to study reading-related brain processes. Indeed experiments utilising the RSVP technique have formed the foundation for the knowledge about brain function during word recognition and higher-level semantic processing. However, the way in which RSVP oversimplifies reading into a static linear process decreases its ecological validity considerably. From eye-movement studies we know that reading consists of multiple dynamic processes, such as visual intake during fixations, the lengths of which are modulated by the processing demands, skipping of words because of their predictability and visual characteristics and frequent gaze regresses to previous parts of the text to re-read material that are ambiguous or hard to understand [4]. Generally, in RSVP experiments presentation times are static, and there is no chance of skipping words or returning back in the text. However, during normal text reading, the parafoveal information of the words surrounding the fixated words is also processed to some extent [4], which is not possible in the standard implementation of RSVP. Alternative versions of RSVP have been proposed to counteract these issues: self-paced RSVP, where [30] the presentation rate of the words is controlled by the participant, and RSVP with flankers [31], where the sentence ‘slides’ over the participants’ foveal field of vision. However, these modified versions also have some problems. Self-paced RSVP arguably requires conscious monitoring of the presentation rate by the participant while eye movements during reading do not have such requirement, as they are largely automatised. Meanwhile, RSVP with flankers requires suppression of the eye movements towards the flankers, which is a cognitive requirement that is not present during natural reading situations. Furthermore it has been shown that pre-saccadic attention prepares the visual system for the next retinal input [32,33]. As pre-saccadic attention is tightly related to impending saccade execution, its role cannot be fulfilled if saccades are not made. Recent findings illustrate that volition in attention allocation and saccade generation towards words facilitate word recognition in a way that is not present in RSVP [34]

Co-registering EEG and eye tracking represent an ecologically valid alternative for studying reading. This is achieved through analysing the EEG signal based on selected gaze behaviours, for example, fixation on a target word. The combination of co-registration and a behaviour-based analysis approach allows the participant to proceed in the reading task at his or her own pace, with the ability to regress back in the text as well as to skip words. From this participant-initiated reading behaviour we can extract FRPs (fixation-related potentials), electrical brain responses that are very similar to visually evoked potentials from more traditional ERP experiments [34,35]. Indeed, studies comparing the RSVP and FRP methodologies have shown that brain activity during naturalistic FRP experiments differs drastically from RSVP equivalents [1,34], which further emphasises the need for brain research during reading with naturalistic experimental designs.

When performing reading studies with the FRP method, two things need to be considered. First, the ocular artefacts and extra-ocular muscle artefacts feature prominently in the co-registered EEG signal [35]. Generally, these artefacts are stronger than the signal of interest and thus need to be removed from the data. This can be achieved with blind-source separation methods such as ICA and temporal selection methods [36,37]. Temporal selection exploits the dependence between the ocular artefact independent component time course and the time course of the optical eye-movement record [37]. Recent research has shown that artefact signals associated with FRPs can be effectively managed [1,35,38,39]. Second, a more complex issue relates to the fact that during reading fixations occur in relatively fast succession, which results in additional spatiotemporal mixing of the scalp-recorded EEG/ERP signals, especially in the parts of the signal that exceed the duration of the fixation that is used as the time-locking event [40]. As cognitive processes have an evident influence on eye movements [4], changes in gaze pattern due to differences in conditions reflect the mixing of brain activity from different sources in the latter parts of the averaged epoch [40]. We propose to untangle this issue by analysing the time courses of these underlying activity patterns by separating them with blind-source separation methods, such as ICA [36,41].

Even though identifying and removing artefactual signals with ICA from the signal of interest is the typical use in contemporary EEG analysis [42], ICA can also be applied to separate brain signals from each other. This can be of particular interest for late latency components, which typically have multiple neural sources, or when there are multiple overlapping sensory responses. The underlying assumptions of ICA place certain restrictions on the nature of the sources detected. First, a signal arising from a spatial source is assumed to be temporally independent from other spatial sources [36,41]. Second, the spatial source is assumed to be in a fixed location throughout the duration of the measurement of the data of interest [43]. As independent components are defined as spatially fixed sources that change in activity through time, several interesting possibilities arise. First, there is the option to forward project an independent component’s activity to a scalp activity and localise it [43,44]. Second, data from several experiments from the same subject can be inserted into the same decomposition to determine whether different experiments share psychological processes and underlying neural mechanisms [42]. Third, a decomposition analysis (run) can also include data from multiple subjects, and thus the obtained group solutions can be applied to all of the subjects [45]. Hence, the application of ICA beyond artefact cleaning is a promising approach for FRP experiments on natural reading, as the stationarity of sources provides a solution to the ever-present spatio-temporal overlap of activity from previous and consequent fixations and their potential confounds to the scalp signal. Running ICA across a sample of participants also reduces measurement-related error variation (e.g. due to random individual differences in arousal state) and should thus improve the results of ICA decomposition when looking for shared brain activity sources across the sample. In the current study, we employed a group ICA procedure to disentangle spatially overlapping processes during sentence reading and when encountering semantic anomalies in the sentences.

Participants

A total of 66 typically reading elementary school students in grade 6 (from 12.0 to 13.5 years; 36 female) were recruited as part of the eSeek!–Internet and Learning Difficulties: A Multidisciplinary Approach for Understanding Reading in New Media–project. In agreement with the Declaration of Helsinki, the study was approved by the Ethical committee of the University of Jyväskylä. Written consent was acquired from the parents of the participants, and the children had the opportunity to discontinue the experiment at any time. All participants reported normal or corrected vision, no history of learning difficulties or neurological abnormalities and were rated higher than the weakest 13% of the whole eSeek population (whole norming population being 542 children of the same age group) in reading fluency. Reading fluency was estimated with three tests and reduced to a single factor with principal axis factoring with PROMAX rotation using the IBM SPSS 24 statistics programme (IBM Inc.). The three tests were the Word Identification Test–a subtest of the standardised Finnish reading test ALLU [46], the Word Chain Test [47] and the Oral Pseudoword Reading Test [48].

Apparatus

The experiment was administered using Experiment Builder (1.10.1630) software running on a Dell Precision T5500 workstation. Eye movements were recorded with a table-mounted Eyelink 1000 eye tracker with a 2000 Hz upgrade (SR Research Ltd). Both eyes were recorded at 1000 Hz. The EEG was recorded with a NeurOne amplifier (Bittium) at a 1000 Hz sampling rate with a 128 channel electrode net (Electrical Geodesics Inc.) using Ag/AgCl electrodes. The synchrony between the eye movements and EEG measures was established with a combination of triggering Ethernet messages and transistor to transistor logic (TTL) pulses, both originating from the workstation running the experiment. The stability of the synchronisation between the eye-tracker recording and EEG was checked by comparing the time differences in the trial onset and offset messages in both data streams. The participants leaned on a chinrest while their eye movements were recorded, with a distance of 60 cm from the participants’ eyes, and the EEG was recorded simultaneously. The participants’ responses were recorded into the EEG event stream and into an individual response file on the workstation running the experiment. The experiments were carried out in a dimly lit and soundproofed room at the laboratory facilities of the University of Jyväskylä.

Stimulus procedure

Before each trial, a black dot appeared on the left side of the screen at the vertical level of the participant’s eyes, and the participant was instructed to remain fixated on it while the experimenter approved the fixation. The black dots also served as confirmation for the validity of the calibration. If the fixation on the dot differed from the calibration by more than 1 degree, the experiment software alerted the experimenter, and the calibration was redone. After the experimenter accepted the fixation, the fixation dot disappeared, and the sentence appeared. The participants were instructed to read the sentence as quickly as possible and then judge whether it was sensible or not by pressing the left button for ‘yes’ and the right for ‘no’ on a two-button response box using the right index and middle fingers. There was no time limit for responding, and the sentence disappeared only after the response. The experiment was divided into four blocks, between which the eye tracker was recalibrated and the quality of the EEG maintained.

Materials

The stimuli consisted of 200 sentences with a median length of six words (ranging from 5 to 9). On the screen, one letter subtended 0.4 degrees visual angle. The sentences were divided into three categories: 100 plausible sentences, 50 sentences where the target word was severely anomalous to preceding sentence context and 50 sentences where the target word was severely anomalous to the preceding context but was an orthographic word neighbour of a plausible word. The categories were balanced in this way to make task response (yes/no) probability 50%. Examples of the sentences are presented in Fig 1. The frequency of the target words and the previous words was controlled for (2x3 repeated measures ANOVA: previous word vs. target word p>0.05, condition p>0.05, condition * previous word vs. target word p>0.05). The frequencies of the target and previous words (Table 1) were extracted from a newspaper corpus [49]. A norming study was completed with an independent sample (N = 10) to estimate the cloze probability of the target words and plausibility of the whole sentences. Also the plausible ending of the sentences in the anomalous word neighbour condition was included in the norming study. Cloze was estimated with a standard cloze task while plausibility was estimated using a seven-point Likert-type scale, where 1 corresponded to ‘not at all plausible’ and 7 to ‘highly plausible’. Unrelated anomalous words did not significantly differ from anomalous word neighbours in either cloze or plausibility (independent samples t-test, p>0.05). The plausible condition did not significantly differ from the plausible ending of anomalous word neighbour sentence in cloze or plausibility (p>0.05). These norms are presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Example stimuli.

The part of the sentence that deviates from the context is highlighted in red. Sentence translations: 1) plausible–‘Main attraction of the museum was the fragile mummy’; 2) unrelated anomalous–‘For building up muscle mass one needs an intense ring’; 3) anomalous word neighbour–‘During the night the cat caught a slow coal’ (‘hiili’, coal, being a neighbour of ‘hiiri’, a mouse).

https://doi.org/10.1371/journal.pone.0209741.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Target word and pre-target word frequencies and sentence characteristics.

https://doi.org/10.1371/journal.pone.0209741.t001

EEG and eye-tracking data preprocessing

Data were preprocessed using Eeglab 13.3.2 [36] with the EYE-EEG extension [35]. The saccades were detected from the gaze location data with a median velocity-based algorithm [50], using 6 standard deviations from the median velocity as a threshold for a saccade (minimum duration of 4ms; if the two saccades were less than 50ms apart from each other, they were merged into a single saccade).

EEG data were off-line filtered, with a high-pass filter of 0.5 Hz and 20 Hz as the low-pass filter. Electric manifestations of eye movements (ocular artefacts) were modelled with ICA, with a PCA (principal component analysis) reduction of the 128 channels to 100 principal components prior to the ICA training. The components were selected to correspond with the optical recording of eye movements using the temporal covariance criterion [37] of 1.1. Selected independent components (ICs) were pruned out of the EEG data.

After the ocular artefact pruning, the EEG epochs with the fixations of interest were selected. These included the first fixation on the target word and the previous fixation. If the previous fixation was not on the word preceding the target word, the whole trial was discarded to make the baseline segments of the FRP similar. Epochs for averaging were -100ms to 900ms, time-locked to the fixation onset. Trials with both correct and incorrect answers were kept. Fixations and their corresponding epochs were discarded if the EEG during the averaging epoch differed by more than 5 standard deviations from the mean on any channel or if the trial ended within the 900ms after first fixation on the target word to avoid confounding activity from trial offset.

Segmented single trial data (-100ms to 900ms) were subjected to a second run of ICA to determine shared fixation-related components. Prior to ICA, the data for each subject were standardised (basic z-score conversion) so that individuals with strong voltages would not drive the ICA decomposition. The data were downsampled to 250 Hz to make computational requirements feasible, and the group matrix was whitened to 30 principal components with PCA. Extended Infomax, an ICA algorithm sensitive to sub-Gaussian distributions [51], was used because it has been shown to produce good results with non-simulated group EEG data [52]. The resulting group-level ICA weights were then applied to the 1000 Hz individual data, and the resulting IC activities were used for statistical inference. It is important to note here that our application of group-level ICA does not attempt to reconstruct topographies of individual subject’s ICs but applies the group-level weights obtained from the population-mixing matrix directly to the individual subject data. Ten of the 30 components were determined by visual inspection to be such that they could be generated by a cortical source by displaying a dipolar pattern. Pattern was judged to be dipolar if it had either 1-pole uniform field on several adjacent channels or 2-pole field on several adjacent channels and nearby polarity reversal. Components judged to be noise were pruned out of the data if they displayed known artefact characteristics (eye movements, single channel pop-out artefacts). Topographies of all components derived with the group ICA are presented in Fig 2. Table 2 contains the PVAF (percentage of variance accounted for) values of each component calculated with the EEGLAB [36] function compvar, PVAF is a measure that describes the amount of variance an independent component explains.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The group-level ICA estimated spatial filters.

Components 1, 2, 3, 4, 5, 6, 7, 9, 11 and 12 were retained in the data because they were judged to have a dipolar field structure and thus were likely be generated by brain tissue. Components 8 and 10 were pruned out of the data since they were judged to be likely to have been caused by eye movements. Components 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29 and 30 were pruned out of the data because they were judged to be random pop-out artefacts and mechanical artefacts mostly constrained to few channels.

https://doi.org/10.1371/journal.pone.0209741.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive values of the percentage of variance accounted for (PVAF) by the group ICA components.

https://doi.org/10.1371/journal.pone.0209741.t002

The average quantity of fixations and the corresponding epochs for averaging for each condition were as follows: plausible 83, unrelated anomalous 41 and word neighbour anomalous 41 fixations.

Statistical analysis of the eye-tracking and response accuracy behavioural variables

The behavioural variables were analysed in MATLAB. The effects of the conditions on response accuracy were analysed using a Wilcoxon sign-rank test, from which we report Z-values, P-values and effect sizes [53]. Based on the eye-movement variables, the previous fixation duration (PFD), the first fixation duration, gaze duration and immediate re-fixations were analysed. Because trials that were ended with a participant response within 900ms of the first fixation onset were discarded, the re-fixation measure here is a binary measure of regression to earlier parts of the sentence vs. immediate re-fixation within the target word. The eye-movement variables were analysed with the linear mixed effects (LME) models (for FFD and GD) and generalized mixed effects (GLME) models (for refixations), which is the standard procedure in eye-movement research to control for random effects from the individual subjects and items (fitlme & fitglme functions of the MATLAB statistics and machine learning toolbox). In the LME results, we use the plausible (PLA) condition as a reference condition, against which we compare unrelated anomalous (URA) and anomalous word neighbour (AWN). The estimated random structure was simple intercept structure for trials and subjects. Possible differences between URA and AWN are explored by comparing whether their effect parameters are different (coefTest function of the MATLAB statistics and machine learning toolbox). The mean values of the eye-movement variables are presented in Table 3 and also in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Behavioural measures.

Bar graph presentation of the behavioural measures; exact values are presented in Table 3. A) Percentage of correct answers, B) Fixation durations of the previous fixation (PFD) and the first fixation (FFD) and C) Re-fixation probability (RFP) after the first fixation on the target word in a semantic sensibility judgement task in a sentence reading context of 12- to 13.5-year-old children (N = 66). The error bars denote 1 standard error. PLA = Plausible, URA = Unrelated anomalous and AWN = Anomalous word neighbour.

https://doi.org/10.1371/journal.pone.0209741.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Eye-movement results.

https://doi.org/10.1371/journal.pone.0209741.t003

Statistical analysis of the first fixation time-locked FRPs

FRPs were analysed with nonparametric cluster-based pairwise permutation statistics in Besa statistics 2.0 (for the method description, see[54]), which provides a solution to the multiple comparison problem present in multi-channel EEG recordings. Nonparametric cluster-based permutation tests have two essential steps. First, the desired test (here normal t-test) is run over all channels (when done at the scalp level) and time-points. Then, values over a certain significance (here the typical p<0.05 threshold) are clustered based on clustering criteria. Here, these criteria were temporal adjacency (significant samples were consecutive) and an electrode distance of 3cm (a significant sample in the electrode was part of a cluster if the distance from another significant sample in another electrode was less than 3cm). All t-values of the clusters (each sample, each electrode) are summed to form the cluster test statistic, which is used to estimate test significance. Second, the distribution to estimate this test statistic is generated by randomly re-assigning the condition labels in each average and running the test and clustering procedure again and storing the results in the permutation distribution. When the real observed sum-t probability in contrast to permutation distribution is smaller than 0.05, then the observed cluster is considered to be statistically significant. We used 10,000 permutations to define the permutation distribution. With three conditions, the pairwise condition contrasts for the permutation statistics were as follows: unrelated anomalous vs. plausible, word neighbour anomalous vs. plausible and unrelated anomalous vs. anomalous word neighbour condition. For the sake of simplicity, when we use the terms negative or positive in the results section, they describe the amplitude difference between the first response and the second response in the paired comparison.

Response accuracy

The mean response accuracy (percentage of sentences correctly identified as sensible/insensible) was 83.34% (SD = 7.27) for the plausible condition, 93.4% (SD = 8.82) for the unrelated anomalous condition and 84.86% (SD = 12.02) for the anomalous word neighbour condition. Unrelated anomalous sentences had a significantly higher response accuracy (RA) than plausible (Z = 5.525, p<0.000001, r = 0.481) or word neighbour anomalous (Z = 6.414, p<0.000001, r = 0.558) sentences. There was no difference in response accuracy between plausible and anomalous word neighbour sentences (Z = 1.028, p = 0.303, r = 0.09).

Eye-movement results

Semantic anomaly and word neighbour effects in FRPs.

The first FRPs (Fig 4) showed several statistically significant clusters where the FRPs differed between the conditions (Fig 5). The FRPs differed between unrelated anomalous and plausible conditions, which was evident in a negative cluster in the central frontal scalp area (p < 0.0001) spanning from 138ms to 900ms, which remained stable throughout presence of the cluster, and a positive parietal occipital cluster (p<0.001) with a duration of 180ms to 900ms. The cluster was initially observed in the occipital fringe channels, from which it moved to central parietal locations (roughly 450ms onwards). Anomalous word neighbour and plausible conditions were also significantly different, as there was a fronto-central negative cluster from 97ms to 900ms (p<0.00001) that remained on the frontal sites for the duration of its presence and a positive cluster in the posterior occipital scalp area from 154ms to 900ms (p<0.001 which was initially constrained to occipital fringe channels whence it moved to central parietal sites (roughly 550ms onwards). Furthermore, unrelated anomalous and anomalous word neighbour were significantly different conditions, as we detected one cluster in the time window from 457ms to 698ms, which was positive in the centro-parietal regions (p <0.05).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. FRPs to the target words.

Left Panel: FRPs from the first fixation on the target word on selected channels (E11, E27, E47, E60, E62, E65, E72, E75, E85, E90, E98 and E123 of the GSN-Hydrocel 128-channel cap (Electrical Geodesics Ltd)). Coloured bars under each channel highlight time-points with significant differences that belong to a cluster in the nonparametric permutation test. Right Panel: the associated topographies (all channels) over time in 12- to 13-5-year-old children (N = 66). The zero time-point is the onset of the first fixation on the target word.

https://doi.org/10.1371/journal.pone.0209741.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Statistical differences in the FRP amplitude for the first fixation on the target word.

The scale depicts differences between the first element of the pair to the second element of the pair in values of the test statistic t. Rectangles signify that the sensor displays a statistically significant difference and belongs to a cluster. The colour of the stars within the squares signifies membership in a specific cluster. PLA = Plausible, URA = Unrelated anomalous and AWN = Anomalous word neighbour.

https://doi.org/10.1371/journal.pone.0209741.g005

Statistical analysis of the source activity

Source waveforms were statistically analysed in MATLAB utilising a method similar to the nonparametric cluster-based permutation approach used for the scalp FRPs above [54]. Condition effects were examined with pairwise comparisons using a Wilcoxon sign-rank test sample by sample. The Wilcoxon test was chosen due to the assumption that considering each time-point of the independent component responses to be normally distributed is unreasonable. The multiple comparisons problem was dealt with as follows. Consequent samples (minimum two) with significant differences were defined to be part of the same difference between the signals, and the duration of this difference is what we base our correction on. These ranges consisting of significant differences are later referred to as consequent samples clusters. After this, we employed a randomisation approach, where we flipped the label of the condition within each given subject randomly, re-ran the Wilcoxon sign-rank test sample by sample and stored the length of each cluster that was observed in the permutation distribution. This randomisation step was then repeated 10000 times. If the randomisation iteration resulted in a solution where there was not a single sample with a significant difference, that iteration was entered as zero into the permutation distribution. After the iterations, the permutation distribution contained the distribution of durations of significant differences when the data were randomised. As the null-observation clusters and the true-observation clusters are based on length and contain no negative values, the correction is one-tailed. Those true-observations that were longer than the 95^th percentile of the permutation distribution were accepted for further analysis, and those that were shorter than the 95^th percentile of the permutation were rejected. In addition to reporting the p-value of the cluster in relation to the permutation distribution, we also report Wilcoxon statistics and effect sizes for the mean of the cluster [53].

Source localisation

In order to visualise the sources behind the group level ICs, we forward projected components with significant differences after the correction procedure and then used cortically restricted LORETA implemented in Brain Electrical Source Analysis (BESA, Besa GmbH, Gräfelfing, German) using an age-appropriate template MRI for 12-year-olds [59,60].

Frontal projecting component

Frontal negativity (Fig 5), which dominates the difference between conditions in scalp waveforms, seems to be generated by a single IC component (component 1, Fig 6D). Source localisation analyses with cortically restricted LORETA suggest that it is localised in the right posterior frontal middle gyrus, in close proximity to Brodman areas 8 and 6. In PET [61] and some fMRI experiments [62,63], this location has been implicated as an FEF, an area that is associated with many features related to voluntary eye movements, including maintaining fixation, releasing fixation, triggering of eye movements, saccade amplitude and velocity [56] and inhibition of return [64]. In addition, contributions to higher cognitive functions have been proposed, for example, retaining saccade target locations over a couple of seconds, which is essentially a form of working memory [65], and deployment of both overt and covert spatial attention [66]. Transcranial magnetic stimulation of the left or right site of FEF causes saccade latencies to the contralateral direction be altered [56], and cortical electrical stimulation conjugates eye movements to the contralateral direction [67]. Our manipulation is associated with a difference in the next saccade target (refixation vs. regression), which in most cases is in the earlier parts of the sentence, contralateral of the right FEF. Localisation of the early frontal activity to the right FEF, the pattern of saccade targeting differences and previous findings on associated functions of FEF together strongly suggest that the early frontal activity difference that we observe is associated with the first fixation behavioural effects (FFD, refixation probability) that we observed.

Parietal and temporal projecting components

In the time behaviour of the independent components projecting to the parietal scalp, a biphasic N400/P600-like pattern can be observed. The biphasic N400/P600 pattern appears in two components (components 3 and 6, Fig 6B and 6A, respectively) for both the anomalous conditions. One component (component 4, Fig 6C) only contributes to the P600 effect of the unrelated anomalous condition. As a whole, the parietal brain source structure (in the LORETA analysis) is highly reminiscent of the beamforming results in recent research on semantic and syntactic processing [26], where N400m was localised to left superior temporal and posterior frontal regions and P600m was found to have a plurality of sources, including the bilateral frontal, posterior temporal and parietal regions. In our data, the pattern was similar, with the two components exhibiting a biphasic N400/P600 pattern localised in the left parietal and left posterior temporal areas (component 6) and in the central parietal areas, spreading to the right anterior temporal regions (component 3). Component 4 contributing only to the P600 pattern of the unrelated anomalous condition was localised in the right parietal regions. We will next discuss each of these parts of the parietal pattern in detail.

Component 4 was localised in the right parietal cortex in the approximate area of the right angular gyrus (AG), in the vicinity of Brodman areas 39 and 40 (Fig 5C). AG has been most consistently associated with semantic processing. This is especially the case for the left AG area, although less strong yet equally consistent findings of right AG activation in semantic processing exist as well [68]. Right AG activity has also been found to be associated with cognitive conflict without semantic constraints. This was apparent in research comparing a flanker, stroop and sentence plausibility comprehension tasks, where it was found that the left AG would react to the semantic conflict only while the right AG would respond to all three types of conflict equally [69]. Inhibition of inappropriate responses during no/go tasks has also been associated with the right AG [70]. Thus, in our case, where independent component 4 is contributing only to the P600 pattern of the unrelated anomalous condition, a right AG origin might be related to the capacity of the right AG to process cognitive conflict in several modalities. In the case of the unrelated anomalous condition, the conflict might be gross enough to also be processed in the right AG in the late P600 phase, although the detection of single letter deviation is not strong enough to elicit a significantly different response in the right AG, which is not linguistically specialised.

The independent component 3 topography shows an archetypical centro-parietal N400/P600 complex, and the time course of the component complements this observation (Fig 6B). Unrelated anomalous words significantly modulate both N400 and P600 patterns, whereas anomalous word neighbours modulate only the P600 pattern. The cortical LORETA model shows a pattern of two distinguishable loci of activation: the bilateral superior parietal cortex and right anterior temporal cortex. However, one needs to be particularly cautious when interpreting distributed source model solutions implicating anterior temporal regions, as noise in the frontal and face region scalp sensors tends to migrate to these locations, with a gradual increase of source strength towards the temporal pole being a specific sign of possible artefactual contamination (as stated in discussion with Monto. S. on February 2, 2018, a University researcher at the Centre of Interdisciplinary Brain research at University of Jyväskylä). This is not, however, the case in our data. The locus of the temporal activity is located at the anterior part of the middle temporal gyrus rather than at the anterior pole. If our localisation on this part was actual rather than artefactual, our effects in component 3 could reflect semantic features of the word, as anterior temporal lobes have been argued to act as semantic storage hubs [71]. Anterior temporal lobe contribution to very precise semantic features is particularly evident in semantic dementia, in which bilateral degradation of anterior temporal structures is typical [72,73]. The other areas implicated in our LORETA source localisation solution of component 3 were the bilateral superior parietal areas, which were also implicated in recent research on semantic violations [26].

Component 6 (Fig 6A) contains a highly interesting temporal pattern of activation. First, it displays a contribution to the biphasic N400/P600 for both anomalous conditions. Unlike the other components, component 6 also displays significant differences between the anomalous conditions, and this is essentially because there is delay in the onset of the N400/P600 pattern for the word neighbour anomalous condition. The difference that contributes more negative voltage for the word neighbour anomalous ending than for the plausible ending begins roughly 50ms later than the negative difference in responses between the unrelated anomalous and plausible conditions. Furthermore, the onset of a positive difference between the word neighbour anomalous and plausible conditions occurs roughly 150ms later than the difference between the unrelated anomalous and plausible words. Second, component 6 localises in the LORETA analysis to a broad patch of the left temporal and parietal cortex. Given their heavy involvement in reading and semantic processing in the current experiment, these regions are expected to show activation. For example, the left AG, which is one of the areas implicated in our cortical LORETA model of component 6, is largely involved in all aspects of semantic processing that require concept retrieval and concept integration [68]. Moreover, the left AG has been claimed to provide semantic constraints during language comprehension [74] and to engage when semantic associations are made [75].

The left temporal lobe was also part of the cortical LORETA model of component 6 (Fig 5A). In previous research, the left temporal lobe has been found to be a major contributor to N400 effects in brain lesion, intracranial and MEG studies [24]. These findings converge with fMRI studies, which commonly find middle temporal gyrus activation associated with semantic context manipulations [24]. Overall, the source pattern of component 6 is very consistent with previous research on semantic processing.

In addition, component 7 (Fig 6F) displays a significant difference between URA and PLA around 400ms, and the LORETA model implicates the left temporal lobe as a potential source for activity reflected in the component. However the significance pattern and the waveform of the component make it difficult to interpret how component 7 relates to the pattern observed at component 6. It may well be that the short-lived difference reflects sensitivity to large-scale anomalies but a lack of sensitivity to recognise anomalies of one letter.

The temporal pattern and LORETA solution of the forward-projected independent components converge on an interpretation that the semantic processing of anomalous word neighbours is delayed in relation to the semantic processing of unrelated anomalous words. It could be that the assignment of the semantic role of the anomalous word neighbour becomes delayed because sentence context would favour the semantic role of the plausible word over the anomalous word. The negative contributing cluster is longer for the contrast between the anomalous word neighbour and the plausible word than between the unrelated anomalous word and the plausible word. This could indicate that the assignment of a semantic role or fitting the anomalous word neighbour to the context is more laborious, as the semantic system receives interference from the plausible word that is just one letter away. The delay of onset and longer duration in the earlier negative-contributing phase builds up to a greater delay at the onset of the positive contributing phase. It could be that difficulty in the assignment of the semantic role results in a heavily delayed onset of the integration processes, reflected by the P600 that independent component 6 seems to be contributing to in the later parts of the waveform.

Limitations, boundary conditions and discussion

There are several methodological limitations in the current study. First, to fully control the contributions of parafoveal processing to the process under study, we could have used parafoveal masking of the target word until it was foveated. The hardware that we utilise certainly makes this possible. Forcing parafoveal prediction error with the parafoveal mask would have allowed us to probe whether the pattern that we observed in the frontal scalp and in component 1 was truly a result of a prediction error based on pre-saccadic attention. Then again, it could be argued that including parafoveal masking in our experiment would require us to go through all combinations of mask type with counterbalancing, resulting in nine conditions instead of three, which would have increased the requirements related to the experiment length and population size radically to achieve required signal quality and statistical power. Nevertheless, such a study would be particularly interesting for testing the hypothesis about pre-saccadic attention generating predictions in FEF of the next retinal input during reading.

Second, there is the crucial issue regarding the temporal overlap of activation from previous and subsequent fixations, which effectively creates a situation where one cannot be exactly certain of the degree to which other fixations besides the fixation of interest are contributing to the pattern observed on the scalp. In some studies, this is resolved by a procedure called ‘fixation matching’, where FRPs are matched to eliminate the possible effects of diverging gaze paths. This means that from the condition that has a shorter fixation duration on average, the longer fixations are kept in the FRP, and from the condition that has longer fixations on average, the shorter fixations are selected for the FRP. However, this creates a psychologically biased dataset, as fixation durations and other gaze events are thought to reflect cognitive operations and their processing costs [35]. We thus adopt the position that ‘fixation matching’ compromises the ecological validity of FRP studies, and thus the overlap issue should be addressed with signal processing techniques. We have addressed this issue through a group-level ICA procedure and a separate analysis of the unmixed components. Examination of component 2 (see Fig 6E) highlights the first clear-cut feature that we can observe in the waveforms, the P1/λ-response. This component localises to the primary visual cortex and reflects the initial processing of the low-level features of the foveated stimuli. Thus, if we expect the subsequent fixations to confound the conditions differently, we should observe signs of it in the later parts of the time course of component 2. Of course, this applies mainly to the timing distribution differences between conditions, not to the semantic or other features to which the primary visual cortex is insensitive. From the statistical results of component 2, we see that there is a late difference between the unrelated anomalous and plausible words between 562ms and 774ms. As it is highly unlikely that this difference would reflect semantic processing, we conclude that it is a manifestation of the difference in the fixation timing of the subsequent fixations between conditions. It would take roughly 100ms (from the first positive peak of component 2 to the first positive peak of component 6 and component 4) for effects from this difference in the primary visual cortex to influence the P600 results. All of our results begin prior to this time-point of 660ms. Hence, if the confounding influence is there, it is very minor. If the activity of component 2 would suggest otherwise, the temporal overlap could be further investigated with recently developed signal processing techniques. For example ADJAR (Adjacent Response algorithm)[82,83] or GLM-based linear deconvolution[84] would be suitable (with concrete adjustments to pre-processing pipeline).

Third, we did not rotate sentence frames across conditions, so each target word was always associated with a specific sentence. Therefore, we cannot eliminate the possibility that our results could be affected by random differences between the sentences rather than by our manipulation within these sentences. However, the behavioural results from PFDs argue against this possibility. If the difference between sentences is driving our effects, it should also be present in the PFDs. We do not see any modulation that would signify confounds from the sentences preceding the target words.

Fourth, the target word was sentence final. This is generally not done in eye-movement studies in order to avoid confounds from wrap-up effects that are associated with the final sentence words. However a recent review of ERP studies illustrates that this assumption of a ‘general wrap up’ process at the sentence end is not supported by empirical evidence and suggests that psycholinguistic research has been damaged by the last word target word avoidance dogma [85]. Moreover, given that our focus is on late responses related to semantic processing, where the context provided by the whole sentence is crucial, we think that our placement of the target word is justified. Semantic anomaly manipulation might cause different reactions if placed in earlier parts of the sentence, which would be an interesting research subject.

Fifth, there is confound in the cloze probability of the plausible condition and the plausible word neighbour of the anomalous word neighbour and thus differences between the plausible and two other conditions can partially be due difference in predictability. However the semantic anomaly in the stimuli is clear and thus we consider it to be the driving influence behind the effects. It is important to note that this confound does not impact comparisons between unrelated anomalous and anomalous word neighbour conditions and thus does not impact the main finding of delayed processing of the anomaly in anomalous word neighbour condition.

Finally we studied 12–13.5-year-old children while typical cognitive neuroscience experiments employ university students around 20 years of age. Our sample is a sub-population of the eSeek-project (eSeek! Internet and Learning Difficulties–Multidisciplinary approach for understanding information seeking in new media), and this article essentially describes how school children process semantic information. The pattern of results might change due to developmental effects if this experiment were to be conducted with a representative adult sample.