Information flow through LAG-1

In the category learning experiments modelled, the task comprises several hundred learning trials, each of which has two phases: a response phase and a feedback phase. In the response phase, a fixation cross is first shown in order to center the participant’s gaze, after which a stimulus is presented. The participant then views the features of the stimulus, and makes a response by clicking a button associated with one of the categories. The category response then initiates the feedback phase wherein both the same stimulus and the correct category are presented. When the participant is ready to start the next trial, they click a button, and the fixation cross for the next trial appears. Both of the phases of the trial are self-paced: the stimulus and the correct category can be observed for as long as the subject (or the model) wishes.

The way that LAG-1 processes information during the experiment is depicted in Fig 1. At the start of each trial a stimulus is presented to the model as input to the Visual Field. Stimulus features have particular values, coupled to that particular location on the screen, and the model, because it represents everything spatially, is sensitive to the locations associated with particular combinations of feature values. As spatial competition for attention gets resolved by the lateral dynamics of local-excitation and global-inhibition, gaze will shift to a saccade threshold-crossing peripheral location of the Visual Field, and extract the value of the feature there. This information then passes into the Learning System, activating the appropriate Feature Detectors whose self-sustaining activation acts as a kind of visual working memory. Feature Detectors then propagate their activation through a weight matrix and into the Category Neurons. Recurrent connections from the Category Neurons (propagating backward through a copy of the same weight matrix) pass activation to Feature Expectation Neurons that are associated with those active categories. A critical feature of the system is that activation of these neurons is modulated by a gain mechanism that gives additional boosts of activation to features based on their expected information gain. The calculation of gain is explained in detail in the “Category Neurons” section of the Learning System overview. This top-down information is integrated with both bottom-up information propagating in from the Visual Field, and directly on to the Spatial Attention Field, such that active categories are driving attentional choices, and not just adapting what is salient [39]. Activity on the Spatial Attention Field passes into the Saccade Motor Field, which, because of an inhibitory signal at the current fixation location, can be thought of as a map of possible saccade targets. The pressure to initiate a saccade (and thus target the peak location in the Saccade Motor Field) is controlled by a trio of neurons, and builds up slowly from the beginning of a fixation. Strong activation at the location currently activated can delay saccades, and strong activation at potential saccade target locations can speed them up. Because correct categorization requires information about the value of more than a single feature in these experiments, and because only one feature can be fixated at a time, the model must make a series of eye movements, as humans must, in order to gather the information necessary to make a correct classification. As the model views the features of the stimulus, information is sustained by both the Feature Detectors and the Category Neurons, until activation of a motor response crosses a fixed threshold (the Decision Click Neuron); in the human experiments this would be like a mouse click or button press response.

Once a category response has been chosen, the feedback phase of the trial begins. The correct category enters the Visuospatial Processing system as a button that appears in the periphery of the Visual Field. If viewed, a boosting signal excites the Category Neuron corresponding to the correct category. LAG-1 may continue to look at features of the stimulus, boosting their activation among the Feature Detectors, and countering any memory decay for those values that may have occurred since having last looked at them. During the feedback phase the connections between active features and active categories are strengthened according to simple Hebbian learning. The longer LAG-1 studies the feedback, the more the synaptic weights connecting the active features and categories are strengthened. Once the Click Decision Neuron has again exceeded threshold, the next trial is initiated.

As changes to the associative weights between the Feature Detector Neurons and the Category Neurons accumulate over learning trials, the dynamics of the system begin to change. The strength of an association, reflecting base rates, and the category selectivity of Feature Detectors, reflecting information gain, work together to modify top-down attention. These top-down signals decrease the chance that irrelevant information will be fixated, as well as decreasing fixation durations. Increased activation of the Category Neurons, and faster accessing of the useful information leads to faster reaction times. Increased activation of the Category Neurons also leads to less time being spent during feedback. LAG-1 thus reflects the idea that learning influences attention and attention influences gaze, as well as the inverse relation: what is viewed, and for how long, changes what is learned.

LAG-1 has only three fitted individual subject parameters—shown in green in Fig 1: Learning Rate, Fixation Impatience, and Trial Impatience. The Learning Rate modulates how rapidly the connections between Feature Detectors and categories change per unit time. The Fixation Impatience parameter modulates the growth rate of the pressure to initiate a saccade, which is known to be dynamically adjusted to improve reward rate (c.f., [58, 59]), and the Trial Impatience similarly controls the growth rate of the pressure to make a response (i.e., either by clicking with a category response, and thus initiating the feedback phase, or, during the feedback phase, by clicking to initiate the next trial). These parameters have the effect of preventing the model from perseverating on a difficult decision and ensuring a level of global stability. These impatience parameters could be thought of similarly to “urgency signals” found in the speeded choice literature (e.g., [60, 61]): the idea being to keep the system moving by applying a strong compulsion to move the eye or make a decision after a consistent amount of time has elapsed.

Detailed description of LAG-1 and neurophysiologial context

In this section we provide the formal description of the model, as well as identify research relevant to its structure and functioning. The model structure, illustrated in Fig 1, can be a useful map to the three subsystems, their component parts, and can even help one parse the various model equations (e.g., it can help to compare the inputs in the equation to the inputs shown in the figure). Readers that are less interested in the computational details may still wish to read the overviews provided for the three subsystems to get a feel for how they work. For readers who are interested in the technical details but may be unfamiliar with, or could benefit from a refresher on, the differential equations that form the foundation of the model, we have produced a detailed primer that can be found in the supplementary information: “S1 Appendix: A Primer to Dynamic Neural Field Theory”.

In the supplementary information titled “S2 Appendix: Companion equations for the formal description of LAG-1and neurophysiologial context.”, we provide the equations for each of the components described at a high-level here. Wherever possible we have attempted to give model parameters intuitive labels—for example, the Fixation Neuron is labeled u_x, the Gaze Change Neuron is u_g, and the Feature Detection Neurons are . The output of each neuron and field is bounded by a sigmoid function. Values having been transformed by a sigmoid function f are indicated with a *, e.g., u* = f(u). More information about this is, provided in the supplementary information S1 Appendix. Finally, references to structural components of the model are capitalized, such as the Visual Field, as opposed to generically talking about the visual field of view.

Modelling complex non-linear systems like LAG-1 requires that the correct balance be found between the various forces affecting the activity of a neuron or field. In LAG-1 we use u_c to indicate a simple scaling parameter. There are many of these throughout the model equations as they are a convenient way to adjust the relative magnitudes between forces at play in the equation. That being said, once fields/neurons were stabilized, these scaling parameters were not changed. That is, u_c parameters were not adjusted at all during the data fitting process. We think of these parameters as a mathematical (and practical) necessity, but not critical for understanding the work as a theory; for this reason they are not discussed in detail at each occurrence. Values for scaling parameters used in any model equations throughout this paper can be found in the supplementary information titled “S5 Appendix: Parameter tables and best fits.”

Spatial Attention Field.

Turning knowledge into behaviour at the right place, and at the right time, is a large part of the functions of the priority maps in the parietal cortex [39, 67]. The posterior parietal region of the parietal cortex coordinates a number of motor actions like reaching, grabbing and saccade targeting [68]. Neurons in lateral intraparietal area (LIP) have been shown to vary systematically with the location [69], category [70], reward [71], posterior likelihood [38, 72] and task relevance [73, 74] of individual eye movements. This makes LIP a very important source of attention related projections to more proximate motor control of the eye like the superior colliculus [75, 76].

The Spatial Attention Field in LAG-1 represents changes in attentional priority to locations in a spatiotopic reference frame. The different frames of reference of the Spatial Attention Field and the Visual Field are depicted in Fig 2. Notice that there is no change in the location of features on the Spatial Attention Field between Fig 2A and 2B, despite the shifting of the bottom feature onto the fovea of the Visual Field.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Information processing schema.

Schematic of the relationships between experiment input, the Feature Detection Neurons, Visual Field, and the Spatial Attention Field. A) The fovea is indicated by the dashed grey circle in the center of the Visual Field. B) After an eye movement to the bottom middle feature, the green sensitive Feature Detector is activated and the Spatial Attention Field is boosted at its associated location. The Visual Field is coding spatial information in retinotopic coordinates while the Spatial Attention Field is coding information in spatiotopic coordinates.

https://doi.org/10.1371/journal.pone.0259511.g002

Changes in activation on the Spatial Attention Field are described by similar equations as those used for the Visual Field, in Equation 13.

Learning system overview

LAG-1 learns during the feedback phase of every trial by Hebbian modification of the weights connecting active Feature Detection Neurons with Category Neurons activated by fixating a correct category presented on the screen. If an incorrect guess was made, anti-Hebbian learning simultaneously decouples the active Feature Detection Neurons and Category Neurons. For example, if a blue, yellow and green set of Feature Detection Neurons is co-active with category A during the feedback phase, applying the learning rule will increase the particular weight that connects the presynaptic Feature Detection Neuron and the correct Category Neuron, proportionate to the current strength of the weight. If the correct category was not A in this example, but B, then the blue, yellow, and green Feature Detection Neurons activated by fixating these features, would reduce their connection strength to A, and increase it to B during the feedback phase.

Attending to feedback signals is an essential part of category learning [30, 37]. The category learning experiments we simulate are self-paced: participants click a response button associated with a particular category which brings up the feedback display, and when they are ready they click again to advance to the next trial. Participants can, if they choose, rapidly click to advance to the next trial without spending time looking at the feedback or stimulus. Human participants who have mastered the categories often choose to do this in order to finish faster—LAG-1 too, has this option. The decision to click and advance the experiment to the next phase is represented by a Click Decision Neuron having two thresholds. Crossing the first threshold moves the trial from the response phase to the feedback phase, crossing the second threshold ends the trial.

At the start of the experiment, when LAG-1 knows nothing about the category and feature relationships, impatience and experiment feedback signals provide the bulk of the input needed for the Click Decision Neuron to cross its thresholds. As knowledge about the category structure develops, this knowledge will result in higher activation of particular Category Neurons, which, when combined with experiment feedback signals, make for faster reaction times in the feedback phase. When the category associations are very strong, feedback signals may be unnecessary for crossing the second threshold, and thus, LAG-1, like the human participants, can stop looking at feedback altogether when it has mastered the categories.

In addition to registering the physical attributes of the features being looked at, Feature Detection Neurons act like a visual working memory in that they can sustain their activation even after looking elsewhere. Each of these neurons is connected to the others, as well as to a set of Category Neurons that do no themselves have a sensory connection to the experiment stimulus except through the the Feature Detection Neurons.

Activity propagates from the Category Neurons into the Click Decision Neuron, as well as feeding back through a mirror of the weights that connect the Feature Detection Neurons with the Category Neurons, to activate the Feature Expectation Neurons. These neurons mirror the Feature Detection Neurons in number but project back toward the spatial locations of the Visual and Spatial Attention Fields where the feature values are known to be associated. This is how LAG-1 allows feature-based attention to influence the selection of eye movements [86, 87].

At the top of the visual attention hierarchy in the brain are category sensitive neurons in areas like inferotemporal (IT) and medial temporal lobe (MTL) which get activated by ongoing sensory combinations of visual input and which have reentrant projections to successive layers of the visual cortex, and categorical projections to LIP respectively. Neurons in these regions effectively sensitize visual neurons according to category expectations [70, 88–92]. Neurons in these regions are reported to have a number of attribute-invariant preferences for all kinds of complex categories like cars and celebrities [93, 94]. When activated, neurons with these categorical preferences alter sensory expectancies across modalities and bias the spatial priority of action competitions resolved in the parietal cortex [73, 95].

Because our primary theoretical interest was integrating learning, attention and gaze, and not replicating a fully developed category learning model we chose to use a relatively simple learning system (cf. [96]). The only weights that are adjusted within each experiment are those between the Feature Detection Neurons and the Category Neurons. This means that LAG-1 is, for now, only capable of linear association, limiting its general ability to test rules and learn categories that are not linearly separable [21, 96, 97]. Implementing a one–level–more–detailed category learning system for LAG-1 would require a richer version of visual feature-based categorization inspired by the functions of IT/MTL/hippocampus [93], reentrant relationships between temporal, visual, and parietal cortices [63, 72, 98], prefrontal cortex for choosing among differing rules and strategies [57, 99], and reward modulated Hebbian plasticity for more explicit temporal difference learning [100–102].

Breadth and precision of the modelling

It is important to recognize that our primary interest is in the breadth of LAG-1, rather than its precision. That is, we are less concerned about the precise quantitative magnitiudes than we are in seeing if LAG-1’s behaviours (both choice and oculomotor) change with learning the way people’s behaviours do. Our core argument is that relatively simple connections between learning and attention and gaze underlie a wide variety of findings. In an effort to assess this idea in the simulations, we make several modelling decisions which undoubtedly makes getting good quantitative fits more difficult, but which are truer to our project goals.

First we keep the model flexibility low so that we can unambiguously attribute successful simulations to our core claims. The mathematics that allows the model to fixate in two spatial dimensions and run continuously in time are obviously complex, requiring many equations and many fixed parameter values. However, parameters used in the fitting process—those that can actually change the model’s behaviour to produce better quantitative fits—are very few. Our three free parameters (Learning Rate; Trial Impatience; Fixation Impatience) are considerably fewer than comparable models, e.g., [32].

Second, consistent with our claim that the myriad of behavioural phenomena shown in these task are related, we fit a very broad range of findings. Yet, each additional measure we include constrains the model further, and will impact the quantitative goodness of our fits. We note the breadth of LAG-1’s predictions in this task are unparalleled to our knowledge. Quantitatively comparing models that do not fit the same phenomena is not common practice, and there are clear problems to doing so. For example, we note that for data that a model does not address, the likelihood of that data given the model is zero. Under this construal, the broad model always wins, even if its fits are terrible. Obviously there is something unsatisfactory here, in that a narrower model may still have value, even if it cannot address every finding that a broad model can. Yet, it seems equally unsatisfactory to quantitatively compare overlapping models only on the datasets to which they both apply: essentially placing zero value on fitting more data.

Finally, we reserve a few measures, leaving them out of the fitting process entirely, so that we can verify the accuracy of LAG-1’s natural predictions. In LAG-1, the connection between attention and learning processes is relatively constrained: both the magnitude of the connection strength between features and categories, and the information value of those connections, are quantities that derive purely from the knowledge contained in one set of associative weights. We reasoned that if this account is correct, then, as long as we captured important neurofunctional relationships, other complex phenomena might emerge from the basic constraints in the model without additional coaxing on our part. Demonstrating that LAG-1 can make appropriate predictions about unfitted measures is an important way of further supporting our core argument.

Experimental data being fit

Testing LAG-1 requires an experiment in which both learning and gaze are simultaneously measured, and in which changes in learning lead to changes in gaze. As we know of no other process models that target the relationships between learning and attention and gaze found in category learning in this particular way, instead of direct model comparisons, as is often done when looking at very specific effects or representational difference between models, the approach here is to try and match the human data for all the findings and effects listed in Table 1.

In this section, we fit LAG-1 to data produced in two different studies: [10] and [25]. We describe them in brief, here, with a fuller description of the methods available below. We chose the [25] data as it uses a category structure that has been run many times in our lab with different experimenters, stimuli, instructions, base-rate manipulations, information access costs, and response types [11, 26, 27, 29, 30]. The structure elicits the theoretically important behaviour called stimulus-specific attention, wherein attention is allocated differently based on the properties of the stimulus itself. This finding is important because many existing models have one set of attention weights that change only during response feedback, and so cannot change the allocation based on the features of present stimulus. Eliciting stimulus-specific patterns of eye movements will be an important test of LAG-1.

The second data set is from the [10] study using the “5/4” categories of [17]. This eye tracking study was one of the earliest category learning experiments to use eye movements as a confirmational measure of attention weight predictions. The results suggested that prototype models underestimate attention to the least relevant stimulus dimension in the 5/4 categories (e.g. [32, 104]). Though the 5/4 categories have a long, if contentious (e.g. [105]), history in the study of categorization, using it here allows us to ensure that LAG-1 can model both the appropriate learning and generalization data as well as eye tracking measures. Simulating the [10] experiment also requires that LAG-1 process variable-length experiments with different numbers of features, different arrangements of the features, and different numbers of categories than the [25] study. The reaction time data for the response and feedback sections of the data were not available, and so these measures were not fit in Simulation 2, even though LAG-1 automatically produces reaction times when fit to the remaining data.

In Simulation 1, four behavioural measures were fit: accuracy, fixation count, the probability of fixating irrelevant information on a trial, and the average fixation durations, over the experiment. The data on each of these measures was summarized by an initial intercept and a slope of change in the measure over the experiment, yielding a target vector with eight elements for each human subject. In line with our breadth-focused modelling goals we leave several aspects of the human behaviour to emerge organically from the model. Reaction times, within-trial fixation probabilities, and time spent on features during feedback were left out of the fit function, allowing the fits to these measures to emerge from the model organically.

In Simulation 2, we used the individual subject transfer probabilities (for all 16 stimuli), their learning points (the trial number on which the participant began two consecutive error-free blocks), and a measure of the changes in the allocation of attention to stimulus features of high and low diagnosticity. We were unable to precisely reconstruct all of the individual subject data reported in [10] for the attentional allocation component of the fit measure, so we instead used the reported averages in three measures of attentional allocation. The first two numbers were the average starting and ending fixation counts to the individual features. The third component was the difference between the least informative (Feature 2) and the most informative features (Feature 1) on this measure (∼0.3). The target vector for Simulation 2 thus had 19 elements. Again we allow the model to predict several aspects of the behavioural data with Simulation 2: in particular the within-trial fixation probabilities and the overall fixation proportions.

The experiments record all of the participants’ various responses, when they occurred, and the location of their gaze at all points in time. The simulations of the model are similarly data-rich. Likewise, the attentional and gaze processes in LAG-1 are also spatial and the output of LAG-1 is thus just like the raw data from humans. In fact, we use the same computer code for both humans and simulations to convert the raw data to the aggregated and binned measures presented below. This applies to gaze also: just as with humans, raw gaze location described in x,y coordinates is recorded at a time scale set to approximate a 120hz eye tracker, and this is aggregated into fixations using the same modified gaze dispersion algorithm [106] that was used for the human participants in Simulation 1. Once the final best-fitting simulations are produced (see the supplementary information “S3 Appendix: Fitting procedure” for a full description of the fitting procedures, and supplementary information “S5 Appendix: Parameter tables and best fits” for the best-fitting parameter values) the result is a simulation yoked to each human subject: matching both the experimental situation, and, as well as possible, the impatience and learning rate characteristics of that subject.

Free parameter relationships with model behaviour

Before we cover the results of the simulations, it might be helpful to the reader to get a sense of how the model behaves under various values of the three free parameters. Model output at varying levels of the Learning Rate, Fixation Impatience and Trial Impatience is shown in Fig 3. Each parameter combination used to generate the plots is chosen from the storage table of LAG-1 for Simulation 1 output according to the closest models to low, medium and high levels of one of the parameters when the other two parameters are held to the overall medians observed in the individuals. Fig 3A shows how Learning Rate modulates accuracy by simultaneously plotting LAG-1’s learning curves given low, medium, and high Learning Rates, taken from the best fits of the 42 subjects of Simulation 1. The free parameters map onto behaviour in relatively straightforward ways. As the Learning Rate increases, the connections between the Feature Detection Neurons and Category Neurons change more rapidly, leading to improved accuracy. In Fig 3B, as the Trial Impatience increases, the activation of the Click Decision Neuron grows more rapidly, resulting in faster decisions, and therefore fewer fixations. Finally, as the Fixation Impatience parameter increases, as shown in Fig 3C, activation of the gaze change neuron grows more quickly, resulting in shorter fixations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Isolated parameter influences on model.

In each of the six figures a comparison of the model’s behaviour is presented at three different levels of the parameters. Each set of parameters is chosen from the closest models available when restricted to the range set by the lowest and highest observed in the best fits of individuals in Experiment 1. In each case, one free parameter is compared while the other two parameters are held to the overall medians observed within that same range. A) Model accuracy over three levels, low, medium and high, of Learning Rate ([2.5 × 10⁻⁶, 1.1 × 10⁻⁵, 1.65 × 10⁻⁵]), when Fixation Impatience is 1.7 and Trial Impatience is 1.8. B) Model fixation counts over three levels, low, medium and high, of Trial Impatience ([1.6, 1.8, 2.05]), when Learning Rate is 1.1 × 10⁻⁵ and Fixation Impatience is 1.5. C) Model fixation durations over three levels, low, medium and high, of Fixation Impatience ([1.5, 1.7, 1.9]), when Learning Rate is 1.1 × 10⁻⁵ and Trial Impatience is 1.8. D) Model accuracy over three levels, low, medium and high, of Trial Impatience ([1.6, 1.8, 2.05]), when Fixation Impatience is 1.7 and Learning Rate is 7 × 10⁻⁶. E) Model feature fixation probability over three levels, low, medium and high, of Trial Impatience ([1.6, 1.8, 2.05]), Fixation Impatience is 1.7 and Learning Rate is 1.1 × 10⁻⁵. F) Model feature fixation durations over three levels, low, medium and high, of Learning Rate ([2.5 × 10⁻⁶, 1.1 × 10⁻⁵, 1.65 × 10⁻⁵]), Fixation Impatience is 1.7 and Trial Impatience is 1.8. Shading represents standard deviations of the model for that level of parameters.

https://doi.org/10.1371/journal.pone.0259511.g003

However, complex interactions amongst the parameters are also important, and meaures of accuracy, fixation duration, and fixation count, can be influenced by changes to the other parameters as well. In Fig 3D, the Trial Impatience parameter strongly influences the learning curve by scaling the amount of time spent on feedback. Fixation counts in Fig 3E are influenced by Fixation Impatience because they change the number fixations that can occur within a particular interval of time. Fixation durations may moderately shrink over the course of experiment as learning increases the recurrent input to the Spatial Attention Field, eliciting the onset of saccades more rapidly. But to underscore how complex the predictions owing to these interactions can be, in the final plot, it is only when Fixation Impatience is high, that this decline is noticeable in Fig 3F.

Simulation 1

The eye tracking and category learning data for the first set of simulations comes from the publicly available data set originally reported in [25]. Participants in this experiment were instructed to classify fictional microorganisms, defined as having three critical organelle features (see Fig 4). Each feature could take on two possible values, yielding eight stimuli in total, which were associated with four different categories (A1, A2, B1, B2). Features are arrayed in one of three locations. As is typical for category learning studies, the assignment of image pairs to feature dimensions (feature one, two, and three), and features to locations was roughly counterbalanced across participants. For LAG-1, we simplify the stimuli from complex cell organelles down to three colored features with two possible values, each with its own layer contributing input to the Visual Field (see Fig 2). After making a decision, the participants were given feedback presenting the stimulus again, their own response, and the correct response. Only participants who had at least 70% of their gaze collected in total were included for fitting, and only trials having more than 75% of gaze collected were included in the analyses. Participants had to meet a learning criterion of 24 trials in a row correct in order to be included, leaving 42 individuals in the data set. Of the 480 trials of data available from each subject, we look at only the first 360 in order to reduce simulation time; human behaviour changed very little after that point. More comprehensive details of this experiment can be found in the original paper and on the data repository website.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Simulation 1: Experiment and model stimuli examples.

A) An example stimulus from the response phase (top) and feedback phase (bottom) from [25]. Each feature type subtends approximately 1.7° of visual angle, separated by 10.6°. B) The features as they look to LAG-1 during the response phase (top), and feedback phase (bottom) showing the location of the feedback button. Features are represented as simple color patches (top right). The feedback button provides the correct answer when foveated by the model. Also depicted by a solid black outline top the top right feature is 1 standard deviation of LAG-1’s fovea, given a fixation to the top right hand feature, where the dashed line reflects 2 standard deviations from highest acuity. Stimuli are counterbalanced between subjects, such that features appear at different locations and have differing relevance.

https://doi.org/10.1371/journal.pone.0259511.g004

Stimulus-specific attention—the useful finding that emerges from this category structure—occurs after the participants learn to classify the categories correctly. As can be seen in Table 2, Feature one is always useful, in that it can tell the difference between categories A and B. Features two and three are contingently useful: that is, if Feature one takes a 0 value, then Feature two is important for telling the difference between A1 and A2, while Feature three is no help at all. Likewise, if Feature one has a value of 1, then Feature three can say whether the stimulus is a B1 or B2 but Feature three is irrelevant. Participants can, and do, learn this: the data of within-trial fixation probabilities after learning this category structure show that Feature one is the most likely to be fixated early in the trial and the second most relevant feature for that stimulus, either Feature two or three, is most likely to be fixated later in the trial. Models using a single set of weights cannot shift attention depending on the currently viewed stimulus within a trial, and so cannot account for this finding [11, 16, 107, 108], but some models can account for this at an aggregate feature level [109–111].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Category structure used in [25].

https://doi.org/10.1371/journal.pone.0259511.t002

Simulation 1 results—direct fits

We first look at model performance on measures that we fit directly—that is, some aspect of the measure was a component of the fit function. These basic measures reflect the learning of the categories, and the corresponding changes in the allocation of attention.

Learning curves.

The first measure we look at is classification accuracy. Chance performance in this task is 25%, setting a baseline from which a learner will increase over the course of the experiment. Fig 5A shows the averaged learning curve for both the model and the human participants’ learning performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Simulation 1 model fits.

Each measure is depicted over 360 trials averaged into 24 bins of 15 trials. Color shading represents standard error of the mean accumulated from individuals and population. A) Trial accuracy means and variability are fit well. B) Fixation duration variability is fit well but the final late reduction means are diverge. C) Fixation count means, variability and change are qualitatively fit well. D) Probability of fixating the irrelevant shows roughly the same change and variability over the experiment, however the scale is off by about 25% at all points. E) Reaction time means, variability and change are qualitatively fit well. F) Total time looking at features during feedback has roughly the right change and variability but the scale is off by 50% at all points.

https://doi.org/10.1371/journal.pone.0259511.g005

Simulation 1 results—indirect/predictive fits

In this section we look at three additional measures which were not in the fit function at all. These measures vary from being moderately constrained by the basic attentional behaviours (reaction time is related to fixation durations and fixation counts, for example), to being mostly unconstrained in its fit to fixation durations and counts (e.g., within-trial fixation probabilities, including the important stimulus-specific attention finding).

Within-trial fixation probabilities.

Within-trial feature fixation probabilities have previously been reported as another way of quantifying attentional priority [10, 11]. This measure, used in Fig 6, reveals the general order in which the features are fixated. Note that this measure describes within-trial behaviour, whereas all of the data used in the fitting procedure are at the level of the changes across the experiment; there is nothing in the fitting procedure that rewards approximating this within-trial data. The category structure is designed to elicit different optimal fixation orders for the A categories (see Fig 6A for the human data and Fig 6B for LAG-1), differently than the B categories (Fig 6C for the human data and Fig 6D for LAG-1). The trends observed in the participants when looking at just the features of interest (where at any given point in the trial the total probability sums to unity—fixations outside AOIs are excluded from these data for both humans and simulations), show that for the A categories, there is a Feature one precedence, followed by Feature two, and the irrelevant Feature three dropping in probability over the trial. For the B categories Features two and three are flipped as a result of the change in feature relevance. These same trends are present in LAG-1. As before, the model fixates irrelevant information a bit too often, and thus the differences are less dramatic.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Within-trial feature fixation probabilities.

Within-trial feature fixation probabilities averaged across all post criterion point trials and scaled to 100 data points for each trial and averaged into five bins. A) Human data for A categories and with LAG-1’s comparable performance in B. C) Human data for the B categories with LAG-1 again beside it in D. The three important features of the data are: the primacy of Feature one through most of the trial; the primacy of the alternative feature at the end of the trial; and the change in the identity of the second feature across categories (For Category A stimuli it is Feature two, and for Category B stimuli it is Feature three). LAG-1 qualitatively captures all three findings.

https://doi.org/10.1371/journal.pone.0259511.g006

Individual fits.

The fits shown thus far have been averages across many individual participants and simulations. To give the reader a better sense of how LAG-1 addresses differences across individual participants, we show fits to three individual subjects chosen for their differences in behaviour. Amongst these three participants, there is fast and slow learning, short and long fixaton durations, and many and few fixations per trial. Fig 7 shows data for three human participants and for the several model simulations run for each participant. The individual fits for the remaining participants can be found in the supplementary information beginning at Fig 21, in “S6 Appendix: Individual fit visualizations.”. Overall LAG-1 seems responsive to individual differences, and produces large and small numbers for counts, durations, and accuracy. As far as individual fits, there are no obvious peculiarities, though, for some participants, LAG-1 fails to fit the magnitude of a measure, or its slope. As in the aggregated data, it is clear that LAG-1 is not capturing the variability of human data, though it seems better for accuracy than for the other measures. We kept noise relatively constrained for these processes, but additional noise, for Trial Impatience in particular, seems warranted. We note, though, that some of this variability is extreme and seems more likely to be strategic, or to be related to phenomena like mind-wandering [116] than to be a matter of simple noise.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Fits to individuals.

Data from three individual participants for fixation count, fixation duration, and accuracy. Below each participant are LAG-1’s fits to that participant’s data. Though the fits are not perfect in all cases, the model is showing both more and fewer fixations (counts), shorter and longer fixations (in milliseconds), and slower and faster learning (proportion correct) in accordance with individual differences.

https://doi.org/10.1371/journal.pone.0259511.g007

Simulation 2

The second simulation is of response data and eye movement data from [10]. Participants in this experiment were instructed to classify fictional schematic drawings of insects, defined as having four diagnostic features (see Fig 8). Two of these features were highly diagnostic (and equivalently so) of the category, one was of medium diagnosticity, and one of low diagnosticity. Each feature could take one of two possible values, yielding sixteen stimuli in total. Nine stimuli were shown during training and were associated with two categories, leaving seven for classification during a feedback-free transfer phase (see Table 3). After choosing a category, the participants would get feedback, and see the stimulus again. After 189 trials or 18 correct trials in a row, whichever came sooner, a transfer phase would begin that twice tested all of the stimuli, including the seven previously unseen stimuli. There were 64 participants that met the learning criterion with minimal loss of gaze data. Further details of this experiment can be found in the original paper.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Simulation 2: Experiment and model stimuli examples.

A) Example stimulus used by [10]. The four features of interest were reported to subtend approximately 4° of visual angle each, with the stimulus height and width being 12°. B The four color features, spaced 24 spatial units horizontally and vertically and 34 spatial units diagonally, presented to the model.

https://doi.org/10.1371/journal.pone.0259511.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Category structure used in [10].

https://doi.org/10.1371/journal.pone.0259511.t003

The best fitting models in Simulation 2 were found by comparing individuals with LAG-1 on 2 subject-level measures and 2 population-level measures. Where Simulation 1 had 8 components to the error function, Simulation 2 uses the 16 transfer responses of each individual, as well as their individual learning speeds, and the average fixation counts at the start and end of the experiment, for a total of 19 components as the objective function (weights listed in Table 8 S5 Appendix). Because we did not have individual data for all measures, we do not show fits to individual subjects as we did in Simulation 1.

Simulation 2 results—direct fits

Again, we first look at model performance on measures that we fit directly—that is, some aspect of the measure was a component of the fit function.

Transfer probabilities.

The first measure we look at is transfer behaviour. Fig 9 shows the transfer responses for both human participants and our LAG-1 simululations. These responses are to both trained A and B category stimuli, as well as the seven transfer stimuli not presented during training.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Simulation 2: Model fits to transfer responses.

Average categorizations on the 16 transfer stimuli, over all models/subjects. Error bars represent standard error of the mean.

https://doi.org/10.1371/journal.pone.0259511.g009

Fixation probabilities and counts.

Trial-level fixation measures reported in [10], are presented in Fig 10 along with the comparable output from the best fitting LAG-1 models. It is clear that by the end of the experiment human participants are fixating the least diagnostic dimension, Feature 2, less often and for less overall time than the other features. The fixation probabilities to each feature are generally higher in LAG-1, just as they were in Simulation 1 (as indicated by the probability of fixating the irrelevant feature measure over the course of the experiment). However, LAG-1 does show analogous reductions in fixation probability and fixation counts, to the least-diagnostic feature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Simulation 2 model fits.

A) The probability of fixating a particular feature reported for each of feature observed in human subjects. B) The probability of fixating a particular feature reported for each of feature observed in the best fits of LAG-1. C) The fixation counts to each feature averaged over all trials observed in human subjects. D) The fixation counts to each feature averaged over all trials observed in the best fits of LAG-1.

https://doi.org/10.1371/journal.pone.0259511.g010

Simulation 2 results—indirect/predictive fits

Fixation proportions.

As the purpose of [10] was to use an overt empirical measure of attentional optimization to test model predictions, one of the measures they reported was the proportion of time spent on each feature in the transfer phase of the experiment (see Fig 11). Most category learning models are fit with a capacity-constrained level of attention, meaning that on any trial the weights to each dimension are normalized in some way; this means that the total time on each feature has to be represented as a proportion. In our simulations, LAG-1, like the human participants, reproduces the rank ordering of the features by diagnosticity, with high diagnosticity features receiving the most observations, then medium, then low.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Simulation 2: Fixation proportion fits.

The fixation proportions aggregated across just the transfer trials of the experiment. LAG-1 data is on the left, while the data from [10] are re-represented on the right.

https://doi.org/10.1371/journal.pone.0259511.g011

Within-trial fixation probabilities.

The last measure we report here is similar to the within-trial fixation probability measure that we reported in Simulation 1. In Fig 12 the first four seconds of LAG-1’s average within-trial fixation probability to each feature is reported alongside the human data presented in [10]. LAG-1 reproduces the approximate ordering of peak fixations to features of differing diagnosticity: the higher diagnosticity features have peak fixation probabilities earlier than less diagnostic features. The lowest diagnosticity feature is fixated much less often in the first few seconds of the trial for both humans and the model. As in Simulation 1, we do not use within-trial information as part of the fitting function; that and, the largely arbitrary relationship between fixation orders and the overall category structure, make these data difficult to fit precisely.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Simulation 2: Within-trial fixation probabilities fits.

A) Human within-trial fixation probabilities averaged over the first four seconds all transfer trials reported in Rehder and Hoffman (2005). B) Comparable within trial fixation probabilities for LAG-1. High (relevance) 1 refers to Feature 1 in Table 3, Low is Feature 2, High 2 is Feature 3, and Medium is Feature 4.

https://doi.org/10.1371/journal.pone.0259511.g012

Opportunities for improvement

While LAG-1 does a fine job fitting the important qualitative findings, which are summarized in Table 4, there is significant room for improvement in getting the magnitudes correct. While many of the findings might be improved by tweaking one or more of the numerous fixed parameters that influence model timings, there are two major areas in particular where LAG-1 diverges from the human data in ways that seem more fundamental. First, the model does not show enough feature neglect (i.e., inattention to critical features), and second, the variability on many measures, like fixation duration and probability of fixating an irrelevant feature, is too small.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Summary of simulation results.

https://doi.org/10.1371/journal.pone.0259511.t004

Feature neglect is most obvious in Fig 10: LAG-1 fixates too many features. At the start of the experiment, humans ignore a stimulus feature about 25% of the time. Intuitively, this seems strange. After all, how can one learn which features are predictive of the category if they are not viewed? In addition to causing difficulty for LAG-1, this finding also contradicts the assumptions made by models such as ALCOVE, which spreads attention evenly across all features at the beginning of an experiment [16]. It is not only in the [10] data that we find this pattern however, we also see this in the [25] data which had a different number of stimulus features. Indeed, [26] reports the probability of ignoring irrelevant information from 10 experiments, and the range of feature neglect seen in the first block of the experiment was between 10% and 40%. One possibility is that this is a function of missing fixations due to eye tracker error; but two facts mitigate against this idea. First, we ran simulations using LAG-1’s gaze data wherein we dropped gaze points (because we used LAG-1’s data, there were no naturally missing values) in the same proportion, and with the same sequence length, as in humans, to see if the missing data would cause reduced fixations, and thus feature neglect. It did not. In fact, the overall number of fixations in the experiment increased. Second, we reanalysed the human data from the [25] study, using only trials in which we had 100% of the gaze data and found there was still significant feature neglect. [10] did note that this may be indicative of participants using rule-based strategies initially, but they ultimately conclude that there’s likely a mixture of strategies at work. Without further study, it is difficult to say for sure, but we note that there is at least some feature neglect even in the case of mouse driven interfaces such as used in the second experiment of [25]. Regardless, LAG-1 does not show significant feature neglect, and this fact influences the model’s fit to several other measures. For instance, the within-trial fixation probabilities do not quite match the human data (Fig 6), primarily because it looks too often at irrelevant features. Overall, LAG-1 does well with the learning-related changes in fixation probabilities, but does not capture all of their overall levels.

The other area where LAG-1 could provide better fits is in predicting the variability across, and within, individuals. This is most noticeable in our attempt to account for fixation durations (Fig 5B), but can also be seen to a lesser extent in measures like the probability of fixating an irrelevant feature (Fig 5D). Given the fact that almost every equation in the model has a noise parameter, it is sensible to ask if we might achieve the appropriate variability by simply increasing the noise on some of these. Possibly. Changing the variability, by increasing noise in one of the components of the model, can have a large impact—one that reverberates throughout all the dynamics of the system. In Fig 13 one can see that the neurons of the Saccade Timing System have very little noise. As an example, imagine that noise was added to the neurons of this system. This could increase the variability in fixation durations, but it would also influence ongoing Hebbian learning between features and categories; a fixation that lasts 800ms would end up having 4 times greater category association than would a 200ms fixation, and that, in turn can increase the variability in the speed of learning. Time sensitive activity is known to modify fundamental motor control over the eye [59]: a fact that causes additional challenges for modelling efforts. We felt it was more important, as a first step, to reproduce the primary attentional learning findings shown in Table 1, than to capture the variablity more generally, though we strongly agree with the idea that variabilty is important to fit [119].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Saccadic timing neurons.

Two examples of the contemporaneous activity of the three neurons of the Saccade Timing System, at two different points in learning. A) Before the model has made any strong feature-category associations, different stimulus features are equally interesting to it. Because there is little preference for particular features at this point, competition to determine the next saccade target takes longer to resolve than later in the experiment, extending the fixation duration in time. In these cases, the Gaze Change Neuron (blue) eventually surpasses the Fixation Neuron (red), just prior to the phasic spike in activity of the Saccade Initiation Neuron (black) which initiates a saccade. B) After the model has established stronger associations between features and categories, the knowledge that particular feature locations contain useful information, speeds targeting by boosting priority at these locations in the Spatial Attention Field. This is reflected in the modified timing relationships of the three neurons, where the Gaze Change Neuron does not reach the same level as the Fixation Neuron before the Saccade Initiation Neuron spikes. As a result, fixation durations are 30–40 ms faster than at the start of the experiment.

https://doi.org/10.1371/journal.pone.0259511.g013

Application of LAG-1 to other category learning situations

There are many variations of basic category learning experiments, and we have fit only a small subset of them. The studies we fit had binary valued stimuli, spatial separation of the stimulus features, self-paced responses, self-paced feedback viewing, and the re-presentation of the stimulus during feedback. But, what if they did not? Can LAG-1 handle task variants?

LAG-1 can easily handle any temporal manipulations—for example, adding a specific stimulus presentation duration rather than the self-paced presentation—with no modifications. Manipulations like changing the feedback presentation duration, presenting each stimulus feature for a different amount of time, presenting stimulus features in different orders can also be easily be implemented. Note that many of these experimental manipulations seem likely to influence the performance of the model, and are thus predictions; in LAG-1, as a theory, time matters for many things. For example, if one were to run LAG-1 through two conditions, one with a short feedback phase, and one with a long feedback phase, LAG-1 would predict that the long feedback phase would lead to stronger associations between active features and categories because of the time dependence of Hebbian learning. We note that this prediction about a temporal manipulation falls outside the scope of what extant category learning models can address because they are generally not modelling cognitive processes in time, and we also note that this prediction has some empirical support [30, 37].

LAG-1 can also deal with spatial manipulations with little to no modifications to the core model. LAG-1 can accommodate different spatial configurations, and make predictions about how this will influence measures like fixation order, reaction time, and even learning. Again, the configuration of the stimulus seems very likely to influence patterns of attention and gaze, and in turn, performance and learning. Imagine an experiment in which, in one condition the features are spread out in a triangle shape, and another condition in which two features are near each other on the left, and the other feature is on the right. This spatial configuration would likely lead to faster reaction times, because they require less time to inspect the features. The activation of two features next to each other (if they are sufficiently close) could join to create a larger peak on the visual field, and thus be more likely to attract attention. What if we compared a situation where the two features on the left were relevant for the correct classification of the stimulus and the feature on the right was irrelevant to one in which the relevant features were separated? It seems plausible that having the relevant features next to each other (and fixated first) might lead to a bit faster learning in that condition, which is to say that the interaction of learning and attention is influenced by the spatial factors that LAG-1 incorporates.

Other kinds of changes to the experiment can be implemented with varying difficulty. Removing the visual feedback indicator (to simulate a sound indicating accuracy, for instance) would be very simple to implement. We would omit the feedback button as input to the Visual Field. The correct category node can still be activated by feedback and the associations between features and categories can change as usual. This requires a change to the experiment (changing what is displayed to the model) but no change to the core model. More substantial changes would be required for something like comparing information sampling using eye movements to information sampling using hand movements. Manual information sampling (e.g., using a computer mouse to click on features to reveal them) is less variable, and more efficient, though slower, and shows similar changes throughout learning [25]. To the extent that we think of information sampling and the allocation of attention using the eyes to be broader phenomena (e.g. “active sampling”, [120]), additional mechanisms that implement manual selection of information—such as clicking of menus in a computer program—would be needed. Accepting stimuli with continuous dimensions rather than binary valued features is something LAG-1 is already coded to do, but were not used in the eye tracking experiments we simulate and added unnecessary complexity for the current purposes, so we omitted it here. Having more complex visual processing sufficient to categorize natural images would require more extensive modification. Work by [51] is a good example of how a DNFT-based model might approach this challenge.

Application of LAG-1 to a broader range of experimental situations

Our goal in creating LAG-1 was to build and test a model that can capture the interactions between learning, attention, and gaze. Category learning is a useful task in which to test this integration: participants learn categories and attend to features of varying relevance; further, both response behaviours and gaze behaviours change in concert throughout the experiment. Only a model that specifies how learning is related to the allocation of gaze, and vice versa, can illuminate this co-evolution. A wide range of experimental paradigms within the broader realm of visual cognition depend on these systems, however, and in this section we will provide a rough sketch of how LAG-1 might be expanded in a variety of ways to address to such situations.

Bottom-up salience.

In category learning tasks there is a clear goal: participants are choosing to fixate particular locations to get information that helps them choose the correct category. The emphasis of the task is on learning to direct top-down attention toward task relevant features, the locations of which are well known. The free-viewing of scenes, in contrast, is a function of bottom-up salience. Though LAG-1 does not include many aspects of bottom-up attention–in category learning experiments stimulus differences in bottom-up salience are controlled for by the choice of features and counterbalancing—nevertheless, adding the influence of bottom-up salience into the model is straightforward.

To demonstrate this we added an additional input to the Visual Field (implemented as an additional layer) that acts as a preprocessed salience map. To handle the wider range of field inputs, we also changed the function that chooses saccadic targets from a greedy, maximum activation based selection, to a probabilistic choice, like was used for the category decision. This allowed the model to fixate a broader range of locations. These small changes to the model allowed us to use an image as a stimulus. To do this, the image must be converted to a salience map that, when input to the Visual Field, can drive fixations to the most salient (in the bottom-up sense) areas of the image. Though we left the category learning system in place, there is no input to the Feature Detection Neurons, and so no top-down attention is generated. Fig 14 shows the original image (Roy Lichtenstein’s 1973 painting: “Things on the Wall”), the salience map it produced, a heat map of a typical human free-viewing the painting, and a heat map of LAG-1 fixations viewing the image. Note that this implementation of the model predicts bottom-up salience influences the Visual Field earlier than top-down forces because the recurrent loop through the whole system takes extra time. Indeed, short latency saccades are associated with bottom-up salience and longer latency saccades are more influenced by behavioural goals [121].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Salience map representation of model during free-viewing.

Top left) “Things on the Wall” by Roy Lichtenstein. Top right) a salience map generated by three boundary detection iterations at different spatial scales, integrated across the dimensions of: color, orientation, and intensity. Bottom left) Fixation heat map from a typical human subject. Bottom right) Fixation heatmap from one trial of LAG-1.

https://doi.org/10.1371/journal.pone.0259511.g014

For expediency, in this demonstration we created the salience map with a common Matlab package [122]. For example, we could implement the components of the salience map (i.e., color, orientation, intensity) as layers of the Visual Field, and calculate the salience from that activation. Abrupt onset and motion are also known to act as a source of salience (e.g., [123, 124]). Both of these factors might be implemented using large changes in activation at a particular location of the Visual Field to boost attention (see [45] for a DFNT-based approach to change detection). It is intriguing to consider that a system with robust, top-down attention may actually require a strong abrupt onset salience mechanism to function effectively because significant top-down attentional boosting of known, or expected features and locations, might otherwise make fixating new, unanticipated stimuli too improbable and slow.

Uncertainty in feature location.

In category learning the main focus is on the learning the connection between features and categories. In other situations the location of the information sought is unknown. One of the most iconic visual cognition tasks is visual search [125]. In visual search the categories used in the task (target and distractor) are simple and explained from the beginning; no learning is necessary, or in most experiments helpful or desired. Another difference is that while feature locations are static and known to learners in category learning tasks, feature locations are unknown in visual search. While the specifics of what is known and what needs to be learned differ between visual search and category learning, there is a general level at which the tasks are similar: one is presented an image, then one looks around for information in the image, and then makes a response. LAG-1 performs search in a guided fashion (e.g., [126]). It has a selective processing stream—the Learning system which only processes information about features that are fixated—but, it also has a non-selective one: the Visuospatial System, which processes information about basic visual properties (e.g. color) and their locations. Scene statistics and other bottom-up influences on salience are not currently part of the model, but could be incorporated in the same way as a saliency map is used in the free-viewing section above.

The most straightforward way of doing visual search in LAG-1 is to take “target” and “distractor” to be categorical designations (in place of category A and B). Features that indicate target and distractor categories are set in advance (i.e., hard coding the difference into the information gain matrix in this demonstration). In the model there are two recurrent pathways from the category level back into the Visuospatial System. One goes to the locations of particular features on the Spatial Attention Field. In category learning, this linkage between features and their location in space makes sense because participants know the feature locations—which do not move—from the instructions. In a visual search task, however, using this to guide search seems unrealistic because the participant has no knowledge of how particular features connect to particular locations on the Spatial Attention Field. To address this LAG-1 has a second connection: this one links back to the various layers of the Visual Field and can thus produce low-level feature-based attention [86]. Search in LAG-1 is thus guided using this second recurrent pathway. Knowledge about the target one is looking for recurrently boosts activation in the corresponding layers in the Visual Field via gain on the appropriate Feature Expectation Neurons. This second pathway raises the activation of the locations of those features and thus the chance that they will be selected as a visual target from the resultant boost on the Saccade Motor Field, in other words, guided search.

Consider a task in which participants are searching for a vertical line amongst lines of 15°. In simulating such a task, as we have done with LAG-1 and shown in Fig 15, the feature-specific layers of LAG-1’s Visual Field would code orientations of differing degrees (instead of the six different colors we used in the category learning simulations, the six layers used in Simulation 1 can code six different orientations. Top-down signals boost the Feature Expectation Neurons (or field, in this case) associated with the target orientation, leading to increased activation on the corresponding slice of the 3D Visual Field. This boost causes an increased likelihood of an eye movement to that location, relative to unboosted locations, and thus more efficient search. In this implementation of visual search, LAG-1 will look around from item to item, and when an item is fixated the Feature Detection Neurons are activated, which in turn activates the associated category nodes (target or distractor). The model will only stop when the model fixates the target pushing the Category Neuron for target to its threshold, or until the Trial Impatience has driven a click decision (e.g. on target absent trials). LAG-1 does not search in parallel (though again, its choice of target locations is biased toward the boosted location): it must confirm that a particular object is the target by fixating the target. This simple version of visual search, again, implemented with only minor changes from the category learning implementation, undoubtedly will produce the common findings of serial search slopes and RT differences between target present and target absent displays. Search will also be less efficient in the case of search for a conjunction of features: distractor locations that share properties with the target would also receive a boost, decreasing the relative advantage of target over distractor locations, and thus decreasing the target’s advantage. Overall, what LAG-1 would predict is that search is more or less efficient depending on the degree of overlap between target and distractor properties [127].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. LAG-1 applied to an example visual search problem.

A) Visual Field input array recreated from [53]. B) An example trial of LAG-1’s visual search scan path. Each frame is centered at the successive fixation location until the target is foveated.

https://doi.org/10.1371/journal.pone.0259511.g015

Foundational findings in visual search include accuracy and response time differences as the items of the search array differ by category, such as letters and colours, how they flank one another, and their variability. Several of these seem amenable to simulation by a visual search version of LAG-1. Knowledge effects, such as the influence of target and distractor similarity, fall out of the category learning system [128] as described above, as presumably would category effects [129]. Flanking, wherein search is slowed if distractor orientations are on both sides of the target orientation, also seems to have a relatively straightforward explanation that derives from the spatial dynamics of neural fields. In a more generalized version of our visual search extension, input from a small line length of 15° could more strongly activate neurons tuned to that orientation and weakly activate ones of nearby frequencies leading to a center surround bump on the field leading to a flanking effect. In LAG-1, this finding would make sense in that activation of a target could slightly activate orientations that are associated with distractors due to overlap (again, we are imagining a 3d visual field—with x and y as spatial dimensions and z is orientation—and a Feature Detection Field that codes orientation continuously). If distractors flank both sides, there is more overlap, and therefore more competition between target and distractor nodes. This competition would straightforwardly weaken the relative advantage of the target within the recurrent channel. This account makes the assumption that the orientations are close enough that there is some overlap in the activation bumps on the field. The model thus makes the prediction that the flanker effect would be strong in cases where the differences between targets and distractors are small, but may be weak, or non-existent if distractors are very different than targets, and thus have no overlap on the field (see Fig 19 in S1 Appendix).

Some findings do not seem to have a natural explanation given the current structure of LAG-1. The finding that search is slowed if there is more variation in distractors, even if that variation is farther from the target, (a 30° target among 15° distractors is found faster than a 30° target among both 15° and 0° distractors) has no obvious explanation for example [53, 130]. We can also see no easy method of accounting for search asymmetries such as the result that a 0° target among 15° distractors is easier to find than the reverse. More substantial modifications to the model seem necessary to account for such phenomena. By design, in most versions of visual search, learning mechanisms do not aid performance. However, in contextual cueing a display in a visual search task is repeated multiple times, and participants in these experiments show improved search times for these repeated displays [131]. LAG-1 has no method of producing this effect in its present form; learning in the model associates features with categories, but the location of the features is presumed to be known (consistent with the category learning tasks to which it is applied in the current work). [132] simulated contextual cueing using a simple two layer network in which distractor locations are associated with the target location upon each presentation of the display. This model restricts learning to those distractors that are nearby the target. Adding a similar mechanism to LAG-1 would be relatively straightforward. An additional field—a Visual Memory Field—could be linked via connections to the Visual Field, such that when the target is found (or more broadly, when reinforcement is presented), connections are strengthened between the Visual Field, and the Visual Memory Field around the area of fixation. Such a modification would more or less embed the [132] model into the larger LAG-1 framework.

Contributions and conclusion

While we have discussed at length the details of the model, the methods used in the simulations and the fits to the data, it is important to emphasize that, at the broadest level, the theme of the present work is integration. It proposes connections between different neurofunctional systems, and between different empirical phenomena operating at different timescales. Alan Newell identified four timescales of cognition, emphasizing that biological, cognitive, rational, and social phenomena, are not just different classes of phenomena, but that they occur at different speeds [19]. One important aim, then, is to understand how biological phenomena that are rapid, give rise to slower cognitive phenomena (such as a classification decision), and in turn give rise to still slower phenomena (such as category learning). LAG-1 gives just such an account; one that spans most of Newell’s temporal bands of cognition. At the finest scale the model resolves changes in neural activation with short timesteps (∼10⁻² seconds). These neural changes build over time to drive saccadic eye movements (∼10⁻¹ seconds), which are in turn ordered according to the expected information gain derived from Hebbian learning at the trial level (∼10⁰ seconds), shaping the perception of abstract categories learned on the order of ∼10¹ seconds. Finally, as associations strengthen and decline over trials, the model describes a continuous relationship between average learning curves and average attentional learning (∼10³ seconds).

LAG-1 is not an attempt to be a better model of category learning, or of oculomotor control, or of attention. Instead, it is a step toward building bridges between established work in these areas by providing a theory of how they might interact. As a result of the integrative approach to the construction of the model, LAG-1 can be applied to more than just category learning performance (e.g. accuracy, transfer probabilities). LAG-1 can simultaneously account for measures that have only occasionally been modelled (e.g. reaction time, probability of fixating features), and further, to measures that have never been directly modelled (e.g. time spent viewing feedback). We fit learning phenomena, and we fit attention phenomena, but most importantly, we account for their interactions (e.g. learning-related changes in fixation durations). We know of no related model that has shown good qualitative fits to as many different empirical findings as LAG-1.

Even just the existence of a dynamic theory of attention, learning and gaze provides theoretical motivation for novel experiments with temporal and spatial manipulations that could reveal a whole host of psychologically interesting phenomena related to learning and attention. Not only can it motivate new kinds of thinking and research about attention and learning in action, but it can also be constrained by, and improved based on, findings from a wide variety of investigations in the cognitive sciences.