Table 1.
Processes and paradigms that have been modeled using the scheme in this paper.
Figure 1.
This figure provides a schematic overview of the message passing scheme implied by Equation 3.
In this scheme, neurons are divided into prediction (black) and prediction error (red) units that pass messages to each other, within and between hierarchical levels. Superficial pyramidal cells (red) send forward prediction errors to deep pyramidal cells (black), which reciprocate with predictions that are conveyed by (polysynaptic) backward extrinsic connections. This process continues until the amplitude of prediction error has been minimized and the predictions are optimized in a Bayesian sense. The prediction errors are the (precision weighted) difference between conditional expectations encoded at any level and top down or lateral predictions. Note that there are prediction errors at every level of the hierarchy, for both hidden states and hidden causes (and sensory states and the lowest level). The synaptic infrastructure proposed to mediate this comparison and subsequent modulation is shown in the lower panel, in terms of a doubly-innervated synapse [84] that is gated by dopamine (cyan). Here, dopamine is delivered by en passant synaptic boutons and postsynaptic D1 receptors have been located on a dendritic spine expressing asymmetric (excitatory) and symmetric (inhibitory) synaptic connections. This represents the synaptic arrangements indicated by the cyan arrows in the upper panel.
Figure 2.
This figure provides a schematic overview of winnerless competition.
These itinerant (wandering) dynamics are used to model sequential neuronal dynamics that, in this paper, encode prior beliefs about sequential changes in hidden states (e.g., affordance). Technically, these dynamics comprise stable heteroclinic channels or cycles that connect unstable fixed points. The fixed points are the colored dots in the upper left diagram. Each unstable fixed point is attractive in one dimension and repelling in another, expelling the state so that it is captured by the next unstable fixed point and so. A common example of these dynamics is provided by predator-prey relationships modeled with Lotka-Volterra equations of motion, denoted by
in the lower panel. The speed with which the fixed points are visited is controlled by a variable
that scales the elements in a transition matrix
, which couples the attractor states. In this paper, the attractor states are mapped to fixed locations in an extrinsic (physical) frame of reference to encode their affordance, using a softmax function of the attractor states
and a matrix
, encoding their locations. This means that the orbit or trajectory in the four dimensional attractor space maps to a two-dimensional trajectory, which cycles through the four locations in a fixed order. We use this trajectory to generate forces that elicit pointing movements: See [87] and [23] for details.
Figure 3.
This figure distinguishes between the equations of the generative model (left-hand side; see Equation 6) and the equations generating sensory information (right-hand side; see Equation 5).
The generative model is trying to predict the sensory states produced by the equations on the right. These sensory states comprise the location of the agent's arm in both proprioceptive (intrinsic) and exteroceptive (extrinsic) coordinates. The locations of the four cues in the previous figure are shown in extrinsic coordinates in the lower right insert. In addition to these sensory inputs, the agent also receives sensory information about the salience of cues at the four locations (e.g., illumination). The equations of the generative model have been divided into those responsible for the selection or generation of a particular context or set and those specifying the relative affordance of cue locations used to select action. Crucially, both sets of equations are based on winnerless competition using the itinerant dynamics of the previous figure. These equations come to life when action (driving movements) becomes a function of conditional expectations about hidden variables in the generative model. See main text for further details.
Figure 4.
This schematic illustrates the connections between prediction units (black) and error units (red) that underlie the simulated reaching movements.
The prediction units encode conditional expectations about hidden states and causes, while the error units encode the associated prediction errors. The connections between these two sorts of units are specified by the message passing scheme in Equation 3 (cf., Figure 1). In brief, error units pass precision weighted prediction errors forward and horizontally (red connections), while prediction units sent predictions backwards and horizontally (black connections). Note that prediction units only communicate with error units and vice versa. In this figure, expectations about hidden states in the first level have been divided into two sets, corresponding to the position of the arm (motor cortex) and the affordance of the cue locations (premotor cortex). The blue circle at the bottom of this figure indicates motor neurons in the ventral horn of the spinal cord that mediate action. The cyan arrows represent various projections from the substantia nigra and ventral tegmental area (SN/VTA). Exteroceptive sensory information enters directly at parietal cortex and the superior colliculus encoding positional information about the arm and the salience of cue locations respectively.
Figure 5.
This figure summarizes the results of simulations under normal levels of dopamine (using a log precision of four for all prediction errors).
The conditional predictions and expectations are shown as functions of time over 128 time bins, each modeling 64 ms of time. The upper left panel shows the conditional predictions (colored lines) and prediction errors (red lines) based upon the expected in states on the upper right. In this panel and throughout, the grey areas denote 90% Bayesian confidence intervals. The inferred speed of itinerant cycling among affordance states corresponds to the first of the hidden causes at the second level (left middle panel). These hidden causes are a softmax function of their associated hidden states (right middle panel). The blue lines encode a sequential context, while the green lines encode the converse (random) context. The switching in these conditional expectations occurs after sufficient sensory evidence has accumulated following a reversal of the presentation order. The lower left panel shows the trajectory (dotted lines) in an extrinsic frame of reference, in relation to the cue locations (green circles), while the lower right panel shows action in terms of horizontal and vertical angular forces causing these movements.
Figure 6.
This figure combines the dynamical results from the previous figure with the supposed functional anatomy in Figure 3.
It shows the conditional expectations about hidden states and causes associated with regionally specific representations. The dotted red time courses associated with the prediction error units in the striatum show a set-related prediction error when the order of the cues was reversed (after the first five presentations). It is these prediction errors that drive the switch in contextual expectations assigned to the prefrontal cortex.
Figure 7.
This figure shows the results of simulations under progressively reduced levels of precision (dopamine) as indicated by the equalities in the lower row.
The display format of these simulated responses is the same as used in the left panels of Figure 5 (conditional predictions and prediction error; hidden contextual causes at the second level and motor trajectories). The left column presents the conditional responses under normal levels of dopamine (as in Figure 5), while the middle and right columns show the equivalent responses for intermediate and low levels of dopamine. As noted in the main text, the main features of these simulations are reciprocal changes in the amplitude of prediction errors at the first and second levels that are associated with a progressive failure set switching (i.e., a failure to recognise that the order of stimulus presentation no longer conforms to sequential expectations).
Figure 8.
This is a blow up of the motor trajectories under low levels of dopamine from the previous figure.
It highlights the fact that when movements are in an expected clockwise direction, the initial trajectory (dotted blue arrow) is directed towards the (correct) next target location. Conversely, when the movements are in a counter clockwise direction, the agent is initially confounded by false expectations about which cue will appear next (dotted red arrow).
Figure 9.
This figure presents the behavioral results from the simulations under different levels of dopamine.
The upper panel shows the reaction times for each trial or cue as a function of cue order (over 10 cues). The reaction time was measured as the time from cue onset to the time that the pointing location fell within a small distance of the target location. The equivalent results for accuracy are shown in the lower panel in terms of the (inverse) average distance from the pointing location to the target location for each trial. The colored lines correspond to different levels of simulated dopamine; with red lines indicating the lowest level and yellow lines the highest. The key things to note here are: (i) the reaction time costs of unpredictable (first five), relative to predictable trials (first five), shown by the yellow line and (ii) the increase in amplitude and duration of switching costs as dopamine is depleted (colored lines); modeled here in terms of the precision of prediction errors on visual salience.
Figure 10.
This figure represents behavioral results in terms of reaction times for depleting dopamine in three regions: the superior colliculus encoding sensory salience (as in previous figure), the motor cortex encoding proprioception (middle column) and the premotor cortex encoding affordance (right column).
These results are shown using the same format as in previous figure and illustrate the qualitatively different effects of dopamine depletion in different parts of the brain (or model). The lower panels indicate the implicit projections, from the substantia nigra or ventral tegmental area, have been selectively depleted (where a red cross highlights the forward prediction errors affected). The key thing to take from these simulations is that reducing the precision of prediction errors on sensory salience induces bradykinesia and perseveration; whereas the equivalent reduction in proprioceptive affordance causes bradykinesia without perseveration. Finally, compromising the precision of changes in affordance increases perseveration and decreases bradykinesia.