Building upon the Rescorla-Wagner (RW) model.

If an agent believed in only one latent state, then a RW model could be used to determine the relationship between cues and rewards. The RW model proposes a linear relationship between cues and expected rewards. This relationship is captured by an associative strength for each cue updated according to: with initial associative strength V_n(1) = 0. Term α_n(t) is referred to as associability and is constant in the RW model. Collecting associative strengths in a vector , we can express this update concisely as

The term A(t) is now a matrix capturing associability. The RW model assumes A(t) is diagonal with terms α_n(t) along its diagonal, whereas we will consider matrices that are not diagonal. Changes in associative strength depends on the learning agent’s current expectation or best guess for rewards given observed cues. If their expectation exceeds the actual reward, then associative strength declines for each cue present. If their expectation is below the actual reward, then associative strength increases for each cue present. By updating associative strength in vector form, we see that rewards are captured by a linear regression model of cues. Hence, cues can be continuous or represent interactions, i.e. the presence or absence of a compound of cues. That is, the same update formula applies for continuous cues and interaction terms. Also note that the RW model learns cue-reward associations rather than action-value associations as in temporal difference reinforcement learning.

Allowing for latent-state learning.

We build upon the RW model by allowing a learning agent to propose L competing (i.e. mutually-exclusive) associations between cues and rewards. The index to each association is referred to as a latent state and is associated with its own RW model for predicting rewards from cues. The agent learns about associations for each latent state while learning about their belief in which latent state best explains current observations. We capture belief in latent state l on trial t as a positive variable p_l(t) such that beliefs sum to one: p₁(t) + … + p_L(t) = 1.

We let represent the vector of associative strengths and A_l(t) represent the associability matrix for latent state l. The learning agent updates associative strength as before except for the subscript l: (1) where we used the shortened notation E_l(t) to represent the prediction error for latent state l on trial t: (2)

An associability process.

Associability is constant in the RW model. By contrast, experiments such as those by Hall and Pearce [9, 24] suggest associative strength is path-dependent: associability, i.e. how quickly associative strength changes, depends on the history of observations and changes over time [10]. We describe associability as depending on current beliefs and a matrix B_l(t) which we refer to as an effort matrix: (3) for a patent-specific parameter α₀ ∈ [0, 1] controlling how much an individual weights older observations versus newer observations. The effort matrix B_l(t) is updated according to (4) with B_l(1) = I. The effort matrix estimates the matrix of cue second moments associated with each latent state. Second moment information of cues has been integrated into other models of learning, but usually in the form of variances and covariances rather than second moments directly [11, 19]. The use of the effort matrix is motivated in our derivation of our model in the Methods section and is similar to matrices found in an online algorithm for solving the machine learning problem of contextual bandits with linear rewards [25].

Updating latent-state beliefs.

Beliefs link latent states in our model. The assumed world view of the learning agent proposes that rewards are generated on trial t as follows: a latent random variable X(t)∈{1, …, L} is drawn from some distribution and rewards are drawn depending on the current latent state X(t) and cues (Fig 1A). Further, rewards are assumed to be mutually independent between trials conditional on latent states, and latent states are assumed to be Markovian, i.e. X(t) depends on X(1:t − 1) only through X(t − 1). Under these assumptions, Bayes law yields a posterior distribution over latent states from observed rewards given by: where we suppress the dependence on cue vectors to shorten notation. The posterior probability ρ_l(t) that latent state X(t) is l based on observations up to trial t is then:

From this expression, we see that optimal Bayesian inference requires enumerating over all possible sequences of latent states, the number of which grows exponentially with t. Such computation is generally considered optimistic for human computation [26]. Efforts are made to reduce computation such as with particle methods [17], assuming one transition between states [27], or using a functional form of memory that does not maintain the entire history of observations.

We capture a more general dynamic learning environment similar to [26], by assuming latent states are a Markov chain in which a latent variable transitions to itself from one trial to the next with probability 1 − γ(L − 1)/L and transitions to a new state (all new states being equally-likely) with probability γ(L − 1)/L. This assumption reflects that learning often involve blocks of consecutive trials, or stages, in which rewards are generated in an identical manner. An optimal Bayesian filtering equation could then be used to update beliefs

Unfortunately, this equation requires the learning agent knows the probability distribution of rewards for each latent state. We thus do not think the agent reasons in an optimal Bayesian way. Instead, we initialize beliefs p_l(0) = 1/L and propose that the agent uses an approximate Bayesian filtering equation (cf. [28]) (5) where ϕ is the probability density function of a standard normal random variable; σ(t) is the agent’s estimate of the standard deviation of rewards at the start of trial t; and (6) is a normalizing constant to ensure beliefs sum to 1. In other words, we replace the distribution/density function of rewards in the optimal Bayesian filtering equation with a normal density function with mean given by the agent’s current estimate and standard deviation given by the agent’s current estimate σ(t). Even though we use a normal density function, rewards do not need to be normally-distributed or even continuous.

Measuring expected uncertainty.

The learning agent may also track the standard deviation σ(t) of prediction errors. With only one standard deviation, we assume the learning agent estimates the standard deviation pooled over each latent state. We use pooled estimates to reduce the number of variables, but it may be more realistic to use a separate estimate for each latent state. We use the following update: (7) where is some initial estimate of the variance; β₀ is a constant associability parameter for variance σ²(t); and is the squared prediction error averaged over latent states: (8)

Variance is also used in other models of latent-state learning, such as the model in Redish et al [11] and the model in Gershman et al [19]. Further, Yu and Dayan [26] proposed a neural mechanism, via acetylcholine signalling, by which a learning agent keeps track of expected uncertainty in changing environments. As a staple to statistical inference, variance is a natural candidate for quantifying expected uncertainty.

Measuring unexpected uncertainty.

Many latent-state learning models assume the number of latent state can grow [11, 15, 17]. Similarly, we use an online mechanism to trigger an addition of new latent states when needed to explain rewards. This mechanism reflects the need to capture unexpected uncertainty, as discussed in Yu an Dayan [26]. It is based on Page’s algorithm [29] for solving a problem known as change point detection (cf. [30]). The idea is to test whether we can reject our current model in favor of an alternative model with an extra latent state. Specifically, we let L be the number of latent states actively considered on trial t. We keep track of model performance using a statistic q(t): (9) with a scalar parameter δ and initialized q(1) = 0. Importantly, ϕ(0)/l₀ is the likelihood ratio between a one-state model with expected rewards given by actual rewards R(t) and the current model, where l₀ was the normalizing constant for beliefs at Eq (6). If q(t) exceeds a threshold η, we let (10)

Note that so that the current reward is the expected reward for the new state. We then replace L with L + 1.

Changes in context.

Learning often involves changes in context whether it be new surroundings, backdrop, or point in time. Context changes are believed to influence learning in a distinctly different way than a cue influences learning [12]. Our model supposes that these changes in context corrodes current beliefs, whereby beliefs in a latent state on a previous trial are thought to be less informative on the present trial. For a temporal shift in context, our model supposes this corrosion increases with time. While the Gershman (2017) model uses a temporal kernel, we replace beliefs p_l(t) at the end of trial t with (11) where ITI is the time between trials within a task phase. This update amounts to repeatedly updating latent state beliefs for ITI − 1 iterations according to the Markov chain transition probabilities. Beliefs would become evenly distributed between latent states in the limit as ITI goes to infinity. For a visual or spatial shift in context, we set ITI = ∞ in equation above, leading to uniform beliefs 1/L, i.e. all latent states are believed to be equally likely. As a result, an agent does not carry their beliefs forward to the next trial when there is a spatial or visual change of context.

A temporal shift in context might also alter how associative strengths and beliefs are updated. Gershman et al [19] describes the influence of retrieval on a memory by way of “rumination”. This mechanism is captured in the model of Gershman (2017) [19] and in our model by repeatedly replacing associative strengths with the new estimate performing the updates at the end of trial t again. This process is repeated for a number of trials given by min{χ, ITI − 1} where χ is a patient-specific parameter.

Putting it all together.

Our latent-state model of learning is summarized in Fig 1B and in Algorithm 1. Briefly, a learning agent undergoes the following each trial: they observe cues, generate expectations for rewards, observe a reward, measure error, update their belief in latent states, and update associative strength and measures of uncertainty. In between trials, a learning agent may adjust their beliefs and associative strengths to account for contextual shifts whether they be visual, spatial, or temporal. Further, parameters can be tuned to describe differences in learning between agents: α₀ influences the rate of learning associative strengths, β₀ influences the rate of learning variance, γ influences transitions between latent state, σ₀ influences initial expected uncertainty, ν and δ influence unexpected uncertainty, and χ determines rumination.

Our model is grounded by six major predictions or assumptions about how an agent learns:

1. An agent learns online. Model variables are updated on each trial using only current observations and current values of model variables. Consequently, an agent can learn without remembering observations of cues and rewards from prior trials. Their memory and computational requirements grow only in the number of latent states rather than the number of trials, which may be a more realistic reflection of how humans learn. The Rescorla-Wagner model and other models also use online updates for certain model variables [8, 11, 17, 19].
2. An agent uses latent states to predict rewards. In deriving our model (see Methods section), we start with the assumption that the agent’s goal is to predict rewards. The role of a latent state is to account for different predictions that can be made upon observing cues. By contrast, the role of a latent state could be to discriminate between different sets of cues-reward observations [11, 17], which in turn could be used to predict rewards.
3. Associability, i.e. how quickly associations are updated, depends on beliefs in latent states and cue novelty. Associative strength updates quickly for latent states believed to be active, but slowly for latent states believed to be inactive. This feature is similar to the Mackintosh model [31] in that associability increases when rewards are accurately predicted. However, associative strength also updates quickly when an agent switches their belief to a new latent state to account for unexpected or surprising outcomes, i.e. outcomes in which prediction error is larger relative to the expected uncertainty. This feature is similar to Pearce-Hall model [9], which supposes associability increases in the presence of uncertainty. Meanwhile, the effort matrix, which tracks how often a cue appears (diagonal entries) and a pair of cues appear (off-diagonal entries), also determines associability. As a result, associative strength is updated more quickly when novel sets of cues are presented.
4. Latent states are relatively stable between trials. Latent states are expected to be the same from one trial to the next with a certain probability. This assumption causes beliefs in latent states to be relatively consistent between trials. An alternative is to assume latent states are exchangeable between trials, allowing beliefs in latent states to shift more sharply between trials [17]. Stability between trials, however, might degrade with time. Our model assumes that with time, the last trial becomes less and less informative about the latent state, to the point that the agent eventually believes each latent state is equally likely regardless of prior beliefs.
5. Beliefs are maintained over multiple latent states. Our model allows an agent to maintain beliefs over multiple latent states on each trial as opposed to believing in only latent state. For example, an agent is able to believe that two latent states are equally likely. This allows the agent to explicitly state their beliefs in competing associations, as is required by some learning tasks [32]. Maintaining beliefs over multiple states is a common feature for Bayesian models [17, 19, 27, 32]. An alternative is to classify each trial to one state [11, 33].

Algorithm 1: Proposed model of latent-state learning which updates variables online, i.e. does not keep track of past cues and rewards.

/* Loop through trials                */

for each trial t = 1, …, T do

 /* State inference                  */

 Update latent-state beliefs p₁(t), …, p_L(t) with Eq (5);

 /* Unexpected uncertainty learning        */

 Update change point statistic q(t) with Eq (9);

 if q(t + 1) > ν then

  /* Add new state                  */

  Set with Eq (10);

  Update number of latent states L ← L + 1;

 end

 /* Value learning                  */

 Update associative strengths with Eq (1);

 Update effort matrices with Eq (4);

 /* Expected uncertainty learning         */

 Update variance σ²(t) with Eq (7);

 /* Account for context               */

 if Context shifts after trial t then

  Repeatedly update associative strength with Eq (1) if temporal shift;

  Update latent-state beliefs with Eq (11);

 end

end

Associability depends on beliefs in latent-states.

Latent-state learning allows an agent to attribute sudden shifts in associations to new latent states, even when the shift is unsignalled. Associability starts relatively high when learning about a new state, thereby explaining the well-known Pearce-Hall effect [9] in which associability remains high in unpredictable circumstances.

To illustrate, we find that a shift in latent state beliefs can explain Experiment 1 from Wilson et al (1992) [38]. This experiment involved two groups, Group C and Group E. For Group C, the experiment consisted of alternating between reinforcing and not reinforcing two cues (a tone denoted by Cue B and a light denoted by Cue A) with a reward (food), followed by reinforcing just the light. Group E had similar experiment conditions, except the tone was omitted in the middle of the experiment on non-reinforced trials. The omission was expected to decrease the associative strength of the light for Group E relative to Group C, which would be observed when the light was paired with food. Surprisingly, Group E responded more favorably to the light than Group C. The leading explanation was that associability of the light was higher for Group E than Group C [10].

Our latent-state model offers an explanation: that while indeed the associative strength of the light does decrease for Group E relative to Group C, the pairing of the light (Cue A) solely with food is so surprising to Group E that they correctly infer the change in experimental conditions thereby shifting to a new latent state to build new expectations for the light (Fig 3). This shift would be observed as higher associability of the light in Group E than Group C, agreeing with experimental results. The Redish (2007) model also captures greater responding to the light in Group E than Group C. By contrast, the RW model, infinite-mixture, and Gershman (2017) models do not show greater responding to the light (cue A) in Group E. Meanwhile, the fRL+decay model captures this experiment but provides an alternative explanation: associative strength of the tone decays less in Group E because it is presented more often, which is compensated by an increase in the associative strength of the light.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Associability depends on latent-state beliefs.

A) Experimental results from Stage 3 of the Wilson et al (1992) experiment [27]. Group E had higher magazine activity, i.e. greater responding, than Group C during the light (Cue A) in Stage 3. Thus, it was believed the light had greater associative strength in Group E than Group C during Stage 3. Reprinted from Paul N. Wilson, Patrick Boumphrey, & John M. Pearce, Quarterly Journal of Experimental Psychology 44:1 pp. 17-36. Reprinted by Permission of SAGE Publications, Ltd. B) Simulation of associative strength of the light (Cue A) and latent state beliefs from the Wilson et al (1992) experiment. Our model predicts that only Group E detects the change in experimental conditions and shifts their beliefs. Because of this shift, associability is higher in Group E than C during Stage 3, leading to higher associative strength of the light. Beliefs in the first latent state (dark and light blue solid lines) and the second latent state (dark and light blue dashed lines) are shown for models with latent states. Gray dashed lines demarcate experimental stages.

https://doi.org/10.1371/journal.pcbi.1007331.g003

Other learning effects can also be similarly explained by our model as a shift in latent-state beliefs (Fig 4). Experiments 1A-B in [39], for example, examined whether a cue (Cue A) would undergo the same change in associative strength as another cue (Cue B) when presented together, as would be predicted by the RW model. Rescorla, however, found that Cue A decreased more in associative strength than Cue B. Our latent-state model explains that differences in associability arises because of a shift in latent states. In Experiment 1B, for example, the omission of the reward is so surprising to the agent—Cue A was always reinforced—that learning agents correctly infer a change in conditions and switch their beliefs to a new latent state in which they do not expect Cues A and B to be reinforced. Since rewards were originally expected to follow Cue A but not cue B, the associative strength of Cue A experiences the greater change. By contrast, the only other models to capture greater associability of Cue B during stage 2 was the infinite-mixture model in experiment 1A and the Redish (2007) model in experiment 1B. Otherwise, the other models for the given set of parameters do not detect the change in experimental conditions, causing cues A and B to change a similar amount during Stage 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Another demonstration of associability depending on beliefs in latent states.

Experimental results from A) Experiment 1A and B) Experiment 1B by Rescorla (2000) [39]. In both experiments, Rescorla concluded that associative strength increased more for Cue B relative to Cue A when presented together based on compound tests that showed greater responding in the compound with the Cue B than the compound with Cue A. This result suggested that associability can differ between cues even when presented together. Reprinted from “Associative Changes in Excitors and Inhibitors Differ When They Are Conditioned in Compound” by R.A. Rescorla, 2000, Journal of Experimental Psychology, 26, p. 430-431. Reprinted with permission from the American Psychological Association. C) Total change in associative strength was simulated during Stage 2 of Experiments 1A-B from Rescorla (2000) [39]. The RW model does not capture these effects since associability is constant whereas our model captures these effects because latent-state beliefs alters associability. Beliefs in the first latent state (black solid lines) and the second latent state (black dashed lines) are shown for models with latent states. Gray dashed lines demarcate experimental stages.

https://doi.org/10.1371/journal.pcbi.1007331.g004

The partial extinction learning effect (PREE) is also similarly explained by our model as a shift in latent-state beliefs (Fig 5). Partial reinforcement involves alternating between reinforcing and not reinforcing a cue. It was found that a response was harder to extinguish after partial reinforcement compared to continuous reinforcement [40]. Our model explains that an agent is better able to discriminate between reinforcement and extinction with a continuous reinforcement schedule. This allows the agent to switch their beliefs to a new latent state and thus experience faster extinction. PREE was also observed even if continuous reinforcement was administered in between partial reinforcement and extinction [41, 42]. In this case, an agent is still better able to discriminate between reinforcement and extinction with a continuous reinforcement schedule, but differences between reinforcement schedules are smaller as an agent shifts their beliefs to a new latent state in both cases. Only the Redish (2007) model was also able to capture greater extinction for continuous reinforcement compared to partial reinforcement in either of the two PREE simulation experiments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Partial reinforcement extinction effect.

A) Experimental results from Jenkins [41] demonstrating that the associative strength of a cue is harder to extinguish after partial reinforcement (Group 20P) compared to continuous reinforcement (Group 20R). Reprinted from “Resistance to extinction when partial reinforcement is followed by regular reinforcement” by H.M. Jenkins, 1962, Journal of Experimental Psychology, 64, p. 443. Reprinted with permission from the American Psychological Association. B) Simulation of partial reinforcement effect (Experiment 1). This effect is observed even when partial reinforcement is followed by continuous reinforcement prior to extinction (Experiment 2). Our model captures these effects, because an agent is better able to discriminate between reinforcement and extinction with a continuous reinforcement schedule. The agent can thus shift their beliefs to a new latent state in order to build new associations for extinction. Beliefs in the first latent state are shown for models with latent states. Gray dashed lines demarcate experimental stages.

https://doi.org/10.1371/journal.pcbi.1007331.g005

Associability depends on history of cue presentation.

Our model encodes the history of cue presentation in effort matrices B_l(t), endowing associability in our model with unique properties. One example is backwards blocking (Fig 6). A compound of two cues (Cues A and X) is reinforced with a reward followed by only one of the cues (Cue X) being reinforced. In the second part, Cue A gains associative strength while Cue X loses associative strength even though it is not presented. The RW model cannot explain backwards blocking, because a cue must be present to change its associative strength. The infinite-mixture and the Gershman (2017) model also did not capture backwards blocking. By contrast, the associative strength of Cue X correctly decreased in the second part for our latent-state model. Our latent-state model first learns the compound predicts a reward. Without additional information, it splits associative strength equally between the two cues. Later, our latent-state model learns Cue A predicts rewards. Reconciling both parts, our latent-state model increases the associative strength of Cue A while decreasing the associative strength of Cue X. In other words, our latent-state model learns about the difference in associative strengths between Cues A and X. Mathematically, applying the inverse of B_l(t) to the cue vector c(t) rotates the cue vector from being in the direction of Cue A to be in the direction of Cue A minus Cue X. Only the fRL+decay model was also able to reproduce backwards blocking, but provided an alternative explanation: associative strength decays for any cue that is absent, such as Cue X in Stage 2 of this experiment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Associability depends on history of cue presentation.

A) Experimental results of a backwards blocking experiment from Miller and Matute [43]. After two stages of the experiment, a backwards blocking group (BB) had significantly slower response time to Cue X than a control group (CON) even though Cue X was presented in an identical manner between groups. Their result suggested the associative strength of Cue X can change on trials it is not presented. Reprinted from “Biological Significance in Forward and Backward Blocking Discrepancy Between Animal Conditioning and Human Causal Judgement” by R.R. Miller and H. Matute, 1996, Journal of Experimental Psychology, 125, p. 374. Reprinted with permission from the American Psychological Association. B) Simulation of a backwards blocking experiment. In our model, associability depends on the history of cue presentation through effort matrices. After the combined associative strength of Cue A and Cue X is learned, these effort matrices rotate the direction of learning into the direction of the difference of Cue A and Cue X. As a result, associative strength of Cue X decreases even though Cue A is presented alone, thereby allowing our model to capture backwards blocking. Gray dashed lines demarcate experimental stages.

https://doi.org/10.1371/journal.pcbi.1007331.g006

Renewal.

Latent-state learning was offered as an explanation of renewal of expectations after extinction and the role that context plays in this renewal [11, 12, 15, 16]. We simulated renewal wherein a cue is reinforced, extinguished, and then reinforced (Fig 7). Two experimental conditions are considered: one in which the same visual/spatial context is provided through each phase and another in which a different visual/spatial context is provided during the extinction phase. Following prior models [8, 11, 17, 19], context was encoded as a separate cue in all models except our latent-state model. Our latent-state model encodes context as a shift in beliefs. Thus, while context may directly modulate both associative strengths and latent state inferences in other models, it only modulates latent state inferences in our model. This latter view is consistent with Bouton [12] who says “contexts modulate or ‘set the occasion’ for the current CS–US or CS–no US association. Put another way, they activate or retrieve the current relation of CS with the US”. Note for this example, expected rewards are given on each trial to account for the contribution of both the cue and the context rather than associative strength which only accounts for the contribution of the cue.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Changes in context influences beliefs.

A) Experimental results from Ricker and Bouton [46] demonstrating a faster response during reacquistion (phase 3) than acquisition (phase 1) after extinction (phase 2). Reprinted by permission from Springer Nature Customer Service Centre GmbH: Springer Nature, Animal learning and behavior, Reacquisition following extinction in appetitive conditioning, Sean T. Ricker & Mark E. Bouton, (1996). B) Experimental results from Bouton and King [47] demonstrating a more robust return of a fear when extinction occurs in different context (EXT-B) as opposed to the same context (EXT-A) as acquisition. Reprinted from “Contextual Control of the Extinction of Conditioned Fear: Tests for the Associative Value of the Context” by M.E. Bouton and D.A. King, 1983, Journal of Experimental Psychology: Animal Behavior Processes, 9, p. 252. Reprinted with permission from the American Psychological Association. C) Simulation results of expected rewards when a response is reinstated after extinction with and without a change in a visual/spatial context. The second experiment examines associative strength of cue when a response is reinstated after extinction with and without a change in a temporal context (i.e. a time delay between trials). Our model shows a rapid reinstatement of expectations as the agent switch their beliefs back to the first latent state. Our model also shows that rapid reinstatement is more robust with changes in context, particular in the first few trials of reinstatement. Expected rewards are depicted rather than associative strengths to account for the influence of context on expectations in addition to the cue, since models other than our model treat context as an additional cue. Beliefs are shown for the first latent state. Gray dashed lines demarcate experimental stages.

https://doi.org/10.1371/journal.pcbi.1007331.g007

In both experimental conditions, our latent-state learner slowly detects the shift when arm-reward contingencies first shift and switches their beliefs to a new latent state, effectively consolidating the memory of the first task phase. By detecting this shift, the agent can both construct new associative strengths for each arm and preserve the old associative strengths that were accurate for the first block of trials. These associative strengths are later recalled when arm-reward contingencies revert back and the agent switches their beliefs back to the original latent state. Introducing a different context during extinction allowed the learning agent to better detect the underlying structure of the task, recalling the first latent state earlier during reinstatement, resulting in higher associative strength in the final phase. This improvement in detection predicted by the model, however, was very slight and most noticeable in the first few renewal trials. The infinite-mixture model predicts that the learning agent switches beliefs in a new latent state only when a different context is introduced otherwise using one latent state to predict all phases of the task. For the given parameters, the Gershman (2017) model uses only one latent state for all three phases of the task for both the no renewal and renewal conditions. Consequently, other than our model, only the Redish (2007) model correctly predicts a rapid return of expectations.

Spontaneous recovery.

Latent-state learning also provides an explanation for how changes in temporal context influences reinstatement (Fig 8). For example, spontaneous recovery is a learning effect wherein the associative strength of a cue is reinstated more strongly after a time delay between extinction and renewal [12]. We simulated spontaneous recovery using the same schedule as renewal, but adding a time delay between extinction and renewal. Our model accounts for time delays by shifting beliefs towards uniform beliefs over latent states. Uniform beliefs reflects that beliefs from a prior trial is less informative on a trial when there is a time delay between trials. As a result, a time delay causes the associative strength of the cue predicted by our model to be about half the associative strength at the end of its initial acquisition (first latent state) and at the end of extinction (second latent state). Without the time delay, the associative strength predicted by our model is simply the associative strength at the end of extinction. Only the Gershman (2017) model adjusts its prediction for context changes due to temporal shifts, but does not capture a more robust return of associative strength due to a temporal shift. The other models also do not capture this more robust return.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Changes in temporal context also influences beliefs.

A) Experimental results from Brooks and King [48] demonstrating a more robust return of a fear when extinction occurs after a 6 day delay as opposed to immediately after acquisition. Reprinted from “A Retrieval Cue for Extinction Attenuates Spontaneous Recovery” by D.C. Brooks and M.E. Bouton, 1993, Journal of Experimental Psychology: Animal Behavior Processes, 19, p. 80. Reprinted with permission from the American Psychological Association. B) Simulation results of associative strength and beliefs when a response is reinstated after extinction with and without a change in a temporal context (i.e. a time delay between trials). Our model shows rapid reinstatement is more robust with changes in temporal context, particular in the first few trials of reinstatement. Beliefs are shown for the first latent state. Gray dashed lines demarcate experimental stages.

https://doi.org/10.1371/journal.pcbi.1007331.g008

Memory modification.

Our last simulation experiment explores the potential role of latent-state learning in memory modification. Monfils et al [44] in a rodent model and later Schiller et al [45] in humans showed that a single retrieval trial between acquisition and extinction can reduce the fear response to the cue in latter tests. The time between retrieval and extinction was further shown to modify this effect. Only certain windows of times reduced the fear response. They proposed that the retrieval trial helped to reconsolidate the acquisition memory to be less fearful within this reconsolidation window. The Gershman (2017) model [19] includes a computational explanation of this phenomenon. With a similar Bayesian framework to the Gershman (2017) model, our model included similar memory modifications.

We simulated the Monfils-Schiller experiment in [19], varying the time between retrieval and extinction (Fig 9). Our model reproduces a qualitatively similar result as the Gershman (2017) model. There is a reconsolidation window which can help reduce the fear memory. For example in both models, associative strength of the cue during testing is smaller for a time of 5 from retrieval to extinction relative to time of 1 or 100. Both models predict that beliefs in the first latent state (corresponding primarily with acquisition) remains relative high in the reconsolidation window, allowing the agent to update the acquisition memory to be less fearful. Outside the reconsolidation window, the agent updates the second latent state (corresponding primarily with extinction), leaving the acquisition memory relatively intact. While qualitatively similar between models, the effect of retrieval is more pronounced in our model in terms of differences in magnitude of cue associative strength during testing.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Memory modification.

A) Experimental results from Schiller et al [45] demonstrating that different delays (10 min, 6 hr, no reminder) of a retrieval trial can significantly modify fear response after extinction. Reprinted by permission from Springer Nature Customer Service Centre GmbH: Springer Nature, Nature Preventing the return of fear in humans using reconsolidation update mechanisms, Schiller et al., (2010). B-C) Simulation results of associative strength and beliefs for three different time delays (1, 5, and 100) between a single retrieval and extinction. Both models correctly predict that certain time delays can weaken the associative strength upon testing. The bottom row depicts a ‘reconsolidation window’ of time delays after retrieval wherein the associative strength on testing is decreased.

https://doi.org/10.1371/journal.pcbi.1007331.g009

Rescorla-Wagner model as an online approach to maximum likelihood estimation.

Before we motivate our latent-state model, we explore how one could perform MLE online to predict rewards without latent-states. In the process, we show that the RW model is an online approach to MLE and illuminate a pathway for performing MLE online with latent-states. The most natural modeling choice for rewards is arguably a standard Gaussian regression model, ubiquitous in statistical inference: where is the cue vector on trial t (independent variables); are unknown parameters (unknown regression coefficients); and ϵ(t) is an independent Gaussian random variable (error) with mean zero and unknown variance ν. Unknown parameters are collected in .

Although we could maximize ℓ directly, we want an online approach to maximization and instead consider applying Newton’s method for optimization (i.e., gradient descent), which involves repeatedly updating an estimate of θ with a new estimate of the form: for some learning rate λ(t) and an appropriate matrix H(t). Matrix H(t) can take many forms and is important for determining how quickly estimates converge to a maximum/minimum. An ideal choice is the Hessian resulting in the Newton-Raphson method, but the Hessian is often noninvertible in MLE. A common alternative is the negative of the Fisher information matrix scaled by 1/t and evaluated at the current estimate [50]. Even with Newton’s method, rewards still need to be stored up to trial t, but Newton’s method can be turned into an online algorithm using the well-known Robbins-Monro method [53]. We simply replace the gradient with an (stochastic) approximation that depends only on the current trial:

Note that each side of the equation is equal in expectation when rewards are independent and identically-distributed.

Upon replacing the gradient with a stochastic approximation, we arrive at the RW model if we use a constant learning rate λ(t) = λ₀ and the matrix

For the Gaussian regression model, matrix H(t) resembles evaluated at We simply replaced with the identity matrix I, which ensures H(t) is invertible. Putting the online approach together with the fact that for the Gaussian regression model, we arrive at the following online update for the estimate :

Critically, estimates are updated exactly as in an RW model:

In other words, the RW model is an online approach to maximum likelihood estimation.

Latent-state learning as an online approach to expectation-maximization.

Motivated by the previous argument, we now explore how one could perform MLE online to predict rewards with latent-states. A natural choice for modeling rewards is to use a Markov chain of Gaussian regression models. Latent random variable X(t)∈{1, …, L} represents the active hypothesis on trial t. Given the active hypothesis X(t) = l, rewards are described by a Gaussian linear regression model: where are unknown parameters and ϵ(t) is an independent Gaussian random variable with mean zero and unknown variance ν. The extension to latent-state learning thus leads to L different Gaussian regression models for rewards rather than one. As discussed in the Model subsection, we assume X(t) is a Markov chain that transitions to itself from one trial to the next with probability 1 − γ(L − 1)/L and transitions to a new state (all new states being equally-likely) with probability γ(L − 1)/L. Parameter γ and initial probabilities for X(0) are assumed to be known/given, leaving unknown parameters collected in .

In a latent-variable setting, a popular algorithm to perform MLE is expectation-maximization (EM) [54]. An iterative algorithm, EM improves upon a current estimate at each iteration in two steps. Fixing the current trial number t, an expectation step uses a current estimate to calculate the expectation of the complete log-likelihood of rewards and latent states given rewards and scaled by 1/t:

A maximization step searches for a maximum of over θ. The maximum is then used as the estimate in the next iteration. The crux of the EM method is that is easier to maximize than the log-likelihood ℓ(θ) of the observed data and that improvement in guarantees improvement in ℓ(θ).

Mirroring our argument for the Rescorla-Wagner model, we can replace the maximization step with a Newton update: for appropriate learning rate λ(t) and matrix H(t). Even in a latent-variable setting, one choice for H(t) is still the negative of the Fisher’s information matrix scaled be 1/t and evaluated at , but Newton’s method still requires all rewards up to trial t. To recover an online approach, we use a stochastic approximation to in the maximization step:

We then replace , which is computed using all the rewards, with latent-state beliefs p_l(t) defined at Eq 5 which is computed online. This replacement yields the stochastic approximation

The last equality follows from our assumption that rewards are a Markov chain of Gaussian regression models.

Upon replacing the gradient in Newton’s method with this stochastic approximation, we can arrive at our latent-state learning model if we use λ(t) = α₀ and matrix where we used the definition of B_l(t) at Eq 4. For a Markov chain of Gaussian regression models, matrix H(t) resembles the negative of the Fisher information matrix scaled by 1/t: where . We simply replaced diagonal blocks with B_l(t) which ensures H(t) is invertible and replaced ρ_l(s, θ) with latent state beliefs p_l(s). With these choices for λ(t) and H(t), we update online according to:

We thus arrive at our latent-state model for updating associative strengths at Eq (1). We can also recover our latent-state model for updating variance at Eq (7), if we replace the learning rate α₀ with a different learning rate β₀. This completes our motivation for our latent-state model.