Advertisement
  • Loading metrics

Representations of Value in the Brain: An Embarrassment of Riches?

  • Jeffrey J. Stott,

    Affiliation: Graduate Program in Neuroscience, University of Minnesota, Minneapolis, Minnesota, United States of America

  • A. David Redish

    redish@umn.edu

    Affiliation: Department of Neuroscience, University of Minnesota, Minneapolis, Minnesota, United States of America

Representations of Value in the Brain: An Embarrassment of Riches?

  • Jeffrey J. Stott, 
  • A. David Redish
PLOS
x

Abstract

Over the past two decades, neuroscientists have increasingly turned their attention to the question of how the brain implements decisions between differently valued options. This emerging field, called neuroeconomics, has made quick progress in identifying a plethora of brain areas that track or are modulated by reward value. However, it is still unclear how and where in the brain value coding takes place. A primate study by Strait and colleagues in this issue of PLOS Biology finds overlapping signals of value coding in two brain regions central to the valuation process: the ventromedial prefrontal cortex and the ventral striatum. This finding reconciles the primate and rodent literatures, provides valuable insight into the complexity of value computation, and helps set the agenda for future work in this area.

The (Neuro)Economic Model of Choice

A central axiom of economic models of choice is that there exists a single, common scale of value that the decisionmaker uses to compare options, and that this common currency can be revealed through the actions of the animal (see Box 1 for a glossary of terms used in this preview). This hypothesis matches our intuitive notion that when we make decisions, we weigh them against each other. Mathematically, this principle is necessary for internally consistent (i.e., rational) choices. Decisions arise from computations occurring in neural systems. Under the assumption of this central axiom of economic choice, one would expect to find abstract representations of value somewhere in the nervous system. Therefore, it generated a great deal of excitement when neuroscientists studying decision making started finding correlates of value within the brain [1].

Box 1. Glossary of Terms

Value: An integration of factors that affect how desirable a reward is, including the magnitude of reward, probability of receiving the reward, delay to the reward, effort needed to obtain the reward, etc. Economics tells us that value has to be revealed through an agent’s actions.

Subjective value: Because value reflects the specific needs and preferences of the individual, value is inherently subjective. Subjective value is contrasted with objective value.

Common currency: A metric for measuring or representing the value of an option so that all things can be measured on the same scale. In neural terms, a common currency coding scheme refers to a reliable relationship between the activity of a neuron (or brain region) and the value of a specific option.

State-based representations: Representations about the “state of the world.” In terms of reward, a state-based representation could include factors beyond immediate value such as reward identity (i.e., flavor), when or where the reward occurs (contextual information), the motor action needed to obtain the reward (i.e., “go left” versus “go right”), etc.

Expectations of reward: Explicit knowledge about potential rewards. In neural terms, this would be reflected in changes in neural firing that are elicited by cues (external or internally generated) that predict reward.

Orbitofrontal cortex (OFC): Part of the prefrontal cortex located behind the orbit of the eyes (in primates) and in the anterior (frontal) region of the cerebral cortex in rodents. The OFC is activated by reward receipt and by stimuli that predict reward.

Ventromedial prefrontal cortex (vmPFC): Part of the prefrontal cortex medial to the OFC. However, some papers use these two terms synonymously. The vmPFC is activated by reward receipt and by stimuli that predict reward.

Ventral striatum (VS): The striatum is the major input nucleus of the basal ganglia, a series of interconnected brain nuclei that form a loop with the cerebral cortex. The ventral (toward the bottom) aspect of the striatum is activated by rewards and reward-predicting stimuli. It also receives dense inputs from the dopamine system, and this input is thought to mediate reward learning.

A number of studies that recorded activity from single neurons in the monkey orbitofrontal cortex (OFC) found neurons that changed their firing rate according to the subjective value of the reward being offered, showing a common currency for value, independent of (or “abstracted from”) the sensory and motor variables of the task [2]. In agreement with this model, previous experiments by Strait and colleagues have found evidence for common currency signaling in a nearby cortical structure, the ventromedial prefrontal cortex (vmPFC) [3].

In contrast, neuroeconomic experiments in rodents have largely failed to find these representations of abstract value signaling in the orbitofrontal cortex, instead finding more state-based representations of expectations of reward [49]. Moreover, the rodent studies have identified a prominent role for value coding in the ventral striatum (VS) [913], calling into question a unique role for the OFC in signaling value.

While it is certainly possible that this discrepancy could be chalked up to species-specific differences in brain architecture [14], and/or differences in task design, the study reported in this issue of PLOS Biology by Strait and colleagues [15] argues against these interpretations. Strait et al. find overlapping signatures of value coding in monkey vmPFC and VS, including evidence for abstract value coding, as required by the common currency model of decision making.

This result highlights a fundamental open question in the field of neuroeconomics: If multiple brain regions contain value signals, are these brain regions performing similar or distinct functions during the decision-making process? How do we account for the multiplicity of value signals found throughout the brain?

Computation

The fundamental question to ask about neural signals is: what computation are these signals providing to the behavior? Neural systems are computational processes evolved to select appropriate actions from information taken from one’s past (memory), one’s needs and goals (motivation), and one’s information about the world (sensory cues) [16].

The Question of Timing

Comparing the timecourse of representations in the vmPFC and VS, Strait et al. found that neural signals related to the value of the chosen action emerged earlier in the VS [15]. Interestingly, neural activity in the VS was able to predict the animal’s decision ahead of the choice period, but the neural activity in the vmPFC did not.

These results closely match a recent study in rodents where neural signals were recorded simultaneously from the OFC and VS in rats performing a neuroeconomic decision-making task [9]. Fig 1 shows a side-by-side comparison of the recording study in monkeys by Strait et al. [15] featured in this issue with a recording study in rodents by Stott and Redish [9]. In the study by Stott and Redish, reward expectation coding occurred earlier in the VS than in the OFC during trials in which the rat deliberated at the choice point (Fig 1D–1F). Strait et al. find a similar result of an earlier appearance of value-related signals in the VS than the vmPFC in the monkey (Fig 1A–1C). Both studies show that the VS contains value-related information before the OFC/vmPFC, hinting at potentially different roles for these structures, despite their similarity along other dimensions of value signaling [9,15]. In particular, the precise timing of activity in these two brain regions suggests that the VS activity arises earlier in the chain of events leading to action selection than the OFC or vmPFC.

thumbnail
Fig 1. Schematic illustration of parallel neuroeconomic tasks in monkey and rat.

The top panel illustrates the task (A), recording locations (B), and one of the significant results (C) from the study highlighted in this issue of PLOS Biology by Strait and colleagues[15]. The bottom panel illustrates the task (D), recording locations (E), and one of the significant results (F) from the study by Stott and Redish, 2014[9]. Image credit: Karin Odell.

http://dx.doi.org/10.1371/journal.pbio.1002174.g001

A Multiplicity of Value Signals in the Brain

Neuroeconomic experiments have identified a number of brain regions that exhibit neural responses related to reward value [9,1722]. The data presented by Strait et al. [15] in this issue of PLOS Biology adds to this body of work, showing similar representations of value in the VS and vmPFC on the same task, albeit with differences in the timing of activity. Still, the broad overlap in value signaling between different structures invites the question of what these multiple brain regions are contributing to choice. There are three hypotheses that could explain the presence of value-related neural firing in multiple brain structures.

First, trivially, multiple structures could be contributing the same valuation computations to the system, at which point one would expect to see multiple copies of value-related signals. This hypothesis is unlikely to be correct, because studies have consistently found subtle differences between the representations of value, and because value-related signals have been found across the brain, including in structures well-known to be involved in other functions (such as sensorimotor integration [17]) and even in primary sensory areas such as the visual cortex [23].

Second, value representations could serve as a modulator of other signals. Value is important in a number of domains, including sensory, motor, and cognitive activities. In perceptual discrimination tasks, increasing the value of a target has been shown to increase neural firing in cells that detect sensory stimuli [23], in cells that are tuned to specific motor actions [24], and in cells that are more aptly described as “sensorimotor,” integrating sensory and motor information [17]. Therefore, value signals are important in focusing attention toward salient stimuli and in biasing motor commands toward more valuable actions [25]. These forms of secondary modulation (wherein a cell has a primary response to a stimulus or concept but shows firing rate modulation of that primary tuning) occur throughout the brain, such as attentional modulation of visual stimuli in the primary visual cortex [23] or task-related modulation of place representations in the hippocampus [26].

Third, when interpreting the role of different brain regions involved in choice, it is important to keep in mind that the brain may accomplish a given task using a variety of different solutions. A broad body of work now supports the view that the brain contains several functionally and anatomically distinct decision-making systems, each of which selects actions (makes decisions) using a different underlying computational algorithm [16,27,28]. These different decision-making systems could also be considered as different valuation systems [16,29], under the assumption that decisions reflect value. This interpretation could explain the existence of overlapping value signals seen in the vmPFC and VS. Both areas may routinely generate representations of value, as seen by Strait et al., but these brain areas may be differentially recruited depending on which action selection system is being engaged.

Each of these hypotheses call into question the centralized common currency hypothesis of value and opens new unanswered questions.

The Ignorance Questions

In his intriguing book Ignorance [30], Stuart Firestein suggests that the purpose of science is less to identify new answers as much as it is to identify new questions: what are the questions that you didn’t know were questions until you made this discovery? These new observations that value representations occur across the brain open up several new questions and call into question even the fundamental concept of value in a common currency.

What Happens When These Valuations Come into Conflict?

If there are multiple representations of value, then there must either be a mechanism to ensure that they always retain the same ordering between options or there must be a process by which the agent can take an action in the face of conflicting value signals. There is an extensive literature in behavioral economics showing irrational behavior (i.e., when choices are influenced by aspects that should not logically factor into the decision) under specific situations. Examples include framing effects, loss aversion, the endowment effect, extremeness aversion, sunk costs, etc. [31,32]. Do these irrationalities arise from conflict between multiple representations of value?

Do These Systems Participate in a Single Valuation Circuit?

One possible explanation for multiple valuation representations in the brain is that they are part of a larger circuit interacting with each other. This, of course, raises the question of the anatomical and functional connectivity between these neural structures. There is now extensive evidence that functional circuits can change on a moment to moment basis within tasks [33,34]. Is the functional flow of information from the VS to the OFC and vmPFC, as suggested by the timing data [9,15]? Or does it depend on the specifics of the task involved? In rats, lesion studies suggest that the VS and OFC are necessary for different aspects of valuation, depending on different task parameters, with the OFC only playing a role when flavor and identity are involved (apples versus oranges), but the VS playing a more general role whenever value is compared (both one apple to three apples and apples versus oranges) [4].

What Are the Computational Differences between These Different Valuation Signals?

The idea that there are multiple decision-making systems that can calculate value in different ways [16, 27, 29] predicts the presence of many different value signals in the brain. Although any given action may be driven by one of these systems, neural systems are often active even in tasks in which they are not playing an immediate computational role. For example, hippocampal neurons show place fields on tasks that do not require an intact hippocampus [26].

The Concept of Value

As intriguing and logical as the common currency concept is, it has recently come under scrutiny [35]. Strait et al. [15] and others [1] have found multiple representations of value in the brain, but we do not yet know if these representations must agree with each other or not. If they do not, then we come to an interesting question of arbitration and mediation and what drives behavior in these conflict situations. The relationship between these valuation signals and the decisions being made by the animal is a very intriguing open and interesting question that will require additional experiments.

References

  1. 1. Levy D.J. and Glimcher P.W., The root of all value: a neural common currency for choice. Current Opinion in Neurobiology, 2012. 22(6): p. 1027–1038. doi: 10.1016/j.conb.2012.06.001. pmid:22766486
  2. 2. Padoa-Schioppa C. and Assad J.A., Neurons in the orbitofrontal cortex encode economic value. Nature, 2006. 441(7090): p. 223–6. pmid:16633341 doi: 10.1038/nature04676
  3. 3. Strait C.E., Blanchard T.C., and Hayden B.Y., Reward value comparison via mutual inhibition in ventromedial prefrontal cortex. Neuron, 2014. 82(6): p. 1357–66. doi: 10.1016/j.neuron.2014.04.032. pmid:24881835
  4. 4. McDannald M.A., et al., Orbitofrontal neurons acquire responses to 'valueless' Pavlovian cues during unblocking. Elife, 2014. 3: p. e02653. doi: 10.7554/eLife.02653. pmid:25037263
  5. 5. Roesch M.R., Taylor A.R., and Schoenbaum G., Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron, 2006. 51(4): p. 509–20. pmid:16908415 doi: 10.1016/j.neuron.2006.06.027
  6. 6. Feierstein C.E., et al., Representation of spatial goals in rat orbitofrontal cortex. Neuron, 2006. 51(4): p. 495–507. pmid:16908414 doi: 10.1016/j.neuron.2006.06.032
  7. 7. Wilson R.C., et al., Orbitofrontal cortex as a cognitive map of task space. Neuron, 2014. 81(2): p. 267–79. doi: 10.1016/j.neuron.2013.11.005. pmid:24462094
  8. 8. Steiner A.P. and Redish A.D., The road not taken: neural correlates of decision making in orbitofrontal cortex. Front Neurosci, 2012. 6: p. 131. doi: 10.3389/fnins.2012.00131. pmid:22973189
  9. 9. Stott J.J. and Redish A.D., A functional difference in information processing between orbitofrontal cortex and ventral striatum during decision-making behaviour. Philos Trans R Soc Lond B Biol Sci, 2014. 369(1655). doi: 10.1098/rstb.2013.0472. pmid:25267815
  10. 10. Lavoie A.M. and Mizumori S.J., Spatial, movement- and reward-sensitive discharge by medial ventral striatum neurons of rats. Brain Res, 1994. 638(1–2): p. 157–68. pmid:8199856 doi: 10.1016/0006-8993(94)90645-9
  11. 11. Lansink C.S., et al., Fast-spiking interneurons of the rat ventral striatum: temporal coordination of activity with principal cells and responsiveness to reward. Eur J Neurosci, 2010. 32(3): p. 494–508. doi: 10.1111/j.1460-9568.2010.07293.x. pmid:20704595
  12. 12. van der Meer M.A. and Redish A.D., Covert Expectation-of-Reward in Rat Ventral Striatum at Decision Points. Front Integr Neurosci, 2009. 3: p. 1. doi: 10.3389/neuro.07.001.2009. pmid:19225578
  13. 13. Roesch M.R., et al., Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J Neurosci, 2009. 29(42): p. 13365–76. doi: 10.1523/JNEUROSCI.2572-09.2009. pmid:19846724
  14. 14. Wise S.P., Forward frontal fields: phylogeny and fundamental function. Trends Neurosci, 2008. 31(12): p. 599–608. doi: 10.1016/j.tins.2008.08.008. pmid:18835649
  15. 15. Strait C.E., Sleezer B.J., Hayden B.Y., Signatures of value comparison in ventral striatum neurons. PLoS Biol, 2015. 13(6): e1002173. doi: 10.1371/journal.pbio.1002173.
  16. 16. Redish A.D., The mind within the brain: how we make decisions and how those decisions go wrong. 2013, Oxford: Oxford University Press. xii, 377 pages.
  17. 17. Platt M.L. and Glimcher P.W., Neural correlates of decision variables in parietal cortex. Nature, 1999. 400(6741): p. 233–8. pmid:10421364 doi: 10.1038/22268
  18. 18. Leon M.I. and Shadlen M.N., Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron, 1999. 24(2): p. 415–25. pmid:10571234 doi: 10.1016/s0896-6273(00)80854-5
  19. 19. Paton J.J., et al., The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature, 2006. 439(7078): p. 865–70. pmid:16482160 doi: 10.1038/nature04490
  20. 20. Kawagoe R., Takikawa Y., and Hikosaka O., Expectation of reward modulates cognitive signals in the basal ganglia. Nat Neurosci, 1998. 1(5): p. 411–6. pmid:10196532 doi: 10.1038/1625
  21. 21. O'Doherty J.P., Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr Opin Neurobiol, 2004. 14(6): p. 769–76. pmid:15582382 doi: 10.1016/j.conb.2004.10.016
  22. 22. Glimcher P.W. and Fehr E., Neuroeconomics: decision making and the brain. Second edition. 2014, Amsterdam Boston: Academic Press. xxviii, 577 pages.
  23. 23. Shuler M.G. and Bear M.F., Reward timing in the primary visual cortex. Science, 2006. 311(5767): p. 1606–9. pmid:16543459 doi: 10.1126/science.1123513
  24. 24. Pastor-Bernier A. and Cisek P., Neural correlates of biased competition in premotor cortex. J Neurosci, 2011. 31(19): p. 7083–8. doi: 10.1523/JNEUROSCI.5681-10.2011. pmid:21562270
  25. 25. Lauwereyns J., et al., A neural correlate of response bias in monkey caudate nucleus. Nature, 2002. 418(6896): p. 413–7. pmid:12140557 doi: 10.1038/nature00892
  26. 26. Redish A.D., Beyond the cognitive map: from place cells to episodic memory. 1999, Cambridge, Mass.: MIT Press. xviii, 420 p.
  27. 27. van der Meer M., Kurth-Nelson Z., and Redish A.D., Information processing in decision-making systems. Neuroscientist, 2012. 18(4): p. 342–59. doi: 10.1177/1073858411435128. pmid:22492194
  28. 28. Daw N.D., Niv Y., and Dayan P., Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci, 2005. 8(12): p. 1704–11. pmid:16286932 doi: 10.1038/nn1560
  29. 29. Rangel A., Camerer C., and Montague P.R., A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci, 2008. 9(7): p. 545–56. doi: 10.1038/nrn2357. pmid:18545266
  30. 30. Firestein S., Ignorance: how it drives science. 2012, New York: Oxford University Press. viii, 195 p.
  31. 31. Plous S., The psychology of judgment and decision making. 1993, Philadelphia: Temple University Press. xvi, 302 p.
  32. 32. Kahneman D., Thinking, fast and slow. 1st ed. 2011, New York: Farrar, Straus and Giroux. 499 p.
  33. 33. Stujenske J.M., et al., Fear and safety engage competing patterns of theta-gamma coupling in the basolateral amygdala. Neuron, 2014. 83(4): p. 919–33. doi: 10.1016/j.neuron.2014.07.026. pmid:25144877
  34. 34. Benchenane K., et al., Coherent theta oscillations and reorganization of spike timing in the hippocampal- prefrontal network upon learning. Neuron, 2010. 66(6): p. 921–36. doi: 10.1016/j.neuron.2010.05.013. pmid:20620877
  35. 35. O'Doherty J.P., The problem with value. Neurosci Biobehav Rev, 2014. 43: p. 259–68. doi: 10.1016/j.neubiorev.2014.03.027. pmid:24726573