Matching as a Sub-optimal Maximizing Strategy in Independent Choice Behaviors

The analysis is easier if we express the matching law as follows[8]:(1)where 〈r〉 is the average reward per choice from all options and 〈r|a〉 the average reward conditioned on choice of option a. We can derive the above expression from the relationship I_a≅〈r|a〉N_a. Thus, the matching law equalizes the expected returns on all the options that are chosen sufficiently many times. Note that the matching law should not be confused with “probability matching”[22], which states that the frequency of choosing option a is proportional to 〈r|a〉 rather than I_a. Probability matching is typically observed in a task in which each expected return 〈r|a〉 is fixed and independent of subject's behavior (i.e., concurrent variable-ratio schedules). In such a simple task, the maximizing behavior satisfies the matching law, but not the probability matching. Hereafter, we focus on the matching law. Moreover, we consider the case where subjects make choices at fixed intervals. We can employ the discrete time steps without much loss of generality, since the framework describes a free-response task on continuous time if the interval is sufficiently short and choosing nothing is an available option.

We analyze the outcome of the decision process without specifying the detail of neural decision system. To this end, we assume a set of ‘synapses’ w = (w₁, w₂, …, w_m) that determines the behavioral policy to make decision. These variables are often called “policy parameters” in mathematical models of decision making. Then, the probability of choosing option a is given as a function p_a(w) of the synaptic weights. To ensure a smooth search for an optimal set of choice probabilities, we require that arbitrary infinitesimal changes of {p_a(w)} allowed in the space of choice probabilities can be caused by some set of infinitesimal changes {dw_j}.

With the above definitions, we can describe the average reward per choice as . Many decision-making algorithms attempt to maximize 〈r〉 by modifying behavioral outputs. Whatever algorithm is used, the synaptic weights to maximize 〈r〉 should satisfy the stationary condition ∂〈r〉/∂w_j = 0 for arbitrary j, i.e.,(2)

The first term contains the explicit dependence of the choice probability on w_j, whereas the second term the possible change in 〈r|a〉 generated implicitly by the change in subject's behavioral policy. The conditional expectation value 〈r|a〉 is obtained by taking an average over all possible patterns of past choices in which the newest choice is option a. In general, the reward probability depends not only on the current choice, but also on the history of the past choices[6], [12]–[15]. In such a case, 〈r|a〉 depends on the choice probabilities that produced the past choices, and hence depends on w_j.

In order to maximize reward, the brain has to explore the correct dependence of the reward probability on the past choices. It seems, however, difficult to infer this dependency correctly with little knowledge on an accurate model of the environment. In such a difficult situation, the brain may simply omit the second term in Eq. 2 in its practical attempt to maximize reward,(3)

Multiplying Eq. 3 by arbitrary variations {dw_j} and taking a summation over j gives , where dp_a(w)≡Σ_j(∂p_a/∂w_j)dw_j represents the infinitesimal change caused by {dw_j}, and R≡(〈r|1〉, 〈r|2〉, …, 〈r|n〉) and dp(w)≡(dp₁(w), dp₂(w), …, dp_n(w)) are vectors in the space of multiple options. If all options have non-vanishing stationary choice probabilities, the probability changes dp(w) may occur in an arbitrary direction that satisfies the probability conservation , where 1≡(1, 1, …, 1) is an n-dimensional identity vector. Therefore, the conditions R·dp(w) = 0 and 1·dp(w) = 0 can simultaneously be satisfied only by such R that is parallel to 1. If the stationary choice probability vanishes for some option, p_a = 0, we can forbid the changes in this direction (dp_a = 0), and R should have identical components for all the options exhibiting non-zero choice probabilities. These results and Eq. 1 imply that the truncated stationary condition given by Eq. 3 is equivalent to the matching law.

Thus, the steady choice behavior exhibits matching when the decision system ignores the influence of subject's past choices on the expected outcome in aiming for the stationary condition of reward maximization. Hereafter, we call this suboptimal maximization strategy to achieve Eq. 3 “matching strategy”. By contrast, we call the strategy to directly solve Eq. 2 “the maximizing strategy”.

To demonstrate the above relationship between the matching and maximizing strategies, we study an alternative choice task (n = 2), in which the expectation value of return on each choice pattern is specified completely by the subject's current (a_t) and most recent choices (a_t−1) as (see Methods). We consider the case where subject's current choice is independent of its past choices. Hereafter, such decision behavior is called “independent choice behavior”. Since p₂(w) = 1−p₁(w), the subject's decision system controls only the choice probability p₁(w) through w, and makes every choice with probability p₁(w). Then the average return on the current choice 〈r_t|a_t〉 is obtained by averaging over the possible patterns a_t−1 = 1,2 as , and hence depends on w through the choice probability p₁(w). Since ∂〈r_t|a_t〉/∂w_j≠0, the matching strategy does not maximize reward in this task. Actually, it gives 〈r〉 = 0.25 whereas the maximizing strategy yields 〈r〉 = 0.45 (Figure 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Dependences of the expectation values 〈r|1〉, 〈r|2〉 (dot-dashed lines) and 〈r〉 (solid curve) on p₁ in a decision task with two options (Methods).

The reward probability is given as a function of the current and most recent choices, but the subject makes each choice independently of the past choices. The task parameters are set as g₁₁ = 0, g₂₁ = 0.2, g₁₂ = 1 and g₂₂ = 0.4. The expectation values are given as 〈r|a〉 = g_a1p₁+g_a2(1−p₁) and 〈r〉 = 〈r|1〉p₁+〈r|2〉(1−p₁). The matching (vertical solid line) and maximizing (vertical dashed line) choice probabilities are obtained as solutions of equations 〈r|1〉 = 〈r|2〉 and d〈r〉/dp₁ = 0 respectively. The matching strategy (〈r〉 = 0.25) earns less than the maximizing strategy (〈r〉 = 0.45) in this task.

https://doi.org/10.1371/journal.pone.0003795.g001

The matching strategy enables us to derive a variety of learning rules that lead to matching behavior (Supporting Text S1). For instance, such a category of learning rules includes the well-known actor-critic in the reinforcement learning theory [1], [20], [21], direct actor[23], melioration[16] and local matching[9]. In particular, the actor-critic and direct actor also belong to the covariance rule[19]. We numerically solved the decision task analyzed in Figure 1 to show that all these learning algorithms generate matching behavior (Figure 2A). By contrast, indirect actor [23] does not exhibit matching in the steady behavior (Figure 2B). The indirect actor belongs to Q-learning without state variables[1] (see below for the state variables). Since Q-learning determines the choice probabilities by estimating “action values”, i.e., the expected returns on individual options, it does not show matching.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Decision behaviors generated by various decision-making systems.

The horizontal and vertical axes indicate the cumulative numbers of choices given to option 1 and 2, respectively. Dashed and solid line segments indicate the slopes corresponding to the maximizing and matching choice probabilities, respectively. See Methods for details of the algorithms. (A) The actor critic (red), direct actor (magenta), local matching (blue) and melioration (green) were numerically simulated with β = 4. (B) The Q-leaning was simulated for β = 2, 4, 8, 16, and 32. At β = 32, the system eventually learns to choose only option 2.

https://doi.org/10.1371/journal.pone.0003795.g002

Matching vs. Maximizing over All Possible Choice Behaviors

The quantitative analysis conducted in Figure 1 was restricted to the case where the subject generates independent choice behaviors. It was shown that the maximizing strategy earns better than the matching strategy. However, the average reward 〈r〉 = 0.45 achieved by the maximizing strategy in Figure 1 is not the global maximum, but is only the best one among independent choice behaviors. For instance, an alternate choice pattern of 1212…, where the current choice depends on the most recent choice, can earn better (〈r〉 = (g₁₂+g₂₁)/2 = 0.6) than the best independent choice behavior in that task. Thus, to produce a better outcome in some situation, the subject is required to make each choice depending on the past choices or other available information. Below, we investigate the relationship between the matching and maximizing strategies, taking all possible choice behaviors into account.

To make the argument as general as possible, we include the case where the subject may receive sensory signals σ_t before making a choice a_t at time t. Then, in a given task, the external and internal information available for the subject at time t consists of the histories of sensory signals, subject's past choices and the past returns: H_t = (σ_t, r_t−1, a_t−1, σ_t−1, r_t−2, a_t−2, σ_t−2,…). A decision-making task specifies the conditional probability distribution P(σ_t+1, r_t|a_t, H_t). In contrast, the general rule to determine subject's choice behavior is described by the conditional probability distribution P(a_t|H_t). The problem is how to explore an optimal behavioral policy Pˆ(a_t|H_t) to maximize the average reward 〈r〉 in a given task.

In practice, however, it is difficult to optimize the dependence of P(a_t|H_t) on the whole history H_t. Hence, subject's decision system may extract partial information s_t from H_t, and restrict the behavioral policy as(4)

We may call the above s_t “state variables”. We assume that the decision system controls the definition of state s_t, H_t↦s_t, and P(a_t|s_t). In order to maximize the average reward, the decision system has to adopt an appropriate definition of state with which an optimal behavioral policy Pˆ(a_t|H_t) satisfies Eq 4. It has been proved [24] that if a map H_t↦s_t satisfies(5)for a given task, then the maximal average reward can be obtained by a behavioral policy that satisfies Eq. 4. The average reward obtained by an arbitrary choice sequence can be expressed by P(s_t+1, r_t|a_t, s_t) that satisfies Eq. 5 and does not depend on the variables that are not reflected in s. Therefore, state s that satisfies Eq. 5 represents crucial information about reward delivery in that task. The above theorem means that the optimal policy Pˆ(a_t|H_t) depends on only the crucial information. Hereafter, we may say that a definition of state variables, H_t↦s_t, is correct if and only if s_t satisfies Eq. 5. Note that the selection of the correct definition may not be unique.

Suppose that the decision system adopts a certain definition of state variables, H_t↦s_t. Let p_as = P(a_t = a|s_t = s) be the choice probability with which the decision system in state s chooses option a. Each state-dependent choice probability is determined as a function of the synaptic weights p_as(w). In order to explore all possible patterns of state-dependent choice probabilities smoothly, we assume that an arbitrary pattern of {p_as} and an arbitrary direction of infinitesimal changes {dp_as} allowed in the space of probabilities can be expressed by some pattern of w and some direction of infinitesimal changes dw, respectively (see Methods).

Taking the state dependence into account, the average reward is written as 〈r〉 = Σ_sΣ_a〈r|a,s〉 p_as(w)P(s), where 〈r|a, s〉 is the average reward conditioned on choice of option a in state s, and P(s) is the distribution of the states that the subject has visited over sufficiently many decision trials with fixed {p_as(w)}. The stationary condition for reward maximization ∂〈r〉/∂w_j = 0 is written as (6)

The maximizing strategy attempts to achieve Eq. 6 taking the whole dependence on w into account. In contrast, as in the previous case, the matching strategy ignores the dependence of the expected outcome of the current choice on w in aiming for the stationary condition. The outcome in the present case consists of the return r_t and the next state s_t+1. Therefore, the matching strategy ignores the dependence of P(s_t+1, r_t|a_t, s_t) on w, and hence ignores ∂〈r|a, s〉/∂w_j and ∂P(s′|a, s)/∂w_j, where P(s′|a, s)≡P(s_t+1 = s′|a_t = a, s_t = s). By transforming the second term repetitively with the recursive relation P(s′) = Σ_s,aP(s′|a,s)p_as(w)P(s) and by setting ∂〈r|a, s〉/∂w_j = ∂P(s′|a, s)/∂w_j = 0, we obtain the stationary condition of the matching strategy (Supporting Text S2): (7)

Note that the terms omitted in the matching strategy differ for different definitions of the state. Then, using Eq. 7 and the probability conservation, we can extend the matching law to the case of state-dependent choice behaviors (Supporting Text S2):(8)

The extended matching law given as Eq. 8 depends also on the definition of the state.

We schematically illustrate the relationships between the maximizing and matching strategies with correct and incorrect definitions of the state variables (Figure 3A). The horizontal plane represents the multi-dimensional space of arbitrary choice behaviors. Defining state variables restricts the state-dependent choice behavior to a certain subspace. If state variables are correctly defined to satisfy Eq.5, the subspace (red curve) includes the optimal choice behavior (red circle). The conditional probability P(s_t+1, r_t|a_t, s_t) takes a fixed value specified by the task, which is actually independent of w. Therefore, the matching strategy coincides with the maximizing strategy, which indeed earns the globally maximal average reward (red triangle) unless the choice behavior is trapped by a local stationary point. In contrast, if an incorrect definition of state variables is chosen, the set of generable choice behaviors (blue curve) does not necessarily include the optimal choice behavior. Therefore, the maximizing strategy can lead to only the best choice behavior (blue triangle) within the restricted set. The conditional probability P(s_t+1, r_t|a_t, s_t) depends on the past choices that are not reflected in state s_t, and hence depends on w. Therefore, the matching strategy (blue cross) in general deviates from the maximizing one (blue triangle).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Relationship between the maximizing and matching strategies for state-dependent decision-making.

(A) The performance of the matching and maximizing strategies based on correctly (red) or incorrectly (blue) defined state variables is shown schematically. (B) Actor-critic systems (Methods) were trained on a decision task in which the subject's current and most recent two choices, a_t, a_t−1 and a_t−2, specify the reward probability according to the following task parameters: g₁₁₁ = 0, g₂₁₁ = 0.6, g₁₂₁ = 0.9, g₂₂₁ = 1, g₁₁₂ = 1, g₂₁₂ = 0.6, g₁₂₂ = 1, and g₂₂₂ = 0 (Methods). Curves and dashed lines display the local temporal averages of the rewards earned by the actor-critic systems and the best average rewards obtainable by the maximizing strategy, respectively, in three cases: no state variable (blue); an imperfect state variable s_t = a_t−1 (magenta); correct state variables s_t = (a_t−1, a_t−2) (red).

https://doi.org/10.1371/journal.pone.0003795.g003

To explain the above results, we conduct numerical simulations of a simple alternative task in which the reward probability is given as a function of the current and most recent two choices (a_t, a_t−1, a_t−2) (see Methods). A correct definition of state variables for making choice a_t is s_t = (a_t−1, a_t−2). An actor-critic system (see Methods) operating on the correct state variables earns the globally maximal average reward (Figure 3B, red dashed line). In contrast, for an incorrectly defined state, such as s_t = a_t−1 or no state variable, the best average rewards (magenta and blue dashed lines, respectively) are smaller than the globally maximal one, and the average rewards earned by the actor-critic systems operating on the incorrect state variables (magenta and blue curves) are still smaller.

Thus, the matching strategy is as efficient as the maximizing one if they are combined with a mechanism to explore and select a correct definition of state variables. However, the matching strategy in general deviates from the maximizing one for the choice behaviors restricted by an incorrect definition of state variables.

Summary of assumptions

Our proof of matching law (Eq. 3) is valid for a wide class of natural learning rules, including those employing a widely-used soft-max function for choice probabilities (see below). In the following, however, we explicitly describe the assumptions necessary to make our proof mathematically rigorous. For decision-making tasks, we assumed 1) discrete time step t at which the subject is required to make decision, 2) a finite number of fixed options (a = 1, 2, …, n) available for the subject at every time step, and 3) a scalar amount of reward given to the subject at every time step. For the decision system, we required the following assumptions: 4) the decision system can control the definition of state s_t and the state-dependent choice probabilities {p_as} through a set of synapses w = (w₁, w₂, …, w_m), 5) it adopts a definition of state s_t with which the number of possible states is finite (l), and 6) on a certain definition of state, an arbitrary pattern of possible {p_as} and an arbitrary direction of possible infinitesimal changes {dp_as} can be expressed by some w and dw, respectively. The assumption 6 requires the following condition:(9)where q(w) represents the ln-dimensional vector function consisting of the state-dependent choice probabilities {p_as(w)}, and J(w) is the Jacobian matrix of q(w): J_ij(w) = ∂q_i(w)/∂w_j. Equation 9 requires m≥l(n−1). Independent choice behaviors are generated in the case l = 1.

Decision-making task for demonstrations

To examine the performance of the matching and maximizing strategies, we introduced a decision-making task in which reward is given (r_t = 1) or not given (r_t = 0) to the subject according to the probability determined by the subject's current (a_t) and most recent one or two choices (a_t−1 and a_t−2). Each choice should be taken from one of two options (a = 1, 2), although it is straightforward to extend the present results to more general tasks with more than two options. The conditional expectation value of return on each choice pattern is given as a task parameter: or . The values of these parameters are given in figure legends. For given task parameters {}, we can calculate the maximum of the average reward 〈r〉 = Σ_a,a′,a″g_aa′a″p_aa′a″P(a′,a″), where p_aa′a″ is the conditional choice probability p_aa′a″≡P(a_t = a|a_t−1 = a′, a_t−2 = a″), and P(a′, a″) is the probability distribution P(a′, a″)≡P(a_t−1 = a′, a_t−2 = a″) obtained as a solution of equation P(a,a′) = Σ_a″p_aa′a″P(a′,a″). The best average rewards obtainable by the restricted choice behaviors with state-definition s_t≡a_t−1 and no state variable can be calculated by restricting p_aa′a″ as p_aa′1 = p_aa′2 = p_aa′ and p_a1 = p_a2 = p_a, respectively.

Learning rules for independent choice behaviors

Synapse-updating rules can be described by change Δw_j in w_j at time t, w_j(t+1) = w_j(t)+Δw_j(t). Melioration[16] proposes to increase the choice probability of the option that has the largest expectation value of return. An implementation of melioration is described as p₁(w) = w₀, p₂(w) = 1−w₀, Δw₀ = α(w₁−w₂) and , where α is a positive constant, and if a_t = a, and otherwise. The average returns 〈r|1〉 and 〈r|2〉 are estimated as w₁ and w₂, and the choice probabilities are determined by w₀ updated by the estimated average returns. Local matching[9] is designed to directly achieve the matching law as and . For actor-critic[1], direct actor[23] and Q-learning[1], we used a soft-max function as each choice probability: , where β is a positive constant. Individual updating rules are described as and Δu = α(r_t−u) (actor-critic), (direct actor) and (Q-learning). The details of the algorithms and the relations to the matching strategy and the covariance rule[19] are discussed in Supporting Text S1.

Actor-critic model with state variables

An iterative method to achieve Eq. 7 was shown in [29], [30]. Assuming a set of synapses corresponding to individual options in individual states {w_as} and defining the choice probabilities in each state as , we can obtain the stochastic gradient ascent rule for Eq. 7 as 〈Δw_as〉 = λβP(s)p_as(Q_as−V_s), where λ is a positive constant, and and V_s≡Σ_aQ_asp_as represent the relative values of choosing a in state s (relative action-value) and of state s (relative state-value), respectively. Using the relations and , we can obtain the actor-critic model as an implementation of the matching strategy:(10)where if s_t = s, and otherwise. The variable u estimates the average reward and the variable ν_s represents the state-value of s estimated with the temporal difference (TD) error algorithm. While the actor-critic system is usually designed for maximizing a discounted sum of future rewards[1], the updating rule in Eq. 10 was derived to maximize the average reward[24], [29], [30].

Numerical simulations

In the simulations shown in Figures 2 and 3B, model parameters were set as α = λβ = 0.05, and the initial values of all dynamical variables were set to 1. The value of β was set as β = 4 by default, while it was varied for the Q-learning simulations (Figure 2B). To show the time evolution of reward in Figure 3B, we updated the local average y according to Δy = (r_t−y)/200 from an initial value of 0.64, which is the average reward obtained with even choice probabilities: p₁ = p₂ = 0.5.