Dynamic integration of forward planning and heuristic preferences during multiple goal pursuit

Selecting goals and successfully pursuing them in an uncertain and dynamic environment is an important aspect of human behaviour. In order to decide which goal to pursue at what point in time, one has to evaluate the consequences of one’s actions over future time steps by forward planning. However, when the goal is still temporally distant, detailed forward planning can be prohibitively costly. One way to select actions at minimal computational costs is to use heuristics. It is an open question how humans mix heuristics with forward planning to balance computational costs with goal reaching performance. To test a hypothesis about dynamic mixing of heuristics with forward planning, we used a novel stochastic sequential two-goal task. Comparing participants’ decisions with an optimal full planning agent, we found that at the early stages of goal-reaching sequences, in which both goals are temporally distant and planning complexity is high, on average 42% (SD = 19%) of participants’ choices deviated from the agent’s optimal choices. Only towards the end of the sequence, participant’s behaviour converged to near optimal performance. Subsequent model-based analyses showed that participants used heuristic preferences when the goal was temporally distant and switched to forward planning when the goal was close.

In the case of the flat prior the approximate posterior factorises as follows where d denotes the number of free parameters.
[ Note that in the above plot we have a good estimate of the biases, but very noisy estimates of other parameters after 200 iterations. In practice, one would expect that a non-hierarchical model has more noiser estimates and potential exhibits slower convergance towards the free energy minimum, compared to hierarchical parametric models.
Here, we will define and test a hierarchical parametric model, where we will use the so-called horseshoe prior as the prior of hyper-parameters. The horseshoe prior assumes that each parameter might have different prior uncertainty, which is independent of the specific subject (all subjects share the same prior uncertainty for a specific parameter). In addition, we will also assume that each parameter is sampled from a different prior mean, where we also assume that the prior mean is defined on the group level, hence  ELBO(η, γ) where ELBO(η, γ) denotes negative variational free energy (evidence lower bound) expressed as To define the posterior distribution over model parameters and hyper parameters, we follow the fact that the the true joint posterior contains dependencies between free parameters. To account for this we will approximate the posterior in the form of the multivariate normal distribution, which is factorised over global hyper parameters, and local parameters for each subject. Hence, We express the approximate posterior over group parameters (⃗ µ, ⃗ σ) as In the case of the subject specific posterior we assume the factorisation between subjects and parameter dependence within subject, hence we express each factor as a multivariate normal distribution In summary, we are performing an approximate inference based on approximate posterior by maxmising evidence lower bound with respect to prior parameters η and posterior parameters γ.
[ Finally we will test the horseshoe+ prior which in addition to the global hyper prior over model parameters contains local (subject specific) hyper priors over prior uncertainty where we introduced another set of subject specific parameters γ n of the hierarchical prior. Hence, η = (m 1 , . . . , m d , s 1 , . . . , s d , γ 1 , . . . , γ N ).
Similarly, as for the horseshoe prior we will use here the approximate posterior that factorises on joint distribution over hyperparameters, and joint distribution over subject specific parameters. Hence, where we use the same functional form as the above for the posterior over hyperparameters and the same functional form for the subject specific posterior in the case of the horseshoe prior, and as p(⃗ x n , ⃗ ρ n |γ ′′ n ) = 1 ρ n 1 . . . ρ n d N 2d (⃗ µ n w , Σ n w ) Note that in both cases we used centered parameterisation of the parametric model, as non-centered parametrisation leads to a numerical instabilities when computing gradients over free model parameters.