Fig 1.
(A) A graph showing the relationships between the rooms in the museum task. The museum is characterized by three different communities or “wings”, each consisting of 5 rooms. Individual rooms are numbered 1–15. (B) “Balanced” action-outcome mappings for each room. Each set of outcomes is randomly assigned to keys <z> and <m> (labelled “i” and “ii”). The same key mapped to the same set of outcomes across the three different wings. Hence, if the participant intended to move out of the current wing into a specific different wing, they needed only to select repeatedly the key leading to the desired wing (labeled as “intention”). Green arrows indicate transitions to the intended wing, orange arrows indicate transitions ending in nodes with green arrows, and red arrows indicate transitions ending in nodes without any green arrows. Note that the probabilistic mappings ensure that, even when the subjects select a response appropriate to their intended direction, the actual transition obtained might or might not be consistent with their intention (see S1 Appendix for more details). (C) Illustration of “preference” in action-outcome contingencies (see Methods). Blue arrows indicate transitions that are present for both actions (as shown in B). Red arrows indicate transitions that are not possible for either action. One room does not have any preferred or removed transitions. This room is labeled with “W” for “wide”. (D) Illustration of an example miniblock. The first image is the goal cue (i). In this example, the participant begins the miniblock in a room with a cactus painting (center; images from ref (112) but illustrations shown here were sourced from openclipart.org), and the goal cue indicates that they must look for the “lamp” painting (top). The participant initiates the miniblock by pressing <space> (ii) (This “start state”, which the participants could anticipate based on the goal-cue (i) step, was excluded from the analyses). Subsequently, the participants were required to select either <z> or <m> to move between rooms (iii, iv) and <space> to indicate that they reached the goal (v), after which they received a reward (points later translated into a financial bonus) (vi). As illustrated, pressing <space> in a non-goal room results in a small penalty (“wrong goal”) (vii). Similarly, pressing <z> or <m> in the goal room also leads to a small penalty (“goal miss”) (iix). The miniblock always ends with either a “goal miss” (iix) or a “reward” (vi) screen, after which the next goal cue (i) is presented.
Fig 2.
(A) Horizontal stacked bar charts for every participant, illustrating the posterior model probability derived from random effects Bayesian model comparison between four cognitive models, “Null” (orange), explicit hierarchical “Exp” (pink), model-based “MB” (yellow), and successor representation “SR” (blue). Participants were sorted by posterior model probability for the “Exp” model for interpretability. (B) Regression coefficient of the “rotation” variable in the explicit hierarchical logistic regression. Density plot shows the full posterior distribution of the population mean, with orange lines indicating the 95% highest density interval. Dots represent posterior means of individual participant (random) effects. (C) Proportion of <m> choices over the course of the testing phase, grouped for every participant by trials where <m> was the optimal rotation vs where <z> was the optimal rotation. Empirical data are shown in blue. Posterior predictions from the explicit hierarchical logistic regression are shown in orange. (D) Regression coefficients for the response switch analysis, describing the probability of switching response key upon “entry” of the goal wing, “leave” of the goal wing, and the number of “repetitions” of pressing the current key. Colors similar to B. (E) Proportion of trials on which the participant switched their response key, grouped by trials where the participant just entered the goal wing or just left the goal wing. Switch proportion of all other (“baseline”) trials is subtracted, so values below 0 reflect a decreased tendency to switch, and values above 0 an increased tendency. Color coding as in C. (F) Given a particular goal outside the current wing, one action corresponds to the correct “rotation” to follow for most of the rooms in the current wing. However, one room in each community actually leads to better outcomes when the participants would follow the opposite action (antirotation, here colored in orange). In this figure, possible outcomes of the rotational action are colored blue, and possible outcomes of the antirotational action are colored orange. As can be observed, the orange room has a preferred transition in the direction of the goal wing (colored with an orange to blue gradient, indicating both choices can lead to this outcome). Interestingly, when following the rotation action (blue transitions), one possible transition leads in the correct direction whereas the other possible transition leads to a room for which the transition in the direction of the goal wing was removed (X in the figure). By contrast, when following the antirotation (orange transitions), the other possible outcome leads to a state that still has a transition in the direction of the goal wing (blue circle). For this reason, model-based and (converged) successor representation models predict participants would pick the antirotational action in this orange room, and the rotational action in the blue room. (G) Proportion of rotational actions chosen in the orange (antirotation) and blue (rotation) rooms (“Node type”, x-axis, colored as in F), considered separately when the correct rotation would be <z> (left panel) or <m> (right panel). Black dots with error bars correspond to mean with 95% confidence interval. (H) Regression coefficients for the (anti)rotation selection analysis, describing the probability of selecting the correct rotation when it is <m> (over <z>, left) and when occupying the blue room (as opposed to the orange room, see F; middle). The interaction term is displayed on the right. Colors similar to B.
Fig 3.
(A) Horizontal stacked bar charts for every participant, illustrating the posterior model probability derived from random effects Bayesian model comparison between four cognitive models, “Null” (orange), explicit hierarchical “Exp” (pink), model-based “MB” (yellow), and successor representation “SR” (blue). Participant data were sorted by posterior model probability for the “SR” model for interpretability. (B) Regression coefficients of the successor representation model. Density plots show the full posterior distributions of the population means, with orange lines indicating the 95% HDI. Dots represent posterior means of individual participant (random) effects. SPE refers to the state prediction error, EV to expected value, and RPE to reward prediction error. See text for definition of remaining terms.
Fig 4.
Posterior predictive checks of the response time successor representation model between communities.
(A) Layout of the museum with transitions colored according to type. The goal painting is colored black and labeled “G”. Green transitions stay within wings that do not contain the goal painting (“outside”). Blue transitions move between non-goal wings (“between”). Orange transitions move from a non-goal wing into the goal wing (“into”). Pink transitions move from a goal wing into a non-goal wing (“out of”). Yellow transitions stay within the goal wing (“inside”). (B) Distributions of regressor values assigned to trials following the different transition types as defined in A. Regressor values were computed based on the discount factor value at the maximum a-posteriori of the joint likelihood of the participant-level successor representation model fit. SPE: state prediction error, EV: expected value, RPE: reward prediction error. (C) Mean (log-transformed) response times (left) and mean posterior predictions (right) for each participant for the different transition types as defined in A. Population means with their 95% confidence interval are shown in black. (D) Mean (orange) and 10th and 90th percentile (blue) of the response time distribution for trials binned by regressor values (as assigned in B) in intervals of 0.3. Binned separately for the different regressors (SPE, EV, RPE, conflict). Conditional response time distributions are repeatedly plotted side by side for real data (left) and the full posterior predictive distribution (right).
Fig 5.
Posterior predictive check of the response time successor representation model within communities.
(A) Layout of the museum with rooms colored according to type. The goal painting is colored black and labeled “G”. Green rooms lie within wings that do not contain the goal painting (“outside”). Blue rooms lie at the boundary between two non-goal wings (“away”). Orange rooms lie at the boundary allowing for a transition into the goal wing (“toward”). Pink rooms lie within the goal wing (“within”). Yellow rooms lie at the boundary allowing for a transition out of the goal wing (“out of”) (B) Mean response times (standardized within participant) in different rooms outside the goal wing, as labeled in A. (C) Same as B, but drawn from posterior predictions of the fitted successor representation for each participant. (D) Same as B, but for rooms inside the goal wing. (E) Same as D, but drawn from posterior predictions of the fitted successor representation for each participant.
Fig 6.
(A) Euclidean distances (normalized) for different pairs of paintings, grouped by whether they were part of the same community and directly connected (community-connected) or not (community-boundary), or whether they were directly connected but not part of the same community (bridge), or not connected and not part of the same community (unrelated). Orange dots with error bars represent mean with 95% standard error. (B) Regression coefficients of the free sort regression. Density plot shows the full posterior distribution of the population mean, with orange lines indicating the 95% highest density interval. Dots represent individual participant posterior means. Note that the “community” effect captures both “community-connected” and “community-boundary” effects as shown in A, and that the “boundary” effect captures their difference. (C) Effects that quantify bias induced by community structure. The “community-direct” effect only investigated directly connected rooms, and asked whether paintings of the same community (community-connected in A) were placed closer together than those of different communities (bridge in A). The “community-indirect” effect is analogous, but investigating pairs of rooms that had a minimal distance of one intermediate room. This corresponds to “community-boundary” paintings in A, and a subset of “unrelated” paintings.
Fig 7.
Modularity of the successor representation model of response times.
(A,B) Relationship between possible discount factor values and “modularity” of the successor matrix after a sequence of states as experienced by two representative participants. Modularity is measured as the ratio of state prediction error for between-wing transitions over within-wing transitions (corrected for “preferred” transitions; see Methods). Notice that A shows a “peak” (orange line) just before 1, whereas B linearly increases with a maximum at 1. These two patterns were ubiquitous in our data set. Not all discount factors lead to modular successor representations (horizontal grey line at modularity = 1). For both these participants, only discount factors above about 0.25 show an effect of community structure. (C) Posterior means of the recovered discount factor parameters for all participants (dots) and kernel density estimate over these. (D) Modularity of the successor matrix for all participants (blue dots) under a null model and under the fitted successor representation model. (E) State prediction errors for all state-action transitions (even those not possible in the actual experiment), based on the estimated successor matrix for an example participant in our dataset (derived from the full posterior distribution; mean estimated discount factor of 0.978). Hotter colors indicate increased prediction error. Both axes index state-action conjunctions, with states labeled 1–15 as in Fig 1A, and actions labeled as (i) or (ii) as in Fig 1B. The community structure is visible as ‘squares’ of decreased prediction error for states (rooms) that are part of the same community (wing). (F) Posterior model probabilities of the successor representation response time model (x-axis) and the modularity measure (y-axis) for all participants (blue dots) with line of best fit (orange). (G) Similar to F with total accumulated reward bonus on the y-axis.
Fig 8.
Modularity correlations with task performance.
(A) Response time slowing (x-axis) for transitions between communities (blue in Fig 4A) compared to transitions within communities (green in Fig 4A), shows a positive relationship with Modularity (y-axis), defined as the ratio of the state prediction error elicited by between-wing transitions over within-wing transitions (corrected for “preferred” transitions; see Methods). (B) Between-community response time slowing (top) and modularity (bottom) show a positive trend and significant positive relationship with acquired total reward bonus respectively. (C) Between-community response time slowing (top) and modularity (bottom) show a positive relationship with the explicit hierarchical “rotation” choice policy. (D) Between-community response time slowing (top) and modularity (bottom) show a positive relationship with action-switching immediately upon leaving the goal wing (“Leave”). (E) Between-community response time slowing (top) and modularity (bottom) do not show a relationship with action-repetition when entering the goal wing (“Entry”). (F) Between-community response time slowing (top) and modularity (bottom) show a negative relationship with distance between paintings of the same community (“Within”) (i.e., the paintings are placed closer together). (G) Between-community response time slowing (top) and modularity (bottom) show a positive relationship with distance between paintings at the boundaries of the same community (“Boundary”). (H) Between-community response time slowing (top) and modularity (bottom) show a negative relationship with distance between paintings connected across communities (“Bridge”). Significance of regression models indicated as *: p < 0.05, **: p < 0.01, ***: p < 0.001. For RT slowing, this significance corresponds to a simple linear regression. For SR modularity, the significance corresponds to a model comparison of a multiple regression model (including both RT slowing and SR modularity as independent variables) against a simple linear regression (including only RT slowing as independent variable).
Fig 9.
Transition types with respect to preferred transitions.
Arrows colored by transition type with respect to the preferred and removed transitions as shown in Fig 1C. Here, preferred transitions are coded as orange (“preferred”) (blue arrows in Fig 1C). These can be the outcomes of selecting both <z> and <m> in their outgoing states. These transitions were included as nuisance regressors in our models, since we expect them to yield faster response times as they are more predictable. It can be seen that only the top node does not have any outgoing preferred transitions, and when referencing Fig 1B, it can be seen this is the only node with 4 possible immediate outcomes (2 distinct outcomes for each action). Hence, this node is referred to as “wide node” and transitions into this node are labeled as such (yellow). Transitions out of this node are labeled as “wide trans” (pink). Both these transitions can be expected to yield slower response times, since the more diffuse nature of transitions out of this node makes them harder to anticipate. Therefore, both these transition types are included as nuisance regressors in our models. It can be seen only one within-wing transition is not preferred and not associated with the wide node (blue, “within”). This transition can be contrasted with the “between” wing transition (green) to yield an unbiased estimate of higher-order surprise-based slowing.