Fig 1.
Network interaction models that could generate localized visual category selectivity.
In all panels, a simplified model schematic of face selectivity in the fusiform face area and posterior superior temporal sulcus (FFA/pSTS, Methods) is depicted as a localized outcome of network interactions. We tested these models by applying activity flow mapping from Cole et al. [27,28] to select held-out “target” regions (including FFA/pSTS). Circles: regions in a network (color legend: bottom [25]). Gray arrows: activity flow processes (Methods), which are weighted (by each region’s connectivity fingerprint) but shown as uniform for visualization purposes. (A) This model predicts that all cortical network interactions (“fully distributed”) converge to generate localized face selectivity. (B) This model predicts that only stimulus-driven interactions with V1 generate face selectivity. We tested this by restricting activity flow mapping initialization to vertex-level data from V1. (C) Extending panel B: This model predicts that stimulus-driven interactions are further shaped by activity flow processes in visual networks. We tested this by simulating the flow of V1-initialized stimulus-evoked activity through the entire visual system. (D) This model predicts that fully distributed interactions (i.e., beyond just the visual system in panel C) are important for generating localized visual selectivity. We tested this by simulating the flow of V1-initialized stimulus-evoked activity through the entire set of cortical brain regions, including recurrent/feedback activity flow. Using mappings initialized in V1 (as in B-C), we assessed how well selectivity was generated when the fully distributed set of network interactions was initialized by stimulus-driven activity flow processes.
Table 1.
Specifications of functional brain areas.
Table 2.
Average MNI coordinates for each functional complex from the literature.
Fig 2.
Activity flow procedure to map task activity in held out brain regions.
(A) Activity flow mapping toy diagram and formula. Task activity for held-out region j (purple) is mapped as the sum of task activity of all other cortical regions, i (coral) (n = total number of regions), weighted by their connectivity estimates with j (gray). (B) Activity flow simulation results (reproduced with permission from [27]) demonstrating that activity flow mapping is most successful when distributed processing mechanisms are high and localized processing mechanisms are low. (C) Example of activity flow mapping with empirical data (steps 1–6) (reproduced with permission from [28]). For a given target region j, estimates of intrinsic (e.g., resting-state) connectivity between j and all other source regions (step 1) are multiplied by all other regions’ actual task activations (step 2). This yields an activity flows map quantifying the contribution of all other regions’ activity flow upon the held-out region, j, for the task of interest (step 3). These are summed to equal the mapped task activity of j (step 4). This procedure is iterated over all regions to generate activity-flow-mapped task activations across the whole cortex (step 5). This is compared with the actual whole-cortex map of task activations via Pearson’s r, MAE, and R2 to estimate accuracy (step 6). Importantly, this approach is flexible to different estimates of connectivity. Source vertices not included in the analyses (10 mm from the target region j; see Methods) are depicted in green.
Fig 3.
Mapped activations across all HCP conditions yielded response profiles for four functional complexes of interest.
(A) Causally confounding graph patterns in standard FC estimation methods (adapted with permission from [127]). CombinedFC incorporates both bivariate and multiple regression measures such that confounders, chains, and colliders are accounted for. (B) The cross-participant (n = 176) average resting-state connectivity matrix (estimated via combinedFC) of 360 MMP regions [56], ordered along each axis per the Cole Anticevic brain-wide network partition (CAB-NP [25]; color-coded on each axis to match panel C). This was the functional network organization utilized in the present study for activity flow mapping. Note that our implementation of combinedFC used multiple regression as the final step, and therefore FC estimates were given by beta coefficients (see Methods). (C) Cortical schematic of the CAB-NP and its 12 functional networks from Ji et al. [25], reproduced with permission. (D) Response profiles (across all 24 HCP conditions) of four complexes of interest to the present study (indicated along the y-axis; note that “r” stands for right hemisphere); mapped versus actual (left and right respectively; mean across participants depicted in each panel). Black boxes highlight the n-back conditions that maintained visual semantic category embeddings germane to a given functional complex (e.g., 0-back bodies and 2-back bodies for the right EBA and right FBA). Activity-flow-mapped response profiles were highly accurate, suggesting that mapped activation patterns of the functional complexes of interest were reliable across multiple cognitive domains.
Fig 4.
Visual categories and identified functional complexes.
(A) Exemplar stimuli from each of the four visual semantic categories shown to participants across the n-back task (bodies, faces, places, and tools). Please note that the face example is a photograph of the author of this manuscript, meant to be representative (but not an exact match) of the original face stimuli while protecting privacy concerns of the original models. In each category, stimuli varied to sample a wide array of representations. Body images included whole bodies (no faces) and isolated body parts (excluding nudity); face images included diverse ages and facial expressions; place images included indoor and outdoor scenes (and combinations, e.g., a patio); and tool images included distinct items (e.g., plier versus drill) as well as similar variants (e.g., two drills: one white, one blue). See Barch et al. [47] and www.humanconnectome.org for further details. (B) A schematic of the cortical surface in three gross orientations, with MMP regions [56]. outlined in silver and functional complexes relevant to processing body features (see Tables 1 and 2, Methods) colored in based on CAB-NP network affiliations ([25], Fig 1C) and identified with text labels and arrows. EBA = extrastriate body area; FBA = fusiform body area. VIS2 = secondary visual network; DAN = dorsal attention network. “r” denotes right hemisphere; “l” denotes left hemisphere. (C) Functional complexes related to face processing. FFA = fusiform body area; pSTS = posterior superior temporal sulcus; LAN = language network; DMN = default mode network. (D) Functional complexes related to places and scenes. RSC = retrosplenial complex; PPA = parahippocampal place area. (E) Functional complexes related to processing images of tools and objects. LOC = lateral occipital complex.
Fig 5.
Distributed activity flows account for the majority of localized selectivity to bodies.
(A) The activity flow mapping procedure (steps match Fig 2C) generating body responses in EBA/FBA (black), projected onto cortical schematics (right hemisphere). Green: source vertices excluded from analyses. Step 2 was not blacked out for visual comparison with step 4, however it was held out in-analysis. Step 4 color scale shows maximum of all regions’ mapped activations to body images for visual comparison with step 2. (B) The connectivity fingerprint [13,15] of the right EBA/FBA via rsFC (black lines). Radial lines: source regions connected to EBA/FBA, clustered by functional network assignments [25] (colored per legend; Fig 3C). 95% confidence interval: across participants. (C) Right EBA/FBA body selectivity: activity-flow-mapped (purple) and actual (coral). Gray dots: individual participants’ scores. Dashed line: no selectivity (1.0); used for comparison in statistical tests. Significant t-statistics are indicated with an asterisk (p<0.00001; see Methods). (D) Estimated contribution of distributed network interactions to body selective responses in EBA/FBA. (E) The activity flows (as in A3) of each source region contributing to mapped EBA/FBA responses to body images (statistical significance asterisks at network-mean level). (F) Variance explained by each network-restricted activity flow model (unmixed partial R2 in gray) of EBA/FBA’s response profile. Black lines: 95% confidence interval across participants. Asterisks: statistical significance versus each other network. E-F suggests that activity flowing over interactions with VIS2 and DAN represents a general network coding mechanism for EBA/FBA. See Methods for full details of each analysis.
Fig 6.
Distributed activity flows account for the majority of localized selectivity to faces.
All figure specifications follow Fig 5. (A) The activity flow procedure mapping activations to face categories, projected onto cortical schematics (right hemisphere). Right FFA/pSTS was the held-out target complex. (B) The connectivity fingerprint [13,15] of the right FFA/pSTS via whole-cortex rsFC (black lines). Radial lines: source regions connected to the FFA/pSTS, clustered by functional network assignments [25] (colored per legend and Fig 3C). (C) Face category selectivity exhibited by the right FFA/pSTS. Significant t-statistics are indicated with an asterisk (p<0.00001; see Methods). (D) Estimated contribution of distributed activity flow processes to face selectivity exhibited by the right FFA/pSTS. (E) Activity flows (as in A step 3) of each source region contributing to the mapping of FFA/pSTS responses to face images. VIS2 regions contributed most to the FFA/pSTS mapped activation magnitude to faces. (F) Variance explained by each network-restricted activity flow model (unmixed partial R2 via dominance analysis; Methods) of the right FFA/pSTS’ response profile. VIS2 accounted for the most variance, altogether suggesting that activity flowing over VIS2 regions represents a general network coding mechanism for FFA/pSTS processing. DAN and DMN regions also accounted for a nontrivial amount of variance at the response-profile level suggesting that, across diverse cognitive domains, FFA/pSTS processing is impacted by activity flowing over DAN and DMN regions, in addition to VIS2 (in the face-specific case).
Fig 7.
Distributed activity flows account for the majority of localized selectivity to places.
All formatting and figure specifications follow Fig 5 (A) Activity flow mapping of activations to place categories, projected onto cortical schematics (right hemisphere only). Right PPA/RSC was the held-out target complex. (B) The connectivity fingerprint of the right PPA/RSC, as in Passingham et al. [13] except with whole-cortex rsFC (black lines). Radial lines: source regions connected to the PPA/RSC, clustered by CAB-NP functional network assignments [25] (colored per legend and Fig 3C). (C) Place category selectivity exhibited by the PPA and RSC in the right hemisphere. Significant t-statistics are indicated with an asterisk (p<0.00001; see Methods). (D) Estimated contribution of distributed activity flow processes to the emergence of place selective responses in the right PPA/RSC. (E) Activity flows (as in A step 3) of each source region contributing to the mapping of PPA/RSC responses to place images. VIS1, VIS2 and DAN contributed most to the right PPA/RSC mapped activation magnitude to place categories. Note that VIS1, VIS2, and DAN were all statistically greater than each other network, except for each other. (F) Variance explained by each network-restricted activity flow model (partial R2 via dominance analysis; Methods) of the right PPA/RSC’s response profile. VIS2, DAN, and DMN accounted for the most variance, suggesting that activity flowing over regions in these networks represents a general network coding mechanism for PPA and RSC processing, while VIS1 contributes to place-specific responses.
Fig 8.
Distributed activity flows account for the majority of localized selectivity to tools.
All formatting and figure specifications as in Fig 5. (A) Activity flow mapping of activations to tool categories in the held-out target—the right LOC—projected onto cortical schematics (right hemisphere). (B) The connectivity fingerprint of the right LOC, as in Passingham et al. [13] except with whole-cortex rsFC (black lines). Radial lines: source regions connected to the LOC, clustered by CAB-NP functional network assignments [25] (colored per legend and Fig 3C). (C) Tool category selectivity exhibited by the LOC in the right hemisphere. Significant t-statistics are indicated with an asterisk (p<0.00001; see Methods). (D) Estimated contribution of distributed activity flow processes to the emergence of tool selective responses in the right LOC. (E) Activity flows (as in A step 3) of each source region contributing to the mapping of LOC responses to tool images. VIS2 contributed most to the right LOC mapped activation magnitude to tool categories (F) Variance explained by each network-restricted activity flow model (partial R2 via dominance analysis; Methods) of the right LOC’s response profile. VIS2 accounted for the most variance, suggesting that activity flowing over regions in these networks represents a network coding mechanism for LOC processing.
Table 3.
Empirical (true) resting-state connectivity fingerprints generate visual category selectivity significantly better than null model connectivity fingerprints.
Fig 9.
Other regions associated with visual category responses in the literature do not drive the fully-distributed model results.
(A) While our procedure for identifying functional complexes from the literature was systematic (see Methods), it was possible that excluded, category-responsive regions were driving the results in Figs 5–8. Such regions were excluded from functional complexes because they were either: (1) not supported by 10 or more peer-reviewed studies at the time of preparing this manuscript, (2) studies used experimental stimuli too distinct from the n-back visual categories, or (3) studies did not provide spatial information systematically consistent with standard volumetric and/or surface-based topography. These regions, held-out from source sets in this control analysis, were as follows (from left to right in panel A): the dorsal visuomotor stream for body selectivity; the occipital face area (OFA), inferior frontal gyrus (IFG), and orbitofrontal cortex (OFC) for face selectivity; the transverse occipital sulcus (TOS) for place selectivity; the intraparietal sulcus (IPS) and middle frontal gyrus (MF) for tool selectivity. Colors of regions are consistent with the functional network assignment used throughout all other figures. (B) For each body, face, place, and tool (left to right) selectivity analysis, individual participant’s (dots) visual selectivity scores were maintained between the results in Figs 5–8 (x-axes) and the same analyses with the regions in panel A held-out from the source set. This suggests that other category responsive regions in the literature did not drive the fully-distributed model findings.
Fig 10.
Activity flows directly from V1 are sufficient to generate visual category selectivity.
(A) Theoretical schematic of stimulus-driven activity flow processes generating visual category selectivity (as in Fig 1B). Given prior literature, mapped activity flow processes (gray arrow) have a refined inference: from V1 to later visual regions, we inferred that activity flow processes were primarily stimulus driven. This contrasts with the prior whole-cortex models, which also included top-down and likely recurrent influences. (B) Activity flow mapping procedure for the stimulus-driven model, conducted at the vertex level. V1 sources were used to map targets across VIS. Note that the usage of “step 1” serves as a prelude to later steps tested in an extended visual system model (Fig 11). (C) The null connectivity fingerprint (rsFC) model used for control analyses. The top depicts the true VIS network, and the bottom depicts pseudo-random (edge degree and strength preserved) network architectures over 100 permutations. For visualization purposes, networks are shown at the region level, but analyses were conducted at the vertex level. (D) Actual (coral) and mapped (purple) visual category selectivity exhibited by the right EBA/FBA, FFA/pSTS, PPA/RSC, and LOC (left to right). Category selectivity exhibited by V1 (for each respective image category) is shown to demonstrate that activation patterns in V1 alone do not account for mapped visual category selectivities (V1 selectivity scores were all nonsignificant; see main text for statistics). Most importantly, these results demonstrate that the activity flow mapping process increased the category selectivity of every functional complex relative to V1 activity patterns, despite source activity originating solely from V1. Category selectivity generated by V1-initialized activity flow processes were significant in each functional complex (see main text for statistics). Significant t-statistics are indicated with an asterisk (p<0.00001; see Methods); see Figs 5–8 for significance of actual category selectivity of functional complexes.
Fig 11.
Visual category selectivity generated by activity flow processes over the extended visual system.
(A) Stimulus driven activity flow processes, further shaped by later visual interactions, generate localized visual category selectivity (Fig 1C). From V1 to later visual regions, we inferred that activity flow processes are stimulus driven (Figs 1B and 10). Within the visual system, we inferred that activity flow processes are bidirectional and/or recurrent. (B) Activity flow mapping procedure for the extended visual system model. All steps were conducted at the vertex level. Step 1: V1 sources were used to map targets across the visual cortex (VIS) (Fig 10). Step 2: mapped VIS activation patterns from Step 1 (weighted by connectivity estimates as in all activity flow models; see Methods) were used as sources to map held-out targets across visual cortex. Steps 3+: step 2 was repeated until a settling threshold was reached – or the point at which mapped values stopped changing (see Methods). (C) The settling threshold was reached at step 3. All further analyses only included steps 1–3 from panel B. (D) Accuracy of mapped activation patterns across all conditions (left: an average of all visual regions; right: functional complexes studied herein). Across visual cortex, explained variance tended to increase with each step, indicating that the extended visual system model was improving mapped response profiles across the cortex. This pattern was also observed across functional complexes, with some exceptions, such as the PPA/RSC (which appears most accurate at step 2). (E-H) Actual (coral) and visual-system-mapped (purple) category selectivity (see Methods) exhibited by the right EBA/FBA (E), FFA/pSTS (F), PPA/RSC (G), and LOC (H). Mapped category selectivity is shown for each step in panel B. Activity-flow-mapped category selectivity estimates were not consistently significant (see main text for statistics), suggesting that extending V1-initialized activity flow processes across the visual system alone does not stably generate localized visual category selectivity. In panels E-F: significant t-statistics are indicated with an asterisk (p<0.00001; see Methods); see Figs 5–8 for significance of actual category selectivity of functional complexes.
Fig 12.
Adding fully distributed network interactions to stimulus-driven activity flows further enhances category selectivity.
(A) A schematic of V1-initialized activity flow processes that are further propagated across all cortical network interactions (as in Fig 1D). Here, fully distributed network interactions are initially established by activity flowing over the connectivity fingerprints of each functional complex with V1. We used the multistep mapping procedure of Fig 11B (but with only two steps), except for at the region level (Methods). (B) Mapped (purple) and actual (coral) body selectivity exhibited by the right hemisphere EBA/FBA (gray dots: individual participants; boxplot line: median). Statistical significance is reported in the main text. (C-E) Same as B, but: (C) face selectivity in FFA/pSTS, (D) place selectivity in PPA/RSC, and (E) tool selectivity in LOC. Across all functional complexes, stimulus-driven + fully distributed mappings generated visual category selectivity that was greater than the stimulus-driven alone model of Fig 10 and closely matched (and in some cases improved upon) the fully distributed alone model of Figs 5–8. This suggests that activity flow processes initialized in V1, that are further processed over all cortical network interactions, are capable of generating highly accurate visual category selectivity, with reduced causal confounds (see main text for full rationale). Panels B-D: significant t-statistics are indicated with an asterisk (p<0.00001; see Methods); see Figs 5–8 for significance of actual category selectivity of functional complexes.