A paradigm to probe sensorimotor integration

To probe the role of sensorimotor feedback on thermoregulatory behaviors we designed a paradigm in which larval zebrafish navigate a virtual thermal environment. Larval zebrafish swim in discrete bouts that are on average 100-200 ms long [26]. These swim bouts are separated by stationary periods, the interbout intervals which last 500-1800 ms. We therefore focused our paradigm on modulating the amount of sensory feedback larval zebrafish would receive during the swim periods versus the intervening interbout periods. To accomplish this we used an infrared laser guided by online tracking [23] (Fig 1A). This setup allowed us to present thermal gradients as a fish might experience in nature by modulating temperature according to fish position, as well as presenting changes in temperature that are decoupled from position which would be difficult to obtain in a gradient setup. Using this setup, we modulated the temperature of larval zebrafish according to one of two rules (Fig 1B). In the first case, temperature was coupled to the position of the fish within a 10 cm sized arena in the form of a “thermal plaid”, forming multiple mini-gradients ranging from at the warmest point to at the coolest over a distance of 21 mm (“Plaid” condition). We chose a plaid-like temperature profile for two reasons. Within the plaid, fish will experience both increases and decreases in temperature even for short movement distances and unlike a circular gradient, there is no defined relationship between temperature and distance to the edge of the dish which avoids confounds introduced by larval zebrafish hugging the edge of the arena (thigmotaxis [32]). Fish in the second group explored the same 10 cm arena, however their temperature was controlled by the temperature experienced by a corresponding fish within the first group (“Replay” condition). This difference in the relationship of the stimulus to the location within the arena (Fig 1C) resulted in a difference in sensorimotor feedback between the two conditions, while maintaining the sensory stimulus itself. In the Plaid condition, changes in temperature were coupled to movements enacted by the fish. In the Replay condition on the other hand changes in temperature were uncoupled from behavior (Fig 1D). From the fish’s perspective, the largest change in temperature should occur while moving (i.e., during swim bouts) in the Plaid condition, while temperature changes should be evenly distributed across bout and interbout periods in the Replay condition. This is indeed what we found. The paradigm led to a median temperature change of for each swim bout in the Plaid condition, while the median temperature change was in the Replay condition. Importantly, within comparable time-intervals during inter-bout periods the change in temperature in either condition was (Fig 1E). That is, the temperature change experienced during swims was higher than during stationary periods in the Plaid condition, whereas there was no difference during Replay conditions. Although these changes are small, larval zebrafish use differences on the order of to modulate behavior during gradient navigation [12,23]. This paradigm allowed us to assess whether larval zebrafish change the processing of thermosensory stimuli dependent on the presence or absence of sensorimotor feedback.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A paradigm to test the influence of sensorimotor integration on sensory processing.

A) Illustration of the behavioral setup in which fish are continuously tracked and their temperature is modulated via an infrared laser that is centered on the fish’s head at all times. B) Illustration of the behavioral paradigm. C) Relationship of arena position of the fish to temperature in Plaid (left) and Replay (right) conditions across experiments. D) Illustration of extracted swim bout kinematics and 4.5 s long example trace for a pair of Plaid (left) and Replay (right) experiments showing delivery of the same stimulus but change in sensorimotor feedback. Stimulus temperature top, swim speed in second row revealing structure of larval zebrafish movement as periods of rests with intermittent swim bouts, third row reveals tail movements that are used as sensitive indicators to delineate the starts of swim bouts, bottom row shows fish heading within the arena revealing turns and straight swims; swim periods are marked in purple. We note that the rapid alternations in heading during swim-bouts are caused by the tail beats of the fish. Inset in Plaid stimulus trace shows an enlarged view of the pattern depicted in C with the example movement depicted across D overlaid as a white line. E) For Plaid and Replay conditions the average temperature change experienced by larval zebrafish during swim-bouts (filled purple boxes) and during random time-intervals of the same length (open boxes) within each experiment. Comparison of Plaid within swim-bout vs. random: Rank Sum test, p-value < 0.001; statistic = -7.81; N = 52 fish. Comparison of Plaid within swim-bout vs. Replay within swim-bout: Rank Sum test, p-value < 0.001; statistic = -7.98; N = 52 fish.

https://doi.org/10.1371/journal.pcbi.1013134.g001

Sensorimotor feedback increases the stimulus influence on behavior

When larval zebrafish are deprived of visual feedback during behavior, they strongly suppress motor output, entering a state of learned helplessness [33]. A reason for this might be that when animals move through the world, they generally experience optic flow [34] and e.g., rodents will compute mismatch signals when this expectation is violated [35,36]. The complete absence of visual feedback may therefore signal that behavior is futile since it appears to the animal as if it is not moving. However, whether or not movements lead to temperature changes is strongly dependent on environmental conditions. Within a temperature gradient any swim likely changes the temperature of the animal while swimming in a constant temperature pool does not. The expectation of thermal feedback with movement is therefore conceivably weaker than the expectation of visual feedback. In line with this we found no gross changes in behavioral output based on stimulus coupling as assessed by comparing swim kinematics across the Plaid and Replay conditions. While there was a reduction in interbout intervals less than 1 s in the Replay condition the overall distributions were not significantly different (Fig 2A). Distributions of swim distances and angles turned per swim were highly similar in the Plaid and Replay conditions indicated by the large overlap in distributions (Fig 2B and 2C). Importantly, in both the Plaid and Replay condition, interbout intervals were shorter and swim distances were longer than during baseline conditions in which fish were swimming at a constant temperature of (Fig 2A and 2B). This indicates that the higher temperature increased swim vigor under both conditions. It is therefore unlikely that the absence of thermal feedback in the Replay condition induced learned helplessness.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Absence of feedback does not induce learned helplessness.

A) Density of interbout intervals across Plaid (blue) and Replay (red) experiments during the stimulus phase as well as combined data during the baseline phase (black dashed). Statistics are based on a boot-strapped KS test see Methods. Plaid vs. Replay: p = 0.2548; statistic = 0.0486; N = 104 experiments. Plaid vs. Baseline: p < 0.001; statistic = 0.2283; N = 156. Replay vs. Baseline: p < 0.001; statistic = 0.1897; N = 156. B) Density of per-swim displacements across Plaid (blue) and Replay (red) experiments during the stimulus phase as well as combined data during the baseline phase (black dashed). Plaid vs. Replay: p = 0.218; statistic = 0.0664; N = 104. Plaid vs. Baseline: p < 0.001; statistic = 0.1975; N = 156. Replay vs. Baseline: p = 0.002; statistic = 0.1362; N = 156. C) Density of swim turn angles across Plaid (blue) and Replay (red) experiments during the stimulus phase as well as combined data during the baseline phase (black dashed). Plaid vs. Replay: p = 0.9003; statistic = 0.0121; N = 104. Plaid vs. Baseline: p < 0.067; statistic = 0.0362; N = 156. Replay vs. Baseline: p = 0.109; statistic = 0.0338; N = 156. In all panels shaded error region demarcates bootstrap standard error within each bin.

https://doi.org/10.1371/journal.pcbi.1013134.g002

The increased prevalence of the shortest interbout intervals in the Plaid over the Replay condition (Fig 2A) suggested that sensory feedback enhances swim generation. To understand the basis of this change we were specifically interested in testing for differences in sensory processing and structuring of the behavior. To quantify these two aspects, we sought to extract sensory and behavioral receptive fields. The former describes the transformation of thermal stimuli into swim generation, while the latter quantifies aspects like refractory periods or the influence of swim history on swim initiation. We had previously extracted such receptive fields using white-noise stimuli and generalized linear models (GLM) which revealed the importance of temperature change in guiding swim initiation and which identified a clear refractory period after swimming [23]. Since white-noise stimuli are random and by construction controlled by the experimenter rather than the animal, we could not use them for our current purpose since they would lack sensorimotor feedback. In the current experiments, where larval zebrafish controlled the temperature stimulus, we instead faced the challenge of long stimulus autocorrelation times (S1 FigA) which makes it challenging to extract receptive fields. We therefore used a method based on a convolutional neural network (CNN) we recently developed, Model Identification of Neural Encoding (MINE) [19]. This approach allows us to recover receptive fields even for stimuli with high autocorrelation times, by fitting a CNN relating the stimulus to the response and subsequently using Taylor expansion to extract the receptive field [19]. We previously demonstrated that this approach requires less tuning than receptive field computations based on regularized linear regression and generally yields a more faithful representation of the true receptive field [19]. Here, we designed a CNN to predict the probability of performing a swim bout based on both sensory and behavioral history (Fig 3A). This network received the following inputs as predictors: A one-second history of past temperatures experienced by the fish and a one-second history of previous swim bouts. These inputs were chosen since we and others had previously shown that they influence swim behavior during thermal stimulation and exploration [23,37].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Coupling of sensation and action modulates receptive fields and increases stimulus influence.

A) Illustration of the convolutional neural network that uses the Temperature stimulus experienced over the last 10 seconds and the swim history to predict the probability of swimming. B) Classifier performance as Area under the receiver-operator-curve (ROC-AUC) of classifying individual frames into swim and non-swim frames for naive networks and those trained on circularly permuted data as well as experimental data (N=50 iterations). Comparison of Plaid vs. Replay ROC AUC: Rank Sum test, p < 0.001; statistic = -8.62; N = 50 iterations. C) Coefficients of the thermosensory receptive field versus time before a swim-bout. Model fit on Plaid data (blue), Replay data (red) and control data (black). Errors are bootstrap standard errors across model fits (N = 50 iterations). D) Coefficients of the bout history receptive field versus time before a swim-bout. Model fit on Plaid data (blue), Replay data (red) and control data (black). Errors are bootstrap standard errors across model fits (N = 50 iterations). E) Histogram of changes in bout frequency induced by the thermosensory receptive fields fit on the Plaid (blue) and Replay (red) conditions. These are the expected changes in bouts per second between the most suppressive and most activating stimuli according to the receptive fields. Statistics are based on a boot-strapped KS test see Methods. p < 0.001; statistic = 0.1245; n = 104. F) Scatter plot of the standard deviation in bout-frequency of the thermosensory receptive field effects in each of 52 experiments when using the Replay receptive fields versus using the Plaid receptive field (purple dots). Dashed line is identity, any dot below the line signifies greater modulation by the Plaid than Replay receptive field. Wilcoxon signed rank test p < 0.001; statistic = 0.0; N = 104.

https://doi.org/10.1371/journal.pcbi.1013134.g003

We trained the CNN on 80% of the data after optimizing some of the hyper-parameters on a separate dataset (see Methods and S1B and S1C Fig). To account for potential artifacts due to the long stimulus autocorrelation times, we generated a control condition by rotating the network outputs (generated bouts) with respect to the inputs in the training data. The idea was that this control would allow us to later estimate the noise-floor of extracted receptive fields. After training, we measured how well the networks classified individual frames into those that contained a swim-bout vs. those that did not. Summarizing the results using the area under the receiver-operator-curve (ROC-AUC analysis) showed that in 68% of cases the networks ranked a randomly selected frame with a swim-bout higher than one without (Figs 3B and S1D). This approaches the performance of the GLMs we previously fit on white-noise stimulus data which did so in 71% of cases [23]. Interestingly, the prediction was slightly worse for the model fit on Replay data, which might suggest that behavior is less predictable during the absence of sensorimotor feedback (Fig 3B).

We subsequently extracted receptive fields as a compact representation of how inputs affect the behavioral outputs by differentiating the networks [19]. Here, these receptive fields described the influence of temperature and swim-bout history across time on the generation of the current swim. The thermosensory receptive field (Fig 3C) had a similar structure under both Plaid and Replay conditions, however sensory coupling significantly increased the magnitude of the coefficients (S2 FigA). As expected based on previous results [23], swim initiation was guided by both absolute temperature and changes in the temperature stimulus as evidenced by the mixture of positive and negative coefficients in the receptive fields. Since the coefficients 600 ms before swim initiation were considerably more negative in the Plaid condition (Fig 3C), we expect that sensorimotor feedback enhances the sensitivity to temperature change. The receptive field describing the influence of swim history on swim generation showed a sharper transition out of the refractory period 300 ms before swim initiation in the Plaid condition (Fig 3D). Specifically, swim generation was significantly more suppressed for interbout intervals below 100 ms and significantly more enhanced for interbout intervals between 400–600 ms in the presence of sensorimotor feedback (S2B Fig). Just like the increased predictability of swims during the Plaid condition (Figs 3B and S1D) this suggests that behavior was more structured in the presence of sensorimotor feedback.

The increased magnitude of coefficients in the thermosensory receptive field during Plaid conditions suggested a stronger modulation of behavior by thermal stimuli. To test if this was the case during our experimental conditions, we used the thermosensory receptive field to determine the predicted modulation in swim frequency mediated by the stimulus. Since the receptive field measures how much the stimulus affects the probability of swim bouts, we used the Plaid and Replay receptive fields to compute the swim probability at each time point in each experiment. This allowed us to determine the change from the average bout frequency mediated by the thermosensory receptive field across time. As expected, the stimuli experienced by larval zebrafish during the experiments were predicted to have a stronger influence on behavior in the presence of sensorimotor feedback than in the Replay condition. This can be seen in the overall distribution of changes in bout frequency (Fig 3E). Here, a wider distribution signifies that the stimulus induces a larger range of bout frequencies between times where the stimulus is opposed to the receptive field and times where it is aligned with the receptive field in the Plaid compared to the Replay condition. In line with this observation, the standard deviation of stimulus bout frequency modulation across time induced by the receptive fields is larger for the Plaid than Replay receptive field across all experimental stimuli (Fig 3F). The overall effects of the sensory receptive field on bout frequency were small, but consistent with the expected modulation given the limited thermal range of we tested (see Limitations below). In summary, these results suggest that sensorimotor feedback enhances the influence of the thermal stimulus on behavior generation.

Mixed selectivity neurons represent sensorimotor feedback

We previously identified neurons across the larval zebrafish brain that jointly encode thermosensory stimuli and information about generated behavior [19]. The neurons could therefore encode information about sensorimotor feedback, e.g., by specifically capturing coincidences of temperature change and behavioral actions. To test this idea, we tested whether a linear model could classify Plaid versus Replay experiments based on the activity of the mixed selectivity neurons (Fig 4A). This follows the notion that important information is often encoded such that it can be linearly decoded by downstream neural circuits [38–40]. Since we didn’t record neural activity during our behavioral experiments we made use of the predictive power of previously fit CNN models [19] to generate predictions of how the mixed selectivity neurons would have responded during the 52 Plaid and Replay experiments. We fed the stimuli as well as elicited behaviors (Fig 4B) as inputs into CNN models we previously fit when we identified mixed selectivity neurons [19]. Each model then predicts the calcium response of an individual mixed selectivity neuron. These models were fit under open-loop conditions (akin to the Replay condition). Therefore in the context of the closed-loop Plaid condition, naturalistic neural representations may not be perfectly recapitulated. However, analysis of these neurons suggested that their responses depend on the coincidence of temperature change and behavioral responses, which occurred during the imaging paradigm [19]. This suggests that the responses we identified approximate the true physiological tuning of these neurons which can therefore be replicated by the CNN model. We computed the responses of 1023 nonlinear mixed selectivity neurons across 100 random train/test splits of the behavioral data (Fig 4C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Mixed-selectivity neurons provide a substrate for identifying sensorimotor feedback.

A) Schematic of the classifier approach, showing how stimuli and behaviors are fed through mixed-selectivity neuron CNN models on the one hand and directly used to classify Plaid and Replay conditions on the other. B) Example stimulus and swim behavior during a 60 s period of a randomly selected experiment. C) Corresponding predicted calcium activity of nonlinear mixed-selectivity neurons, clustered according to response correlation for display purposes. D) Classifier performance (as ROC-AUC) of a logistic regression based classifier trained on predicted neural activity (purple) or the inputs across the listed integration times. Dashed line indicates median performance of neural classifier. (N = 100 random train/test splits).

https://doi.org/10.1371/journal.pcbi.1013134.g004

In Costabile et al., 2023, we found that neuronal subtypes were broadly represented in all animals [19]. Consequently, we performed PCA on the complete set of 1023 neurons to reflect the average activity space characteristic of all fish, rather than the precise neuronal composition of any single individual. We then trained a linear classifier on the first 10 principal components across the neural predictions. This classifier therefore answers the question whether the Plaid or Replay condition can be linearly decoded from neural activity in mixed selectivity neurons. Notably, our CNN models represent intervening processing between the inputs (temperature stimulus and behavior) and the activity of mixed selectivity neurons. This nonlinear processing could either make it easier to decode the presence or absence of sensorimotor feedback or obscure this information. In the latter case we would expect that the presence or absence of sensorimotor feedback is easier to decode from the inputs than from the neural activity. Therefore, for comparison, we trained an equivalent linear classifier on the inputs themselves, i.e., the stimulus and the generated behavior. Comparing the performance of the two classifiers then tested whether mixed selectivity neurons represented the information in a manner that is easier to decode by a linear classifier than a classifier operating on the raw inputs as encoded in sensory temperature neurons and motor neurons respectively.

A linear classifier trained on the activity of mixed selectivity neurons was better than chance level in separating Plaid and Replay experiments, indicating that these neurons carry information relevant to detecting sensorimotor feedback (Fig 4D). The comparison classifier trained on input data however failed to match the performance of the neuron classifier unless integration time was extended to 10 s. Importantly, this input classifier was trained to optimally integrate the sensory and behavior information across the 10 s. This argues that mixed selectivity neurons optimally integrate information about stimuli and behavior across time to allow for the detection of sensorimotor feedback by downstream circuits.

Limitations

Here we demonstrate that larval zebrafish adjust the processing of thermal stimuli when their own behavioral actions control the temperature they experience. We suggest that this allows the animal to optimize behavioral output for thermal gradient navigation and hence thermoregulation. To directly address this point, it would be highly desirable to fit a model on the data that explains not only swim generation but also swim kinematics. This would allow to directly compare gradient navigation abilities of models that process stimuli as the fish does in the Plaid versus Replay condition. If the model fit on the Plaid condition would overall remain closer to the preferred temperature, it would strongly suggest that the changes we observe indeed optimize sensorimotor transformations for gradient navigation if there is sensorimotor feedback. Unfortunately, technical limitations prevent us from probing a large enough temperature range to make such a comparison feasible. Our thermal stimuli span a range around the preferred temperature of larval zebrafish (). While this is a large enough range to observe modulation in swim generation frequency (as modeled here), kinematic parameters such as swim distance barely vary within this range. In line with this, an attempt to fit a model that predicts swim distance was unsuccessful. Furthermore, the temperature range is not large enough to model thermal navigation. So even if we could fit models for all swim features, we still would not be able to use them for simulations of thermal navigation. These would require gradients in the 20–30  range [11,12,23] and there would be no guarantee that our current models would generalize over this temperature range.

We currently infer the importance of mixed selectivity neurons through indirect means. In a previous study a CNN model predicted that mixed selectivity neurons encode integrated sensorimotor information, specifically the coincidence of temperature change and behavior. Since these CNN models have high predictive power, we used them here to infer the response of these neurons during Plaid and Replay behavior. However, direct neural recordings during virtual navigation could clarify how these neurons represent sensory motor feedback. Combining functional calcium imaging with a head-embedded version of the thermal plaid paradigm could reveal whether these neurons selectively enhance sensitivity to self initiated feedback. These experiments would directly link the computational role to underlying circuit mechanisms that support adaptive thermoregulatory behavior.

Ethics statement

Animal handling and experimental procedures were approved by the Ohio State University Institutional Animal Care and Use Committee (IACUC Protocol #: 2019A00000137-R1).

Fish strains

All experiments were performed in pigmented offspring of incrosses between mitfa +/-; Elavl3-H2B:GCaMP6s [22] animals.

Behavioral setup and temperature calibration

The behavioral setup used was as described previously [23]. Due to slight modifications in the equipment used, we describe the components again below.

While fish were freely exploring a circular arena with a diameter of 100 mm and a depth of 4 mm, we acquired images at 250 Hz using a Mikrotron 1362 camera (SVS-Vistek GmbH, Germany) utilizing a NI PCIe-1433 frame grabber (National Instruments Corporation, USA). The arena was illuminated from below using an array of 880 nm IR LEDs. Visible light and reflections of the laser were blocked using a combination of three filters: A 25 mm diameter 900 nm shortpass filter (Thorlabs, USA), a 50 mm diameter 900 nm shortpass filter (Edmund Optics, USA) and a 50 mm diameter 750 nm longpass filter (Edmund Optics, USA). Custom written software in C# (Microsoft, USA) extracted the fish position and heading angle in realtime (average time from image acquisition to position: 0.3 ms). The position information was used to send voltage commands via an NI PCIe-6323 DAQ board (National Instruments Corporation, USA) to a set of 6210H Galvos rotating 3 mm diameter X/Y scan mirrors (Cambridge Technology, USA). At the same time the output power of an SDL-980-LM-8000T (Shanghai Dream Lasers, China) infrared laser operating at 980 nm with a maximum output power of 8 W was controlled according to the behavioral paradigm by supplying appropriate voltage commands to the laser current driver. The laser beam was cleaned by first focusing the beam onto a pinhole using a 50 mm focal length lens (Thorlabs, USA) and subsequently collimating using another 50 mm focal length lens (Thorlabs, USA). The beam was then slightly focused using a 750 mm focal length lens (Thorlabs, USA) to a spot diameter of 5 mm at sample, measured using an IR fluorescent alignment disc (Thorlabs, USA).

Temperature calibration was performed as described in [23].

Experimental paradigm

Each experiment was performed in a different fish. Compared to running the Plaid and Replay paradigm in the same fish this had the disadvantage of introducing more noise due to behavioral variability between fish. This choice was made because otherwise the Replay experiments would always need to be run in fish that already went through the Plaid paradigm. In other words, the results could have been confounded by longer-term habituation effects. A “thermal plaid” was presented to the fish rather than a circular gradient, to better decouple the relationship between the temperature experienced by the fish and the distance to the edge of the arena. This was done to mitigate confounds by thigmotaxis behavior in which larval zebrafish track edges. Having multiple mini-gradients allowed excluding all data close to the edge from the analysis (see below).

In the Plaid condition the laser power at sample, and therefore temperature, delivered to the fish was determined by the position within the arena as follows:

(1)

where x and y are the fish centroid coordinates in mm from the top left corner.

This led to a plaid with a period of 21 mm and laser powers at sample ranging from 500 mW to 2000 mW, effectively presenting multiple mini-gradients to the fish. For each fish run in the Plaid condition, one corresponding fish was run in the Replay condition. This fish got an exact copy of the laser powers (and hence temperatures) delivered to the Plaid fish.

Each fish was first habituated to the chamber for 10 minutes and then subjected to a baseline condition with constant power of 500 mW for 20 min followed either by the Plaid or the Replay condition for another 20 min.

Data analysis

All data analysis was performed in Python using Tensorflow [24] and scikit-learn [25].

Receptive field extraction and effects.

Linear receptive fields were extracted as described previously. Specifically, we calculated the derivative of the output of the network (the log probability of emitting a swim-bout) with respect to its inputs (10 s of sensory history and 10 s of previous swim bout ends), which is equivalent to the extraction of a spike-triggered average [19]. We note that the receptive fields show the influence of the inputs on the log-probability of emitting a swim-bout - the effect on probability is non-linear.

To assess the effect of the thermosensory receptive fields on bout frequency, the Plaid and Replay receptive fields were correlated point-by-point with the stimulus presented in each of the 52 experiments (since each Plaid/Replay pair received the same stimulus). Since the receptive field was extracted via Taylor expansion, its effect encodes the change in log-probability relative to the average log-probability of producing a swim. At each point the average log-probability was added and the overall swim-probability was calculated using a logistic transform:

(2)(3)(4)(5)

The resulting probabilities were used to calculate a histogram of swim bout probability modulation around the mean for the Plaid and Replay receptive fields. At the same time, the receptive field effects for each of the 52 stimuli were determined as the standard deviation of bout probabilities induced by the receptive field.

Relevant Python files in repository: utility.py, rf_analysis.py