Fly strains and rearing

Flies were housed under a 12 h:12 h light:dark cycle at 25°C and 60–70% humidity on cornmeal, oatmeal, yeast and sucrose food. For all experiments 3-6 day old males were used, which were starved for 10-12 h prior to testing, while supplying water via a wet cloth. Flies were then transferred to the arena using an aspirator and left in the arena for 2-10 hours. The following strains were employed: Gr5a-LexA (gift from Kristin Scott [23]), LexAop-Chrimson ([21], w1118; P{13XLexAop2-IVS-CsChrimson.mVenus} attP40, Bloomington 55138), Canton S (from A.v.Philipsborn). The flies expressing Chrimson were fed all-trans retinal (ATR, Sigma Aldrich, CAS Number: 116-31-4) for 2-3 days before the starvation period. ATR food was prepared by mixing normal food with ATR to reach a 400 μM solution and then covered with aluminum foil to avoid degradation. Flies fed on ATR food were kept in the dark under aluminum foil cover.

Experimental conditions

Chrimson >Gr5a flies were tested in eight different single-sided stimulation conditions; with 0, 5, 15, 30, 45, 60, 75 and 100% stimulation probability. In a second series of experiments, flies were tested under double-sided stimulation conditions, with 5-5% and 15-15% stimulation probability.

Post-processing of walking data

Walking traces were cleaned of missing data points and jumps in the centroid contrast tracking and filtered with a butterworth filter using a cutoff frequency determined from camera jitter. Next, a trial structure was defined and data from flies with less than 50 trials was excluded from further analysis.

Definition of observables

Stops were defined by speeds below a value of |v| ≤ 0.01mm/s, which is governed by the resolution of our tracking system and corresponds to a movement of less than one pixel between two timestamps. Turns were defined by velocity sign changes since our setup is effectively one-dimensional.

Logistic regression

Regression analysis was performed on return choices against their reward history for individual flies and fly populations by averaging over individuals from the same experimental condition. Due to the binary output variable we used logistic regression. Here a weighted sum of the input vector X_i, i ∈ {1, …, M} that consists of two elements for each trial back i of past rewards and return choices is assumed to be a logit function of the dependent binary output variable y (return choice). To estimate the weights β_i for each element of the reward and choice history, the weighted sum h(x) is computed, (1) and used to define (2) where ϵ is the remaining difference (error) between y′ and the estimate of y′, h. y′ is a continuous latent variable that needs to be mapped to the binary output variable y. Thus, the probability of seeing y = 1 is a logistic function of h, (3) Logistic regression yields estimates of the parameters β_i from the data which can be used to make predictions.

To understand the values of the regression weights and what can be concluded about the fly behavior from them, we generated 100000 element reward vectors with different reward probabilities (5-30%). Under the assumption that the regression weights are determined by how often the flies returned to a stimulation and neglecting any reward correlations, we generated corresponding return choice vectors. The percentage of return choices following a reward was set to approximate “medium responsiveness”, with 50% correspondence. There were no choices on unrewarded trials. The regression weights can be seen in S3 Fig. Logistic regression was implemented using Matlab function glmfit.

Reinforcement learning models

The following reinforcement learning models [6] were applied to the data to identify potential underlying algorithms: a Rescorla Wagner (RW) model [12], a forgetting model where learning and forgetting rates are equal (termed FQ model) and a forgetting model where learning and forgetting happen at different rates (termed ).

The RL models were fit to each individual fly using maximum likelihood estimation with the following log likelihood function (4) c(t) = 1 corresponds to a return choice and c(t) = 0 corresponds to no choice on trial t. The simple RW model has three parameters, α, β and bias, where α is the learning rate, determining the impact of the reward-prediction error, R(t) − Q(t − 1), on the value update, where R(t) is the reward at trial t and Q(t) is the value corresponding to a choice. β is the weighting factor of the value in the choice probability, (5) A bias parameter was included, to account for the fact that the baseline return probability for a fly is below 50%. In this simple model, the value of a choice c = 1 is only updated, when the fly made a choice, and remains constant otherwise (ϕ = 0 in Eq 6). To make the model slightly more realistic, a second RL model, the FQ model, was implemented, where the value of a choice was forgotten, if the fly didn’t make a choice, with the same learning parameter ϕ = α as in the value update equation. The third model had one additional parameter, a forgetting parameter ϕ = α_F, to allow for the more general case of different strengths of the learning and forgetting processes. (6) Every fly was fit with 100 random initializations of these parameter sets for each model and the best parameters were selected by the corresponding highest log likelihood values, ln(L). Subsequently, the Akaike Information Criterion [24] (AIC) score was computed, to select the one that best fits the data, while taking the number of parameters into account.

To allow for predictive testing of the models, only half of every fly data was used to fit the parameter values and the other half was used to predict the flies’ choices. The F₁ score was used as accuracy measure for every fly, (7) (8) (9) with TP the rate of true positives (fraction of trials that the model predicted correctly fly’s return choices), FP the rate of false positives (fraction of trials when model predicted returns but fly did not return) and FN the rate of false negatives (fraction of trials when model did not predict returns, but fly returned).

To test the models’ generative power, 1000 sequences of 1000 trials each for the different experimental probability conditions were simulated.

Place preference depends on optogenetic stimulation probability of sugar receptor neurons

Numerous studies show that the rewarding experience from a specific location in space changes behavior of animals so that they spend more time in that location, in other words, it induces place preference to that location [25–27]. In order to see how place preference changes as a function of experienced rewards we set up a single-fly, closed-loop optogenetic stimulation assay. Flies were food-restricted (10-12hr restriction) and received as a reward the sensation of sweet taste by the brief stimulation of trehalose Gr5a [20] sugar receptor neurons that expressed the Channelrhodopsin Chrimson [21]. In this assay a fly was free to walk in a linear track arena. We defined trigger and reset zones placed at each end of the arena to deliver light pulses conditioned on the fly’s behavior (Fig 1A). Single pulses of optogenetic stimulation with a fixed probability were delivered only when the fly crossed the reset and trigger zones as described in the inset in Fig 1A. Thus, simply staying in the trigger zone did not provide additional optogenetic stimulation. Each fly experienced only one stimulation probability condition and all flies were initially loaded into the center of the arena. Similar to previous studies [28–30], we observed clear effects of the light stimulation on behavior.

First, we looked at how kinematic variables evolved as a function of stimulation frequency. Since our setup can be seen as one-dimensional, we first focused our analysis on the walking traces along the x-axis (long axis). The optogenetic stimulation changed the walking patterns of the flies, from a more symmetrical coverage of the arena shown for one example fly in Fig 1B (magenta traces), to a stimulation zone localized occupancy (Fig 1B, green traces). By testing stimulation probabilities from 0—100%, we showed that place preference (measured by occupancy probability of x positions) positively correlates with the stimulation frequency, as shown for an individual fly (Fig 1C) and for the population (Fig 1D). At 5% stimulation probability, the population occupancy distribution (Fig 1D) was very similar to unstimulated and genetic controls, showing that a low stimulation probability is not sufficient to induce a significant place preference. We observed a temporal decay of the place preference (Fig 1E) with a probability dependent magnitude. This indicated a saturation or behavioural adaptation effect in response to the optical stimulation.

To quantify the flies’ preference for the stimulation side we compared the flies’ zone preference indices across different stimulation probabilities (Fig 1F). Preference indices were computed from the occupancy (time spent in one of the zones) distributions for each zone (within the reset zone boundaries), using (10) which produces preference index values between -1 and 1, meaning strong zone 2 or strong zone 1 preference, respectively, and indifference at PI = 0. There was a positive correlation of the stimulation probability and zone preference index. All stimulated genetic control animals (that did not express Chrimson) had preference indices around 0 and did not show difference (p > 0.05) in their preference for zone 1 vs zone 2, proving that this preference effect doesn’t stem from simple attraction to the light. To further test if flies had developed the preference for one side of the arena during optical stimulation, we performed double-sided stimulation experiments. We observed similar levels of occupancy and place preference with these flies (S2A and S2B Fig).

These results suggest, that flies associate place with the rewarding experience of sugar receptor neuron stimulation.

Optogenetic stimulation triggers local search behaviour

The optical stimulation may affect the place preference by stopping animals movement, by increasing local turning behaviour without linear displacement, or both. We looked at additional kinematic parameters to disambiguate these possibilities.

The optical stimulation decreased speed compared to non-stimulated trials (S1A Fig left and right panel) and increased the staying time duration of the fly around the stimulation zone (S1C Fig). However, we did not see any effect of optical stimulation on the average walking speed of the fly (S1B Fig). We neither saw a clear difference in speed distribution for walking paths toward the rewarded zone (in-walking path) vs away from the rewarded zone (out-walking path) (S1D Fig). Thus, the measured place preference could be due to an increase in the frequency of stop events that happen upon optical stimulation. We defined stops as the speeds below a set threshold level, determined by the resolution of the camera (for precise definition of stops refer to the Materials and methods section). The frequency of stops increased with stimulation probability (Fig 1G). Increase in stop events was accompanied by an increase in turning events around the rewarded location (Fig 1H). Here we treated fly arena as effectively one-dimensional and therefore turns were defined as velocity sign changes that indicate reversal of walking direction. The probability of turns increased upon optical stimulation on the stimulated side, while it decreased for the unstimulated side (Fig 1H). The turn frequency of stimulated flies surpassed turn frequency of the control population at the arena walls (Fig 1H). We also looked at the temporal dynamics of turns on stimulated vs unstimulated trials. S3A Fig shows that turns persisted for longer periods in stimulated trials than in unstimulated trials. The same effect was seen in stimulated vs unstimulated flies (S3A Fig, rightmost panel). Therefore, we concluded that flies respond to optical stimulation by stopping and reversing their walking direction (turning). This behaviour is similar to local searches [31] previously described as a behavioural response to rewarding stimuli [29, 30] and suggested that the stimulation in our experiments was also rewarding.

Next, we asked if local searches exhibited adaptation to the reward probability. We analysed the polar plots of the angular distribution of the walking paths in the reset zone in S3C Fig. We have separated rewarded (solid lines) and unrewarded (dashed lines) trials. The search path of the rewarded fly had a larger variability in turn angles than an unrewarded fly’s path, that only turns at the arena wall. The angular distribution of walking traces were significantly different for rewarded vs unrewarded flies, however we failed to see any significant difference on rewarded trials across different probability conditions (S3C Fig). The same was true for probability conditions of 5 and 15% (not shown here). This could be due to the fact that turns and stops on rewarded trials happen immediately following the optical stimulation and may elicit initial stereotypical behaviour. Therefore, we also looked at the unrewarded trials. The unrewarded angular distributions across conditions were significantly different from the unstimulated control flies (0%). These results suggested that local searches emerge upon reward encounter and they elicit probability independent responses.

Flies return to the rewarded location as a function of reward probability, consistent with a value accumulation process

We reasoned that local searches signal active seeking of rewarded locations and asked if they resulted in animals returning to that location on a longer time scale. To measure this, we split the continuous walking trajectories into discrete trials. We defined a trial to be the time between two crossings of the same reset zone and the accompanying reward zone from the same direction, see Fig 2A. This means that within a trial, the fly visits a reward zone, makes the choice to either return to the same zone again without reaching the other zone (return decision), or to sample the other reward zone before returning. Trials also differed in whether or not the fly was rewarded when it entered the reward zone. In this way, we created a sequence of binary events consisting of a probabilistic reward followed by a binary choice to return or not. Fig 2B shows the return probability for all trials (rewarded and unrewarded) to each zone (one and two) for all tested probability conditions. In all stimulated conditions, there were significantly more returns to the rewarded zone than to the unrewarded zone, which was not the case for the unstimulated and genetic controls (Fig 2B).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Flies return to optogenetic stimulation site.

A Walking trace of one example fly over time in blue. Zone boundaries for reset zone are marked with dashed and for reward zone with dotted lines. Definition of a trial (highlighted trace in black) that consists of travel trajectory from entering the reset and reward zone, exiting the reset zone until entering the reset zone again in that order. Returns are defined as trials with walking trajectories that leave the reward and reset zone and return to the same reward zone, before reaching the other side’s reset zone boundary. Black trace with the curly bracket represents a case for non return behaviour. Two examples of returns are shown in green curly brackets. B Total return behavior per probability condition to zones 1 and 2 (*: p < 0.05; **: p < 0.01; ***: p < 0.001, Mann Whitney U test) for stimulated flies and genetic controls (LexAop-Chrimson/+ and Gr5a-LexA/+) shown as box plots (median, quartiles, full range and outliers in red crosses are shown). Number of animals for each condition is the same as in (Fig 1F). C Return probability versus average reward probability within 30 minute behavioral session. Black line: Pearson correlation, R = 0.5, p = 1e-16 (Robust Correlation package by [32]). D Return behavior on rewarded trials (red light bulb), unrewarded trials (grey light bulb) and unrewarded zone 2. (*: p < 0.05; **: p < 0.01; ***: p < 0.001, Kruskal-Wallis test with multiple comparisons). Same number of animals were used as in (Fig 1F). E Logistic regression against the reward history. Solid lines: Population averages, shaded regions: ± SEM. Only significant coefficients (p < 0.01, computed using t-statistics) are shown. F Logistic regression of the return choices in the 5%, 15% and 30% condition against choice history for 10 trials into the past. Solid lines: Population average, shaded regions: ± SEM.

https://doi.org/10.1371/journal.pone.0239616.g002

The experienced reward rate and set reward probabilities differ due to the stochastic nature of the reward delivery. We also saw a positive correlation of returns with the experienced reward rate (Fig 2C), resulting in correlations between rewards and choices (S4C Fig) for different probability conditions. This result suggests that flies accumulate values over rewarded trials as proposed by a previous study [29]. However, our optical stimulation protocol might sensitize or desensitize activated neurons to subsequent stimulations, thus masking the internal value accumulation process. Alternatively, fictive rewards in the form of optical stimulation may induce a phenomenon described as Caloric Frustration Memory (CFM). CFM is documented to reduce behavioral responses to the non-caloric artificial sweeteners [33]. In either case, we observed a reduction in behavioural responses over time in high frequency optical stimulation sessions. The behavioural reduction of responses to optical stimulation was most pronounced in the 100% stimulation case (S4A Fig) and was absent at lower probabilities both for single (S4A Fig) and double sided stimulation (S2G Fig). When present, these behavioural effects make it look as if animals show diminishing action values. To mitigate this problem most of our subsequent analysis was focused on performance of animals with low (up to 60%) stimulation probability conditions within a 30 min period from the start of the session. We observed that the return rate for a 30 min period drops by 35% (S4B Fig, upper panel), largely driven by flies that experience 100% stimulation condition (S4A Fig). The probabilistic reward delivery allowed us to further minimize the problem of behavioural reduction of responses to high frequency optical stimulation and analyze how returns changed as a function of reward probability, on trials where animals had not been subjected to optical stimulation. Our data shows that as reward probability and reward rate increased, returns scaled up to the rewarded location on non-rewarded trials compared to the non-rewarded location (Fig 2D). Thus, we saw a strong evidence that despite behavioural reduction of responses to high frequency optical stimulation (S4B Fig) flies were accumulating an internal value of actions as a function of reward rates.

To look more directly into value accumulation we used logistic regression analysis to see how rewards in past trials contributed to current choices and computed the past reward effects (reward kernel) on current return choices [34]. We show that immediate rewards (0 trial ago rewards) had the strongest effect on current choices while rewards further back in the trial history had smaller contributions (Fig 2E). The same effects were seen with two-sided optical stimulation (S2E Fig, left and middle panels and S2F Fig). We limited this analysis to low probability stimulation conditions due to behavioural reduction of responses to high frequency optical stimulation as mentioned earlier. Based on our analysis we concluded that flies mostly rely on current rewards to make choices, but also incorporate rewards into their choices that happen further back in trials. A simulated fly that only responds to immediate rewards with 0.5 probability generates reward kernels unlike the ones we see in the animal data (S4D Fig). Namely, it has all zero regression coefficients for past rewards except for the immediate rewards. This is consistent with the idea that flies accumulate reward value over trials.

In some of the reward foraging studies using a probabilistic reinforcement structure [34, 35] not only rewards, but also past choices contribute to the animals’ current choices. This is sometimes termed decision inertia [36]. To test if flies also exhibited decision inertia we regressed current choices on past choices. Please note that regression analysis was carried out using both past rewards and past choices using a generalized linear model as described in Eqs 1–3. This way we could identify independent contributions of past rewards and past choices to current choices (returns). Our analysis failed to detect any effects of past returns on current returns (Fig 2F, S2E Fig, right panel).

Overall, these results show that flies integrate reward history into choices.

Reinforcement learning models that use forgetting and learning rates capture fly behaviour

Regression analysis of rewards and returns revealed that immediate rewards had strongest effects on choices. However, this analysis did not distinguish how flies update the value of chosen vs unchosen options (refer to the Methods section for details). Discounting or forgetting value of unchosen options would be advantageous for animals that forage in natural habitats where the potential number of feeding sites (options) is much higher than two. In such habitats keeping the memory of all visited options may compromize learning the new value of recently visited options. Therefore, flies, like mammals as documented in previous studies [37], may forget values of unchosen options. To see how unchosen option values are updated, we modeled the choice behavior within a reinforcement learning framework by comparing three RL models that use different update rules for unchosen options. One model freezes the value, one forgets the value with the same parameter as the learning rate α and the third forgets the value with a separate forgetting parameter α_F (Fig 3A). In order to compare the different RL models we used two different metrics. First, on each trial we computed the upcoming choices by each model (return or no return) given the reward and choice history that the fly experienced. We next computed the F1 score (see Materials and methods) that balances sensitivity (correct prediction of true positives) and specificity (correct rejection of true negatives) in assessing each model’s predictive power. Second, we compared different RL models using generative tests. For this we first selected each model’s parameters that best explained the observed data (Materials and methods). Next, using these parameters we used each model to generate choices in a simulated environment with two alternative options using the same reward probabilities as experienced by flies. We compared the number of consecutive return choices to the same option that each of our models and flies generated.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Reinforcement learning model captures reward probability dependent returns.

A Schematics of RL algorithm that computes value (Q) and choice (same as return) probability (P(c)) for each trial (t) based on reward (R) history. Q value update on chosen options (c(t) = 1) is controlled by the learning rate α. If choice probability is above a random number (drawn from uniform probability distribution from 0 to 1) the agent returns to the rewarded location (choice equals 1) otherwise it does not. The different RL models differ in how they update the value of unchosen options, controlled by the parameter ϕ, also illustrated in B. B Examples of value evolution over trials for each choice of ϕ. RW model: blue, FQ model: yellow, model: magenta. Dashed lines show average value for each model. C Generative test of the FQ model. Return probability separated by stimulated (red light bulb) and unstimulated (grey light bulb) trials to the rewarded zone per probability condition for the data (colored) and the model (grey). Bars: Population mean ± SEM. D Return choice run length histogram for population 5, 30, 60% fly data (blue) and the model (100 simulations, grey). Bars: Mean ± SEM. Run lengths are defined as consecutive returns to the same side. E Return choice run length generated by FQ model compared to deterministic (det) and random models for different reward probability conditions. The deterministic model generates returns with 0.5 probability when experiencing a reward, while the random model generates returns with probability of 0.5 independent of rewards. For each model Mean ± SEM are are shown.

https://doi.org/10.1371/journal.pone.0239616.g003

Examples of the evolution of the value over trials for all three models are depicted in Fig 3B. To account for the fact that the flies have a baseline return probability below 50%, we included a bias parameter. Model selection using the Akaike information criterion (AIC) score [24], a measure that describes how well a model fits the data by accounting for the number of parameters (S5A and S5B Fig), as well as predictive (S5C and S5D Fig) and generative tests (S5E and S5F Fig) slightly favored the second model which we termed forgetting-Q model, or FQ. The best-fit parameter values show a high variability across flies and similar mean values across experimental conditions (S5B Fig). Predictive testing of the best-fit FQ model on the data yielded rather poor overall accuracy (S5C and S5D Fig). However, F₁ accuracy reached 80% when the model was fit on data that had roughly equal, or higher, numbers of returned to not-returned trials. Nevertheless, under generative testing the model was able to produce similar return probabilities as the flies (Fig 3C) and reproduced return run lengths (Fig 3D). The return run length was markedly different in FQ model that extracted parameters from fly behavior from a deterministic model that responded to rewards with 0.5 probability of returns and a random model that generated random returns independent of reward outcome (Fig 3E). We defined return run lengths as the number of consecutive returns to the same side. The same analysis performed on flies with two-sided stimulation also favored the FQ model (S2H Fig) that showed good predictive (S2J Fig) and generative performance (S2K Fig). However, we observed a much smaller spread of the FQ parameter values (S2I Fig). We think that this discrepancy stems from the data limitation for one-sided stimulation trials.

Overall our analysis favoured the FQ model as the more adequate description of fly behaviour.

RL model explains time dependent nature of returns on first rewarded trials

In standard RL models the value of options starts from zero and is updated according to experienced reward outcomes (Eq 6) (Fig 1A). However, animals may not be completely neutral to environmental or internal cues in the very beginning of the trial and may instead be attracted or repulsed by the mere fact that these stimuli are novel [38]. For example, flies may show some initial attraction to the edges of the arena in the beginning of the assay. In this case the value update process would start from a positive initial value rather than from zero. To gain further insights into the value update process we looked how flies responded to first rewarded trials.

First we noticed that returns on the first rewarded trials were influenced by the reward probability (Fig 4A). One possible explanation could be that the flies were sensitive to the timing of the first reward in the session as mentioned earlier. To further elucidate behavioral mechanisms that drive this phenomenon we looked at the delay of the first rewards from the start of the session using both trial- (Fig 4B and 4E) and time-based analysis (Fig 4C and 4D). Both analyses showed a similar trend of decreased return probability with increased delay. We observed the same effect of the first rewarded trial on return probability when both sides were used to deliver optogenetic stimulation (S2D Fig). The arena on its own may have exerted a somewhat attractive effect on flies as control flies that never experienced rewards showed above baseline level of returns that decreased to baseline (Fig 4D and 4E, insets). This decay was fast as no change in returns were observed on first and later trials (Fig 4C, magenta circles connected with grey line) on a time-scale of minutes. The time course of returns could be also explained by an alternative hypothesis. Namely that flies when introduced to the arena at the beginning of the session have generally increased locomotion resulting in higher turning behavior that ultimately results in increased return behaviour. If so we should have seen the same effects on time evolution of turning behavior. We failed to detect any observable decay on time evolution of turning behavior on non rewarded trials (S3A Fig, lower panel).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. The timing of first rewards affect return frequency.

A Return behavior upon first reward per probability condition. For unstimulated controls (0%, magenta) the return probability was computed for the first trial. (*: p < 0.05; pairwise Fishers exact test.) B First rewarded trial number per probability condition. Black lines: mean. C Return behavior upon first reward within the first minute of recording and after the first minute. Blue circles without a connecting grey line correspond to conditions where the first reward always happened within the first minute. Red line: average return probability. Magenta circles and dark grey line: unstimulated controls. Error bars: ± SEM. (*: p < 0.05; **: p < 0.01; ***: p < 0.001, Kruskal-Wallis test with multiple comparisons and Welch’s t-test). D Fraction of flies (independent of stimulation probability) that returned to a first reward within the first 200s (time bins: 0.03s between 0 and 1s, 0.1s between 1 and 9s, 50s from 10 to 200s and 100s from 300 to 1000s). Inset: Fraction of unstimulated flies that returned for the first time since the session start against time (time bins of 0.1s). Error bars: ± SEM. E Fraction of flies that returned to a first reward within in the first 30 trials. Inset: Fraction of unstimulated flies that returned for the first time against the trial index. Error bars: ± SEM. F Return probability to all other rewarded trials when the first reward happened within the first 3 trials (orange violin) or after the first 3 trials (blue violin). G Return probability to all other rewarded trials depending on whether the fly returned upon the first reward (orange violin) or not (blue violin). (***: p < 0.001, Welch’s t-test) H Two examples for the evolution of the RL value Q (blue curve) over trials. Upper figure: 5% stimulated fly. Lower figure: 30% stimulated fly. Red stems: stimulation events. Red dashed line: average value. I Transparent grey bars: RL model’s prediction of first reward return rate. (Color bars as in A).

https://doi.org/10.1371/journal.pone.0239616.g004

Our previous analysis suggested that the value accumulates as the number/probability of rewarded trials increases (Fig 2D). If so, the timing of the first rewarded trial may have affected the subsequent return probability as the value should be higher for early vs later occurring first rewards. For this we looked at the return probability on all subsequent trials when first rewards happened within the first 3 trials (this number was chosen since the return probability does not change if the fly is rewarded after the 3rd trial (Fig 4E)) or later (Fig 4F). We could not detect a significant difference in return probability between these groups, suggesting that except for the first few trials the behavior of the animal was not affected by the timing of the first rewards. There could be individual differences to novelty that may indicate the flies’ sensitivity to rewards in general. Therefore, we separated flies that showed return on first rewarded trial from flies that did not return on the first rewarded trial and looked at the return probability on all subsequent rewarded trials (Fig 4G). We show that return behavior on the first rewarded trial is a good predictor of future returns and may reflect individual differences among flies.

To formally account for the observed responses of flies on the first rewarded trials we incorporated this in our RL models and assumed that option values (in our case zone 1 and zone 2 of the arena) are not set to zero initially, but rather start with some default positive value that over time decays (Fig 4H). Note, that this simple model qualitatively explains return behavior in control flies that never experienced optical stimulation. We also tested if such RL model could correctly predict flies’ returns to first rewards. The FQ RL model indeed generated similar return probabilities (Fig 4I).

The high return behaviour at the beginning of the session could have been due to the animal’s motivation to escape the arena and not due to the attraction of flies to the novel environment. Because of this flies may have initiated more turns at the beginning of the session. We think this scenario is less likely, although we can not completely rule it out. If animals were in escape or aversive state they would have likely assigned less value to the first experienced rewards and thus showed less returns to those rewards than to the subsequent rewards. Our data showed opposite trends.

The curvature of walking trajectories depends on accumulated value

The experienced reward rate or option values had a clear impact on return behavior in flies. However, due to the binarization of returns this analysis did not reveal how accumulated value affected walking trajectories of flies on a trial by trial basis. We therefore looked at the two-dimensional walking paths of flies as a function of accumulated value.

Previous work [39] has shown that inbound paths of flies to their feeding sites are more straight than outbound paths, suggesting that path integration mechanisms help animals reach their feeding sites using shorter routes. Here we used a similar approach and decided to look at how flies navigated towards and away from their rewarded location as a function of accumulated value. We looked at the angular distribution of walking paths on rewarded and non-rewarded trials as a function of reward probability. Angular distribution was calculated in the following way: For each individual walking x- and y-traces, we compute the angle between the fly’s position at time t (x_t, y_t) and the next time step t + 1 (x_t+1, y_t+1), by computing the arctan between them. To ensure all angles are computed from the same reference vector, the angles are shifted by pi when the orientation of the trajectory changes. The more curved a path is, the more uniform the corresponding angular distribution gets, which translates into a higher angular distribution entropy (Fig 5A). First we noted that out-walking (walking away from the rewarded location) paths generally had slightly higher spread in angular distribution compared to in-walking paths (walking towards rewarded location) (Fig 5B and 5C). This difference did not reach statistical significance. We noted, however, a consistent trend in the reduction of the entropy in angular distribution of in-walking paths as a function of the reward probability (Fig 5D) (p < 0.05 for 5 and 15% reward probability compared with the 100% reward probability). Thus, flies choose to walk more straight paths towards the rewarded locations as a function of accumulated value.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. The accumulated value affects walking direction of flies.

A Sketch of path angular distribution analysis. The “straighter” path 1) is characterized by a narrow angular distribution, while the more curved path 2) has a broader angular distribution. If the distribution is narrowly peaked, it has a smaller entropy S than a broader distribution. B Angular distribution entropy of in- and out-walking paths (box plots with median and quartiles shown), on rewarded (red light bulb) and unrewarded (grey light bulb) trials. For out-walking (away from the rewarded zone, measured from the reset zone) trajectories, there is no difference between rewarded and unrewarded trials. For in-walking trajectories (from the position of return to the reset zone), the rewarded trials have a smaller entropy, corresponding to more straight paths. This is significant for 100% compared to unstimulated controls (*: p < 0.05, Kruskal-Wallis test with multiple comparisons). C Same data (box plots with median and quartiles) as in B but sorted by unrewarded (top) and rewarded (bottom) trials. (*: p < 0.05, Kruskal-Wallis test with multiple comparisons). D There is a trend of more straight in-walking after a reward (indication of path integration) with increased reward probability. Top figure: only rewarded trials from B bottom (*: p < 0.05, Kruskal-Wallis test with multiple comparisons). Bottom figure: Same data plotted against each fly’s experienced average reward rate. Red line: Pearson correlation, R = -0.18, p = 0.005 (Robust Correlation package by [32]).

https://doi.org/10.1371/journal.pone.0239616.g005