1 Introduction

In their daily lives, humans are confronted with tasks in complex and time-continuous environments, leading to the development of the ability to quickly react to high-dimensional information with goal-directed actions. While the conventional experimental approach, characterized by its discrete structure and use of low-dimensional stimuli, has offered important insights into human behavior, modeling human behavior in time-continuous environments has been challenging.

In recent years, a new approach has emerged: using deep neural networks (DNNs) in cognitive science to capture real-world processing. DNNs offer a powerful framework for addressing these limitations [1]. Initial studies have demonstrated similarities between DNN representations and fMRI activation patterns in the visual domain of object recognition using a generalized linear model (GLM) with features generated by a DNN [2,3]. Recent investigations suggest that the DNN-based modelling approach can be extended beyond pure perception to tasks involving motor responses. In particular, features generated by a deep Q-network (DQN) have shown their ability to scale to time-continuous, ecologically valid tasks that are often inaccessible to traditional analytical methods [4,5]. DQNs have been created through the use of deep learning models in the field of reinforcement learning (RL) [6]. They have enabled the investigation of how the brain processes time-continuous tasks by parameterizing the process of stimulus-response (S-R) transformations. This is achieved by Q-learning [7]: In each state s_t, an agent selects an action a_t according to a certain stochastic policy π that maps states to actions, and observes a reward generated by the game at time step t to proceed to the next state s_t + 1. For the policy π, the corresponding Q-function of the state-action pair (s,a) is defined as the expected, discounted future reward:

with the discount factor , which determines the importance of future rewards. Deep Q-learning allows the agent to estimate the Q-values and to approximate the optimal policy that maximizes the expected, discounted future reward by primarily following the strategy of selecting the action with the highest Q-value. There is preliminary evidence that human gaming behavior can be modeled using features of a basic DQN [8]. Beyond modeling human behavior, a basic DQN model can also be used to predict human brain activity [5]. Whether the representations learned by DQNs are predictive for human behavior during S-R processing in visuomotor tasks has received limited attention [4,5,9,10]. To address this, we first applied the approach to behavioral data to validate the method itself.

The ongoing rapid improvements in the field of RL have led to more advanced DQN models that solve complex tasks at human-level and beyond [11,12]. Based on the previous results, the question arose as to whether advancements in machine learning can be leveraged to enhance the modeling of human behavior. Arcade games serve as a suitable benchmark for evaluating the performance of DQNs and for studying time-continuous S-R transformations with high-dimensional visual stimuli, as they typically demand immediate actions. Here, we consider three different DQNs: a baseline DQN [13], Ape-X [14], and SEED [12]. All three DQNs possess the capability to acquire the skill of playing arcade games through deep Q-learning. They differ in their structure, training, and model complexity. The baseline DQN is a “vanilla" feed-forward convolutional neural network (CNN) combined with a Q-learning algorithm. The CNN comprises an input layer, followed by a series of hidden layers, namely convolutional layers and fully connected layers, each with a non-linear activation function, and an output layer. Each neuron of the output layer is assigned to a possible action of a game and thus indicates the Q-value associated with the action for an input state. While Ape-X and SEED share some structure with the baseline DQN, they incorporate additional elements in architecture and learning methods that lead to a significant increase in performance. In the case of Ape-X and SEED, a dueling architecture has been integrated. The estimation of Q-values is split into two components: one stream for the evaluation of different actions and another for evaluating a certain state. Moreover, SEED introduces a substantial enhancement to its architecture by implementing a long short-term memory (LSTM), a recurrent neural network (RNN) that incorporates past experiences into decision-making. Ape-X and SEED have implemented additional techniques beyond their architecture to enhance Q-learning. For instance, they employ double Q-learning by utilizing separate networks for action evaluation and action selection. Additionally, they collect more gameplay data, evaluate the data according to its importance, and store it in a shared buffer. Unlike the other two DQNs, SEED does not suffer from the limitation of reward clipping, restricting the received reward to a certain range. We individually trained the baseline DQN and Ape-X to play arcade games and used a pre-trained SEED. The baseline DQN, compared to the other two DQNs, does not show stable learning curves. After sufficient training, Ape-X and SEED achieved performance significantly higher than human-level. In contrast, despite its basic architecture, the baseline DQN achieved game scores close to human levels but slightly underperformed them.

We analyzed behavioral data recorded from subjects playing the arcade games Breakout, Space Invaders, and Enduro. In Breakout, the player must control a paddle to smash a wall of bricks with a ball. In Enduro, a racing game, the player aims to overtake as many cars as possible across different track conditions. In Space Invaders, the player must fight aliens with a spaceship before they reach Earth. These games were selected to encompass a wide range of game types and skills.

We used the trained DQN models as a nonlinear, feature-generating mapping by processing the video screens seen by subjects through the DQNs. We used the features generated by DQNs to predict human responses using a GLM, which fitted the Q-values to human data. We hypothesize that the improvements in performance of Ape-X and SEED translate into better prediction accuracy of behavioral data in the form of human motor responses.

To test this hypothesis, we evaluated and compared the prediction accuracy of the linear model using features of Ape-X and SEED against the linear model using features of the baseline DQN. Additionally, as the experimental tasks were time-continuous, we analyzed the impact of temporal resolution on prediction accuracy by smoothing DQN and human data to varying degrees. We also explored whether an adequate amount of training time for the DQNs is essential for achieving high prediction accuracies. We successfully showed that GLMs with features from a trained DQN are suitable models for predicting human behavior at a fine-grained temporal scale.

2 Results

We investigated the behavior of 23 human participants (15 female, 8 male; mean age: 24.7 years, age range: 20-34 years) playing three arcade games: Breakout, Space Invaders, and Enduro. We recorded gaming data, including motor responses, gameplay videos, which were sequences of observed screens, and the corresponding rewards. The task and the collected data were sampled at 45 Hz. From a technical perspective, the implementation of the task was discrete due to the presentation of individual frames, although from a human perceptual standpoint, the gameplay appeared time-continuous [15]. The participants played each of the games for 7 sessions, with each session lasting 7 minutes of gameplay after they had practiced the game. The analysis consisted of two steps (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. DQN-based linear model.

(A) Behavioral data were collected while subjects played the arcade games Breakout (top), Space Invaders (middle), and Enduro (bottom). The human-generated videos were processed by a baseline DQN (gray), Ape-X (green), and SEED (red). Each neuron of the output layer of a DQN corresponds to the Q-value of a possible type of action, such as ‘no operation’ (‘noop’), move left, move right, fire, and brake. The time series of Q-values were used as predictors in a GLM, while the actual time series of human motor responses served as the predicted variable. For each DQN, the GLMs were fitted to six out of seven sessions, and the Pearson correlation between the predicted human time series and the actual one was calculated on the left-out session. (B) In the preprocessing step, human actions were binary coded and smoothed. A separate GLM was fitted for each type of human action, with the corresponding human time series as the dependent variable.

https://doi.org/10.1371/journal.pone.0338034.g001

In the first step, the DQN models were used for feature generation. Preprocessed human-generated videos of a particular game were processed by the DQNs trained on this game, resulting in a time series of activation values for each neuron in the top layer. Each neuron within the output layer represents a possible action in the game, with the activation serving as Q-value of a video frame. These time series were then used as predictors in a GLM to map the stimulus features to human actions. Before using them in a GLM for predicting human data, the human and DQN time series had to be preprocessed, including smoothing using a Gaussian kernel. For modeling human motor responses, a smoothing kernel with a full width at half maximum (FWHM) of 0.79 seconds proved to be optimal (see Sect 2.2), which was used for smoothing in Sects 2.1 and 2.3. Further details can be found in Sect 4.2.4, and a visualization of these time series is presented in S1 Fig. In the second step, for each DQN, each game, each subject, and each type of action, we employed a separate logistic model to fit the generated features to human data. The time series of Q-values served as predictors, while the time series of human actions was used as the dependent variable. To evaluate prediction accuracy, a 7-fold cross-validation procedure was employed. We calculated the Pearson correlations between the preprocessed, actual human time series and the predicted time series as mean values across all test blocks, subjects, and types of actions. In the following, when we refer to the prediction accuracy of a DQN, we are specifically referring to the prediction accuracy of the GLM model, which utilizes predictors generated by the DQN.

2.1 Improvements in DQNs contribute to improved predictive power

Advancements in deep RL have significantly enhanced the performance of DQNs in playing arcade games. SEED and Ape-X have achieved levels of performance surpassing those of humans [12,14]. However, it remained unclear whether the more advanced DQNs generate features for the GLM that yield higher correlations compared to the baseline DQN. Therefore, we compared the prediction accuracy of human motor responses.

The results show that all three models predicted human behavior significantly above chance level (see Table 1) in all three games.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Results of the one-sample t-test.

https://doi.org/10.1371/journal.pone.0338034.t001

A comparison between the games revealed that all three models achieved their highest prediction accuracy in the game Enduro ( with r_DQN denoting the Pearson correlation coefficient of the time series predicted by the DQN), compared to Breakout () and Space Invaders () (see Fig 2). To assess the impact of advancements in DQNs on predictive power, we used a one-way analysis of variance (ANOVA) with repeated measures to conduct a comparison of prediction accuracy between the baseline DQN, Ape-X, and SEED. There was a statistically significant difference between the three DQNs in Breakout (Fisher’s z-transformed, F(2,40) = 270.46, p < 0.001), Space Invaders (Fisher’s z-transformed, F(1.47, 32.3) = 95.29, p < 0.001, Greenhouse-Geisser-corrected), and Enduro (Fisher’s z-transformed, F(1.30, 28.65) = 158.41, p < 0.001, Greenhouse-Geisser-corrected). A post hoc paired t-test revealed a significantly higher correlation of SEED compared to the baseline DQN and Ape-X in Breakout, Space Invaders, and Enduro (Fisher’s z-transformed, p < 0.001, Bonferroni-corrected). These findings demonstrate that SEED, the most advanced of the three DQNs, provided the most effective feature-generating mapping for the GLM. Furthermore, we observed that the correlation of Ape-X significantly differed from the baseline DQN in the games Breakout and Enduro (Fisher’s z-transformed, p < 0.001, Bonferroni-corrected) but not in Space Invaders (Fisher’s z-transformed, p = 0.062, Bonferroni-corrected).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Prediction accuracies of the models.

Pearson correlation of the three DQNs evaluated for Breakout (left), Space Invaders (middle), and Enduro (right). Error bars represent the 95% confidence interval for the correlation’s mean across subjects. The statistically significant differences between the DQNs (paired t-test, p < 0.001, Bonferroni-corrected) are denoted by ‘*’. Among the three models, SEED exhibited the highest ability to model human actions in all three games.

https://doi.org/10.1371/journal.pone.0338034.g002

2.2 The balance between loss of information and predictive power: The optimal smoothing kernel

To analyze human gameplay behavior, the time series of human motor responses were modeled using the time series of DQN’s Q-values. However, examining individual frames of the time series did not seem meaningful for these kind of games due to the high temporal resolution of 45 Hz. Additionally, the DQN time series had a different structure, jitter, and characteristics due to the method of calculation, compared to those of humans. Hence, as an important preprocessing step, the DQN and the human time series were convolved with a Gaussian kernel with a smoothing parameter σ, which is a measure for the strength of smoothing. Smoothing a time series , the k-th element of the convolution vector is then defined by

for . This weighted averaging, where neighboring values are assigned greater weight based on their proximity, removes high-frequency components from the time series, making them smoother and more stable. In the standard cross-validation procedure of the GLM, we fitted the smoothed predictors to the smoothed human time series, both with the same smoothing parameter σ (see Fig 3A). We found that the smoothing parameter significantly influenced prediction accuracy. In this section, we aimed to quantify the degree of information loss caused by smoothing, thereby determining the attainable temporal resolution when modeling human behavior. Therefore, we also analyzed the prediction of the unsmoothed, noisy human time series (see Fig 3B). For this purpose, we calculated prediction accuracies as functions of the smoothing parameter σ.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The influence of the smoothing parameter on prediction accuracy.

Graphs illustrate the Pearson correlation coefficient concerning the smoothing parameter of the Gaussian kernel . A logistic model was fitted with its smoothed predictors in a cross-validation procedure to the smoothed human time series. Predictive power was assessed through Pearson correlation between the time series being modeled and the smoothed (A) and unsmoothed (B) human time series for Breakout (top), Space Invaders (middle), and Enduro (bottom) across the three DQNs. As the smoothing kernel increased, the prediction accuracy of the smoothed human time series increased (A). Conversely, as the information loss in higher frequencies in the predictors increased, the predictive power of the actual human time series decreased (B).

https://doi.org/10.1371/journal.pone.0338034.g003

When predicting the smoothed human time series in the games Space Invaders and Enduro, it is noticeable that the more information removed in the high-frequency domain (more smoothing), the better the prediction. However, in Breakout the baseline DQN and Ape-X had already achieved the maximum predictive power when smoothing with a Gaussian kernel with FWHM  = 1.57 seconds and FWHM = 1.05 seconds, respectively.

When predicting the original, unsmoothed human time series by smoothing the predictors using a Gaussian kernel with FWHM less than 0.79 seconds for Breakout and Enduro, and less than 0.26 seconds for Space Invaders, the prediction accuracy increased as the smoothing kernel increased. However, using a Gaussian kernel with a larger width led to a decrease in prediction accuracy.

An analysis of the data using fast Fourier transform (FFT) provided further insights into the effects of smoothing on prediction accuracy (see S7 Text). Smoothing the time series of the DQNs and those of humans resulted in a convergence of the frequency spectra of the predictors and the dependent variable. Such alignment may play a significant role in the observed increase in predictive performance. When predicting the original, unsmoothed human time series with its high frequencies, smoothing the predictors led to a divergence of the frequency spectra. This smoothing-induced loss of information in the high-frequency domain of the predictors is possibly the explanation for why the unsmoothed human time series became challenging to model with progressive smoothing. Small Gaussian kernels initially had a positive impact on the predictive power of the actual human time series, as smoothing aligns the different structures within the time series, while also preventing overfitting during model fitting in the 7-fold cross-validation procedure. Overfitting could be attributed to the high frequencies present in the unsmoothed predictors, which captured the inherent noise in the unsmoothed human time series. This further highlights the importance of this preprocessing step.

To address this trade-off between information loss due to smoothing and reduced predictive power due to high temporal resolution, we opted for a Gaussian kernel with an FWHM  = 0.79 seconds for data preprocessing of the human and DQN time series.

2.3 Prediction accuracy of DQNs relies on gaming performance

Before using DQNs as feature-generating mappings, they required several days of training to acquire the ability to play arcade games. We hypothesized that training DQNs on progressively larger amounts of data, achieved through longer training times, would not only lead to higher game scores but also improve the predictive performance of the DQN. Throughout the training process, we stored multiple checkpoints for the baseline DQN and Ape-X, capturing the state of the DQNs at different time points. Since checkpoints were not available for SEED, our analysis exclusively focused on the baseline DQN and Ape-X. At each checkpoint, we evaluated the model’s prediction accuracy using the Pearson correlation coefficient between the actual human time series and the time series predicted by the model at that checkpoint (see Fig 4). We evaluated gaming performance using the average game score achieved by the DQN at that checkpoint.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The influence of the training time on prediction accuracy.

Relationship between the training time of a DQN, the gaming performance of a DQN, and the predictive performance of human time series using features generated by a DQN for Breakout (top), Space Invaders (middle), and Enduro (bottom). Each plot illustrates the learning curves (blue), showing the average game scores achieved in an episode, averaged across 50 episodes. Prediction accuracy is represented by the Pearson correlation coefficient for the baseline DQN (left, gray) and Ape-X (right, green). We calculated the Pearson correlation r_DQN between gaming performance and predictive power as means across all subjects (lower right of each chart). Correlations significantly differing from zero were identified using one-sample t-tests, with ** p < 0.001 and * p < 0.01.

https://doi.org/10.1371/journal.pone.0338034.g004

When examining the score curve trajectory of the baseline DQN, instability in the training process was observed. Therefore, an improvement in prediction accuracy with extended training time could not be expected and was not observed. A linear relationship between the average score achieved in an episode by the baseline DQN and its prediction accuracy was only evident in Breakout. In contrast, Ape-X showed significantly more stable learning curves that increased steadily, with occasional minor downward deviations as training time increased. However, the prediction accuracy was correlated with the gaming performance of Ape-X, confirming our initial intuitive hypothesis, which was particularly evident in Breakout. In Space Invaders, the model achieving the highest average score also reached its maximum predictive power. In Enduro, achieving a steady state in terms of performance did not necessarily correspond to a constant maintenance of prediction accuracy. However, it is somewhat surprising that even networks with low training duration and thus randomly initialized weights already exhibited positive correlations (see S2 Text).

3 Discussion

Modeling human behavior within complex and time-continuous environments has been challenging. However, recent advancements in machine learning, specifically the use of DQNs, have enabled the investigation of how humans process high-dimensional visual stimuli into appropriate motor responses in such environments.

In this work, we aimed to explore a DQN-based modeling approach for predicting human motor responses in complex and time-continuous tasks. We demonstrated that features of DQNs provide a powerful tool for investigating time-continuous human behavior in visuomotor tasks, which has not been feasible with traditional analytical methods [5,8]. Building on these results, the primary objective was to determine whether advancements in deep learning would translate into better predictions of human behavior. We evaluated the prediction accuracy of features generated by two advanced DQNs (Ape-X and SEED) in comparison to those of a baseline DQN when modeling human behavior during gameplay of three arcade games. To enable a comparison between state-of-the-art DQNs and a simpler, more traditional DNN for this research question, we used one of the first DQNs, a breakthrough in 2015, as a baseline due to its simple and fundamental architecture, as well as its human-comparable performance, making it a reasonable reference model [13]. Although it did not achieve strong task performance and exhibited instability during training (see Fig 5), it was sufficient to establish the feasibility of our investigation and, due to its basic architecture, it was ideal for comparisons between the DQNs and previous studies. This makes it all the more intriguing that, despite its simplicity, it achieved predictions above chance level. Our results provide a basis for exploring potential similarities between human and DQN representational spaces, which may in turn offer useful insights into certain aspects of human decision-making. By a comparison of the three network architectures, we were able to investigate the strengths and limitations of this approach, and we examined the importance and potential influence of certain network components, not only on game performance but also on predictive power.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Learning curves of the DQNs.

Learning curves for the baseline DQN (gray), Ape-X (green), and SEED (red). Each plot displays the game score achieved in an episode, averaged across 50 episodes per checkpoint across the training period and the human-level performance, represented by the average game score achieved by the participants per episode (light gray). Ape-X and SEED achieved human-level performance in all three games; the baseline DQN reached human-level in Enduro.

https://doi.org/10.1371/journal.pone.0338034.g005

In recent years, DQNs have experienced a significant increase in power, reaching or surpassing human-level performance in various tasks [11,16]. It seems plausible that advancements in machine learning can be leveraged to improve predictive capabilities [17,18]. Our findings support this hypothesis. We demonstrated that SEED, the top-performing model, generated features that led to the most accurate predictions of human actions. We speculate that potential reasons for variations in prediction performance are mainly attributable to the differences in network architecture. Compared to the complex processes in the human brain, the baseline model and Ape-X being constructed with a feed-forward architecture and SEED featuring a combination of feed-forward and recurrent components, presumably represent significant simplifications [19]. Ape-X and SEED additionally incorporated a dueling DQN architecture to distinguish between the value of actions and the value of the current state [20]. While this innovation might have contributed to the notable performance improvements in playing arcade games, it unexpectedly did not lead to a significant enhancement in the predictive power of the generated features of Ape-X in Space Invaders (see Sect 2.1). Building upon the Ape-X architecture, SEED introduced a major advancement. In contrast to the feed-forward network, SEED, with its LSTM, employed an RNN, equipping the model with a temporal comprehension [21]. RNNs play a crucial role in modeling the human brain [22], for example, studies have shown that RNN components are associated with human working memory [23]. They have also been argued to contribute to the creation of more human-like models [24,25]. To evaluate the impact of temporal context on prediction accuracy, we additionally calculated the accuracy while disabling the pre-trained SEED’s LSTM from storing past game information (see S9 Text). The results show that without temporal context, SEED’s prediction performance was comparable to that of Ape-X and the baseline DQN. This suggests that the temporal context plays a crucial role in the model’s calculations and highlights the importance of this advancement for achieving high prediction accuracy.

Since DQNs first enabled the modeling of time-continuous behavior without manually designed features [5,8], it would also be interesting to extend our hypothesis to a broader range of RL algorithms. The algorithms are categorized as model-based and model-free [26]. In our study, we used the model-free approach with Q-learning as the basis for comparing the three DQNs and their advancements. In addition to Q-learning, policy optimization directly optimizes the agent’s policy. This algorithm has the advantage of applying to continuous action spaces, allowing for more complex tasks, whereas DQNs suit discrete actions [27]. While model-free algorithms require the optimal policy to be discovered through exploration, model-based algorithms have a functional representation of their environment and make decisions based on simulated outcomes. Model-free algorithms require significantly more training data than humans (see Sect 4.2.3). Model-based algorithms can bridge this gap in Atari games, for example, by using stochastic video prediction techniques [28], though their performance remained relatively limited. We assume that similar learning efficiency enables insightful feature-generating mappings. A related approach is theory-based RL [29], which extends model-based approaches by incorporating human-like, abstract models of causal relationships. Since humans use internal models to plan and generalize in complex environments, the finding that the model-based algorithm EMPA can capture similarities with the brain’s model representation while learning Atari games [10] offers another interesting approach for feature-generating mappings. These RL algorithms can also be combined [30] as psychological and neuroscience studies showed their coexistence and interaction [31,32]. Transformer models [33] and multi-task learning [34,35] are expanding RL research, offering new ways to test our hypothesis.

The prediction accuracy relied not only on the DQN but also on the chosen smoothing kernel during data preprocessing. Smoothing with a Gaussian kernel was an essential preprocessing step in our analysis of time-continuous behavior. The Gaussian kernel acts as a low-pass filter, suppressing high-frequency components. This process reduces noise and makes trends more apparent. This step was crucial because conducting a frame-by-frame comparison at a resolution of 45 Hz did not seem meaningful. Despite the discrete implementation of the task, humans perceive tasks at such finely-grained resolutions as time-continuous [15]. However, at this temporal resolution, subjects were unable to perceive individual frames and thus could not execute a conscious motor response to each frame [36]. Furthermore, the DQN time series inherently differed in structure, jitter and characteristics from those of humans, as they exhibited high frequencies, for instance, due to the calculation of Q-values within the RL process (see S1 Fig), as well as the lack of a cost function, which led to unstable action selection and a jittery game flow (see S1 Video). As a result of smoothing, the time series became more stable for predictions, as the high-frequency components were removed. Therefore, in Sect 2.2, we investigated the optimal smoothing kernel to ensure precise and meaningful predictions of human motor responses, while also maximizing the information content of the human time series. The impact of smoothing on the discrepancies and issues mentioned above can be observed in Fig 3. When the human and DQN time series were smoothed in parallel, the frequency spectra of both variables converged (see S7 Text), which could play a crucial role in the observed increase in prediction accuracy. For this reason, we were also able to show that smoothing the jittery DQN time series with an FWHM of 0.79 seconds was necessary to optimally predict the actual human time series, while also preventing overfitting during model fitting. Since predicting high frequencies is generally more challenging, the results demonstrate that smoothing is a valid approach. The correlation peaked for a Gaussian smoothing kernel with FWHM  = 0.79 seconds, demonstrating that human behavior can be modeled on a fine-grained temporal scale. This introduces a novel and important aspect, raising a key question in the analysis of time-continuous studies, which is relevant not only in the context of arcade game environments but also in time-continuous experimental tasks in general.

The use of DQNs represents a modern method for addressing questions in cognitive science [2,3,37]. However, there exists a scientific debate regarding whether DQNs can serve as models for cognitive processes due to the similarities and disparities they exhibit compared to the human brain [19,38]. The degree to which a DQN can effectively serve as a model to explain human behavior and cognitive processes remains also controversial [1,19,39]. In the following, we will address these points, focusing on key aspects relevant to our research.

While the brain serves as a biological blueprint for artificial neural networks, and the networks attain human-level performance across a wide range of tasks, there are various disparities between the human brain and neural networks, such as differences in training behavior and architecture [19,38,40]. For example, humans possess the ability to interpret objects depicted on screens and leverage prior knowledge about the temporal and spatial properties of these objects [41,42]. They could use their knowledge of cars in Enduro, for instance, to efficiently learn the effects of the actions ‘steering right’, ‘braking’, and ‘accelerating’. In contrast, our feed-forward DQNs do not possess these processes of cognitive abstractions inherent in the human brain and must acquire outcomes through S-R learning.

However, DQNs also leverage certain advantages. During the training process, they process information from millions of frames [13] within a few days of training, a task that surpasses the temporal capacity of humans. To compensate for this in our main experiment, participants were given an informational advantage before the training period, including details about the game’s objective, the scoring system, and available actions and their effects in the game. The DQNs were also unaffected by factors like fatigue [43] or response time [44]. Therefore, it would be interesting to equip the DQN with constraints that mimic human limitations and investigate whether this has positive effects on predictive power. For example, restricting a DQN’s response time to human levels resulted in a decrease in game performance [45], but whether this translates to an increase in predictive power is still unclear. In our experiment, adjusting the human time series to match the response times of the DQN through a post-shift of the human time series did not lead to an improvement in prediction (see S6 Text).

All these features of DQNs led to superhuman performance as their complexity increased. Intuitively, one might assume that proximity to human-level performance could also be associated with increased predictive power. However, as human-level performance was significantly lower than that of Ape-X and SEED and comparable to the baseline DQN, the results in Figs 2 and 5 speak against this intuitive assumption. While Fig 4 demonstrates a positive correlation within Ape-X between a DQN’s task-solving ability and its prediction accuracy, it also highlights that upon reaching human-level performance, the prediction accuracy remained significantly below that of a model checkpoint with superhuman performance. Additionally, even highly efficient players, who might be expected to align closely with optimal strategies, did not show higher prediction accuracy when using modern DQNs (see S8 Text). This confirms that SEED’s high predictive power cannot simply be explained by its higher game scores. Achieving human-level scores was not sufficient to generate suitable features for prediction (see Figs 2, 4, and 5). This may be due to differences in the intrinsic processing and calculation of Q-values compared to human behavior. For this reason, we aimed to test this hypothesis empirically in our study. Our results demonstrate that despite the large gap in the game scores achieved by SEED and humans, SEED generated features that closely align with the representation of human behavior.

DQNs are primarily designed to automate complex tasks across diverse domains, focusing on rapid training on large datasets rather than explicitly modeling human brain processes [19,38,46]. Studies have shown that adapting the DQN to aspects such as human visual attention not only enhanced task performance in playing arcade games but also improved action prediction accuracy [47,48]. In our experiment, we trained the DQNs without human data and did not instruct them to emulate human behavior. Although the DQNs we used received the same input as the participants, consisting of grayscale screens, and the DQNs and the human participants shared the goal of maximizing game scores, their gaming behavior in reaching this goal differed.

This became apparent when we visually analyzed the DQN behavior (see S1 Video) and when categorizing the gaming behavior in comparison to that of humans. At first glance, Ape-X and SEED exhibited gaming behavior patterns that resembled those of humans. However, upon closer visual investigation of the gameplay, disparities in the control mechanisms and the selection of actions became evident, deviating from human gameplay behavior. For example, the DQNs did not account for internal costs that subjects associated with executing an action (see S1 Text), resulting in arbitrary movements in “free choice” situations. In contrast, humans showed a tendency to perform no action in such situations [49]. One approach to address this divergence could involve the incorporation of regularization by penalizing the selection of an action other than ‘noop’ with a small negative reward. In Enduro, each action executed at any time step influenced the progression of the game, whereas in Breakout, the paddle’s control mainly affected gameplay only when the ball was near the paddle. Given Enduro’s significantly higher predictive power across all three DQNs, we speculate that implementing the cost function in the DQN algorithms could enhance prediction accuracy. When observing SEED’s gaming behavior, we noticed that it primarily followed specific strategies. For instance, in Breakout, it aimed to target the high-scoring top layer of the wall as early as possible, while in Space Invaders, it consistently aimed at the bonus spaceship. We observed similar strategic behavior in human gameplay. However, this behavior was not observed in the other two models, suggesting that reward clipping represents an additional technical limitation.

Reward clipping is a feature implemented in the baseline DQN and Ape-X to stabilize the training process by restricting the received reward to a certain range, preventing the agent from choosing high-value targets [50]. A whole research field is dedicated to possible modifications of the objective function. While we applied reward clipping as a stabilization technique, other approaches exist to modify the reward signal to encourage desired behaviors and improve learning stability and efficiency [51]. The emerging research area of reward shaping focuses on optimizing the reward function without changing the optimal policy, and these methods have been shown to increase performance in Atari games [52]. We suppose that reward shaping techniques aligned with human behavior and brain function might not only enhance task performance but could also lead to more human-like action patterns, thereby providing a way to empirically test hypotheses derived from theory and potentially improve the prediction of human behavior. One example is an algorithm called Learning Intrinsic Reward for Policy Gradient (LIRPG) [53], which augments DQNs with a learnable intrinsic reward signal optimized alongside the policy. The aim is to find an intrinsic reward that, when combined with the task’s extrinsic reward, stabilizes learning and improves the overall performance of policy gradient agents. This reflects human motivations, which can also be divided into extrinsic and intrinsic components [54,55]. Additionally, analyzing response patterns of the DQNs could offer deeper insights into the internal computations and game strategies that allow SEED to reach a feature space closer to modeling human behavior, improving prediction accuracy. To better understand which aspects of human action selection are captured by the DQNs, a valuable avenue for future research would be to develop an analytical approach for exploring similar state-space representations and examining the response behaviors within them.

While certain proposed adjustments to enhance the human-likeness of DQNs might enhance prediction accuracy, our study did not aim to implement a DQN that mimics human behavior. Rather, our goal was to demonstrate that DQNs can generate stimulus features that serve as suitable predictors in the GLM for modeling human behavior.

When dealing with neural networks, a second debate arises regarding the evaluation of DQNs as cognitive process models and their explanatory properties [1,39]. Can black boxes, such as DQNs, effectively explain other black boxes, such as the human brain? The more complex DQNs become, the more opaque they tend to be. That the information obtained from neural networks can provide relevant insights into cognitive processes has become evident through a growing body of research that has addressed the comparison of representations of stimuli in neural networks with those in the human brain [4,9,19,23]. Particularly in object recognition, studies have explored the similarities between activations in DNN units and fMRI voxel patterns to investigate the processing of visual stimuli [2,3,37]. In our case, when comparing DQNs of varying complexity regarding their predictive capabilities, we aimed to gain insights into the respective strengths and limitations of each approach and the possibility of identifying an additional explanatory facet that could be ascribed to these models [1]. A systematic comparison of the architectures, such as evaluating the predictive power with and without individual components and quantitatively assessing the importance of each exceeds our computational resources, as the DQNs would need to be retrained. Due to our limited resources, replicating this is not feasible for us, even though it could provide exciting new insights. Therefore, we compared and interpreted our results in the context of findings from other studies. Although the explanatory power of this behavioral approach is limited, when using DQN-based models to study S-R transformations in visuomotor tasks, comparing the model’s output to human behavior can make similarities between the internal model dynamics and neural processes more informative. Otherwise, apparent overlaps would be of limited interpretive value, whereas behavioral alignment provides a first validation step for any further interpretation.

However, focusing solely on behavioral data in this study has its limitations. While behavioral data provide valuable insights into behavior, they do not address the underlying neural processes and mechanisms. Incorporating additional neuroscientific measurement techniques, such as EEG or fMRI, could enhance this approach by investigating the temporal and spatial visual brain representations [5,37]. By integrating the hidden layers of a DQN, these methods could offer a fine-grained characterization of the individual steps of S-R transformations, potentially providing deeper mechanistic insights.

DNNs currently offer unique value. In this context, the prevailing direction is characterized by ongoing improvements of these models while simultaneously developing techniques to enhance their transparency [56,57]. Future developments in DNNs may benefit from integrating alternative assumptions, biologically inspired constraints, and objective functions derived from theories of brain function, thereby making them even more human-like [38,58]. All these approaches make DQNs suitable models for processes in the human brain and behavior and such exploratory studies can reveal new regularities and patterns that support theoretical concept formation [1]. Hence, we see great potential in this modeling approach.

The method presented here offers a valuable complement to conventional experiments, characterized by their trial-based study designs and the use of categorical, low-dimensional stimulus spaces. The approach provides higher ecological validity, albeit at the cost of internal validity, by expanding the possibilities for investigating human behavior in time-continuous and complex environments. This DQN-based modeling approach has also been applied to address other questions: Neural networks have also provided detailed insights into neural activations within task-related brain regions while subjects played arcade games, using fMRI data [5,10]. Motivated by our comprehensive validation of the DQN-based modeling approach within a time-continuous framework, an interesting question opens up whether the hidden layers of these advanced DQNs can be used to obtain more accurate predictions of the activations of brain regions involved in S-R transformation during relatively complex behaviors, such as gameplay based on neuroimaging data. We aim to investigate this question in future work.

4 Methods

Supporting information