2.1. Reinforcement learning

Reinforcement Learning (RL) represents a foundational pillar of artificial intelligence (AI), with its core focus on enabling agents to learn optimal decision-making strategies through interactions with their environment [30–32]. Unlike supervised learning, where labeled data guides the model, RL agents rely on trial-and-error processes to maximize cumulative rewards, forming policies that dictate their behavior. Rooted in the Markov Decision Process (MDP) framework, RL has evolved significantly, giving rise to diverse methods capable of addressing the increasing complexity of real-world tasks [33,34].

The theoretical basis of RL lies in the Bellman equation, which recursively defines the optimal policy by breaking down the problem into sub-problems. Early algorithms, such as Q-learning and SARSA (State-Action-Reward-State-Action), utilized tabular methods to approximate the value of state-action pairs [35]. While effective in simple, discrete environments, these methods encountered scalability issues when applied to problems with large state-action spaces due to their reliance on maintaining exhaustive state-action tables. The advent of function approximation marked a turning point in RL research. By replacing tables with parameterized functions, such as linear approximators or neural networks, RL methods could handle larger and more complex environments [36]. Temporal Difference (TD) learning, an essential component of modern RL, introduced methods for updating value estimates by combining observed rewards with bootstrapped estimates of future rewards, bridging the gap between Monte Carlo methods and dynamic programming.

The integration of deep learning into RL, commonly referred to as Deep Reinforcement Learning (DRL), has revolutionized the field by enabling agents to process raw, high-dimensional inputs such as images and videos. Deep Q-Networks (DQN) exemplify this paradigm shift [37,38]. By employing convolutional neural networks (CNNs) to process raw pixel inputs, DQN agents achieved superhuman performance on a range of Atari 2600 games. This breakthrough illustrated the potential of end-to-end learning, where agents directly map sensory inputs to actions without requiring hand-crafted feature engineering. Subsequent advancements refined the DQN architecture to improve stability and efficiency. Double DQN mitigated the overestimation bias inherent in Q-learning by decoupling action selection and evaluation. Dueling DQN introduced a two-stream architecture to separately estimate state values and action advantages, enhancing learning efficiency in environments with many redundant actions. Prioritized Experience Replay improved sample efficiency by prioritizing transitions with higher learning potential, addressing the uniform sampling limitations of standard replay buffers.

Despite their success, DQN-based methods are primarily designed for discrete action spaces, limiting their applicability to continuous control tasks. This limitation spurred the development of algorithms such as Deep Deterministic Policy Gradient [39] (DDPG) and Twin Delayed DDPG (TD3) [40], which extend the capabilities of DRL to continuous domains by incorporating actor-critic architectures. Policy gradient methods represent another major class of RL algorithms, focusing on directly optimizing the policy rather than deriving it from a value function. The policy is parameterized by a neural network, and the optimization objective maximizes the expected cumulative reward. The introduction of Trust Region Policy Optimization [41] (TRPO) and Proximal Policy Optimization [42] (PPO) addressed challenges in policy optimization, such as instability and divergence. TRPO constrains updates to ensure monotonic improvement, while PPO simplifies this process by employing a clipped surrogate objective, balancing exploration and exploitation more effectively. PPO, in particular, has become a preferred choice for many RL tasks due to its simplicity and robust performance. Its ability to handle both discrete and continuous action spaces, combined with ease of implementation, has established it as a standard benchmark in RL research.

Traditional RL methods, while effective in specific tasks, often struggle with generalization and require extensive retraining when transitioning to new environments. Meta Reinforcement Learning (MRL) addresses this limitation by introducing mechanisms that enable agents to “learn how to learn.” Instead of training policies from scratch, MRL leverages prior experience to accelerate adaptation to novel tasks. This paradigm is particularly advantageous in environments characterized by dynamic objectives or rapidly changing conditions. Algorithms such as Model-Agnostic Meta-Learning (MAML) [43] and its RL variants have demonstrated the efficacy of meta-learning in diverse settings. MAML optimizes initial policy parameters such that a few gradient steps suffice to adapt to new tasks.

Despite these advancements, RL methods face several challenges that hinder their applicability in real-world scenarios. Sample inefficiency remains a significant issue, as most algorithms require extensive interactions with the environment to achieve satisfactory performance. This inefficiency is exacerbated in tasks with sparse rewards, where agents struggle to identify meaningful signal patterns to guide learning [44,45]. Another limitation lies in the scalability of RL to high-dimensional, continuous state-action spaces. While actor-critic methods address this to some extent, they often suffer from instability during training, necessitating careful hyperparameter tuning and engineering. Furthermore, generalization remains a persistent challenge. Agents trained on specific tasks frequently fail to adapt to unseen environments, highlighting the need for more robust and flexible learning frameworks.

The proposed MRL-STDP model aims to address several limitations of traditional RL and DRL approaches by integrating meta-learning capabilities with biologically inspired synaptic tuning mechanisms. Unlike DQN, which relies on fixed Q-value approximations, the MRL-STDP model dynamically adjusts its policy through STDP-based synaptic updates. This real-time adaptability is particularly advantageous in rapidly changing environments, where fixed policies struggle to maintain performance. Compared to PPO, which optimizes policies through constrained updates, MRL-STDP introduces an additional layer of plasticity by incorporating synaptic timing correlations. This mechanism allows the model to fine-tune its decision-making at a more granular level, potentially leading to faster convergence and higher adaptability. Additionally, the MRL-STDP framework addresses sample inefficiency through its meta-learning component, which leverages prior knowledge to reduce the exploration burden in new tasks. This capability distinguishes it from traditional methods like DQN and PPO, which often require extensive retraining to achieve similar performance in novel environments.

2.2. Atari game

The Atari 2600, released in the 1970s, has emerged as a foundational platform for evaluating reinforcement learning (RL) algorithms in the field of artificial intelligence [46–51]. The suite of games available on the Atari 2600, each offering distinct mechanics and visual characteristics, presents a broad and challenging range of environments to test RL algorithms, as depicted in Fig 2. This diversity establishes Atari as an optimal testbed for evaluating the versatility and resilience of these models.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. In-game screenshot from ‘Combat’, a classic game on the Atari 2600 console.

This image illustrates the characteristic, minimalist visual style of early video games, with two tanks navigating a maze-like environment. At the top of the screen, the player’s scores are shown: ‘2’ for the green tank and ‘0’ for the white tank. This game is often used in reinforcement learning (RL) research to assess and refine algorithms designed to navigate and compete within simple, static environments. The game’s straightforward graphics and clear gameplay mechanics offer a controlled environment ideal for evaluating the adaptability and decision-making skills of RL models.

https://doi.org/10.1371/journal.pone.0320777.g002

The image above comes from Combat, a timeless game for the Atari 2600, showcasing the minimalist yet recognizable visual style of early video games. It features two tanks navigating a maze-like environment, with the player scores displayed at the top: ‘2’ for the green tank and ‘0’ for the white tank. This game serves as a benchmark in reinforcement learning (RL) research, helping to develop and test algorithms capable of performing in constrained and static environments. The game’s simple graphical design and direct mechanics offer a controlled space for assessing the adaptability and decision-making abilities of RL agents.

The importance of Atari games in reinforcement learning (RL) research was emphasized in the groundbreaking paper [52], which introduced the Deep Q-Network (DQN), a model that achieved superhuman performance in multiple Atari games [53]. This study was a landmark achievement, as it showed that end-to-end learning, from raw pixels to actions, was possible through deep learning techniques, establishing a new benchmark for reinforcement learning (RL) algorithms. By employing a convolutional neural network to process pixel data and directly generate game actions, it demonstrated that RL could effectively handle raw sensory inputs in complex and dynamic environments [54]. This approach significantly enhanced the generalization ability of RL models, enabling them to perform effectively across various Atari games, a feat that traditional methods had previously struggled to achieve.

Despite its successes, RL faces several limitations when applied to Atari games, particularly in terms of sample inefficiency and the difficulty of training agents to handle high-dimensional sensory inputs. Traditional RL algorithms, while effective in controlled settings like Atari, demand extensive interactions with the environment to achieve optimal policies [55]. The challenge is further exacerbated by the fact that, while Atari games offer variety, they remain relatively simple compared to real-world tasks, which raises concerns about the scalability of RL algorithms to more complex, high-dimensional problems. Recent research has sought to mitigate these issues through techniques like Double Q-learning [56], Dueling DQN [57], and Prioritized Experience Replay [58], all of which aim to improve the stability and sample efficiency of DQN in Atari environments.

In addition, model-based RL approaches have been investigated as a solution to these limitations, allowing agents to simulate and plan future actions more efficiently, thereby reducing the need for extensive exploration in real-world environments [59]. Despite these advancements, RL agents continue to struggle with more realistic, dynamic environments, emphasizing the need for further research into more robust and sample-efficient methods.

Hypothesis 1: The combined MRL-STDP model achieves faster convergence compared to traditional RL models.

This hypothesis is grounded in MRL’s ability to utilize prior knowledge for initializing parameters, thereby reducing the number of episodes needed to reach optimal performance. Let T_c denote the number of episodes required for convergence, where T_c is defined as the episode at which the total reward R(t) stabilizes within a margin ε:

(1)

It is hypothesized that:

(2)

Hypothesis 2: STDP integration enhances fine-grained decision-making in dynamic environments by dynamically adjusting synaptic weights.

STDP introduces a biologically inspired rule for updating synaptic weights w_ij based on the relative timing Δt of neuronal spikes:

(3)

where A₊ and A₋ are scaling factors, and τ₊, τ₋ are time constants. We hypothesize that this rule enhances the model’s capacity to prioritize actions leading to rewards, resulting in improved performance metrics such as cumulative reward R and decision latency L:

(4)

Hypothesis 3: The MRL-STDP model generalizes better across different tasks than single-method RL models.

The meta-learning component enables knowledge transfer, while STDP fine-tunes responses to new environments. Let ΔR denote the performance drop when transitioning from training games to novel test games:

(5)

We hypothesize:

(6)

The theoretical insights on Meta Reinforcement Learning (MRL) and Spike-Timing-Dependent Plasticity (STDP) highlight their complementary potential in addressing key limitations of traditional reinforcement learning. MRL’s capability to rapidly adapt through meta-learning synergizes with STDP’s biologically inspired synaptic adjustments, providing a robust framework for dynamic and complex environments. These hypotheses lay the foundation for a methodological design that rigorously tests the combined MRL-STDP model’s efficacy in enhancing learning efficiency, decision-making precision, and task generalization. The subsequent methodological framework ensures that these hypotheses are systematically evaluated, guiding the exploration of the proposed model’s adaptability and performance across varied scenarios.

In this study, Meta Reinforcement Learning and Spike-Timing-Dependent Plasticity are integrated to enhance the adaptability and learning efficiency of AI agents. MRL enables agents to learn how to learn by leveraging past experiences to adapt to new tasks quickly, significantly improving learning efficiency in dynamic environments. On the other hand, STDP fine-tunes synaptic weights based on the precise timing of neuronal spikes, offering a biologically inspired mechanism that optimizes the learning process at a local level. This tuning process increases the agent’s ability to adapt to short-term changes and refine its decision-making in real time. The combination of these mechanisms informs the experimental design by providing a dual approach to learning: MRL facilitates fast task adaptation and cross-game generalization, while STDP optimizes the learning process by fine-tuning the agent’s behavior through neural plasticity.

4.1. Problem definition, environment setup and preprocessing

The primary goal of this research is to tackle the challenge of training AI agents to perform optimally across multiple Atari 2600 games, utilizing the frameworks of Meta Reinforcement Learning (MRL) and Spike-Timing-Dependent Plasticity (STDP), as illustrated in Fig 3. This involves a thorough setup of the learning environment and careful preprocessing of game data to ensure that the input to the AI models is both rich in information and conducive to learning sophisticated strategies.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Schematic of a meta-learning system integrated with a recurrent neural network (RNN) for adaptive learning.

The diagram shows two primary components: the meta-learning system (on the left) and the subordinate system (on the right). The meta-learning system includes a subordinate system that incorporates a recurrent neural network to process inputs x(j) and previous outputs , And the meta-learning system incorporates a supervisory component with a fixed learning algorithm (Backpropagation Through Time - Real Time Recurrent Learning, BPTT-RTLR). This component oversees the learning of the subordinate system, which produces predictions. Meanwhile, the subordinate system on the right, featuring an adjustable model and learning algorithm, fine-tunes its parameters according to the current input x(j) and the preceding label . The interconnection between these systems emphasizes continual adaptation and feedback-driven modification of learning parameters, facilitating dynamic learning and prediction refinement over time.

https://doi.org/10.1371/journal.pone.0320777.g003

This research addresses the challenge of enabling AI agents to quickly adapt and generalize across different gaming environments without the need for extensive retraining. In traditional reinforcement learning, agents often become overfitted to specific tasks or need prolonged training periods to perform well across varied scenarios. By integrating MRL and STDP, this study aims to improve the agent’s ability to learn efficiently from fewer interactions and adapt its strategies using both pre-existing knowledge and dynamic synaptic changes.

The primary platform for this study is the Atari Learning Environment (ALE), available through OpenAI Gym. ALE provides a standardized API that facilitates direct interaction with a collection of Atari games, which serve as the tasks for the AI agents. These games differ in both their dynamics and visual complexity, ranging from the simple “Pong” to the more intricate “Montezuma’s Revenge.” For each game, the environment returns frames of 210 × 160 pixels using a 128-color palette, which the agent must interpret to make decisions. To reduce the computational burden and focus learning on relevant features, the raw frames are preprocessed by reducing dimensionality and simplifying color complexity. Specifically, each frame is converted from RGB to grayscale, eliminating the third color dimension and simplifying the input data using a standard luminance model.

(7)

where R, G, and B are the red, green, and blue channel values of the pixels. After conversion to grayscale, each frame is resized to 84 × 84 pixels using bicubic interpolation, which smooths the image and preserves the quality of visual information. The choice of 84 × 84 pixels is a balance between maintaining sufficient spatial resolution to capture necessary game details and minimizing input size to optimize computational efficiency.

To capture the temporal relationships between frames, which are essential for recognizing motion and changes in the game state, a stack of four consecutive frames is utilized as the state representation for the agent. This stacking approach enables the model to detect movement and transitions in the environment, offering a more comprehensive input than a single frame alone. Consequently, the state tensor has dimensions 84 × 84 × 4, integrating both spatial and temporal information, this preprocessing pipeline plays a crucial role in enhancing the learning process. It significantly affects the efficiency and effectiveness of training, as it enables the model to focus on key features from a simplified input. This optimization reduces the computational load and accelerates the overall training time.

4.2. Meta Reinforcement Learning Implementation

The application of Meta Reinforcement Learning (MRL) in Atari game environments involves designing a model capable of quickly adapting to the distinctive challenges of different games without requiring extensive retraining. A key innovation in our approach is the combination of a meta-policy network with a base-policy network. This dual-network framework utilizes the fast adaptability of MRL to enhance the performance and efficiency of traditional reinforcement learning models.

The meta-policy network acts as a high-level controller that generates the initial parameters for the base-policy network, based on the current state of the game. The architecture of the meta-policy consists of two fully connected layers: the first layer maps the processed game frame inputs to a hidden representation, and the second layer outputs the parameters that initialize or modify the base-policy network. Specifically, the input to the meta-policy network is a vector with dimensions corresponding to the pre-processed game frames. Specifically, the input to the meta-policy network is a vector of dimensions 84 × 84 × 4 (representing stacked frames for capturing motion) which is processed to reduce dimensionality and color complexity, thereby focusing the network on relevant features for decision-making. The hidden layer comprises 256 neurons, which has been found to offer a robust compromise between computational efficiency and capability to capture complex patterns.

The base-policy network interacts directly with the Atari environment by taking the current game state as input and outputting action probabilities. It also includes two fully connected layers with ReLU activations, introducing necessary non-linearity to allow for the learning of complex strategies. The output layer uses a softmax function to convert the raw output into a probability distribution over possible actions, directing the agent’s decisions in the game. The output dimension corresponds to the number of possible actions, which typically ranges from 4 to 18, depending on the specific Atari game.

Training involves a coordinated interaction between the two networks. The meta-policy network generates parameters that initialize or adjust the parameters of the base-policy network at the start of each game or episode. This approach enables the base-policy network to begin each game with strategies informed by past experiences, speeding up the learning process. The meta-policy network’s parameters influence key aspects of the base-policy network, such as the weights of neuron connections. The updating formula for the parameters of the base-policy network in the context of meta-learning is given by:

(8)

where θ_base are the parameters of the base-policy network, θ_meta are the parameters output by the meta-policy network, S_t is the current state, f represents the function mapping meta-policy outputs to base-policy parameter updates, and α is the learning rate governing how quickly the base-policy adapts.

The meta-policy is trained through a meta-learning strategy, where its effectiveness is gauged by the performance of the base-policy across a series of games and episodes. This is accomplished by optimizing a loss function that captures the cumulative rewards achieved by the base-policy, ensuring that the meta-policy’s goals align with sustained success across various games.

4.3. Integration of spike-timing-dependent plasticity (STDP)

The incorporation of Spike-Timing-Dependent Plasticity (STDP) into neural networks for Atari game environments marks a key development in enhancing reinforcement learning (RL) approaches. STDP, inspired by biological learning mechanisms, adjusts synaptic strengths based on the relative timing of neuronal spikes, enabling a more flexible learning process that can potentially improve the adaptability and efficiency of RL agents in complex and dynamic environments, such as video games.

In our approach, STDP is implemented within a multi-layer neural network, where synaptic connections at each layer are modified according to the timing differences between pre-synaptic and post-synaptic spikes. This adaptation process is governed by custom parameters that adjust the network’s behavior. Built using PyTorch, the architecture consists of several fully connected layers, each followed by a ReLU activation function to ensure non-linearity. The first layer receives an input vector from the preprocessed Atari game frames, typically of dimension 84 × 84 × 4, representing stacked grayscale frames to capture spatial and temporal information, as shown in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Illustration of a dynamic system model used in time series analysis.

The diagram depicts the interaction between input variables x_t and context variables c_t across time steps t = 1: T_p within a single period p from 1 to P. The variable y_t represents the output at each time step, influenced directly by x_t and c_t. Additionally, the output is modulated by the hyperparameter h_p, indicating a parametric influence on the process. The function f(·) at the top represents a transformation applied to the outputs y_t over infinite time steps, aggregating them into a final result. This framework is typically employed in computational models where time and context-sensitive processing is required, such as in recurrent neural networks or dynamic systems simulations.

https://doi.org/10.1371/journal.pone.0320777.g004

Each connection within the neural network is associated with an additional STDP parameter, which is initialized close to zero and constrained to small values. This ensures that the updates to the synaptic weights are gradual and do not disrupt the stability of the learning process. The update rule for STDP is applied as follows:

(9)

where Δw is the change in weight, η is a small learning rate specific to STDP (typically set to 0.0001 for stability), Δt represents the difference in activation timing between neurons, and α is a scaling factor to modulate the effect of timing differences. According to this rule, the synaptic weight is increased when the pre-synaptic neuron fires before the post-synaptic neuron, signifying a possible causal relationship with the action taken. If the firing order is reversed, the synaptic weight is reduced, thereby strengthening connections that consistently result in favorable outcomes.

4.4. Training loop and network optimization

In the context of reinforcement learning applied to Atari games, the training loop and optimization of the neural network are essential for building efficient AI agents capable of mastering complex tasks. This section outlines the advanced techniques implemented in Python using PyTorch, highlighting the distinct features of our approach, including the network architecture, hyperparameter tuning, and dynamic modifications throughout the training process, as illustrated in Fig 5.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Graphical representation of the optimization process in machine learning, focusing on the adjustment of weights during gradient descent.

The graph plots the cost function J(w) against the weight parameter w. The curve shows how the cost function decreases to reach a global minimum J_min(w), indicating the optimal weight where the cost function reaches its minimum. The initial weight is marked to show its position relative to this minimum. The dashed line represents the gradient of the cost function at the initial weight, highlighting the direction and speed of the steepest descent towards the global minimum. Arrows along the gradient path demonstrate how the optimization algorithm adjusts the weights to reduce the cost function. This diagram emphasizes the iterative use of gradient information to refine model parameters for enhanced performance.

https://doi.org/10.1371/journal.pone.0320777.g005

Our neural network, named AtariNet, is specifically designed to process the spatial-temporal dynamics of Atari game frames. The network is structured with three convolutional layers followed by two fully connected layers. The convolutional layers are configured to progressively decrease the spatial dimension while increasing the depth, which is essential for capturing hierarchical features from raw pixel data. The first layer uses 32 filters of size 8 × 8 with a stride of 4, the second layer uses 64 filters of size 4 × 4 with a stride of 2, and the third layer uses 64 filters of size 3 × 3 with a stride of 1. This configuration allows the network to effectively learn from the 84 × 84 pixel input frames, representing four stacked grayscale frames of the game environment.

To train the model, we use an epsilon-greedy strategy for selecting actions, which strikes a balance between exploration and exploitation. The epsilon value, which controls the probability of choosing a random action, begins at 1.0 (completely random) and gradually decays exponentially until it reaches a minimum of 0.1 (10% random), as described by the decay formula:

The exploration rate ε is defined using an exponential decay model to balance exploration and exploitation during training:

(10)

where ε_start =1.0, ε_end =0.1, and the decay rate is calibrated to ensure that exploration decreases adequately over the course of training.

The network utilizes a replay buffer to store transition tuples containing state, action, next state, and reward. This helps reduce the correlation between consecutive training samples, thereby stabilizing the learning process. Training batches are randomly sampled from the buffer, which holds the 10,000 most recent transitions. Each batch, consisting of 32 transitions, is used to update the network weights through backpropagation. The loss function employed is cross-entropy loss, modified by the discount factor γ = 0.99 for future rewards:

(11)

where action_prob is the probability of the taken action as predicted by the network.

To prevent the problem of exploding gradients, which frequently occurs in deep neural networks, gradient clipping is applied during backpropagation. This technique restricts the gradients to a maximum norm of 1, ensuring that the weight updates remain stable and do not cause instability in the network.

(12)

where g is the gradient computed for each parameter during backpropagation. The training loop is conducted over 1,000 episodes, with the network saving checkpoints every 10 episodes. These checkpoints enable periodic evaluation and adjustments of the model, offering a way to revert to a previous state if needed, which helps prevent negative impacts from undesirable updates during training.

5.1. Baseline comparisons

We assessed the performance of each algorithm through a comprehensive set of metrics. The average total rewards per episode provided a direct indication of the agent’s ability to optimize its score based on its decisions. The “episodes to convergence” metric was used to evaluate the speed at which each model reached a threshold level of performance, offering insights into learning efficiency. Moreover, the computational resources required, expressed in GPU hours, were tracked to gauge the operational costs of training, which is vital for understanding the scalability and practical deployment of the models, as shown in Table 1 and Fig 6.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Performance comparison of different algorithms in Atari games.

https://doi.org/10.1371/journal.pone.0320777.t001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Performance comparison of different algorithms in Atari Games.

https://doi.org/10.1371/journal.pone.0320777.g006

The results for the MRL+STDP model were highly encouraging across all evaluation metrics. In Breakout, for example, the model achieved an average of 430 total rewards per episode, outperforming DQN (350), Double DQN (380), and A3C (400). This superior performance can be attributed to the STDP mechanism, which enables dynamic adjustments to synaptic strengths based on temporal differences in neuron activation, allowing the agent to better respond to critical environmental cues. Additionally, the MRL framework facilitated swift adaptation, promoting efficient transfer of learning across various game scenarios. As a result, the MRL+STDP model required only 250 episodes to converge, compared to the 300-550 episodes needed by other models.

The learning efficiency of the MRL+STDP model was further highlighted in Space Invaders, where it not only achieved significantly higher rewards (250 vs. 180-225 from other models) but also converged faster, with only 320 episodes compared to 380-500 for the other algorithms. This demonstrated the model’s ability to generalize effectively in games that require both quick reflexes and strategic planning.

As shown in Table 2, MRL-STDP achieves faster convergence than both DQN and PPO. The agent requires approximately 1500 episodes to reach optimal performance, whereas DQN takes around 2000-2500 episodes and PPO takes about 1800-2200 episodes. MRL-STDP outperforms both DQN and PPO in terms of performance. It shows a 40% improvement in reward efficiency, compared to 20-30% for DQN and 25-35% for PPO. In terms of computational resources, MRL-STDP is the most efficient, requiring around 50-60 GPU hours. DQN is more resource-intensive, requiring 60-80 GPU hours, while PPO needs 55-70 GPU hours. This lower computational cost suggests that MRL-STDP is more optimized for use in resource-constrained environments, without sacrificing performance. Memory usage is relatively consistent across the models, with MRL-STDP using approximately 4.5 GB, which is slightly lower than DQN’s 5-6 GB and comparable to PPO’s 4.8-5.5 GB. While memory consumption for all models is similar, MRL-STDP’s slightly lower memory usage gives it an edge in environments where memory capacity is limited. Training time is correlated with convergence speed. MRL-STDP reaches optimal performance in approximately 12-15 hours, while DQN requires 15-20 hours and PPO 14-18 hours. The reduced training time for MRL-STDP further emphasizes its efficiency in adapting to new tasks. The effect size for MRL-STDP is the highest at 0.8-0.85, indicating that the reward improvements are more significant compared to DQN (0.55-0.65) and PPO (0.6-0.75). The 95% confidence interval for MRL-STDP is [0.82, 0.88], which is narrower compared to DQN ([0.5, 0.6]) and PPO ([0.58, 0.72]), indicating that MRL-STDP exhibits more stable and reliable performance across trials.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of MRL-STDP with DQN and PPO.

https://doi.org/10.1371/journal.pone.0320777.t002

Similar improvements were observed in Pong and Ms. Pac-Man, where the MRL+STDP model consistently outperformed other algorithms in terms of both rewards and convergence speed. In Ms. Pac-Man, where long-term planning and strategic movement are crucial, the MRL+STDP model scored 2900 in just 420 episodes, significantly surpassing its competitors.

Furthermore, the model’s computational efficiency was evident in its lower GPU consumption. For example, in Breakout, the MRL+STDP model required only 16 GPU hours, while traditional models such as A3C used up to 18 GPU hours. This reduction in resource usage, without compromising performance, makes the model highly suitable for resource-constrained environments and suggests its potential for real-world applications.

The results for the MRL+STDP model were highly encouraging across all evaluation metrics. In Breakout, for example, the model achieved an average of 430 total rewards per episode, outperforming DQN (350), Double DQN (375), and A3C (400). This superior performance can be attributed to the STDP mechanism, which enables dynamic adjustments to synaptic strengths based on temporal differences in neuron activation, allowing the agent to better respond to critical environmental cues. Additionally, the MRL framework facilitated swift adaptation, promoting efficient transfer of learning across various game scenarios. As a result, the MRL+STDP model required only 250 episodes to converge, compared to the 300-550 episodes needed by other models.

The learning efficiency of the MRL+STDP model was further highlighted in Space Invaders, where it not only achieved significantly higher rewards (250 vs. 180-225 from other models) but also converged faster, with only 320 episodes compared to 380-500 for the other algorithms. This demonstrated the model’s ability to generalize effectively in games that require both quick reflexes and strategic planning. Similar improvements were observed in Pong and Ms. Pac-Man, where the MRL+STDP model consistently outperformed other algorithms in terms of both rewards and convergence speed. In Ms. Pac-Man, where long-term planning and strategic movement are crucial, the MRL+STDP model scored 2900 in just 420 episodes, significantly surpassing its competitors. Furthermore, the model’s computational efficiency was evident in its lower GPU consumption. For example, in Breakout, the MRL+STDP model required only 16 GPU hours, while traditional models such as A3C used up to 18 GPU hours. This reduction in resource usage, without compromising performance, makes the model highly suitable for resource-constrained environments and suggests its potential for real-world applications.

5.2. Adaptation tests

The experimental methodology began with training the MRL+STDP model, along with traditional models like DQN and Double DQN, on a set of well-understood games. After this, the models were tested on more complex games such as Asteroids and Centipede. The primary evaluation metrics included the number of episodes required to reach a competitive performance level, the rate at which scores improved per episode, and a qualitative assessment of how gameplay strategies evolved over time. This thorough evaluation approach aimed to capture both quantitative performance gains and the qualitative improvements in the sophistication of gameplay strategies, as summarized in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Adaptation performance of different models in new games.

https://doi.org/10.1371/journal.pone.0320777.t003

In Asteroids, the MRL+STDP model required only 400 episodes to achieve competitive performance, significantly outperforming DQN and Double DQN, which needed 800 and 650 episodes, respectively. The average score improvement per episode for MRL+STDP was 6 points, well ahead of the 3 and 3.5 points seen in DQN and Double DQN. Qualitatively, the MRL+STDP model demonstrated a swift mastery of advanced maneuvering and shooting accuracy, highlighting its capacity to quickly adopt and apply effective strategies.

Similarly, in Centipede, the MRL+STDP model again showed superiority, needing just 450 episodes to reach competitive performance, compared to the 750 and 700 episodes required by DQN and Double DQN. The score improvement rate for MRL+STDP was 7 points per episode, compared to 5 and 5.5 points for the other models. Qualitative analysis revealed that MRL+STDP quickly adapted to the game’s dynamics, excelling at both targeting and dodging enemies—demonstrating its dynamic learning ability.

The enhanced performance of the MRL+STDP model can be attributed to several factors. The meta-learning component facilitates the transfer of knowledge from one game to another, enabling faster adaptation to new environments. Furthermore, the STDP mechanism allows for dynamic adjustments to synaptic strengths based on neuron activation timing, making the model more responsive to rapid changes in the environment, as depicted in Figs 6 and 7.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Performance over epochs with a 95% confidence interval.

https://doi.org/10.1371/journal.pone.0320777.g007

Furthermore, the integration of MRL+STDP enhances the agent’s ability to develop and apply more refined strategies, as demonstrated in the qualitative gameplay analysis. In contrast to traditional models, which tend to improve slowly through repeated exposure to game scenarios, the MRL+STDP model quickly adapts and formulates complex strategies tailored to the unique challenges of each new game. This capacity for fast learning not only reduces the time required to reach proficiency but also deepens the strategic complexity of the gameplay, as shown in Fig 8.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Comparison of Mean Absolute Error (MAE) in Millimeters as a Function of Kernel Size.

Fig 8 presents the mean absolute error (MAE) in millimeters as a function of kernel size (in pixels) for two model configurations. The red line, marked with circles, corresponds to the ‘Full Model’, while the blue line, marked with ‘X’ symbols, represents the ‘Reduced Depth’ version. The graph tracks the performance of both models across kernel sizes ranging from 100 to 300 pixels. The Full Model generally maintains a lower MAE, suggesting higher accuracy than the Reduced Depth model. Both models show a decrease in MAE as the kernel size increases up to approximately 200 pixels. After this point, the Reduced Depth model’s MAE increases significantly, while the Full Model’s performance stabilizes before slightly increasing again.

https://doi.org/10.1371/journal.pone.0320777.g008

5.3. Generalization ability

The training set encompassed games of varying difficulty, including “Breakout” (requiring precise timing and pattern recognition), “Space Invaders” (focused on strategic positioning and shooting), “Pong” (centered on reflexes and anticipating the opponent), and “Pac-Man” (necessitating advanced maze navigation and ghost evasion). This variety aimed to expose the model to a wide range of game mechanics. After training, the models were evaluated on “Asteroids” and “Centipede,” which were not part of the training set, to examine how well the models could generalize their learned strategies.

The evaluation metrics included the following: Performance consistency across games, measured as the average score achieved by models expressed as a percentage of an optimal score set by top human benchmarks; Strategic behavior comparison, a qualitative analysis of how well models applied learned behaviors from training games to test games; and Performance loss during the transition to new games, calculated as the percentage decrease in scores from training to test games, highlighting adaptation efficiency.

The MRL+STDP model excelled in generalization, scoring 78% of the optimal in test games compared to 85% in training games, with only a 7% performance loss. This model showed strong strategic flexibility, adapting quickly to the demands of “Asteroids” and “Centipede” with minimal errors in targeting and maneuvering.

The A3C model, with a similar 7% performance drop, achieved slightly higher test-game consistency at 80% of the optimal score. It demonstrated effective adaptation to the fast-paced nature of the test games, showcasing quick strategic recalibrations.

In contrast, Double DQN and DQN exhibited greater performance losses of 14% and 17%, respectively. These models struggled with adapting strategies from training games, such as using ineffective shooting patterns and evasion tactics from “Breakout” in “Asteroids.” Their difficulties in applying learned strategies to new contexts hampered performance in the test games.

The strategic behavior analysis highlighted a clear distinction: traditional models, while competent at mastering specific games, were less capable of transferring their strategies across contexts. Meanwhile, MRL+STDP demonstrated superior adaptability, likely due to the MRL module’s ability to enable rapid strategy deployment and the STDP mechanism’s dynamic tuning of synaptic strengths. This allowed the model to respond effectively to the timing and sequence of events in the new games, as shown in Fig 9.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Real-time inference.

Fig 9 illustrates a bar chart showing real-time inference times (in milliseconds) for three models—A40, V100, and A5000—based on varying intermediate snapshot counts. The graph plots the inference times for snapshot counts of 9, 18, 27, 36, and 45, with each model represented by different colors: gray for the A40, light blue for the V100, and green for the A5000. The results indicate that the A5000 typically experiences the longest inference times across all snapshot counts, suggesting higher processing complexity or computational load. In contrast, the A40 consistently demonstrates the shortest inference times, reflecting a more efficient processing performance under the same conditions. This analysis is key for assessing how different hardware configurations perform as the number of snapshots increases and computational demands change.

https://doi.org/10.1371/journal.pone.0320777.g009

6.1. Experiments

Although the MRL-STDP model shows considerable improvements in learning efficiency, adaptability, and generalization within Atari game environments, its performance in more complex scenarios remains to be fully explored. Specifically, it has not yet been tested in environments with dynamic objectives, continuous action spaces, or high-dimensional state representations, such as those found in open-world games or robotics applications. The use of predefined game-specific state encoding could also present challenges when dealing with hierarchical decision-making or real-time goal adaptation. Moreover, while the computational overhead introduced by integrating STDP is manageable in discrete settings, it may become a limiting factor when scaling to tasks involving large neural architectures or extensive datasets. Future work should focus on adapting the model for continuous and hierarchical environments and optimizing the STDP mechanism to improve computational efficiency. Such advancements will be essential for expanding the MRL-STDP framework to a wider range of real-world applications beyond gaming environments (Fig 10).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Heatmaps displaying the distribution of values obtained from nine separate experiments, each divided into multiple segments labeled N, O, P, Q, and R, corresponding to different conditions or parameters tested within each experiment.

Each cell in the heatmap corresponds to a specific value, with color intensity representing the magnitude as per the scale on the right. Lighter shades indicate lower values, while darker shades represent higher values. The numbers within the cells reflect the measurements or counts relevant to each experimental condition. These heatmaps provide a visual representation of variability and trends across experiments, helping to analyze data distribution. Labels like ‘50>x>40’ beneath certain experiments show specific criteria or value ranges applied to those segments. The visualization in Fig 10 enables quick comparison of experimental results across diverse conditions, highlighting trends and anomalies.

https://doi.org/10.1371/journal.pone.0320777.g010

The scalability of the MRL-STDP model to high-dimensional continuous state spaces, such as those encountered in MuJoCo environments, presents both opportunities and challenges. While the meta-learning framework enables rapid adaptation to new tasks, the computational demands of STDP increase significantly in environments with extensive state-action spaces. To address this, optimizing the STDP mechanism becomes crucial. One approach involves incorporating sparse synaptic updates by selectively applying STDP to critical connections identified through importance sampling, reducing computational overhead without sacrificing performance. Additionally, adaptive scaling of STDP learning rates based on the magnitude of reward gradients can further enhance efficiency. Future work should also explore integrating model-based planning with STDP to leverage environmental dynamics, enabling more efficient learning in complex tasks. The computational demands of the MRL-STDP model increase with larger datasets or more complex environments, such as MuJoCo or continuous control tasks. For instance, in high-dimensional environments, training the model required 20-25 GPU hours on an NVIDIA V100, compared to 15-18 GPU hours for simpler discrete action tasks like Atari. This highlights the need for efficiency improvements to enhance scalability. Optimizing batch processing by implementing dynamic mini-batch sizes based on gradient variance can reduce GPU memory usage and training time. Additionally, leveraging mixed-precision training can decrease computation costs while maintaining model accuracy. Further, optimizing STDP updates through sparse activation or grouping similar updates can lower the overhead associated with synaptic adjustments. These strategies collectively improve the model’s scalability, making it suitable for real-world applications with large-scale data and high-dimensional environments. To mitigate these computational challenges, several strategies were employed. These included batch size tuning, gradient accumulation, and mixed-precision training, which collectively reduced training time by an average of 15% without compromising model accuracy. Additionally, the architecture was designed with efficiency in mind, leveraging lightweight neural network components such as depthwise separable convolutions and selective activation mechanisms.

6.2. Theoretical Insights to Methodology Design

MRL is fundamentally characterized by its capacity to leverage past experiences to optimize learning for new tasks. The adaptability of MRL is formalized through its meta-objective:

where represents the task-specific loss for the i-th task, θ are the shared meta-parameters, and α denotes the learning rate for task-specific adaptation. This formulation ensures that the meta-parameters are optimized to minimize task-specific losses across a distribution of tasks p(τ). By initializing the base-policy network with θ, the MRL framework enables rapid adaptation to new environments, reducing the need for extensive retraining. In the proposed methodology, this capability is implemented by training a meta-policy network that outputs parameters for each task i. These parameters initialize the base-policy network, which subsequently refines its parameters via task-specific updates during gameplay. The MRL framework’s role is to align the meta-policy network with the broader objective of minimizing cross-task generalization errors, thus accelerating convergence in dynamic environments.

STDP provides a biologically inspired mechanism for fine-grained synaptic adjustments based on temporal neuron spike correlations. The STDP rule for updating synaptic weights w_ij between pre-synaptic neuron i and post-synaptic neuron j is expressed as:

where is the time difference between post-synaptic and pre-synaptic spikes, and and are scaling and temporal constants. This mechanism reinforces synaptic connections for causal relationships and weakens them for reverse causation . In the proposed model, STDP is incorporated into the base-policy network to dynamically refine its parameters during gameplay. By continuously adjusting synaptic weights, the network learns to prioritize actions that maximize cumulative rewards, a process formalized as:

where ε represents the set of all synaptic connections. This dynamic weight adjustment augments the network’s capacity to respond to environmental changes, complementing the meta-policy’s broader adaptation. The combined use of MRL and STDP bridges the gap between global task-level adaptation and local decision-level refinement. MRL ensures that the agent rapidly adapts to new tasks by initializing effective parameters, while STDP fine-tunes these parameters in real-time to optimize performance within the task.