3.2.1 Meta-level MDP.

The meta-level MDP is defined as a tuple , where:

• S₀ represents the high-level state space capturing the overall security status of the network. Each state includes aggregate security metrics, threat assessments, and resource availability.
• A₀ is the set of available sub-policies that the meta-controller can activate. Each action corresponds to selecting and parameterizing a specific sub-policy for execution.
• is the transition function that defines the probability of transitioning from state s₀ to state after taking action a₀.
• is the reward function that evaluates the effectiveness of high-level decisions based on overall security objectives.
• is the discount factor for future rewards at the meta-level.

The meta-controller’s objective is to learn a policy that maximizes the expected cumulative discounted reward:

(1)

3.2.2 Sub-level MDPs.

For each sub-task i, I define a corresponding MDP , where:

• S_i is the state space relevant to sub-task i, containing detailed information specific to that task.
• A_i is the action space for sub-task i, comprising the primitive actions available to the corresponding agent.
• is the transition function for sub-task i.
• is the reward function specific to the objectives of sub-task i.
• is the discount factor for future rewards in sub-task i.

Each sub-level agent aims to learn a policy that maximizes its expected cumulative discounted reward within the context of its specific sub-task.

I define four primary sub-tasks in our framework:

1. Reconnaissance: Monitoring network traffic and system logs to detect potential threats.
2. Analysis: Evaluating detected anomalies to determine their nature and severity.
3. Response: Executing defensive actions to mitigate confirmed threats.
4. Recovery: Restoring affected systems to normal operation after an attack.

3.2.3 Hierarchical policy learning.

To learn effective policies at both levels, I employ a hierarchical reinforcement learning approach based on the options framework [17]. The meta-controller learns to select appropriate sub-policies based on the current high-level state, while each sub-level agent learns to execute its specific task effectively.

For the meta-controller, I use Proximal Policy Optimization (PPO) [33] to learn a policy that maximizes the expected return:

(2)

where is the distribution of initial states.

For each sub-level agent, I also employ PPO to learn task-specific policies:

(3)

where is the distribution of states encountered during the execution of sub-task i.

The hierarchical structure enables efficient learning by reducing the complexity of the overall task and allowing specialized agents to focus on specific aspects of defense.

3.3.1 Semantic attribute space.

I define a semantic attribute space that captures the fundamental characteristics of cyber attacks, including:

• Attack vectors (e.g., phishing, exploitation, lateral movement)
• Target resources (e.g., databases, authentication servers, endpoints)
• Behavioral patterns (e.g., data exfiltration, privilege escalation, denial of service)
• Temporal characteristics (e.g., persistence, frequency, duration)

Each attack pattern, whether known or novel, can be represented as a point in this semantic space based on its observable characteristics.

3.3.2 Semantic mapping function.

I define a mapping function that projects observed attack features X into the semantic attribute space . This function is learned from a diverse dataset of known attacks and their corresponding semantic attributes.

For a novel attack pattern x_new with unknown class, the system predicts the appropriate response by:

(4)

where:

• Y is the set of possible response strategies
• maps response strategies to the semantic space
• is a similarity function in the semantic space, typically implemented as cosine similarity:(5)

3.3.3 Zero-shot policy transfer.

To enable zero-shot transfer of defense policies to novel attacks, I learn a conditional policy function that takes as input not only the current state s but also a semantic embedding z of the attack being defended against.

During training, the agent learns to associate effective defense strategies with specific regions of the semantic space. When a novel attack is detected, its semantic embedding is computed, and the conditional policy generates an appropriate response based on the similarity to known attacks.

The zero-shot policy transfer is formulated as:

(6)

where:

• z_new is the semantic embedding of the novel attack
• Z_known is the set of semantic embeddings for known attacks
• w_k is the similarity weight between z_new and z_k:

(7)

This approach enables the system to generate appropriate responses to previously unseen attack patterns by leveraging knowledge about semantically similar known attacks.

4.1.1 Federated learning architecture.

Our federated learning architecture consists of three main components:

1. Local Agents: Deployed across different network domains, each with its own local dataset and model.
2. Regional Aggregators: Responsible for aggregating models from local agents within a specific region or organization.
3. Global Coordinator: Responsible for aggregating regional models and distributing the global model.

This hierarchical aggregation structure reduces communication overhead and enhances privacy by limiting the exposure of local models.

4.1.2 Federated averaging with differential privacy.

I implement a privacy-enhanced version of the Federated Averaging (FedAvg) algorithm [24] to aggregate model parameters across domains. The standard FedAvg algorithm computes the global model parameters as a weighted average of local model parameters:

(10)

where:

• is the global model parameters
• is the local model parameters from domain k
• n_k is the number of data points in domain k
• n is the total number of data points across all domains

To enhance privacy, I incorporate differential privacy by adding calibrated noise to the local model parameters before aggregation:

(11)

where:

• is the privacy-preserved model parameters
• is Gaussian noise
• C is the clipping threshold for gradients
• σ is the noise multiplier controlling privacy level

The privacy guarantee of this approach is quantified using the -differential privacy framework [34], where ε represents the privacy budget and δ is the probability of privacy violation. We set these parameters based on the sensitivity of the security data being protected.

4.1.3 Secure multi-party computation for model aggregation.

To further enhance privacy during model aggregation, I implement a secure multi-party computation (SMPC) protocol based on homomorphic encryption [35]. This protocol enables the computation of the weighted average of model parameters without revealing the individual parameters to any party.

The SMPC protocol operates as follows:

1. Each local agent encrypts its model parameters using a homomorphic encryption scheme: .
2. The encrypted parameters are sent to the aggregator.
3. The aggregator computes the weighted average on the encrypted parameters: .
4. The aggregator distributes the encrypted global model to all agents.
5. Each agent decrypts the global model using its decryption key.

This approach ensures that raw model parameters are never exposed during the aggregation process, providing strong privacy guarantees.

4.1.4 Knowledge distillation for policy transfer.

To facilitate the transfer of knowledge between different network domains without sharing sensitive data, I implement a federated knowledge distillation approach [36]. This approach allows agents to learn from the behavior of other agents without accessing their training data.

The knowledge distillation process operates as follows:

1. Each local agent trains its policy on local data.
2. The agent generates a set of synthetic states and computes the corresponding action probabilities.
3. These state-action probability pairs are shared with other agents.
4. Each agent updates its policy to match the aggregated action probabilities on the synthetic states.

This approach enables knowledge transfer while minimizing the exposure of sensitive information about local network environments.

4.2.1 Dynamic reward formulation.

I implement a dynamic reward function that adapts to the evolving threat landscape:

(12)

where:

• R_security rewards successful threat mitigation
• R_efficiency rewards resource-efficient responses
• rewards effective responses to previously unseen attacks
• are dynamic weights adjusted based on the current threat landscape

The dynamic weights are updated using a meta-learning approach that optimizes the trade-off between different objectives based on recent performance:

(13)

(14)

(15)

where J is a meta-objective that measures overall defense effectiveness and is the learning rate.

4.2.2 Adversarial training for robustness.

To enhance robustness against adversarial attacks, we implement an adversarial training approach that simulates sophisticated attack strategies. This approach involves training defensive agents against a simulated adversary that continuously adapts its attack strategies to exploit vulnerabilities in the defense system.

The adversarial training process is formulated as a min-max optimization problem:

(16)

where:

• are the parameters of the defense policy
• are the parameters of the attack policy
• R_D is the reward function for the defender
• is the defense policy
• is the attack policy
• ρ is the state distribution

This adversarial training approach ensures that defensive agents are prepared for sophisticated and adaptive attack strategies.

4.2.3 Continual learning with experience replay.

To maintain effectiveness against evolving threats while avoiding catastrophic forgetting of previously learned defense strategies, I implement a continual learning approach with experience replay [37]. This approach involves maintaining a replay buffer of experiences from different threat scenarios and periodically retraining the model on a balanced sample from this buffer.

The experience replay buffer contains tuples from various threat scenarios. During training, I sample mini-batches from this buffer to update the policy:

(17)

where is the loss function and is a sample from the replay buffer.

To ensure balanced representation of different threat scenarios, I implement a prioritized sampling approach that gives higher probability to scenarios that are underrepresented in recent training iterations.

4.2.4 Adaptive zero-shot learning algorithm.

I implement an adaptive zero-shot learning algorithm that continuously refines the semantic mapping function based on new observations. The algorithm is presented in pseudocode below:

Algorithm 1. Adaptive zero-shot learning.

Require: Initial semantic mapping function ϕ, known attack set

  Z_known, policy function

Ensure: Updated semantic mapping function , updated policy

  function

1: Initialize replay buffer

2: while not converged do

3:   Observe current state s_t

4:   Detect potential attack pattern x_t

5:   Compute semantic embedding

6:   if similarity then

7:    // Novel attack detected

8:    Compute response using zero-shot transfer:

9:   

10:    Execute action a_t and observe reward r_t and next state

      s_t + 1

11:    Store experience in buffer

12:    Update

13:   else

14:    // Known attack detected

15:   

16:    Execute action a_t and observe reward r_t and next state

      s_t + 1

17:    Store experience in buffer

18:   end if

19:   // Periodically update semantic mapping and policy

20:   if update_interval reached then

21:    Sample batch from buffer

22:    Update semantic mapping function ϕ to minimize

  embedding loss

23:    Update policy function to maximize expected reward

24:   end if

25: end while

26: return

This algorithm enables continuous adaptation to evolving threats by updating both the semantic mapping and the policy functions based on observed experience. Algorithm 1 is executed by each local defender agent during online inference. Table 1, shows the mapping between textual description and Algorithm 1 components. The key components involved are as follows:

1. Semantic Mapper ϕ: Projects observed features into the learned semantic space.
2. Policy Matching Module: Compares the mapped vector to existing known responses using similarity functions.
3. Knowledge Base: A local copy or federated view of known defense-action mappings.
4. Adaptation Module: If no match is above threshold, the agent uses fallback exploration or queries the federation for new mappings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Mapping between textual description and Algorithm 1 components.

https://doi.org/10.1371/journal.pone.0329969.t001

The algorithm runs continuously on agents deployed in edge hosts or gateways, depending on the testbed scenario. The logic is integrated as part of the agent’s decision cycle.

4.3.1 System architecture.

Fig 1 illustrates the overall architecture of our implementation, showing the hierarchical structure of agents, the federated learning components, and the zero-shot adaptation module.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. System architecture of the Adaptive Zero-Shot Hierarchical MARL framework with Federated Knowledge Sharing.

https://doi.org/10.1371/journal.pone.0329969.g001

The system is deployed in a simulated network environment consisting of multiple segments, each containing various devices and services. Each network segment is protected by a team of specialized agents coordinated by a local meta-controller. The meta-controllers from different segments participate in federated learning to share knowledge while preserving privacy.

4.3.2 Neural network architecture.

For the hierarchical policy learning, I implement neural network architectures for both the meta-controller and the specialized agents:

• Meta-Controller: A transformer-based architecture [39] that processes aggregate security metrics and outputs sub-policy selections. The network consists of a multi-head attention mechanism followed by feed-forward layers with ReLU activations.
• Reconnaissance Agent: A convolutional neural network (CNN) that processes network traffic patterns and system logs to detect potential threats.
• Analysis Agent: A graph neural network (GNN) that analyzes the relationships between detected anomalies to identify coordinated attack patterns.
• Response Agent: A deep Q-network (DQN) that selects appropriate defensive actions based on the current threat assessment.
• Recovery Agent: A policy gradient network that determines the optimal sequence of recovery actions to restore affected systems.

4.3.3 Zero-shot learning component.

For the semantic mapping function, I implement a Siamese network [40] that projects observed attack features into a semantic embedding space. The network is trained using a contrastive loss function that minimizes the distance between semantically similar attacks and maximizes the distance between dissimilar attacks.

The zero-shot policy transfer is implemented using a conditional policy network that takes as input both the current state and the semantic embedding of the detected attack. This network is trained to associate effective defense strategies with specific regions of the semantic space.

4.3.4 Federated learning implementation.

I implement the federated learning component using PySyft [41], a library for privacy-preserving machine learning. The implementation includes:

• Secure aggregation protocols based on homomorphic encryption
• Differential privacy mechanisms for parameter sharing
• Hierarchical aggregation structure for efficient communication
• Knowledge distillation for policy transfer

4.3.5 Semantic mapping function implementation.

The semantic mapping function is implemented using a Siamese neural network architecture. The training dataset consists of pairs of attack instances represented by feature vectors extracted from network logs, packet-level information, and host behaviors. Each pair is labeled as either semantically similar or dissimilar based on domain expert annotations and established attack taxonomies. During training, the Siamese network employs a contrastive loss function, formally defined as:

(18)

where represent the embeddings of paired samples, y is the binary label (1 for similar pairs, 0 for dissimilar), denotes the Euclidean distance function, and m is a predefined margin. This formulation ensures that semantically similar attack pairs have embeddings close to each other, while embeddings for dissimilar pairs are encouraged to separate. This approach ensures the semantic attribute space accurately captures critical characteristics of cyberattacks, facilitating effective zero-shot classification and policy transfer.

4.3.6 Experimental setup.

I evaluate our approach using a combination of simulated and emulated network environments:

1. CybORG Simulation Environment: A reinforcement learning environment for cyber security developed by the CAGE Challenge [42], which provides realistic network topologies and attack scenarios.
2. DARPA CRATE Dataset: A dataset of cyber attacks and defenses collected from realistic network environments [43], which I use to train and evaluate our zero-shot learning components.
3. Custom Federated Learning Testbed: A distributed testbed consisting of multiple virtual network environments, each with its own security policies and attack patterns, which I use to evaluate the federated learning components.

The experiments are conducted on a cluster of machines, each equipped with an NVIDIA A100 GPU, 128GB RAM, and 32 CPU cores. To specifically evaluate the federated learning and privacy-preserving aspects of AZH-MARL, I developed a custom distributed testbed. This testbed simulated a collaborative defense scenario involving three independent organizational networks. Each virtual network environment within the testbed comprised approximately 30-40 nodes, with unique security policies, varying levels of baseline security posture, and distinct, locally generated attack patterns (e.g., phishing campaigns targeting one network, ransomware in another, or data exfiltration attempts in the third).

This heterogeneity was crucial to test the framework’s ability to share knowledge effectively and adapt defenses in a federated manner while preserving the privacy of each participating entity’s local data. In the custom federated learning testbed, each of the three simulated organizational networks had its own deployment of this hierarchical agent structure which included a Meta-Controller and four sub-level agents: Reconnaissance, Analysis, Response, Recovery. The local models from these agents were then aggregated via the federated learning mechanism. Fig 2 illustrates a network architecture, for this federated learning testbed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Custom federated testbed.

https://doi.org/10.1371/journal.pone.0329969.g002

4.4.1 Evaluation metrics.

I use the following metrics to evaluate our approach:

1. Security Metrics:
   • Detection Rate (DR): The percentage of attacks successfully detected.
   • False Positive Rate (FPR): The percentage of benign activities incorrectly classified as attacks.
   • Mean Time to Detection (MTTD): The average time from attack initiation to detection.
   • Mean Time to Response (MTTR): The average time from detection to defensive action.
2. Zero-Shot Performance Metrics:
   • Zero-Shot Detection Rate (ZSDR): The detection rate for previously unseen attack patterns.
   • Zero-Shot Response Effectiveness (ZSRE): The effectiveness of defensive actions against novel attacks.
   • Semantic Mapping Accuracy (SMA): The accuracy of the semantic mapping function in classifying attack patterns.
3. Federated Learning Metrics:
   • Communication Overhead (CO): The amount of data transferred during federated learning.
   • Privacy Leakage (PL): The amount of sensitive information that can be inferred from shared models.
   • Convergence Time (CT): The time required for the federated learning process to converge.
4. Resource Efficiency Metrics:
   • Computational Overhead (CPO): The computational resources required for real-time operation.
   • Memory Usage (MU): The memory required for model storage and execution.
   • Energy Consumption (EC): The energy required for continuous operation.

4.4.2 Baseline approaches.

I compare our approach with several baseline approaches:

1. Traditional MARL: A flat multi-agent reinforcement learning model without hierarchy or transfer mechanisms [2,9];
2. Hierarchical MARL: A structured MARL approach that incorporates hierarchical control but lacks generalization capabilities [15–18];
3. Federated Learning: A collaborative training method where agents share models without centralizing raw data [24,25,31,32];
4. Zero-Shot Learning: A generalization-based model trained to respond to unseen attack types without prior specific training [19–23].

4.4.3 Results and analysis.

Zero-day exploits:.

In our experiments, zero-day exploits refer to attacks leveraging previously unknown vulnerabilities for which no explicit signature or patch was available. These were simulated across CybORG, the DARPA dataset, and our custom federated testbed by withholding specific attack variants from training. This forced agents—especially those using our AZH-MARL framework with zero-shot learning—to generalize from known behaviors and respond adaptively to novel threats, such as obfuscated malware or unseen TTPs.

Advanced Persistent Threats (APTs):.

APTs were modeled as multi-stage, stealthy campaigns with persistent access goals. Using CybORG and selected DARPA scenarios, I scripted chained attack phases (e.g., phishing, lateral movement, data exfiltration), testing agents’ ability to detect the campaign across time. In our federated testbed, APTs unfolded gradually within different nodes, evaluating the meta-controller’s capacity to recognize strategic threats and orchestrate coordinated responses beyond isolated alerts.

Fig 3 shows the performance of our proposed AZH-MARL approach compared to the baselines across different attack scenarios, including known attacks, zero-day exploits, and advanced persistent threats (APTs).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Performance comparison of different approaches across attack types.

https://doi.org/10.1371/journal.pone.0329969.g003

While both federated learning and zero-shot learning independently offer strong performance in terms of generalization and decentralized training, they each have limitations when applied in isolation. On its own federated learning suffers from difficulty in adapting to unseen attack patterns, particularly those not present in the training data across agents. Conversely Zero-shot learning, while effective at detecting novel attacks, often lacks context-aware response strategies and is susceptible to noisy or ambiguous semantic mappings when not anchored to local experience. The proposed AZH-MARL framework builds on the strengths of both techniques. It uses federated knowledge sharing to continuously refine distributed policies and semantic mappings, while leveraging a hierarchical multi-agent architecture to locally adapt to emergent threats. This tight coupling of ZSL within a federated MARL framework enables faster response times and greater detection robustness, especially in the presence of adversarial behavior or concept drift. As shown in Fig 3, AZH-MARL outperforms each technique in isolation by providing adaptive coordination between semantic embedding and policy optimization across distributed agents. While Federated Learning (FL) and Zero-Shot Learning (ZSL) together offer strong generalization, they lack adaptive coordination and localized response intelligence, which AZH-MARL provides. Table 2 shows overview of the capabilities of each approaches against some aspects

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison between FL + ZSL and the proposed AZH-MARL framework.

https://doi.org/10.1371/journal.pone.0329969.t002

Detection performance:.

Table 3 presents the detection performance of our approach compared to the baselines. Our approach achieves a detection rate of 94.2% for known attacks and 82.7% for zero-day exploits, significantly outperforming the baselines. The false positive rate is maintained at 3.8%, which is comparable to the best-performing baseline.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of detection performance metrics across different defense approaches.

https://doi.org/10.1371/journal.pone.0329969.t003

The improved detection performance can be attributed to the combination of hierarchical structure, which enables specialized agents to focus on specific aspects of detection, and zero-shot learning, which enables recognition of novel attack patterns based on semantic similarities.

Response effectiveness:.

Fig 4 illustrates the effectiveness of defensive responses across different attack scenarios. Our approach achieves a mean time to response of 2.3 seconds for known attacks and 3.7 seconds for zero-day exploits, representing improvements of 35% and 42% respectively compared to the best-performing baseline.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Zero-shot learning integration for recognizing and responding to novel attack patterns.

https://doi.org/10.1371/journal.pone.0329969.g004

The response effectiveness is particularly notable for advanced persistent threats, where our approach achieves a containment rate of 87.3% compared to 62.1% for the best-performing baseline. This improvement is due to the hierarchical coordination of defensive actions and the adaptive response mechanisms that adjust to evolving threat patterns.

Zero-shot performance:.

Table 4 presents the zero-shot performance of our approach compared to baselines with zero-shot capabilities. Our approach achieves a zero-shot detection rate of 82.7% and a zero-shot response effectiveness of 76.5%, outperforming all baselines.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparison of zero-shot learning performance metrics across different approaches.

https://doi.org/10.1371/journal.pone.0329969.t004

The semantic mapping accuracy of our approach is 89.3%, indicating that the semantic mapping function effectively captures the fundamental characteristics of attack patterns and enables accurate classification of novel attacks.

Federated learning performance:.

Fig 5 shows the convergence of the federated learning process across different privacy settings. Our approach achieves convergence within 15 communication rounds while maintaining strong privacy guarantees ().

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Hierarchical MARL framework with meta-controller and specialized sub-policies.

https://doi.org/10.1371/journal.pone.0329969.g005

The communication overhead of our approach is 45% lower than standard federated averaging due to the hierarchical aggregation structure. The privacy leakage, as measured by the success rate of membership inference attacks, is maintained below 5% across all experiments.

Ablation study:.

To understand the contribution of each component to the overall performance, I conduct an ablation study by removing individual components from our approach. Table 5 presents the results of this study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Ablation study showing the contribution of each component to overall performance.

https://doi.org/10.1371/journal.pone.0329969.t005

The removal of the hierarchical structure results in a 12.3% decrease in detection rate and a 18.7% increase in mean time to response, highlighting the importance of task decomposition for efficient learning and coordination.

The removal of the zero-shot learning component results in a 37.5% decrease in zero-shot detection rate and a 42.1% decrease in zero-shot response effectiveness, confirming the critical role of semantic mapping for adapting to novel threats.

The removal of the federated learning component results in a 15.2% decrease in detection rate for environments with limited local data, emphasizing the value of privacy-preserving knowledge sharing for improving overall defense effectiveness.

Resource efficiency:.

Table 6 presents the resource efficiency of our approach compared to the baselines. Our approach requires 2.3 GB of memory and 4.5 GFLOPS of computational power for real-time operation, which is higher than rule-based approaches but comparable to other learning-based approaches.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Comparison of resource efficiency metrics across different defense approaches.

https://doi.org/10.1371/journal.pone.0329969.t006

The energy consumption of our approach is 15% lower than flat MARL approaches due to the efficient task decomposition and hierarchical decision-making structure.

4.4.4 Discussion.

The evaluation results demonstrate that our Adaptive Zero-Shot Hierarchical MARL approach with Federated Knowledge Sharing significantly outperforms existing approaches across various metrics. The key advantages of our approach include:

1. Enhanced Adaptability: The zero-shot learning component enables effective response to previously unseen attack patterns, addressing a critical limitation of traditional approaches.
2. Improved Coordination: The hierarchical structure enables efficient coordination among specialized agents, leading to more effective defense strategies.
3. Privacy-Preserving Collaboration: The federated learning component enables knowledge sharing across organizational boundaries without compromising sensitive data.
4. Scalable Defense: The hierarchical aggregation structure reduces communication overhead and enables scalable deployment across large network environments.

These advantages make our approach particularly well-suited for defending against sophisticated and evolving cyber threats in complex network environments with privacy constraints.

However, our approach also has limitations that warrant further investigation. First, the computational requirements are higher than those traditional rule-based approaches, which may limit deployment on resource-constrained devices. Second, the effectiveness of zero-shot learning depends on the quality and diversity of the training data used to learn the semantic mapping function.

In the next section, I conclude the paper and discuss future research directions to address these limitations and further enhance the capabilities of our approach.