Graph-based state representation

The post-fault service restoration problem is modeled as a MDP characterized by the tuple , where denotes the state space, denotes the action space, represents the state transition dynamics governed by the power system and restoration actions, is the reward function evaluating restoration performance, and is the discount factor. Specifically, and denote the system state and action at time step t, respectively.

Although the MDP is formulated in a general stochastic form, the transition dynamics in this study are primarily determined by the switching logic and AC power flow calculations. While the framework supports stochasticity, the simulation environment here assumes deterministic transitions unless exogenous disturbances are explicitly introduced.

To avoid ambiguity between graph learning and physical simulation, two graph representations are defined at each time step: a physics graph for network simulation and constraint enforcement, and a learning graph for graph neural network inference.

The physics graph is defined as

(1)

where denotes the set of physical buses, denotes the set of energized physical branches and switches at time step t, denotes the physical node-state feature matrix, and denotes the physical edge-attribute matrix containing branch parameters such as line resistance and reactance. This graph is exclusively used for AC power flow calculation, radiality checking, and operational constraint enforcement.

To facilitate information aggregation between DERs and the grid in the graph attention network (GAT), a learning graph is further constructed as

(2)

where

(3)

and

(4)

Here, denotes the set of virtual DER nodes, and denotes the set of virtual connection edges linking each DER virtual node to its point of common coupling (PCC).

It should be emphasized that the virtual edges in are introduced solely for message passing and state representation learning in the GAT. They are not physical branches and are therefore excluded from AC power flow equations, topology feasibility checks, radiality constraints, and branch flow limit evaluation. In particular, the learning graph inherits all physical buses and physical branches from the physics graph, and further augments them with DER virtual nodes and virtual edges. In this way, the two graphs remain consistent through their shared physical subgraph while serving different purposes in neural inference and physical simulation.

Each node is associated with a feature vector , defined as

(5)

where includes device-specific operational states such as available power capacity, state-of-charge, and controllability indicators. Consequently, the global node feature matrix is constructed as .

Each physical branch is associated with an attribute vector

(6)

For physical branches, and denote the actual line resistance and reactance, respectively. For virtual edges in , the corresponding edge attributes are assigned as a negligible constant only as placeholder edge attributes for neural message passing, rather than as physical impedance parameters in the power flow model.

The collection of physical branch attribute vectors forms the physical edge-attribute matrix

(7)

while the collection of all edge attribute vectors in the learning graph forms

(8)

Here, each row corresponds to the transposed edge attribute vector associated with an edge in the corresponding graph.

Accordingly, the state input to the policy network is represented by the learning graph

(9)

while AC power flow calculation, environment transition, and feasibility verification are carried out on the physics graph .

During each restoration step, the selected switching action is first applied to by updating the energized physical branch set . Topology masks, radiality constraints, and operational feasibility checks are then evaluated only on the updated physics graph. After the AC power flow is solved and the new physical operating state is obtained, the learning graph for the next decision step, , is reconstructed by augmenting the updated physics graph with DER virtual nodes and virtual edges.

Hybrid action space and state transition

To enable coordinated control of network reconfiguration and resource scheduling, the action space is constructed as a heterogeneous action space defined as

(10)

At each decision step, the hierarchical agent selects a single action . If , it represents a switching operation that modifies the network topology . If , it corresponds to an active power regulation command for the virtual power plant (VPP).

The VPP is modeled as an aggregated entity. This aggregation provides a tractable abstraction for system-level decision-making, while preserving essential physical constraints through time-varying power boundaries.

Its active power injection is constrained by the aggregated physical limits of its internal components:

(11a)

(11b)

Here, denotes the maximum available photovoltaic power determined by solar irradiance. The zero term in the lower bound explicitly indicates that PV output is fully curtailable if required. The ESS limits are constrained by inverter ratings and state-of-charge. For EVs, the charging flexibility is constrained by the user’s energy demand and departure time.

Specifically, the immediate charging boundaries () are dynamically updated to ensure that the required energy is satisfied before departure, thereby enforcing the energy sufficiency constraint through the aggregated boundary model.

For VPP regulation, the DER action set is defined as

(12)

where denotes a predefined normalized adjustment ratio.

At each time step t, the selected action represents a relative decrease, hold, or increase command with respect to the adjustable capacity of the VPP.

The actual power setpoint of VPP i is updated as

(13)

where denotes the available adjustable capacity of VPP i, and and are the minimum and maximum admissible power limits, respectively.

The projection operator enforces feasibility by ensuring that the updated power setpoint remains within the VPP operational bounds.

The VPP-level action generated by the VPP-control agent serves as an aggregate regulation command for the internal DER resources. After the aggregate active-power setpoint of VPP i is updated, the requested VPP-level power adjustment is calculated as

(14)

where is the updated aggregate active-power setpoint obtained from the VPP-level regulation command, is the current active-power injection of VPP i, and denotes the requested aggregate power adjustment to be realized by the internal DER resources.

To physically realize this VPP-level regulation command, the controllable DER units inside VPP i are collected as

(15)

where denotes the set of controllable DER units in VPP i, including PV units, ESS units, and EV charging units. Here, , , and denote the numbers of controllable PV units, ESS units, and EV charging units contained in VPP i, respectively.

The requested aggregate adjustment is realized by updating the internal DER setpoints subject to their real-time regulation margins and operating constraints. The realized aggregate VPP power adjustment after device-level allocation can be expressed as

(16)

where is the realized aggregate VPP power adjustment after intra-VPP allocation, and , , and denote the setpoint adjustments of PV, ESS, and EV units, respectively. The negative sign before the EV term is used because EV charging is modeled as controllable load. Therefore, reducing EV charging demand increases the net VPP active-power injection, whereas increasing EV charging demand decreases it.

After the VPP-level regulation command is determined by the VPP-control agent, it is passed to the intra-VPP dispatch module for physical implementation. According to the real-time operating constraints, the intra-VPP dispatch module first evaluates the feasible regulation margins of controllable PV, ESS, and EV units. Then, the module identifies the DER units that can contribute to the required regulation direction and determines their allocation weights according to their available regulation margins and predefined dispatch priority coefficients. Based on the normalized allocation weights, the aggregate VPP-level regulation signal is allocated to one or more feasible units. Each device-level regulation command is bounded by the corresponding device operating range during execution. When the regulation margin of some units is insufficient, the dispatch module further selects other units with feasible margins to share the remaining adjustment, thereby improving the executability of the aggregate command. Finally, the obtained device-level commands are used to update the setpoints of PV active-power output, ESS charging/discharging power, and EV charging load.

When , the VPP is required to increase its net active-power injection. The admissible device-level realizations include increasing PV output when curtailed power is available, increasing ESS discharging power or reducing ESS charging power, and reducing EV charging demand. Conversely, when , the VPP is required to decrease its net active-power injection. The admissible device-level realizations include curtailing PV output, reducing ESS discharging power or increasing ESS charging power, and increasing EV charging demand. When , the current DER setpoints are maintained.

After the device-level realization is determined by the intra-VPP dispatch module, the corresponding DER setpoints are adjusted in the same normalized incremental manner as the VPP-level command and then projected onto their admissible operating ranges. For PV units, the admissible range is determined by the available photovoltaic power and curtailment limits. For ESS units, the admissible range is determined by the charging/discharging power ratings and SOC constraints. For EV units, the admissible range is determined by the charging power bounds and the remaining energy requirement before departure. After the bounded device-level updates, the realized active-power injection of VPP i is recalculated by aggregating the updated internal DER setpoints. Therefore, although the learned policy outputs the VPP-level aggregate regulation command, the physical execution is explicitly mapped to device-level DER operations, including the adjusted DER units, their regulation directions, and their setpoint updates.

The system state transition is implemented according to the selected action. If a switching action is selected, the physical topology is updated. If a DER regulation action is selected, the internal DER setpoints are updated through the intra-VPP dispatch module according to the corresponding device-level realization rule. Then, the environment solves the AC power flow on the updated physics graph to obtain the new operating point, including nodal voltages and branch power flows. Finally, the learning graph is constructed by augmenting the updated physics graph with DER virtual nodes and virtual edges for state encoding.

Therefore, the physical transition is written as

(17)

where denotes the environment-level physical transition mapping, which updates the physical network topology or the VPP aggregate power injection according to the selected action , and then solves the AC power flow to obtain the updated physical operating point. The state used by the agent at the next step is constructed as

(18)

where denotes the learning-graph construction mapping, which augments the updated physical graph with DER virtual nodes, virtual edges, and the updated DER operating states for state encoding.

Operational constraints and multi-objective reward design

All operational constraints are imposed on the physics graph . In particular, the branch-flow equations, voltage magnitude limits, branch thermal limits, and radiality requirement are defined only over physical buses and physical branches in . Virtual edges in the learning graph are excluded from all physical feasibility checks.

Given a control action, the simulation environment evaluates the post-action operating state on the physics graph through AC power flow analysis. For radial distribution systems, the corresponding network constraints can be expressed in the following branch-flow form.

At each time step t, for each physical branch , the network variables satisfy the following power balance and voltage drop equations:

(19)

(20)

(21)

The last terms in Eqs. (19)–(21) represent the quadratic power losses and voltage drops on branch (i,j), with resistance and reactance .

Given a control action, the simulation environment solves the full AC power flow equations (e.g., via the Newton–Raphson method) to obtain the system operating point, which is then used to verify operational security.

Specifically, nodal voltage magnitudes must remain within permissible bounds,

(22)

and branch apparent power flows must respect thermal ratings,

(23)

Regarding resource capability, the dispatch of DERs is constrained by their inverter capacity and active power limits , defined as:

(24)

Additionally, the network topology is strictly constrained to maintain a radial structure for coordination with protection schemes.

Within the proposed framework, hard constraints, including radiality and switching feasibility, are enforced via action masking over controllable physical switches only. Virtual edges are fixed auxiliary connections in the learning graph and are not subject to switching actions or topology constraints. Soft constraints (e.g., voltage violations and thermal overloads) are strictly penalized in the reward function.

The reward function is defined as

(25)

where represents the proportion of restored load, penalizes active power losses, and accounts for DER operational and degradation costs. The specific values of the reward-related hyperparameters used in the experiments are summarized in Table 1. These coefficients were selected based on preliminary trials and the relative scales of different reward components, so as to emphasize load restoration and physically feasible operating states while retaining line-loss reduction and DER regulation cost as auxiliary objectives.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Experimental hyperparameters used in this study.

https://doi.org/10.1371/journal.pone.0343196.t001

The safety-related discrete reward component is defined as:

(26)

Here, is a weighting coefficient that scales this explicit feedback to guide the agent toward physically feasible operating states.

The optimal control policy is obtained by maximizing the expected cumulative discounted return:

(27)

which formalizes the objective of identifying a coordinated sequence of topology reconfiguration and DER dispatch actions to achieve rapid, efficient, and secure post-fault restoration.

Hierarchical reinforcement learning framework for coordinated restoration

This subsection describes the internal decision-making process of the proposed Expert-Guided Hierarchical Graph Reinforcement Learning (EH-GRL) framework. The post-fault restoration problem involves heterogeneous and tightly coupled control decisions, including discrete network topology reconfiguration and stepwise incremental DER power regulation. Directly optimizing a composite action space that simultaneously covers both decision types leads to a combinatorial explosion and unstable policy learning. To address this challenge, EH-GRL adopts a hierarchical decision-making architecture that separates strategic task allocation from safety-aware execution, thereby reducing decision complexity while preserving coordination between structural and operational restoration actions [25].

At each decision step, the environment provides the current graph state , which is encoded into the latent representation by the shared GAT–Transformer encoder. Based on , the high-level policy selects a task type , determining whether the control focus should be placed on topology reconfiguration or DER regulation. Conditioned on the selected task, the corresponding low-level controller generates a feasible action within its task-specific action subspace. Actions associated with the inactive task are masked out by assigning −∞ logits (or Q-values), ensuring that only task-consistent decisions are executed. The selected action is then applied to the environment, leading to a state transition and a scalar reward reflecting restoration progress, operational safety, and constraint satisfaction. This shared reward signal is used to update both the high-level task-selection policy and the active low-level controller, thereby enabling long-term priority allocation and safe task-specific execution across hierarchical levels.

The high-level policy operates on a compact graph-level state representation, denoted as , which is extracted by a topology-aware GAT–Transformer encoder. Specifically, we denote the graph encoding function implemented by the GAT–Transformer encoder as . In accordance with the state modeling introduced in the previous section, the environment state at time step t is represented as a graph . All decision-making policies in the proposed framework operate on the encoded representation

(28)

rather than directly on the raw graph structure.

Instead of directly generating the composite action , the high-level policy first selects a restoration task type:

(29)

where is the selected task type at time step t. represents a task focused on topology reconfiguration, which involves structural changes to the network, while corresponds to a task focused on DER regulation, optimizing the operation of energy resources within the network. This task-level gating mechanism dynamically allocates control focus between topology reconfiguration and DER regulation based on the evolving network condition. By doing so, it effectively decouples the discrete structural decisions (topology changes) from operational power support actions (DER regulation), all while maintaining a unified MDP formulation.

Accordingly, only the sub-action associated with the selected task is activated at each step. When , a low-level topology controller generates the switching action ; when , a DER controller produces the regulation action , while the other sub-action remains inactive. This hierarchical execution scheme preserves the original environment action interface but substantially reduces the effective action space explored at each step, improving learning efficiency and stability during long restoration horizons.

Importantly, although the decision logic is hierarchically structured, the underlying environment exposes a unified discrete action space. Specifically, the global action space comprises mutually disjoint subsets corresponding to network reconfiguration and DER regulation, and since these subsets are mutually exclusive, each action uniquely determines its associated task category. Consequently, the hierarchical decomposition is realized internally through task-conditioned action masking, rather than by modifying the fundamental interaction interface with the environment.

The high-level policy is parameterized as a categorical distribution

(30)

where denotes the high-level task selected at time step t, is the global graph-level state embedding extracted by the encoder, and denote the weight matrix and bias vector of the linear output layer, respectively, and represents the learnable parameters of the high-level policy.

To improve early-stage convergence and reduce exploration variance, the high-level policy is initialized via the proposed expert-guided imitation learning strategy, and subsequently optimized using the Proximal Policy Optimization (PPO) objective.

(31)

where denotes the expectation over sampled time steps, is the PPO clipping threshold, is the probability ratio between the current policy and the previous policy with parameters , and is the estimated advantage function.

In both the high-level task-selection policy and the low-level DER controller, the advantage function is estimated using Generalized Advantage Estimation (GAE) with the corresponding independent critic network.

Once the task type is determined, execution is delegated to specialized low-level controllers that explicitly enforce physical feasibility. When topology reconfiguration is selected, a DQN-based controller explores the discrete and combinatorial switching action space. A physics-informed action masking mechanism is applied to exclude infeasible operations. Specifically, at each time step, the environment computes the feasible action set based on the current graph state , excluding actions on faulted components, violations of network radiality, and switching sequences that may induce electrical islands.

The masked Q-values are defined as

(32)

where a is a candidate switching action, is the action-value function, and represents the set of physically feasible actions determined by the environment under the current network configuration.

These masked Q-values are consistently used during both action selection and target value computation. The topology controller is trained by minimizing the temporal-difference loss

(33)

where r is the immediate reward, denotes the encoded next state, is the discount factor, and is the target Q-network used to stabilize training.

When DER regulation is selected, a PPO-based controller performs stepwise power adjustments of heterogeneous DER units. Instead of directly selecting absolute power setpoints, the DER action represents discrete incremental changes in active power, facilitating stable learning and smooth regulation under uncertainty.

PPO is applied here to a discrete categorical policy over incremental DER adjustment actions. The DER policy is parameterized as

(34)

where represents the policy parameters.

The DER controller is optimized using the PPO objective

(35)

where is the policy probability ratio and denotes the parameters of the previous DER policy.

The high-level task-selection policy and the low-level DER controller are equipped with independent critic networks for advantage estimation, avoiding value function interference across different decision timescales and action semantics.

A device-aware action masking mechanism is applied directly to the policy logits, in which infeasible actions are assigned −∞ log-probabilities based on the heterogeneous physical constraints of different DER types. Similar to topology reconfiguration, the corresponding feasibility masks are generated by the environment based on the current graph state . For energy storage systems and electric vehicles, the masking rules enforce state-of-charge bounds, charging and discharging power limits, and mutually exclusive operating modes. For photovoltaic units, actions exceeding the instantaneous available generation determined by irradiance conditions and inverter capacity are excluded. Through this capability-aware masking scheme, the DER controller is restricted to a dynamically feasible action set, ensuring strict compliance with device-level operational constraints.

In addition, system-level operational constraints, including voltage magnitude limits and branch thermal ratings, are enforced through a reward penalty mechanism, guiding the agent toward safe operating regions without requiring explicit constraint checks during action selection.

Overall, the proposed EH-GRL framework achieves an effective separation between strategic task allocation and safety-aware execution. The high-level policy adaptively allocates control priority between topology reconfiguration and DER regulation, while the specialized low-level controllers ensure that all executed actions remain feasible and physically compliant throughout the restoration process.

Topology-aware state embedding via GAT

As a core perception module of the proposed EH-GRL framework, a topology-aware state embedding is constructed to encode the structural and operational characteristics of post-fault load transfer in active distribution networks. Unlike conventional grid modeling, the state representation here must account for the heterogeneous integration of VPPs. Based on the graph-based MDP formulation introduced earlier, the environment state at time step t is represented as a graph . Each node feature vector explicitly encodes both its functional role (e.g., load bus, DER, switch) and its real-time operational state, including voltage magnitude, load supply status, and critical VPP constraints (e.g., State of Charge, inverter capacity margin). In this dynamic graph, load transfer feasibility is jointly constrained by local electrical states and global network connectivity. These irregular, topology-dependent, and resource-constrained dependencies render conventional convolution-based architectures unsuitable.

To effectively capture such interactions, a GAT is adopted as the backbone of the state encoder. The core motivation for employing GAT in this specific context lies in its ability to explicitly model the heterogeneous electrical influence between components. In a post-fault scenario, controllable elements such as VPPs and tie switches typically exert a stronger impact on restoration feasibility than passive load buses. The attention mechanism naturally mimics this physical logic by adaptively weighting neighboring information based on electrical proximity and operational relevance (e.g., highlighting paths with sufficient VPP headroom).

Specifically, let denote the embedding of node i at layer l, where denotes the initial node feature vector. The unnormalized attention coefficient between node i and its neighbor j is computed as:

(36)

where and are trainable parameters. Physically, reflects the relative relevance of power flow capability or information exchange from component j to i. These coefficients are normalized to obtain attention weights:

(37)

where denotes the set of neighboring nodes, and the self-loop term allows each node to retain its own operational state during aggregation. Through learning, evolves to highlight valid energization paths, effectively filtering out infeasible connections in the faulted network.

To enhance representational capacity, a multi-head attention mechanism with K heads is employed. The updated node embedding is obtained by concatenating the outputs of all attention heads:

(38)

where and correspond to the attention weight and linear transformation of the k-th attention head, respectively.

A critical challenge in VPP-assisted restoration is preserving the boundary constraints of flexible resources. Deep GNN layers tend to over-smooth node features, which poses a risk of blurring the precise status of VPPs (e.g., distinguishing a fully charged ESS from a depleted one). To mitigate this, a residual connection from the original node features is introduced:

(39)

where denotes the initial feature vector of node i, and is a learnable linear projection for dimension alignment. This residual design ensures that critical low-level electrical attributes are preserved for accurately evaluating load transfer feasibility under safety constraints.

While the GAT encoder captures local electrical couplings, switching operations often induce global voltage profile changes through long-range dependencies. For example, closing a tie switch in an upstream feeder may indirectly enable downstream load pickup through a sequence of power flow redistributions. To explicitly model such non-local and multi-step couplings, the node embeddings generated by the GAT are further integrated with a Transformer-based encoder within the overall EH-GRL architecture, enabling global information aggregation across the network.

Finally, the node-level representations are aggregated via a global pooling operation to obtain a compact graph-level embedding . This global vector encapsulates both the topological connectivity and the aggregate flexibility of VPPs, providing a comprehensive context for the high-level agent to arbitrate between topological reconfiguration and DER dispatch tasks.

Expert-guided training strategy

To facilitate efficient and stable training of the proposed EH-GRL framework, an expert-guided imitation learning (IL) strategy is introduced to initialize the high-level task-selection policy [26]. In post-fault restoration tasks, purely reward-driven reinforcement learning often suffers from low sample efficiency and unstable early-stage behavior due to the high dimensionality of system states and strict operational constraints. To address these challenges, expert demonstrations derived from restoration trajectory logs generated in the OpenDSS-based simulation environment are leveraged to provide a reliable warm start for policy learning. By distilling feasible and high-quality expert decisions into the neural policy, this strategy effectively guides early exploration toward safe and meaningful regions of the action space, thereby accelerating convergence without replacing the subsequent reinforcement learning phase.

Consistent with the unified action interface, expert demonstrations are provided at the level of concrete control actions rather than abstract task labels. Each expert action belongs to exactly one such subset, thereby implicitly determining the corresponding task type.

Formally, the expert dataset is defined as

(40)

where denotes the graph-structured system state, and represents the expert-selected control action.

The expert demonstration dataset is constructed from restoration trajectory logs generated in the OpenDSS-based distribution system simulation environment. In the current dataset version, 1000 candidate trajectories are first generated, from which 180 trajectories are retained after quality scoring and balanced screening. The final dataset contains N = 1290 state-action pairs, with an average of 7.17 pairs per trajectory (range: 1–10), covering 66 fault scenario types. Data quality is ensured through completeness and type-consistency checks for state, action, and reward records, consistency verification between state-action sequence lengths and action legality, and trajectory quality scoring based on restoration rate, power-loss rate, cumulative reward, and action structure. In addition, balanced screening with upper-bound constraints across fault types and trajectory types is adopted to avoid a small number of scenarios or action patterns dominating the dataset. In the final dataset, the expert actions consist primarily of 866 topology-control actions and 422 DER-regulation actions, with the remaining 2 samples corresponding to terminal actions at restoration completion. The retained demonstrations cover diverse fault types, restoration stages, and action structures, including both topology-control and VPP-level DER-regulation actions. Since each demonstration is represented as a graph-structured state-action pair generated from the OpenDSS-based restoration environment, the corresponding operating-state information is embedded in the graph states, which provides feasible and high-quality initialization samples for expert-guided policy learning.

The associated high-level task label is induced from the action semantics by a deterministic and predefined mapping

(41)

where assigns the task category according to the action subset to which belongs.

During expert-guided pretraining, imitation learning is primarily applied to the high-level task-selection policy. Specifically, the graph-structured state is encoded using the shared topology-aware encoder described in the state encoding module, yielding a graph-level embedding . The parameters are optimized by minimizing the cross-entropy loss

(42)

The detailed expert-guided pretraining process with topology-aware state encoding is summarized in Algorithm 1. In accordance with PLOS ONE submission guidelines, this algorithm is presented as a non-floating text block.

Algorithm 1. Expert-Guided Pretraining with GAT-Based State Encoding.

Require: Expert dataset , encoder parameters , high-level policy parameters , number of epochs E, minibatch size B

Ensure: Pretrained task-selection policy

1: for epoch = 1 to E do

2:  for each minibatch do

3:   Compute embedding

4:   Infer task label

5:   Predict task distribution

6:   Compute loss

7:   Update and by minimizing

8:   end for

9:  end for

10: return Pretrained policy

By supervising task-selection behavior through structured action semantics, the IL phase provides a reliable initialization for the high-level policy without requiring additional expert labeling effort. After pretraining, the policy is further refined through reinforcement learning, allowing the agent to adapt beyond the expert demonstrations and optimize long-term restoration performance.

Experimental setup

This study was conducted on a high-performance computing platform equipped with an AMD Ryzen 7 7800X3D octa-core processor, 32 GB of DDR5 memory, and an NVIDIA GeForce RTX 4060 Ti graphics card, providing sufficient computational power for training and inference of deep reinforcement learning agents.

The test system is a real-world 10 kV distribution network supplied from a 115 kV substation in China, modeled as an 83-node, 97-line graph with two main transformers and dynamic topology reconfiguration capabilities. To emulate a high-penetration active distribution network, Virtual Power Plant (VPP) units are deployed at strategic nodes with specific configurations based on OpenDSS modeling: a 500 kW photovoltaic (PV) station at Node 15; a 500 kW/1000 kWh electrochemical energy storage (ESS) system with 95% round-trip efficiency at Node 45; and an electric vehicle (EV) charging station at Node 23. To reflect demand-side flexibility, the EV station is modeled as a controllable load comprising 300 kW of fixed base load and 200 kW of adjustable charging power, enabling the VPP to modulate power consumption for grid support in unidirectional (grid-to-vehicle) mode.

To capture temporal dynamics beyond static operating conditions, a 24-hour time-series scenario is constructed based on typical daily profiles of PV generation and EV charging behavior (see Fig 2). Notably, the midday period features high PV generation relative to local load, inducing reverse power flow and voltage violations—critical conditions for evaluating the VPP’s auxiliary restoration and voltage support capabilities.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Dynamic test scenario with 24-hour profiles of PV generation and EV charging behavior.

The shaded noon interval highlights the risk of reverse power flow and overvoltage.

https://doi.org/10.1371/journal.pone.0343196.g002

The main training and reward-related hyperparameters are summarized in Table 1. Shared environment and reward settings were kept consistent across all methods, while algorithm-specific optimization parameters followed the configurations listed in Table 1.

All agents shared a common graph encoder for state representation. The encoder consisted of two edge-aware GATConv layers with residual projection, followed by a Transformer encoder and graph-level mean pooling. On top of this shared encoder, DQN used an action-value Q-prediction head, whereas PPO used separate actor and critic heads. Each episode represented a single post-fault restoration trajectory initialized after environment reset with a randomly sampled fault scenario. Episodes were capped at 10 decision steps and could terminate earlier when the terminal action was executed, restoration was successfully completed, or other predefined environment stopping conditions were satisfied.

Comparative evaluation against learning-based and rule-based restoration baselines

To validate the effectiveness of the proposed Expert Knowledge–guided Hierarchical Graph Reinforcement Learning (EH-GRL) method, comparative experiments were conducted against both learning-based and rule-based restoration baselines. For the learning-based comparison, current mainstream deep reinforcement learning methods, PPO and DQN, were first evaluated under identical experimental environments. The reported results of the learning-based baseline comparison among EH-GRL, PPO, and DQN were averaged over three independent training runs with different random initializations.

For fair comparison, EH-GRL and all baseline methods were implemented under the same observation scope, control scope, and candidate action set. Specifically, the heterogeneous control space was unified as a discrete action set consisting of topology-control actions and VPP-oriented DER-regulation actions. For the DER part, each selected action first represents an aggregate VPP-level regulation command, indicating whether the net active-power injection of the corresponding VPP should be increased, maintained, or decreased. This VPP-level command is then realized through the intra-VPP dispatch mechanism described in Section 2.2, where feasible device-level regulation commands and setpoint updates are determined for the controllable PV, ESS, and EV units according to their operating constraints. PPO and DQN operated on the same unified action set, while EH-GRL performed hierarchical task selection followed by task-conditioned action execution. Infeasible topology-control actions and infeasible DER-regulation realizations were filtered using the same legality-check mechanism for all methods, so that the compared approaches differed in decision architecture rather than action availability. The experiments ran for 5000 episodes. Key indicators such as reward, loss function, recovery ratio, and line loss were observed to evaluate the performance of each algorithm in terms of training efficiency and control performance. The overall results are shown in Fig 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Training performance comparison of EH-GRL, PPO, and DQN algorithms: (a) Reward trend showing faster convergence of the proposed method; (b) Loss trend indicating higher training stability for EH-GRL; (c) Recovery ratio trend demonstrating superior restoration capability of the proposed method; (d) Line loss trend reflecting a slight increase due to higher power recovery.

https://doi.org/10.1371/journal.pone.0343196.g003

In terms of convergence during training, the proposed algorithm demonstrates a faster learning speed in the early stages, with the reward curve stabilizing at a high level after approximately 1000 episodes. Based on the statistics of the last 1000 episodes, the average reward reaches 9.8858 ± 0.0301, exceeding those achieved by PPO (9.4679) and DQN (9.4557). The evolution of the loss function further reflects differences in training stability among the three methods: the proposed algorithm attains an average loss of 0.6466 ± 0.0124, which is lower than that of PPO (0.8038) and substantially lower than that of DQN (1.4072). Consistently, the recovery ratio, as the key task execution metric, remains at the highest level throughout training, as shown in Fig 3(c). During the stable phase, the proposed algorithm achieves an average recovery ratio of 96.46% ± 0.0014 with minimal fluctuation, indicating strong robustness, whereas PPO and DQN reach 94.76% and 92.90%, respectively, with DQN exhibiting noticeable instability in the early training stage.

With respect to operational efficiency, Fig 3(d) shows that the average line loss rate of the proposed algorithm is approximately 0.0040, which is slightly higher than that of PPO (0.0039) and DQN (0.0035). This increase is attributable to the algorithm’s strategic preference for restoring a larger amount of load, which shifts the system operating point toward higher power flow levels. Consequently, a moderate increase in line losses is a reasonable trade-off when a substantially higher recovery ratio is achieved.

These results demonstrate that EH-GRL outperforms the learning-based baselines in terms of learning speed, training stability, and restoration effectiveness.

To further examine the influence of the VPP-level DER regulation granularity, a sensitivity analysis was conducted by varying the normalized adjustment ratio in the DER action set . The remaining environment settings, reward coefficients, and training settings were kept unchanged. The tested values of were set to 0.1, 0.2, and 0.3. For each value of , five independent training runs were conducted, and the averaged results were used for plotting and performance evaluation. The evaluation metrics are consistent with those used in the learning-based baseline comparison, including reward, loss function, recovery ratio, and line loss.

As shown in Fig 4, provides the most favorable overall trade-off among the tested settings. Based on the statistics of the last 1000 training episodes, obtains the highest average reward of 10.0328 and the lowest loss of 0.6018, while achieving a recovery ratio of 96.88%. When is increased to 0.2, the average reward decreases to 9.2828, the loss increases to 0.9776, and the recovery ratio decreases to 95.10%. A further increase to leads to lower reward and higher loss, with an average reward of 9.1030 and a loss of 1.1923, while the recovery ratio remains lower than that obtained with , at 95.42%.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Sensitivity analysis of the VPP regulation step size : (a) reward; (b) training loss; (c) recovery ratio; and (d) line loss. All metrics are evaluated under the same training and reward settings, with set to 0.1, 0.2, and 0.3.

https://doi.org/10.1371/journal.pone.0343196.g004

Although the line loss slightly decreases as increases, this should be interpreted together with the restored load level. A lower line loss in this case is mainly associated with a lower recovery ratio and reduced power delivery, rather than indicating a better restoration strategy. Overall, a smaller regulation step size provides finer VPP-level DER adjustment and more stable policy learning, whereas a larger may lead to coarser DER regulation and less stable training. Therefore, is adopted in the main experiments as a balanced setting between regulation granularity, restoration effectiveness, and training stability.

To further assess the practical restoration capability of the proposed EH-GRL method, an additional comparison was conducted against a DER-aware branch-exchange heuristic with power-flow screening and post-restoration DER dispatch, denoted as the DER-BE heuristic. Unlike learning-based methods, this heuristic relies on predefined distribution network operation rules and deterministic screening of candidate restoration schemes. For this rule-based comparison, all 80 fault types considered in the test set were evaluated in each repeated run. The reported performance metrics were then obtained by averaging the results over both the fault types and the four repeated evaluation runs, so as to keep the statistical reporting consistent with the learning-based comparisons.

After a fault occurs, the faulted line is first isolated. The DER-BE heuristic then selects tie switches and sectionalizing switches according to branch-exchange rules, so that the restored network topology maintains a radial operating structure. For each candidate topology, OpenDSS power-flow calculation is performed to verify its feasibility. Candidate schemes that fail to converge, form loops, or violate basic operational constraints are discarded. After a feasible topology is obtained, DER dispatch is further considered. Specifically, the controllable DER resources, including PV, ESS, and EV units, are adjusted through candidate output or load-regulation levels. For each candidate DER dispatch scheme, the DER-supported supply capability, unrestored load, line loss, and voltage constraints are evaluated. The final heuristic solution is selected as a feasible restoration scheme that jointly includes switching operations and DER dispatch.

As shown in Fig 5, EH-GRL achieves a higher average recovery rate of 98.37%, compared with 97.91% obtained by the DER-BE heuristic. Meanwhile, EH-GRL results in a lower unrestored load of 151.7 kW, whereas the DER-BE heuristic leaves 194.5 kW unrestored. This indicates that the proposed method can identify more effective restoration strategies through hierarchical coordination between topology control and VPP-oriented DER regulation. In terms of network operating efficiency, EH-GRL also achieves a lower line loss of 42.5 kW, compared with 61.5 kW for the DER-BE heuristic. More importantly, the average number of switching operations is significantly reduced from 1.91 for the DER-BE heuristic to 0.91 for EH-GRL. This result suggests that EH-GRL can exploit DER regulation more effectively to support load restoration, thereby reducing the reliance on frequent switching operations while maintaining a high restoration level.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Restoration performance comparison between EH-GRL and the DER-BE heuristic: (a) recovery rate; (b) unrestored load; (c) line loss; and (d) switching operations. DER-BE denotes a DER-aware branch-exchange heuristic with power-flow screening and post-restoration DER dispatch.

https://doi.org/10.1371/journal.pone.0343196.g005

Overall, the comparative results show that EH-GRL not only outperforms learning-based baselines such as PPO and DQN, but also achieves superior restoration performance compared with a DER-aware rule-based branch-exchange heuristic. The results further confirm that the proposed expert-knowledge-guided hierarchical decision framework can improve both restoration effectiveness and operational efficiency.

Analysis of VPP-assisted support effectiveness

As illustrated in Fig 6, the bus voltage profiles with and without VPP participation are compared during peak PV output. This section presents simulation-based validation using the actual distribution system topology. The studied distribution network exhibits a robust structure with a large short-circuit capacity, providing a strong inherent capability to suppress voltage fluctuations.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of bus voltage profiles with and without VPP support.

The top panel shows voltage violations caused by PV reverse power flow without VPP participation, while the bottom panel demonstrates effective mitigation of overvoltage and a flattened voltage distribution with VPP-assisted regulation.

https://doi.org/10.1371/journal.pone.0343196.g006

To investigate the voltage support capability of the VPP under stressed operating conditions, the PV penetration level is temporarily increased by scaling the installed PV capacity to 1 MVA, thereby creating a severe reverse power flow scenario for voltage regulation assessment.

Voltage regulation performance under high PV penetration. First, the system operating condition without VPP support is analyzed. As shown in the top panel of Fig 6, when the 1 MVA PV unit operates at full output, limited local load consumption leads to significant reverse power flow. The per-unit voltage at several buses exceeds the upper operational limit of 1.05 p.u., reaching a maximum value of approximately 1.0565 p.u., which results in voltage violations.

Subsequently, the VPP is introduced to participate in grid auxiliary services. As illustrated in the bottom panel of Fig 6, bus voltages are effectively regulated through the coordinated control of heterogeneous DERs. Through this targeted, node-level DER coordination, voltage violations are completely eliminated, voltage peaks are significantly suppressed, and the overall voltage profile becomes smoother and more balanced. These results demonstrate that even in a strong grid environment, VPP-assisted DER coordination plays a critical role in maintaining voltage security.

Nighttime fault restoration performance without voltage violations. To further examine whether the benefits extend beyond voltage regulation, additional experiments are conducted under nighttime conditions (21:00), where PV generation drops to zero. In the non-VPP baseline, DER units remain installed in the system but are not aggregated or actively dispatched during restoration; only network reconfiguration actions are allowed. The performance comparison of EH-GRL-based fault restoration strategies is summarized in Table 2.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Performance comparison of EH-GRL-based fault restoration strategies with and without VPP support under nighttime conditions.

https://doi.org/10.1371/journal.pone.0343196.t002

As shown in Table 2, the VPP-supported EH-GRL strategy achieves better load recovery performance under nighttime conditions. Since PV generation is unavailable at night, the improvement mainly comes from the coordinated use of ESS and flexible EV load. For fault lines 009041, 014078, and 005015, the VPP-assisted strategy restores all interrupted loads, while the non-VPP strategy cannot achieve full restoration. For fault lines 073075 and 069071, although the recovery ratio is still below 100%, it is higher than that of the non-VPP baseline. This indicates that VPP resources can provide additional local support and help cover the remaining load that cannot be restored by network reconfiguration alone.

Compared with the non-VPP strategy, which mainly relies on multiple switching operations to establish alternative supply paths, the VPP-assisted strategy can coordinate ESS discharge, ESS charging, and EV load adjustment to improve restoration flexibility. In terms of power loss, the results should be interpreted together with the restored load level. For example, the VPP-supported strategy reduces the power loss in the case of fault line 005015, while in some other cases the loss slightly increases because more load is restored and more power is delivered through the network. Therefore, the main contribution of VPP support in these nighttime scenarios is reflected in improving load recovery capability and providing additional operational flexibility, while maintaining acceptable operating performance.

Validation of expert-guided optimization mechanism

This section evaluates the effectiveness of the expert-guided optimization mechanism, with a particular focus on how imitation learning (IL) influences policy decision confidence and training stability. For this purpose, the EH-GRL models used in this subsection are trained as a separate set of experimental instances, with training configurations specifically tailored to the analysis of expert guidance. The evaluation metrics are kept consistent with those adopted across other experimental sections to ensure comparability.

To further examine the training behavior of the imitation learning stage, the expert demonstration dataset was divided into training and validation subsets with a ratio of 4:1. Fig 7 shows the behavior cloning loss curves during expert-guided imitation learning. The dashed curve represents the total optimization loss used for updating the policy network, while the solid curves denote the behavior cloning losses on the training and validation subsets, respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Behavior cloning loss curves during expert-guided imitation learning.

The expert demonstration dataset is divided into training and validation subsets with a ratio of 4:1. The dashed curve denotes the total optimization loss used for policy-network updating, while the solid curves represent the behavior cloning losses on the training and validation subsets.

https://doi.org/10.1371/journal.pone.0343196.g007

As shown in Fig 7, the total optimization loss decreases rapidly during the first few epochs and then reaches a relatively stable level, indicating that the policy network can effectively learn from the expert demonstrations. The training loss remains lower than the validation loss throughout the imitation learning stage, while both curves exhibit stable trends without severe oscillation. Since the validation loss remains stable and does not show an increasing trend, no obvious overfitting is observed under the adopted 4:1 data split. This result suggests that the expert-guided pretraining process provides a stable supervised initialization for the policy.

After confirming the stable convergence of the imitation learning process, we further analyze its effect on policy decision confidence. Fig 8 illustrates the distribution of the action probability gap between the most likely action and the second most likely action in the policy output, before and after the imitation learning phase.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Distribution of action probability gap before and after imitation learning.

(a) Initial random exploration with low decision confidence; (b) After expert-guided imitation learning, showing increased confidence and clearer separation between the most likely and second most likely actions.

https://doi.org/10.1371/journal.pone.0343196.g008

As shown in Fig 8(a), before introducing imitation learning (i.e., during the initial random exploration phase), is extremely small, with its distribution mainly concentrated near 0, corresponding to a magnitude of only 10⁻⁵. This indicates that without expert guidance, the agent struggles to effectively distinguish between optimal and sub-optimal actions, exhibiting high uncertainty in the decision-making process.

After introducing expert knowledge for imitation learning, the decision characteristics of the policy changed significantly. As shown in Fig 8(b), the distribution of shifted entirely to the right, centered around 0.610, and presented a concentrated single-peak morphology. Most sample probability differences were concentrated in the 0.600 to 0.620 interval. This result indicates that imitation learning enables the agent to assign higher confidence to actions recommended by experts, thereby forming a clearer and more stable decision preference. This demonstrates that expert knowledge provides effective supervision signals for policy learning, laying a good initialization foundation for the subsequent reinforcement learning phase.

To isolate the contribution of expert knowledge guidance, an ablation study is conducted by removing imitation learning and expert supervision from the proposed EH-GRL, yielding a baseline denoted as Hierarchical Graph Reinforcement Learning (H-GRL). The reported results of this ablation comparison were averaged over three independent training runs with different random initializations. Fig 9 compares the two methods in terms of training stability and fault restoration performance. EH-GRL demonstrates faster convergence and superior stability in the core restoration metric. As shown in Fig 9(c), EH-GRL achieves an average recovery ratio of 94.51%, whereas H-GRL attains 89.26%, with the standard deviation reduced from 0.17% to 0.08%. Consistent improvements are also observed in the loss function (Fig 9(b)), where EH-GRL converges to a lower and smoother level (1.0023 ± 0.0087), while the corresponding value for H-GRL is 1.1572 ± 0.0051, indicating more accurate value estimation and enhanced training stability.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Ablation study comparing EH-GRL and H-GRL methods.

(a) Reward distribution showing reduced variance and more consistent performance in EH-GRL; (b) Loss trend indicating smoother convergence and lower estimation bias for EH-GRL; (c) Recovery ratio trend demonstrating higher restoration success rate and policy stability with expert knowledge guidance.

https://doi.org/10.1371/journal.pone.0343196.g009

Regarding the reward metric (Fig 9(a)), H-GRL attains a slightly higher average reward during the stable period, reaching 9.5820, while EH-GRL yields an average reward of 9.5117. However, the reward variance of H-GRL is nearly twice that of EH-GRL, amounting to 1.1966 and 0.6095, respectively. This observation suggests that although the removal of expert guidance may encourage more aggressive exploration, it also introduces substantially higher uncertainty and operational risk. In contrast, the more concentrated reward distribution of EH-GRL reflects safer and more reliable decision-making, which is more desirable for safety-critical fault restoration scenarios.

Synthesizing the above results, for power system restoration scenarios with extremely high requirements for safety and robustness, EH-GRL incorporates safety constraints derived from expert knowledge into the policy learning process. While making a moderate trade-off in average economic indicators, it achieves an approximately 5.25% increase in restoration success rate and significantly reduces the volatility of policy output. Although EH-GRL exhibits a slightly higher line loss compared with H-GRL, which achieves 0.0033 ± 0.0006, its overall performance demonstrates clear advantages in terms of safety and stability. These results demonstrate that the proposed Expert-Guided Optimization learning mechanism better meets the requirements for safe and stable system operation in practical engineering applications.