2.1. System and algorithm design

We consider a local smart grid community composed of N prosumers and one Mobile Edge Computing (MEC) server co-located with the distribution substation. Each prosumer is equipped with a rooftop PV panel and an embedded computing device that connects to the MEC server via a single-hop wireless uplink. We assume orthogonal frequency-division multiple access (OFDMA), where the total system bandwidth B is equally partitioned among the prosumers that choose to offload in a given slot. Specifically, if prosumers offload at slot t, each receives an effective bandwidth of , so that heavy simultaneous offloading directly increases per-user transmission delay. There is no intermediate gateway; each prosumer communicates with the MEC server directly. The system operates in discrete time slots of duration 𝜏.

As shown in Fig 1, each prosumer decides whether to execute a task locally or offload it to the MEC server. Let be the task size (bits) of prosumer i at slot t, and the fraction executed locally.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. The proposed hybrid Deep Reinforcement Learning framework.

https://doi.org/10.1371/journal.pone.0342888.g001

For task delay, local computation time is

(1)

where is CPU cycles per bit and is local CPU frequency. The offloaded part is sent over a wireless link. For the wireless transmission, we assume a quasi-static Rayleigh fading channel where the channel gain remains constant within a time slot but varies across slots. The transmission rate is given by Shannon’s formula in (2).

(2)

yielding transmission delay . Total delay is .

Regarding the energy model, the total energy consumption comprises two parts: CPU processing and wireless transmission. It is important to note that we also consider a baseline static power consumption for maintaining device connectivity. For energy and trading, local computation consumes energy. Thus, the refined energy consumption model is:

(3)

where transmission uses .

It is important to note that the present model does not include local battery storage. All harvested solar energy must be either consumed locally within the same slot or traded with peers; any unconsumed and untraded surplus is curtailed. Consequently, the agent cannot physically “store” energy across time slots. The economic benefit of the learned policy arises instead from computational load shifting: by choosing between local execution and offloading, the agent reshapes the prosumer’s instantaneous net demand to align with prevailing price signals within each slot. Incorporating battery dynamics—with state-of-charge transitions, efficiency losses, and degradation—is a natural extension that we leave to future work.

We also note that since each prosumer’s CPU frequency is a fixed hardware parameter rather than a decision variable, the energy expression in Eq. (3) is linear in the offloading fraction , which is the actual control variable. If dynamic voltage and frequency scaling (DVFS) were introduced, the well-known cubic relationship would need to be adopted. We treat the fixed-frequency assumption as a simplification appropriate for the class of embedded IoT devices considered here, and flag DVFS integration as a modeling extension.

Let be harvested solar energy. The net imbalance determines trading: if positive, buy at price ; if negative, sell at .

To avoid grid dependence, we enforce community energy balance:

(4)

where the MEC server has capacity is constrained by Equation (5).

(5)

In practice, collecting per-prosumer state information (channel gains, task sizes, harvested energy) and disseminating offloading decisions introduces signaling overhead. Under typical 5G New Radio control-plane procedures, the round-trip signaling latency is on the order of 1–2 ms with sub-milliwatt energy consumption per exchange. Given that the slot duration 𝜏 in our setting is on the order of seconds, this overhead is negligible relative to computation and data-transmission times. We therefore treat coordination as instantaneous in our model. We acknowledge, however, that in dense deployments with hundreds of prosumers, signaling congestion could become non-negligible, motivating decentralized or hierarchical coordination architectures.

To ensure the realism of the simulation, we utilize high-resolution smart meter data analytics methodologies [12] to preprocess the input profiles. Furthermore, the integration of edge intelligence into smart grids necessitates a robust multi-agent resource allocation framework, which has been extensively studied in the context of edge computing [13]. Our work aligns with the broader concept of Transactive Energy (TE) systems, which facilitate decentralized coordination among prosumers [14]. Recent advances in this domain highlight the importance of addressing both architectural challenges and market mechanisms in distribution systems [15]. The proposed edge-enabled smart grid architecture also reflects the current trends towards IoT-enabled infrastructure, as discussed in recent surveys [16].

The system aims to maximize a utility combining throughput, energy use, and trading profit:

(6)

where is the monetary gain (or cost) from energy trading, and , , are non-negative weights reflecting operational priorities. This linear combination is widely used in multi-objective optimization for smart grids due to its interpretability and ease of tuning.

Since the problem is sequential and uncertain, we use a DQN to choose , followed by a rule-based step to adjust trades and satisfy Equation (4). The per-slot procedure is formalized in Algorithm 1.

Algorithm 1: Per-Slot Joint Offloading and Energy Trading

Input: , , , ,

Output: ,

1  // Step 1: Offloading via DQN

2    

3    ,

4  // Step 2: Energy imbalance

5    for i = 1 to N do

6      Compute

7          

8    end for

9  // Step 3: Balance trading

10    

11    

12    

13    for i = 1 to N do

14     if

15        then

16    else

17        

18    end for

19    return ,

The proportional adjustment in Steps 10–18 of Algorithm 1 functions as a centralized community-level settlement layer, operationally analogous to the pro-rata curtailment rules used in pool-based local energy markets when aggregate supply and demand are mismatched. While this mechanism enforces strict per-slot energy balance, it does not constitute a fully decentralized peer-to-peer negotiation. We adopt it here for tractability and to guarantee constraint satisfaction, but recognize that replacing it with an incentive-compatible mechanism—such as a double auction or bilateral matching protocol with network-aware pricing—would better align with the P2P paradigm and is an important avenue for future work.

2.2. Consolidated problem formulation

Bringing together the models above, the joint optimization problem over a horizon of T slots is stated as follows:

(7)

subject to:

(8)

The discrete offloading levels make the problem mixed-integer, while the Shannon rate in Eq. (2) introduces nonlinearity into both the delay and energy expressions, rendering the formulation a mixed-integer nonlinear program (MINLP). The temporal coupling through the discount factor 𝛾 and the stochastic nature of renewable generation, channel gains, and prices make exact solution intractable in real time, motivating the DRL-based approach described next.

3.1. Simulation setup

We model the system as a Markov Decision Process (MDP) defined by the tuple .

1. 1. State Space : The state at time t is a vector that captures all information necessary for decision-making:

(9)

where is an estimate of the MEC server’s current workload (e.g., the number of CPU cycles queued). This provides the agent with awareness of both local conditions and shared resource pressure.

1. 2. Action Space : The action is the joint offloading decision for all prosumers, . To make the problem tractable for DQN, we discretize each into three levels: full local execution (), half offloading (), and full offloading (). Thus, the total number of joint actions is 3^N. For N = 20, this is large but manageable via function approximation.
2. 3. Reward Function : The immediate reward is the system utility defined in Equation (2), adjusted to penalize constraint violations:

(10)

where is a penalty coefficient. This soft-constraint handling encourages the agent to learn feasible policies without explicitly restricting the action space.

1. 4. Transition Dynamics : The state transition is governed by exogenous processes (e.g., solar irradiance, task generation) and is unknown to the agent, which is precisely why a model-free RL approach is suitable.

3.2. DQN architecture and training

The core of our framework is a fully connected neural network that approximates the action-value function at the individual prosumer level. Rather than maintaining a single output layer of 3^N nodes—which would be computationally prohibitive—we adopt a parameter-sharing scheme: one neural network is trained and shared across all prosumers. At each slot, the network is called N times, once per prosumer, with a prosumer-specific state vector that includes the local task size , channel gain , harvested energy , market prices and , and an estimate of the current MEC workload . Each call produces Q-values over three actions {0, 0.5, 1}. This reduces the effective per-call action space from 3²⁰ to 3, making standard DQN entirely tractable. The joint effect across prosumers emerges implicitly: the shared MEC load feature provides each prosumer with awareness of aggregate offloading pressure, and the capacity-violation penalty in Eq. (10) discourages collectively infeasible outcomes. The network consists of three hidden layers with 128, 64, and 32 neurons respectively, using ReLU activations.

Training follows the standard DQN algorithm with two key enhancements:

1. Experience Replay: Transitions are stored in a replay buffer of size 10,000. For each training step, a mini-batch of 64 transitions is sampled uniformly to break temporal correlations.
2. Target Network: A separate target network with weights is used to compute the target Q-values, and its weights are updated by slowly tracking the main network: with .

The loss function for a mini-batch is the mean squared Bellman error:

(11)

where is the discount factor. The agent is trained offline using historical or synthetic data streams that capture realistic diurnal and seasonal patterns. Specifically, we use one month of scaled PV generation data from the NREL SAM database and corresponding PJM market price data. After training, the learned policy is deployed online for real-time decision-making.

It is important to note that our framework does not employ meta-learning, multi-agent coordination, or online fine-tuning. The focus is on learning a single, robust policy that can generalize across different days within the same operational context, which aligns with the practical deployment scenario of a fixed edge-grid testbed.

3.3. Complexity analysis

The computational complexity of the proposed framework consists of two phases: offline training and online inference. During training, the complexity is dominated by the backpropagation through the Deep Q-Network. For a network with L layers and N_neu neurons per layer, the complexity per step is . In the online inference phase, the prosumer only needs to perform a forward pass, which is computationally lightweight () and well-suited for edge devices. The heuristic energy balancing step (Algorithm 1, Steps 10–18) involves simple sorting and summation operations, with a complexity of where N is the number of prosumers. This low latency ensures that decisions can be made within the strict timing requirements of the smart grid control loop.

4.1. Simulation setup

We simulated a local energy community comprising N = 20 prosumers, each equipped with a rooftop PV panel and an embedded computing device. The key simulation parameters are listed in Table 1:

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Key Simulation Parameters.

https://doi.org/10.1371/journal.pone.0342888.t001

Three baseline methods were compared against our proposed DQN-based approach:

1. Local-only execution: All tasks are executed locally; energy trading is performed only to cover local deficits.
2. Greedy heuristic: Offload if channel gain exceeds a threshold; trade energy greedily based on current price signals.
3. Proposed DQN+Heuristic: Combines DQN for offloading decisions with a rule-based strategy for energy trading.

4.2. Experimental results

The performance of each method was evaluated using the following metrics:

1. Average system utility: Measures the overall benefit across all prosumers in terms of computation throughput, energy conservation, and economic return.
2. Average task delay: Reflects the average time taken to complete tasks.
3. Energy trading revenue: Captures the net monetary gain from selling surplus energy or purchasing deficit energy.

As shown in Fig 2, the proposed DQN+Heuristic approach converges to a high-utility regime after approximately 1000 episodes. In the initial 500 episodes the agent explores the action space, producing volatile rewards; beyond episode 1000 it consistently outperforms the Greedy Heuristic. The key mechanism is computational load shifting rather than energy storage. During slots with high electricity prices, the agent preferentially offloads tasks to the MEC server, thereby reducing local CPU energy consumption and increasing the instantaneous solar surplus available for sale to peers at the prevailing high price. During low-price slots, the agent favors local execution, productively absorbing surplus solar energy rather than selling it cheaply. This price-responsive reshaping of the demand profile—within each slot, without any inter-temporal storage—is the primary driver of the long-term utility improvement. The Greedy Heuristic, which reacts only to instantaneous channel quality, cannot exploit this price-computation coupling.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Average System Utility Over Time.

https://doi.org/10.1371/journal.pone.0342888.g002

As shown in Fig 2, the proposed DQN+Heuristic approach achieves a rapid convergence to a high utility state. In the initial 500 episodes, the agent explores the trade-off space, resulting in fluctuating rewards. However, beyond episode 1000, the proposed method consistently outperforms the Greedy Heuristic. The superiority of the DQN stems from its foresight: strictly greedy strategies often deplete battery energy during low-price periods to save computation time, leaving no energy to sell when prices peak later in the day. The DQN agent learns to “hoard” energy or offload tasks conservatively in anticipation of future price spikes, thereby maximizing long-term utility.

Average task delay for each method is exhibited in Fig 3. The proposed approach shows a significant reduction in task delay compared to the Local-only and Greedy heuristics, highlighting the benefits of intelligent task offloading.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Average Task Delay.

https://doi.org/10.1371/journal.pone.0342888.g003

Fig 3 illustrates the average task delay. While the “Local-only” strategy suffers from the limited CPU capacity of IoT devices (approx. 450ms), the proposed method reduces this to 310ms. Interestingly, the Greedy approach achieves 380ms, which is worse than ours. This is counter-intuitive since greedy methods usually prioritize immediate speed. The reason lies in congestion avoidance: the greedy heuristic tends to offload tasks whenever the channel is good, causing bottlenecks at the MEC server (violating the soft capacity constraints). Our DQN agent learns to estimate the MEC workload and chooses to compute locally when the server is crowded, effectively balancing the load and reducing queuing delays.

The financial impact is detailed in Fig 4. The proposed method improves revenue (or reduces cost) by approximately 11%. By jointly optimizing offloading, the system effectively treats “computation” as a flexible load. During periods of high solar generation and low electricity prices, the agent encourages local computation to consume surplus energy. Conversely, when grid prices are high, the agent offloads tasks to the edge (powered by the utility scale grid or its own resources), freeing up local solar energy to be sold to the community for profit.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Energy Trading Revenue.

https://doi.org/10.1371/journal.pone.0342888.g004

Crucially, the DQN agent exhibits price-sensitive behavior: during intervals of high grid electricity prices, the agent tends to offload more tasks to reduce local consumption, thereby maximizing the volume of solar energy available for sale. Conversely, when prices are low, it reverts to local computing, preserving the limited edge bandwidth for more critical tasks. This strategic load shifting is the primary driver of the 11% revenue improvement.

Table 2 summarizes the key performance metrics for each method. The proposed DQN+Heuristic approach outperforms both baselines, achieving a 12.3% higher average utility, a 16.5% reduction in average task delay, and approximately 11% improvement in energy trading revenue.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Summary of Performance Metrics.

https://doi.org/10.1371/journal.pone.0342888.t002

To validate robustness, we varied each weight by ±50% from its nominal value while holding the other two fixed. Table 3 reports the results. Average utility and delay are relatively insensitive to perturbations in and : a 50% increase in raises utility by only 3.1% while delay drops by 4.2%, reflecting a mild shift toward throughput maximization. The system is more sensitive to : increasing it by 50% boosts trading revenue by 8.7% but raises average delay by 6.3%, since the agent aggressively offloads to free solar energy for sale at the cost of MEC congestion. These results confirm that the nominal weights provide a balanced operating point, but also that practitioners can tune to prioritize economic objectives when grid conditions warrant it.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Sensitivity of Performance to Utility Weights (±50% Perturbation).

https://doi.org/10.1371/journal.pone.0342888.t003

Since the objective maximizes aggregate utility, one may ask whether certain prosumers are systematically disadvantaged. We computed Jain’s fairness index over per-prosumer time-averaged utilities.

(12)

The proposed method achieves 0.91, indicating that benefits are distributed reasonably evenly. This is partly attributable to the proportional rationing in Algorithm 1, which scales all trades by the same factor 𝛽 and thus avoids preferential treatment. Nonetheless, the current formulation does not explicitly guarantee individual rationality—i.e., that every prosumer is better off participating than opting out. Incorporating such guarantees via game-theoretic mechanism design is an important direction for future work.

4.3. Constraint satisfaction

To ensure practical feasibility, we monitored the satisfaction of two critical constraints. Firstly, the edge server capacity needs to be considered, the total workload on the MEC server did not exceed its processing capacity throughout the simulation period. Violation rates were below 5%, primarily occurring during peak load periods. Secondly, we should notice the community-level energy balance. Net energy exchange with the main grid was zero at each time slot, confirming the effectiveness of our proportional adjustment mechanism for energy trading. Fig 5 depicts the temporal dynamics of edge server capacity utilization. The proposed method maintains compliance with the edge server capacity constraint, with occasional minor violations during peak hours.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Temporal Dynamics of Edge Server Workload and Capacity Violation.

https://doi.org/10.1371/journal.pone.0342888.g005

Fig 6 exhibited the net energy exchange with the main grid. The proposed DQN method (red) strictly maintains the net grid exchange around zero, satisfying the self-sufficiency constraint. In contrast, the baseline strategies exhibit significant fluctuations (Duck Curve pattern), with the Local-only strategy (brown) incurring higher grid dependence due to intensive local computation.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Verification of Community-Level Energy Neutrality.

https://doi.org/10.1371/journal.pone.0342888.g006

This strict adherence to the zero-net-exchange boundary implies that the community operates as a virtual island from the grid operator’s perspective. This characteristic is highly valuable for grid stability, as it eliminates the need for the upstream distribution system operator (DSO) to reserve spinning reserves for this community’s load fluctuations.

4.4. Scalability discussion

The per-prosumer parameter-sharing scheme described in Section 3 Proposed DRL-Based Optimization Framework makes inference linear in N and avoids the combinatorial explosion of the joint action space. However, this decomposition treats each prosumer’s decision as approximately independent given the shared MEC load feature, which sacrifices explicit inter-agent coordination. For moderate community sizes (𝑁 ≤ 20), the capacity-violation penalty provides sufficient implicit coupling, as evidenced by the low constraint violation rates reported above. For larger communities, a Multi-Agent Reinforcement Learning (MARL) framework, such as QMIX or MAPPO operating under the centralized-training-decentralized-execution (CTDE) paradigm, which would enable richer coordination while retaining per-agent scalability at inference time. Mean-field approximation, which replaces individual agent interactions with an average effect, offers another promising path when N grows to hundreds. Investigating these extensions is a priority for future work.