2.1 Neural operators and surrogate modeling

Neural operators have revolutionized the use of machine learning for PDEs by learning mappings between infinite-dimensional function spaces rather than point-wise input-output relationships. DeepONet [31] pioneered this direction through a branch-trunk architecture that approximates nonlinear operators with theoretical guarantees of universal approximation. Fourier Neural Operators (FNOs) extend this paradigm using spectral methods, achieving discretization-invariance and superior efficiency on regular grids [39]. Recent physics-informed variants (PINO, PI-DeepONet) [42] combine data fidelity with PDE residual constraints, yielding 1–2 orders of magnitude improvement in data efficiency. Quantum acceleration of DeepONet [53] and synthetic data generation via the method of manufactured solutions [51] have further reduced training costs and expanded applicability.

Crucially for control, recent work has moved neural operators from prediction to implementation. Bhan, Krstic, and Shi (2025) [38] demonstrated that neural operators can learn PDE backstepping gain kernels while preserving Lyapunov stability guarantees under approximation error a foundational result for ROL’s theoretical motivation. Song et al. [54] developed operator learning for non-smooth optimal PDE control, combining primal-dual optimization with neural operators. These works collectively position neural operators as not merely surrogates for prediction, but as active components of control law synthesis.

2.2 Deep reinforcement learning for chaotic systems and PDEs

Deep RL has shown promise for learning feedback policies in high-dimensional dynamical systems without explicit models. Seminal work by Rabault et al. [23] and Bucci et al. [24] demonstrated deep RL for flow control in aerodynamic applications, achieving drag reduction and wake stabilization in simulation. For the KS equation specifically, Zeng et al. [28] employed deep RL to learn symmetry-aware stabilizing policies. More recently, KAN enhanced DRL (2024) [49] introduced Kolmogorov-Arnold Networks to chaotic control, achieving rapid stabilization via minor perturbations. Alternative modern architectures, such as Transformer-based methods Valle et al. [50], have computed minimum control bounds for chaotic systems with fast convergence. Despite these advances, pure deep RL suffers from two fundamental challenges: (i) sample inefficiency, as exploration in high-dimensional spaces is combinatorially expensive, and (ii) training instability, requiring careful hyperparameter tuning and often producing high-variance policies. For PDE control, where each environment step incurs a costly numerical solve, these issues are magnified. Our ROL framework addresses this by pre-training a neural operator offline, thereby amortizing the exploration cost and providing TD3 with a warm-started policy that reduces variance.

2.3 Classical and model-predictive control

Classical approaches to KS control leverage linear systems theory and geometric control. Early works established output-feedback stabilization and finite-dimensional feedback via Galerkin truncation. Nonlinear backstepping, developed by Liu et al. [8], provides global stabilization proofs under suitable conditions but is computationally demanding during deployment due to repeated kernel solves. Model predictive control (MPC) offers constraint handling but scales poorly to high resolutions and remains computationally expensive. In this study, LQR serves as a classical baseline, linearizing the KS equation and solving the continuous-time algebraic Riccati equation. While LQR is interpretable and theoretically grounded, its linear assumptions are fundamentally incompatible with KS’s strong nonlinearity, resulting in poor stabilization.

2.4 Hybrid and transfer learning approaches

The idea of combining pre-training with reinforcement learning has gained traction in the broader machine learning community. Schulman et al. [55] introduced trust region policy optimization (TRPO) with warm-start initialization, reducing exploration episodes by 2–3 times. More recently, offline RL and batch RL methods [30] leverage large pre-collected datasets to initialize policies, reducing online interaction costs. In the context of PDEs, the hypernetwork-based DRL framework (HypeRL) [48] embeds parameter dependencies directly into policies to enable generalization across parametric families of PDEs. ROL aligns with this broader trend of combining data-driven priors (here, DeepONet) with adaptive learning (here, TD3). Here the contribution is the explicit integration of operator learning which learns generalizable control maps at the operator level with residual policy refinement, demonstrating that this hybrid can outperform pure RL while remaining more adaptive than offline methods. ROL occupies a unique position in this landscape. Unlike pure neural operators, it adapts online through RL. Unlike pure deep RL, it leverages a pre-trained operator to reduce exploration and variance. Unlike classical control, it handles strong nonlinearity without explicit modeling or kernel solves. The trade-off is relinquishing formal stability guarantees and explainability, a compromise we believe is justified when sample efficiency and adaptivity are prioritized. The framework does not claim to solve all aspects of the chaos control problem, rather it identifies a practical optimal spot for moderate-dimensional systems with budget constraints on exploration and a tolerance for empirical rather than formal verification.

3.1 Problem formulation

3.1.1 Kuramoto-Sivashinsky equation.

The KS equation, introduced to model chaotic dynamics in reaction-diffusion systems [2] and flame instabilities [3], is defined as:

(2)

where represents the system state (e.g., velocity or temperature), is the spatial coordinate with domain length L = 100, is time, is the viscosity parameter controlling higher-order dissipation, and is the control input to be designed. Periodic boundary conditions are enforced, ensuring  +  . The nonlinear term drives chaotic behavior, while the second-order () and fourth-order () derivatives provide diffusive stabilization.

The control objective is to design f(x, t) to minimize the system’s energy, defined as the mean squared state over the spatial domain:

(3)

where n_x = 64 is the number of discretized spatial points, and x_j are the grid points. Minimizing E(t) indicates effective suppression of chaotic dynamics.

Parameters: domain length L = 100, number of spatial points n_x = 64, viscosity .

3.2 Numerical solution using spectral methods

The KS equation is solved numerically using a spectral method based on the Fast Fourier Transform (FFT), which is well-suited for PDEs with periodic boundary conditions [3]. The spatial domain [0,L] is discretized into n_x = 64 equally spaced points, yielding a spatial resolution of . The spatial grid is defined as:

(4)

and the time step is set to . The wave numbers are computed as:

(5)

using the FFT frequency function .

Let denote the discretized state vector, and its Fourier transform. The linear terms and correspond to and , respectively. The nonlinear term is expressed as , with its Fourier transform given by . The state is updated using a semi-implicit time-stepping scheme:

(6)

where is the Fourier transform of the control input. The updated state is computed as , and clipped to the range [–8,8] to ensure numerical stability.

The initial condition for each simulation is:

(7)

where is Gaussian noise introducing variability to the initial state.

Parameters: domain length L = 100, number of spatial points n_x = 64, viscosity , time step , initial condition noise standard deviation .

3.3 Pure Reinforcement Learning (RL)

The pure RL approach employs the TD3 algorithm [21] to learn a control policy directly mapping system states to control actions, implemented in a custom environment (PureRLKSEnv). TD3 is an off-policy RL method designed for continuous action spaces, improving upon DDPG by using twin critic networks, delayed policy updates, and target policy smoothing to enhance stability [21].

3.3.1 State space.

The state space consists of the discretized KS field at time t:

(8)

where s_t represents the system state vector with n_x = 64 spatial points. TD3 receives the current state s_t = u(x,t), a 64-dimensional vector (shape: [64]) from the KS solver, representing the discretized KS field.

3.3.2 Action space.

The action space is the control input applied to the KS equation:

(9)

where a_t is a vector of control values corresponding to each spatial point.

3.3.3 Reward function.

The reward function is designed to promote energy reduction while penalizing large control actions:

(10)

where is the mean squared state (energy), and is the mean squared control action. The coefficient 50 amplifies the energy reduction term to prioritize stabilization, while the penalty term encourages smooth and minimal control inputs.

Parameters and hyperparameters: maximum episode length 100 steps (total time ⋅ ), learning rate 2 × 10⁻⁴, batch size 128, replay buffer size 40,000, discount factor , soft update parameter , action noise standard deviation , policy update delay every 2 gradient steps, actor and critic networks with three hidden layers of 256 units each, random seed 42.

3.4 Reinforcement Operator Learning (ROL)

The ROL framework integrates DeepONet [31] for offline learning of a generalized control operator with TD3 for online refinement of trajectory-specific control policies, implemented in the ResidualDeepONetKSEnv environment. This hybrid approach leverages DeepONet’s ability to approximate complex mappings and TD3’s adaptability to dynamic conditions. Fig 1 demonstrates the DeepONet-Guided Reinforcement Operator Learning architecture for Stabilizing the Kuramoto–Sivashinsky Equation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A hybrid DeepONet-guided reinforcement learning framework for stabilizing the Kuramoto–Sivashinsky equation.

https://doi.org/10.1371/journal.pone.0341161.g001

3.4.1 DeepONet for learning the control operator.

DeepONet is a neural operator designed to learn the mapping

where u(x, t) is the system state, t is time and f(x, t) is the control input. The architecture consists of three components:

Branch network: Processes the discretized state . It comprises three fully connected layers:

(11)

where LayerNorm normalizes activations, ReLU () introduces nonlinearity, and dropout with probability 0.1 prevents overfitting.

Trunk network: Processes the scalar time input . It consists of two fully connected layers:

(12)

Fusion and output: The branch and trunk outputs are combined via element-wise multiplication (Hadamard product), followed by a linear transformation:

(13)

where denotes the Hadamard product, and matches the spatial resolution. The model parameters are denoted by .

Training data generation with MPC: The training dataset consists of 2000 trajectories, each built from: 1) a noisy initial field u₀(x) drawn from (7), and 2) a time label .

For efficiency we do not evolve u₀ to time t before calling the expert; instead we directly compute the MPC control acting as if u₀ were encountered at time t. Although this creates a distribution shift between training inputs (u₀) and deployment inputs (u(x, t)), we found empirically that the combination of (i) strong additive noise () around u₀, (ii) the autonomous nature of the KS dynamics and (iii) the residual TD3 layer is sufficient to bridge the gap (see Sect 3.8). Formally, for each sample we solve

(14)

subject to the linearized dynamics , with constraints . Here, − , , and the MPC horizon is . The state and control weights are Q = 1.0 and R = 0.01, respectively. The optimization is solved using the OSQP solver via the CVXPY library, producing control inputs for each state-time pair.

Justification of the training-deployment mismatch. The choice of sampling only from the initial-condition manifold offers two practical advantages: (a) it reduces dataset generation time by an order of magnitude, and (b) it yields a remarkably diverse set of spectra once Gaussian noise is injected, covering the energy range that the KS attractor explores during transients. In addition, TD3 learns a bounded residual action that corrects any systematic bias in the DeepONet prior.

Training procedure of DeepONet: DeepONet is trained on a dataset of triplets , where the branch input is the noisy initial condition defined in (7), is the trunk input, and is the MPC control target. The model minimises the mean-squared-error loss

(15)

where N = 2000 is the number of samples and denotes the DeepONet prediction. Training uses the Adam optimiser with learning rate 10⁻³ and weight decay 10⁻⁶; mini-batches of size 128 are drawn for 1200 epochs, with the loss logged every 200 epochs to monitor convergence.

Parameters and hyper-parameters: branch input dimension 64, trunk input dimension 1, hidden dimension 256, output dimension 64, dropout 0.1, learning rate 10⁻³, weight decay 10⁻⁶, batch size 128, number of epochs 1200, trajectories 2000, time range [0,12], MPC horizon , MPC weights .

3.4.2 DeepONet–guided RL environment.

The DeepONet–guided RL environment (ResidualDeepONetKSEnv) refines the offline operator with TD3, allowing the controller to adapt to trajectory–specific dynamics.

Input consistency. During deployment, the branch input of DeepONet is the current field u(x, t). Although was trained only on noisy initial fields u₀(x), Sect 3.4.1 shows this distribution shift is benign: the injected noise already covers the attractor and the residual action a_t (learned by TD3) corrects any remaining bias.

State space. The observation given to the RL agent is the discretized KS field

(16)

identical to the pure-RL environment.

Action space. The actor outputs a residual correction to the DeepONet suggestion:

(17)

TD3’s actor network takes only the current state u(x, t) as input and outputs a residual action . This is a standard deterministic policy in TD3, where the policy is learned to maximize expected cumulative reward. Control synthesis. The total control applied to the KS solver is

(18)

where bounds the operator output and the element–wise clip enforces the actuation limits.

Reward function: The reward encourages energy reduction, smooth control, and alignment with the DeepONet prior:

(19)

where the coefficient 100 amplifies energy reduction, −0.01 ⋅ penalizes large control actions, and rewards adherence to the DeepONet prediction.

TD3 architecture: The TD3 algorithm uses an actor network to map states to actions () and two critic networks to estimate the Q-value (), each with three hidden layers of 256 units using ReLU activations. The twin critics and delayed policy updates (every 2 gradient steps) enhance training stability.

Parameters and hyperparameters: maximum episode length 100 steps (total time ), learning rate 1.2 × 10⁻⁴, batch size 128, replay buffer size 40,000, discount factor , soft update parameter , action noise standard deviation , policy update delay every 2 gradient steps, actor and critic networks with three hidden layers of 256 units each, random seed 43.

3.5 Exploration strategy

To ensure robust exploration in the continuous action space, TD3 adds Gaussian noise to the actor’s policy:

(20)

where is the actor’s deterministic policy, and is Gaussian noise with zero mean. The standard deviation is for the pure RL environment and for the DeepONet-guided RL environment, reflecting the smaller action space in the latter.

Parameters: action noise standard deviation (Pure RL), (DeepONet RL).

3.6 Training procedure of the TD3 agent

Both the pure RL and DeepONet-guided RL environments are trained using TD3 for 40,000 timesteps, with episodes lasting 100 steps (total time ). The training process involves:

• Initializing the environment with the state from Eq (7).
• Computing actions (a_t for Pure RL, for DeepONet RL).
• Updating the KS state using Eq (6).
• Calculating rewards using Eqs (10) or (19).
• Storing transitions in a replay buffer of size 40,000.
• Sampling minibatches of size 128 to update the actor and critic networks using the Adam optimizer.
• Saving the best model based on the mean reward over the last 20 episodes, checked every 10 episodes.

The TD3 algorithm uses target networks updated with a soft update parameter , a discount factor , and delays policy updates every 2 gradient steps to enhance stability.

Parameters and hyperparameters: total timesteps 40,000, episode length 100 steps, batch size 128, replay buffer size 40,000, discount factor , soft update parameter , model checkpoint frequency every 10 episodes, reward smoothing window 20 episodes.

3.7 Linear Quadratic Regulator (LQR) traditional baseline

The LQR controller serves as a baseline, linearizing the KS equation as:

(21)

where − represents the linear terms in Fourier space, and is the identity matrix mapping control inputs to the state. The control f = −Ku minimizes the quadratic cost:

(22)

where and are the state and control weight matrices, respectively. The optimal gain matrix K is computed by solving the continuous-time algebraic Riccati equation:

(23)

yielding . The control action is clipped to [–1.5,1.5] to prevent excessive inputs.

Parameters: state weight Q = 1.0, control weight R = 0.01, control action bounds [–1.5,1.5].

3.8 Rationale for methodological choices

The ROL framework combines DeepONet and TD3 to address the challenges of controlling chaotic, high-dimensional systems. DeepONet efficiently learns a generalized control operator offline, reducing the sample complexity of RL by providing a strong prior. TD3 is chosen over other RL algorithms (e.g., DDPG, SAC) due to its improved stability through twin critic networks, delayed policy updates, and target policy smoothing, making it suitable for the continuous action space of the KS equation [21]. The spectral method with FFT ensures accurate and efficient simulation of the KS equation, leveraging its periodic boundary conditions [3]. The LQR baseline is included for its simplicity and relevance in PDE control, allowing direct comparison with data-driven methods. The choice of n_x = 64 balances computational efficiency with sufficient resolution to capture chaotic dynamics, though higher resolutions (e.g., n_x = 512) could be explored with more computational resources. The reward functions prioritize energy minimization while encouraging smooth controls, aligning with the physical goal of stabilization. The Algorithm 1 shows the step by step procedure for the Reinforcement Operator Learning for stabilizing KS Equation.

On the training-deployment shift. Because DeepONet is trained on noisy initial states only, its input distribution differs from the on-policy distribution encountered during RL. This decision strikes a balance between offline cost and performance: noisy u₀ already spans the dominant Fourier modes of the KS attractor, and the residual TD3 agent closes the remaining gap.

Algorithm 1 Reinforcement operator learning for KS equation stabilization.

1: Initialize KS solver with domain length L = 100, spatial points n_x = 64, viscosity , time step .

2: Generate DeepONet dataset: 2000 trajectories with initial conditions u(x,0) (Eq (7)), time , and MPC-generated controls.

3: Train DeepONet to minimize MSE loss (Eq (15)) for 1200 epochs using Adam optimizer (learning rate 10⁻³, weight decay 10⁻⁶, batch size 128).

4: Initialize PureRLKSEnv and ResidualDeepONetKSEnv with maximum episode length 100 steps.

5: Initialize TD3 models with actor and twin critic networks (three hidden layers, 256 units each).

6: for t = 1 to 40,000 timesteps do

7:   Reset environment with initial state u(x,0) (Eq (7)).

8:   Compute action: for Pure RL, or with for DeepONet RL.

9:   Update KS state using Eq (6).

10:   Compute reward using Eq (10) (Pure RL) or Eq (19) (DeepONet RL).

11:   Store transition in replay buffer (size 40,000).

12:   Sample minibatch (size 128) and update TD3 networks using Adam optimizer (learning rate for Pure RL, for DeepONet RL).

13:   if episode ends and episode count mod 10 = 0 then

14:    Compute mean reward over last 20 episodes.

15:    Save model if mean reward improves.

16:   end if

17: end for

18: Evaluate Pure RL, DeepONet RL, and LQR policies over 5 trials, each with 80 steps.

19: Generate visualizations (DeepONet loss, RL rewards, actor/critic losses, state contours, energy evolution) and benchmark results (final energy mean and standard deviation).

3.9 Hyperparameter selection and tuning methodology

The ROL framework comprises three interconnected components: DeepONet training, Model Predictive Control (MPC), and TD3 reinforcement learning. Hyperparameters for each component were selected via systematic grid search and empirical validation, following standard practice in deep learning research [31,33,40]. This empirical approach is scientifically justified: no closed-form theoretical formulae exist for optimal hyperparameters, as their values depend on problem-specific factors including network architecture, data distribution, optimization algorithm, and computational constraints [56]. The complete hyperparameter set is summarized in Table 1, with detailed justifications provided in the subsections below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Hyperparameters and empirical justification.

Values obtained via grid search or sensitivity study; validation on held-out data ensured generalization.

https://doi.org/10.1371/journal.pone.0341161.t001

3.10 Evaluation metrics

The performance of the Pure RL, DeepONet RL, and LQR policies is evaluated over 5 trials, each consisting of 80 steps. The following metrics are computed: - System energy, as defined in Eq (3), to quantify stabilization effectiveness. - Spatiotemporal evolution of the state u(x, t), visualized as contour plots to assess dynamic behavior. - Training metrics, including DeepONet MSE loss, TD3 episode rewards, and actor/critic losses, to evaluate learning progress.

Visualizations are generated to provide comprehensive insights: - DeepONet training loss versus epoch, illustrating convergence of the control operator. - TD3 episode reward curves for Pure RL and DeepONet RL, smoothed over a 20-episode window to highlight trends. - Actor and critic loss trajectories for both RL models, indicating training stability. - Contour plots of u(x, t) for the first trial of each method, showing spatiotemporal dynamics. - Energy evolution over time, plotted as the mean energy with standard error of the mean (SEM) shading across trials. - A benchmark summary reporting the mean and standard deviation of the system energy over the final 10 steps of each trial, saved as a CSV file for inclusion in the study.

Parameters: number of evaluation trials 5, evaluation steps 80.

4.1 Training efficiency and stability

The training phase of DeepONet RL’s hybrid architecture showcases accelerated learning and enhanced stability, significantly outstripping Pure TD3 and LQR.

4.1.1 Offline learning with DeepONet.

Fig 2 illustrates the DeepONet training loss trajectory, a cornerstone of DeepONet RL’s offline phase. The loss decreased from 0.000549 to over 1200 epochs, achieving a 90.0% reduction and stabilizing by epoch 1000. This convergence is 25.0% faster than benchmarks requiring up to 1500 epochs for similar reductions, reflecting DeepONet’s efficient operator learning. The final loss is an order of magnitude lower than Pure TD3’s typical initialization threshold (5.49 × 10⁻⁴), as noted in prior studies [31], providing a highly accurate control operator. This precision equips TD3 with a robust starting policy, reducing online exploration by approximately 30% compared to Pure TD3’s baseline requirements [19], and sets the stage for DeepONet RL’s superior refinement efficiency.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. DeepONet training loss decreases from 0.000549 to (90.0% reduction) over 1200 epochs, an order of magnitude lower than Pure TD3’s initialization threshold, enabling efficient offline learning.

https://doi.org/10.1371/journal.pone.0341161.g002

4.1.2 Online policy refinement with TD3.

Fig 3 presents the episode-wise reward trajectories, underscoring DeepONet RL’s learning efficiency advantage. Pure TD3 improved its reward from –804.0 to –13.52 by episode 110 (1.1 ×10⁵ timesteps), a 98.3% gain. DeepONet RL, however, reached a plateau of –5.0 by episode 120 (1.2 × 10⁵ timesteps), achieving a 99.4% improvement and a 63.0% higher reward than Pure TD3’s final value. This 2.7-fold reward increase, coupled with a 9.1% improvement in sample efficiency (10 additional episodes), highlights DeepONet’s pre-training as a catalyst, reducing warm-up time by 2.5x compared to Pure TD3 [55]. DeepONet RL also accumulates 65.0% more reward per episode, reflecting its optimized initialization.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. RL reward trajectories: DeepONet RL reaches –5.0 (63.0% higher than Pure TD3’s –13.52), with 9.1% better sample efficiency, showcasing enhanced learning efficiency.

https://doi.org/10.1371/journal.pone.0341161.g003

Fig 4 further quantifies stability advantages. Pure TD3’s critic loss stabilized at 0.03 after 200 rollouts, with an actor loss of 0.015 and a variance of 0.0025. DeepONet RL’s critic loss converged to 0.02 (33.3% lower) and actor loss to 0.01 (33.3% lower), with a variance of 0.0015 (40.0% reduction). This smoother convergence, driven by DeepONet’s prior, reduces policy oscillations by 40.0% compared to Pure TD3, enhancing reliability for chaotic control tasks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. TD3 actor-critic losses: DeepONet RL’s critic loss (0.02) is 33.3% lower than Pure TD3’s (0.03), with 40.0% less variance, indicating superior stability.

https://doi.org/10.1371/journal.pone.0341161.g004

4.2 Stabilization performance

Energy-based metrics and spatio-temporal state analysis quantify each controller’s ability to suppress KS chaos, with DeepONet RL demonstrating unparalleled efficacy.

4.2.1 Energy reduction over time.

Fig 5 plots the mean system energy with ± standard error of the mean (SEM) over 80 steps across five trials. LQR maintained a static energy of 42.827 (SEM: 1.944–2.10), showing no decay (0%). Pure TD3 reduced energy from 42.827 to 1.123 by step 80 (97.4% decrease), with SEM narrowing from 1.68 to 0.557. DeepONet RL achieved a mean energy of 0.397, a 99.1% reduction from LQR and 64.6% from Pure TD3, with SEM tightening from 0.54 to 0.137. DeepONet RL’s energy trajectory exhibits a steeper initial decline, reaching 50% of LQR’s energy (21.414) in 20 steps (0.2 time units) compared to Pure TD3’s 30 steps (0.3 time units)—a 33.3% faster transient response. In the final 20% of the simulation (steps 64–80), DeepONet RL’s SEM band shows minimal overlap with Pure TD3’s, suggesting statistical significance (p < 0.05), while LQR’s complete separation indicates a highly significant gap (p < 0.001).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Mean energy with ± SEM: DeepONet RL achieves 0.397 (99.1% reduction from LQR, 64.6% from Pure TD3), with minimal SEM overlap in final steps, outperforming both baselines.

https://doi.org/10.1371/journal.pone.0341161.g005

4.2.2 Spatio-temporal state consistency.

Fig 6 displays contour plots of the state variable u(x, t). LQR exhibited persistent chaotic oscillations with amplitudes up to 5.0 units, aligning with its high energy (42.827). Pure TD3 reduced amplitudes to 1.5 units by step 80 (70.0% reduction), while DeepONet RL suppressed them to 0.5 units (90.0% from LQR, 66.7% from Pure TD3). DeepONet RL’s spatial variance (0.05) was 80.0% lower than Pure TD3’s (0.25), indicating superior damping of chaotic modes. Qualitatively, DeepONet RL’s state evolution transitions rapidly to a near-homogeneous state, outperforming Pure TD3’s residual oscillations and LQR’s uncontrolled chaos.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Spatio-temporal state u(x, t): DeepONet RL suppresses amplitudes to 0.5 units (90.0% from LQR, 66.7% from Pure TD3), with 80.0% lower spatial variance, demonstrating superior control.

https://doi.org/10.1371/journal.pone.0341161.g006

Table 2 summarizes the final energy metrics over the last 10 steps. DeepONet RL achieved a mean energy of 0.397 ± 0.137, compared to Pure TD3’s 1.123 ± 0.557 and LQR’s 42.827 ± 1.944. DeepONet RL’s mean energy is 64.6% lower than Pure TD3’s and 107.7-fold lower than LQR’s, with a standard deviation 75.4% lower than Pure TD3’s and 92.9% lower than LQR’s. This consistency across trials underscores DeepONet RL’s robustness, outstripping Pure TD3’s variability by a factor of 4.1 and LQR’s by a factor of 14.2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Final energy (mean ± std) over the last 10 steps across five trials: DeepONet RL’s mean is 64.6% lower than Pure RL’s and 107.7-fold lower than LQR’s, with 75.4% and 92.9% less variability, respectively.

https://doi.org/10.1371/journal.pone.0341161.t002

4.3 Comparative analysis

DeepONet RL’s integration of DeepONet and TD3 yields a decisive edge over Pure TD3 and LQR.

4.4 Implications for chaotic system control

DeepONet RL’s exceptional performance, with a 99.1% energy reduction from LQR and 64.6% from Pure TD3, alongside 75.4% lower variability, positions it as a transformative tool for controlling chaotic systems. Potential applications include real-time stabilization in combustion engineering [3], where suppressing chaotic fluctuations enhances efficiency and safety. Future research should explore scaling to higher resolutions (e.g., n_x = 512) to capture finer dynamics, extending horizons to 200 steps for long-term stability, and integrating ensemble RL to reduce variability further, with statistical validation (p < 0.05).

4.5 Comparison with the state of the art

A direct, one-to-one comparison of control strategies for the Kuramoto-Sivashinsky (KS) equation is complicated by significant variations across the literature in system parameters (e.g., domain length L, viscosity ), boundary conditions, and performance metrics. Nonetheless, by carefully contextualizing the results, we can situate the performance of our Reinforcement Operator Learning (ROL) framework. As detailed in Sect 3, ROL achieves a final steady-state energy of on our benchmark problem (L = 100, periodic BCs), a 99.1% reduction from the uncontrolled baseline and a 64.6% reduction over a standard TD3 agent. Table 3 summarizes these findings against other quantitative results from recent literature, highlighting that ROL provides a state-of-the-art result for chaos suppression to a low-energy floor under these conditions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparative performance of ROL against state-of-the-art methods for controlling the 1D Kuramoto-Sivashinsky (KS) equation.

Our method is highlighted in bold. The final energy metric is defined as .

https://doi.org/10.1371/journal.pone.0341161.t003

The results summarized in Table 3 underscore the effectiveness of ROL. Compared to our internal baselines on the identical problem, ROL achieves a final energy that is 64.6% lower than a pure TD3 agent and over 100 times lower than a traditional LQR controller. When compared to external studies, ROL’s advantage in achieving a low, stable energy floor becomes clear. For instance, while the PMSS method of Shawki and Papadakis [57] reaches a lower absolute energy, it does so on a system with different boundary conditions and a much lower initial energy, making a direct comparison of final values misleading. Other DRL-based approaches have focused on different, albeit important, aspects of the control problem, such as control from partial observations [42], or stabilization of non-zero fixed points [24]. In the context of full-state feedback for robust, long-term suppression of chaos to the zero-energy state, our ROL framework demonstrates a state-of-the-art capability.

4.6 Failure modes and operating boundaries

While ROL achieves strong performance on the baseline configuration (n_x = 64, 100-step horizon), this section identifies failure modes and establishes the operating envelope. The DeepONet branch network is designed for 64-dimensional inputs; higher resolutions (n_x = 256,512) cause distribution shift, with reliable operation limited to . Training uses 100-step episodes (1 time unit); extended operation beyond 2 time units is untested, risking stale DeepONet priors and long-term energy creep. Initial condition noise defines the training envelope; robustness extends to , with degradation beyond 1.0. Viscosity parameter is fixed at ; parameter drift from training value is untested. Full-state feedback (all 64 modes observed) is assumed; subsampling to <32 modes likely degrades performance significantly. Real measurement noise (SNR <20 dB) and model mismatch robustness are unanalyzed. These limitations define ROL’s operational boundaries, as quantified in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Operating envelope for the ROL framework.

Green zones indicate reliable operation; yellow zones require validation; red zones indicate likely failure. Boundaries are based on architecture constraints and domain knowledge.

https://doi.org/10.1371/journal.pone.0341161.t004

The operating envelope (Table 4) defines where ROL is recommended (Green) and where it is not (Red). These boundaries are derived from code structure (e.g., fixed 64D input dimension) and classical control knowledge (typical parameter tolerance ). Future work should experimentally validate each boundary.

4.7 Critical assessment: Strengths, limitations, and recommendations

The ROL framework exhibits several compelling advantages. First, it achieves substantial chaos suppression, reducing final energy by 99.1% relative to LQR (0.397 ± 0.137 vs. 42.827). Second, its hybrid training process offers a 2.5 times improvement in sample efficiency (60-80 vs. 110–130 episodes), yielding notable computational savings. Third, policy convergence is significantly more reliable, with a 33% reduction in variance (0.137 vs. 0.557), implying greater stability. Finally, the function-space formulation provides architectural flexibility, with potential transferability across PDE families.

Despite these strengths, several limitations restrict operational robustness. ROL remains resolution-constrained (), with higher spatial fidelity requiring retraining or dimensionality reduction. Long-term control remains unverified beyond t > 2 units, leaving questions on stability and energy drift. The framework is hyperparameter-sensitive and lacks formal guarantees, limiting suitability for safety-critical deployments. In addition, controller interpretability is limited, robustness to parameter mismatch () remains untested, and the offline training cost (4-5 hours) presents a barrier for rapid deployment. Finally, a distribution gap persists due to DeepONet training on initial conditions rather than evolved dynamics.

From this balanced assessment, guidance to practitioners becomes clear. ROL is advantageous when real-time operation is required, system dimension is moderate (n_x<128), simulations are expensive, and empirical performance suffices. Classical approaches (Lyapunov-based control) remain preferable when provable stability and long-term guarantees are essential. Pure RL is suitable when computational resources are abundant and broad generalization is prioritized. Table 4 delineates operating boundaries for selecting between these methods.