https://orcid.org/0009-0004-6012-4846
Figures
Abstract
Building on recent insights that augmenting reinforcement‐learning policies with disturbance estimates improves robustness and sim-to-real transfer, this paper proposes a disturbance-aware actor–critic RL framework for high‐precision robotic manipulators. We derive the dynamics of manipulators ranging from two to six degrees of freedom and design a nonlinear disturbance observer that provides real-time estimates of lumped uncertainties. Unlike conventional feedforward-only designs, the proposed DACRL framework deeply integrates DOB information into the actor-critic’s learning state, cost function, and update laws to explicitly compensate for unknown dynamics and disturbances. A Lyapunov-based analysis proves that tracking errors, observer errors and neural-network weight errors remain uniformly ultimately bounded. Extensive simulations show that the proposed controller achieves sub-degree tracking errors across multiple DOF cases, with the 2-DOF example achieving steady-state errors below ±0.02 rad (Joint 1) and ±0.05 rad (Joint 2) and delivering faster convergence and smoother torques than conventional RL and DOB baselines. The disturbance-aware controller generalizes across trajectories and payloads, offering improved robustness while retaining learning flexibility. While the current study focuses on performance and robustness, future work could explore the integration of disturbance-observer-based control barrier functions to formally address safety constraints during the learning process. Simulation results suggest that the disturbance-aware controller can improve robustness while retaining learning flexibility. Real-platform validation remains an important direction for future work.
Citation: Nguyen VN, Le TMT, Pham DH, Nguyen HC (2026) Robust disturbance-aware actor–critic reinforcement learning for multi-DOF robotic manipulators. PLoS One 21(10): e0354740. https://doi.org/10.1371/journal.pone.0354740
Editor: Longhui Qin, Southeast University, CHINA
Received: February 3, 2026; Accepted: July 11, 2026; Published: October 6, 2026
Copyright: © 2026 Nguyen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data needed to reproduce the simulation results are included within the within the paper and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: Conflict of Interest Statement: The authors have no conflicts of interest to disclose.
1. Introduction
Reinforcement learning (RL) has been increasingly adopted in robotics as a data-driven paradigm for sequential decision-making and control in complex nonlinear systems [1]. Broad overviews of RL algorithms and methodological families further consolidate this trend and clarify common design choices [2]. Recent monographs also provide rigorous foundations for sequential decision-making and optimal control perspectives that are frequently used in continuous-time robotic control formulations [3].
Within this context, model-based RL has attracted growing attention due to its potential for improved sample efficiency and stronger links to control-theoretic structure [4]. In addition, unified viewpoints connecting RL with dynamic programming and model predictive control help motivate control-oriented RL formulations and performance indices for real-time control [5]. Meanwhile, continuing progress in algorithmic design and discovery has expanded the practical deployment space of RL in complex systems [6], and complementary advances in scalable learning architectures and training heuristics further broaden applicability in robotics settings [7].
Despite these developments, achieving high-precision trajectory tracking for robotic manipulators using RL remains challenging in the presence of strong nonlinear coupling, parameter variations, friction, payload changes, and time-varying disturbances. Safety and constraint satisfaction motivate safe-RL mechanisms such as control barrier functions (CBFs) for enforcing state/input constraints [8]. Learning-based CBF formulations and DOB-assisted CBF designs further provide practical routes for integrating learning with safety guarantees under uncertainty [9].
In parallel, disturbance observers (DOBs) provide a computationally efficient means to estimate lumped uncertainties and reject matched disturbances in nonlinear systems [10]. DOB-based robust tracking control has been widely studied for manipulators under modeling errors and external disturbances [11]. Additional DOB/observer-enhanced robust control designs demonstrate improved tracking performance and robustness under practical disturbance conditions [12].
These considerations have motivated observer-integrated adaptive optimal control and reinforcement learning frameworks for uncertain nonlinear systems [13]. DOB-based actor–critic learning is one representative direction that combines online policy improvement with disturbance compensation [14]. Other DOB-enhanced ADP/RL schemes emphasize robustness and practical stability properties while learning in continuous time [15]. Disturbance-aware objective shaping has also been explored to better reflect uncertainty in the performance index used for learning and optimization [16]. Further, stability-oriented analyses are commonly developed to establish boundedness/UUB-type guarantees under approximation and observer assumptions [17]. Observer-integrated adaptive optimal control for robotic manipulators further extends these ideas toward real robotic platforms under disturbances and uncertainties [18].
However, in several DOB–RL implementations, the DOB is primarily designed and tuned independently, and its estimate is used mainly for feedforward compensation, while the learning objective and policy updates are only indirectly influenced by observer mismatch. Under time-varying disturbances, this separation may require the learning component to accommodate disturbance-induced transients and fast components, which can reduce learning efficiency and adversely affect control smoothness. These observations motivate a co-designed disturbance-aware actor–critic framework in which DOB information is embedded into the learning loop at both the state and objective levels, while preserving a control-theoretic structure suitable for real-time implementation.
The main contributions of this paper are summarized as follows.
- 1. Disturbance-aware objective shaping: A disturbance-aware performance index is formulated by explicitly incorporating the measurable DOB filter residual as a penalty term into the RL cost function. Unlike prior works [13,16–18] that construct the learning objective solely from tracking error and control effort — leaving the learning loop without a direct signal when the DOB estimate is inaccurate the proposed cost explicitly penalizes estimation mismatch, incentivizing the policy to operate in regions where the observer performs well.
- 2. Co-designed DOB–Actor–Critic learning rule: The actor update law is augmented with a disturbance-aware gradient term derived from the measurable residual, and the learning state includes both the disturbance estimate and the filter residual. This addresses two gaps absent from [13,14,17,18]: (i) cascade designs provide no gradient signal preventing the policy from selecting torques that amplify the DOB residual; and (ii) augmenting the state with the disturbance estimate alone cannot distinguish a well-converged observer from a poorly converged one, whereas including the filter residual allows the critic to condition value estimation on current observer quality.
- 3. Composite control architecture for manipulators: A composite torque law integrates a nominal/robust baseline, DOB compensation, and an actor corrective term, consistent with [10–12,19,20]. Unlike cascade architectures where the DOB and AC blocks operate independently, the proposed structure couples the DOB output into the learning loop at both the state level and the objective level, ensuring the actor corrective term is learned with explicit awareness of current observer performance rather than treating the DOB-compensated plant as a fixed, uncertainty-free system.
- 4. Stability-oriented analysis: A Lyapunov-based analysis is provided to establish boundedness (UUB-type) of closed-loop signals under standard observer and function-approximation assumptions, aligning online learning with stability requirements in continuous-time control [13–18].
- 5. Progressive evaluation and scalability: The method is evaluated on a 2-DOF manipulator and further assessed on a 6-DOF industrial manipulator under composite disturbances, with comparisons to DOB-only, standard actor–critic, and DOB-based ADP/RL baselines [13–18]. Fault-tolerant RL control of manipulators under actuator faults further motivates evaluation under non-ideal actuator conditions [21]. Modern dynamics and simulation/control references are also included to support modeling and implementation practices [22–24].
2. Related work
This section provides a brief review of the foundational research informing the present study. While RL for robotic control has been extensively studied for optimal control [1]– [6], achieving robust, real-time deployment under modeling errors and disturbances remains a persistent challenge [7]. To ensure reliability, safety and constraint enforcement through Control Barrier Functions (CBFs) has emerged as a key area of interest, particularly when integrated with RL and disturbance-aware filtering [8,9]. Complementary to this, DOB for robust tracking offers an efficient means of estimating lumped uncertainties in nonlinear manipulators, though its efficacy is often limited by bandwidth-noise trade-offs [10]– [12].
The key technical distinction of the proposed framework is threefold. First, prior DOB-RL designs ([13,16–18]) formulate the RL objective independently of DOB estimation quality: the cost function penalizes tracking error and control effort but not disturbance mismatch. In DACRL, the measurable filter residual appears directly in the cost via a
penalty, creating an explicit learning incentive to reduce estimation mismatch. Second, the actor update in DACRL includes a gradient term
, which discourages policies that amplify the DOB residual a mechanism absent from all cascade designs reviewed. Third, both
and
are included in the learning state, allowing the critic to condition value estimation on current observer quality rather than just the estimated disturbance magnitude.
Table 1 summarizes representative lines of work and highlights that this paper explicitly couples DOB mismatch into the learning objective and actor update, and evaluates the method on both 2-DOF and 6-DOF manipulators under modern control metrics.
3. Formulation and analysis
3.1. Dynamics, tracking objective, and lumped disturbance model
Consider an (n)-DOF rigid manipulator described by the Euler–Lagrange dynamics:
where is the joint position,
is the commanded torque, and d(t) denotes a matched lumped disturbance aggregating friction, external perturbations, and modeling errors. Such modeling is standard in multibody and robot dynamics and industrial robot simulation studies [22].
Given a desired trajectory define the tracking errors:
and the filtered and sliding tracking variable:
Assumption A1. The lumped disturbance d(t) is bounded and piecewise differentiable: as commonly assumed in disturbance-observer-based robust control [10].
3.2. Measurable residual and disturbance observer smoothing
Because d(t) is not measurable, we define a measurable residual computed from the nominal model and measured signals:
Residual and DOB constructions are widely used for robot disturbance estimation and robust tracking [11]. In practice, the residual contains modeling and sensing errors.
where is a bounded lumped error term,
.
To obtain a smooth disturbance estimate, we employ a first-order DOB filter:
Define the (unmeasurable) true estimation error and the measurable filter residual:
Combining (5) and (7) yields:
and hence the two-sided bound:
Since d(t) is not measurable, directly minimizing is not implementable online. A principled measurable surrogate is instead:
because (9) shows that reducing
reduces
up to the bounded modeling/sensing term
. Penalizing
is weaker because it mixes disturbance magnitude with
; therefore, this work uses
as the disturbance-aware mismatch penalty, consistent with DOB-based RL co-design motivations in the recent literature [13].
3.3. Disturbance-aware RL objective, learning state, and analysis-compatible parameterization
We form a disturbance-aware learning state:
Define the instantaneous utility:
which combines tracking accuracy, control effort, and the measurable disturbance mismatch penalty.
The penalty in
constitutes the primary departure from standard actor-critic objectives. Whereas conventional RL objectives for manipulators penalize tracking error and control effort [1,2,14], the inclusion of the measurable residual energy creates a direct coupling between observer performance and the learning signal. This incentivizes the policy to operate in regions where the DOB provides accurate estimates, rather than relying solely on the nominal+robust baseline to absorb remaining estimation error.
In discrete time, the discounted value function is:
This objective is consistent with optimal-control and RL formulations grounded in dynamic programming and modern RL surveys [3].
To ensure strict consistency between stability-oriented analysis and simulation and implementation, both actor and critic use linear-in-weights approximators
Actor (policy):
Critic (value):
where is a bounded feature and basis vector and
is an RBF feature vector standard in disturbance-aware actor–critic and robust RL control for nonlinear systems [13].
where is component-wise saturation and normalization and
are the RBF centers and widths.
Assumption A2. On a compact set there exist ideal weights
such that:
This assumption and the bounded-weight mechanism below are consistent with robust actor–critic learning under matched disturbances [14].
3.4. Unified TD error and implementable update laws
Define the temporal-difference (TD) error:
To guarantee bounded weights and align with UUB-style reasoning, use gradient descent with projection onto compact sets .
Critic update:
Actor update:
where denotes a standard actor–critic policy-improvement surrogate used in robust RL control and the disturbance-aware term is defined through the measurable mismatch:
The disturbance-aware actor gradient term represents a second departure from standard policy improvement. In conventional actor-critic updates, the policy is updated to reduce the TD error, which reflects cumulative discounted cost. The additional gradient
provides a direct regularization signal that penalizes policies generating large DOB residuals, independent of the TD error. This discourages the actor from selecting torques that, while effective in reducing tracking error, destabilize DOB estimation or amplify unmodeled dynamics.
In the proposed DACRL framework, the DOB has a dual role. In the control loop, the disturbance estimate is used directly in the composite torque law to compensate for matched lumped disturbances. In the learning loop,
and the measurable DOB mismatch and residual are incorporated into the disturbance-aware state and objective function, so that the actor–critic updates are influenced by disturbance-estimation quality. Beyond mere compensation, the DOB’s disturbance estimates directly shape both control torque and policy adaptation. For clarity and reproducibility, the online implementation of the proposed DACRL controller is summarized in Algorithm 1. At each sampling instant, the controller first computes the tracking errors and the measurable disturbance residual. The DOB’s disturbance estimate provides feedforward compensation and builds an augmented state, driving the actor’s control output and the critic’s evaluation. The composite torque is applied to the manipulator, and the actor–critic weights are updated using the TD error with projection to maintain bounded adaptation.
Algorithm 1.
Inputs: , parameters
and bounds
.
At each sampling instant k:
1. Compute via (2), and
via (3).
2. Compute residual via (4).
3. Update DOB estimate (Euler discretization of (6)
compute
via (7).
4. Form learning state via (10).
5. Compute features and
via (15).
6. Evaluate critic: via (14).
7. Synthesize composite control torque:
from (13).
8. Compute utility via (11).
9. After observing compute TD error
via (17).
10. Update and
via (18) – (20) with projection and clipping.
Algorithm 1 highlights that the proposed method is implemented fully online. The DOB, actor, critic, and composite torque law are updated at every sampling instant using measurable signals. The projection operation in the actor–critic updates is included to prevent parameter drift and to maintain consistency with the boundedness assumptions used in the stability analysis.
4. Controller design
This section provides a reproducible design recipe for the proposed DACRL controller. It presents the manipulator model and assumptions, the DOB and DACRL components, and the final composite torque law for real-time implementation on 2-DOF and higher-DOF platforms. The 2-DOF manipulator is specified in Table 2 and illustrated in Fig 1.
Consider an (n)-DOF rigid-link manipulator with joint coordinates , joint velocity
and input torque
. The dynamics can be written in Euler–Lagrange form as:
where is symmetric positive definite,
collects Coriolis/centrifugal terms,
is the gravity vector,
is the input-channel mapping, and
denotes lumped matched disturbances.
Assumption 1. is nonsingular for all q in the operating region, which ensures disturbances entering through the same channel as control can be compensated. For standard torque-controlled manipulators,
.
The lumped disturbance aggregates multiple non-idealities:
where denotes external disturbances,
includes viscous and Coulomb friction, and
denotes unmodeled dynamics and parametric mismatch. In the proposed design, all terms in (22) are treated as a single matched disturbance entering the input channel of (21).
Given a desired trajectory define tracking errors:
and the sliding-like variable used for baseline robustness:
where
.
A residual signal that is measurable from model terms and sensor signals is defined as
If the model were exact and noise-free, then . In practice, r also contains modeling error and measurement noise; therefore, we obtain a smoothed disturbance estimate via a first-order low-pass filter:
applied component-wise to each joint. The estimate is used (i) for disturbance compensation in the torque input, and (ii) as an additional disturbance context signal for learning.
Implementation note (acceleration availability). If is not directly measured, it can be obtained from a filtered differentiator
with
chosen to balance noise amplification. In that case,
should be tuned conservatively to avoid injecting differentiation noise into
. Larger
yields faster disturbance tracking but higher sensitivity to noise; smaller
yields smoother
and torque but slower rejection.
To embed DOB information into the learning loop, we define the disturbance-aware state:
This augmentation enables the policy and value function to explicitly condition on disturbance estimates and tracking “robustness” signals (s), supporting co-design rather than cascade compensation.
To maintain consistency with Lyapunov-style boundedness analysis, we employ linear-in-weights function approximation.
Actor (policy) approximation.
where is the actor weight matrix, and
is a chosen basis vector as low-order polynomials and RBFs of
. This form is linear in
which is standard in stability-oriented adaptive and actor–critic control. Critic approximation:
where and
is typically an RBF basis. Define an instantaneous utility that reflects tracking and control effort objectives:
Using a discrete-time implementation with sampling is defined the TD error:
Critic update gradient descent with projection:
where is the critic learning rate and
denotes projection onto a compact set
to prevent parameter drift.
Actor update (disturbance-aware co-design). We design the actor update to include (i) a standard policy-improvement term, and (ii) a measurable coupling penalty driven by DOB mismatch. A practical measurable surrogate is the squared residual energy , since
is measurable from (24). The actor update is:
where is the actor learning rate and
weights the coupling.
A concrete choice consistent with measurability is: Use giving; For the coupling term, from (24) we have
. Since
contains the actor term,
can be computed as:
, which is measurable because
and
are known online. For linear actor (27),
is directly related to
.
The final control input is a composite torque:
where is the actor term.
Nominal (computed-torque-like) term.
This term provides baseline model-based tracking.
Robust term (smooth sliding saturation).
where is applied element-wise and
controls smoothness (larger
→ smoother torque, smaller
→ stronger sliding action but may induce chatter).
Remarks. ensures a stabilizing baseline and torque smoothness.
offloads fast disturbance rejection to the DOB.
improves performance online under the disturbance-aware state x, while the coupling in (32) discourages policies that increase residual mismatch.
The design in (21) – (35) is directly applicable to (n)-DOF manipulators by using vector, matrix forms with . The primary scaling considerations are: (i) basis size
for
,
, (ii) DOB filter gain
per joint or joint-dependent
, and (iii) robust.
The engineering motivation for the disturbance-aware coupling term in the actor update is to prevent the policy from selecting torque commands that, while reducing tracking error in the short term, amplify uncompensated disturbance components and degrade observer convergence. A standard actor update driven solely by the TD error carries no such signal. By explicitly penalizing the DOB filter residual in the policy gradient, the actor is guided to operate the plant in regimes where the disturbance observer remains effective, thereby preserving the disturbance rejection capability that the composite control architecture relies upon.
5. Simulation scenarios and results
5.1. Simulation setup
All simulations were implemented in Python 3.10 using PyTorch 2.2 and SciPy 1.13 for numerical integration. The DACRL controller is implemented online with a fixed sampling time of Ts = 1 ms. The plant dynamics are integrated using a fourth-order Runge–Kutta method, while the actor, critic, and DOB are updated at every sampling instant. The actor adopts a linear-in-weights policy using normalized and clipped handcrafted polynomial/RBF features of the disturbance-aware state. The critic uses an RBF-based linear-in-weights value function. RBF centers are selected by k-means clustering from a short warm-up rollout, and RBF widths are chosen from the mean distance to neighboring centers. The actor and critic weights are updated by projected gradient descent, with projection bounds used to avoid parameter drift. Convergence is evaluated empirically through bounded actor/critic losses, bounded weights, reduced TD error, bounded DOB estimation error, post-transient tracking RMSE, peak tracking error, and torque-energy behavior. To maintain consistency with the Lyapunov/UUB-oriented analysis, both learning components follow the linear-in-weights approximation used in (27)–(32). This design prioritizes analytical tractability, bounded online adaptation, and real-time feasibility. Nevertheless, it may sacrifice approximation capacity compared with deep neural networks, and the impact of higher-capacity approximators will be investigated in future work. Specifically, the actor is parameterized as where
is a handcrafted basis vector. The critic is parameterized as
where
is an RBF feature vector. Online updates use per-step gradient descent with projection (31)–(32) to keep weights bounded and avoid parameter drift, rather than deep-network backpropagation. RBF centers were obtained by k-means clustering over state samples collected from a short warm-up rollout; widths were set to the mean distance to the k-nearest centers. The nominal parameters in Tables 3 and 4 were selected to obtain stable online learning and smooth torque responses; however, the influence of these parameters on tracking accuracy, torque energy, and disturbance-estimation performance should be further quantified through a dedicated sensitivity study. Controller parameters were fixed as in Table 3. In all experiments, features were normalized and clipped to predefined ranges to satisfy boundedness assumptions in the analysis. The actor basis size was set to
and the critic RBF size was set to
as reported in Table 4. Projection was implemented via component-wise clipping
.
For fair benchmarking and to isolate the contribution of the proposed co-design, all learning components in AC, OADP+DOB, and DACRL employed the same feature sizes and learning hyperparameters reported in Table 4.
Statistical robustness. Current results show DACRL’s effectiveness under fixed settings. To better quantify robustness and repeatability, future work will conduct multiple independent runs with varying conditions and report comprehensive statistical metrics.
5.2. Simulation results for scenario 1 with 2-DOF manipulator model
The proposed controller was tested on a realistic 2-DOF manipulator (Table 1) under complex time-varying disturbances composed of sinusoidal terms, periodic step changes, and amplitude variations to emulate payload shifts and persistent environmental interactions. The desired trajectory for both joints was set to a smooth, exponentially converging sinusoidal signal to avoid unrealistic initial jerks: with A=[0.6, 0.8]T rad/s and ω = π rad/s.
(Joint 1)
(Joint 2)
The trajectory tracking performance of the proposed DACRL controller is depicted in Fig 2. It is observed that both joint angles (q1, q2) accurately track their respective reference trajectories (q1d, q2d) after a brief initial transient phase.
The left panel shows the arm configuration alongside the actual and reference end-effector trajectories, while the right panel provides a zoomed Cartesian view. Qualitatively, the actual trajectory closely follows the reference path with no visible steady-state offset, drift, or high-frequency chattering, confirming effective disturbance rejection by the composite controller. Quantitatively, the close Cartesian agreement is consistent with the joint-space steady-state errors of ±0.02 rad (Joint 1) and ±0.05 rad (Joint 2) reported in Fig 4, and the smooth trajectory curvature confirms that the DOB successfully suppresses the injected time-varying disturbances without inducing oscillations. From an engineering perspective, the results demonstrate that joint-level disturbance rejection translates directly into task-space tracking precision, which is the practically relevant performance criterion for manipulation applications.
Fig 3 demonstrates exceptional tracking performance, with both joints converging to their references within 0.5s. The small overshoot <5% in Joint 2 indicates well-tuned control gains.
With the DOB + AC configuration exhibiting a noticeably smaller initial transient than the Actor-Crisis-Only baseline, and both methods achieving near-identical steady-state tracking thereafter, confirming that the DOB contribution is most significant during the transient phase.
The corresponding tracking errors are presented in Fig 4. The steady-state tracking errors are confined within ±0.02 rad for Joint 1 and ±0.05 rad for Joint 2, which represents a high degree of tracking precision. The controller successfully suppresses the effects of significant nonlinearities and the complex, injected external disturbances, confirming the theoretical UUB stability result.
The control torques, shown in Fig 5, remain within feasible limits of approximately ±50 Nm for Joint 1 and ±15 Nm for Joint 2. The control signals are smooth and do not exhibit high-frequency chattering, which is a critical advantage for practical implementation and actuator longevity.
The DOB + AC controller produces smooth, continuously varying torque profiles within ±50 Nm (Joint 1) and ±30 Nm (Joint 2) throughout the motion, whereas the AC Only configuration generates high-frequency torque spikes during the initial transient before collapsing to near-zero output, indicating that without DOB compensation the actor alone is insufficient to sustain adequate disturbance rejection over the full operating period.
5.3. Simulation results for scenario 2 with 6-DOF manipulator model
To further validate scalability and robustness, we conducted simulations on the Mitsubishi RV-2FBD 6-DOF industrial manipulator, demonstrating that the proposed controller can handle strongly coupled, high-dimensional nonlinear dynamics with uncertainties. Fig 6 shows that the end-effector trajectory closely tracks the reference in the planar workspace as well as in the x(t) and y(t) components, with a small position error after a brief transient. Fig 7,Fig 8 confirm accurate joint-space tracking: all six joint trajectories follow their references and the tracking errors rapidly converge to a small neighborhood around zero and remain uniformly bounded, supporting the theoretical stability claims. Fig 9 indicates smooth torque profiles within practical limits, and Fig 10 presents stable online training curves, implying effective policy and value-function adaptation. Overall, these results verify that the proposed framework scales effectively to a 6-DOF industrial manipulator and is promising for real-time deployment and extension to other multi-DOF robots.
The simulation results for the 6-DOF case collectively demonstrate the scalability and robustness of the proposed DACRL framework under strongly coupled, high-dimensional nonlinear dynamics. In the Cartesian space (Fig 6), the end-effector trajectory closely follows the reference path in both X and Y components, with the position error rapidly decreasing from a peak of approximately 0.12 m at the initial transient to a bounded steady-state band of 0.008–0.02 m after t ≈ 0.5 s, confirming effective task-space regulation despite the increased kinematic complexity. At the joint level (Fig 7), all six joints accurately track their sinusoidal references throughout the 15 s simulation, with Joint 6 exhibiting the largest visible deviation due to its distal position and higher sensitivity to cumulative kinematic coupling. Correspondingly, Fig 8 shows that all joint tracking errors converge within approximately 0.2–0.5 s to a uniformly bounded steady-state band within ±0.02 rad for Joints 1–5 and ±0.03 rad for Joint 6, consistent with the UUB stability guarantees established in the Lyapunov analysis. The control torques (Fig 9) remain within physically feasible bounds across all joints peaking at approximately 100 Nm for Joint 1 during the initial transient and settling to smooth steady-state profiles within ±20 Nm with no sustained high-frequency chattering, confirming that the smooth sliding saturation term in the composite torque law effectively prevents actuator excitation. Finally, Fig 10 shows that both actor and critic losses decrease rapidly within the first 50 training steps and remain bounded near zero thereafter, providing empirical evidence of stable online adaptation under the linear-in-weights parameterization, consistent with the bounded weight assumptions of the theoretical analysis.
5.4. Discussion
The online learning process is illustrated in Fig 10. The critic loss decreases rapidly from a peak of approximately 27 within the first 10 training steps and converges to near-zero by step 30, while the actor loss stabilizes near zero by step 20, both remaining stably bounded throughout the remaining 470 steps. This rapid convergence confirms that the disturbance-aware learning state and residual-penalized cost provide sufficiently informative gradient signals for online adaptation within the 1 ms sampling constraint, while the projection-based update mechanism effectively prevents parameter drift under the injected time-varying disturbances. These results should be interpreted as empirical evidence of stable online adaptation under the tested scenario rather than proof of global optimality over the entire state space.
Fig 11 reports the actor and critic losses during online learning. Both losses decrease rapidly and remain bounded, indicating stable numerical learning behaviour under the selected hyperparameters. Loss convergence is interpreted as an empirical signal of learning progress rather than evidence of global optimality.
Fig 12 shows that the DOB estimate quickly tracks the injected time-varying disturbance and maintains a bounded estimation error, confirming effective disturbance compensation and reduced learning burden. The frequency-domain response indicates disturbance attenuation over the experiment’s bandwidth, reflecting the usual speed–noise trade-off. Table 5 provides a quantitative comparison using MSE of joint tracking. The proposed DACRL achieves the lowest MSE for both joints (Joint 1: 1.25 × 10 − ⁴ rad²; Joint 2: 4.12 × 10 − 4 rad2) and improves upon OADP+DOB [13] under the reported settings.
Table 6 details the steady-state tracking errors and control effort for the 6-DOF case after the initial transient phase.
Table 7 further indicates that co-designing the DOB with the learning loop via disturbance-aware state augmentation and objective shaping offers a plausible explanation for improved tracking and smoother control compared with sequential combinations.
The simulations used fixed controller and learning parameters to ensure reproducibility and fair comparison among the baselines. However, DACRL performance may be affected by several key hyperparameters, including the dob gain, actor/critic learning rates, disturbance penalty, robust gain, and projection bounds. These parameters influence disturbance-estimation speed, noise sensitivity, learning convergence, torque smoothness, and tracking accuracy. therefore, a systematic sensitivity analysis is planned for future work. The main parameters, nominal values, suggested test ranges, and expected effects are summarized in Table 8.
To quantify the contribution of each disturbance-aware component, we performed an ablation study comparing the full DACRL (state augmentation + disturbance-aware cost) with three baselines: (i) robust+dob without rl, (ii) standard actor–critic without disturbance awareness, and (iii) actor–critic with disturbance-aware state only (using d ^ as input but not penalizing it in the cost). All variants used identical network architectures, learning rates, and disturbance profiles. as summarized in Table 3 the full dacrl achieved the lowest rms tracking error and the most efficient torque-energy usage, while the disturbance-aware cost term notably reduced transient oscillations versus state-only augmentation, confirming the benefit of the proposed co-design. The current implementation uses a first-order dob, linear-in-weights actor–critic approximators, and projection-based gradient updates to reduce computational complexity and support real-time control. nevertheless, the computational cost increases with the number of dofs, the size of the disturbance-aware state, and the number of actor/critic basis functions. For higher-dof manipulators, execution time per sampling step, memory usage, and real-time feasibility should be explicitly evaluated. Future work will investigate sparse features, reduced-order basis functions, parallel dob updates, and optimized implementation for high-dimensional robotic systems.
Limitations and experimental validation. Although the proposed DACRL framework shows robust tracking performance in both 2-DOF and 6-DOF simulations, the present study is limited to numerical validation. Therefore, the results should be interpreted as a simulation-based proof-of-concept rather than full evidence of real-world deployment readiness. Practical factors such as sensor noise, communication delay, actuator saturation, friction uncertainty, payload variation, and real-time computational constraints may affect closed-loop performance. Future work will validate the proposed controller on a physical manipulator, including payload-change tests, external disturbance rejection tests, trajectory generalization, and comparisons with DOB-only and conventional actor–critic controllers under identical hardware conditions. Safety supervision, including torque limits and possibly control-barrier-function-based constraints, will also be considered before online learning is deployed on hardware.
The present stability analysis assumes bounded and piecewise differentiable matched disturbances. This assumption is standard in many DOB-based robust control designs, but it does not fully cover abrupt impacts, discontinuous disturbances, severe actuator saturation, or actuator faults such as bias, partial loss of effectiveness, or stuck actuators. Although the simulations include sinusoidal disturbances, periodic step changes, and amplitude variations, more aggressive non-smooth disturbances and fault scenarios require additional fault-tolerant and safety mechanisms. Future work will extend the proposed DACRL framework by incorporating fault-detection and isolation, actuator-saturation handling, adaptive fault-tolerant compensation, and safety constraints such as control barrier functions.
6. Conclusion and future work
This paper presents DACRL, a co-designed reinforcement learning framework for high-precision robotic trajectory tracking under nonlinear dynamics and disturbances. By integrating a Nonlinear Disturbance Observer (NDOB) directly into the actor–critic learning loop via state augmentation and objective shaping, the system enables real-time disturbance rejection and explicit reaction to uncertainties. Supported by a Lyapunov-based stability analysis ensuring UUB convergence, the method was validated on 2-DOF and 6-DOF (Mitsubishi RV-2FBD) manipulators. Simulation results demonstrated improved tracking accuracy, faster convergence, and smoother torque profiles compared with standard RL and cascade designs under the tested disturbance profiles. However, experimental validation on a physical manipulator is still required to assess real-time implementation issues, actuator constraints, sensor noise, and sim-to-real transfer performance. The current DACRL framework does not explicitly enforce actuator limits or formal safety constraints. Although the simulated torques remain bounded and relatively smooth, practical deployment requires constraints on joint positions, velocities, torques, and collision avoidance to be incorporated into the controller. Future work will integrate actuator saturation handling, safety filters, and control-barrier-function-based constraints to ensure safe online learning and hardware implementation.
Supporting information
S1 File. Simulation source code for the disturbance-aware actor–critic reinforcement learning framework.
https://github.com/hung83ddt-glitch/PONE-D-26-05825R2-Plosone. The “S1 File.zip” archive contains the Python source code used to perform the 2-DOF and 6-DOF robotic-manipulator simulations.
https://doi.org/10.1371/journal.pone.0354740.s001
(ZIP)
References
- 1. Tang C, Abbatematteo B, Hu J, Chandra R, Martín-Martín R, Stone P. Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes. arXiv preprint arXiv:2408.03539. 2024.
- 2. Shakya AK, Pillai G, Chakrabarty S. Reinforcement learning algorithms: A brief survey. Expert Systems with Applications. 2023;231:120495.
- 3. Li SE. Reinforcement Learning for Sequential Decision and Optimal Control. Singapore: Springer. 2023.
- 4. Luo F-M, Xu T, Lai H, Chen X-H, Zhang W, Yu Y. A survey on model-based reinforcement learning. Sci China Inf Sci. 2024;67(2).
- 5. Bertsekas DP. Model predictive control and reinforcement learning: A unified framework based on dynamic programming. IFAC-PapersOnLine. 2024;58(18):363–83.
- 6. Zhang D, Yuan Q, Meng L, Xia R, Liu W, Qin C. Reinforcement learning for single-agent to multi-agent systems: from basic theory to industrial application progress, a survey. Artif Intell Rev. 2025;59(2).
- 7. Oh J, Farquhar G, Kemaev I, Calian DA, Hessel M, Zintgraf L, et al. Discovering state-of-the-art reinforcement learning algorithms. Nature. 2025;648(8093):312–9. pmid:41125136
- 8. Guerrier M, Fouad H, Beltrame G. Learning control barrier functions and their application in reinforcement learning: A survey. arXiv. 2024.
- 9.
Cheng Y, Zhao P, Hovakimyan N. Safe and efficient reinforcement learning using disturbance-observer-based control barrier functions. In: Proceedings of the Learning for Dynamics and Control (L4DC). 2023;1–12.
- 10. Ren H, Ma H, Li H, Lu R. A disturbance observer based intelligent control for nonstrict-feedback nonlinear systems. Science China Technological Sciences. 2023;66:456–67.
- 11. Ha W, Park J-H, Back J. An internal model disturbance observer based robust trajectory tracking control for articulated manipulators. Int J Control Autom Syst. 2025;23(2):479–88.
- 12. Al-Mola MHA, Abdelmaksoud SI. Disturbance Observer-Based Intelligent Control for Trajectory Tracking in Redundant Robotic Manipulators. Int J Robotics Control Syst. 2025;5(3):1743–57.
- 13. Liang X, Yournal of Aeronautics. 2023;36(11):271–80.
- 14. Liang X, Yao Z, Ge Y, Yao J. Disturbance observer based actor-critic learning control for uncertain nonlinear systems. Chinese J Aeronautics. 2023;36(11):271–80.
- 15. Geng J, Fu Z, Tao F, Wang N. Observer-based adaptive prescribed time optimal control for constrained nonlinear systems via actor-critic neural network. Neurocomputing. 2026.
- 16. Li L, Li J, Cao J. Data‐driven disturbance compensation control for discrete‐time systems based on reinforcement learning. Adaptive Control & Signal. 2024;39(7):1364–79.
- 17. Liu L, Lv J, Lin H, Zhan R, Wu L. ADP-based fault-tolerant control with stability guarantee for nonlinear systems. Entropy (Basel). 2025;27(10):1028. pmid:41148986
- 18. Nam DP, Hong Quang N, Phuong Nam T, Hai Yen TT. Adaptive dynamic programing based optimal control for a robot manipulator. IJPEDS. 2020;11(3):1123.
- 19. Wen Q, Yang X, Huang C, Zeng J, Yuan Z, Liu PX. Disturbance observer-based neural network integral sliding mode control for a constrained flexible joint robotic manipulator. Int J Control Autom Syst. 2023;21(4):1243–57.
- 20. Hu F, Ma T. Disturbance observer-based adaptive fuzzy finite-time cooperative control for high-order multi-agent systems with input saturation. Expert Systems with Applications. 2025;276:127179.
- 21. Yang P, Zhang S, Yu X, He W. Reinforcement-learning-based finite time fault tolerant control for a manipulator with actuator faults. IEEE Trans Cybern. 2025;55(6):2621–32. pmid:40272963
- 22. Iqbal J. Modern control laws for an articulated robotic arm: Modeling and simulation. Eng Technol Appl Sci Res. 2019;9(2):4057–61.
- 23. Fouzia M, Khenfer N, Boukezzoula NE. Robust adaptive tracking control of manipulator arms with Fuzzy Neural Networks. Eng Technol Appl Sci Res. 2020;10(4):6131–41.
- 24. Akil A, Nouaiti A, Touati A, Rabbah N. Modeling and nonlinear backstepping control of a 3-DoF robot manipulator. Eng Technol Appl Sci Res. 2025;15(3):22459–65.