Research background and technological evolution

Early rehabilitation robot control mainly relied on classical control theories, such as Proportional-Integral-Derivative (PID) controllers and Sliding Mode Control (SMC). These methods achieved trajectory tracking through manual parameter adjustments, which, although computationally efficient, were inadequate in adapting to system uncertainties caused by neural conduction delays and abnormal muscle coordination in stroke patients [5–7]. This often resulted in weak robustness of control parameters and noticeable oscillation effects. With the development of machine learning technologies, fuzzy logic and traditional neural networks (e.g., Support Vector Machines) were introduced to improve motion classification accuracy using data-driven methods [8,9]. However, these shallow models struggled to capture the temporal dependencies in complex action sequences, such as “grasp-release-grasp,” and often exhibited intent analysis deviations and control delays in clinical applications.

In recent years, deep learning methods have brought new breakthroughs in rehabilitation robot control. Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), have significantly enhanced the ability to analyze movement intent by capturing the temporal dynamics of surface electromyography (sEMG) [10,11]. Convolutional Neural Networks (CNNs) have effectively handled the spatial features of joint angle data, improving the accuracy of complex motion trajectory modeling [12,13]. For example, Glove-Net [14] has validated multisensory fusion for grasp classification, laying a foundation for our work. However, the high computational complexity of deep networks conflicts with the computational limitations of embedded devices, leading to increased real-time control delays and reduced device battery life. Meanwhile, multimodal data fusion still primarily relies on simple concatenation or early fusion, lacking explicit modeling of the cross-modal causal relationships between “electromyographic intent - joint motion - force feedback,” which introduces safety risks during fine operations, such as grasping fragile objects.

At the clinical application level, neural network-based adaptive control methods (e.g., adaptive neural PID) have shown potential for adapting to individual differences by adjusting parameters through online learning [15,16]. For example, using Radial Basis Function (RBF) networks to approximate system dynamics has improved trajectory tracking accuracy to some extent [17]. However, these methods generally suffer from poor model interpretability—rehabilitation practitioners find it challenging to understand the relationships between neural network weights and sensor contributions, resulting in trial-and-error clinical adjustments and increased training risks. Additionally, existing technologies lack the adaptability to dynamic changes during the rehabilitation stage, failing to automatically switch between ensuring trajectory accuracy during passive training and intent response speed during active training, thus limiting the continuity and effectiveness of rehabilitation training.

Currently, the core development direction of intelligent rehabilitation robots focuses on the deep fusion of multimodal data and biomechanical adaptation of control strategies. The combination of contrastive learning and attention mechanisms provides a new approach to cross-modal semantic alignment. By explicitly modeling the causal relationships between modalities, this approach ensures that changes in electromyographic intent are accurately reflected in joint motion and force feedback. This enables more effective integration of multiple data sources and enhances the overall system’s adaptability to diverse rehabilitation scenarios. By enforcing temporal consistency constraints between different modal features, it significantly enhances the causal associations of neural activation, motion execution, and mechanical feedback [18,19]. However, achieving interpretable modulation of modality weights while ensuring computational efficiency remains a technical challenge. Additionally, with the advancement of technologies such as brain-machine interfaces and virtual reality, multimodal control architectures integrating visual feedback and EEG signals have become new research hotspots [20,21]. However, such solutions require higher synchronization precision of sensors and data preprocessing algorithms, and clinical implementation still faces challenges related to hardware integration and noise suppression.

Research methodology and core contributions

To address the aforementioned challenges, this paper proposes the Contrastive Cross-Modal Attention - Stacked Gated Recurrent Unit - Adaptive PID Control Architecture (C-GAP), which achieves closed-loop optimization from electromyographic intent perception to precise motion execution through a full-link design of “multimodal semantic alignment - hierarchical temporal modeling - clinical-oriented control.” The architecture first dynamically fuses surface electromyography (sEMG), joint angles, and force signals through the Contrastive Cross-Modal Attention (C-CMA) mechanism, using cosine similarity loss to enforce temporal consistency across modalities and resolve the semantic misalignment of heterogeneous data. It then uses a stacked GRU network to extract hierarchical temporal features, capturing short-term dynamics, such as muscle synergy activation during grasping, at the lower layers, while learning long-term trends, such as muscle strength changes during continuous training, at the upper layers. Residual connections are incorporated to enhance the modeling efficiency of complex action sequences. Finally, an adaptive PID control module is used to dynamically adjust proportional, integral, and derivative parameters based on modality weights, constructing a “intent perception - effect feedback” dual-loop control strategy and ensuring precise control of grip strength and joint trajectories through a safety threshold protection mechanism.

Core Contributions:

• This paper proposes a contrastive cross-modal attention mechanism that explicitly constrains the temporal consistency of sEMG signals, joint angles, and force signals, effectively resolving the semantic misalignment of multimodal data. This provides physically meaningful fused features for control strategies, significantly enhancing the mapping accuracy of “neural intent - action execution.”
• The design of the stacked GRU network with residual connections reduces computational complexity while enhancing the ability to model long-term dependencies. This enables efficient representation of complex action sequences and provides a lightweight solution for real-time control in embedded rehabilitation devices.
• This paper constructs a dual-loop PID control framework driven by modality weights, supporting automatic adaptation during the rehabilitation phase and incorporating safety threshold protection. This effectively balances the trajectory accuracy of passive training with the intent response speed of active training, improving the safety and individual adaptability of rehabilitation training.

Research progress on multimodal data fusion methods

Multimodal data fusion is a core technology for achieving precise control of rehabilitation robots. Its central challenge lies in integrating the spatiotemporal correlations of heterogeneous data such as surface electromyography (sEMG), joint angles, and force feedback. Early studies relied on manual feature engineering, where the time-domain features of sEMG (such as root mean square, zero-crossing rate) and kinematic parameters of joint angles (such as joint angular velocity, acceleration) were concatenated and combined with traditional machine learning methods like Support Vector Machines (SVM) or Random Forest for motion classification or force prediction [22,23]. Although these methods demonstrated some effectiveness for simple actions, their limited nonlinear modeling capabilities of shallow models made it difficult to capture the dynamic coupling relationships between multimodal data. In complex grasping tasks, the instantaneous variations in muscle signals and delayed responses in joint movements could not be effectively synchronized, leading to insufficient robustness in control strategies [24].

With the development of deep learning, attention mechanisms have been introduced to enhance the interaction between modalities. Self-attention mechanisms dynamically assign weights by calculating the similarity between different modal features. For instance, in the fusion of sEMG signals and joint angles, attention is focused on muscle group activation patterns and joint degrees of freedom most relevant to the current action, thus improving intent analysis accuracy [25,26]. However, traditional attention mechanisms are data-driven and fail to explicitly constrain the physical consistency between modalities which specifically involves the biomechanical causal relationship between neural activation intent and motion execution effect. Relevant studies have explicitly analyzed grasp synergies and low-dimensional coordination patterns, linking multimodal hand data to interpretable coordination structures [27,28]. For example, the activation of specific muscle groups during grasping should correspond to specific changes in joint angles and force output [29]. This lack of explicit physical modeling reduces the interpretability of the fused features, making it difficult for rehabilitation therapists to adjust control strategies based on attention weights.

Recent studies have attempted to integrate contrastive learning theory by designing consistency loss functions to align the temporal distributions of multimodal data. For example, in the fusion of sEMG - joint angles - force signals, cosine similarity loss is used to constrain the temporal synchronization between sEMG signals and force signals, thereby enhancing the causal correlation between muscle activation and force output [30]. However, these methods depend on complex network architectures and pretraining processes, and their model generalization ability and real-time adaptability still need improvement in clinical scenarios with significant individual differences among stroke patients and dynamically changing data distributions.

Application of temporal modeling techniques in rehabilitation control

Recurrent Neural Networks (RNNs) and their variants (such as LSTM networks and GRUs) are the mainstream methods for handling sequential data and are widely used for dynamic feature extraction of sEMG signals. LSTM effectively captures long-term dependencies in sEMG signals through the cooperation of input, forget, and output gates, showing strong modeling capabilities for long-term changes like muscle fatigue and neural adaptation during continuous grasping motion force prediction [31]. However, the complex gating structure of LSTM leads to higher computational complexity, and the inference delay on embedded rehabilitation devices (such as Jetson Nano) often exceeds 50 ms, which fails to meet the strict 20 ms latency requirement for real-time control [32,33]. GRU simplifies the gating structure by merging the forget and input gates into an update gate, thus maintaining temporal modeling capabilities while improving computational efficiency, with a 40% faster inference speed compared to LSTM [34]. As a result, GRU has become the preferred choice for lightweight rehabilitation devices. However, a single-layer GRU network is insufficient for hierarchical feature extraction in multi-stage action sequences such as “finger extension - grasping - release.” The lower layers struggle to capture the short-term dynamics of muscle synergy activation at the moment of grasping, while the upper layers inadequately model long-term trends such as muscle strength changes and joint coordination during continuous training [35].

To balance modeling accuracy and computational efficiency, stacked architectures and residual connections have been introduced into GRU networks. Stacked GRUs process data at multiple levels, with the lower layers focusing on high-frequency instantaneous changes in sEMG signals and subtle fluctuations in joint angles, while the upper layers learn overall trends and phase transition logic in action sequences [36]. Residual connections alleviate the vanishing gradient problem in deep networks and preserve detailed information from the lower layers [37]. However, existing studies have mainly designed stacked architectures for single-modal data (such as sEMG signals or joint angles) and have not fully utilized the complementary nature of multimodal data, resulting in incomplete temporal feature extraction in complex scenarios, which limits the performance improvement of control strategies.

Challenges in matching adaptive control strategies with clinical needs

Rehabilitation robot control strategies need to achieve a dynamic balance between trajectory accuracy, intent response speed, and operational safety. Traditional Proportional-Integral-Derivative (PID) controllers are widely used due to their simple structure and strong robustness, but their fixed parameters cannot adapt to individual patient differences and changes in rehabilitation stages [38,39]. For example, stroke patients require strict joint trajectory constraints during the passive training phase to avoid secondary injuries, while during active training, quick response to muscle intent is required to enhance neural feedback [40,41]. Manually adjusting PID parameters is time-consuming and prone to control errors due to improper parameter settings, such as trajectory deviation or force overshoot. Fuzzy PID controllers dynamically adjust PID parameters using fuzzy logic rules, improving the system’s adaptability to uncertainties such as muscle strength fluctuations and neural conduction delays, especially in force-control scenarios [42]. However, the design of fuzzy rule sets relies on the experience of rehabilitation medical experts and is difficult to generalize to complex clinical situations like muscle spasms and joint adhesions. Additionally, it lacks integration of multimodal data, and the real-time changes in sensor signals and parameter adjustments are disconnected.

Data-driven adaptive control methods, such as neural network-based PID, optimize control parameters by online learning of the patient’s muscle dynamics model. For instance, combining Radial Basis Function (RBF) networks to approximate nonlinear system dynamics can improve trajectory tracking accuracy and reduce the cost of manual tuning [43,44]. However, these methods generally suffer from two main issues: first, they fail to associate the fusion results of multimodal data with control parameter adjustments, creating a disconnect between sensor weight distribution and control strategies; second, they lack a clinically-oriented safety protection mechanism. When a patient experiences muscle spasms leading to excessive grip force or abnormal joint angles, the system cannot timely trigger safety thresholds, such as torque limits or motor shutdown, based on the integrated multimodal signals, which poses risks of equipment damage and secondary patient injuries [45,46]. Furthermore, existing strategies do not implement automatic recognition of rehabilitation stage transitions and control mode switching, and the same parameters perform poorly across different training phases (passive, active, autonomous), limiting personalized rehabilitation.

Existing technologies still face critical gaps in the physical interpretability of multimodal fusion, temporal modeling efficiency, and clinical adaptability. There is an urgent need for systematic solutions that integrate biomechanics and clinical needs. The proposed C-GAP architecture in this paper, through the collaborative design of contrastive cross-modal attention, stacked GRU, and adaptive PID, aims to overcome these bottlenecks and provide a new paradigm for clinical applications of rehabilitation robot control technology.

Dataset

This study uses the publicly available MUSED-I and Ninapro DB5 datasets, which provide abnormal electromyographic (EMG) signals from stroke patients during rehabilitation and multi-modal synchronized gripping action data. These datasets offer rich clinical and physiological signal support for model training and control strategy validation.

The MUSED-I dataset focuses on upper limb motor dysfunction rehabilitation and contains 8-channel surface EMG signals (sEMG) collected by Myo armbands, covering 6 basic hand movements such as wrist flexion and extension, and finger opening and closing. The subjects include 10 healthy individuals and 2 stroke patients. Each movement is repeated with the Fugl-Meyer motor function score attached, enabling the analysis of the relationship between EMG signal features and movement quality at different stages of rehabilitation. The key value of this dataset lies in the annotation of abnormal EMG signals from stroke patients, such as signal distortions due to muscle spasms or compensatory movements, providing critical data for the model to handle clinical noise and individual differences.In particular, the addition of artificial noise aims to simulate real-world distortions often observed in stroke patients, reflecting signal degradation and variability caused by muscle spasms or abnormal muscle coordination. This enhances the robustness of the model, ensuring that it can generalize well to real clinical scenarios.

The Ninapro DB5 dataset features multi-modal synchronized data collection, integrating 8-channel sEMG (200Hz), 3-axis grip force sensor data (50Hz), and 5 degrees of freedom joint angle signals (100Hz). It covers 20 types of gripping movements, including forceful grips and precise pinches. The subjects consist of 10 healthy individuals and 5 stroke patients, with some data recording abnormal mechanical feedback and joint motion trajectories during muscle spasm states. This dataset’s strength lies in its provision of “EMG intent - joint movement - force output” temporal alignment data, supporting the verification of temporal synchronization for multi-modal fusion algorithms and closed-loop testing of control strategies. The inclusion of stroke patient data, particularly during muscle spasm states, further enhances the clinical relevance of the dataset by reflecting the challenges of varying muscle coordination and abnormal movement during rehabilitation.

The combined use of these two datasets covers rehabilitation training scenarios for stroke patients, ranging from basic movements to complex gripping tasks, while providing time-space aligned multi-modal signal samples. This ensures the generalization ability of the model under both normal and abnormal physiological states, laying a data foundation for validating the effectiveness of the “intent perception - action execution” control framework.

Multimodal data preprocessing

The aim of multimodal data preprocessing is to address the heterogeneity of sensor signals by performing temporal alignment, noise suppression, and feature normalization, providing high-quality inputs for model training. For the MUSED-I and Ninapro DB5 datasets, the specific processing steps are as follows:

Temporal synchronization. Since each modality’s sensor has different sampling rates (sEMG at 200Hz, joint angles at 100Hz, and force signals at 50Hz), the sEMG time series is used as a reference. The zero-order hold interpolation method is applied to upsample the joint angle and force signals to 200Hz, ensuring that the multimodal data is fully aligned with a 5ms time step, preventing temporal misalignment that could affect cross-modal correlation analysis.

Surface EMG Signal (sEMG) processing. First, a 50Hz notch filter is applied to remove power line noise. Then, a 20-450Hz band-pass filter is used to retain the frequency components associated with muscle activation. Next, full-wave rectification and a 10ms sliding average filter are applied to convert the AC signals into DC signals reflecting muscle activation intensity, reducing high-frequency noise interference.

Joint angle and force signal processing. For joint angles (e.g., wrist flexion and extension angles, finger opening and closing) and force signals (e.g., grip force, fingertip pressure), a median filter is first used to remove abnormal spikes. Then, the minimum-maximum normalization method is applied to scale the values to the [0, 1] range, eliminating sensor range discrepancies. For stroke patient data, additional Gaussian noise with a mean of 0 and standard deviation of 0.1 is added to simulate signal distortions caused by muscle spasms or compensatory movements, enhancing model robustness.

Data splitting and sequence construction. The preprocessed multimodal time-series data is split into training, validation, and test sets in a 7:2:1 ratio, with stratified sampling to ensure balanced distributions of action types and subjects. A sliding window technique (window size: 1s, step size: 0.5s) is applied to segment the data, generating input sequences that include current and past 1s of multimodal signals. The corresponding future 50ms of force signals and joint angles are used as output labels, constructing “temporal input - control target” mapping pairs to support the model’s short-term action trend prediction training.

Through the above preprocessing steps, the issues of temporal misalignment, noise interference, and scale differences in multimodal data are effectively addressed, resulting in a unified training sample set. This provides a solid data foundation for subsequent comparisons of cross-modal attention mechanisms and stacked GRU network-based feature extraction.

Overall model description

This section presents the overall architecture of the proposed C-GAP (Contrastive - Gated Attention - based Pathway) model, which is designed to address the challenges of hand function rehabilitation robot control. The model integrates data preprocessing, feature extraction, temporal modeling, and control output, providing a unified framework for both healthy and pathological scenarios. The architecture begins with the data input and preprocessing stage. For healthy subjects, multi-modal data including electromyography (EMG), force, and angle signals are collected from the Ninapro DB5 dataset. For stroke patients, single-modal EMG data from the MUSED-I dataset are utilized. All data undergo essential preprocessing steps such as denoising, normalization, and temporal alignment to ensure data quality and consistency. Next, the Contrastive Cross-Modal Attention mechanism plays a crucial role in fusing multi-modal features while handling the differences between healthy and pathological data. This is followed by the Stacked GRU Temporal Modeling module, which extracts the sequential dynamics of hand movements through multiple layers of Gated Recurrent Units. Finally, the Adaptive PID Control Module generates precise control trajectories based on the output of the Stacked GRU, adapting to the specific movement characteristics of different patients. The overall architecture of the C-GAP model is illustrated in Fig 1, which comprehensively demonstrates how each component collaborates to achieve accurate hand function recognition and rehabilitation control.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic diagram of the overall architecture of the C-GAP model for hand function rehabilitation robot control, illustrating the workflow from multimodal signal processing and feature extraction via contrastive cross-modal attention and stacked GRU temporal modeling to adaptive PID control output.

https://doi.org/10.1371/journal.pone.0342802.g001

Contrastive Cross-Modal Attention (C-CMA) module

The Contrastive Cross-Modal Attention (C-CMA) module is a core component for achieving deep fusion of multi-modal data, whose design goal is to explicitly model the temporal consistency of “EMG intention, joint movement, and force feedback” to address the issue of modal semantic disconnection in traditional methods. As shown in the structure Fig 2 of the Contrastive Cross-Modal Attention module, this module constructs cross-modal interaction relationships in a shared feature space, and through contrastive learning and dynamic weight assignment, it forces different modal features to maintain physical consistency in the temporal dimension, thereby providing fused features with biomechanical interpretability for the subsequent stacked GRU network.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Structure diagram of the Contrastive Cross-Modal Attention (C-CMA) module, illustrating the process from multi-modal signal embedding to cross-modal interaction and feature fusion for temporal consistency modeling.

https://doi.org/10.1371/journal.pone.0342802.g002

Let the input be the surface electromyography (sEMG) signal sequence , joint angle sequence , and force signal sequence , where T denotes the number of time steps, and represent the feature dimensions of each modality, respectively. First, the module maps each modal signal to a shared feature space via an independent encoder (implemented by a multi-layer perceptron), yielding the modal feature sequences:

(1)

where , and D is the dimension of the shared feature space. This process converts heterogeneous modal signals into comparable feature representations, laying the foundation for cross-modal interaction.

In the cross-modal similarity calculation stage, the module calculates the local similarity matrix between modality i and modality j at time step t:

(2)

This formula captures the instantaneous similarity between modalities using cosine similarity, which is more robust to noise and temporal delays compared to traditional methods.

To achieve dynamic weight assignment, the module inputs the cross-modal similarity matrix into a weight generation network , and performs softmax normalization with a temperature parameter τ to generate the attention weight of each modality at time step t. Taking the EMG modality weight as an example:

(3)

Similarly, the joint angle modality weight and force modality weight are calculated, and they satisfy the normalization condition . The weight generation process strengthens the modal features most relevant to the current movement through nonlinear transformation, while suppressing the interference of irrelevant noise.

The dynamic weight assignment mechanism is designed to align with fundamental biomechanical principles of human hand movements. Specifically, the weight generation network is constrained to prioritize modalities that dominate the biomechanical characteristics of the target movement: sEMG modality is enhanced for muscle activation-driven movements, force modality for force-stabilized movements, and joint angle modality for joint coordination-dominated movements. This design ensures the learned attention weights retain biological plausibility, rather than being purely data-driven without physical interpretability.

Based on the attention weights, the module performs multi-modal feature fusion to obtain the fused feature that contains cross-modal correlations:

(4)

This fused feature not only retains the original information of each modality but also embeds the temporal dependency between modalities, providing a more informative input for subsequent temporal modeling.

To enforce the temporal consistency between modalities, the module designs a contrastive consistency loss function to distinguish between positive sample pairs (modal signals within the same movement cycle) and negative sample pairs (modal signals from different movement cycles or with noise interference):

(5)

This loss function forces cross-modal features to maintain consistency in the temporal dimension by maximizing the logarithmic probability of similarity for positive sample pairs and minimizing that for negative sample pairs, thereby strengthening the causal correlation among “neural activation, movement execution, and mechanical feedback”.

Considering the dynamic continuity of movement sequences, the module introduces a temporal difference constraint term to penalize abnormal jumps of modal features between adjacent time steps:

(6)

where represents the temporal difference for each modality. This term ensures that feature changes conform to the movement continuity assumption in biomechanics, avoiding unreasonable feature mutations caused by noise.

Finally, the total loss function of the Contrastive Cross-Modal Attention module is:

(7)

where λ is a balance coefficient. By adjusting this parameter, an optimal balance can be achieved between modal consistency and movement dynamics.

Through the above mechanisms, the C-CMA module transforms the fusion process of multi-modal data from simple data concatenation to semantic alignment based on biomechanical laws. It not only improves the physical interpretability of features but also provides more discriminative inputs for the temporal modeling of the subsequent stacked GRU network, thus forming a complete technical chain of “perception, fusion, modeling” and laying a key foundation for the precise control of rehabilitation robots. The design of this module not only considers the heterogeneity of multi-modal data but also incorporates causal constraint relationships in biomechanics, providing a new technical path to solve the cross-modal mapping problem in rehabilitation scenarios.

Stacked GRU temporal modeling

The Stacked Gated Recurrent Unit (Stacked GRU) temporal modeling module is a core temporal feature extraction component of the C-GAP architecture. It processes multi-modal fused features through a multi-layer network to achieve joint modeling of short-term muscle synergy activation and long-term neural adaptation processes in rehabilitation movements. This module takes the fused feature sequence output by the contrastive cross-modal attention module as input and enhances gradient flow in deep networks through residual connections, addressing the inadequacy of traditional single-layer recurrent networks in modeling long-term dependencies of complex action sequences.

Let the fused feature sequence output by the contrastive cross-modal attention module be . The stacked GRU contains L layers of networks, with each GRU layer having a hidden state dimension of H. The input of the l-th layer GRU at time step t is the hidden state of the previous layer (the input of the first layer is ).

The gating mechanism of a single GRU layer forms the basis of temporal modeling. For the l-th layer GRU, the update gate and reset gate at time step t are defined as:

(8)

(9)

where is the Sigmoid activation function, are weight matrices, and are bias vectors. The gating mechanism enables selective memory of historical information.

Based on the gating signals, the update formulas for the candidate hidden state and the final hidden state are:

(10)

(11)

where is the hyperbolic tangent activation function, and denotes element-wise multiplication. This mechanism realizes dynamic update of hidden states through gating weights, preserving key temporal information.

To capture multi-level temporal features, the stacked GRU structure introduces residual connections, where the input of the l-th layer is the residual fusion of the output of the previous layer and the original features:

(12)

where is the residual weight coefficient. When l = 1, . Residual connections effectively alleviate the gradient vanishing problem in deep networks, ensuring the preservation of underlying detailed features.

To address the need for modeling long-term trends such as muscle strength changes in rehabilitation training, a temporal integration operation is introduced after the output of the top GRU layer:

(13)

This integration operation aggregates short-term temporal features into long-term trend representations, suitable for capturing slow changes in neural adaptation processes and enhancing the model’s ability to perceive dynamic changes in rehabilitation stages.

The training objective of the stacked GRU is achieved through a loss function containing temporal smoothness constraints:

(14)

where is the model’s predicted value, is the true label, the first term is the mean squared error loss, and the second term constrains the temporal smoothness of hidden states. μ is a balance coefficient, ensuring that the predicted results conform to the movement continuity in biomechanics.

To preserve both short-term dynamics and long-term trends, the module weightedly fuses the outputs of each GRU layer with integral features:

(15)

where are layer weights, which adaptively learn to determine the contribution of features from different layers to the current movement, achieving organic integration of multi-level temporal features.

Through the above mechanisms, the stacked GRU temporal modeling module enhances the ability to extract hierarchical features from complex action sequences while maintaining computational efficiency. The bottom layers capture short-term dynamics such as muscle synergy activation during grasping moments, while the top layers, combined with integration operations, learn long-term trends such as muscle strength changes in continuous training. This provides precise feature representations containing temporal context for subsequent adaptive PID control, effectively addressing the incomplete temporal modeling of traditional single-layer recurrent networks in rehabilitation scenarios, and laying a temporal feature foundation for accurate mapping from movement intention to action execution.

Adaptive PID control module

The adaptive PID control module is a core execution component of the C-GAP architecture for achieving precise motion control. Its design goal is to dynamically adjust proportional (Kp), integral (Ki), and derivative (Kd) parameters based on the temporal feature sequence output by the stacked GRU, realizing dual closed-loop control of force and trajectory for the end-effector of the rehabilitation robot. Through the joint driving of modal feature weights and rehabilitation stage features, this module addresses the problems of poor individual adaptability and stage switching lag caused by fixed parameters of traditional PID, while embedding a safety threshold constraint mechanism to ensure the safety of the training process.

Let the temporal feature sequence output by the stacked GRU be , which contains key information such as muscle activation intensity and joint movement trends. The control targets are to track the desired force signal and desired joint angle , with actual outputs and respectively.

The basic PID control law forms the foundation of the module. The outputs of the force control loop and angle control loop are defined as:

(16)

(17)

where and are the force error and angle error respectively. and are time-varying PID parameters adjusted dynamically.

To achieve parameter adaptation, the module inputs the temporal features into a parameter prediction network (implemented by a two-layer fully connected network), outputting the adjustment amounts of PID parameters at each moment:

(18)

(19)

(20)

where are modal weights output by the C-CMA module, and T_c is the integral time constant. The parameter adjustment amounts are dynamically correlated with modal importance and error characteristics. For example, the EMG weight dominates the adjustment of the proportional term to quickly respond to intention changes.

The PID parameter update formulas based on adjustment amounts are:

(21)

(22)

(23)

where η is the learning rate, and the function limits the integral parameter within the safe range to avoid control overshoot caused by integral saturation. The same update mechanism is applied to the angle loop parameters .

The safe range boundaries , along with the force safety threshold and joint angle constraint (corresponding to in the subsequent safety constraint formula), are determined based on the natural motion boundaries of human hand joints and the physiological load tolerance of upper limb muscle groups. These values are set to align with the essential safety requirements for hand rehabilitation training, preventing excessive force output or joint overextension that may lead to secondary injuries for patients. To ensure training safety, the module introduces dual safety constraints on force and angle, triggering attenuation control when the actual output exceeds the safety threshold:

(24)

where u(t) is the original control output, x(t) can be or , is the safety threshold, and is the attenuation coefficient. Progressive control attenuation is achieved by integrating the degree of overrun, avoiding impacts on patients caused by sudden stops.

The final control quantity is obtained through weighted fusion of the force loop and angle loop, with weights dynamically determined by the rehabilitation stage feature s(t) (passive/active/autonomous):

(25)

where takes a smaller value in the passive stage (focusing on trajectory constraints) and a larger value in the active stage (focusing on force feedback), realizing stage adaptation of the control strategy.

The training loss function of the module includes constraints on control accuracy and parameter smoothness:

(26)

where is the parameter smoothness coefficient, ensuring gradual adjustment of PID parameters and avoiding sudden changes in control quantities.

Through the above mechanisms, the adaptive PID control module dynamically correlates multi-modal temporal features with control parameters, realizing closed-loop mapping from “intention perception” to “precise execution”. Its core advantages are: parameter adjustment based on modal weights enhances the biomechanical interpretability of the control strategy; the integral form of safety constraints realizes a smoothly attenuated protection mechanism; stage-adaptive fusion solves the problem of control target switching in different rehabilitation stages, providing a safe, precise, and dynamically adaptive control scheme for personalized rehabilitation training.

Experimental platform

The experimental platform is constructed in a three-tier architecture, consisting of the rehabilitation robot hardware, multimodal sensor system, and embedded computing unit, providing both hardware and software support for the validation of the C-GAP framework.

The hardware system includes the upper limb exoskeleton rehabilitation robot (model: RehabArm-EX7), multimodal sensors, and an embedded computing unit. The following Fig 3 provides an overview of the rehabilitation robot setup, including the key hardware components and their interactions within the experimental platform. The robot is equipped with a 7-DOF robotic arm and integrates a 6-axis force sensor (ATI Nano17) at the end-effector for real-time force measurement. A 16-channel MyoWare surface electromyography (sEMG) sensor (sampling rate 1000Hz) covers the major upper limb muscle groups and synchronously collects muscle activation signals. Joint angles are obtained by fusing high-precision encoders (resolution: 0.1^°) with an inertial measurement unit (IMU). The computing unit uses the NVIDIA Jetson AGX Orin, which integrates a 6-core ARM CPU and a 128-core GPU, running a real-time operating system to achieve a 20ms control cycle for hard real-time computing. Safety performance validation utilizes publicly available datasets, covering simulated clinical scenarios including basic hand movements and muscle spasm-induced signal distortion cases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Overview of the experimental platform, including the rehabilitation robot hardware, multimodal sensors.

https://doi.org/10.1371/journal.pone.0342802.g003

The software environment is based on Ubuntu 20.04, using Python 3.8 and PyTorch 1.12 for model training and inference, with TensorRT 8.4 used to optimize neural networks, ensuring that GRU inference latency per step is less than 5ms. The control algorithms are developed based on the ROS framework, with force control and trajectory control modules implemented in C++ to ensure real-time performance. A self-developed data processing library supports multimodal signal synchronization (timestamp error < 1ms), real-time filtering, and feature extraction.

The system addresses sensor clock drift issues through a hardware synchronization trigger mechanism (GPIO signals) and the IEEE 1588 precision time protocol. A pipeline architecture for data acquisition, model inference, and control output is established. The module interactions based on shared memory and dynamic priority scheduling strategies ensure that the highest-priority tasks are scheduled with a delay of less than 1ms, providing reliable real-time performance guarantees for the experiments.

Experimental parameter settings

Experimental parameter settings are optimized around the core modules and training process of the C-GAP architecture, with key hyperparameters determined through grid search and cross-validation to balance control accuracy and real-time performance. In the Contrastive Cross-Modal Attention (C-CMA) module, the shared feature dimension D is set to 128, and the time window length is 200ms to capture short-term dynamics of movements. The temperature parameter τ is tuned to 0.8 using the validation set, and the contrastive loss balance coefficient λ is set to 0.5 to balance modal consistency and motion dynamics constraints.

The stacked GRU module adopts a 3-layer network structure, with each layer having a hidden state dimension H of 256. The residual weight coefficients decrease layer by layer (, , ) to enhance the retention of underlying features, and the long-term dependency integration window Ω is set to 1s to aggregate neural adaptation trends in rehabilitation training.

For training configurations, the optimizer uses AdamW (weight decay 0.001) with an initial learning rate of 1e-4 and a cosine annealing scheduling strategy. The batch size is set to 32 to adapt to the memory constraints of the embedded platform. The number of training epochs is 100 with an early stopping mechanism (termination if the validation loss does not decrease for 10 consecutive epochs). Data augmentation strategies include random time shifts (ms) and signal amplitude perturbations () to improve the model’s robustness to clinical noise.

In the adaptive PID control module, the initial range of the proportional parameter K_p is set to [0.5, 5.0], the integral parameter K_i is limited to [0.1, 2.0] to avoid saturation, and the derivative parameter K_d ranges from [0.05, 1.0]. The parameter update learning rate η is 0.01. Safety thresholds are set to 30N for force signals and 15^° for joint angle overruns, with a constraint time window t_s of 100ms. The attenuation coefficient is tuned to 0.5 through simulating muscle spasm scenarios. Force limit events are defined as actual grip force exceeding target force ±20% with target force ranging from 5–30N for pinch and 10–50N for grip. Emergency stop is triggered when force overrun lasts more than 50ms or joint angle overrun exceeds 15^° for more than 30ms. All parameter settings are validated for generalization on a mixed dataset of healthy subjects and stroke patients to ensure the stability of control strategies across different rehabilitation scenarios.

Comparison model settings

To ensure a fair comparison, all models included in this study were adapted to our upper-limb rehabilitation task. We aligned the input data (synchronized sEMG, joint angles, and force feedback) and adjusted their architectures and hyperparameters as needed to ensure consistency with our experimental setup. This process allowed us to evaluate each model under similar conditions, ensuring that performance comparisons were valid and meaningful.

LLMT [47]: An improved Transformer architecture is used to process multi-channel sEMG and IMU signals, with sparse attention mechanisms employed to reduce computational complexity, enabling lower limb motion pattern classification.

Guo et al. [48]: This model constructs an end-to-end time-series prediction framework based on Transformer, using musculoskeletal simulation to generate hemiplegic motion data and predict the future 100ms of assistive torque.

OSCS [49]: An integrated fuzzy logic pain detection module dynamically adjusts the exoskeleton stimulation parameters based on patient feedback, optimizing personalized stimulation schemes in rehabilitation training.

Wang et al. [50]: This model proposes an online optimization strategy for the membership function of the Takagi-Sugeno fuzzy model to improve the position tracking control performance of rehabilitation robots.

FL-HPR [51]: A federated learning architecture that integrates image and point cloud data, optimized through dynamic graph edge convolutions, enabling human pose recognition with multi-center data.

Zafar et al. [52]: A federated learning framework suitable for edge devices, combined with model quantization technology, to enable lightweight deployment of sEMG-based gesture recognition.

DAMMRL [53]: A multi-model adaptive control strategy using a dual-agent mechanism to dynamically match changes in the patient’s motion abilities, achieving human-robot collaboration adaptation in robot-assisted rehabilitation.

LP-STGCN [54]: Introduces a self-supervised pLSTM module and an attention-guided mechanism, based on graph convolutional networks, to achieve quantitative evaluation in lower limb motion rehabilitation.

Evaluation metrics

The evaluation metrics include the mean absolute error (MAE) of force control, the mean absolute error (MAE) of joint angle tracking, the trajectory completion rate for assessing control accuracy; the control cycle compliance rate (20ms hard real-time), neural network inference latency, and multimodal data synchronization error for evaluating real-time performance; and the force exceedance event rate, safety threshold response time, and emergency stop success rate for measuring safety.