3.1 Physical riveting and welding work cell and digital twin platform

During the production of stiffened H-beams in the H-beam riveting and welding work cell, stiffener plate gripping and riveting, along with the subsequent welding operation, are critical steps in the production workflow. As shown in Fig 1, the gripping robot first executes the stiffener plate gripping operation and then transports the plate to one side of the H-beam for riveting. After the stiffener plate is riveted to the H-beam, the welding robot performs the welding operation at the designated position to secure it. To complete the riveting and welding of a single stiffener plate, the gripping robot collaborates with a welding robot for a specific duration. Upon completing the collaboration, the gripping robot moves to the next designated position, carrying another stiffener plate, and collaborates with a nearby welding robot to perform the riveting and welding operations. This process continues until the entire production sequence is fully executed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Production workflow of stiffened H-beams.

https://doi.org/10.1371/journal.pone.0331515.g001

The physical H-beam riveting and welding work cell and its digital twin platform are illustrated in Fig 2. The system primarily consists of a gripping robot that grips and rivets the stiffener plates, as well as four welding robots. The gripping robot, along with the welding robots on both sides, is equipped with servo-controlled bases, allowing them to traverse along the robot track. On both sides of the work cell, H-beams are securely positioned. Stiffener plates are pre-stacked at the front of the work cell. This enables the gripping robot to pick up a new plate from the stacking area after completing the previous riveting and welding task. In the H-beam riveting and welding work cell, the gripping, riveting, and welding of stiffener plates are performed by the respective robots at designated positions according to pre-recorded teaching instructions. The stiffener plate riveting and welding operations simulated on the digital twin platform are illustrated in Fig 3. Once the gripping robot and a welding robot complete a riveting and welding cycle, the gripping robot returns to its initial position. It then picks up a new stiffener plate and moves to the next designated position to collaborate on the subsequent task. This process continues until both H-beams are completely assembled. Once the process is completed, all robots return to their initial positions, marking the end of the production sequence.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Illustration of the physical riveting and welding work cell and digital twin platform.

https://doi.org/10.1371/journal.pone.0331515.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Demonstration of stiffener plate riveting and welding operations.

https://doi.org/10.1371/journal.pone.0331515.g003

Within the digital twin platform of the work cell, the steel structure framework (excluding the robots) was modeled in SolidWorks. The positions of all objects in the model were determined based on precise measurements. According to the measurement data, their spatial positions in cyber space were determined using global or local coordinates. Additionally, the 3D models of the robots were defined in unified robot description format (URDF) files. Collision detection and feedback are enabled by integrating collision boxes into the robot model. Finally, the entire digital twin platform was developed in Unity 3D and connected to the physical work cell via an industrial computer.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The connection of physical space.

https://doi.org/10.1371/journal.pone.0331515.g004

For data transmission from the physical work cell to the digital twin platform, six-axis data is acquired via socket communication [31]. This data is then used to control the corresponding digital twin robots in real time through C# scripts. The industrial computer controls the physical work cell and facilitates data exchange between the two systems. It serves as the master station, while the gripping and welding robots function as slave stations. The master station transmits request frames to slave stations over IP, which, upon reception, execute the corresponding operations. As shown in Fig 4, the industrial computer communicates with a 16-port industrial Ethernet switch via the TCP/IP protocol. It transmits six-axis control commands and receives real-time six-axis data through this connection. After configuration, the Ethernet switch establishes a data link layer connection with the motion controller. Motion commands are transmitted from the motion controller to the robot layer via the EtherCAT protocol, enabling precise control of the servo-controlled base and robot posture.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Collision risk warning.

https://doi.org/10.1371/journal.pone.0331515.g005

By leveraging digital twin technology to link the physical and virtual spaces, intelligent monitoring of the production process at the physical work cell is enabled. Upon detecting equipment failures or other anomalies, the digital twin platform identifies collision risks, as illustrated in Fig 5. It immediately issues an alert, prompting the industrial computer to control the physical work cell to stop production and await manual intervention. Production resumes only after the fault is cleared.

3.2 Development of multi-agent interactive environment and system framework

During actual production, while maintaining safe and stable operations, the operating speed of each servo-controlled base is set to the highest feasible fixed value , enabling robots to reach their target positions in the shortest possible time. Let denote the total production time. Since the riveting and welding time starts after the gripping and welding robots reach their designated positions, it is influenced by the process itself. Optimizing without compromising quality is challenging. Therefore, to optimize production efficiency, adjusting the scheduling time of each robot is the most effective approach.

(1)

During riveting and welding operations, the servo-controlled bases of both the gripping and welding robots remain stationary, while their six-axis arms perform active movements. During the gripping operation, the gripping robot grips the stiffener plate at its initial position, with active six-axis arm movements. During the scheduling process, the servo motors of each robot base move, while the six-axis joints of all robots remain in their respective zero-position configurations. The key to optimizing multi-robot collaborative scheduling lies in optimizing the movement of the servo-controlled bases of the five robots. Therefore, in constructing the multi-agent reinforcement learning environment, as shown in Fig 6, only a 2D top view is considered. The 3D model of the work cell is transformed into a 2D top view multi-agent interactive environment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Abstraction of the 3D scene into a 2D multi-agent interactive environment.

https://doi.org/10.1371/journal.pone.0331515.g006

Fig 6 illustrates the 2D environment for multi-agent reinforcement learning. The blue square represents the gripping robot, while the four red squares represent the welding robots. Each robot is restricted to movement along the y-axis and has a predefined range of motion, which is consistent with the settings in the 3D scene. Collision detection is applied to welding robots sharing the same track. Meanwhile, the two H-beams are positioned on both sides, and purple dots indicate the designated riveting and welding positions. A rule is defined that when the x-coordinates of the gripping robot and a welding robot simultaneously align with that of a welding point, both robots halt at the position. They remain stationary for a fixed number of timesteps to accommodate the execution of the riveting and welding operations. Afterward, the gripping robot returns to its initial position and remains there for a certain number of timesteps to account for the stiffener plate gripping operation during actual operation. Once it has remained at the origin for a certain number of timesteps, the gripping robot resumes its movement and collaborates with another welding robot to perform the next riveting and welding task for a new stiffener plate. Finally, once all stiffener plates have been riveted and welded, all five robots return to their initial positions, marking the end of the production process.

After appropriate simplification of the problem and construction of the 2D environment, the rMAPPO algorithm is applied to solve the multi-agent scheduling problem. The task is addressed in a continuous action space under a fully cooperative framework. Additionally, it integrates a digital twin system to validate the model’s performance. The overall system framework is illustrated in Fig 7. The rMAPPO network is designed based on the Actor-Critic framework, where the actor networks take the agents’ observations as input and outputs the corresponding actions that the agents need to execute at the current timestep Δt. Each action represents the distance an agent needs to move, ranging from 0 to v_max ∙ Δt. Meanwhile, the critic network evaluates the agents’ actions based on accumulated rewards and global state information, guiding the actor networks to optimize their policies. As shown in Fig 7, after defining the production mission objective for the riveting and welding work cell, the corresponding information is transmitted to the industrial computer and the 2D environment. The trained rMAPPO network generates actions for each agent, which are integrated into a unified scheduling policy for the riveting and welding work cell. This policy is transmitted to the industrial computer and translated into corresponding servo base control commands. The commands are then dispatched to both the digital twin platform and the physical work cell for synchronized execution. The servo-controlled bases of the robots in the physical work cell execute precise movements to reach the designated positions. Upon reaching the target positions, the industrial computer controls the robots to perform the corresponding six-axis motions (such as riveting, gripping, or welding), while concurrently transmitting the six-axis data of the respective robots to the digital twin platform. This enables the corresponding robots in the digital twin platform to replicate the six-axis motions in real time. Ultimately, a cyber-physical co-control mechanism is established between the digital twin platform and the physical work cell. This framework not only enhances production efficiency using the rMAPPO model but also facilitates real-time monitoring of the work cell’s production status. Monitoring signals can be transmitted to the industrial computer by the digital twin platform. The framework also supports offline validation. By using six-axis data from the robot’s teaching motions, the correctness of rMAPPO decisions can be verified within the digital twin platform of the work cell. Once validated, the system can be connected to the physical work cell to achieve a real-time control loop.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Framework of riveting and welding work cell based on rMAPPO and digital twin.

https://doi.org/10.1371/journal.pone.0331515.g007

4.1 Proximal policy optimization

The primary objective of reinforcement learning algorithms is to maximize the expectation of the cumulative rewards , which is given by:

(2)

Here, and represent the agent’s state and action at timestep , respectively. represents the maximum number of timesteps for agent exploration. is the immediate reward at timestep , while is the discount factor, which controls the trade-off between long-term and short-term rewards. The sequence defines the agent’s trajectory, consisting of states and actions. represents the agent’s policy with parameters .

However, the PPO algorithm does not directly maximize the expected return , Instead, it follows the Actor-Critic framework, where the actor uses a neural network to determine the current policy and select the action , while the critic models the value function using another neural network, where represents the parameters of the value function. The critic assesses whether the action chosen by the actor is optimal or advantageous.

In traditional policy gradient (PG) methods [32], the gradient of the policy is defined as:

(3)

Here, represents the advantage function, which measures the relative advantage of the current action . In PPO, the actor’s objective function is derived from it and further refined into the objective function , which is given by:

(4)

PPO iteratively approximates the optimal policy by optimizing the above objective function, and thus indirectly optimizing . It replaces the conventional log probability formulation with the probability ratio of the new policy to the old policy:

(5)

This replacement mitigates update instability. The clipping mechanism is controlled by the hyperparameter to limit the magnitude of policy updates. With a properly chosen clipping coefficient , the return is non-decreasing [33]. The advantage function is computed using the generalized advantage estimation (GAE) method [34], formulated as:

(6)

Here, is a hyperparameter, .

In PPO, the critic’s objective function is defined as:

(7)

denotes the Critic network’s state value function, and represents the cumulative discounted rewards. The critic enhances the precision of state value estimation through minimizing .

The complete objective function of PPO, , integrates the objective functions of the actor and critic, while incorporating the policy entropy function to promote exploration. The final formulation is:

(8)

where and are scaling factors that control the trade-off among different components. In PPO, the policy network (actor) and value network (critic) collaborate to maximize . The policy network updates the policy through maximizing , while the value network refines state value estimation through minimizing .

4.2 Multi-agent proximal policy optimization

MAPPO is an adaptive variant of the PPO algorithm that adapts it for the multi-agent reinforcement learning (MARL) domain [35]. In multi-agent settings, the decision of one agent can affect both the environment observed by other agents and their subsequent decisions [36]. As shown in Fig 8, MAPPO adopts the centralized training and decentralized execution (CTDE) framework. During the training phase, all agents share a centralized critic, which leverages global information (including each agent’s observation , the action generated by the actor networks) and the discounted reward . The critic estimates the overall state value based on global information and optimizes the actor’s policy updates using the objective function . During the execution phase, each agent independently selects action based solely on its local observation , ensuring agent autonomy and scalability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Framework of MAPPO.

https://doi.org/10.1371/journal.pone.0331515.g008

The core objective of MAPPO is to maximize the expectation of the joint cumulative reward of all agents, , formulated as:

(9)

Here, denotes the joint policy of all agents, and is the set of parameters defining the joint policy. and denote the state and corresponding action of the i-th agent at timestep , respectively. represents the reward received by the i-th agent at timestep .

MAPPO also indirectly optimizes through policy optimization for individual agents. It incorporates a clipping mechanism to constrain the magnitude of policy updates, mitigating instability during training. The objective function for each actor, , is defined as:

(10)

where

(11)

Here, represents the probability of the i-th agent selecting action given its observation at timestep t. denotes the probability ratio between the new and old policies for the actor. is the advantage function, computed via the GAE according to equation (6).

The objective function of the entire actor network, , consists of the objective functions of individual actors and the entropy function of their policies, , incorporating batch normalization:

(12)

where is a hyperparameter that scales the entropy term, represents the number of agents, and denotes the maximum number of timesteps for agent exploration. The exploration phase consists of a fixed number of trajectories, each containing experience data from different timestep. The actor policy network optimizes the policy by maximizing .

MAPPO adopts a centralized critic network, where the joint state of all agents is used as input for the critic network to estimate state values. Additionally, a clipping mechanism is introduced to limit the magnitude of value updates. The objective function of the critic network, , is defined as follows:

(13)

where represents the discounted reward, and denote the value estimates of the joint state under the new and old policies, respectively. During critic network training, is minimized to improve value estimation accuracy. In MAPPO, the actor network and the critic network are typically optimized separately, which helps mitigate interference between networks and stabilize training.

4.3 Recurrent multi-agent proximal policy optimization

In MAPPO, if agents have full observability or their decision-making is minimally dependent on historical observations, both the actor and critic networks can be implemented using a multi-layer perceptron (MLP). However, in the multi-robot scheduling environment of the riveting and welding work cell studied here, agents operate under partially observable conditions. Additionally, after completing a riveting and welding task, the gripping robot must return to its initial position to grip the next stiffener plate. Therefore, the welding robots need to analyze historical data to determine whether the gripping robot is in a ready-to-cooperate state. MAPPO, as an on-policy algorithm, can only utilize data generated by the current policy during policy optimization. Unlike off-policy algorithms that utilize an experience replay mechanism, MAPPO cannot efficiently reuse past experiences. To address multi-agent scheduling in partially observable environments, a recurrent neural network (RNN) is integrated into the MAPPO framework. The actor and critic networks are built using RNNs, which increase the utility of individual samples by capturing more information from historical hidden states. This allows agents to effectively utilize historical information when handling temporal sequences and partially observable environments, enhancing policy optimization.

rMAPPO needs to retain the hidden states of the RNN when collecting data, as the RNN’s output depends on its past hidden states. In rMAPPO, the actor network employs an RNN to process the current observation along with the previous hidden state , which is then used to generate the action probability distribution:

(14)

where represents the action probability distribution produced by the corresponding actor network, and is the updated hidden state of the RNN layer, which retains historical observation information.

The critic network also incorporates an RNN to integrate historical information. It computes the state value estimation based on the global observation state and the previous hidden state , while simultaneously updating the hidden state of the RNN layer :

(15)

In the local observation processing unit of the actor network, the RNN processes past hidden states, enabling agents to incorporate past observations and address decision-making challenges in partially observable environments. In the state value estimation unit of the critic network, the RNN processes global states and past information, providing more accurate state value estimates to guide policy optimization.

In cooperative partially observable Markov decision processes (POMDPs), the convergence of rMAPPO is supported by two key theoretical foundations: the convergence of the centralized critic network and the synchronized policy updates across agents. According to the POMDP policy gradient theorem, the critic must first converge to the true team value function to ensure the unbiased estimation of advantages. This requirement is satisfied through centralized training under the Robbins-Monro conditions [37]. When all agents synchronously update their policies using the clipped objective in PPO, the joint policy improvement is monotonic. This is because the shared critic enables consistent credit assignment among agents, while recurrent policy gradient updates preserve the Markov property of belief states when sufficient historical information is available [33,38,39]. Although the joint policy space is formally the Cartesian product of individual policy spaces, the effective space is a strict subset due to policy coupling and the shared team objective. Given that the team return is continuous and bounded over this space, and under the assumption of compactness and appropriate learning rates, the joint policy can converge to a locally Pareto-optimal solution. Thus, the convergence of rMAPPO arises from the interplay between a stable value estimator and coordinated policy improvement among agents.

During the process of using rMAPPO to solve the robot scheduling problem, certain environmental constraints impose predefined action constraints on agents in specific scenarios. In such cases, the environment enforces the action directly, bypassing the neural network’s action generation. To handle this, action masking is employed to mask out actions generated by the actor network. For instance, when a welding robot and a gripping robot—in a collaborative state—reach a designated welding position, the environment imposes a waiting period for a certain number of timesteps to simulate the riveting and welding process. During these timesteps, action masking is applied, disabling actions generated by the actor network. Similarly, after a gripping robot completes a collaborative task, it must return to the initial position to grip a new stiffener plate. Since this movement sequence is explicitly defined in the environment, action masking is applied to prevent the actor network from generating actions during the corresponding timesteps. All such timesteps are treated as invalid timesteps during network computation. After masking invalid actions, it is essential to ensure that these invalid timesteps are excluded from loss calculation and gradient updates.

By introducing the mask , invalid timesteps are excluded from the objective function computation. In rMAPPO, the per-agent objective function of the actor network is formulated as follows:

(16)

The overall objective function of the Actor network is still given by equation (12).

Accordingly, in the critic network, after applying a mask to the joint state , the objective function is revised as follows:

(17)

where denotes the masked joint state, defined as: , where represents the element-wise multiplication operation. During backpropagation, as the mask is already incorporated into the objective function, data from invalid timesteps is excluded from gradient computation, thus avoiding disruptions in the optimization process.

The rMAPPO algorithm framework is depicted in Fig 9, providing a detailed representation of the algorithm’s structure and execution process. The left module describes the interaction between agents and the environment, including the process of generating observations , actions , and discounted rewards . The right module describes the detailed process by which the rMAPPO algorithm utilizes the aforementioned information. The critic network takes the joint state as input, along with the previous hidden state from its RNN, and outputs the global value function estimation at the current timestep. The critic network then updates the hidden state through the RNN module, and is forwarded to the GAE module for advantage estimation. The critic’s error is optimized via backpropagation using a clipped loss function. The actor network receives the local observation and the previous hidden state , then outputs the action probability distribution ( ). A clipping mechanism is applied to regulate updates, ensuring policy stability. Additionally, parameter updates in both the critic and actor networks are applied only to valid states and actions. Action masking is used to filter out invalid information, mitigating its impact on training stability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Framework of rMAPPO.

https://doi.org/10.1371/journal.pone.0331515.g009

In multi-agent scenarios, the joint value function may exhibit significant variations due to the complexity of states and actions. Value normalization mitigates this issue by constraining the value function within a narrower range. This improves gradient stability during training, enabling the critic network to align more effectively with its optimization objective and thus accelerating convergence. The normalized value function is defined as:

(18)

where and represent the updated mean and squared mean, respectively [40].

4.4 Reward function

The reward function provides a quantitative assessment of the utility of an agent’s action in a given state. A well-designed reward function can enhance the critic’s estimation accuracy. To enable efficient cooperation among agents, and to ensure the successful execution of stiffener plate riveting and welding tasks, the reward functions are defined separately. They are tailored for two distinct agent types: gripping robots and welding robots, as detailed below.

1. Gripping robot: The reward function for the gripping robot consists of two terms. The first is the negative sum of absolute differences between its y-coordinate and those of the remaining riveting positions. The second is a constant reward term . To encourage cooperation among robots, a fixed reward is assigned during the non-cooperative period, when the gripping robot returns to the origin to grip the next stiffener plate after completing the previous riveting task. When the gripping robot is in a ready-to-cooperate state, is set to 0. The reward function is formulated as follows:

(19)

where is a constant, represents the number of stiffener plates yet to be riveted, is the y-coordinate of the gripping robot, is the y-coordinate of the i-th stiffener plate.

1. Welding robot: the reward functions , , , for the four welding robots are determined by the gripping robot’s y-coordinate and a cooperation reward. The reward function is defined as follows:

(20)

where is the y-coordinate of the i-th welding robot, is the total number of stiffener plate welding tasks. When the welding robot is in a cooperative state, ξ is a positive value; otherwise, ξ = 0. The cooperation reward ξ for welding robots is designed to improve the overall efficiency of completing the welding tasks.

Since the research scenario involves multi-agent collaboration, a shared reward mechanism is employed. The final reward at timestep , denoted as , is the cumulative reward of all agents within the same timestep, with additional penalties for elapsed timesteps and unfinished tasks. The reward function is formulated as follows:

(21)

where and are constants. A well-designed penalty can significantly enhance coordination efficiency among agents. However, if the penalty is too large, it may diminish agents’ motivation to complete tasks efficiently.

5.1 Experimental setting

During the training phase, the servo-controlled base of the gripping robot is set to a movement speed of 2 m/s, while the servo-controlled bases of the welding robots are set to 3 m/s. The environment timestep is defined as Δt = 0.1s, with a collaboration duration of 20 Δt, and the gripping robot’s stiffener plate gripping time at its initial position set to 10 Δt. The maximum length of H-beams that can be processed at the work cell is 8 meters. Based on actual production conditions, each H-beam typically requires 6–10 stiffener plates, with a minimum spacing of 0.2 meters between adjacent plates. Accordingly, five groups of experiments are designed, with the number of stiffener plates set to 6, 7, 8, 9, and 10, respectively. Each experimental group is repeated 5 times, and the evaluation metrics are averaged. In each episode, the stiffener plate positions are randomly generated within a range of 0–8 meters.

All experiments are conducted on a workstation featuring an Intel Core i9-13980HX CPU and an NVIDIA RTX 4060 GPU. The software versions used are as follows: python(3.8), pytorch (1.5.1), Unity 3D (2021.3.36f1c1), CUDA (10.1). Details regarding the learning rates for the actor and critic networks, along with the clipping parameter ε, will be provided in subsequent sections. Unless otherwise stated, parameter settings are shown in Table 1. The training procedure consists of 80 episodes, each with a maximum length of 2500 timesteps.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Simulation settings of rMAPPO algorithm.

https://doi.org/10.1371/journal.pone.0331515.t001

5.2 Convergence performance

Root mean square propagation(RMSprop) is adopted to adaptively adjust the learning rates of the actor and critic networks during training. RMSprop adjusts learning rates by applying an exponentially weighted moving average, decreasing the rate for parameters with large gradients and increasing it for those with smaller ones. This optimizer is well-suited for algorithms involving temporal models and non-stationary optimization tasks.

A proper initial learning rate for RMSprop can significantly improve convergence speed and stability. One core feature of PPO-based algorithms is the clipping parameter ε, which controls policy updates and is typically set between 0.1 and 0.3 based on empirical evidence [33]. To determine appropriate initial learning rates for the actor and critic networks, as well as the clipping parameter ε, five experimental groups are configured. Each experimental group is repeated 5 times with different random seeds. The mean cumulative reward of the final 10 episodes is used as the performance metric. If training does not converge, the average reward is recorded as 0.

As shown in Fig 10, all subfigures (a-e) demonstrate that the reward values are sensitive to the parameter combinations. The color gradient from blue to yellow indicates the presence of distinct performance peaks. Across the five experiments, as the number of stiffener plates increases, the task becomes more complex. Accordingly, the distribution of high-reward regions (yellow bars) shifts. Based on the results of all five experiments, lower initial learning rates ensure more stable updates and lead to higher cumulative rewards. Therefore, the initial learning rates for the actor and critic are set to , and the clipping parameter is set to 0.2 for the remaining performance evaluations, as this configuration consistently achieves convergence and yields the highest rewards.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Initial learning rate and clipping parameter selection.

In (a), (b), (c), (d) and (e), different numbers of stiffener plates are designated for riveting and welding on the H-beam, with positions randomly assigned.

https://doi.org/10.1371/journal.pone.0331515.g010

5.3 Ablation experiment and algorithm comparison

The ablation experiment is conducted for rMAPPO by removing the RNN layers from the actor and critic networks. This evaluates the role of the recurrent module in rMAPPO under a partially observable multi-agent scenario. In the MAPPO network, which replaces the RNN layers with a multi-layer perceptron (MLP), the relevant parameters of the actor and critic are kept consistent with those in rMAPPO.

Fig 11 illustrates the convergence performance curves of rMAPPO and MAPPO. Experimental results indicate that increasing the number of stiffener plates leads to greater task complexity, resulting in more pronounced training instability in MAPPO. In contrast, rMAPPO demonstrates stronger adaptability, outperforming MAPPO in both convergence speed and reward accumulation. It reaches a stable state 10–15 episodes earlier and achieves higher steady-state reward values. The experimental results indicate that rMAPPO better captures the temporal dependencies of the environment. Particularly in the later training stages, the RNN module enhances agents’ ability to retain historical information, enabling them to learn superior policies. This highlights rMAPPO’s stronger adaptability to sequential dependencies in partially observable environments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Convergence performance of rMAPPO and MAPPO.

In (a), (b), (c), (d) and (e), different numbers of stiffener plates are designated for riveting and welding on the H-beam, with positions randomly assigned.

https://doi.org/10.1371/journal.pone.0331515.g011

Fig 12 depicts the value loss trends of rMAPPO and MAPPO across the five experimental groups. Clear differences in training dynamics are observed between rMAPPO and MAPPO. Across all five subfigures, both algorithms exhibit value loss curves that transition from high volatility to gradual convergence. However, rMAPPO shows more stable convergence in most cases. In the early training stages, both algorithms experience significant oscillations in value loss. As training progresses (around episodes 20–50), rMAPPO’s loss curve declines more rapidly, with reduced fluctuations, and eventually stabilizes around 0.1–0.2 after episode 50. In subfigure (e), where the number of stiffener plates is highest, rMAPPO demonstrates superior stability. This indicates a more robust value function estimation capability when facing increased task complexity. Such performance gain can be attributed to the RNN module in rMAPPO, which captures temporal dependencies in the critic network and significantly enhances estimation accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Value loss of rMAPPO and MAPPO.

In (a), (b), (c), (d) and (e), different numbers of stiffener plates are designated for riveting and welding on the H-beam, with positions randomly assigned.

https://doi.org/10.1371/journal.pone.0331515.g012

The convergence performance and cumulative rewards of the proposed rMAPPO algorithm are further evaluated in comparison with two baseline methods: MADDPG and MASAC. Due to differences in task characteristics and the design of the reward function, the maximum achievable reward varies across different riveting and welding scenarios. As illustrated in Fig 13, five comparative experiments were conducted. With increasing task complexity—represented by a growing number of stiffener plates from subfigures (a) to (e)—the performance differences among the algorithms become more pronounced. In all cases, rMAPPO consistently achieves the highest cumulative rewards, demonstrating stronger generalization and reaching optimal performance earlier than other algorithms. Its final convergence performance demonstrates a clear advantage over the baselines, especially as task complexity increases. Moreover, rMAPPO exhibits greater training stability. It enters the stable improvement phase more quickly, maintains smoother training, and adapts more effectively to fluctuations caused by exploration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Comparison of convergence performance of rMAPPO with other baseline methods.

In (a), (b), (c), (d) and (e), different numbers of stiffener plates are designated for riveting and welding on the H-beam, with positions randomly assigned.

https://doi.org/10.1371/journal.pone.0331515.g013

Across the five experimental groups, the completion time for different riveting and welding tasks varies depending on task complexity. As shown in Fig 14, the rMAPPO algorithm demonstrates a clear advantage in time efficiency. Throughout the five experiments with increasing task complexity, it consistently requires the fewest timesteps. From subfigure (a) to (e), all algorithms show a general increase in timestep consumption. However, rMAPPO shows the slowest growth in timestep consumption and the smallest fluctuation range, indicating excellent stability. Upon entering the convergence phase, it completes operations in just 700–900 timesteps for most simpler tasks (subfigures a and b). For more complex tasks, it still performs significantly better than other algorithms, requiring only 1200–1400 timesteps. Moreover, rMAPPO converges faster during training, reaching lower timestep levels earlier. This reflects its high efficiency and robustness in handling randomly distributed riveting and welding tasks. Overall, these results highlight rMAPPO’s clear advantages in both stability and policy quality, particularly in complex or partially observable environments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Comparison of timesteps of rMAPPO with other baseline methods.

In (a), (b), (c), (d) and (e), different numbers of stiffener plates are designated for riveting and welding on the H-beam, with positions randomly assigned.

https://doi.org/10.1371/journal.pone.0331515.g014

5.4 Digital twin implementation

In the testing phase, the trained model is deployed on the industrial computer and integrated with both the physical work cell and its digital twin platform for joint debugging. This aims to assess the impact of the rMAPPO model on production efficiency when integrated with the digital twin system. All experiments involving the digital twin are conducted in Unity 3D.

This study examines the system’s real-time performance in practical deployment. During task execution, real-time latency primarily stems from three sources: communication, computation, and physical delays [41]. Testing shows that the rMAPPO model incurs a decision-making delay of approximately 10 ms when generating scheduling strategies. The physical delay of the servo drives or mechanical joints is around 3 ms. Since the rMAPPO model is deployed on the industrial computer, the scheduling policy is translated into servo base control commands and dispatched to both the digital twin platform and the physical work cell. In the physical work cell, the industrial computer sends six-axis robot control or servo base control commands to drive the robots. Meanwhile, the digital twin platform receives the six-axis motion data from the physical robots via the industrial computer. Combined with the transmitted servo base control actions, this data enables the twin robot to execute corresponding actions in synchronization. The digital twin platform can also transmit monitoring signals back to the industrial computer. The average end-to-end transmission delays between modules, as measured in tests, are shown in Table 2. After the industrial computer processes and dispatches the generated scheduling policy to both the physical and virtual systems, transmission delays occur as the six-axis data is returned to the digital twin platform for execution. However, physical delays arise during the actual execution of robotic actions. These combined effects reduce the execution time gap between virtual and physical robots, improving synchronization. Ultimately, system latency remains within a few milliseconds, meeting the real-time requirements of riveting and welding tasks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. End-to-end data transmission delay.

https://doi.org/10.1371/journal.pone.0331515.t002

Under the integration of the digital twin and physical work cell, ten production rounds were conducted using different methods for five task groups, each with varying stiffener quantities and positions. The performance of rMAPPO is assessed and compared with other baseline algorithms, as well as with the fixed policy previously implemented in the work cell. The fixed policy operates based on a predefined sequence: the gripping robot first moves to the nearest riveting and welding target, then signals the nearest welding robot to arrive and jointly executes the riveting and welding task. Afterward, the gripping robot returns to the initial position to retrieve the next stiffener plate and repeats the process.

Fig 15 compares the task completion times of four multi-agent reinforcement learning algorithms across five task groups of increasing complexity using box plots. Overall, as task complexity increases, all algorithms show a rising trend in execution time. Among them, rMAPPO consistently achieves the shortest completion times and greater stability in most groups, maintaining relatively low median times (approximately 75–110 seconds). In the integrated testing environment involving both the physical riveting and welding work cell and the digital twin platform, rMAPPO reduces task completion time by approximately 7% to 10% compared to other reinforcement learning algorithms. Fig 16 illustrates the average cumulative rewards for each algorithm across the five testing groups. rMAPPO outperforms all others in every group, indicating stronger overall performance and better compatibility with the reward function design adopted in this study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Completion times for different algorithms.

https://doi.org/10.1371/journal.pone.0331515.g015

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. The average cumulative reward of algorithms.

https://doi.org/10.1371/journal.pone.0331515.g016

Table 3 compares the average task completion times between reinforcement learning algorithms and a rule-based fixed policy across five experimental groups. In the integrated testing environment involving the physical riveting and welding work cell and its digital twin platform, all reinforcement learning methods exhibit improved efficiency over the fixed policy. Among them, rMAPPO achieves the most significant reduction in task duration, decreasing actual execution time by 12.4% to 15.8% compared to the fixed policy. Based on both the training performance in the 2D multi-agent interaction environment and the evaluation results in the physical–digital integrated system, rMAPPO exhibits greater stability and efficiency. As such, it is more suitable for solving the robotic scheduling problem in the H-beam riveting and welding work cell.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. The average completion time of each method.

https://doi.org/10.1371/journal.pone.0331515.t003