3.1. Optimization of reinforce algorithm incorporating strategic gradient

The REINFORCE tries to maximize Cumulative Rewards (CRs). In ADR-VN tasks, robots need to choose the optimal action based on visual input and current state. Strategic Gradient refers to a gradient calculation method in RL where the parameters of the policy function are directly optimized to maximize the CR. Its classic mathematical definition is shown in equation (1).

(1)

In equation (1), calculates the mathematical expectation of all the trajectories generated by strategy . is a complete trajectory, is the strategy gradient, and represents the strategy parameter. represents the probability of choosing action under the state . is the CR at time step . Compared with the standard policy gradient and REINFORCE methods, the proposed optimization policy gradient in this paper distinguishes itself in three key aspects: The classical policy gradient typically uses an independent value network as the benchmark, while this paper significantly reduces the gradient variance by designing a shared feature extraction layer in the ACA based on the state value function; The standard REINFORCE relies on complete trajectories to update parameters, while this paper combines the single-step update mechanism of Actor-Critic to improve training efficiency; Traditional policy gradient does not optimize for visual feature extraction, while this paper embeds the residual group – attention cascading unit in the visual front-end, making the gradient update more focused on high-dimensional semantic features. At the same time, through deep separable convolution and channel pruning, it achieves lightweighting and ensures the real-time inference capability of the embedded platform. The RGB images are normalized, and the pixel values are mapped to the [0,1] interval. The depth image is subjected to noise filtering and invalid value filling. Finally, features are extracted through CNN to obtain a visual feature vector with a dimension of 256. is the robot state, including the real-time position, speed, heading angle, and residual battery of the robot. After standardization, these state parameters are concatenated into a state feature vector with dimension 8 [18–20].

The reward function value of ADR-VN is based on the weighted importance of navigation tasks and statistical analysis of historical navigation data, and is quantitatively derived from the core task semantics and actual user needs of ADR-VN. The reward design follows the systematic principles of prioritizing the core task, constraining the safety bottom line, and assisting with efficiency optimization. The core task priority principle corresponds to the core needs of users using ADR to complete navigation tasks; The safety bottom line constraint principle corresponds to the strict requirements for collision-free operation of ADR in industrial scenarios; The efficiency-assisted optimization principle corresponds to the conventional requirements for ADR path stability and motion energy efficiency in practical applications. Based on these design principles, combined with the weight ratios of indicators such as task completion rate, collision occurrence rate, and path deviation rate in historical navigation data, specific reward values are derived. The terminal reward for successfully reaching the target (distance ≤ 0.3m) is set at + 100. Since reaching the target is the core task semantics of ADR-VN and the primary requirement of users for the navigation task, this value is 10 times the base reward unit, highlighting the core priority of task completion. The collision penalty (obstacle distance ≤ 0.5m, duration ≥ 0.5s) is set at −50. Since collision directly leads to the interruption of the navigation task and equipment damage, it is a strict safety risk to be avoided in industrial applications. This value is 50% of the core task reward and twice the general reward. Through a strong penalty mechanism, ADR’s obstacle avoidance behavior is strengthened, reflecting the hard constraint of the safety bottom line. The immediate reward of +2 for path adherence (deviation ≤ 0.2m) and penalty of −1 for deviation (>0.5m) are calibrated to balance path stability and movement flexibility. The ± 1 and ±0.5 rewards/penalties for speed control (1-3m/s) align with energy efficiency and motion safety constraints. A sensitivity analysis is conducted by adjusting each reward/penalty value by ±30%: results show that the model’s NSR fluctuates by less than 3%, and decision latency remains within 0.05s, confirming the robustness of the value settings. The interaction between multiple reward factors follows a hierarchical logic: safety (collision penalty) and task completion (target reward) are primary constraints, while path adherence and speed regulation act as secondary optimizations. When conflicts arise (e.g., path deviation to avoid obstacles), the negative penalty for collision (−50) outweighs the path deviation penalty (−1), guiding the model to prioritize safety. The CR formula begins with the core logic of RL: an ADR obtains an immediate reward at each time step during navigation. Since future rewards are uncertain, a discount factor is introduced to weight future rewards, balancing the model’s focus on short-term feedback and long-term navigation goals. The CR starting from time step t includes the immediate reward at (weighted by ) and discounted rewards from subsequent time steps, which is integrated into a continuous summation form until the navigation task terminates at time step . Through controlled variable experiments, is determined as the optimal value to ensure the model focuses on long-term target arrival without ignoring short-term actions like obstacle avoidance. The CR starts from time step , as shown in equation (2).

(2)

In equation (2), is the CR. denotes the discount factor, is the immediate reward, and is the time step. The optimal value of γ is determined to be 0.99 through controlled variable experiments in the study. When γ = 0.99, the model can focus on the long-term navigation target without overly ignoring the immediate feedback of short-term actions. The specific process of the REINFORCE is shown in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Flow of the REINFORCE algorithm.

https://doi.org/10.1371/journal.pone.0347775.g001

In Fig 1, the policy network is initialized as CNN + fully connected layers: three convolutional layers (kernels 5 × 5, 3 × 3, 3 × 3; strides 2; outputs 16/32/64 channels) with two max-pooling layers (2 × 2), followed by three dense layers (256, 128, 6 neurons). The input is a 264-dimensional feature vector (composed of 256-dimensional visual features and 8-dimensional robot state features), and the basic output is 6 discrete actions, corresponding to straight driving, 15° left turn, 15° right turn, 30° left turn, 30° right turn, and braking. These six actions can be mapped to specific speed instructions for the left and right wheels of the robot. When driving straight, the speed of both wheels is the reference value. When turning, the speed of the single-side wheel decreases linearly according to the turning angle. When braking, the speed of both wheels is zero. At the same time, through the action space interpolation strategy, the discrete actions are smoothly mapped to continuous actions, supporting continuous adjustment of speed and steering angle. It can meet the requirements of smooth trajectory planning in high-dimensional action spaces. Parameters use the He normal initialization (bias = 0). Training starts with 1,000 diverse problem instances (straight, curved, multi-obstacle paths). Actions are sampled via roulette wheel from the policy’s probability distribution; A greedy path (max-probability actions) provides an upper-bound baseline. A moving average baseline based on CR is used to reduce policy gradient variance. Gradients are computed and parameters updated with Adam (initial lr = 1e-4, cosine annealing to 1e-5). Training stops when any of the following occurs: 100k iterations are reached, test success rate stabilizes (<1% fluctuation over 10 iterations), or gradient norm drops below 1e-5.

This study introduces the Actor-Critic algorithm to optimize the REINFORCE algorithm, further optimizing its performance to improve navigation efficiency and stability. The Actor-Critic algorithm is an RL method that combines policy gradient and value function estimation. In ADR-VN, the Actor is responsible for generating actions, while Critic is responsible for evaluating the value of these actions [21–23]. The strategy gradient selection ACA algorithm is shown in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Schematic diagram of ACA in VN of ADRs.

https://doi.org/10.1371/journal.pone.0347775.g002

In Fig 2, the ACA network receives visual input and environmental state as inputs, and generates policy functions and value functions through its internal network structure. These functions are further used to guide the behavior decisions of robots in navigation environments. Each unit includes a strategy function, a value function, and a corresponding network structure. They synchronize optimization strategies and value functions to achieve more effective learning of optimal strategies. Each processing unit interacts with the navigation environment, receiving environmental feedback such as visual status, obstacle information, and reward signals, to continuously adjust and optimize its strategy and value evaluation, and improve the navigation performance of ADR in complex dynamic environments [24–25]. The ACA algorithm defines the state behavior value function as the expected CR that can be obtained by performing an action in a specific visual state, as given by equation (3).

(3)

In equation (3), is the action-state value function. is the discount factor. is the visual state at time , including the robot’s position, speed, heading, and environmental information obtained through the camera. is the action taken at , such as steering angle, acceleration, etc. is the probability distribution of strategy, that is, action selection. is the immediate reward obtained at time , usually designed based on factors such as whether the robot successfully avoids obstacles and travels along a predetermined path [26]. The state value function represents the expected CR of the state at time , updated as shown in equation (4).

(4)

In equation (4), and are the state value function at time and . is the weight vector at . is the deviation at . is a function related to value updates, used to adjust value estimates. Deviation is used to measure the difference between predicted and actual values, as shown in equation (5).

(5)

In equation (5), is the immediate reward obtained at time . is a small positive number used to control the smoothness of deviations. is the expected value in the new state. is the expected value in the current state. The parallelized ACA for ADR-VN is shown in Fig 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Parallel ACA for VN of an autonomous robot.

https://doi.org/10.1371/journal.pone.0347775.g003

In Fig 3, two intelligent agents are located in their respective VN environments and communicate with the policy logic control module through data transmission pipelines. The strategy logic control module includes a strategy predictor and a strategy trainer, which work together to optimize the autonomous driving model. The strategy predictor is responsible for predicting navigation strategies, while the strategy trainer updates model parameters based on the prediction results, thereby improving the VN ability of ADR [27–28]. The process of improving the REINFORCE algorithm based on ACA is shown in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Flowchart of improved REINFORCE algorithm based on ACA.

https://doi.org/10.1371/journal.pone.0347775.g004

In Fig 4, Actor and Critic networks are initialized with He normal and Xavier normal distributions, respectively, and biases set to 0.1 to avoid near-zero initial outputs. The system then acquires the current state via synchronized sensor data, visual input at 15 fps, and robot state at 50 Hz. The Actor selects actions using an ε-greedy strategy, which are executed via the motion control module to produce motor commands. New states and rewards are collected in real time. The Critic estimates state values for both current and next states, computes the TD error, and uses it directly as the advantage signal to evaluate action quality. Critic parameters are updated via Adam (lr = 5e-5) by minimizing squared TD error, while Actor parameters are updated via gradient ascent on the policy gradient (same learning rate). The loop continues until termination conditions are met: reaching the goal, collision, timeout, or battery below 5%. This collaborative ACA operates within a REINFORCE-based policy optimization loop.

3.2. Construction of vn model based on improved reinforce

This paper effectively optimizes the strategic gradient of the REINFORCE by combining the ACA. This model needs to process complex visual information and make real-time decisions in dynamic environments, thus requiring higher performance of the algorithm. This study constructs an efficient VN model by integrating ResBlock and AM into an improved REINFORCE. ResBlock preserves input information through skip connections, allowing for constructing deeper network structures and improving model performance [29–31], as shown in Fig 5.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. ResBlock structure diagram.

https://doi.org/10.1371/journal.pone.0347775.g005

In Fig 5, the upper ordinary direct connection structure takes the feature map as the input, which then passes through a convolutional layer with a kernel size of 1 × 1 and an output channel number of 64. After being nonlinearly transformed by the ReLU activation function, it is connected to another convolutional layer with a kernel size of 3 × 3 and an output channel number of 64. Similarly, after the ReLU activation, it finally passes through a convolutional layer with a kernel size of 1 × 1 and an output channel number of 256, and is activated by ReLU. Eventually, the feature map is output. The main path of the lower ResBlock structure is exactly the same as the ordinary direct connection structure. At the same time, an additional shortcut connection is added from the input to the end addition operation. The original input feature is added element-wise with the output of the main path, and the final output is . Through this residual connection design, while retaining the original feature information, effective deep feature extraction is achieved, which can alleviate the gradient disappearance problem that may occur during the training of deep neural networks. The design of ResBlocks allows the network to effectively avoid gradient problems when processing deeper features, improving the stability of network training [32–33]. The , output after ResBlock processing, is shown in equation (6).

(6)

In equation (6), means the input feature data. is the ReLU function utilized to introduce nonlinear characteristics. AM further enhances the model’s ability to capture key visual features. The Channel Attention Module (CAM) and Spatial Attention Module (SAM) re-calibrate the channel and spatial feature response. This combination makes the model to focus more on important visual information, thereby rising the accuracy of navigation decisions [34–35]. Fig 6 shows the module structure combining CAM and SAM.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Module combining CAM and SAM.

https://doi.org/10.1371/journal.pone.0347775.g006

In Fig 6, the input feature map first enters the first-level processing unit, which contains parallel convolution branches and CAMs. The convolution branches consist of three different-sized convolution kernels of 1 × 1, 3 × 3, and 1 × 1. After the convolution operation completes, feature extraction is fused with the output of the CAM. The CAM re-weights the features extracted by the convolution in the channel dimension to enhance the feature information of key channels. The features after the first-level fusion continue to be passed downward to the second-level processing unit, whose structure is the same as the first level, retaining the parallel convolution branches composed of 1 × 1, 3 × 3, and 1 × 1 convolution kernels, while replacing them with a SAM. This module enhances the weights in the spatial dimension for the features extracted by the convolution, highlighting important spatial position features. Finally, the convolution branch of the secondary processing unit and the SAM output are fused, and the fused attention feature map is output after the final 1 × 1, 3 × 3, and 1 × 1 convolution kernel processing. Accurate enhancement and effective extraction of high-level visual features are achieved. This design strengthens navigation decisions by prioritizing informative environmental features [36]. The strategy network generates the probability distribution of actions built on the extracted features, while the value network assesses the current state’s value to assist the strategy network in making decisions. Eventually, the output layer outputs the final navigation action, as displayed in Fig 7.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Diagram of the ADR-VN model based on the improved REINFORCE algorithm.

https://doi.org/10.1371/journal.pone.0347775.g007

The model in Fig 7 uses improved REINFORCE technology to process 15 fps sequences with optical flow to predict obstacle trends. Preprocessing normalizes, removes distortion/noise, and enhances illumination for feature consistency. The feature layer outputs a 256D vector via convolution, ResBlocks, and attention, extracting features from edges to semantics like obstacles/lane lines. The Actor outputs Softmax-normalized actions with regularization; The Critic outputs [0, 100] state values for TD errors. Actions selected via ε-greedy are sent via ROS to the chassis controller and converted to wheel speeds using a differential model. Training uses asynchronous parallel updates and gradient clipping (threshold 10) for efficiency/stability.

4.1. Comprehensive performance analysis of improved reinforce algorithm

To further assess the performance of the improved REINFORCE in ADR-VN tasks, many experiments are performed. To simulate the driving environment in the real world, the CARLA environment is used for autonomous driving simulation. This environment provides a configurable virtual city map, including elements such as roads, traffic signals, pedestrians, and other vehicles. The experiment selects the Town05 map, which includes various scenes such as urban roads, highways, tunnels, and parking lots. The weather conditions are set to three types: sunny, rainy, and foggy, respectively simulating different lighting and visibility scenarios. The 1,000 problem instances generated in the experiment cover three typical scenarios: straight roads (30%), curves (40%), and areas with dense obstacles (30%). Each scenario undergoes systematic changes in terms of lighting (daytime strong light, nighttime weak light, sudden changes in tunnel brightness), dynamic obstacle density (0–5 moving objects), and path complexity (curvature radius of 2–10 m). The training set and test set are divided in an 8:2 ratio, and the test instances do not appear in the training set in terms of layout, obstacle type, and lighting conditions, ensuring out-of-distribution evaluation. All instances are independently generated through random seeds to avoid data leakage. All instances are independently generated through random seeds to prevent data leakage. To analyze the sample efficiency, quantitative tests are conducted on the core indicators of the model’s NSR and decision accuracy under different training rounds. The results show that when the training rounds are reduced to 70%, 50%, and 30% of the original setting, the model’s NSRs are 91.3%, 88.2%, and 82.5%, and the decision accuracies are 94.5%, 92.1%, and 87.6%, respectively. The performance shows a smooth decline as the training rounds decrease, without a sudden drop, demonstrating good sample utilization efficiency. Table 1 shows the experimental configuration.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Experimental settings.

https://doi.org/10.1371/journal.pone.0347775.t001

The improved REINFORCE algorithm is compared with the REINFORCE algorithm, Deep Deterministic Policy Gradient (DDPG), and Deep Double Q-Network (DQN). The selection of the baseline algorithm strictly follows the principles of domain representativeness and task adaptability: The traditional DRL navigation method is selected as the basic comparison group, covering two core frameworks of early policy gradient and value iteration. Modern mainstream methods are included as advanced comparison groups, covering stability optimization, exploration ability enhancement, and global feature modeling, ensuring the coverage of typical algorithms of different technical paths in the ADR-VN domain. The parameters of each algorithm are all set to the optimal configuration verified in the original literature and unified the experimental environment parameters: REINFORCE’s learning rate: 1e-3, with discount factor of 0.98; DDPG experience replay buffer size: 1e6, target network update rate: 0.001; DQN dual network update interval: 1000 steps, ε decay rate: 0.995; PPO trust region clipping coefficient: 0.2, default learning rate: 3e-4 (after optimization for navigation task, it becomes 1.5e-4); SAC default temperature parameter: 0.2 (after optimization for navigation task, it becomes 0.15), discount factor: 0.99; Transformer-based VN methods (Transformer-DRL) number of multi-head attention heads: 8, feature dimension: 256. All algorithms are implemented based on the PyTorch framework, with 50,000 training iterations, batch size 64, and the physical parameters and sensor parameters of the experimental environment remain consistent to ensure the fairness of the comparison. The ACC and recall rates of several algorithms are shown in Fig 8.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Comparison of ACC and recall rates of several algorithms.

https://doi.org/10.1371/journal.pone.0347775.g008

In Fig 8 (a), as the iterations increase, the ACC of the improved algorithm rapidly rises and reaches its highest ACC after approximately 300 iterations, approaching 0.95. In Fig 8 (b), the recall rate of the improved algorithm rapidly increases and reaches its highest point after approximately 400 iterations, approaching 0.91. The improved algorithm outperforms other algorithms in both ACC and recall metrics, demonstrating its faster learning and convergence ability during the iteration process. This shows that the improved algorithm has a higher NSR and recall rate in ADR-VN tasks, thereby improving overall navigation performance. The convergence and runtime comparison of several algorithms are shown in Fig 9.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of convergence and runtime of several algorithms.

https://doi.org/10.1371/journal.pone.0347775.g009

In Fig 9 (a), the improved algorithm quickly converges when the number of iterations reaches about 100, with a convergence rate close to 1.0, demonstrating its efficiency during the training process. The REINFORCE converges at around 200 iterations, with a convergence rate of approximately 0.65. The convergence rates of DDPG and DQN reach approximately 0.6 and 0.5 at 300 and 400 iterations, indicating a slower convergence speed. The research algorithm has significantly better convergence speed than the compared algorithm, and can achieve higher convergence rates faster. In Fig 9 (b), the improved algorithm has the shortest runtime throughout the entire iteration process, gradually increasing from about 5 seconds to about 15 seconds from the beginning to 500 iterations. The runtime of the REINFORCE increases from about 10 seconds to about 20 seconds, DDPG increases from about 15 seconds to about 35 seconds, and DQN increases from about 30 seconds to about 45 seconds. The improved algorithm exhibits the best performance in ADR-VN tasks, with fast convergence speed and short runtime. This demonstrates that the algorithm can well improve the training and running efficiency of the model, and is suitable for application in autonomous driving scenarios that require fast response. The Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) of several algorithms are shown in Fig 10.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Comparison of RMSE and MAE of several algorithms.

https://doi.org/10.1371/journal.pone.0347775.g010

In Fig 10 (a), as the iterations rise, the RMSE value of the improved algorithm rapidly decreases and stabilizes after approximately 20 iterations, ultimately stabilizing at around 0.10. The improved algorithm performs the best in reducing RMSE, followed by REINFORCE and DDPG, while DQN performs relatively poorly. In Fig 10 (b), as the iteration grows, the MAE value of the research algorithm rapidly decreases and stabilizes after approximately 20 iterations, ultimately stabilizing at around 0.09. The improved algorithm outperforms the compared algorithm in reducing both RMSE and MAE error metrics, demonstrating faster convergence speed and higher accuracy during the iteration process. This shows that the improved algorithm has higher navigation accuracy and stability in ADR-VN tasks, which can improve overall navigation performance. The accuracy of the algorithm in the value function and strategy function is shown in Fig 11.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Accuracy of the algorithm in the value function and the policy function.

https://doi.org/10.1371/journal.pone.0347775.g011

In Fig 11 (a), at the beginning of iteration, the accuracy of the value function of the improved algorithm rapidly increases, reaching over 0.8 after approximately 300 iterations, and maintaining stable growth in subsequent iterations, ultimately approaching 1.0. The improved algorithms perform best in value function evaluation, enabling faster and more accurate assessment of the value function, thereby improving decision quality. In Fig 11 (b), the improved algorithm also performs outstandingly in terms of the accuracy of the policy function, reaching over 0.8 after approximately 300 iterations and maintaining stable growth in subsequent iterations, ultimately approaching 1.0. The improved REINFORCE algorithm also performs the best in policy function evaluation, which can evaluate policy functions faster and more accurately, improving the accuracy and overall performance of policy selection.

4.2. Evaluation of adr-vn application effectiveness

To assess the practical application effect of the improved algorithm in ADR-VN tasks, the performance of the algorithm in simulated environments and real scenes is further examined. The average reward value of ADR-VN in the simulated environment and real scene is shown in Fig 12.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. Average reward value of VN of autonomous robot in simulated environment and real scene.

https://doi.org/10.1371/journal.pone.0347775.g012

In Fig 12 (a), as the number of data points grows, the sliding average reward value gradually rises and tends to stabilize, while the actual reward value shows significant volatility. After approximately 1,000 data points, the sliding average reward value tends to stabilize, indicating that the model’s learning in the simulated environment gradually converges. However, although the actual reward value has shown an overall upward trend, it fluctuates greatly, indicating a performance difference between the simulated and real environments. In Fig 12 (b), in the real scene, the sliding average reward value also tends to stabilize after about 1,000 data points, but the average reward value is slightly lower than in the simulated environment. The actual reward value fluctuates more in real scenarios and shows a significant decrease at certain data points. This is due to the complexity and uncertainty in the real environment. Nevertheless, the overall trend shows an increase in reward values, indicating an improvement in performance in real-world environments. The improved REINFORCE has shown good learning performance in both simulated and real-world environments, achieving high average reward values with fewer data points and demonstrating good convergence and stability. The success rate of ADR-VN is evaluated in both simulated and actual environments, as listed in Table 2.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. NSR between simulated environment and real scenario.

https://doi.org/10.1371/journal.pone.0347775.t002

In Table 2, the improved REINFORCE achieves 92.5% NSR in simulated environments and 92.3% in real scenarios, with 2.1% path deviation rate and 93.5% navigation stability. The 7.7% of operation failures mainly stems from two situations: the failure of visual feature extraction under extreme lighting conditions and the lag in predicting sudden changes in dynamic obstacles. Among them, the former accounts for 62% and the latter accounts for 38%. The system is equipped with real-time fault detection and recovery mechanisms for such failure scenarios. It completes visual feature restoration through deep image distance information, predicts the movement trend of obstacles based on historical trajectories, and triggers temporary deceleration strategies. The average fault recovery time is 1.2 seconds, enabling a quick return to the normal navigation state. The success rates of REINFORCE in the two scenarios are 90.1% and 90.5%, with path deviation rate and navigation stability of 3.1% and 86.1%. The four indicators of DDPG are 81.7%, 81.2%, 2.8%, and 85.2%, while DQN is 81.2%, 88.4%, 3.5%, and 73.6%. The improved algorithm has high NSR in both simulated and real environments, demonstrating its superior performance in VN tasks. To comprehensively evaluate the performance in ADR-VN tasks, the Navigation Decision Time (NDT) in different scenes is analyzed, as exhibited in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. NDT between simulated environment and real scenario.

https://doi.org/10.1371/journal.pone.0347775.t003

In Table 3, the improved algorithm has a decision time of 0.85 seconds in simulated environments, 0.68 seconds in real scenarios, a response time of 0.45 seconds, and a path planning time of 0.35 seconds. Compared to simulated environments, the decision time is reduced by 20.6%. The four values of the REINFORCE algorithm are 1.0 s, 0.8 s, 0.50 s, and 0.40 s, resulting in a 20.0% reduction in decision time. The various indicator times of DDPG and DQN algorithms are (1.15 s, 0.86 s, 0.60 s, 0.25 s) and (1.05 s, 0.87 s, 0.50 s, 0.30 s), reducing decision time by 25.2% and 17.1%. The improved algorithm has demonstrated faster decision times in both simulated environments and real scenarios, with the shortest decision time, response time, and path planning time being relatively short in real scenarios. This demonstrates the algorithm’s good adaptability and efficiency in real scenarios. To further verify the performance of the improved REINFORCE algorithm in extreme scenarios, the study adds extreme weather and complex obstacle tests in the CARLA simulation environment. The extreme weather scenarios are heavy rain (visibility ≤ 50m) and thick fog (visibility ≤ 30m). In CARLA, the precise simulation of extreme weather is achieved by adjusting the WeatherParameter set. In the heavy rain scenario, the precipitation intensity is set to 100.0, the size of raindrops is 0.2, and the speed of raindrops is 20.0. At the same time, the atmospheric scattering effect is enabled to simulate the refraction and obstruction of rainlight. In the thick fog scenario, the fog concentration is set to 0.8, and the attenuation coefficient is 0.05. The global fog effect and local atmospheric perspective are combined to reduce the environmental visibility. The sensor noise is modeled separately for the visual camera and the depth camera. For the visual camera, Gaussian noise (SNR = 20dB) and salt-and-pepper noise (density = 0.03) are added. For the depth camera, distance-related Gaussian noise (error increases linearly with the detection distance, with a maximum error ≤ 0.3 m) is added. This simulates the noise interference of sensors in the real environment. The complex obstacle scenarios include dense static obstacles (obstacle spacing ≤ 1m) and dynamic obstacles randomly changing lanes. The results are shown in Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Comparison of Navigation Performance of various algorithms in Extreme Scenarios.

https://doi.org/10.1371/journal.pone.0347775.t004

In Table 4, the improved REINFORCE algorithm still maintains a high NSR in extreme scenarios: the success rate in the rainstorm scene is 88.2%, and it focuses on nearby obstacles and lane lines through image enhancement and AMs, reducing the interference of rainwater. The success rate in thick fog scenes is 85.5%. The distance information of depth images is utilized to assist in determining the position of obstacles, compensating for the insufficient visibility of visual images. The success rate in the dense static obstacle scene is 90.3%. Through the fine features extracted by the ResBlocks, the obstacle gaps are accurately identified, and the detour paths are planned. The success rate in the dynamic obstacle lane-changing scene is 89.7%. The movement trend of obstacles is analyzed through historical frames, and the navigation strategy is adjusted in advance. In addition, the average fault recovery time of the improved algorithm is 1.2 seconds. The fault recovery time refers to the time it takes for the robot to return to the normal navigation state after deviating from the path or approaching an obstacle. The improved algorithm achieves rapid recovery through rapid value assessment and strategy adjustment, which is much lower than that of other comparison algorithms, further verifying its robustness and practicality in complex and extreme scenarios. A special evaluation is conducted on the dynamic obstacle prediction method based on the optical flow algorithm adopted by the model. The results show that after enabling this method, the NSR in scenarios with dynamic obstacle changes increases by 9.2%, and the prediction error of the obstacle movement trend is controlled within 0.15 m. Compared with the basic model without enabling this method, the decision response in dynamic scenarios is 0.2 seconds earlier, effectively reducing the navigation failure caused by the lag in dynamic obstacle prediction.

To verify the reliability of the experimental results, a one-way analysis of variance (ANOVA) and post-hoc Tukey HSD test are conducted on the core performance indicators of each algorithm. The significance level is set at 0.05. The statistical analysis results are shown in Table 5.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Statistical significance analysis results of performance indicators for each algorithm.

https://doi.org/10.1371/journal.pone.0347775.t005

The results show that there are statistically significant differences between the improved REINFORCE algorithm and the comparison algorithm in all core indicators, confirming the reliability of its performance advantage rather than being accidental. To clarify the independent contributions of ACA, ResBlock, and AM, ablation experiments are conducted. The performance of different module combinations is shown in Table 6.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Performance comparison table of ablation experiments.

https://doi.org/10.1371/journal.pone.0347775.t006

Based on REINFORCE+ACA, merely introducing the residual structure can increase the NSR by 2.2%, the accuracy rate by 2.1%, and the path deviation rate by 0.9%. This verifies its core gain in extracting deep visual features. Introducing only the AM can increase the NSR by 0.8%, the accuracy rate by 1.4%, and the path deviation rate by 0.4%, demonstrating its additional effect of focusing on key semantic features. When the two are combined, performance is further enhanced. Compared with a single residual structure, the NSR increases by 2.0%, confirming that the deep feature extraction of residual structures complements the key feature selection of AM, rather than the independent effect of a single mechanism. To comprehensively verify the performance level of the improved algorithm in modern DRL methods, a comparative experiment is conducted by comparing it with PPO, SAC, and Transformer-DRL. The experimental environment and parameter settings are consistent with those in Table 1. The specific performance comparison results are shown in Table 7.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Performance comparison of improved algorithm and modern DRL methods.

https://doi.org/10.1371/journal.pone.0347775.t007

The comparison results show that the improved REINFORCE algorithm outperforms other methods in all key performance indicators: the success rates of navigation in the simulated environment and the real scene reach 92.5% and 92.3%, respectively, which are 2.3–4.0% higher than those of PPO, SAC, and Transformer-DRL methods. The decision time is only 0.85 seconds, significantly shorter than 1.18 seconds of Transformer-DRL, making it more suitable for embedded platform requirements in terms of real-time performance. The accuracy and recall rate remain leading at 95.7% and 91.2%, respectively, the path deviation rate is as low as 2.1%, and the navigation stability reaches 93.5%, demonstrating a more balanced VN ability in complex environments. Its performance advantage stems from the collaborative optimization of the ACA, ResBlocks, and AM, as well as the lightweight visual front-end design that balances real-time performance and accuracy. To comprehensively verify the algorithm’s robustness under practical disturbance scenarios, cross-validation tests are conducted on the embedded platform (NVIDIA Jetson Xavier NX), covering variable kinematic states of the robot and sensor noise interference. The kinematic states include different speed ranges (0.5-2m/s, 2-4m/s) and motion modes (straight-line, continuous turning, start-stop alternating). Sensor noise includes Gaussian noise (SNR = 20dB/30dB) and salt-and-pepper noise (density = 0.01/0.03) added to visual input. The robustness is evaluated by averaging core performance indicators and resource consumption across all test scenarios. The test results are shown in Table 8 below.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Cross-validation comparison of algorithm robustness and resource consumption.

https://doi.org/10.1371/journal.pone.0347775.t008

As shown in Table 8, the improved REINFORCE algorithm demonstrates extremely high robustness in cross-validation tests: its multi-scenario average NSR reaches 91.7%, which is 3.5% higher than that of the original REINFORCE algorithm, 1% higher than DDPG, 2% higher than Transformer-DRL, and maintains high reliability even under changing motion states and sensor noise interference. In terms of real-time performance stability, the average inference delay across multiple scenarios is only 18.9 ms, with a fluctuation range of less than 0.6 ms for different scenarios, and the average computing load remains at 43.2%, indicating that the performance degradation caused by environmental and state changes is minimal. The stable memory usage remains at 48.5MB, and the average path deviation rate is controlled at 2.3%, indicating its strong resistance to sensor noise and adaptability to various motion modes. This cross-validation result confirms that this algorithm not only achieves a balance between performance and resource consumption under ideal conditions, but also remains stable and reliable in the face of common practical disturbances such as variable motion states and sensor noise, fully meeting the robustness requirements of ADR-VN in complex practical applications.

To quantitatively verify the computational efficiency and cross-platform deployment adaptability of the lightweight design of the improved REINFORCE algorithm, three mainstream embedded platforms in the current ADR-VN field are selected for the actual test (NVIDIA Jetson Nano, NVIDIA Jetson TX2, NVIDIA Jetson Xavier NX). The core computational efficiency and resource occupation indicators of the algorithm under each platform are simultaneously collected. All indicators are the average values obtained from multiple scenarios of testing. The actual test data are shown in Table 9.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Comparison of algorithm deployment and computational efficiency on mainstream embedded platforms.

https://doi.org/10.1371/journal.pone.0347775.t009

In Table 9, the improved REINFORCE algorithm, with its lightweight design including deep separable convolution, achieves full-dimensional optimization on three embedded platforms: memory footprint ranges from 45.2 to 48.5MB, a reduction of 39.9% to 42.5% compared to traditional algorithms; peak GPU utilization drops by 17.7% to 26.2%, eliminating the problem of resource overload; inference latency reduces by 44.6% to 49.7%, with the entire platform’s latency under 60 ms, meeting real-time requirements; average power consumption decreases by 30.1% to 30.7%, all controlled within 16W, without any heat dissipation pressure; compared with traditional REINFORCE, the temperature is reduced by 14–16°C, demonstrating the thermal performance advantages of the lightweight design. The improved REINFORCE algorithm is adapted to different embedded platforms with varying computing capabilities, achieving stable optimization results and a dual improvement in computational efficiency and hardware compatibility. To verify the performance of the algorithm in an unordered environment, such as urban streets where there are unpredictable pedestrians and vehicles, a city street scene with randomly crossing pedestrians, lane-changing vehicles, and intersections for cross-traffic is constructed in CARLA. Navigation performance tests are conducted for each algorithm, and the results are shown in Table 10.

Download:

• PNG
  larger image
• TIFF
  original image

Table 10. Comparison of navigation performance of various algorithms in chaotic urban street environments.

https://doi.org/10.1371/journal.pone.0347775.t010

In Table 10, the improved REINFORCE algorithm still maintains the best navigation performance in the disordered urban street environment. Due to the random behaviors of dynamic targets and the cross-interference among multiple entities, the performance of each algorithm has decreased compared to the controlled environment. However, the NSR, decision-making speed, and fault recovery ability of this algorithm are still significantly superior to those of the other comparison algorithms.