Implementation for ASTS.

ASTS is described in the form of tuples, a set of norms N, state S, and event E. (1)

• Norm N. N = {r₁, r₂, …} represents a set of norms, which are the mapping relationships between predefined conditions and agent behavior and are the key to the effective operation of ASTS. Norm r_i is expressed as r_i = {Tri_i, Act_i, Exp_i}.
  • Tri_i is the trigger condition and is used to determine if the associated actions can be performed. Tri_i consists of a series of logical judgment, which can be expressed as .
  • Act_i is the execution action. Act_i consists of a series of methods or actions, which can be expressed as .
  • Exp_i is the expected result to verify the result of Act_i. Exp_i consists of a series of expected conditions, which can be expressed as .
• State S. The data are read by State S. State S represents a set of instantaneous states denoted by S = {s₁, s₂, …}, which is arranged in chronological order. The instantaneous state s_i consists of a series of expressions, which are expressed as .
• Event E. E = {e₁, e₂, …, e_i} represents a set of instantaneous events sorted in chronological order. The instantaneous event e_i consists of a series of sub-events, which are expressed as . When an event e_i occurs, the instantaneous state s_j is read to determine the Tri_k of r_k.

Operation process of ASTS.

Definition 1. Norm, Event, and State. Eq (1) describes the components of ASTS. For example, a logistics truck named Tom starts work at 1o o’clock every day. Tom must follow the norms listed in the Table 1. According to norms r_a and r_b, Tom maintains a safe distance W from the object in front of it (set at 0.5m). If W is greater than 0.5 m and the power is greater than 30% when e₁ occurs, then Tom moves forward. To verify the Act_a result of the norm r_a, we expect W to be equal to 0.5 m. If W is less than 0.5 m and the power is less than 30% when e₁ occurs, Tom stops. The instantaneous events and states are listed in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Example of norm.

https://doi.org/10.1371/journal.pone.0287791.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Recording of instantaneous events and states.

https://doi.org/10.1371/journal.pone.0287791.t002

Definition 2. Triggered norm. When e_i occurs, the instantaneous state s_i is read. If all logical judgment expressions in Tri_x are satisfied, then Norm r_x is denoted as the ‘triggered norm’ at instantaneous state s_i. For example, an instantaneous event e₁ occurs at 9:00. Tri_a is satisfied at the instantaneous state s₁; then Tom starts moving forward. Norm r_a is denoted as the ‘triggered norm’.

Definition 3. State transition. When e_i occurs, norm r_x is triggered at s_i, and Act_x is executed. This process causes the instantaneous state to change from s_i to s_j, which can be expressed as: (2) Where the notation ‘→’ denotes an instantaneous state changed from s_i to s_j as a result of triggering r_x. If the instantaneous state changed from s₁ to s₂ as a result of triggering r_a. The instantaneous state changes from s₂ to s₃ as a result of triggering r_a. These processes are expressed in Formula (3). (3)

Definition 4. Strategy. Norms are the key to the effective operation of multiple agent systems and are a predefined mapping relationship between conditions and behaviors. A strategy is a plan of action based on the current environment and goals. It is used to guide the agent’s decisions and behaviors in a specific domain and helps the agent achieve its goals in that domain. A strategy consists of a set of action plans, and norms can be used as part of developing a strategy. For example, Norms r_a and r_b form a goal-following strategy in Table 1. Norm r_c forms an obstacle-avoidance strategy.

Definition 5. Switch norms or strategies. When e_i occurs, norm r_x is triggered at s_i. Then, e_j occurs and r_y is triggered at s_j. The norm switches from r_x to r_y. Multiple switching norms accumulate when one norm is switched to another. For example, when e₁ occurs, Norm r_a is triggered at s₁. Then, e₂ occurs, and the norm r_a is triggered at s₂. Finally, e₃ occurs, and the norm r_b is triggered at s₃. The norm switches from r_a to r_a and eventually to r_b.

Based on the above description, this study summarises the pseudocode of ASTS as follows:

1. i. Input N, S, E, T;
2. ii. Initialisation: Path = ϕ, Res_Act = ϕ, Res_Exp = ϕ, i = 0, j = 0, k = 0.
3. iii. If an instantaneous event e_i ∈ E occurs, the ASTS data are read by the transient state s_j ∈ S.
4. iv. Judge Tri_k of Norm N.
5. v. If Tri_k = True, Norm r_k is triggered; perform Act_k and judge Exp_k.
6. vi. The results for Act_k and Exp_k are restored as Res_Act and Res_Exp, respectively.
7. vii. s_i, e_j, r_k, Res_Act, Res_Exp denote the restored paths.
8. viii. j + +, i + +.
9. ix. Until i > T, exit ASTS.

Components of FERD system.

The FERD system system is based on the framework of ASTS. The components include a set of tasks M, a set of fire equipment A, a set of sensors Q, and a set of agent norms N′. FERD system can be expressed as: (4)

• Task M. The FERD system is applied to handle fire extinguishing tasks, Task M = {m₁, m₂, …}. Each task m_i comprises a series of attributes denoted as .
• Agent A. A certain number of agents is needed to achieve the ultimate goal, which can be expressed as A = {a₁, a₂, …}. a_i denotes the i-th fire equipment. a_i = {u_i1, u_i2, …, u_ik} denotes the attribute of a_i. u_ik denotes the k-th attribute of a_i.
• Sensor Q. Each agent carries sensors to sense the environment. Sensor Q = {q₁(τ₁), q₂(τ₂), …}, and q_i denotes the i-th sensor. The sampling periods of multiple sensors are different. τ_i is the sampling period of sensor q_i. The sensor q_i can obtain real values and observed values of the attribute u_jk of the fire equipment a_j during each sampling period τ_i. It can be expressed as . denotes real values of sensor q_i for measure attribute u_jk of fire equipment a_j during the first sampling period τ_i1. denotes observed values of sensor q_i for measure attribute u_jk of fire equipment a_j during the first sampling period τ_i1. These sensors have different sampling periods. At each time step , observed value of each sensor is obtained in FERD system. This process is shown in Fig 1.
• Agent Norm N′. The norm set N′ for the agent, which is denoted by . Where norm is expressed as . The internal elements of , , and the internal elements of Tri_k, Act_k, Exp_k are consistent.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Time alignment in FERD system.

The time alignment process of n sensors.

https://doi.org/10.1371/journal.pone.0287791.g001

Objectives of FERD system.

The objectives of FERD system are expressed as Eq (5). Minimize the maximum time for FERD system rescue to complete tasks. And all tasks in M are completed. (5)

Constituent of FERD system.

The FERD system operates based on the GBBopen platform. In a FERD system, a set of multiple KSs make up a strategy. The CS selects the appropriate KS for the multiple agents to switch. After the strategy is executed, the data on the BB are changed until the fire extinguishing tasks are completed. Certain properties of the agent are obtained from external sensors. The components of the FERD system are described in detail below.

1. (1) Blackboard. In the FERD system, a task and a type of agent are defined as classes, respectively. Each agent is defined as an instance. The attribute value of each agent is either directly or indirectly defined.
2. (2) Knowledge Source. KS is a core component of the FERD system. In GBBopen, the attributes of KS are in correspondence with the Agent Norm N′, and the properties of the KS are extended, ‘postcondition-function’. The relationship between N′ and KS can be formally described as follows: and ks_k.pre, , . KSs are divided into two submodules: the Switch Module and Strategy Module.
   1. 1) Classification of Switch Module. The agent can switch between the strategies based on multiple switch modules. The control modes are divided into Automatic Mode and Command Mode. Multiple agents can switch strategies based on their own identification or environmental identification, or based on user requirements or external guidance.
   2. 2) Classification of Strategy Module. Different strategies are generated from the agent state and the collected environmental information. These mainly include the Following Strategy, Motor Strategy, Evolution Strategy, Obstacle Avoidance Strategy, and Mobilization Strategy.
      1. a. Movement Strategy. The movement strategies are divided into Forward, Stop, and Fallback.
      2. b. Following Strategy.Trajectory Following and Goal Following are developed for agents to arrive at fire scenarios rapidly.
      3. c. Mobilization Strategy. In different scenarios, multiple agents can select Formation, Leave Team, or Return To Team.
      4. d. Obstacle Avoidance Strategy. Multiple agents can select Virtual Circle, Virtual Right Triangle, and Virtual Isosceles Triangle.
      5. e. Evolutionary Strategy. The FERD system provides two evolutionary strategies for autonomously evolving norms: Crossover and Other Method. In Crossover, the expectation of one mutated norm enrich the trigger of another mutated norm.
3. (3) Control Shell. In the FERD system, after ksa_k.exe is executed, ksa_k.post is executed. The CS triggers various norms based on event changes to achieve fast switching strategies.

Denoising.

Interference values exist for every sensor. Interference values are values that describes the scenario inaccurately. In this paper, interference values are collectively referred to as random errors. Then the observed values contains a large number of random errors. The random errors not only increase the computer memory and computation overhead but also increase the computation error. So observed values of each sensor need denoising. The purpose of denoising for sensor q_i is to reduce the Standard Deviation σ_i.

Spatial alignment.

The observed value of each sensor in the same platform using different coordinate systems. The observed value of different sensors involve different coordinate systems. Therefore, observed value must be converted into data in the same coordinate system before fusion. In fact, this process is to calculate the transformation relationship between coordinate systems. The system converts the attribute u_jk of the fire equipment a_j into a uniform type of sampled data from sensor q_i by Spatially Aligning KS. In this paper, the transformation matrix of sensor q_i is set to trans_i. The observation for each sampling period after spatial alignment is . The real value of the sampling period after spatial alignment is .

Confidence degree update KS.

The contribution of each sensor to data fusion is also different. Therefore, this paper proposes the concept of confidence degree. The fusion contribution of each sensor is defined by the confidence degree. For example, sensor q_m and q_n are both sensors that make observations of u_jk. In each sampling period, their confidence degrees are updated by this KS. Their confidence degrees are denoted by w_mg and w_ng, respectively. Where .

Observation filtering.

The error of observed values from each sensor is different. Therefore, the concept of error threshold is proposed in this paper. It serves as a precondition for fusion. The error of observed values of sensor q_i on u_jk in each sampling period is expressed as follows. (9)

For example, sensor q_m and q_n measure for u_jk. Then, this paper calculates the error between observed values and real values. Their errors are denoted as ε_mg and ε_ng, respectively. Select their error thresholds by this KS. Their error thresholds are denoted as ε_m and ε_n, respectively. Finally, observed values of two sensors in each sampling period are compared with the corresponding error thresholds separately. The filtering operation is performed mainly for Data Fusion KS.

Data fusion.

This KS is a process of multi-level, multi-spatial information complement and optimal combination processing by multiple sensors. In this process, multiple sources of data are fully utilized for rationalization and use. This not only takes advantage of multiple sensors operating in concert with each other but also integrates data from other information sources. This improves the intelligence of the whole system, as well as the accuracy and comprehensiveness of the information, and reduces the uncertainty of the information. This KS combines observations from multiple sensors into one data. This operation makes the measured values more accurate.

This paper assumes that there are Γ different fusion algorithms in this module. The observed values of sensor q_m and q_n for the same attributes (u_jk) are fused. At this point, the observed values from both sensors have been denoised and spatially aligned. During each sampling period , FERD system can obtain observed values of all sensors. ∃ε_mg ≤ ε_m. Where .

By the fusion algorithm γ ∈ Γ, the result of Data Fusion KS during each sampling period can be expressed as . Then, (10)

Among them, the symbol ‘⦾’ is a symbol for data fusion, which indicates that two observed values are fused. The real values of the same property for the same object is the same for different sensors. The real values of the same property of the same object in each sampling period can be expressed as .

Fusion decision.

For the results of different data fusion algorithms, the role of this KS is to select the appropriate fusion algorithm γ. This algorithm is a fusion algorithm that is the closest to the real values. That is: (11) Where, j = 1, 2, …, x; k = 1, 2, …, y′; γ ∈ Γ.

Experiment scenario.

This experimental sets the experimental scenario as a rectangular environment, as shown in Fig 3. This scenario is a rectangular base station laid by four UWB anchors located at the same horizontal position. One of anchors is used as the origin of rectangular.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Experiment scenario.

The size of this experiment scenario is 15m × 2.4m. Multiple obstacles are also placed within this experiment scenario.

https://doi.org/10.1371/journal.pone.0287791.g003

Experiment platform and equipment.

The experiments in this paper are conducted By taking the Chengdu Kestrel Artificial Intelligence Institute’s Interface Description Language(IDL)-Mapping tool (KIS-CORBA) as a cross-language data communication protocol, acquiring real-time information with multiple sensors, and publishing or subscribing to the information through the serial port of the Robot Operating System (ROS) to finally implement GBBopen to control firefighting equipment. This IDL is the IDL of Object Management Group (OMG).

1) Experiment Platform. All experiments require a PC platform and an ARM-embedded platform. The ARM-embedded platform used in this experiment is the Raspberry Pi (RPI). The Python environment (3.7.0), KIS-CORBA, and ROS platforms are installed into the RPI. The Visual Studio environment (VS2017), KIS-CORBA, Lisp environment (Allegro Common Lisp 10.1 express), LinkTrack technology (NoopLoop for NAssistant applications), and GBBopen are installed on the PC.

2) Experiment Equipment. This experiment uses an unmanned vehicle (RoboMaster) as the firefighting equipment for the FERD system, as shown in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Firefighting equipment.

An unmanned vehicle (RoboMaster).

https://doi.org/10.1371/journal.pone.0287791.g004

i. Sensors. Each piece of unmanned vehicle carried five items, including an IDS, a camera, an UWB anchor, a module of an IMU, and a RPI, as shown in Fig 4.

ii. Application Programming Interface (API). DJI-Innovations (DJI) provides several API instances (https://robomaster-dev.readthedocs.io/zh_CN/latest/) written in Python. This experiment is conducted to realize communication between the unmanned vehicle and GBBopen and to control the wheels of the unmanned vehicle. Because unmanned vehicle is written in Python and GBBopen is written in Lisp, cross-language communication is required. These experiments focused on the communication between Python and Lisp and finally applied the FERD system.

iii. Attributes. Various attributes exist for each firefighting task and each unmanned vehicle. These attributes can be expressed as follows: (12) (13)

iv. Data transmission. The remote server is connected to control the unmanned vehicle through Secure Shell (SSH). The position of the unmanned vehicle, the three-axis attitude angle, and the distance from the front are obtained through UWB, IMU, and IDS, respectively. Among them, UWB achieves high-precision indoor positioning by sending ultra-short pulse signals that exploit multipath propagation and the Doppler effect in indoor environments. The unmanned vehicle obtains images through the camera’s API and embeds a coordinate transformation program to output coordinates. The ROS serial port then reads the obtained data and publishes it in the form of topics. Finally, they are transmitted to GBBopen through KIS-CORBA. The data of the unmanned vehicle is sent to GBBopen, including pose, velocity, distance, etc. GBBopen sends the data to the unmanned vehicle, such as target point, speed, etc.

v. Pose information. Positioning errors occur due to various factors (e.g., the deployment mode of the base station, obstracles and the weakening of the signal) in the indoor environment. Therefore, the experiment assumes that the information measured by the UWB and IMU is the actual location information of the unmanned vehicle. In Fig 3, the number of unmanned vehicles is 2. The position of two unmanned vehicles are (10.8, 1.8) and (15,0.6), respectively. And the position of the firefighting task is (4.2, 0.6).

Experiment contents.

In FERD system, the fusion norms are designed, which guide each unmanned vehicle to perform tasks and the fusion positioning of multiple sensors. Briefly introduce the 22 norms that already exist within FERD system. Norm r₁ creates multiple instances of the unmanned vehicle. Norm r₂ and r₃ are rules for setting the switching ways and sorting the fire locations. The switching ways include Command Mode and Automatic Mode. Norm r₄ ∼ r₈ are rules about the following strategy. Among them, Norm r₄ and r₅ are the ways to determine the following of the unmanned vehicle a_i. Norms r₇ and r₈ are parameters to set the unmanned vehicle a_i following mode. Norm r₉ ∼ r₁₃ are rules about the Mobilization Strategy. These strategies include Formation, Departure, and Return To the Team. Norm r₁₂ and r₁₃ are rules for setting the role of the unmanned vehicle a_i. Norm r₁₄ ∼ r₁₆ are three rules about the Motion Strategy. Here it is recommended that the linear velocity of the unmanned vehicle is between 0.05 ∼ 0.5m/s. The angular velocity of the unmanned vehicle is between 0 ∼ 90rad/s. Norm r₁₇ ∼ r₁₉ are the three rules of the Obstacle Avoidance Strategy. Norm r₂₀ ∼ r₂₂ are rules about Evolutionary Strategy. Table 4 shows the added norms and revised norms within FERD system in this paper. Norm r₁₄ ∼ r₁₆ are rules about three revisions of the Motion Strategy. Norm r₂₃ ∼ r₃₀ are rules about the Following Strategy. Norm r₃₁ ∼ r₃₈ are rules about updating confidence degree and filtering of observations. Norm r₃₉ ∼ r₄₀ are rules about different algorithms about the fusion of multiple sensors. For example, the fusion algorithm of the Kalman Filter based confidence degree and the weighted average-based fusion algorithm are applied in this paper. Norm r₄₁ is the rule for fusion decision-making. This norm is that the fusion results from different algorithms are compared with gt_j. The optimal fusion result is selected.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Norms of fusion moudle.

https://doi.org/10.1371/journal.pone.0287791.t004

In indoor scenarios with multiple obstacles, each unmanned vehicle chooses the appropriate norms based on its state and assists in firefighting tasks through the sensors it carries itself. This experiment sets a fixed route for two unmanned vehicles. They pass through (4.8,1.8), (4.8,0.6), and (4.2,0.6) in turn. Eventually reaching the fire location (4.2, 1.2), they perform the fire extinguishing task.

Experiment limitations.

The experiment design can verify the effectiveness of the proposed module, but this module has certain limitations (e.g., timeliness). In this experiment, the relative velocity between two unmanned vehicles is set to check the real-time performance. Specifically, the relative velocity of two unmanned vehicles is used as an independent variable. The fusion results of multiple sensors are used as the dependent variable. Finally, the fusion results of multiple sensors are compared with the real values. This experiment sets different relative velocities between two unmanned vehicles to perform the fire extinguishing task. The relative velocities in this experiment are 0.03m/s, 0.06m/s, 0.09m/s, 0.12m/s, 0.15m/s, and 0.2m/s, respectively.

Evaluation metrics.

In this experiment, the FER is proposed as the fusion index of this fusion module. GBBopen receives the number of localization coordinates of all sensors simultaneously, expressed by Z. Among them, the distance between the coordinates localized by UWB and the ground truth is large; while the distance between the coordinates localized by other sensors and the ground truth is small. The distance between the coordinates localized by other sensors and the ground truth is small. The number of coordinates of UWB localization is larger than the distance to the ground truth, denoted by z. The FER is expressed as × 100%. With effectively fused data, one evaluation index is the error range of ED and MSE between observed values and the ground truth of a single sensor. Another evaluation index is the error range of ED and MSE between the fusion results of multiple sensors and their ground truth.

Observed values of different sensors in different scenarios.

This step analyzes the different sensors in different scenarios. And the observed values are compared with the real values.

1) UWB Positioning. A unmanned vehicle a₁ carries a UWB anchor to position itself. a₁ starts with the real position (7.8,1.8). a₁ is moved 10 times from right to left at 600mm intervals in sequence. a₁ ends at the real position (2.4,1.82). The real positions of a₁ are (7.8,1.8), (7.2,1.8), (6.6,1.8), (6,1.8), (5.4,1.8), (5.4,1.8) (4.8,1.8), (4.2,1.8), (3.6,1.8), (3,1.78), (2.4,1.82). Finally, the values of the positioning are compared with the real values.

2) IDS and Camera Ranging. First, the mapping relationship between world coordinates and pixel coordinates is obtained by Formula 6. The internal and external reference matrices of the camera are obtained by camera calibration. Then, a₁ carries Camera and IDS to range unmanned vehicle a₂. a₂ moves 10 times in sequence at 600mm intervals from 600∼6000mm. The distances between a₁ and a₂ are 600mm, 1200mm, 1800 mm, 2400mm, 3000mm, 3600mm, 4200mm, 4800mm, 5400mm, and 6000mm respectively. Finally, the values of the ranging are compared with the real values.

3) UWB, IDS and Camera Positioning. a₁ carries Camera and IDS to range a₂. Then, two ranging results are fused. Next, a₁ was positioned using both UWB and Camera. a₂ robot positioning starts with the real position (7.8,1.8). a₂ is moved 10 times from right to left at 600mm intervals in sequence. a₂ ends at the real position (2.4,1.82). The real positions of a₂ are (7.8,1.8), (7.2,1.8), (6.6,1.8), (6,1.8), (5.4,1.8), (5.4,1.8) (4.8,1.8), (4.2,1.8), (3.6,1.8), (3,1.78), (2.4,1.82). Finally, the values of the positioning are compared with the real values.

Train error thresholds and confidence degrees.

The accuracy and error of the observed values of each sensor are different. Each sensor also contributes differently to the fusion of multiple sensors. Therefore, some parameters of some norms are trained. They include the confidence degree of each sensor and the error threshold of all sensors. In this experiment, the number of training is 10. In the initial stage of training, the confidence degree of each sensor is equal. And the error threshold of all sensors is set to null. The results at the end of the first training are the initial values for the second training. Until the end of the tenth training session, the trained results are the initial values of the unmanned vehicles performing tasks.

Performing tasks.

The two unmanned vehicles follow a set fixed route and assist in firefighting tasks through the parameters of the norms that have been trained and the sensors that each carries.

Observed values of different sensors in different scenarios.

The results of the observed values and true values are analyzed by different perspectives (quantitative and qualitative) in different scenarios.

1) UWB Positioning. Figs 5 and 6 show the results of the observed values and real values of unmanned vehicle a₁ analyzed by qualitative. Horizontal axis indicates the length of the rectangle. Vertical axis indicates the width of the rectangle. The positioning of a₁ are compared 10 times in order from right to left. The red line indicates the real values of a₁. The purple line indicates the observed value of UWB. Table 5 shows the results of the observed values and true values of unmanned vehicle a₁ analyzed by quantification. Vertical indicates the number of experiments. Horizontal indicates the data analysis from different scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. In indoor scenario without no obstacle.

Comparison of UWB position and real values in different scenarios.

https://doi.org/10.1371/journal.pone.0287791.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. In indoor scenario with multiple obstacles.

Comparison of UWB position and real values in different scenarios.

https://doi.org/10.1371/journal.pone.0287791.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Data analysis of between observed values of UWB position and real values.

https://doi.org/10.1371/journal.pone.0287791.t005

In Fig 5, the red line is smoother, while the purple line has fluctuations. So, there is instability in the observed values of UWB. In Fig 6, the red line is smoother. the purple line fluctuates more. So, there is instability in the observed values of UWB. And it is less stable than in an indoor scenario without obstacles. In Table 5, the quantitative results are analyzed as follows. In an indoor scenario without obstacles, the error range on the x-axis, y-axis, and European distance is -0.34∼-0.08 m, -0.15∼0.53 m, 0.01∼0.29 m. The SD on the x-axis and y-axis ranges from 0.0∼0.03 m, and 0.03∼0.10 m, respectively. The range of MSE on the x-axis and y-axis is 0.01∼0.12m² and 0.01∼0.28m², respectively. In an indoor scenario with multiple obstacles, the error range on the x-axis, y-axis, and European distance is -0.49∼-0.12 m, -2.35∼0.71 m, 0.05∼5.62 m. The SD on the x-axis and y-axis ranges from 0.01∼0.13 m, and 0.06∼0.90 m, respectively. The range of MSE on the x-axis and y-axis is 0.01∼0.24m² and 0.02∼6.34m², respectively. This quantitative result illustrates that the observed values of UWB fluctuate more on the y-axis than on the x-axis. The UWB observations fluctuate more in an indoor scenario with multiple obstacles than without obstacles. This is due to the fact that the accuracy of the short edge (y-axis) is worse than the long edge (x-axis) in the unobstructed case. The impact on the ranging accuracy in an indoor scenario with multiple obstacles and the impact of the ranging algorithm can cause the short edge to be affected more than the long edge.

2) IDS and Camera Ranging. Figs 7 and 8 show the results of the observed values and real values analyzed by qualitative. The real values are distance between a₁ and a₂. The observed values are a₁ ranging a₂ through the carried IDS and the Camera. Horizontal axis indicates the number of experiments. Vertical axis indicates the observed values of ranging. The red line indicates real values. The purple line indicates that observed values are a₁ ranging a₂ through the carried IDS. The green line indicates that observed values are a₁ ranging a₂ through the carried the Camera. Table 6 shows the results of the observed values and real values analyzed by quantification. Vertical indicates the real values. Horizontal indicates the error between observed values and real values by different sensors in different scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. In indoor scenario without no obstacle.

Comparison of observed values and real values of ranging by different sensors in different scenarios.

https://doi.org/10.1371/journal.pone.0287791.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. In indoor scenario with multiple obstacles.

Comparison of observed values and real values of ranging by different sensors in different scenarios.

https://doi.org/10.1371/journal.pone.0287791.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Data analysis of ranging in different sensors.

https://doi.org/10.1371/journal.pone.0287791.t006

In Figs 7 and 8, there is no data for ranging by IDS in 4800∼6000mm, while the data is available for ranging by Camera. The experimental results show that ranging by IDS does not work, while it works by Camera in 4800∼6000mm. The experimental results show that observed values of ranging by IDS are smaller than the real values. Table 6 is analyzed from the quantitative perspective as follows. In an indoor scenario without obstacles, the error range between ranging by IDS and real values is 134.86∼420.35mm. The SD of these errors is 111.70. The range of error between ranging by Camera and real values is -1334.713∼489.13mm. The SD of these errors is 489.13. The SD of the Camera is 4.38 times higher than the IDS. In an indoor scenario with multiple obstacles, the error range between ranging by IDS and real values is 115.74∼647.74. The SD of these errors is 202.34. The range of error between ranging Camera and real values is -1154.43∼268.55mm. The SD of these errors is 529.73. The SD of ranging by Camera is 2.62 times higher than ranging by IDS. These results show that ranging by IDS and Camera has almost no effect in different scenarios. However, ranging by IDS is more stable than ranging by Camera. This is due to the detection of the target by a Camera that has errors (i.e., errors in the pixel width), resulting in errors in the numerator of Formula (7).

3) UWB, IDS and Camera Positioning. Figs 9 and 10 show the results of the observed values and real values analyzed by qualitative. Horizontal axis indicates the length of the rectangle. Vertical axis indicates the width of the rectangle. The unmanned vehicle a₂ is compared ten times in order from right to left. The red line indicates the real values of a₂. The pink line indicates the observed values of a₁ by UWB to position itself. The purple line indicates that a₁ by IDS to range a₂. At the same time, the observed values of a₁ by Camera to position a₂. The green line indicates the monocular range results of a₁ by Camera to a₂. Also, the observed values of a₁ by Camera to position a₂.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. In indoor scenario without no obstacle.

Comparison of observed values and real values in different scenarios.

https://doi.org/10.1371/journal.pone.0287791.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. In indoor scenario with multiple obstacles.

Comparison of observed values and real values in different scenarios.

https://doi.org/10.1371/journal.pone.0287791.g010

Tables 7 and 8 shows the data analysis between the observed values and real values by different sensors in different scenarios. In these table, vertical represents real values, horizontal indicates the error between observed values and real values by different sensors.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Data analysis of observed values and real values by different sensors in indoor scenario without obstacles.

https://doi.org/10.1371/journal.pone.0287791.t007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Data analysis of observed values and real values by different sensors in indoor scenario with multiple obstacles.

https://doi.org/10.1371/journal.pone.0287791.t008

In Figs 9 and 10, the trend of red line is flat, while the trends of pink, purple, and green lines are fluctuation. The trend of pink line are greater than purple and green lines. When the real coordinates of a₂ (8.4,1.8) and a₁ are (3.6,1.8), (3,1.78) and (2.4,1.82) in that order, there is no data for ranging by IDS. So no observed values are Camera position (IDS ranging). While the data is available for ranging by Camera, so observed values are Camera position (Camera ranging). In Tables 7 and 8, this experiment compares the observed values of a₁ by different sensors with the real values. In indoor scenario without obstacles, the error ranges of a₁ by UWB position itself on the x-axis, y-axis, and ED are -0.34∼-0.08 m, -0.15∼0.53 m, and 0.01∼0.29 m, respectively. The range error of a₁ by Camera position (IDS ranging) position a₂ on the x-axis, y-axis, and ED is -0.54∼-0.27 m, -0.21∼-0.03 m, 0.07∼0.33 m, respectively. The range error of a₁ by Camera position (Camera ranging) position a₂ on the x-axis, y-axis, and ED is -0.29∼1.17 m, -0.21∼-0.04 m, 0.02∼1.38 m, respectively. In indoor scenario with multiple obstacles, The range error a₁ by UWB position itself on the x-axis, y-axis, and ED is -0.49∼-0.12 m, -2.35∼0.71 m, 0.05∼5.62 m, respectively. The range error of a₁ by Camera position (IDS ranging) position a₂ on the x-axis, y-axis, and ED is -0.68∼-0.12 m, 0.21∼0.45 m, 0.13∼0.53 m, respectively. The range error of a₁ by Camera position (Camera ranging) position a₂ on the x-axis, y-axis, and ED is -0.36∼1.15 m, 0.18∼0.45 m, 0.09∼1.35 m, respectively.

From the above qualitative and quantitative analysis results, it is clear that the stability of UWB observations varies from scenario to scenario. This experiment allows the IDS and Camera to complement or fuse each other for ranging the unmanned vehicle. When the Camera (IDS ranging) measurement fails, Camera position (Camera ranging) and UWB can complement each other.

Train error thresholds and confidence degrees.

Figs 11 and 12 show the change curve of training confidence degree and error threshold.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Training confidence.

Training variation curves of confidence degree. The different color lines show the change training trend of different sensors’ confidence degrees. The pink line shows the change training trend of UWB positioning. The purple line shows the change training trend of Camera position (IDS ranging). The green line shows the change training trend of Camera position (Camera ranging). The yellow line indicates the change training trend of Camera position (Camera and IDS fusion ranging).

https://doi.org/10.1371/journal.pone.0287791.g011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Training error threshold.

Training variation curves of error thresholds.

https://doi.org/10.1371/journal.pone.0287791.g012

In Fig 11, the four different channels have the same confidence degree of 0.25 in the initial stage of training. The confidence degrees of the four different channels gradually differ as the number of training sessions increases. The confidence degree of the Camera position (IDS ranging) shows a fluctuating trend of rapid rise. The confidence degree of UWB’s position shows a fluctuating trend of slow rise. The confidence degree of the Camera position (Camera ranging) shows a slow downward trend. The confidence degree of the Camera position (Camera and IDS fusion ranging) shows a rapid downward trend. After training, their confidence degrees are as follows. Camera position (IDS ranging) has the highest confidence degree of 0.52. This is followed by the confidence degree of UWB position with 0.36. The confidence degree of Camera position (Camera and IDS fusion ranging) was 0.12. The lowest confidence degree of Camera position (Camera ranging) was 0.04. Fig 12 indicates the training trend of the error threshold for all sensors. At the initial stage of training, the error threshold of all sensors is 0 m. In the first training, the training error threshold of all sensors fluctuates greatly. The maximum is 0.50m, and the minimum is 0.35m. With the increase in training times, the fluctuation amplitude of all sensor error thresholds tends to be stable. The stable value reaches 0.47m. After training, the error threshold of all sensors reaches 0.46m.

Timeliness.

Table 9 shows the timeliness results of the fusion module. The table indicates the relative speeds of the two unmanned vehicles in the horizontal direction. The longitudinal direction indicates whether the position after fusion is lagged and the lag value. It is obvious from the table that there is no lag in the fusion results of multiple sensors when the relative speeds between a₁ and a₂ are set at 0.03m/s, 0.06m/s, 0.09m/s, and 0.12m/s. While the relative speed between a₁ and a₂ is 0.15m/s and 0.2m/s, the fusion results of multiple sensors show lag. At a relative velocity of 0.15m/s, the fusion results differ from the real values by 0.3m. At a relative velocity of 0.2m/s, the fusion results differ from the real values by 2m. From the analysis in the table, it is clear that the relative velocity is within 0.12m/s to ensure the validity of the fusion results of multiple sensors.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Results of timeliness.

https://doi.org/10.1371/journal.pone.0287791.t009

Performing tasks.

Figs 13–15 shows the visualization results of the FERD system performing task. a₁ and a₂ perform firefighting tasks in an indoor scenario with multiple obstacles. Horizontal axis indicates the length of the rectangle. Vertical axis indicates the width of the rectangle.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Real trajectory of a₁ and a₂.

This figure shows the real trajectory of a₁ and a₂ are perform a task.

https://doi.org/10.1371/journal.pone.0287791.g013

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Positioning trajectory of a₁ and a₂ by UWB.

The red and blue lines indicate the positioning trajectory of a₁ and a₂, respectively. The red line is available at https://doi.org/10.5061/dryad.31zcrjdrt (DOI: 10.5061/dryad.31zcrjdrt).

https://doi.org/10.1371/journal.pone.0287791.g014

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Comparison between UWB positioning and fusion positioning of a₁.

https://doi.org/10.1371/journal.pone.0287791.g015

In Fig 13, a₁ follow a fixed route. It starts at (10.8,1.8) and passes through J₁, J₂ and J₃ in sequence. Finally, it reaches fire T (4.2,1.2) to extinguish the fire. In fact, it is unavoidable that a₁’s wheels have a slight deviation when moving forward. As a₁ moves forward, simultaneously and quickly adjust its deviation, so that a₁ does not differ too much from this route. It is equivalent to a₁ moving along a track. Therefore, the real trajectory of a₁ coincides with the fixed route. The red line indicates the real trajectory of a₁. a₂ starts from (15,0.6). a₂ passes through A′, B′, C′, and D′ in that order. a₂ finally reaches (4.05,1.08). The blue line indicates the real trajectory of a₂. The trajectories of a₁ and a₂ are from right to left. The yellow line indicates the a₂ by multiple sensors to observe a₁. The yellow line is also a fusion trajectory fused of a₁.

However, the positioning trajectory obtained by UWB is not consistent with the real trajectory in Figs 13 and 14. This is due to the fact that the experiment is conducted in an indoor environment with many obstacles. The accuracy of UWB positioning is affected by signal interference, reflections, and other factors.

Fig 15 shows the valid coordinates of a₁ localized by different sensors. The figure includes the visualization results of single-sensor and multi-sensor fusion. The red dots are the coordinates of a₁ localized by the UWB carried by a₁. Purple dots are the coordinates localized by the fusion module for a₁. Specifically, a₁ is fused through the positioning coordinates of a single sensor and a₂ through multiple sensors observing the positioning coordinates of a₁. In addition, the Fig 15 is somewhat different from the result of locating a₁ by fusion in Fig 14. The trajectory of a₁ by UWB localization in Fig 14 is a continuous line, as shown in the red line in Fig 14. While it is a discontinuous point, as shown in the red dots in Fig 15. This is due to the fact that the coordinates obtained by GBBopen are different from the frequency of the localization coordinates obtained by UWB. Among them, UWB obtains the positioning coordinates more frequently than the coordinates obtained by GBBopen. And the data obtained by GBBopen is also susceptible to the influence of net clusters.

In this paper, Fig 13 is analyzed from different perspectives. First, compare the distances of a₁ and a₂ to J₁. It can be seen that the ED from a₁ to J₁ is shorter than a₂ to J₁. From the above analysis, a₁ is the leader and a₂ is the follower. From the figure, it can be seen that a₁ gradually approaches J₁. According to the Norm r₄, then a₂ uses the goal following approach toward a₁. In this figure, a₂ by multiple sensors can observe a₁ when a₁ has traveled to A. Meanwhile, a₂ has traveled to A′. According to the Norm r₂₅ ∼r₂₇, so a₂ takes a goal following towards a₁. Until a₂ by multiple sensors do not observe a₁, at this point a₁ travels to B. Meanwhile, a₂ has traveled to B′. Since both a₁ and a₂ are in the execution task area and a₂ by multiple sensors do not observe a₁. According to Norm r₅, then a₂ takes a trajectory following towards a₁. a₂ travels from B′ to C′. Similarly, a₂ by multiple sensors to observe a₁ when a₁ has traveled to C. Meanwhile, a₂ has traveled to C′. According to the Norm r₂₅∼r₂₇, so a₂ takes a goal following towards a₁. Until a₂ by multiple sensors do not observe a₁, at which point a₁ travels to D. a₂ has traveled to D′. Since both a₁ and a₂ are in the execution task area and a₂ by multiple sensors do not observe a₁. According to Norm r₅, then a₂ takes a trajectory following towards a₁. a₂ starts traveling from D′. Until both a₁ and a₂ complete their firefighting tasks.

Evaluation.

This paper statistics the raw data acquired by GBBopen. GBBopen receives the number of localization coordinates of all sensors simultaneously is 4070. The number of coordinates of UWB localization that are larger than the distance to the ground truth is 3608. So FER is calculated as 11.35%.

Table 10 shows compare results of the range of error and MSE between single sensor and multi-sensor fusion. The range of error of ED between the single sensor observation and the true value is 0.80∼1.47 m. The error range of ED between the observed and true values after multi-sensor fusion is 0.59∼1.02 m. The Min of the error of the ED after multi-sensor fusion increased by 26.25% over the single-sensor observations. The Max of the error of the ED after multi-sensor fusion decreased by 30.61% over the single-sensor observations. The range of MSE between the single sensor and real values of the a₁ after multi-sensor fusion is 1.06∼1.35m². The MSE of the multi-sensor fusion results over the real value of a₁ is 0.66∼0.86m². The Min of the MSE after multi-sensor fusion is 37.74% higher than that of the single-sensor observations. The Max of the MSE after multi-sensor fusion is increased by 36.30% compared to that of the single-sensor observation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Compare results between single sensor and multi-sensor fusion.

https://doi.org/10.1371/journal.pone.0287791.t010

The analysis of the above experiment results shows that the fusion of multiple sensors is achieved in this paper. This fusion module ensures precise positioning between adjacent unmanned vehicles using multiple sensors. The unmanned vehicles are coordinated to accomplish the firefighting task. This fusion module reduces the impact of FERD system in open environments. This module also improves the environmental awareness of each unmanned vehicle.