The physical environment.

The physical environment is defined by its spatial, temporal and dynamic properties. In particular, in the aftermath of a disaster, a number of specific sites might need urgent attention and access to these sites may be limited to certain areas (e.g., due to trees, debris, or natural obstacles). Hence, we can capture such features in terms of paths along which agents can travel from one disaster site to another. Specifically, the spatial property of the environment is encoded by a graph, which specifies how and where agents can move.

Definition 1 (Graph) We model an area of the environment as an undirected graph G = (V,E), where each vertex V representing spatial coordinates are embedded in Euclidean space and edges E encode the movements that are possible between them. Here, we denote N = ∣V∣.

In disaster response, each disaster site is a vertex in the graph, and a traversable area between a pair of sites is an edge of the graph.

Definition 2 (Time) Time is modelled by a set of time steps {1,2,…,T} and at each time step t ∈ {1,2,…,T} the agents visit some sites in the environment.

To capture the dynamic attributes of the environment, we assume that each vertex holds two states: one for information and one for threats.

Definition 3 (Information State Variable) An information state variable indicates different levels of the information at a given vertex.

For example, how many people need help and what is the status of a bridge are information state variables in disaster response scenario.

Definition 4 (Threat State Variable) A threat state variable reflects the level of damage an agent suffers when visiting a given vertex.

For example, the level of fire and the degree of smog are typical threat state variables in disaster response.

Definition 5 (Markov Model of Information and Threat) The two state variables at each vertex change over time according to discrete-time multi-state Markov chains.

To capture the transitions of the state variables, we employ a Markov chain model. Specifically, for a Markov chain with K states S = (S₁, S₂, …, S_K), the matrix of transition probabilities for pairs of states is defined as: where p_ij is the probability that threat state S_i transitions to S_j in one time step and S_i, S_j ∈ S. An example of information and threat models at a vertex is shown in Fig 1. Thus, Fig 1(a) shows a threat model with 2 states (i.e., R₁ and R₂) and Fig 1(b) shows an information model with 3 states (i.e., I₁, I₂ and I₃), where the probabilities of each information/threat state changes over a time step are given (e.g., the probability of R₁ changes to R₂ is 0.1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Example of information and threat models at a vertex.

(a) A threat model with 2 states (i.e., R₁ and R₂) and (b) an information model with 3 states (i.e., I₁, I₂ and I₃), where the probabilities of each information/threat state changes to another over a time step are given (e.g., the probability of R₁ changes to R₂ is 0.1).

https://doi.org/10.1371/journal.pone.0130154.g001

The set of information states for location v_n correspond to an amount of information which agents obtain when visiting v_n. The value of information is determined by the function fⁿ:Iⁿ → ℝ⁺, and increases monotonically with , which indicates that the states of information are ordered in terms of their value. The information state at a given vertex independently evolves as a -state Markov chain model with a matrix of transition probabilities .

Similarly, the set of threat states indicate the threat levels of vertex v_n ∈ V. The “damage” that an agent suffers when visiting vertex v_n is captured by the function cⁿ:Rⁿ → ℝ⁺, and increases monotonically with . The threat state at a given vertex independently evolves over time as a -state Markov chain and the matrix of transition probabilities is .

Having modelled the environment in which the agents operate, we next elaborate on the agents’ goals.

Patrolling Agents.

We define a patrolling agent (agent for short) as a physical mobile entity situated in the environment defined above, capable of gathering information, and maybe damaged by the threat when visiting a vertex. The set of all agents is denoted as A = {1,…,∣A∣}. Then, the movement and visit capabilities of agents are formulated as follows. When patrolling in a graph G, each agent is positioned at a given vertex in G at each time step t. The movement of each agent is atomic, i.e., takes place within the interval between two subsequent time steps, and is constrained by G, i.e., agent m positioned at a vertex v_i ∈ V can only move to a vertex that is adjacent to v_i in G. We assume that ∀v_i ∈ V, v_i ∈ adj_G(v_i), i.e., an agent can also stay at the same vertex. The speed of the agents is sufficient to reach an adjacent vertex within a single time step. Time can be discretised according to the speed of the UAVs. Thus if the UAVs can travel between sites in a five minutes, then a time step may be set at 5 minutes in the model.

Given this, an agent visits a vertex v_n when it is positioned at that vertex. On the one hand, a visit results in the agent being aware of the current information and threat state at v_n, such as and respectively. On the other hand, this agent obtains a reward and suffers a loss for a visit. The time it takes to visit a vertex is assumed to be negligible. We let denote the information value vector, where fⁿ(I_k) is the information value that an agent could get if the information state is (e.g., information at a vertex has 3 states and corresponds to 3 information values [0, 2, 5]). Similarly, we let denote as the damage value vector at vertex v_n, where is the damage value that an agent will lose if the threat state is (e.g., fire level at a position has 4 states which corresponds to 4 levels of damage [0, 4, 6, 10], and smog degree at a position has 3 states which corresponds to 4 levels of damage [0, 2, 5]). For each visit, the information at that vertex is obtained by the agent and we regard that the information state at a given vertex v_n will reset to when an agent visits this vertex ( is the information state which means no new information was generated at v_n after last visit). As the states at each vertex change over time and agents can only access the exact states at the vertices that they visit, the patrolling environment can be considered non-stationary (i.e., joint probability distribution of its states may change when shifted in time) and partially observable.

Furthermore, in this paper, we make two assumptions about the communication and cooperation among agents as follows.

Assumption 1 All the agents can share their collected observations with each other via communication. Such peer to peer communication is free of noise, costs, and delays.

Consider a centralised station is organized to coordinate a team of UAVs for monitoring the continuously changing state of a disaster area, where each UAV can full communication with this station and Assumption 1 is satisfied in these domains. However, in some real scenarios, UAVs can only coordinate with each other using limited communication and decentralised approaches may be more appropriate (but this is beyond the scope of this paper and will be considered in future work).

Assumption 2 When more than one agent is visiting a vertex, only one information value is obtained for the team but each agent suffers the same damage that may be generated at that vertex.

This assumption is satisfied in the scenarios where the information gathering capability of one agent at a vertex is equal to that provided by multiple agents with the same sensors, and agents independently suffer the damage caused by threats. In future work, a model of information fusion for multiple (heterogeneous) agents will be considered. Thus, the team of agents need to coordinate with each other based on their observations while patrolling. Specifically, the goal of the agents is to gather as much information as possible while minimise the damage incurred.

We now provide a simple example to explain how the agents would operate in this scenario. Consider an agent that enters into a building on fire. In our setting, this is equivalent to the agent visiting a node in the graph. The fire level (threat state variable) and valuable information about victims and assets (information variable) changes over time. While exploring the building, the agent may acquire some information and suffer some damage due to the fire. At each time step, an agent selects one adjacent building to visit based on the estimated information value and the prior observation of threat states at each location. It then obtains a reward based on the value of the information, and suffers a loss which is associated with the threat state. Then, the information state at the visited vertex is immediately reset.

Having defined the patrolling problem, we now need to plan the sequential patrolling actions for agents based on the history of actions and observations, and the model of the environment. Hence, in what follows, we first propose a POMDP formulation for single-agent patrolling within a graph and design an algorithm to solve it. Building upon this, we propose a scalable multi-agent patrolling algorithm.

The POMDP Framework.

We now set up the single-agent patrolling problem as a POMDP ⟨𝓢,𝓐,𝓞, T,Ω, r⟩ as follows:

• 𝓢 is the set of states. A state is defined as a tuple , where v is the current position of the agent, and are the threat and information states at vertex v_n ∈ V. We denote , as the state that captures the information and threat states at all of the positions. Given this, the number of the states in 𝓢 increases exponentially with the number the vertices.
• 𝓐 is the set of all actions. The agent select an adjacent vertex to visit as an action.
• 𝓞 is the set of observations. We define an observation as the current position v_i and the information and threat state at this position.
• 𝓣 is the set of conditional transition probabilities. We assume that v is deterministic and only determined by the destination of movement of the agent. Based on the Markov models defined in the patrolling problem, s_e follows a discrete-time Markov process with states.
• Ω is the set of observation probabilities. As an observation o is directly a part of some states, the observation probability Ω(o∣s′, a) = 1 if o is consistent with the corresponding part of s′ and Ω(o∣s′, a) = 0 otherwise.
• r:𝓐 × 𝓞 → ℝ is a reward function. r(a, o) is the sum of the rewards obtained by the agent which associates to the action a and observation o: (1) where α is a weight parameter of the two objectives.

In this model, the states are not directly observable. Hence, a standard belief vector B(t) = [b₁(t), …, b_M(t)] is defined as the posterior probability distribution over the possible states S, where b_m(t) is the conditional probability that the environment state is at the mth state at the current time step t. For any t, it has been shown in [27] that this belief vector is a sufficient statistic for the design of the optimal action for each time step. A policy π specifies the action that will be executed in any given belief state and the optimal policy π* is a policy by which the agent gets the maximum total expected reward accumulated over T steps. However, as each environment state is an joint state of the information and threat states at all of the vertices, the number of possible states S that defined in our POMDP is , which increases exponentially with the number the vertices. Moreover, as the belief vector is defined as the posterior probability distribution over these possible states, the dimension of this belief vector also increases exponentially with the number the vertices.

To address this, we propose an online method by introducing a belief vector of reduced dimension and develop a predictive heuristic to reduce the search space and still produce high quality solutions (as we show later).

Compact Belief Representation.

As the threat state and information state variables at each vertex evolve independently and v is deterministic, we can find a sufficient statistic for the optimal policy whose dimension linearly grows with N, similar to [23, 24]. We introduce a compact representation of belief state and its transition function in this section.

We define a sufficient statistic belief vector of the environment states at time t as the vector of the conditional probabilities (conditioned on the observation and decision history) Ψ(t) = [Ψ_R(t),Ψ_I(t)], where Ψ_R(t) is defined as: (2) where is the probability that the threat state at vertex v_n is , and Ψ_I(t) is defined as: (3) where is the probability that the information state at vertex v_n is and . Then Ψ(t) is a sufficient statistic of optimal decision making [23, 24]. By exploiting the statistical independence among vertices, we reduce the dimension of the sufficient statistic from to , which grows with N linearly. This allows us to reduce the computational and storage complexity of the optimal patrolling policy from exponential to linear.

Theorem 1 For any time t, Ψ(t) is a sufficient statistic for the design of optimal policy for our POMDP formulation.

Proof We show that when the information and threat at the N vertices evolve independently, each element b_m(t) in the standard belief vector B(t) can be obtained from Ψ(t), where b_m(t) is the conditional probability that the environment state is at the ith state. Without loss of generality, we consider N = 2. Let ℐ(t) denote the history up to the beginning of slot t. Let τ_n denote the most recent time instant when vertex v_n is visited. We can thus write an entry of b_m(t) as in Eq (4). Quantities in Eq (4) are entries of Ψ(t). Hence, Ψ(t) is a sufficient statistics. (4)

Initially, we assume that we have probabilistic information about the state of each of the N vertices Ψ(0) = [Ψ_R(0),Ψ_I(0)]. Then, the elements of belief vector Ψ(t) are updated to Ψ(t+1) upon action a = v_i and observation as: (5) where ∀v_n ∈ V, , , and is a unit vector with the k^th item is 1, and are respectively the matrices of transition probabilities of threat and information at position v_n. As shown in Eq (5), the threat belief vector at one vertex v_n that some agent is visiting is updated to based on the observation at this vertex, while at some other vertex that no agent is visiting is updated by the current threat belief vectors and threat Markov model at this vertex. A similar explanation holds for the update to the information belief , as for v_n ∈ V.

Based on the transition function above, a policy π specifies a sequence of actions π = [π(1), π(2),…], where π(t) is the position selected to visit at time t. Given this, the optimal policy can be computed as: (6) where ℛ^π(t)(Ψ(t)) is the reward obtained when the belief state is Ψ(t) and γ ∈ [0, 1] is the discount factor.

Although the dimensionality of the belief state is reduced, the problem is still a POMDP and finding the optimal solution is intractable. Based on this reduced belief vector, we next develop a predictive heuristic and present the online single-agent algorithm that implements this heuristic.

The Predictive Heuristic.

In order to develop a predictive heuristic for online policy selection, we first introduce the assumption that the Markov state transition matrices are monotone matrices, which means that the higher the information/damage value of the vertex’s current state the higher is the likelihood that the next state of this vertex will be of high information/damage value. Then, we show how to define the predictive heuristic as the predictive expected future reward based on the monotonicity of the transition matrices.

Stochastic dominance is a central theme in a wide variety of applications in economics, finance and statistics [28]. Similar assumption has been made to model the states of the channels in communication systems [23, 24] and the states at targets for UAVs monitoring [12]. Stochastic dominance ≻ between two Z dimension probability vector x, y is defined as x ≻ y, if: (7)

We assume that the Markov information model and Markov threat model are monotonic matrices, i.e., the matrix of transition probabilities and satisfies: (8)

If the matrix of transition probabilities and satisfy the assumption above, then and are monotone matrices[29]. Under this assumption, the higher the information value of the state of the current vertex the higher is the likelihood that the next state of this vertex will be of high information value, i.e., if , then . From (5), we know that probability vectors for information states of two vertices keep the relationship of stochastic dominance when no agent visits any of them. Obviously, if , then , which means that a stochastically dominant information belief vector is likely to have a higher information value. The same is true that a stochastically dominant threat belief vector is likely to have a higher damage value. In particular, as the information state at a given vertex will reset to I₁ when there is an agent visiting this vertex, the belief vector of information states (1,0,…,0) is stochastically dominated by the belief vector of any vertex which is not being visited, so the more recently visited vertex always has a lower expected information value.

To note, our monotonicity assumption is not a constraint that makes the information value (or the damage of the threat) increasing with time, but a model that the probability vector of the information (or threat) transition matrices satisfy the feature of stochastic dominance. We now provide a example of a 4-state Markov threat model at a vertex as follows: It can be seen that the Vectors of P_R satisfy the condition of Eq (8), i.e., P_R4 ≻ P_R3 ≻ P_R2 ≻ P_R1, where for P_R3 ≻ P_R2 as an example, the elements of P_R2 and P_R3 match the condition for stochastic dominance of Eq (7) as: For example, if the threat states at vertices v₁ and v₂ are respectively and , i.e., . Then, v₁ is likely to have a higher next threat state than v₂. However, after a time step, it is possible that any threat state may switch to not only a higher state, but also a lower state.

Then given the monotonicity assumption, we can use the relationship between the belief states at different vertices in order to “predict” the belief state at an unvisited node. Hence, we can estimate the expected reward agents may get from one vertex of the graph when visiting it at a near future step. We denote a feasible policy of length D at time t as π_D(t) = (π_t+1,…, π_t+D), which consists of D consecutive deterministic vertices/actions.

Here, we define the predictive heuristic as the predictive expected future reward for policy π_D(t), which is the aggregate of the expected reward of each step in π_D(t) as: (9) where, is the predictive belief vector at the vertex π_t+i and time t+i. For the step t+1, we can get the predictive belief vector by the current belief vector Ψ(t), current action a(t) and observations θ(t), i.e. , which is the belief vector at t+1 and obtained from Eq (5). For {t+2,…, t+D}, we get the predicted belief vector based on a transition which omits observations in Eq (5) as follows: (10) where τ = {t+1,…, t+D−1}.

Given the predictive heuristic and policies that looks ahead D time periods, the agent compares all feasible paths of length D and chooses the next location to visit according to the path that gives the highest predictive expected reward gained over that path. The details of how to use the heuristic in our online single-agent algorithm is presented in the next section.

The Online Algorithm.

Based on the predictive heuristic, we propose an online algorithm for single-agent patrolling problem (Algorithm 1) in this section.

Algorithm 1 Single-Agent Patrolling

Require: : the Markov threat models

Require: : the Markov information models

Require: Ψ(t): the belief state of current time step

Require: o(t): the observation at the current position

Require: v(t): current position.

Ensure: a*(t+1): next action of the agent

  ▷ Step 0: get all feasible policies Π_D(t);

  ▷ Step 1: computing best policy:

1: for π_D(t) ∈ Π_D(t) do

  ▷ Step 1.1: Get predictive belief state for next D steps:

2:   Ψ(t+1) ← δ(Ψ(t)∣v_t, θ(t))

3:   for τ ∈ {t+1,…, t+D−1} do

4:    for v_n ∈ V do

5:     

6:    end for

7:   end for

 ▷ Step 1.2: Compute the predictive reward for π_D(t):

8:   

 ▷ Step 1.3: Compare π_D(t) with the stored best policy:

9:   if then

10:    

11:   end if

12: end for

 ▷ Step 2:return the next action from the best policy

13: return

First, we compute Π_D(t), which is the set of all the feasible policies that start from current position v(t) (step 0), where we name the parameter D as the maximum horizon, i.e. the number of horizons we look ahead in the POMDP. Then, we compute the predictive expected reward for all the policies. For each policy, the belief state at t+1 is updated by the belief state, position and observations at t by Eq (5) (line 2) and the predictive belief state at {t+2,…, t+D} is computed by Eq (10) (line 3–7). Given this, we compute the predictive reward for the policy (line 8). Thus, the best policy is: (11) The best next action here is computed as , which is the first action of best policy (line 13).

Having defined the online single-agent algorithm for our formulation of patrolling under uncertainty and threats, we extend it to compute policies for multi-agent problems next.