Contributions

The contributions of the present study can be summarized as follows:

1. ○ To address PDP, we propose a novel technique that relies on coordinate features and its constraints to enable end-to-end training for the problems with a constraint like PDP without any hand-crafted features.
2. ○ We numerically demonstrate how such a simple approach can handle challenging problems in OP, such pickup and drop-off problems.
3. ○ We conduct experiments on two different applications to demonstrate effectiveness of the proposed technique.
4. ○ We conduct several ablation studies to investigate essential parameters that contribute to high performance.

The remainder of the paper is structured as follows. Section II provides a brief overview of traditional approaches and deep models for routing challenges. Section III presents an introduction to the PDP through mathematical formulation. In Section IV, we reformulate the problem in RL terms and present our DRL solution. Computational experiments and results of our analyses are presented in Section V. Finally, the results are discussed and conclusions are drawn in Section VI.

A. Formalization of PDP as a reinforcement learning problem

The aim of reinforcement learning is to select the best-known action for each given state, which means that the actions should be ranked and assigned corresponding values. Given that such acts are state-dependent, in essence, we should assess the value of state-action pairs. Then, in our task, a certain objective function should have been minimized. This objective function was the total distance traveled by the agent without a violation of any specified constraint.

The route-building process may be thought of as a set of decisions that can be naturally expressed in the form of RL and solved. The route-building procedure was previously described as a Markov Decision Process [45]. The state space, action space, transition rule, and reward function in the MDP were formulated as follows:

State: State s_t = M_t reflected the partial solution built at time step t, where M_t included all visited nodes up to step t, and M₀ refers to the depot.

Action: Section a_t was represented as v_j, i.e., picking node v_j at step t from the subset of the nodes allowed to visit (but not yet served).

Transition: The next state s_t+1 = M_t+1 = (M_t;) was derived from s_t by picking a node at step t, where “;” denoted concatenation of the partial solution from the previous step with a newly picked node.

Reward: The reward was defined as the negative of the entire tour length computed by adding negative values of all step travel distances. As a result, the reward was denoted as shown in Eq (1). (1) where d_t is the negative value of the incremental travel distance between v_i and v_j selected at step t and t+1, respectively. Of note, in DRL, we trained the network to learn the policy that maximized the expected sum of rewards. Therefore, in this problem context, we trained the network to learn the policy that minimized the total travel distance.

Policy: The stochastic policy p_θ chose a node at each time step based on the precedence criterion. This procedure was followed until all pickup-and-delivery services were completed. The ultimate result of enacting the policy was a permutation of all nodes, which specified the order of each node for the vehicle to visit, i.e., . The probability of an output solution was factorized using the chain rule as shown in Eq (2). (2) where V is the set of nodes of the instance. Decision making about the node selection based on the learnt policy p_θ was then performed.

B. Policy Network based on Hybrid Pointer Networks (HPN)

○ Policy’s encoder.

A routing problem’s action space is discrete and increases exponentially with an increase of the problem size. The policy-based reinforcement technique, which consists of an actor network and a critic network [46], is frequently used to address such problems. At each time step, the actor network constructs a probability vector across all actions based on the current state and then samples an action from the allowed action set; this is recursively repeated until the termination condition is reached. The reward of the actor network is computed by accumulating the discounted reward at each step. To reduce variance, the critic network is used, as it criticizes the actions made by the actor; said differently, the critic network acts as a judger of the actor’s action, thus showing how effective (good or bad) its action was as a baseline of the actor network and calculating the baseline reward. After obtaining the actor network’s reward and the critic network’s baseline reward, the policy gradient approach [47] is used to update the parameters of the two networks, with the actor network being taught to produce higher-quality solutions.

In order to learn policy π, following several previous studies [10] where the HPN model was built upon the pointer networks, we built a policy network with an encoder-decoder structure. Given the features of PDP, the HPN was expected to learn the link between the nodes of various roles, allowing the precedence constraint to be captured intrinsically. As a result, the parameters in this deep architecture also corresponded to those in π. We then rebuilt the architecture to make it more suitable for PDP.

Technically speaking, HPN works by passing the context through two different encoders before combining the two contexts using the summing function. For instance, Stohy et al. used a GNN for context encoding in the HPN model, as the graph can extract complex relationships from a graph, and the other encoder is the transformer’s encoder [48] made up of several self-attention layers. The LSTM [49] was used in the point encoder, which encodes the currently selected point concatenated with the truck’s remaining capacity.

As can be seen in Fig 2, for the transformer’s encoder, there was the multi-head attention sublayer where we left out the depot. For simplification, was taken to denote the node embedding of attention layer L−1 (L ∈ {1,…,N}), where i was the node index. We then assumed d_k, d_v to be the query/key dimension and d_v to be the value dimension, where and M = 8 was the number of heads. For the number of heads, we followed the common and parameters used in the literature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Multi-headed attention.

https://doi.org/10.1371/journal.pone.0267199.g002

Given input sequence V, in our case, V was a sequence of the cities’ coordinates concatenated with the cities’ demands. In order to improve feature extraction, the self-attention mechanism learns the relationships between sequence elements to construct a representation of the sequence. Based on input V, three vectors—namely, query, key, and value—were constructed to capture the relationships (see Eq (3)–(7)). (3) (4) (5) (6) (7) where and are trainable parameter matrices, H^enc is a matrix containing the encoded nodes, Q^l, K^l and Val^l are a query, key, and value of the self-attention, respectively, and l denotes the number of self-attention layers.

The graph encoder is the second. The context of the graph embedding layer was obtained by directly encoding the context vector or feature vector. We then simplified the equation and assumed that the graph was complete. As a result, the graph embedding layer could be written as shown in Eq (8) [50]: (8) where is the aggregation function, γ is a trainable parameter, is trainable weight matrix, and N(i) the adjacency set of node i. Next, l denotes the number of layers used for the graph embedding.

For the point encoder encoding the currently selected city concatenated with the truck’s remaining capacity, each city coordinates v_i (i.e. (v_i,1, v_i,2)) concatenated with c_t where c_t were taken to denote the truck’s remaining capacity at time stamp t the concatenated vector is embedded into a higher dimensional vector , where d is the hidden dimension. An LSTM then encoded the vector for the current city v_i. The hidden variable of the LSTM was passed to both the decoder of the current stamp and the encoder of the next time stamp.

○ Policy’s decoder.

The decoder was built on a pointer network’s attention mechanism and outputs the pointer vector u_i, which was then sent through a Softmax layer to generate a distribution over the following candidate points. The formulation of the attention mechanism and the pointer vector is shown in Eq (9). (9) where Softmax normalizes the vector u_i to be the “attention” mask over the inputs, u_i^(j) considered as the j-th element of the vector u_i. A, W_r and W_q are trainable parameters, q is the query vector from the hidden state of the LSTM, r_i is a reference vector containing the contextual information from all cities.

The distribution policy over all candidate cities is provided in Eq (10).

(10)

We then predicted the next visited city by sampling or choosing greedily from policy π_θ(v_i|s_i).To this end, we just reconstructed that architecture to meet the requirements of the problem (see Fig 3). In our problem, we had four features x, y, d, and c, x and y were the 2-dimonational coordinates for the points, d represented the points’ demand and finally c represented the truck’s remaining capacity. As our context included x, y, and d, we passed these three features through the hybrid encoder and obtained two contexts: one from the GNN and the other from the transformer’s encoder. We then concatenated the truck’s remaining capacity with the currently selected point, taking the mean of the GNN context, the transformer context and projecting them into a high dimensional space. We then added both tensors together with the concatenated one and passed them through the LSTM. The architecture of our model to tackle PDP is shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Illustration for how features flow inside HPN.

Feature vector consists of thee vectors: two city’s coordinates and the cities’ demand. | means concatenate both currently selected points with the truck’s remaining capacity.

https://doi.org/10.1371/journal.pone.0267199.g003

C. Training algorithm

Algorithm 1 summarizes the training of the suggested policy for solving PDP where we used the reinforcement learning approach with a roll-out baseline [8]. The policy gradient method was characterized by the following two networks:

1. The actor network drove node determination actions by making a vector of probabilities over those actions and inspected them as indicated by those probabilities to investigate the action space.
2. In the self-critic network, which is a roll-out baseline with a topology similar to that of the actor network, the reward was determined given the beginning state by selecting the node with the best likelihood to eliminate variance. After obtaining the reward of actor network R and the baseline reward of the critic network, the reinforcement learning technique used the policy gradient approach to adjust parameters of two networks.

Furthermore, when the performance of the latter was considerably better according to the results of a paired t-test on a specified number of batches, the parameters of the critic network were substituted with those of the actor network. Policy P was trained to identify higher-quality solutions by upgrading the two networks.

Algorithm 1. REINFORCE with Rollout Baseline [8]

1: Input: number of epochs E, steps per epoch T, batch size B, significance α

2: Init θ, θ^BL ← θ

3: For epoch = 1,…,E do

4: For step = 1,…,T do

5: s_i ← RandomInstance () ∀i ∈ {1,…,B}

6: π_i← SampleRollout (s_i, p_θ) ∀i ∈ {1,…,B}

7: ← GreedyRollout

8:

9: θ ← Adam (θ, ∇ℒ)

10: End for

11: If OneSidedPairedTTest (p_θ, ) < α then

12: θ^BL ← θ

13: End if

14: End for

Experimental settings

As mentioned above, PDP was defined through an undirected graph G = (V, E) where node i = {0, 1,…,m} was re-presented by features n_i. The index i = 0 represented the depot node, and i > 0 represented the i-th customer node. The vehicle had capacity D > 0 and each customer node i = {1,…,m} had a demand δ_I, 0 < δ_i < D. We assumed that the depot demand δ₀ = 0. Both depot and customer nodes were randomly generated inside the unit square [0, 1] × [0, 1]. We generated 21 and 51 (the first node was the depot) for instances with a size of m = 20 and 50; due to memory constrain, we did not go through size 100. We used Kaggle’s Tesla P100 with 16 GB of memory. The corresponding vehicle capacities were 30 and 40, respectively. The customer node demands were uniformly sampled from {1,2,…,9}. We normalized the customer node demand to [0, 1] by the truck’s max capacity through The input, masks, and decoder context vectors for PDP were adjusted as follows:

1. Input: Handling PDP needed only to expand the node feature n_i to a four-dimensional input that included the normalized demand , node feature n_i (nodes’ coordinates), and the nodes’ Euclidian distance from depot (see Eq (11)).
   (11) where || represents concatenation.
2. Vehicle remaining capacity update: We masked out the customer nodes that were served, so there was no need to update the needs of served customer nodes. The decoder selected the customer node π_t at timestep t, and the remaining capacity of the vehicle was represented by . We assumed that the vehicle started at the depot when the decoding timestep t = 1 and the vehicle was zero loaded (with the remaining vehicle capacity ). The remaining vehicle capacity was updated using Eq (12).
   (12)
3. Decoder context vector: The context vector c_t of the decoder at timestep t ∈ {1,⋯,m}, where m was the number of nodes without depot, of three components: hybrid embedding , embedding of node π_t and vehicle remaining capacity (see Eq (13)): (13) where W_x is a trainable parameter.
4. Mask update mechanism: For PDP, our mask consisted of the following two parts: (1) the customer node mask and (2) the depot node mask. For the customer node mask, we masked out:
   1. ○ Customer node that was served;
   2. ○ Any pickup city if its demand plus the remaining capacity exceeded the truck’s limit (1);
   3. ○ When the demand of a customer node exceeded the remaining capacity of the vehicle. That is, customer node i’s attention coefficient u_i,t = −∞ when or .
   4. ○ For the depot mask, as the depot would not be the next chosen node when the vehicle left the depot, the depot node 0’s attention coefficients u_i,t = −∞ for t = 1 or π_t−1 = 0

Next, in order to test our model capability and to generalize through different settings, we then did the following two separate applications:

1. In the first application, considered for bike sharing system BBS, we assumed that all demands summed to zero, and our goal was to satisfy the demands as much as we could, and the truck would leave the depot with a zero capacity.
2. In the second application, we assumed more randomness in demands and randomized the demands while balancing the number of the pickup points with the drop-off one.

The points of the stations and clients (pickup and drop-off) nodes were then arbitrarily and freely created involving a 2-layered uniform circulation in the scope of 0 and 1, where the distance between two hubs was determined in view of Euclidean space. Hyperparameters used during training are summarized in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Hyperparameters used for training both experiments.

https://doi.org/10.1371/journal.pone.0267199.t001

i. Bike Sharing Systems (BBS).

In order to ensure customer satisfaction and system effectiveness, it is essential to rebalance BSS bike distribution [51]. In previous extensive research on redistribution, numerous efficient methods that maintain an equilibrium of bikes at each station have been proposed.

The three forms of BSS rebalancing include static, dynamic, and incentivized rebalancing. A fleet of trucks is frequently used to redistribute bikes in static and dynamic rebalancing models. First, static rebalancing, alternatively referred to as SBRP, is usually performed at night or during periods of low demand. This is so because SBRP assumes that the number of bikes needed at each station remains constant or shows only light fluctuations. Second, the rebalancing outcome in the Dynamic Bicycle Repositioning Problem (DBRP) is influenced by bike movement, which has a substantial impact on bike demand at each station. Consequently, solving DBRP requires accurate demand forecasting. Third, incentivized rebalancing promotes users to participate in system rebalancing by advising modest changes to their intended itinerary via control signals, which provides alternative routes that promote rebalancing, or even paying users for returning bikes to a station. In recent years, BSSs have outgrown the need for stations/docks, with users becoming able to deposit and pick up bikes anywhere within designated municipal boundaries. This sort of bike-sharing, also known as a free-floating bike-sharing system (FFBSS), has several important advantages over traditional BSSs, such as cheaper capital costs and lower risk of bike theft. The system’s efficient rebalancing is critical to its success [52].

The results on BBS21 and BBS51 where the truck’s capacity will be 30 and 40, respectively, are summarized in Table 2. As can be seen in the table, our model achieved a low tour length as compared with the other two models with a large margin. For the sack of clarity and to demonstrate effectiveness of our approach, we used previously proposed models [34, 50] that handled the traveling salesman problem (TSP) with our settings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Our result on BBS21 and BBS51, obj denotes for the total tour length and the Time denotes for the time for evaluation the result averaged over 10k instances.

https://doi.org/10.1371/journal.pone.0267199.t002

Several representative instances for BBS21 and BBS51 are shown in Fig 4. As can be see from the the line plot in Fig 4, at the end of the tour, the truck’s capacity returned to zero, and the overall truck’s capacity during the tour did not exceed its limit (1). This results demonstarted feasibility of the proposed solution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Sample instances for BBS21 and BBS51, with each point labeled with its demand and the vehicle remining capacity once the vehicle served this point.

https://doi.org/10.1371/journal.pone.0267199.g004

ii. PDP with random demands.

In our second application, we tested our methedology against more randomness in generation demands. To this end, we generated equal drop-off and pickup points; yet, we did not ensure that their sum would be equal to zero. In order to esnure finding a feasible soluation for each batch, we also assumed that the agent was allowed to visit the depot and refill half of the truck during to gurantee. Our results on PDP21 and PDP 51 where the truck’s capacity was 30 and 40, respectively, are summarized in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results on PDP21 and PDP51.

https://doi.org/10.1371/journal.pone.0267199.t003

As can be seen in Table 3, our model had the lowest tour length with a large margin for both sizes as compared to GPN and Pointer Network. Fig 5 shows some representative instances for PDP21 and PDP51. As can be seen from the line spot in Fig 5, the truck’s capacity did not execeed its limit, so the solution can be evaluated as feasible.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Sample instances for PDP21 and PDP 51, with each point labeled with its demand and the vehicle remining capacity once the vehicle served this point.

https://doi.org/10.1371/journal.pone.0267199.g005

iii. Large instances.

Given that handling huge instances is a significant challenge, when dealing with real-world applications, a high number of points is to be expected, whi is the criterion that makes one model better than others. In our experiments, we sought to generalize our approach for large instances. In order to achieve the best potential generalization from our model, we used a simple approach of diffusing the generalization capability throughout our model’s parameters. We then trained our model on instances of size 50 and validated our training on instances of size 500 to fine-tune the model’s parameters for bigger instances; our assumption was that the sum of the demands would tend to be equal to zero. Upon setting the batch size of the trainig data to 128 and the validation data to 64, we trained our model only for 4 epochs to avoid overfitting. Finally, in order to avoid memroy limitation, we decreased the number of hidden unites to 64. A summary of our training hyperparameters during training of large instances is provided in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Hyperparameters used for training of large instances.

https://doi.org/10.1371/journal.pone.0267199.t004

Table 5 shows the result of BBS101,BBS201,BBS301,BBS401,and BBS501. Our assumption was that the truck’s max capacity for all of these instances was 40. Due to the memory constraint, we averaged only over 10k intances. Fig 6 illustrates several simple tours for BBS101,BBS201,BBS301,BBS401, and BBS501.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Sample tour for BBS101, BBS201, BBS301, BBS401 and BBS501.

https://doi.org/10.1371/journal.pone.0267199.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of BBS101, BBS201, BBS301, BBS401, and BBS50.

https://doi.org/10.1371/journal.pone.0267199.t005

a. Model choice

While a trade-off between speed and accuracy may be achieved by adjusting the model selection, we discovered that the HPN has a superior Encoder-Decoder combination to deal with complicated input relations. Therefore we opted to tweak this model to deal with PDP.

b. Result analysis

The total demand for a standard bike-sharing system (BBS) tended to be zero. We drew both the truckload and the point’s requests during the journey. Following intuition, for our process to be correct, the truckload should have stayed inside its. Our findings revealed that exceeding the truck limit was impossible, and the model attempted to balance loading during the route, which resulted in a loading plot with a line that rarely exceeded 80% of the truck capacity. Then, in order to see evaluate training stability, we increased the model’s burden and randomly generated demands for both the pickup and drop-off points; however, to ensure visibility of the solution, we added a more relaxing option for the agent and let the agent return to the depot and refill half of the truck capacity. The results showed that all model’s solutions were feasible, including the condition with the capacity margin. Although we experienced difficulties when applying this approach to large-scale instances due to memory constraints, using the validation trick during model training yielded good results with only 16G of RAM. It can be speculated that increasing batch size during training large scale of instance would provide more stable training process and better results than those we obtained.