Problem statement and proposed approach

The authors of this paper are members of a larger team responsible for designing the largest on-road trial in Australia of cooperative vehicles and infrastructure. This FOT will be deployed on the public road from 2020 for a year. One goal of this FOT is evaluating the safety benefits of the Cooperative Intelligent Transport Systems (C-ITS). The FOT will test seven use-cases applications that hopefully will make roads safer and hence contribute towards the Vision Zero goal of no road deaths and serious injuries on Australian roads.

This FOT will include a large number of passengers car which are equipped with C-ITS and IMU devices. Moreover, many C-ITS roadside devices will be deployed on arterial roads and motorways. The installed devices will allow real-time vehicle-to-vehicle and vehicle-to-infrastructure communication to exchange information related to the road and vehicles; this combined information will generate relevant safety warning messages for drivers. In this paper, we are interested in two use-case applications, namely the Slow/Stopped Vehicle (SSV) and The Road Works Warning (RWW).

The Slow/Stopped Vehicle (SSV) event provides warning to the Subject Vehicle (SV) (upstream) which is triggered by a slow or a stopped C-ITS enabled Target Vehicle (TV) (downstream). Every cooperative vehicle is consistently broadcasting cooperative awareness message (CAM) [25] to other cooperative vehicles within communication range. These CAM messages include position, heading, speed, acceleration, and other parameters from the GPS chipset of the emitting vehicle. It is the responsibility of the SV to detect the SSV vehicle hazard by processing received CAM messages. The in-vehicle HMI will then display an SSV warning to the driver, as shown in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. General operation of SSV.

https://doi.org/10.1371/journal.pone.0229289.g001

The Road Works Warning (RWW) provides an advanced warning to the SV approaching a roadworks zone. The RWW DENM (decentralized environmental notification message) is created and broadcasted by C-ITS-F which interfaces to a Transport and Main Roads road works database service. The DENM contains the work zone information such as the speed limit, lane closures, and location. The SV will receive the RWW DENM and decide on the relevance of its location to the work zone and, in particular, the approach zone to the road works. When the location of the roadwork becomes relevant, the HMI will display an RWW warning in addition to the speed limit and lane closure (if any) to the driver, directing them to take suitable action, as shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. General operation of RWW.

https://doi.org/10.1371/journal.pone.0229289.g002

In the non-C-ITS environment, SSV is similar to lane closure due to roadworks because both can cause safety risk and traffic capacity drop. In the case of lane closure due to roadworks, the driver will see the posted warning before the position of the lane closure but this kind of warning does not exist in the SSV case. In the above two cases, in the previous two figures, the event speed is zero; however, the speed of SSV event is not zero. Thus, to our mind, it has less safety risk compared to the zero speed safety events.

In the case of Slow/Stopped Vehicle and lane closure from roadworks, we test the hypotheses that the driver will change lane early before the location of the lane closure. In other words, we need to test the independence of the lane change and the SSV warning as well as the independence of the lane change and the lane closure warning for RWW.

In the remaining of this paper, C-ITS warnings or application will only mean SSV or RWW. Moreover, lane change and early lane change will be used interchangeably. Testing the independence of the lane change and C-ITS warning displayed on the HMI will require constructing a 2x2 contingency table using the IMU data collected from the C-ITS vehicles and non-C-ITS (control) vehicles during the safety events related to a particular C-ITS application. Then using the chi-square statistical test to check the independence of lane changing and C-ITS warning. This approach looks straightforward; however, estimating the elements of the 2x2 contingency table is not trivial because we only have IMU data.

In this paper, given a set of labelled (lane change/lane-keeping) IMU reading, we proposed using the Fast Fourier Transform to estimate power spectrum for each of the IMU signals (Acceleration in Y [Gs], Acceleration in Z [Gs], Roll [degrees], and Yaw [degrees]) within a window of a predefined length. In a real scenario, this window will be started at the onset time of the C-ITS warning. Consequently, we concatenate the power spectrum of all signal of each example to form the corresponding features vector, which is a row in the training matrix. The training data is used to build a Random Forest model to classify the new unseen and unlabelled features vectors. For each unlabelled features vector, the classification output could be one and zero (which we called a “hard” decision), or the probability of each label (which we called a “soft” decision). Finally, we use the output of the classifier to construct the 2x2 contingency matrix and test the C-ITS and lane change and the direction of the association. In the following sections, we will describe in details each of the above steps and show the experimental results.

UAH-DriveSet dataset

In this section, we describe the dataset used to train and test the proposed framework to evaluate the safety benefits in the planned FOT. At the time of writing, we are in the planning phase for the FOT, when we proposed the hypotheses based on the FOT research questions. Moreover, the sensors measures and the models/frameworks to test the hypotheses are identified. Because we proposed a new framework, testing is needed before starting the data collection from the FOT; for that, we used the UAH-DriveSet [26]. The UAH-DriveSet it exhibits the same characteristics as the dataset we will collect for the FOT. This dataset was collected using six different drivers and vehicles. The dataset has more than 500 minutes of naturalistic driving on highways and arterial roads. The data set is collected with the smartphone application DriveSafe, which uses the smartphone’s inertial sensors, GPS, camera, and internet access. The smartphone with the DriveSafe is put up on the centre of the windshield and its rear camera is facing the road. DriveSafe executes its calibration routine at startup in order to make the smartphone perpendicular to the ground such that the Y-axis aligns with the lateral axis of the vehicle and Z-axis with the longitudinal axis of the vehicle. The experimenters used another phone and put it up on the windshield beside the first phone to capture the video of the road during the drive session. For each trip, the available data includes lane changes list, raw accelerometer data at 10 Hz, and the video of the trip, which is reduced to generate the lane changes list. The lane changes list is used as the ground truth labels in our experiment.

In the context of the C-ITS safety benefits evaluation, we are interested only in detecting lane changes versus lane keeping. Thus we consider slow right, fast right, slow left, and fast left lane changes one single class (lane change) and assign them the same label.

Random forest

The random forest (RF) [27] is an ensemble approach that is very efficient in dealing with continuous response and labels. The most significant advantage of the RF approach is that it does not suffer from overfitting because of the Law of Large Numbers, where the error rate of RF will get smaller as more CARTs are added. By adding more CATRs, the error rate will be saturated but will never get larger no matter how many CARTs are added.

The main concept behind RF is building a large group of weak models that will give a resultant strong model. The RF is a large group of un-pruned decision trees with a randomised selection of predictors’ subset at each split. Each un-pruned decision tree is trained using a bootstrap sample from the original dataset. The well-known machine learning technique called Classification And Regression Tree (CART) [28, 29] is one of the common decision trees used in RF. An RF starts with the CART which, in ensemble terms, corresponds to the weak model. CART is a greedy and recursive top-down binary partitioning that divides the feature space into sets of disjoint regions. These regions should be pure with respect to the response variable. Given trained RF classifier, the label assigned to a test example is obtained based on the majority votes from all trees, [30, 31].

Features extraction

We have four channels/sensors. Each one provides us with a time series, namely:

1. Acceleration in Y filtered by KF (Gs)
2. Acceleration in Z filtered by KF (Gs)
3. Roll (degrees)
4. Yaw (degrees)

In order to build a data set that has the lane changing examples, for each time series, we take a short window (2w+1) around the event as shown in Fig 3 and extract features from it. The output of the four windows from the time series is concatenated to form an example of a lane change in the dataset. We created several examples from the same lane change event by shifting the window left and right such that the location of the event inside the window would be different. Creating several examples from the same event has two advantages. First, this data set will have the event at different locations within the window and hence the model built using this data will be independent of the exact location of the event within the window. Second, we will overcome the problem of a relatively small number of events in the UAH-DriveSet. We tried two sets of features. The first set is based on calculating the following quantities of each window; note that we applied the measure to both the values inside the window and the difference of the values inside the window (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Illustration of constructing a lane change training example using the time of the event and the raw accelerometer data.

https://doi.org/10.1371/journal.pone.0229289.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Measures applied to create features.

https://doi.org/10.1371/journal.pone.0229289.t001

The second set is based on calculating the FFT and then calculating the single side spectrum as the input feature for each time series within the window, as shown in Fig 3.

Chi-square test

As will be shown in the experimental work section, the sensitivity of the RF classifier is not good and our goal is only to detect if the number of lane changes and the usage of the C-ITS are independent. So instead of using zero/one output of the RF classifier (i.e. hard decision), we use the probabilities of lane change and lane-keeping (i.e. soft decision). The lane change and lane-keeping probabilities of the C-ITS and Non-C-ITS examples are used to construct the 2x2 contingency table. In this case, we are violating the assumption of the cells of the table being drawn from one multinomial distribution or two binomial distributions. However, the cell probabilities can be estimated using the sample mean. In the context of our problem, the explanatory variable is the presence (i.e. C-ITS) or absence (i.e. non C-ITS) of the C-ITS and the response is the lane change or lane-keeping. To test the independence of the number of the lane change and the C-ITS we used the Pearson chi-squared statistic

Where

n_ij is the observed cell (i,j)

μ_ij is the expected frequency of cell (i,j)

Given n₁₊ of C-ITS examples and n₂₊ of non C-ITS examples. We do not know how many lane changes n₁₁ or n₂₁ and we want to test the independence of the lane change and the C-ITS.

Recall that the given examples are in the form of raw measurement from the IMU. Moreover, we are given a trained RF classifier which has low sensitivity. In this paper, the raw data is processed as described in the above section to extract the input features to the RF classifier. The output of the RF are the probabilities of the lane change and the lane-keeping given certain data example. For the n₁₊ C-ITS examples, the output of the RF is n₁₊ pairs of probabilities. Each pair consists of the probability of lane change and the probability of lane-keeping for certain example in n₁₊. The summation of the n₁₊ lane change probabilities are the observed n₁₁ where the summation of the n₁₊ lane keeping is the observed n₁₂. Similarly for n₂₊we have n₂₁ and n₂₂ which are the lane change and lane-keeping observed numbers in the case of the non C-ITS.

The above contingency table can be used to estimate three different probabilities: joint, marginal, and conditional. Let π_ij = P(presence of C−ITS = i, early lane change = j) denotes the probability of observing cell (i,j). These set of probabilities form the joint probability of the explanatory variable and the response and they satisfy ∑_i,jπ_ij = 1. We can get the marginal distribution of the lane change by calculating the column total π₊₁ = π₁₁+π₂₁ and π₊₂ = π₁₂+π₂₂. Where π₊₁ and π₊₂ are the marginal probability of lane change and lane-keeping respectively. Since we observed a sample from the population, so we need to estimate the π_ij using the sample mean , where N = n₊₁+n₊₂ = n₁₊+ n₂₊

Moreover, we can use Table 2 to estimate a separate probability distribution for lane change at the given level of the explanatory variable (i.e. C-ITS or Non-C-ITS). These probability distributions are the conditional distribution. The lane change and C-ITS are said to be statistically independent if the population conditional distributions of lane changes are identical at each level of C-ITS. In other words, When these two variables are independent, the probability of lane change is the same at C-ITS and Non-C-ITS.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Illustration of the 2x2 contingency table.

https://doi.org/10.1371/journal.pone.0229289.t002

The odds ratio test

The odds of lane change are defined as where π is the probability of lane change. In the context of our 2x2 table, within the first row (C-ITS) the odds of lane change are and within the second row (non C-ITS) the odds of lane change are . Then the odds ratio from the two rows is defined as, . The odds ratio can equal any positive number. The odds ratio θ equals one when the C-ITS and the lane change are independent (π_{Non−C−ITS} = π_C−ITS). So that θ = 1 is the baseline for comparison. When θ>1 this means that the probability of lane change (π_C−ITS) in the presence of C-ITS is higher than the probability of lane change in the absence of C-ITS (π_{Non−C−ITS}). When θ<1 this means that the probability of lane change (π_C−ITS) in the presence of C-ITS is smaller than the probability of lane change in the absence of C-ITS (π_{Non−C−ITS}).

As shown, the odds ratio has the advantage of showing the direction of the association between the C-ITS and the lane change which is important for evaluating the safety benefits of the C-ITS. In this paper, the sample odds ratio equals the ratio of the sample odds in the two rows . As mentioned in the above subsection, the observed cell counts are obtained by summing the output probabilities of the C-ITS and the non C-ITS examples using the RF. Then using the permutation test, we construct the null hypothesis. Consequently, we calculate the p-value of the observed sample odds ratio and drive the conclusion about the direction of the association between the C-ITS and the lane change.

Permutation test

The basic idea of the permutation test is randomly rearranging the observed sample to generate the sampling distribution of the test statistic under the null hypothesis. In the context of this paper, the null hypothesis is the lane change and C-ITS are independent. Under this hypothesis, all possible permutations are equally likely, and the Pearson chi-squared statistic is computed for one thousand of permutation to estimate its sampling distribution under the independence of lane change and C-ITS. Below are the steps to estimate the p-value using the permutation test:

1. Given the RF outputs of C-ITS and Non-C-ITS examples (original data), construct the 2x2 contingency table;
2. Compute the original test statistics using the original 2x2 contingency table;
3. Randomly permute RF outputs, where we randomly move RF outputs among the C-ITS and Non-C-ITS as shown in Fig 4;
4. Construct the 2x2 contingency table using the permuted data—notice that columns and rows sums are not changed (i.e. marginals are the same as the original data);
5. Compute the test statistics using the 2x2 contingency table of the permuted data;
6. Repeat steps 3, 4, and 5 large number of times to get the distribution of the test statistics under the null hypothesis.

Finally, we compare the original test statistic to the estimated null distribution to estimate the p-value as will be explained in the experimental work section.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Illustration of the permutation test.

https://doi.org/10.1371/journal.pone.0229289.g004

Experimental work

This section is divided into two subsections. In the first subsection, we describe the RF classifier training. In the second subsection, we use the classification output of all examples in the dataset to test the proposed test at different sample and effect size.

Classification

The time series output of the four selected sensors is used to extract the features needed to train the RF. In this paper, we investigated two sets of features. The first features set is based on the calculation of the summary quantities shown in Table 1. These features are calculated using a window of the time series so it called “time-domain features”. We should highlight that this window includes the lane change event if it is corresponding to a lane change example. The second features set is the single side spectrum of a window of the time series around the lane change event.

The goal of the first set of experiments is to find a good window size (2w+1) for each of the features set. In order to do so, we used RF with 100 trees. We changed w from 2 to 20. This corresponds to a window size from 0.4 sec to 4 sec. We used leave-one-out-cross-validation to divide the data into a training set and a testing set. Where in each fold we keep all the examples belonging to one trip as a test dataset and all other trips are used as the training dataset. The training dataset is used to build the RF model which is tested using the unseen testing dataset. Then the classification outputs for the examples in the testing trip are saved. For each trip, we repeat this sequence of training-testing sequence and save the classification outputs. Finally, the classification outputs of all trips (i.e. the whole dataset) and the ground truth labels are used to calculate the True Positive Rate (TPR) and the False Positive Rate (FPR) where lane change is coded as positive. As shown in the top panel of Fig 5, the best RF using the time domain features is at = 16, w = 14 or w = 18 which is corresponding to 3.2 sec, 3.4 sec, and 3.6 sec respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The Receiver Operator Curve (ROC) of the RF using different window size where the top panel is the ROC of the RF using the time domain features and the bottom panel is the ROC of the RF frequency domain features.

https://doi.org/10.1371/journal.pone.0229289.g005

The bottom panel in Fig 5 shows the best RF using the frequency domain features is at w = 16, w = 14, w = 18, or w = 20 which is corresponding to 3.2 sec, 3.4 sec, 3.6 sec, and 4 sec respectively. It seems the frequency domain features are better than the time domain in terms of sensitivity. In addition, we decided to set w = 15 which equals 3 seconds window. We choose 3 seconds window because it has a relatively good TPR and FPR. In addition, because we are interested in the early lane change a 3 seconds window after the onset of the C-ITS warning is a reasonable choice, especially if we consider that the 85^th percentile of the perception reaction time (PRT) of the drivers is in the range of 1.1 to 1.3 seconds [18, 19].

Statistical testing

In this subsection, we show how the power of the adapted test change with the sample and effect size. We defined the effect size as and the sample size is set equal to 4M. The dataset has 3,994 lane changes examples and almost 8,000 lane-keeping examples. In addition to the ground truth labels of these examples, we have the hard decision and soft decision classification output from the RF. We set up the experiment as shown in Table 3. We assume the sample size of the Non-C-ITS is 2M where randomly draw M lane change examples and M lane keeping examples without replacement. In the case of the C-ITS row, we randomly draw M+k lane change examples and M−k lane keeping examples without replacement. This makes the C-ITS examples equal the Non-C-ITS examples equal 2M. Notice that, for the randomly chosen examples, if we summed the ground-truth label, it would give us the numbers shown in Table 3. However, if we summed the hard decision or soft decision outputs of the RF of these examples, they would have different numbers than those shown in Table 3 due to the inaccuracy of the classifier.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. The ground truth 2x2 contingency table used in this subsection.

https://doi.org/10.1371/journal.pone.0229289.t003

The randomly selected subset of examples are used to estimate the p-value of the chi-square test as described in the following steps:

1. The constructed 2x2 contingency table from soft decision output of the RF is used to calculate the original chi-square statistic (Stat^org);
2. Randomly permute the subset of the examples as described in the Methods section;
3. Construct the 2x2 contingency table using the permuted data;
4. Calculate the and save the chi-square statistic of the permuted data (Stat^perm);
5. Repeat steps 2,3 and 5 B times, where B = 1000;
6. Calculate the where I is the indicator function.

We repeat the sequence of examples of the random draw, 2x2 contingency table construction, permutation test, and p-value calculation 1,000 times. Then we count the number of times we detected significant differences between C-ITS and Non-C-ITS given that there is a difference between the two groups. Besides, we count the number of times, we fail to detect significant differences between C-ITS and Non-C-ITS, given that there is no difference between the two groups. These counts are divided by 1,000 to estimate the pr(reject H₀|H₁ is true) and pr(accept H₀|H₀ is true) respectively.

The following figures show the pr(reject H₀|H₁ is true) and pr(acceptH₀|H₀ is true) at different M∈{250,500,1000,1500} and different effect size ∈[−0.2,0.2]. In addition, we tried positive and negative k value in order to show that the ground truth odds ratio and the predicted odds ratio are in the same direction, as shown in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6.

The pr(reject H₀|H₁ is true) and pr(accept H₀|H₀ is true) at (a) M = 250, (b) M = 500, M = 1000 and M = 1500.

https://doi.org/10.1371/journal.pone.0229289.g006

In addition to the pr(reject H₀|H₁ is true) and pr(accept H₀|H₀ is true) shown in the above figures, the probability of correct odds ratio are shown as well. The above curve shows only the probability of the ground truth odds ratio and the predicted odds ratio are both greater than one, or less than one without any statistical testing.

We conducted the following steps to test for the odds ratio:

1. The constructed 2x2 contingency table from soft decision output of the RF is used to calculate the original statistics ;
2. Randomly permute the subset of the examples as described in the methods section.
3. Construct the 2x2 contingency table using the permuted data;
4. Calculate the test statistics of the 2x2 contingency table of permuted data and save it.;
5. Repeat steps 2,3 and 5 B times, where B = 1000;
6. Calculate where I is the indicator function.

We repeat the sequence of examples of the random draw, 2x2 contingency table construction, permutation test, and p-value calculation 1,000 times. Then we count the number of times the odds ratio is significantly different from one (i.e. C-ITS and lane change are dependent). Additionally we count the number of times we fail to reject the null hypothesis given that there is no difference between the two groups. These counts are divided by 1,000 to estimate the pr(Correct association direction|H₁ is true) and pr(Correct association direction|H₀ is true) respectively, as shown in Table 4

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. The pr(Correct association direction|H₁ is true) in the green cells and pr(Correct association direction|H₀ is true) in the blue cells.

https://doi.org/10.1371/journal.pone.0229289.t004

As shown in Table 4 above, pr(Correct association direction|H₁ is true) is high at the smallest sample size and reach one as the sample size gets bigger. In addition, the pr(Correct association direction|H₀ is true) is very high for all tested sample sizes at 0.02 effect size.