3.1 Data characterization of VLPD

VLPD contains information such as license plate numbers, the time of capturing images, and the location, vehicle type, and recorded image, which are generated by the HD equipment deployed at traffic checkpoints. Traffic checkpoint data has the following important characteristics: 1) It is massive: As each vehicle passing through a traffic checkpoint generates a data record, given the thousands of checkpoints with recording equipment throughout a city, a large amount of passing traffic data is generated every moment. With the passage of time, the accumulated data volume becomes larger and larger. 2) The data forms a time-series: Traffic checkpoint data are continuously arranged by time, where the moment of the vehicle’s appearance and the vehicle’s unique identification information (license plate number) can be used to uniquely define information about a passing vehicle. 3) It is unevenly distributed: Influenced by various factors such as morning and evening travel peaks, holidays, and weather, the distribution of data along the time axis has an obvious unevenness.

In order to ensure fast access to large-scale license plate data on a big data platform, we first need to solve the problem of access hotspots that are caused by the temporal character and unevenness of the temporal distribution of the data.

3.2 Jump hash consistency algorithm

Based on the preceding analysis of the data characteristics of VLPD, the combination of the moment a vehicle passes through a certain checkpoint and the vehicle’s license plate number (moment + license plate number) can be used to uniquely determine any piece of VLPD. Due to many factors such as weekday travel mornings and evening peaks, holidays, and so forth, the data is unevenly distributed in time. Based directly on the moment + license plate number combinations, it is easy to trigger data access hotspots, which result in load imbalances and seriously affect the performance of data reads and writes.

In the following, we assume that N slots are preset in each cluster of a big data platform to store the VLPD and that the slots are evenly grouped in order, corresponding to the physical servers (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Relationship of logical slots and physical machines.

https://doi.org/10.1371/journal.pone.0288427.g001

Achieving a uniform distribution of data across the physical clusters and ensuring load balancing during writes and queries is equivalent to the equal probability that each piece of VLPD is located in any one slot.

Letting key uniquely define a row of the VLPD and letting n denote the slot number where a piece of data is located, any target slot number n must satisfy the following equation: (1)

That is, for any key, the probability that n is distributed in the target slot interval [0,N-1] is 1/N.

To ensure that Eq (1) holds, we require a function f such that: (2)

That is, for any key, there is a unique slot numbered n among the total slots N such that the probability of key being located in slot n is 1/N.

Our implementation of the function f uses the jump hash consistency algorithm (JHCA) [32,33]. JHCA is a hashing algorithm for data distribution that maps keys into a defined range of integer values using a hash function, and we map the integer value to a predefined set of servers, thus achieving load balancing and efficient of data access.

Assuming that the final hash function that satisfies balancing and consistency is ch(key, buckets), the core idea of the hash consistency algorithm can be described by the following recursive relation:

• When n = 1, all keys are to be mapped into the same bucket, i.e. ch(key, 1) = 0.
• when n = 2, K / 2 keys need to be mapped into each of the two buckets for uniformity (where K is the total number of keys), so K / 2 keys need to be remapped.
• ……
• When the number of buckets increases from n to n + 1, there are K / (n + 1) keys that need to be remapped.

Based on this algorithmic idea, the pseudo-random numbers generated by the linear congruence generator (LCG) can be used to determine which keys are remapped into new buckets [34]. As the number of buckets n changes from 1 to the maximum value, fewer and fewer key redistributions need to occur. That is, there is no need to determine whether to migrate the key for each value of n, but rather a need to jump forward, and this is done by the JCHA. This algorithm can be described using the Java language as follows.

Algorithm 1;: Jump Hash Consistency Algorithm

1 private static int ch(long key, int buckets) {

2   long b = -1;

3   long j = 0;

4   while (j < buckets) {

5    b = j;

6    key = key * 2862933555777941757L + 1;

7    j = (long) ((b + 1) * (double) (1L << 31) / ((double) ((key >>> 33) + 1)));

8   }

9   return (int)b;

10 }

In the code, the ch method receives a long integer key and an integer buckets, and it returns the mapped value of the key in the buckets. If the buckets are defined as the preset slots in the big data platform that are used to store the VLPD, and the key is the unique identifier of any piece of the VLPD, then the ch function’s return value is the slot number where the piece of data is located.

The JHCA has the advantage of being efficient and scalable. For a given key, it requires only one hash calculation to determine the mapping server for that key, rather than multiple calculations like the traditional hashing algorithms. It has been theoretically and empirically shown that the time complexity of the JHCA for computing slots is O(ln(n)). It also has high scalability with relatively little data movement when adding or removing servers.

3.3 Storage structure design

HBase is a Hadoop-based columnar storage system designed to handle large-scale unstructured and semi-structured datasets. HBase manages huge datasets as key–value tuples and uses key–value mapping to achieve highly consistent, elastic, real-time read and write access to big data. Because of its open source, mature development, and wide application in various industries, HBase has been widely used in the field of big data platforms.

The HBase database is an open-source implementation of Google’s paper "Bigtable: A Distributed Storage System for Structured Data" [35]. The row key of an HBase table is the focus of the design, which determines how the data is stored and accessed. Row keys should be unique, stable, sortable, and divisible. It is also necessary to avoid row keys that are too long, too complex, or contain sensitive information. Considering many factors such as traffic car data characteristics and application requirements, the designed HBase table structure is shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Storage structure for VLPD on HBase.

https://doi.org/10.1371/journal.pone.0288427.g002

Row keys: The row key design is the core of the HBase table structure design. The row key for storing VLPD consists of three parts: a slot number, the vehicle’s time of passing the checkpoint, and the license plate number. The slot number is used as the header of the row key to indicate where this data is located, and the combination of the time the vehicle passed through the checkpoint and the license plate number is used to uniquely identify a passing vehicle’s information.

Column families: Each row key corresponds to two column families: One column family holds the attribute information, and the other holds the image that is captured when the vehicle passes by.

3.4 Row key generation algorithm

Considering the time-series nature of the VLPD and the uneven distribution of data in the time series, the VLPD as arranged in the time series is processed in minute slices. That is, a vehicle’s time of passing through the traffic checkpoint expressed as a minute is converted into a timestamp that is used as the first parameter key for JCHA, and the number of preset slots in the big data platform is used as the algorithm’s second parameter buckets to calculate the slot number corresponding to the vehicles passing during the minute slice. The result of the calculation is used as the row key header for all of the data in the time period, and the combination of the time of the vehicle passing through the checkpoint and the captured license plate number forms the unique row key for the data saved in the database.

The slot calculation algorithm is described by the following Java code.

Algorithm 2;: Row Key Generation Algorithm

1 private byte[] generateRowkey(JSONObject row) {

2   //Accurate time to the minute of the vehicle passing through the checkpoint

3   String slotString = row.getString("JGSK").substring(0, 16) + ":00";

4   //Convert time to a long timestamp

5   long timeStamp = getTimeStamp(slotString);

6   int slotIndex = ch(timeStamp, buckets);

7   String rowKey = String.valueOf(slotIndex+10000).substring(1) + "|"

8     + row.getString("passTime") + "|" + row.getString("plate");

9   return rowKey.getBytes(StandardCharsets.UTF_8);

10 }

In the code, row is the VLPD data in JSON object format, and buckets is the number of slots preset in the big data platform to save the data.

3.5 VLPD storage on HBase

HBase uses its core component, the region servers, to manage data access. Each region server manages multiple regions, each of which represents a partition of an HBase table. The region server manages regions according to the range of the row keys in an HBase table, and when the table data grows, HBase splits the region into two or more sub-regions in accordance with certain rules to achieve load balancing. The relationship between rows, regions, and region servers can be expressed in dictionary order by the HBase region server architecture, as shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. HBase region server’s architecture.

https://doi.org/10.1371/journal.pone.0288427.g003

Since there are 1440 minutes in a day, we can set the number of slots where the traffic cars’ plate data is stored to the constant 1440. The slot identifiers located at the head of the row key vary continuously from 0000, 0001,… to 1439. The traffic car data sliced by the minute of a day will be randomly distributed in these slots with equal probability.

In HBase, pre-partitions (regions) on the server are generated using the following script:

$>create ‘Traffic, {NAME=>A’, NAME=>’I’, VERSIONS=>1, SPLITS=>[‘0060’,‘0120’,‘0180’,…,‘1380’].

Executing this script in the HBase shell creates a large table named “Traffic” in the database. The large table has two column families (A and I), which are used to store the attribute information and the captured images, respectively, for the passing cars. At the same time, the table is pre-partitioned to create 24 regions, and HBase will save the data to the corresponding regions based on the slot numbers located at the head of the row keys.

4.1 Data distribution range location

VLPD applications can be summarized by two categories: statistical analyses of vehicle traffic and vehicle tracking. For the statistical analysis of vehicle traffic, the general demand is to compute how many times vehicles pass through a certain traffic checkpoint, on a certain road, and in an entire region. For vehicle tracking, the general demand is to extract location information about all checkpoints that vehicles have passed within a certain time period from the massive VLPD. Both the statistical analyses for multiple conditions and the path tracking for vehicles need to be meaningful within a specific time frame. In other words, for all kinds of VLPD applications, the time factor is a necessary condition.

Based on this categorization, the time period information in the query request for all VLDP application problems can be used to determine the distribution range of the query data in the line key sequence. After receiving a user query request, the client determines the data distribution range as follows:

• Extract the time span information;
• Slice the time period and calculate the slot where the slice is located;
• Combine the slot number and time slice to form a line key prefix;
• Build a scan-task stack in which each element (scan) contains a row key prefix.

Fig 4 shows a schematic representation of the position of the slice data in the row key sequence for a certain time range.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The location of the slice data within the time range.

https://doi.org/10.1371/journal.pone.0288427.g004

As can be seen in Fig 4, the data is discrete and evenly distributed over the entire sequence of row keys. Using the row key prefixes defined by the combinations of slot numbers and time slices, the distribution range of the data can be precisely defined. The data scan range generation algorithm can be described by the following Java code.

Algorithm 3: Data Distribution Range Generation Algorithm

1. private Stack<Scan> getScans(Long fromStamp,Long toStamp){

2.   Stack<Scan> scans = new Stack<>();

3.   for(Long l = fromStamp;l < = toStamp;l + = 60000){

4.    Scan scan = new Scan();

5.    Timestamp timestamp = new Timestamp(l);

6.    DateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

7.    String dateStr = sdf.format(timestamp);

8.    String prefix = String.valueOf(ch(l,numServers)+10000).substring(1) +"|" +

        dateStr.substring(0,16);

9.    scan.setRowPrefixFilter(prefix.getBytes());

10.    scans.push(scan);

11.   }

12.  return scans;

13. }

However, since the data is discretely distributed over the row key sequence, this creates a problem for data scanning queries. Obviously, to improve the query efficiency, the entire sequence of row keys should not be scanned from beginning to end, but rather, the scan should jump forward (as shown in Fig 4). For this reason, there is a need to design a parallel algorithm that can support jump scanning in order to improve the scanning efficiency.

4.2 Multi-threaded sliding window algorithm for a parallel request

Big data platforms are equipped with extreme parallel data processing capabilities. Using parallel algorithms to process multiple scan tasks can significantly improve the query efficiency. Suppose that T scan tasks have been constructed based on the time period in the query request and that these tasks are stored in the scan-task stack. The algorithm for the parallel processing of the scan tasks in the stack can be described as follows:

• Initialize a process pool with a window width of n;
• For each process in the window, repeatedly perform a stack-out operation on the scan-task stack and the corresponding scan operation;
• End the thread when the pop() return value is empty;
• Print the scan result.

The algorithm execution process is illustrated in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Multi-threaded sliding window for parallel requests.

https://doi.org/10.1371/journal.pone.0288427.g005

In this algorithm, the thread count (the window width) needs to be determined jointly with the client-side performance and the number of data node servers on the server side (empirical parameters for determining the window width value are given in the experimental section 5). As the number of scan tasks in the stack keeps decreasing, the process pool window keeps sliding to the right until the stack is empty, ending all processes and completing the query. The algorithm can be further described by the following Java code.

Algorithm 4: Multi-threaded Sliding Window for Parallel Request Algorithm

1. private Result scanConcurrentlySlidWindow(Table table,Stack<Scan> scans) {

2.   Result result = new Result();

3.   ExecutorService executor = Executors.newFixedThreadPool(numThreads);

4.   CountDownLatch latch = new CountDownLatch(numThreads);

5.   for(int I = 0;I < numThreads;i++){

6.    executor.submit(()->{

7.     ResultScanner scanner;

8.     while (!scans.empty()){

9.      Scan scan = scans.pop();

10.       scanner = table.getScanner(scan);

11.       for (Result rs:scanner){

12.        //get the result

13.        result.merge(rs);

14.       }

15.       scanner.close();

16.      }

17.      latch.countDown();

18.     });

19.   }

20.   latch.await();

21.   executor.shutdown();

22.   return result;

23. }

In this code, the method scanConcurrentlySlidWindow receives two parameters, an HBase large table object and a scan-task stack, and it returns the query result after the method execution is completed. In the method, the for-loop initializes numThreads threads for the parallel processing of the scan tasks. When there are no new tasks to be processed, the threads are terminated.

5.1 Physical hardware and application processes

The physical servers in the cluster are divided into two categories, NameNode servers and DataNode servers. Servers running NameNode are allocated more memory and fewer storage resources, while servers running DataNode are allocated more storage and fewer memory resources. The specific hardware resource allocations are shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Experimental hardware resource allocation.

https://doi.org/10.1371/journal.pone.0288427.t001

5.2 Analysis of even data distribution

The uniformity of data distribution can be measured by the variance of the number of rows distributed on the region server. About 29 million rows VLPD for the entire month of April 2021 in the Tianjin Sino-Singapore Eco-city were selected as the experiment sample, and the total amount of data in the slots was obtained by counting the number of data rows with the same slot number in the row key header. The distribution of the VLPD clustered on the region servers is shown in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Distribution of VLPD on region servers.

https://doi.org/10.1371/journal.pone.0288427.t002

Using the assigned region number on the region server as the horizontal axis and the number of rows of VLPD saved on the different region server as the vertical axis, a histogram of the data on the region server was drawn (as shown in Fig 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Data distribution on region server.

https://doi.org/10.1371/journal.pone.0288427.g006

We used the variance equation (3) to compute the variance value, and the variance value(40.3) is very low compared to the average value(1216.1).

It can be seen that the data pieces of VLPD, which has been divided into slices by "minutes", have been evenly distributed over the region server. The read/write hotspot problem has thus been effectively avoided by the storage model based on JHCA.

5.3 Analysis of data retrieval efficiency

The analysis of the comprehensive query conditions of VLPD can all be achieved by scanning the data within a certain time period. Therefore, the system’s efficiency in processing data retrieval query requests can be measured by the length of time needed for the data scanning operations made by the cluster. Based on the data access algorithm model described in this paper, the factors affecting the data scanning efficiency include both the parallel scanning window width (the number of parallel request threads) and the query data time range.

(1)Analysis of scan time with different window widths.

Using sliding window widths ranging from 1 to 12, VLPDs with time spans of 1, 2, and 3 days were scanned separately, and the time taken to complete each scan was recorded. The experimental results are shown in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Impact of concurrent scan operations on query efficiency.

https://doi.org/10.1371/journal.pone.0288427.g007

The experiments show that when the sliding window width is set to 1–3, the time used for scanning varies the most; and when the sliding window width is greater than 3, the effect on the scanning time is not very obvious.

The window width of 3 coincides with the number of data nodes in the experimental cluster. Optimal performance results can be obtained by setting the width of the moving window to the number of physical data node servers in the cluster when implementing access to large data from traffic car crossings using the traffic car data algorithm model described in this paper.

(1) Comparative analysis of query efficiency for different data scales.

A comparative analysis of the time taken for the same query condition with different data sizes can visually reflect the algorithm’s execution efficiency. This paper compares the differences in query efficiency between the two types of storage solutions by storing the day-by-day cumulative sample VLPD in the HBase database and in the popular PostgreSQL database under the condition of performing the same query function operation. In the experiments, we use the data for the April 2021 VLPD in the China-Singapore Tianjin Eco-city as the experimental sample data, and we determine the time spent on querying the vehicle with license plate number "津 AYH6**" on April 1 by accumulating the data storage size of the PostgreSQL database and the HBase database day by day. For the PostgreSQL database, we used the following script:

select * from vehicle t where t. "HPHM" = ’津AYH6**’ and t. "JGSK" BETWEEN to_timestamp(’2021-04-01 00:00:00’,

’ YYYY-MM-DD HH24:MI:SS’) and to_timestamp(’2021-04-01 23:59:59’,’YYYY-MM-DD HH24:MI:SS’);

to obtain the query result(Considering the privacy issue, we hide the last two digits of the license plate number in the sql script). For the HBase database, we used the algorithm described in this paper to perform a scan operation on the same volume data size. The query result was obtained via a filter. The experimental results are shown in Fig 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Comparative analysis of server performance using different solutions.

https://doi.org/10.1371/journal.pone.0288427.g008

The experiments show that both PostgreSQL and HBase return the same result with a total row count of 28. For the PostgreSQL database, the time used for querying increases linearly with the accumulation of the data volume day by day, so that the time complexity is O(n), while for the HBase database, which accesses data using the algorithm described in this paper, the data size has no significant effect on the query efficiency, giving a time complexity of O(1). Although under the condition of a small data size (cumulative data for 8 days, fewer than 1×10⁷ rows), use of the PostgreSQL database to store the VLPD provides better performance. However, as the accumulated data in the database increases, the query time used by the PostgreSQL database is significantly higher than that used by the HBase database, and eventually it cannot complete the demand for real-time data retrieval.

5.4 Characteristics comparison

By comparing and analyzing the research results of current VLPD access models, this paper summarizes the characteristics of different models from the perspectives of whether they support multi-conditioned queries, space usage, memory consumption, horizontal scalability, and real-time access. Compared with models based on auxiliary indexing technology, MapReduce, and Spark, JHAC based VLPD access model has the characteristics of fully supporting multi condition comprehensive queries, saving storage space, reducing memory consumption, supporting horizontal scalability, and meeting real-time data access. Table 3 presents the comparative analysis results of the characteristics of different models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics comparison among different models.

https://doi.org/10.1371/journal.pone.0288427.t003