Introduction

The use of efficient algorithms for the high efficiency video coding (HEVC) standard to achieve high compression has significantly increased computational time of the resultant encoder [1–3]. The HEVC comprises the ME, motion compensation, 2D-DCT, inverse 2D-DCT, quantization, inverse quantization, and context adaptive binary arithmetic coding (CABAC). Among these units, both ME and 2D-DCT are considered computationally expensive [4–7]. Fig 1 shows the basic block diagram of HEVC encoding process [1–5], where “Q” is the quantization process, “Inv Q” is the inverse quantization process and “Inv DCT” is the inverse 2D-DCT process.

The ME exploits the temporal redundancy across the input images of a sequence to achieve compression. In this process, a part or the macroblock in the current frame is searched in a region (search area) in the earlier frame to find the best match of current macroblock. The criteria used for best match is known as sum of the absolute difference (SAD) metric defined as, (1) where, the parameters: curr and ref represent the current and reference frames, respectively.

The m and n are coordinates of the current frame macroblock; and p and q are coordinates of the reference frame macroblock compared to current frame macroblock. The size of macroblock is (16×16) pixels [8, 9]. In HEVC, sizes of macroblocks can be (MB_SIZE×MB_SIZE), where the MB_SIZE ϵ [8, 16, 32, 64]. In this work, value of MB_SIZE is chosen to be 16, because small macroblocks capture motion of small objects more efficiently leading to better compression. The motion vector (MV) is defined as location of the best macroblock in the search area with minimum SAD and is defined as (2) where arg in (2) stands for the arguments, i.e., arguments of minimum SAD, which are coordinates (p, q) of the minimum SAD [8]. On the other hand, the process of ME is based on the rate distortion optimization in HEVC. It consists of the minimization of following cost metric J [8], which is defined as (3) (4)

The second term in Eq (3) is the cost of MV obtained, where the parameter λ is the Lagrange multiplier whose value depends upon the chosen quantization parameter. The R(MV−MV_P) is the number of bits required to encode MV difference (MV−MV_P) and MV_P is the predicted MV. Similarly, DCT [7, 10] is normally used in removing the spatial redundancy and can be considered as real part of the discrete Fourier transform (DFT). Mathematically 2D-DCT is computed as follows: (5)

Here, 0≤p≤M−1; 0≤q≤N−1;

In this case, M is taken equal to N. The parameters m and n are the spatial coordinates, while p and q represent the discrete frequencies. The 2D-DCT can be implemented as transform of the transform using separable property, because this technique results in reduced computational complexity. It implies that the 1D-DCT is first applied along the rows of a 2D data-matrix (E) and then 1D-DCT is applied along the columns of resulting matrix. Mathematically, it is computed as follows: (6) where C is a matrix of the DCT cosine values and C^T is transpose of C.

Exploiting inherent parallelism in a target application may potentially result in reducing the overall computational time. The single-core CPUs supporting multithreading and caches can reduce execution time of an application up to some extent thereby running multiple threads in parallel. However, when the real time constraint on an application is strict, then one must resort to more robust and resilient schemes. Literature reveals different methodologies are utilized to boost up performance of the ME/2D-DCT tasks at the algorithmic and architectural levels [8, 11, 12]. At the algorithmic level, designers often try reducing computational complexity of the relevant algorithm [6, 8, 10], but for architectural level, designers generally implement the parallel/pipelined dedicated VLSI architectures, such as the ASICs, SoC, and/or FPGAs [8, 11, 12].

In addition, the relevant research community has also utilized the single instruction multiple data (SIMD) vector coprocessors and multiprocessing in this concern [9, 13]. Several fast ME algorithms have been devised to alleviate computational complexity of the ME units in contemporary video encoders [6, 8]. Similarly, fast DCT computational algorithms having reduced number of additions and multiplications have also been developed [10, 11]. Besides, utilization of the graphics processing units (GPUs) to accelerate the video encoders’ speed is also now a hot research arena [14]. Instead of dedicating transistors to implementing large caches and efficient control flow units for CPUs, more transistors are dedicated for data processing units in GPU. Due to this reason, GPUs can efficiently execute large data in parallel. Compared with generic type CPUs, GPU is surely the best choice when accelerating a computationally complex application is needed. It is because CPUs perform sequential tasks quickly and the GPUs use parallel processing while computing tasks simultaneously with greater efficiency and speed.

In general, a typical GPU consists of the highly threaded streaming multiprocessors (SMs). Each SM consists of many streaming processors (SPs), arithmetic units (fully pipelined integer units and single precision/double precision floating point units), control logic, and instruction cache shared by SPs [14, 15]. Currently, each GPU has a graphic double data rate (GDDR) DRAM known as the global memory, which is available off-chip. In addition, a SM has a shared memory, an L1 cache, a large capacity register-file, warp scheduler(s), instruction dispatch unit(s), special function units (e.g., sine, cosine, log, and exp), load/store units shared by SPs. Shared memory and register file are partitioned among many threads. There are also memory controllers and an L2 cache, which is off-chip and shared by all SMs [14]. Though there exists a local memory, which is based on the global memory: L1 and L2 caches. The Nividia GeForce GTX 1080 GPU (based on Pascal architecture and implemented in 16 nm FinFET process) have SPs capable to run @1.6 GHz. It can provide overall throughput of 8873 Giga floating point operations per second (GFLOPs) [14]. The high-speed DRAM has 256-bit interface and can run at a data rate of 10 Gbps.

A thread is basically a program or a part of the program. In addition, a thread block consists of multiple threads oriented both horizontally and vertically. Similarly, a grid of thread blocks consists of multiple thread blocks oriented both horizontally and vertically. An SM can execute multiple threads or thread blocks concurrently depending upon the available resources [14, 16–22]. All threads inside a thread block can cooperate with each other through a barrier synchronization mechanism. They also share data using the shared memory. It is also found that threads belonging to different blocks cannot communicate with each other. All threads, whether belonging to the same or different block can share the global memory. At the hardware level, threads of a block are divided into groups of threads, where each group consists of 32 consecutive threads known as warp. All threads of a warp execute the same instruction at the same time in a single instruction multiple thread (SIMT) manner.

There is a fast context switching between warps. Depending upon resources, such as amount of the shared memory and register file etc., there are certain number of warps that can reside inside an SM simultaneously. However, there must be enough warps to achieve thread occupancy and hide memory latency [14, 16–22]. If consecutive threads access consecutive locations of global memory, then it leads to a coalesced memory access. In addition, the instruction level parallelism is also supported by SMs. No doubt, all these highlighted factors lead to high throughput if implemented [14, 16–22].

Moreover, it is also worth mentioning that the NVIDIA Corporation has also developed a parallel computing and programming platform known as the Compute Unified Device Architecture (CUDA) [23, 24]. It is a parallel computing platform and application programming interface (API), which allows software to use certain types of the GPUs for general purpose processing, i.e., an approach called general-purpose computing on GPUs (GPGPU). The researchers and developers can easily port their sequential program to GPUs using the CUDA parallel constructs for acceleration [14, 16–22]. After highlighting GPU architecture, the following text reviews contemporary work on the video encoders using GPUs.

Proposed methodology

The flowchart for the FS ME process (Algorithm 1) can be seen in Fig 2. The parameter frame_num denotes the number of frames. The MB_Y and MB_X denote the number of vertical macroblocks and horizontal macroblocks, respectively. Similarly, the SA_Y and SA_X denote the vertical and horizontal dimension of the search area. Lastly, the CMV represents candidate motion vector (MV). There are two more inner most loops (not shown) for computing the SAD metric sequentially. This is a sequential algorithm involving billions of the arithmetic operations and memory accesses corresponding to search area range of [32 x 32] and the UHD sequences [6, 8].

Algorithm 2 shows pseudocode of FS ME algorithm in which the second and third for-loops of Algorithm 1 are parallelized. For this purpose, a 2D grid of the global threads, with dimensions (MB_X×MB_Y) is created in the CUDA C [14]. Depending upon the SMs available in GPU, these threads will be executed concurrently and will execute the following code in parallel. Fig 3 shows block diagram for Algorithm 2, explaining the parallelization provided by the GPU threads for FS ME process.

Algorithm 2: Pseudocode for the macroblock level parallel FS ME algorithm.

for i in 1 to frame_num loop

 {

 • Load frames into GPU global memory.

 • 2D grid (MB_X×MB_Y) of global threads executes the following instructions simultaneously.

    for l in 1 to SA_Y loop

   for m in 1 to SA_X loop

    {

     ■ Compute SAD corresponding to CMV(l,m)

     ■ Compute minimum SAD and the corresponding MV(l,m)

    }

 }

The pseudocode in Algorithm 3 is same as that of Algorithm 2, except the shared memory is used to store the current macroblock (CMB) and search area (SA) for fast access and efficient data reuse. Shared memory is the fastest memory after register-file in GPU. Corresponding data is loaded into the corresponding shared memories in a coalesced manner. Fig 4 shows Block diagram for Algorithm 3, explaining the parallelization provided by the GPU threads for FS ME process. macroblock level parallelization has been depicted.

Algorithm 3: Pseudocode for Macroblock level parallel FS ME algorithm with shared memory.

for i in 1 to frame_num loop

 {

   •Load frames into GPU global memory.

   •2D grid (MB_X×MB_Y) of global threads is executing the following instructions simultaneously.

   •Load corresponding CMB and SA in the corresponding shared memories.

 for l in 1 to SA_Y loop

    for m in 1 to SA_X loop

   {

    • Compute SAD corresponding to CMV(l,m).

    • Compute minimum SAD and the corresponding MV(l,m).

   }

 }

Algorithm 4 shows pseudocode for the FS ME algorithm in which second, third, fourth, and fifth for-loops of Algorithm 1 are parallelized. For this purpose, a 2D grid of the global threads having dimensions (MB_X×MB_Y) pixels is created in CUDA C. Similarly, a 2D thread block of the local threads with dimensions (SA_X×SA_Y) pixels per global thread is created. Macroblock level loops (second and third loops) are parallelized using global threads while the search-area level loops (fourth and fifth loops) are parallelized using local threads. Search areas (SA) and current macroblocks (CMB) are loaded into corresponding shared memories in a coalesced manner. A barrier mechanism for synchronizing the local threads is also used. Fig 5 shows block diagram for Algorithm 4, explaining the parallelization provided by the GPU threads for FS ME process. SA level and Macroblock level parallelization has been depicted.

Algorithm 4: Pseudocode for search area and Macroblock level parallel FS ME algorithm.

for i in 1 to frame_numb loop

 {

  • Load frames into GPU global memory.

  • 2D grid (MB_X×MB_Y) of global threads is executing following instructions simultaneously.

  • Load corresponding CMB and SA in the corresponding shared memories using four pixels per local thread for SA and one pixel per local thread for CMB.

  • 2D block (SA_X×SA_Y) of local threads executes the following instructions simultaneously.

  • Each local thread computes one SAD corresponding to one CMV.

  • Synchronize threads.

  • The corresponding main global thread executes following instructions:

    for l in 1 to SA_Y loop

      for m in 1 to SA_X loop

    • Compute minimum SAD and the corresponding MV(l,m).

      }

Algorithm 5 shows parallelization of the frame-level loop (first loop) of the FS ME algorithm. A 2D grid of threads with dimensions (2×MB_X×MB_Y) pixels is used, i.e., ME process corresponding to 3 frames is being executed in parallel. Here 2 ME processes are in parallel. During the first iteration of frame level for-loop, 3 consecutive frames are loaded into the device global memory. Next, with each succeeding iteration, 2 consecutive frames are loaded. Similarly, when 3 ME processes are parallelized, then 4 consecutive frames in the first iteration will be loaded into device memory. With each succeeding iteration, 3 consecutive frames will be loaded and so on. Fig 6 shows block diagram for Algorithm 5, explaining the parallelization provided by the GPU threads for FS ME process. Frame level and Macroblock level parallelization has been depicted.

Algorithm 5: Pseudocode for parallelizing frame-level loop (1^st loop) of FS ME algorithm.

for i in 1 to frame_num/2 loop

 {

  • Load frames into GPU global memory.

  • 2D grid (2×MB_X×MB_Y) of global threads executes following instructions simultaneously.

  • Load corresponding CMB and SA in the corresponding shared memories.

      for l in 1 to SA_Y loop

       for m in 1 to SA_X loop

                   {

   • Compute SAD corresponding to CMV(l,m)

   • Compute minimum SAD and the corresponding MV(l,m)

                    }

            }

Algorithm 6 shows pseudocode for the sequential SAD computation. As can be seen there are two for-loops. A (16×16) pixels macroblock has 256 absolute difference and addition operations. In addition, each for-loop has associated 3 instructions, i.e., condition checking, iteration increment, and goto instruction per iteration. So, for a (16×16) pixels macroblock, it amounts to (256×2+(16×3+3)×16) instruction executions, which equals 1328 instruction executions, where memory access instructions are excluded and though absolute difference operation is a multi-instruction operation, it is assumed here a single operation for simplicity.

Algorithm 6: Pseudocode for the SAD computation sequentially.

 for l in 1 to MB_SIZE loop

  for m in 1 to MB_SIZE loop

   • SAD1 = SAD1+abs(curr(l,m)−ref(l,m))

Algorithm 7 shows pseudocode for parallelization of the SAD computation by unrolling the inner for-loop and multithreading the external for-loop. Assuming MB_SIZE equals 16, block of (16×16) pixels local threads is executing the instructions shown. The x and y are the horizontal and vertical coordinates of thread block, respectively. AD is the absolute difference and is a 2D array mapped to shared memory. A total of 256 absolute operations are computed in parallel. Next, these threads are synchronized. Then, 16 threads compute the SADs, where each of 16 threads compute SAD corresponding to the macroblock row as shown by the shared memory array variable SAD1. Threads are again synchronized, and at the end, a single thread computes 16 resultant SADs. Now maximum operations per thread are reduced significantly, excluding conditional instructions for threads. The maximum operations per thread per macroblock are now approximately (1+16+16), which equals to 33. As before the memory access instructions are excluded. There is further possibility of reducing these instructions per thread, e.g., by using 32 threads instead of 16 threads for computing SAD, where (16×16) absolute differences block is broken down into four (8×8) absolute differences blocks, and each (8×8) block is being added by 8 threads. Next, threads are synchronized.

Algorithm 7: Pseudocode for the parallelization of the SAD metric calculation.

 • Block of (MB_SIZE×MB_SIZE) pixels local threads execute the following instructions simultaneously.

  • AD[y][x] = abs(curr[y][x]−ref[y][x])

 Synchronize threads.

 Following instructions are executed by MB_SIZE threads.

  • SAD1[y] = SAD1[y]+AD[y][0]

  • - - - - - - - - - - - - - - - -

  • SAD1[y] = SAD1[y]+AD[y][MB_SIZE−1]

  Synchronize threads.

The following instructions are executed by a single thread.

  for i in 0 to MB_SIZE−1 loop

  • s = s+SAD1[i]

 return (s)

Then, again (2×2) thread block is utilized, in which each thread adds the 8 resultant intermediate SADs out of 32 resultant intermediate SADs in parallel. Threads are synchronized again, and finally, a single thread adds 4 resultant intermediate SADs. Though in this case the maximum number of instruction executions per thread are reduced to (1+8+8+4), which equals to 21 excluding memory access and conditional instructions. However, this later case has more complex coding structure and involves more conditional instructions.

After identifying 4 levels of parallelism in FS ME algorithm, i.e., frame level, macroblock level, search area level, and SAD level parallelization; this work now proceeds towards the parallelization of a recently proposed fast ME algorithm by the authors known as the EHDS ME algorithm [6]. The first step in EHDS ME algorithm is to produce multiple resolution images of the subject frames. Fig 7 shows block diagram for Algorithm 7, explaining the parallelization provided by the GPU threads for FS ME process. SAD level parallelization has been depicted.

Algorithm 8 shows pseudocode for producing multiple resolution (two) images of the frames sequentially. Here, the size of macroblock is MB_SIZE, which is 16 and MB_SIZE1 is 8. Resolution of (8×8) pixels is obtained from (16×16) pixels macroblock by averaging every (2×2) non-overlapping blocks. Similarly, the second resolution of (4×4) is obtained from (8×8).

Algorithm 8: Pseudocode for producing multiple resolution (two) images of frames, sequentially.

for y in 1 to MB_Y loop

 for x in 1 to MB_X loop

  {

    for i in 1 to MB_SIZE/2 loop

    for j in 1 to MB_SIZE/2 loop

    • MB1 = average 2 x 2 non-overlapping squares of the macroblock.

  for i in 1 to MB_SIZE1/2 loop

   for j in 1 to MB_SIZE1/2 loop

    • MB2 = average (2×2) non-overlapping squares of the macroblock1.

     }

Algorithm 9 shows pseudocode for parallelizing for-loops of Algorithm 8. For this purpose, a grid of global threads of dimension (MB_X×MB_Y) pixels is initiated. In addition, the third and fourth for-loops are parallelized using a block of local threads of dimensions (MB_SIZE/2×MB_SIZE/2). In this case, each thread will compute average of the non-overlapping (2×2) block of pixels. These local threads are also responsible for loading macroblock into shared memory in parallel. The threads are then synchronized. Following that, the fifth and sixth for-loops are parallelized using block of local threads of dimensions (MB_SIZE1/2×MB_SIZE1/2), where MB_SIZE is 16 and MB_SIZE1 is 8. In this case again, each thread computes the average of non-overlapping (2×2) block of pixels. Here, x and y are the horizontal and vertical coordinates of thread block, respectively. Fig 8 shows block diagram for Algorithm 9, explaining the parallelization provided by the GPU threads for EHDS ME process. Creation of hierarchy of multiple resolution images is depicted.

Algorithm 9: Pseudocode for parallelizing the external macroblock level for loops of Algorithm 8.

• Following instructions are being executed by a grid (MB_X×MB_Y) of global threads simultaneously.

• The following instructions are being executed by a (MB_SIZE/2×MB_SIZE/2) block of local threads.

• Load macroblock in shared memory

  

• Synchronize threads.

• The following instructions are being executed by a (MB_SIZE1/2xMB_SIZE1/2) block of local threads.

  

The pseudocode given in Algorithm 10 shows serial implementation of the EHDS ME algorithm, while the pseudocode given in Algorithm 11 shows that second and third for-loops in Algorithm 10 are replaced by 2D grid of global threads of dimensions (MB_X×MB_Y) pixel. Fig 9 shows block diagram for Algorithm 11, explaining the parallelization provided by the GPU threads for EHDS ME process. Macroblock level and SAD level parallelization is depicted.

Algorithm 10: Pseudocode for the serial implementation of EHDS ME algorithm.

for i in 1 to frame_num loop

  for j in 1 to MB_Y loop

    for k in 1 to MB_X loop

      {

       • Execute EHDS in a given search area.

       • Compute minimum SAD and the corresponding MV(l,m).

               }

Algorithm 11: Pseudocode for the parallel implementation of EHDS ME algorithm.

for i in 1 to frame_num loop

{

 • The following instructions are being executed by a grid of global threads of dimension (MB_X×MB_Y)

 • The following instructions are being executed by a block of local threads of dimension (MB_SIZE×MB_SIZE)

 • Load corresponding CMB and SA in the corresponding shared memories.

               {

  • Execute EHDS in a given search area.

  • Compute minimum SAD and the corresponding MV(l,m).

                }

   }

The SA and CMB are loaded into corresponding shared memories in a coalesced manner by the local threads. After finding parallelism at macroblock level, second parallelism exists at the SAD level, as was the case in FS ME algorithm given in Algorithm 7. The block of local threads are computing the SAD in parallel. Though frame-level parallelization can also be utilized, but utilizing all levels of parallelization did not give any significant improvement as compared to utilizing few levels of parallelization. It is because search area parallelization cannot be used together with SAD level parallelization. Second, as the number of SMs are limited, using too many threads instead creates serialization effect and hence after certain number of threads, any further multithreading does not give significant benefits.

In addition, search area CMVs in EHDS algorithm have less regularity and are not deterministic, i.e., CMV locations are dependent on the nature of images [6], hence search area level parallelization is not utilized for EHDS algorithm. Algorithm 12 shows pseudocode for serial implementation of motion compensation or reconstruction while pseudocode in Algorithm 13 shows that the first and second for-loops in Algorithm 12 are replaced by 2D grid of global threads of dimensions (MB_X×MB_Y) pixel. Corresponding motion vector is loaded into register or shared memory for fast processing and data reuse. The previous_MB is the reference (previous) frame macroblock and mv_x and mv_y are the corresponding motion vectors. Fig 10 shows block diagram for Algorithm 13, explaining the parallelization achieved using GPU threads, for motion compensation process.

Algorithm 12: Pseudocode for the motion compensation or reconstruction.

    for i in 1 to MB_Y

     for j in 1 to MB_X

      {

           for x in 1 to MB_SIZE loop

        for y in 1 to MB_SIZE loop

            •

        }

Algorithm 13: Pseudocode for parallel implementation of motion compensation or reconstruction.

• The following code is being executed by a grid (MB_X×MB_Y) of global threads simultaneously.

• Load motion vector into register or shared memory.

   for x in 1 to MB_SIZE loop

      for y in 1 to MB_SIZE loop

      •

Algorithm 14 shows pseudocode for the serial implementation of image differencing, i.e., difference between current frame and its reconstruction, while Algorithm 15 shows its parallel version. The current_MB is the current frame macroblock. It is important to mention here that two for-loops shown in Algorithms 13 & 15 can further be parallelized when local threads are utilized, but the resultant improvement is not much significant. After image differencing comes the stage of 2D-DCT. As shown in Eq 6, due to the separable property of 2D-DCT [7], it can be implemented as transform of the transform. Fig 11 shows block diagram for Algorithm 15, explaining the parallelization achieved using GPU threads, for image differencing process.

Algorithm 14: Pseudocode for the serial implementation of the image differencing.

 for i in 1 to MB_Y

  for j in 1 to MB_X

     {

        for x in 1 to MB_SIZE loop

      for y in 1 to MB_SIZE loop

       •

     }

Algorithm 15: Pseudocode for the parallel implementation of image differencing.

• The following code is being executed by a grid (MB_X×MB_Y) of global threads simultaneously.

  for x in 1 to MB_SIZE loop

    for y in 1 to MB_SIZE loop

      •

Algorithm 16 shows pseudocode for the serial implementation of 2D-DCT as transform of transform. Here 1D-DCT is applied along rows of blocks. Then in the second step, 1D-DCT is applied along columns of result, leading to 2D-DCT. The number of blocks horizontally are (2×MB_X) and number of blocks vertically are (2×MB_Y), since size of block is (8×8) pixel, which is half size of macroblock in ME process.

Algorithm 16: Pseudocode for serial implementation of 2D-DCT as transform of transform/

  for m in 1 to 2* MB_Y loop

    for n in 1 to 2* MB_X loop

      {

        • Compute 1D-DCT

        • Compute 2D-DCT

           }

Algorithm 17 shows pseudocode for the serial implementation of conventional 1D-DCT applied along the rows of data block. Here MB_SIZE1 has a value of 8, as 2D-DCT is applied over (8×8) blocks. The coeff represents the 2D matrix of DCT coefficients (cosine values).

Algorithm 17: Pseudocode for serial implementation of conventional 1D-DCT.

 for row in 1 to MB_SIZE loop

  for k in 1 to MB_SIZE1 loop

    {

     • temp = 0

    for k1 in 1 to MB_SIZE1 loop

     {

     •

     }

     • dct_1d[row][k] = temp

 }

Algorithm 18 shows pseudocode for conventional 1D-DCT applied along the columns of 1D-DCT already computed in Algorithm 17 to compute 2D-DCT.

Algorithm 18: Pseudocode for serial 2D-DCT from 1D-DCT.

 for col in 1 to MB_SIZE1 loop

  for k in 1 to MB_SIZE1 loop

   {

    • temp = 0

   for k1 in 1 to MB_SIZE1 loop

  {

    •

   }

    • dct_2d[k][col] = temp

   }

Algorithm 19 shows pseudocode for parallelization of conventional serial 2D-DCT as given in Algorithms 17 & 18. The values of col and row in Algorithm 19 are 8 each in context with size of (8×8) pixels block. The external two for-loops in Algorithms 17 & 18 are replaced by 2D thread block of local threads. The difference image block is read into shared memory in parallel in a coalesced manner, i.e., each local thread reads in a single pixel from the global memory. Similarly, DCT coefficients are read into shared memory in a coalesced manner, as there is a lot of data reuse. Then threads are synchronized. Then each local thread computes one 1D-DCT value from the block rows.

Algorithm 19: Pseudocode for the parallelization of the conventional serial 2D-DCT.

• 2D grid (2xMB_X×2xMB_Y) of global threads is executing the following instructions simultaneously.

• 2D block (col×row) of local threads is executing the following instructions simultaneously.

• Read corresponding difference macroblock in a shared memory in parallel.

• Read DCT coefficients in a shared memory in parallel.

• Synchronize threads.

• Each 1D DCT value is computed by a separate thread (block rows and coefficient columns)

• Synchronize threads.

• Each 1D-DCT value is computed by a separate thread (1D-DCT columns and coefficient columns).

When all threads are finished computing 1D-DCT values, then each of threads computes 1D-DCT values from 1D-DCT values already computed, along the columns, to yield 2D-DCT. Here each thread computes eight multiplications and eight additions for a (8×8) block. Figs 13 and 14 show block diagram of the Loeffler 1D-DCT algorithm [10], which has four stages and total 11 multiplications and 29 additions for 8-point 1D-DCT. Though four stages are in a sequential manner, each stage can be parallelized. Fig 12 shows block diagram for Algorithm 19, explaining the parallelization achieved using GPU threads, for conventional 2D DCT process.

Algorithms 20 & 21 show pseudocodes for the multithreaded Loeffler 1D-DCT algorithm. A (col×row) block of local threads is executing instructions in parallel, where col is 4 and row is 8. The difference image block is read into shared memory in parallel in a coalesced manner from global memory by local threads. Similarly, eight relevant DCT coefficients according to Figs 13 and 14 are read into shared memory.

Algorithm 20: Pseudocode for multithreaded (parallelized) Loeffler 1D-DCT algorithm.

 • 2D block (col×row) of local threads is executing the following instructions simultaneously.

 • Read corresponding difference macroblock in a shared memory in parallel from global memory.

 • Read eight DCT coefficients into shared memory.

// stage 1

   

               

 • Synchronize threads.

// stage 2

  

  

  

  

  Synchronize threads

  

  

 •Synchronize threads.

Algorithm 20 (Continued): Pseudocode (continued) for multithreaded Loeffler 1D-DCT algorithm.

// stage 3

  

  

  

  

  Synchronize threads.

  

  

  

  Synchronize threads.

// stage 4

  

  

  

  

  

Algorithm 21: Pseudocode for the parallelization of 2D-DCT using Loeffler DCT algorithm.

 • 2D grid (2×MB_X×2×MB_Y) of global threads is executing the following instructions simultaneously.

 • 2D block (4 x row) of local threads is executing the following instructions simultaneously.

 • Read corresponding difference macroblock in a shared memory in parallel.

 • Synchronize threads.

 • The 1D-DCT values are computed by four threads (block rows) using Loeffler DCT

 • Synchronize threads.

 • The 1D-DCT values are computed by four threads (1D-DCT columns) using Loeffler DCT.

Among the eight DCT coefficients are given as

{, }, for n equal to 3 and 1 and N equal to 8.

It can be noted that the last DCT coefficient is repeated. The reason is that it made multithreading coding efficient. The th_ind is the x-coordinate of the 2D local threads, which varies from 0 to 3. Similarly, y is y-coordinate of local threads varying from 0 to 7. The diff is difference image block. From stage 1 of Fig 13, the top four rows of output of stage 1 can be computed in parallel using four threads per block row. Similar is the case for bottom four rows. This is done in stage 1 of pseudocode 20 in Algorithm 20. Stage 1 has two arithmetic instructions per thread.

The result of stage 1 is stored in the shared memory array variable (s) as shown in pseudocode 20 in Algorithm 20. The eight additions (subtractions) of stage 1 of Fig 2 are reduced to two additions (subtractions) per thread per block row. All threads are synchronized before moving to stage 2. As can be seen from Fig 13, the top two rows of output of stage 2 can be computed in parallel. Similarly, the case is for third and fourth rows. This is shown by the first two instructions of stage 2 of the pseudocode 20 in Algorithm 20. These two instructions are executed by two threads, the th_ind equal to 0 and th_ind equal to 1, per block row.

From stage 2 of Fig 13, the next four rows involve eight multiplications with DCT coefficients and four additions (subtractions). The eight multiplications are executed by four threads, i.e., two multiplications per thread per block row. This is shown by third and fourth instructions of stage 2 of pseudocode 20 in Algorithm 20. Following that, threads are synchronized. Then, these multiplications are added together by using two threads, th_ind equal to 0 and th_ind equal to 1. This is shown by sixth and seventh instructions of stage 2 of pseudocode 20 in Algorithm 20. The eight multiplications and eight additions (subtractions) of stage 2 of Fig 13 are reduced to two multiplications and four additions (subtractions) per thread per block row.

The result of stage 2 is stored in shared memory array variable (s1) as shown in pseudocode 20 in Algorithm 20, where c is the shared memory array variable holding required DCT coefficients. All threads are synchronized before moving to stage 3. The first two rows of stage 3 in Fig 13 have one addition and one subtraction and cannot be parallelized, so these are executed by a single thread as shown by the first two instructions of stage 3 of pseudocode 20 in Algorithm 20 (Continued). Rows 3 and 4 of stage 3 in Fig 13 have four multiplications and two additions (subtractions). The four multiplications are executed by two threads in parallel, th_ind equal to 0 and th_ind equal to 1. This is shown by third and fourth instructions of the pseudocode 20 in Algorithm 20 (Continued). The threads are synchronized, and multiplications are added again by two threads in parallel, th_ind equal to 0 and th_ind equal to 1, as shown by the sixth instruction of the pseudocode 20 in Algorithm 20 (Continued). The last four rows of stage 3 of Fig 13 have four additions (subtractions) that can be parallelized by two threads, th_ind equal to 0 and th_ind equal to 1. This is shown by seventh and eighth instructions of the pseudocode 20 in Algorithm 20 (Continued). The four multiplications and eight additions (subtractions) of stage 3 in Fig 13 are reduced to two multiplications and five additions (subtractions). All the threads are then synchronized. The result of stage 3 is stored in shared memory variable (s2). The first four rows of stage 4 of Fig 13 are rearrangement operations and can be executed by four threads, with two threads per assignment. The first and third assignments (first and third rows) in stage 4 of Fig 13 can be executed by two threads, th_ind equal to 0 and th_ind equal to 2. This is shown in pseudocode 20 in Algorithm 20 (Continued) by first instruction of stage 4. Similarly, the second and fourth assignments (second and fourth rows) of Fig 13 can be executed by two threads, th_ind equal to 0 and th_ind equal to 1. This is shown by the second instruction in stage 4 of pseudocode 20 in Algorithm 20 (Continued). Fifth and eighth rows of stage 4 of Fig 13 cannot be executed in parallel as they have different operations so are executed by a single thread as shown by third and fourth instructions of pseudocode 20 in Algorithm 20 (Continued).

However, the sixth and seventh rows of stage 4 of Fig 13 which involves two multiplications can be executed by two threads, th_ind equal to 0 and th_ind equal to 1. This is shown by the last instruction of pseudocode 20 in Algorithm 20 (Continued). The eight operations of stage 4 of Fig 13 are reduced to five operations. It is important to mention here that not all instructions of the multithreaded (parallelized) Loeffler DCT are executed by four threads. Some instructions are executed by two threads and some by a single thread. So, some instructions are conditional, however the conditional instructions are not shown in the pseudocode 20 of Algorithm 20 and Algorithm 20 (Continued). Overall, the maximum instructions per thread are 20, out of which there are five multiplications, 13 additions (subtractions) and 2 assignments. Algorithm 21 shows pseudocode of 2D-DCT using multithreaded Loeffler DCT with (4×8) threads per block.

If the multithreaded 2D-DCT from Algorithm 19 is compared with multithreaded 2D-DCT of Algorithm 21, in former case each thread is computing a single DCT coefficient with sixteen operations, in case of 1D-DCT, out of which there are eight multiplications and eight additions. However, it can be seen from Algorithm 17 that there is an inherent for-loop (inner most) which is being executed by each thread. Hence, there are more instructions for condition checking, iteration variable increment and the goto statement. So, there are total of 48 instructions per DCT coefficient per thread excluding synchronizing instructions and memory access instructions.

Also, 2D-DCT per block is being computed by 64 threads, (8×8) in parallel, so there can be lot of shared memory access contentions. However, multithreaded Loeffler DCT is unrolled and has total of 20 instructions per 1D-DCT executed by each of four threads per block row, on average, excluding synchronizing instructions and memory access instructions. Also, 2D-DCT per block, in this case, is being executed by 32 threads, (4×8) in parallel, so there are comparatively less shared memory contentions. Because a multiplier has more delay as compared to an adder, multiplication operation can be a multicycle operation. Hence, reduced multiplication operations in the case of Loeffler DCT can lead to faster implementation. Since inverse 2D-DCT is also a part of video encoder and since it has almost same complexity as 2D-DCT, its implementation is not given here. Its parallelization follows the same steps as given above. Fig 15 shows block diagram for overall ME, motion compensation, image differencing and 2D DCT processes explaining the parallelization achieved using GPU threads.

Results and discussion

The above-mentioned pseudocodes are implemented using the C and CUDA C languages [14]. The CPU utilized is the Intel(R) Core i7 running @ 2.9 GHz. The GPU utilized is the Nvidia GeForce GTX 1080. This GPU is based on the Pascal architecture, and it consists of twenty Pascal SMs and eight memory controllers. Each SM has 128 CUDA cores, 256 KB of register file, a 96 KB of shared memory unit and 48 KB of total L1 cache. Tied to each memory controller is 256 KB of L2 cache, i.e., total size of L2 cache in the GPU is 2048 KB. The base clock frequency is 1.6 GHz, i.e., the frequency at which CUDA cores are running.

The image sequence utilized in this study is the “park joy” with resolution (3840×2160) pixels [29]. The size of macroblock is (16×16). Thus, there are 3840/16, i.e., 240 macroblocks horizontally and 2160/16, i.e., 135 macroblocks vertically. Size of search area is (31×31) pixels, which is consistent with the maximum number of threads in a block in CUDA.

Table 1 shows time consumption in seconds of different implementations as described above. The FS_Serial is serial implementation of FS or exhaustive search ME algorithm (Pseudocode 1, Algorithm 1 and Pseudocode 6, Algorithm 6) on CPU. For 25 frames, the implementation consumes 874 seconds. The FS_MB_par_direct is parallelized implementation of FS ME algorithm (Pseudocode 2, Algorithm 2), at macroblock level, on GPU, in which 2D grid of global threads of dimension (240×135) pixels is utilized. However shared memory is not used, instead only global memory access is utilized.

The corresponding time consumption is 11.4 seconds. The speedup achieved is 874/11.4, which is 77. The speedup is defined as (7) where, T_CPU is the time consumption of the CPU’s implementation and T_GPU is time consumption of GPU’s implementation.

The FS_MB_par_shared_mem is parallelized implementation of FS ME algorithm (Pseudocode 3, Algorithm 3), like implementation 2 above, however shared memory is utilized for fast and efficient data reuse by first loading search area and CMB in it. The time consumption is 5.9 seconds, and the corresponding speedup is 149. As can be seen the speedup is increased by making use of shared memory, which is the fastest memory after register file. The FS_MB_SA_par is the parallelized implementation of FS ME algorithm at macroblock and search area level (Pseudocode 4, Algorithm 4).

The time consumption is 1.4 seconds, and the corresponding speedup is 625. It is important to mention and note that SAD level parallelization (Pseudocode 7, Algorithm 7) cannot be implemented along with search area level parallelization. Since search area dimensions are greater than dimensions of a macroblock here, i.e., using search area parallelization is more beneficial than utilizing SAD level parallelization. Also frame level parallelization (Pseudocode 5, Algorithm 5) along with macroblock and search area level parallelization did not show significant reduction in computational time. The reason is that use of too many threads blocks serializes the implementation since there are fixed number of SMs, i.e., incorporating more threads after certain number of threads does not show significant benefit.

Though a significant speedup is achieved in FS ME algorithm, it is still not enough for a real time implementation. Thus, EHDS algorithm is utilized which has almost the same PSNR or bit rate distortion performance as that of FS ME algorithm, but with much reduced computational complexity. The EHDS_Serial is the serial implementation of EHDS algorithm (Pseudocode 8, Algorithm 8 and Pseudocode 10, Algorithm 10 and Pseudocode 6, Algorithm 6). As can be seen the time consumption is 16 seconds on CPU. The EHDS_MB_par is the parallelized implementation of EHDS ME algorithm at macroblock level (Pseudocode 11, Algorithm 11 and Pseudocode 9, Algorithm 9).

The time consumption is 0.24 seconds and the corresponding speedup relevant to EHDS_Serial is 67, whereas relevant to the FS_Serial it is 3642. The EHDS_MB_SAD_par is the parallelized implementation of EHDS ME algorithm at macroblock and SAD level (Pseudocode 11, Algorithm 11 and Pseudocode 9, Algorithm 9 and Pseudocode 7, Algorithm 7). Here global threads are parallelizing the macroblock level loops and local threads are parallelizing the SAD loop. As can be seen the time consumption is reduced to 0.15 seconds and the corresponding speedup relevant to EHDS_Serial is 107.

Here, search area level parallelization is not utilized as it is not very beneficial, because the search area has distant candidate motion vectors leading to non-coalesced memory access. Also, part of the candidate motion vectors is nondeterministic as is according to the EHDS algorithm. Since search area is serialized so it is now possible to exploit the SAD level parallelization, which is the case above.

For comparison purposes, authors implemented the TZS ME algorithm and parallelized it at macroblock level. As discussed in previous section, the algorithm has inherent dependency which hinders effective parallelization. Even if assumptions are made about predictive motion vectors, i.e., using temporal motion vectors (motion vectors of earlier frames) instead of using spatial motion vectors (spatially adjacent motion vectors in the same frame), the algorithm still has enough computational complexity making it not much suitable for real-time implementation of an encoder. As can be seen time consumption of parallelized implementation, TZS_MB_par, at macroblock level is one second, whereas time consumption of the TZS_Serial implementation, i.e., TZS on CPU, is 69 seconds. The speedup in this case is 69.

The 2D_DCT_Serial_conventional is serial implementation of the conventional 2D-DCT algorithm implemented as transform of transform (Pseudocode 16, Algorithm 16 and Pseudocode 17, Algorithm 17 and Pseudocode 18, Algorithm 18). The time consumption corresponding to 25 frames is 9 seconds. Similarly, 2D_DCT_Serial_Loef fler is the serial implementation of 2D-DCT in which Loeffler algorithm for 1D-DCT is used. The achieved time consumption is 3 seconds. The 2D_DCT_par is parallelized implementation and version of 2D_DCT_Serial_Loef fler. Its implementation is described in Pseudocode 21, Algorithm 21. As can be seen the efficient usage of shared memory and global and local threads, the time consumption has been reduced to 0.1 seconds. Pseudocode 19 in Algorithm 19 shows parallel 2D-DCT without Loeffler algorithm and is slightly inferior to that with the Loeffler algorithm.

The FS_MB_SA_2D_DCT_par is parallelized implementation of the parts of encoding process involving ME, reconstruction (motion compensation) (Pseudocode 13, Algorithm 13), image differencing (Pseudocode 15, Algorithm 15) and 2D-DCT. As can be seen the time consumption is 1.54 seconds. It also should be noted that the computational time of the reconstruction and image differencing has not been shown in Table 1, because the serial implementation of reconstruction process (Pseudocode 12, Algorithm 12) and the image differencing process (Pseudocode 14, Algorithm 14) has time consumption of one second for each process.

The parallelization of these processes leads to highly reduced and negligible time consumption. In addition, the inverse 2D-DCT is also a part of the encoding process and due to having symmetry with 2D-DCT, the algorithm has same computational time as 0.1 second which can be added in the overall time reduction. As can be seen with FS ME algorithm, the real-time constraint could not be achieved fully for 25 frames per second. However, for the EHDS_MB_SAD_2D_DCT_par, which is parallelized implementation of the parts of encoding process involving ME, reconstruction (motion compensation), image differencing and 2D-DCT; it can be seen that the encoding process has a time consumption of 0.25 seconds. The speedup achieved with respect to the serial implementations of the processes involving EHDS ME algorithm is 76.

The speedup of EHDS_MB_SAD_par with respect to serial implementation of FS ME algorithm, is 874/0.15, which is 5826.6. It can be concluded that there is enough time margin to cover the execution of remaining parts of encoder, i.e., quantization, inverse quantization and CABAC, in real time. It is also important to mention that flow control instructions such as the if, switch, do, for, and while can significantly reduce performance of a GPU since such instructions lead to thread divergence and the execution will have to be serialized [21]. Since CABAC algorithm is quite complicated with many flow control instructions, it would be better to implement it on CPU.

Conclusion

To boost performance of the contemporary video encoder (i.e., high efficiency video coding (HEVC) encoder), authors proposed an efficient real-time implementation of the ME and 2D-DCT structures using the Graphics processing unit, i.e., the NVIDIA GeForce GTX 1080. In addition, authors also implemented the reconstruction (i.e., ME compensation) and image differencing processes of the encoder for real-time applications. First, the FS algorithm which is thought exhaustive is the golden standard for ME process in terms of quality; so, it was considered for parallelization. Using global and local threads, tiling and shared memory reuse, the computational time is significantly reduced, but it did not meet real-time constraints. Next, authors considered a low complexity efficient ME algorithm known as EHDS ME algorithm for GPUs’ implementation. Unlike the TZS algorithm of HEVC, this algorithm does not have dependency on spatially adjacent motion vectors and so, it is best choice for efficient parallelization. Experimental results show that throughput of EHDS ME algorithm alone on GPUs corresponding to UHD sequences is greater than 150 frames per second. Then, authors implemented 2D-DCT structure on GPU. Two implementations are presented: i) first one is maximal multithreading of 2D-DCT and ii) second one is an efficient multithreaded (parallelized) 2D-DCT implementation based on the Loeffler’s DCT algorithm; the second implementation is found faster experimentally. The encoding process excluding CABAC stage take 0.45 seconds corresponding to 25 frames of the UHD sequences. Hence, this proposed encoder implementation using GPU can safely be applied in numerous real-time applications.

Acknowledgments

Authors are thankful to the organizations, corporates, and/or individuals whose databases and/or other material have been utilized in this study.