Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A No-Reference Adaptive Blockiness Measure for JPEG Compressed Images

  • Chaoying Tang ,

    cytang@nuaa.edu.cn

    Affiliation College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China

  • Biao Wang

    Affiliation College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China

Abstract

Digital images have been extensively used in education, research, and entertainment. Many of these images, taken by consumer cameras, are compressed by the JPEG algorithm for effective storage and transmission. Blocking artifact is a well-known problem caused by this algorithm. Effective measurement of blocking artifacts plays an important role in the design, optimization, and evaluation of image compression algorithms. In this paper, we propose a no-reference objective blockiness measure, which is adaptive to high frequency component in an image. Difference of entropies across blocks and variation of block boundary pixel values in edge images are adopted to calculate the blockiness level in areas with low and high frequency component, respectively. Extensive experimental results prove that the proposed measure is effective and stable across a wide variety of images. It is robust to image noise and can be used for real-world image quality monitoring and control. Index Terms—JPEG, no-reference, blockiness measure

Introduction

Although digital images provide great convenience to our daily life, storing and transmitting these images are problematic. Consumer cameras can produce images with size more than 12 mega pixels. Some of them can even produce images with over 20 mega pixels. The standard JPEG algorithm installed in consumer cameras are commonly used to compress these images. JPEG is a discrete cosine transform (DCT) -based compression scheme. The input image is divided into 8×8 blocks, each block being transformed independently to DCT coefficients. The DCT coefficients are then quantized using a scalar quantization matrix. At low bitrates, the DCT-coded images generally suffer from visually annoying blocking artifacts as a result of coarse quantization. It can dramatically degrade the image quality.

Over the past decades, research has been carried out to alleviate blocking effects in JPEG-compressed images. Effective measurement of blocking artifacts plays an important role in the development and evaluation of these algorithms. The most widely used point-wise comparison measures, namely mean square error (MSE) and peak signal to noise ratio (PSNR), are not very well matched to perceived visual quality [1]. A lot of effort has been put in the development of effective quality measures.

In this paper, we propose a no-reference adaptive blockiness measure for JPEG compressed images. For areas with low frequency component, we use the difference of entropies across blocks as a measure. For areas with high frequency component, we use the variation of block boundary pixel values in edge images as a measure. The final index is the weighted average of the two measures. The rest of this paper is organized as follows. Section 2 gives a short review of the related work. Section 3 introduces our blockiness measure. Section 4 reports experimental results. Section 5 offers some concluding remarks.

Literature Review

Image quality assessment techniques can be divided into two classes: subjective assessment and objective assessment. In subjective assessment, a group of testers are required to rate a batch of images compressed under different ratios. Objective methods try to estimate the compression amount using mathematical techniques. They can be further divided into two types: full-reference and no-reference methods.

Full-reference methods give an assessment through a comparison between reference and compressed images. They mainly utilize the sensitivity of human visual system and a combination of multiple features [2]. Wang and Bovik [3] proposed a universal image quality index which models image distortion as a combination of three factors: loss of correlation, luminance distortion, and contrast distortion. In [4], the measure was generalized to the Structural SIMilarity index (SSIM) to characterize the saturation effects of the visual system at low luminance and contrast regions and to assure numerical stability.

No-reference methods directly evaluate the compressed images. They are more useful because the reference image is usually unavailable. Moorthy and Bovik [5] presented two strategies—visual fixation-based weighting, and quality-based weighting to weight image quality measurements by visual importance, and demonstrated improvements on the SSIM index in both its multi-scale and single-scale versions. Wang and Bovik [6] proposed an algorithm which evaluates blockiness by computing the FFT along the rows and columns. Pan et al. [7] proposed a method based on the fact that for images with more severe blocking artifacts, more edge pixels are arrayed horizontally and vertically. Lee et al. [8] classified blocks by analyzing block boundaries and converted boundary strength into blockiness score. Li et al. [9] presented a blockiness metric which computes the regularities of pseudo structures. In [10], Li et al. proposed a quality measure for JPEG-compressed images based on Tchebichef kernels. Golestaneh and Chandler [11] presented a measure that gets the number of zero-valued discrete cosine transform (DCT) coefficients within each block. Then it uses a quality map to represent these numbers. Freitas et. al [12] proposed a machine-learning-based method that uses the histograms of local ternary pattern (LTP) as features for the training procedure. Li et al. [13] proposed a deep-neural-network-based algorithm that extracts features using shearlet transform and evolves the features using stacked auto-encoders. Then the differences of evolved features are identified by a softmax classifier. Ren et al. [14] developed a method which uses shearlet transform to localize the distributed discontinuities induced by image degradation, and exhibits the alteration of image quality by the nature scene statistics of shearlet coefficients.

A common problem with the current blockiness measures is that none of them have adaptability to image characteristics. Different areas in an image usually have different amount of high frequency component. The compression process has different influences on these areas. This property should be considered in the blockiness measure.

Proposed Algorithm

In this paper, we propose a no-reference blockiness measure which is adaptive to different areas in an image. Low frequency component in images corresponds to pixel values that change slowly over space, while high frequency component means pixel values that change rapidly. Edges in a digital image are the points at which the image brightness changes sharply or has discontinuities. Therefore sharp transients like edges represent high frequency component, while long unchanging spaces correspond to low frequency component. For the areas with low frequency component, the compression process reduces intensity variability inside a block, sometimes even generates blocks of constant gray scale, especially under high compression ratio. The block boundaries produce abrupt change of intensities across blocks. Fig 1 illustrates the influence of JPEG compression on different image areas. Fig 1(A) is an original image of a building, where the square shows an area with low frequency component (see Fig 1(B)). After compression, blocks of constant gray scale are produced (see Fig 1(C)). Comparing Fig 1(B) and 1(C), we find that before compression, the entropies of pixel values inside a block and across blocks are almost the same. However, they are quite different after compression, because of the generated block boundaries. Considering this property, we use the difference between the entropies of pixel values inside a block and across blocks to represent the compression level.

thumbnail
Fig 1. Illustration of the influence of JPEG compression on different image areas.

(a) is an original image of a building, where the square shows an area with low frequency component; (d) is the same original image, where the square shows an area with high frequency component; (b) and (e) are the selected image areas in (a) and (d), respectively; (c) and (f) are the JPEG compressed versions of (b) and (e) at 0.071 bpp, respectively.

https://doi.org/10.1371/journal.pone.0165664.g001

For the areas with high frequency component, the entropy difference is not influenced by the compression process, since the entropy of pixel values inside a block is already high before compression (see Fig 1D–1F). However, the generated blocks do change the edge information. Discontinuities appear around block boundaries in the edge image, and the higher the compression ratio is, the larger the discontinuities are. Therefore, in this case, we use Sobel operator to get horizontal and vertical edge images first. Then we capture the difference of pixel values around block boundaries in the edge images to represent the compression level.

Let I be a compressed image, and Mx and My be the Sobel masks defined as follows: (1)

The convolution results between I and Mx, My are denoted as: (2) where * is the convolution operator, and Cx and Cy are the horizontal and vertical edge images, respectively. Then Cx and Cy are decomposed into 8×8 non-overlapping blocks, since DCT transform is performed by 8×8 blocks in JPEG compression.

For the kth 8×8 block in the image, if it has edges (with high frequency component), four regions in Cx and Cy, Rih, Riv, Roh, and Rov, are considered. Rih and Riv denote the pixels just inside the horizontal and vertical boundaries, respectively. Roh and Rov denote the pixels just outside the horizontal and vertical boundaries, respectively. Fig 2A–2D illustrate these regions. We define measure s1 for the areas with high frequency component. It is calculated according to the following equations: (3) (4) (5) (6) where Ni and No are the number of elements in the summation (here Ni = 32, No = 40). Nk is the number of blocks in this area. The measure s1 is obtained from the luminance variance of pixels inside and outside block boundaries.

thumbnail
Fig 2. Illustration of the block areas involved in the blockiness measure.

Dark elements represent the regions (a) Rih, (b) Riv, (c) Roh, (d) Rov, (e) Ib and (f) Ibo.

https://doi.org/10.1371/journal.pone.0165664.g002

For the tth 8×8 block in the areas without edges (with low frequency component), the regions Ib and Ibo are considered. Ib denotes an 8×8 block, while Ibo represents a 10×10 pixel area with the 8×8 block in the center. Fig 2E and 2F illustrate these regions. We define measure s2 for the areas with low frequency component. It is calculated according to the following equations: (7) (8) where En(·) is the function to compute entropy. Nt is the number of blocks in this area. The measure s2 is obtained from the entropy difference of pixels inside and across blocks.

The final blockiness measure is defined as the weighted average of the two measures: (9) where wi (i = 1, 2) is the weight of the two measures which is determined as the total numbers of the two different kinds of blocks in an image.

Take Fig 1(A) as an example. After compression, Nk, Nt, s1 and s2 are 637, 1721, 0.220 and 0.812, respectively. In the areas with low frequency component, the average values of En(Ib) and En(Ibo) are 0.366 and 0.734, respectively. It can be seen that the entropy increases after compression, because of the generated block boundaries in Ibo. The final blockiness measure s is 0.652.

Experimental Results and Discussion

Extensive experiments were conducted to evaluate the performance of the proposed blockiness measure. In the first experiment, the images “sailing2” from LIVE database [15] with different levels of blocking artifacts were adopted. The bitrates of the three compressed images are 0.3929, 0.1710 and 0.1638, respectively. The subjective evaluating results of the images are represented using difference mean opinion score (DMOS). In general, the higher DMOS an image has, the lower the image quality is. Therefore a reasonable blockiness measure should be proportional to the DMOS values. The blockiness measures given by Wang’s [6], Pan’s [7], Lee’s [8], Li’s [9] and the proposed method are summarized in Table 1. It can be seen that images No. 2 and 3 have severe blocking artifacts, and image No. 3 is a little bit more serious. Therefore, the blockiness measure of No. 3 should be larger than that of No. 2. However, in Wang’s, Pan’s and Lee’s results, the score of No. 3 is smaller. It can be seen that these measures cannot accurately evaluate images with similar DMOS values. Li’s and the proposed method achieve correct results. However, the difference between Li’s scores of No. 2 and No. 3 is too large compared with the difference between the corresponding DMOS values. The scores given by the proposed method is more consistent with DMOS values. To evaluate the entropy component in the proposed measure, we calculated measure s2 in Eq (8) for the three images and put them in the last column of Table 1. It can be seen that s2 increases monotonically with the compression ratio, which means that entropy is able to describe compression level.

thumbnail
Table 1. Comparison of blockiness measures for the image“sailing2” from LIVE database with different compression levels

https://doi.org/10.1371/journal.pone.0165664.t001

In the second experiment, we used six benchmark images with different themes to do evaluation. The image sizes vary from 256×256, 320×240, 512×512, to 480×720. These images were compressed under quality factors of 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, and 95, using the baseline JPEG standard in MATLAB. The corresponding bitrates can be obtained accordingly. We compared our results with a classical no-reference measure, Perra’s [2], and two full-reference measures, Qi index [3] and SSIM index [4]. Because Qi index and SSIM index give higher results for images with better quality, we use their reciprocals as blockiness indices, i.e., higher index corresponds to more serious blocking artifacts. To have a better observation, we normalize their variations to the range of 0 and 1. Fig 3 reports the experimental results. All of them have areas of different amount of high frequency component. The sensibility to the JPEG blockiness distortion was evaluated using the images compressed at different bitrates. The results show that Perra’s [2] does not change monotonically with the increase of bitrates for any of the images. In other words, its value does not represent the blockiness level in an image. Qi index has the property of monotonicity only for the image of ‘lena’ (see Fig 3(A)). For the images of ‘couple’ (Fig 3(B)), ‘bird’ (Fig 3(D)), and ‘statue’ (Fig 3(F)), the index varies drastically up and down for bitrates between 0.5 and 2.5. For the image of ‘landscape’ (Fig 3(C)) and ‘lighthouse’ (Fig 3(E)), it even increases for bitrates higher than 1. SSIM index decreases monotonically with the increase of bitrates for all the images. However, it is a full-reference measure, so it requires the original image as a reference. The proposed measure has a good property of monotonicity for all the images.

thumbnail
Fig 3. Relationship between normalized blockiness measures and bitrates of some benchmark images.

(a) lena, (b) couple, (c) landscape, (d) bird, (e) lighthouse, and (f) statue. For the copyright issue, the original images are not shown here.

https://doi.org/10.1371/journal.pone.0165664.g003

In the third experiment, we evaluated the influence of noise on the proposed blockiness measure. The six benchmark images were adopted again. In this experiment, we added Gaussian white noise of mean 0 and variance 0.01 into the images, and then compressed them using the baseline JPEG standard in MATLAB. The following procedure was the same as in the second experiment. Fig 4 reports the experimental results. It can be seen that even SSIM does not have correct result for the image of ‘couple’ (Fig 4(B)). The proposed measure still has better performance than other no-reference measures. To evaluate whether the entropy component can correctly represent the compression level for images with different noises, we calculated measure s2 in Eq (8) for images added with Gaussian white noise of different variances and salt & pepper noise with different densities, and compressed with different levels. For the sake of space limit, we show the results of one image (“rushmore” from CSIQ database [16]) in Table 2 and Table 3. The bitrates, the blockiness measure and the corresponding s2 are shown for each variance and each compression level. It can be seen that for any case of the noises, both of the measures increase monotonically with the decrease of bitrates, which means that neither of them is interfered by noise. Therefore, the block differentiation mechanism in the proposed measure which depends on high frequency component will not be influenced by noise. In the fourth experiment, the JPEG images from four image-quality databases, including MICT[17], IVC[18], TID2008 [19], and TID2013 [20], were used to evaluate the overall performance of the proposed measure. The JPEG subsets in MICT, IVC, TID2008, and TID2013 contain 98, 60, 125, and 150 images, respectively. In these databases, 14 of 98, 10 of 60, 25 of 125, and 25 of 150 are reference images. Pearson linear correlation coefficient (PLCC) and root mean square error (RMSE) were used to evaluate the prediction accuracy, and Spearman rank-order correlation coefficient (SROCC) was used to evaluate the prediction monotonicity. These criterions were computed between the predicted scores and the subjective scores, where nonlinear fitting was first conducted to bring them on the same scale [10][21]. For comparison, the results of some popular no-reference blocking artifact metrics were also provided, including Perra’s [2], Wang’s [6], Pan’s [7], Lee’s [8], Li’s [10], Bovik’s [22], Wang’s [23], Liu’s [24], Chen’s [25], Mittal’s [26], Ye’s [27], Liu’s [28], and Liu’s [29]. A good quality measure should achieve high values in PLCC and SRCC, while low values in RMSE. Tables 4 and 5 provide the results, where the best result is marked in boldface. As shown in Table 4, the proposed measure yields the best results of both PLCC and SROCC for MICT, IVC, and TID2013. For TID2008, its SRCC is also the best, and its PLCC is very close to the best result. It can be seen that the proposed measure has a very high correlation with the subjective quality ratings. As for RMSE, the proposed measure also provides the best result for TID2008.

thumbnail
Fig 4. Relationship between normalized blockiness measures and bitrates of some benchmark images interfered by Gaussian noise.

(a) lena, (b) couple, (c) landscape, (d) bird, (e) lighthouse, and (f) statue. For the copyright issue, the original images are not shown here.

https://doi.org/10.1371/journal.pone.0165664.g004

thumbnail
Table 2. Evaluation of the proposed measure and the corresponding s2 for the image“rushmore” from CSIQ database with Gaussian white noise of different variances and with different compression levels

https://doi.org/10.1371/journal.pone.0165664.t002

thumbnail
Table 3. Evaluation of the proposed measure and the corresponding s2 for the image“rushmore” from CSIQ database with salt and pepper noise of different densities and with different compression levels

https://doi.org/10.1371/journal.pone.0165664.t003

thumbnail
Table 4. PLCC and SROCC of blockiness measures for the JPEG images in MICT, IVC, TID2008, and TID2013 databases.

https://doi.org/10.1371/journal.pone.0165664.t004

thumbnail
Table 5. RMSE of blockiness measures for the JPEG images in MICT, IVC and TID2008 databases.

https://doi.org/10.1371/journal.pone.0165664.t005

The experimental results show that the proposed blockiness measure is effective and stable across a wide variety of images. It has a good estimation of compression level over different image areas. Moreover, it is robust to image noise and can be used for real-world image quality monitoring and control.

Conclusions

In this paper, we proposed a no-reference JPEG blockiness measure which is adaptive to the high frequency component in different image areas. It uses difference of entropies across blocks and variation of block boundary pixel values in edge images to estimate the blockiness level in an image. Extensive experiments were conducted to evaluate the performance of the proposed measure. The results indicated that compared with other blockiness indices, the proposed measure has satisfying and stable performance across a wide variety of images. It is robust to image noise and has positive perspectives in applications such as image quality monitoring and control.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (61403196), the Natural Science Foundation of Jiangsu Province (BK20140837), the Specialized Research Fund for the Doctoral Program of Higher Education of China (20133218120018), and the Fundamental Research Funds for the Central Universities (NS2014030).

Author Contributions

  1. Conceptualization: CT.
  2. Data curation: CT.
  3. Formal analysis: CT.
  4. Funding acquisition: CT BW.
  5. Investigation: CT.
  6. Methodology: CT.
  7. Project administration: CT BW.
  8. Resources: BW.
  9. Software: CT.
  10. Supervision: CT.
  11. Validation: CT.
  12. Visualization: BW.
  13. Writing – original draft: CT.
  14. Writing – review & editing: CT BW.

References

  1. 1. Wang Z. and Bovik A.C., “Mean squared error: Love it or leave it? A new look at signal fidelity measures, IEEE Signal Processing Magazine, vol.26, no.1, pp.98–117, 2009.
  2. 2. C. Perra, F. Massidda, and D.D. Giusto, “Image blockiness evaluation based on sobel operator,” in Proceedings of IEEE International Conference on Image Processing, pp.I-389–392, 2005.
  3. 3. Wang Z. and Bovik A.C., “A universal image quality index,” IEEE Signal Processing Letters, vol.9, no.3, pp.81–84, 2002.
  4. 4. Wang Z., Bovik A.C., Sheikh H.R., and Simoncelli E.P., “Image quality assessment: from error measurement to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp.600–612, 2004. pmid:15376593
  5. 5. Moorthy A.K. and Bovik A.C., “Visual Importance Pooling for Image Quality Assessment,” IEEE Journal of Selected Topics in Signal Processing, vol.3, no.2, pp.193–201, 2009.
  6. 6. Z. Wang, A.C. Bovik, and B.L. Evans, “Blind measurement of blocking artifacts in images,” in Proceedings of IEEE International Conference on Image Processing, vol.3, pp. 981–984, 2000.
  7. 7. Pan F., Lin X., Rahardja S., Ong E. P., and Lin W. S., “Using edge direction information for measuring blocking artifacts of images,” Multidimensional Systems and Signal Processing, vol. 18, no. 4, pp. 297–308, 2007.
  8. 8. Lee S. W. and Park S. J., “A new image quality assessment method to detect and measure strength of blocking artifacts,” Signal Processing: Image Communication, vol. 27, no. 1, pp. 31–38, 2012.
  9. 9. Li L., Lin W., and Zhu H., “Learning Structural Regularity for Evaluating Blocking Artifacts in JPEG Images,” Signal Processing Letters, vol.21, no.8, pp.918–922, 2014.
  10. 10. Li L., Zhu H., Yang G., and Qian J., “Referenceless Measure of Blocking Artifacts by Tchebichef Kernel Analysis,” Signal Processing Letters, vol.21, no.1, pp.122–125, 2014.
  11. 11. Golestaneh S.A. and Chandler D.M., “No-Reference Quality Assessment of JPEG Images via a Quality Relevance Map,” Signal Processing Letters, vol.21, no.2, pp.155–158, 2014.
  12. 12. P. G. Freitas, W. Y. L. Akamine, and M. C. Q. Farias, “No-reference image quality assessment based on statistics of Local Ternary Pattern,” in Proceedings of International Conference on Quality of Multimedia Experience, pp. 1–6, 2016. 6–8 June 2016.
  13. 13. Li Y.,Po L.M.,Xu X.,Feng L.,Yuan F., Cheung C. H., et al, “No-reference image quality assessment with shearlet transform and deep neural networks,” Neurocomputing, vol. 154, pp. 94–109, 2015.
  14. 14. Ren Y., Lu W., He L., and Gao X., “Blind Image Quality Assessment in Shearlet Domain,” Lecture Notes in Computer Science, vol. 9242, pp 472–481, 2015.
  15. 15. H. R. Sheikh, Z. Wang, L. Cormack, and A. C. Bovik, “LIVE Image Quality Assessment Database Release 2,” [Online]. Available: http://live.ece.utexas.edu/research/quality.
  16. 16. E. C. Larson and D. M. Chandler, “Categorical image quality (CSIQ) database,” [Online]. Available: http://vision.okstate.edu/csiq.
  17. 17. Y. Horita, K. Shibata, Y. Kawayoke, and Z. M. P. Sazzad, “MICT: Image quality evaluation database 2000,” [Online]. Available: http://mict.eng.u-toyama.ac.jp/mict/index2.html.
  18. 18. A. Ninassi, P. Le Callet, and F. Autrusseau, “Subjectivej Quality Assessment IVC Database 2005,” [Online]. Available: http://www2.irccyn.ecnantes.fr/ivcdb.
  19. 19. Ponomarenko N., Lukin V., Zelensky A., Egiazarian K., Carli M., and Battisti F., “TID2008-a database for evaluation of full reference visual quality assessment metrics,” Advances of Modern Radioelectronics, vol. 10, no. 5, pp. 30–45, 2009.
  20. 20. Ponomarenko N., Jin L., Ieremeiev O., Lukin V., Egiazarian K., Astola J., et al, “Image database TID2013: Peculiarities, results and perspectives,” Signal Processing: Image Communication, vol. 30, pp. 57–77, 2015.
  21. 21. Chandler D. M. and Hemami S. S., “VSNR: A wavelet-based visual signal-to-noise ratio for natural images,” IEEE Transactions on Image Processing, vol. 16, no. 9, pp. 2284–2298, Sep. 2007. pmid:17784602
  22. 22. A. C. Bovik and S. Z. Liu, “DCT domain blind measurement of blocking artifacts in DCT-coded images,” in Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 1725–1728, 2001.
  23. 23. Z. Wang, H. R. Sheikh, and A. C. Bovik, “No-reference perceptual quality assessment of JPEG compressed images,” in Proceedings of IEEE International Conference on Image Processing, pp. 477–480, 2002.
  24. 24. Liu H. T. and Heynderickx I., “A perceptually relevant no-reference blockiness metric based on local image characteristics,” EURASIP Journal on Advances in Signal Processing, vol. 2009, pp. 1–14, 2009.
  25. 25. C. H. Chen and J. A. Bloom, “Image blockiness evaluation based on Sobel operator,” in Proceedings of Pacific-Rim Conference on Advances in Multimedia Information Processing, pp. 112–123, 2010.
  26. 26. Mittal A., Moorthy A. K., and Bovik A. C., “No-reference image quality assessment in the spatial domain,” IEEE Transactions on Image Processing, vol. 21, no. 12, pp. 4695–4708, 2012. pmid:22910118
  27. 27. P. Ye, J. Kumar, L. Kang, and D. Doermann, “Unsupervised feature learning framework for no-reference image quality assessment,” in Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1098–1105. 2012.
  28. 28. Liu L., Dong H., Huang H., and Bovik A. C., “No-reference image quality assessment in curvelet domain,” Signal Processing: Image Communication, vol. 29, no. 4, pp. 494–505, 2014.
  29. 29. Liu L., Liu B., Huang H., and Bovik A. C., “No-reference image quality assessment based on spatial and spectral entropies,” Signal Processing: Image Communication, vol. 29, no. 8, pp. 856–863, 2014.