http://orcid.org/0000-0002-3236-3143
Figures
Abstract
Most of real-world image distortions are multiply distortion rather than single distortion. To address this issue, in this paper we propose a quaternion wavelet transform (QWT) based full reference image quality assessment (FR IQA) metric for multiply distorted images, which jointly considers the local similarity of phase and magnitude of each subband via QWT. Firstly, the reference images and distorted images are decomposed by QWT, and then the similarity of amplitude and phase are calculated on each subband, thirdly the IQA metric is constructed by the weighting method considering human visual system (HVS) characteristics, and lastly the scores of each subband are averaged to get the quality score of test image. Experimental results show that the proposed method outperforms the state of art in multiply distorted IQA.
Citation: Li C, Li Y, Yuan Y, Wu X, Sang Q (2018) Quaternion wavelet transform based full reference image quality assessment for multiply distorted images. PLoS ONE 13(6): e0199430. https://doi.org/10.1371/journal.pone.0199430
Editor: Yuanquan Wang, Beijing University of Technology, CHINA
Received: February 11, 2018; Accepted: June 7, 2018; Published: June 27, 2018
Copyright: © 2018 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available from the the Laboratory for Image and Video Engineering (LIVE) at The University of Texas at Austin (http://live.ece.utexas.edu/).
Funding: This work was supported by the National Natural Science Foundation of China (No. 61771223), and NSF of Hebei Province under Grant F2016202144, and the youth fund from the Department of Education of Hebei Province under grant QN2016217. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
With the large-scale use of intelligent mobile phone and computer in our modern society, evaluating the images after compression and transmission has become an increasingly important issue, and image quality assessment (IQA) has the great practical significance.
IQA can be divided into three types: full-reference IQA, reduced-reference IQA, and no-reference IQA. The full reference IQA is developed earliest, which uses the original image as a reference. Full reference IQA can be roughly divided into error visibility and structural similarity methods. Peak signal-to-nosie (PSNR) is the simplest full reference IQA method. The structural similarity (SSIM) [1] index was proposed base on the human visual system (HVS) characteristics. Also, Wang et al. [2] further proposed a multi-scale version of SSIM, called MS-SSIM. Zhang et al. [3] proposed a feature similarity (FSIM) index that uses phase congruency to weight the quality score based on SSIM index. Kolaman et al. [4] used a quaternion matrix to express a color image, and then calculated the structural similarity to evaluate color image quality. Liu et al. [5] proposed an IQA scheme based on the concept of gradient similarity (GSIM) to alleviate the shortcoming of relevant schemes. Wu et al. [6] integrated the merits of existing IQA metrics with the guide of internal generative mechanism (IGM). Xue et al. [7] devised a full reference IQA model called gradient magnitude similarity deviation (GMSD). Saad et al. [8] presented a no-reference IQA algorithm named BLIIND-II based on a natural scene statistics model of discrete cosine transform (DCT) coefficients.
In recent years, quaternion wavelet transform (QWT) has been widely used in image processing. For example, Chen et al. [9] used hybrid phase congruency extracted by QWT and gradient magnitude to calculate the similarity of images. Traoré et al. [10] proposed a reduced-reference metric based on QWT coefficients and confirmed that QWT produces a better coefficient of correlation with HVS than discrete wavelet transform (DWT). Tang et al. [11] proposed a novel dual-tree QWT based blind camera image quality assessment metric.
Motivated by recent progress in IQA and QWT, in this paper we propose a QWT based full-reference IQA metric called QWT-IQA, which jointly takes into account the local similarity of phase and magnitude of each subband via quaternionic wavelet transform. In our QWT-IQA, we make use of a weighting method to compute the image quality score, inspired by human visual characteristics. A lot of experimental results demonstrate that our proposed QWT-IQA outperforms existing full-reference IQA method.
The remaining part of the paper is organized as follows. In the section Background, we introduce the quaternion and QWT. In the section QWT-based full reference IQA metric, we propose a full-reference IQA metric based on QWT. Experiments on the LIVEMD and MDID2013 databases are carried out in the section Experimental results and analysis. At last, we give the conclusion in the section Conclusion.
Background
Review on quaternion
The quaternion is a mathematical concept proposed by an English mathematician in 1843. We all know real number and imaginary number in mathematics; and quaternion is an expansion of imaginary number. The imaginary number has a real part and an imaginary part, and the quaternion has one real part and three imaginary parts similarly. If q is a quaternion, it can be expressed as [12]: q = a+bi+cj+dk, where a, b, c, and d are the real numbers, a is the real part of quaternion, and bi+cj+dk is the imaginary part of quaternion, and i, j, and k are imaginary numbers satisfying the following
(1)
A quaternion can also be expressed by amplitude and phase, i.e., q = |q|eiφejθekψ, where |q| is the amplitude, and φ,θ,ψ are the phase angles whose range are [−π,π], [−π/2,π/2], and [−π/4,π/4] respectively.
Review on QWT
Quaternion wavelet transform is a new wavelet transform, which combines quaternion and Hilbert transform together. It is approximately shift-invariant and has abundant phase information. The four orthonormal bases of QWT can be expressed in the matrix form as follow.
(2)
where φh and φg are scale functions, ψh and ψg are wavelet functions. According to [13], each row of the matrix G in (2) represents the independent wavelet of QWT, and each column of the matrix represents a subband of QWT. By using the algebra about quaternion, the four groups in each column can be grouped into a wavelet function
(3)
QWT-based full reference IQA metric
As above, QWT is a new dual-tree wavelet that has shift invariance and phase information. In this work, based on the local similarity of phase and magnitude of each subband via QWT, we construct a FR IQA metric. Firstly, the reference image and corresponding distorted image are, respectively, decomposed into a low frequency subband (LL) and three high frequency subbands (LH, HL, HH) at each scale via QWT. Then, an amplitude and three phases are got at each subband. Through a large number of experiments, we acquire the best performance when the image is decomposed by 3 scales QWT. The amplitude of low frequency subband reflects the condition of the original image and has approximate shift invariance. The phases (φ,θ,ψ) of low frequency subband indicate the vertical, horizontal and diagonal texture information, respectively. The amplitude of high frequency subband reflects the contour of the image in some particular direction. The phases (φ,θ) of high frequency represent local shift information, and the phase ψ of high frequency obtains the texture feature of image. Fig 1 shows the images of amplitude and phase (φ,θ,ψ) of each subband via QWT.
Now, we calculate the similarity of magnitude and three phases of the reference image and distorted image. The similarity of magnitude is defined as
(4)
where Mag1 and Mag2 represent the amplitude of LL subband of the original image and corresponding distorted images by 3-scale QWT, and T1 is a positive normal number, which aims to make the denominator non zero. Following the same way as (4), the similarities of phase (φ,θ,ψ) are defined by
(5)
(6)
(7)
where φ1,θ1,ψ1 and φ2,θ2,ψ2 represent phases of LL subband of the original image and distorted images, and T2,T3,T4 are positive normal numbers, which aim to maintain the fraction stability. Note that the similarity range of (5), (6) and (7) is (0,1]. The local similarity of LL subband is defined as follows:
(8)
where the coefficients α, β, χ, δ represent the importance of amplitude and phases. The amplitude of low frequency subband reflects the condition of the original image, and which of high frequency subband denotes the contour of the image in some particular direction. By setting phase β, χ, δ to 1, and changing the value of magnitude α from 0 to 20 step by 1, we gain the plot between SROCC and amplitude shown in Fig 2. It can be seen the SROCC rose gently when the amplitude is set to 10, so we set the magnitude α to 10, and the phase β, χ, δ to 1, then the Eq (8) become following Eq (9).
After getting the similarity of each pixel, the similarity of the LL subband can be calculated. However, the human visual system has different perceptual effects on different regions of the picture. The more obvious the pixel amplitude is, the more prominent its corresponding phase is. The larger the amplitude is, the more important the corresponding pixel is, and the more likely it is to be in the texture changing structure. The smaller the amplitude is, the more likely it is to be in the smooth image region, and the value of the corresponding phase tends to be more unstable [14]. In other words, the human eyes are always devoting more attention to the area which has larger amplitude. Thus, we add the item Magm(x) = max(Mag1,Mag2) to make the IQA metric more consistent with human visual characteristics. Therefore, the SImilarity Metric in LL subband (SIMLL) considering HVS is calculated as following.
(10)
where Ω represents the entire space domain of the image.
Following the same computation way in (10), the SImilarity Metrics in LH, HL and HH subbands, i.e., SIMLH, SIMHL, SIMHH can also been calculated in a similar form. Once the foregoing four similarity metrics are obtained, the QWT-based IQA metric (QWT-IQA) can be calculated by a weight sum of similarity metrics in all subbands as follows.
(11)
(12)
(13)
(14)
where a, b, c and d are used to adjust the importance of each subband.
From Fig 1 the three high frequency subbands are similar each other, so we keep them the same value, and change from 0 to 0.3 step by 0.05, then the coefficient of low frequency from 1 to 0.1 correspondingly, and gain the plot between SROCC and subband coefficients shown as Fig 3. The curve becomes smooth when the coefficients are close. It can be seen the SROCC reaches maximum when coefficients of high frequency subbands are 0.25, which suggest each subband is the same important for our proposed IQA metric.
So here we set all a, b, c and d to 0.25, and the final IQA formula is defined as follows:
(15)
The whole flow chart of our proposed QWT-IQA is illustrated in Fig 4.
Experimental results and analysis
Experiment on the LIVEMD image database
LIVE multiply distorted (LIVEMD) image database [15] consists of 15 reference images and 450 distorted images, which has two multiple distortion scenarios: blur followed by JPEG and blur followed by noise. Each image set has 90 singly distorted images and 135 multiply distorted images, and the size of the images is 1280×720. The difference mean opinion score (DMOS) of LIVEMD is between 0 and 100.
There are several measures to evaluate the correlation between the quality scores and DMOS, such as Spearman rank order correlation coefficient (SROCC), Pearson linear correlation coefficient (PLCC), Kendall’s rank order correlation coefficient (KROCC), and Root mean squared error (RMSE). Note that, the closer correlation coefficient is to 1 and the lower RMSE is, the better the algorithm performs.
Experiments are carried out on two image subsets and the entire database, respectively. We compare our proposed QWT-IQA with the FR PSNR, SSIM, FSIM, MS-SSIM and NR SISBLIM, DIIVINE, BLIINDS-II, NIQE methods. The results are listed in Tables 1, 2 and 3. From Tables 1, 2 and 3 it can be seen that QWT-IQA performs better than these FR and NR algorithms on both image subsets and whole database, no matter what measure is used.
We also give the scatter plot of several FR IQA scores on the LIVEMD image database against DMOS in Fig 5, which also shows the QWT-IQA algorithm has a better agreement with the human subjective perception than PSNR, SSIM, FSIM and MS-SSIM.
Experiment on the MDID2013 image database
Multiply distorted image database (MDID2013) [16] consists of 12 reference images and 324 distorted images simultaneously distorted by JPEG compression, blurring and noise injection. One half of pristine images of size 768×512 are from Kodak database, and the other half pristine images of size 1280×720 from LIVEMD database. The difference mean opinion score (DMOS) of MDID2013 is between 0 and 1.
The comparison results of FR QWT-IQA, PSNR, SSIM, FSIM and MS-SSIM are given in Table 4, as we can see from Table 4, our proposed QWT-IQA algorithm significantly outperforms these FR IQA algorithms on all cases.
We also compare the QWT-SIM metric with some NR metrics such as SISBLIM, DIIVINE, BLIINDS-II and NIQE, listed in Table 4. It can be seen our QWT-SIM is a little inferior to NR SISBLIM, but NR SISBLIM is test on only 20% test images of database, so it is unfair comparison.
On the MDID2013 image database, we also plot the scatter of QWT-IQA scores against DMOS, as shown in Fig 6. As seen clearly from Fig 6, the QWT-IQA algorithm has a good agreement with human subjective perception. This conclusion is consistent with that drawn from the previous experiment in LIVEMD database. In a word, many results have demonstrated that our method is a good technique for IQA.
Experiment on the single distortion LIVE IQA database
LIVE IQA database [17] is single distortion database, which consists of 29 reference images and all 982 images, including five different distortion categories: JPEG2000 (JP2K) and JPEG compression, white Gaussian noise (WN), Gaussian blur (blur) and a Rayleigh fast fading channel distortion (FF).
For further test our proposed QWT-IQA metric, we also compare several IQA algorithms on the LIVE single distortion IQA database, and the results are listed in Table 5. It can be seen our QWT-IQA is only a little inferior to FSIM, and still gets good linear relationship with the human subjective scores.
Time efficiency of IQA metrics
Time efficiency is another important index for algorithm. We give the time of compared algorithm listed in Table 6. It can be seen most of the NR methods need more time to train images, and the FR methods only need to compute the relevance or deviation in a very short time. Our proposed QWT-IQA is more time consuming than FR PSNR, SSIM and MS-SSIM, but also efficient for real time application.
Conclusion
In this paper, we have proposed a QWT-based full reference IQA metric called QWT-IQA for multiply distorted images. It first calculates the local similarity of phase and magnitude of each subband via QWT, and then uses a weighting method to gain image quality score through considering human visual characteristics. Many experimental results have demonstrated that our QWT-IQA has a higher consistency with the subjective measurement on multiply distortion images, compared with the state-of-the-art full reference IQA methods.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 61771223), and NSF of Hebei Province under Grant F2016202144, and the youth fund from the Department of Education of Hebei Province under grant QN2016217.
References
- 1. Wang Z., Bovik A. C., Sheikh H. R., Simoncelli E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612. pmid:15376593
- 2.
Z. Wang, E. P. Simoncelli, A. C. Bovik. Multi-scale structural similarity for image quality assessment. In: 37th Asilomar Conference on Signals, Systems and Computers, 2003, pp. 1398–1402.
- 3. Zhang L., Zhang L., Mou X., Zhang D. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 2011, 20(8): 2378–2386. pmid:21292594
- 4. Kolaman A., Yadid-Pecht O. Quaternion structural similarity: a new quality index for color images. IEEE Transactions on Image Processing, 2012, 21(4): 1526–1536. pmid:22203713
- 5. Liu A., Lin W., Narwaria M. Image quality assessment based on gradient similarity. IEEE Transactions on Image Processing, 2012, 21(4): 1500–1512. pmid:22106145
- 6. Wu J., Lin W., Shi G., Liu A. Perceptual quality metric with internal generative mechanism. IEEE Transactions on Image Processing, 2013, 22(1): 43–54. pmid:22910116
- 7. Xue W., Zhang L., Mou X., Boviket A. C. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Transactions on Image Processing, 2014, 23(2): 684–695. pmid:26270911
- 8.
M. A. Saad, A. C. Bovik, C. Charrier. DCT statistics model-based blind image quality assessment. In: IEEE International Conference on Image Processing, 2011, pp. 3093–3096.
- 9. Chen Q., Xu Y., Li C., Liu N., Yang X. An image quality assessment metric based on quaternion wavelet transform. In: IEEE International Conference on Multimedia and Expo Workshops, 2013, pp. 1–6.
- 10.
A. Traoré, P. Carré, C. Olivier. Reduced-reference metric based on the quaternionic wavelet coefficients modeling by information criteria. In: IEEE International Conference on Image Processing, 2015, pp. 526–530.
- 11. Tang L., Li L., Sun K., Xia Z., Gu K., Qian J. An efficient and effective blind camera image quality metric via modeling quaternion wavelet coefficients. Journal of Visual Communication & Image Representation, 2017, 49: 204–212.
- 12. Muraleetharan B., Thirulogasanthar K. Coherent state quantization of quaternions. Journal of Mathematical Physics, 2015, 251(8): 21–57.
- 13. Chan W. L., Choi H., Baraniuk R. G. Coherent multiscale image processing using dual-tree quaternion wavelets. IEEE Transactions on Image Processing, 2008, 17(7): 1069–1082. pmid:18586616
- 14. Soulard R., Carré P. Quaternionic wavelets for texture classification. Pattern Recognition Letters, 2011, 32(13): 1669–1678.
- 15.
D. Jayaraman, A. Mittal, A. K. Moorthy, A. C Bovik. Objective quality assessment of multiply distorted images. In: 46th Asilomar Conference on Signals, Systems and Computers, 2012, pp. 1693–1697.
- 16. Gu K., Zhai G., Yang X., Zhang W. Hybrid no-reference quality metric for singly and multiply distorted images. IEEE Transactions on Broadcasting, 2014, 60(3): 555–567.
- 17.
H. R. Sheikh, Z. Wang, L. Cormack and A. C. Bovik. LIVE Image Quality Assessment Database Release 2. Available: http://live.ece.utexas.edu/research/quality.