Multi-exposure image fusion methods

Since the MEF image can inherit the advantages of multiple images with different exposure degrees, the MEF method has received a lot of attention. Burt et al [5] utilized the Laplacian pyramid decomposition and the pyramid decomposition for binocular and MEF images, respectively. Goshtasby et al [6] partitioned the multiple images into uniform blocks and chose the maximum information of each block for fusion. Mertens et al [7] calculate the weighting maps to blend multiple exposures via color saturation and contrast. Raman et al [8] designed a fusion method based on the bilateral filter to obtain the MEF image. Li et al [9] introduced a new quadratic optimization-based method for fusion, the great contribution is that the fine details are extracted from a vector field. Gu et al [10] fused the multi-exposure images in the gradient field, since the human visual system (HVS) is sensitive to contrasts that can be represented by local gradients. To propose fast and effective methods, Li et al [11, 12] carried out two pieces of research. The first one is a two-step work including the weight map calculation and the fused image construction. The second one decomposed the image into a base layer and a detail layer to make full use of spatial information. Ma et al [13] utilized a gradient ascent-based algorithm to implement the iterative optimization for MEF. Meanwhile, to obtain better results, Ma et al [14] designed a method based on deep learning.

Image quality assessment methods

As mentioned in the introduction, the IQA methods include FR, RR, and NR forms [15]. The NR ones are more practical in real applications. With regard to the NR quality assessment for ordinary images, many methods rely on natural scene statistics (NSS). For example, Moorthy et al. [16] utilized NSS to design the method named distortion identification-based image verity and integrity evaluation (DIIVINE). Saad et al. [17] processed the image in the discrete cosine transform domain and established an NSS model, which is named BLINDS-II. Mittal et al. [18] proposed BRISQUE method based on the features extracted from the NSS distribution. Liu et al. [19] extracted features from curvelet transform domain, named CurveletQA. Fang et al. [20] considered the natural statistical characteristics of images with contrast distorted must be different from those of normal images and designed a method called ContrastQA. In image processing, the gradient is very important for images, which can represent the structure information of images. In view of this, to sense structure distortion, Xue et al. [21] extracted the features combining the Laplacian of Gaussian response and gradient magnitude map, named GradLog. Li et al. [22] describe the structure distortion by the gradient-weighted histogram of local binary pattern (GWH-GLBP). Liu et al. [23] considered the role of oriented gradient (OG) in quality prediction. Besides, Gu et al. [24] proposed the NIQMC method considering the contrast distortion of the image from the local and global perspectives. Jiang et al. [25] obtained a set of features, i.e., histogram, entropy, and structure, to achieve the goal of quality prediction, which also characterized the contrast. Oszust [26] presented a feature description form by image derivatives of different orders.

Above all are designed for ordinary images, and there is still a lack of NR-IQA methods for MEF images. For the FR methods of MEF images, Ma et al. [27] measured structure consistency and luminance consistency based on structural similarity index measure (SSIM). Xing et al. [28] built a multi-scale contrast-based model inspired by the fact that the human visual system (HVS) is highly sensitive to contrast. Martinez et al. [29] implemented two stages including multi-scale computation and structural similarity score. Xu et al. [30] generated the local and global intermediate references from the input multiple images to extract features, reflecting the visual quality of fused images. In real application, the visual effect of MEF images is similar to tone-mapped images. For tone-mapped images, there exist some NR-IQA methods. For instance, Gu et al. [31] developed a tone-mapped quality index (BTMQI) taking the information, naturalness and structure into consideration. Inspired by the NSS model, Kundu et al. [32] measured the differential NSS and proposed the high dynamic range image gradient-based evaluator (HIGRADE). Yue et al. [33] simulated the color information process in the human brain to propose a novel quality assessment method, meanwhile, they further presented another method [34] in terms of color, naturalness, and structure. Besides, Jiang et al. [35] pointed out the viewing habit of humans and evaluated the quality of tone-mapped images from a local and global perspective. Wang et al. [36] also considered the local degradation characteristics and global statistical properties to present a method. Although there are a few NR-IQA methods for ordinary and tone-mapped images, it is still necessary to propose NR-IQA methods specifically for MEF images.

Image decomposition

It is well known that the averaging techniques including average filter and lowpass filter are widely used to decompose the image. However, these schemes may introduce detail loss more or less, which will damage the MEF image twice. To obtain the complementary information among the MEF image, a two-scale decomposition scheme is used based on the joint bilateral filter. As the intensity information and structure are important for the well-preservation MEF image, it is decomposed into two layers (i.e., the energy layer and structure layer). The process includes global blur and edge recovery.

For global blur, the important purpose is to disseminate the detailed information of the MEF image to the decomposed structural layer to the maximum extent. Given a MEF image I, it can be processed by Gaussian filtering to achieve the blur effect, which is denoted as F_σ, defined as

(1)

(2)

where G_σ(x,y) is Gaussian filter, and the variance is σ².

Then, to generate the global blurred MEF image G, a weighted average Gaussian filter is utilized, which can be represented as

(3)

(4)

where m and n denote the index of pixel coordinates, S(m) is a set of neighboring pixels of n, σ_s is the standard deviation, and N_m is the normalization operation.

Subsequently, the joint bilateral filter is introduced to recover the large structure of the MEF image which is destroyed by the above smooth operation, so as to obtain the energy layer, denoted as I_E. The process is defined as

(5)

(6)

where

(7)

(8)

where g_d and g_r are the spatial distance function and intensity range function, respectively. They are used to set weights based on the distance among pixels and intensity differences, respectively. σ_s and σ_r control the spatial and distance weights of the bilateral filter [37], respectively.

For a MEF image I, let its structure layer be I_S, which can be calculated as

(9)

Fig 2 shows examples of the energy and structure layers decomposed from the MEF images with different mean opinion score (MOS) values. It can be observed that Fig 2(A) shows the dim visual effect with the lowest MOS value, it loses some detailed information. This is also reflected in its corresponding energy and structure layers. The visual quality of Fig 2(B) is at a normal level, which preserves well details and structural information, and its MOS value is at a moderate level. For Fig 2(C), its detailed and structural information seems similar to Fig 2(B), but obviously, the color information is more abundant than Fig 2(B), which determines that it has the highest MOS value. The phenomenon of similar information between Fig 2(B) and 2(C) is reflected in the corresponding energy layers. However, small detail and structure differences between them can be reflected in the corresponding structure layers. From the above observation, it can be drawn that the detail, structure, and color information are important for a well-quality MEF image. Therefore, the energy-related, structure-related and color-related features will be extracted to describe the distortion of the MEF image.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The examples of energy and structure layers obtained from the MEF images.

The first line represents the MEF images which are generated by three MEF algorithms, i.e., Raman09 [8], Gu12 [10], Li12 [9]. The second line shows the corresponding energy layers of the MEF images in the first line. The third line shows the corresponding structure layers of the MEF images in the first line. (a) Balloons_Raman09, MOS = 3.17; (b) Balloons_Gu12, MOS = 5.87; (c) Balloons_Li12, MOS = 8.74.

https://doi.org/10.1371/journal.pone.0283096.g002

Energy-related features

The energy layer I_E contains the details of the MEF image. Through the decomposition process, I_E has a slightly ambiguous effect. Different energy layers will have different blurring representations. We calculated the contrast values from the edge blocks of I_E.

First, I_E is divided into k×k blocks, k is set as 64, which is suggested in [38]. To judge whether the block is the edge block or the flat block, the Sobel operator is used for edge detection by calculating the proportion of edge pixels in the whole block. If greater than 0.2%, the block is the edge one. Then, let F_c be the feature of contrast, which can be defined as

(10)

where I_E^’(i, j) is the i-th, j-th element of the edge block with the size of k×k. T is the number of edge blocks in I_E, and is the average intensity of all pixel values in the whole I_E.

In addition, let F_e be the feature of energy to represent the amount of detailed information, which can be calculated as

(11)

where I_E(i, j) is the pixel value of I_E in the position of (i, j).

From the viewpoint of entropy, it can also discriminate the detailed information loss. Therefore, the entropy value of I_E is calculated to assist the features F_c and F_e. Let F_t be the feature of entropy, and the calculation formula is

(12)

where l is the pixel value of I_E, and p(l) is the probability density of l.

Through the above calculations, F_c, F_e and F_t form a feature group, i.e., energy-related features.

Structure-related features

Generally, the edge, corner and texture are important for the structure of an image. Based on the structure tensor salient detection operator from reference [39], gradient information can be detected efficiently, and the result obtained is denoted by S_T. In addition, to strengthen the gradient feature map obtained by structure tensor salient detection, the neighbor energy is calculated, denoted as N_E, and it can be defined as

(13)

where (x, y) denotes the central position of I_S, v controls the size of the window, and (2v+1)×(2v+1) is the size of the neighborhood.

Then, the enhanced gradient feature map S_NE is calculated as

(14)

Fig 3 gives examples of S_NE obtained from three different MEF images corresponding to the first line of Fig 2. It can be found that Fig 3 increases the difference of structure layers among three MEF images, which is crucial for the discrimination of different distortions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The examples of S_NE maps and distribution maps of their corresponding SVD results, where the corresponding MEF images are listed in the first line of Fig 2.

The first line represents S_NE maps, the second line represents distribution maps. (a) Balloons_Raman09, (b) Balloons_Gu12, (c) Balloons_Li12.

https://doi.org/10.1371/journal.pone.0283096.g003

To quantitate such differences, singular value decomposition (SVD) is used to calculate the primary information of S_NE, denoted as , where q is the number of the SVD diagonal coefficients. The distribution of S_svd is enumerated in the second line of Fig 3. It can be found that different S_svd values show different and regular distributions, which can be captured by the Weibull distribution model. It can be defined as (15)

where χ and ε denote the shape parameter and scale parameter.

The parameters constitute the first group of structure-related features, denoted as F_w.

To capture the statistical differences more comprehensively, the moment features including variance, kurtosis and skewness are extracted to assist the statistical distribution properties. The obtained features are denoted by F_m together and regarded as the second group of structure-related features.

Except for the statistical differences’ calculation, the Local Binary Pattern (LBP) operator of local rotation invariant uniform is used on the S_NE map to code the texture information, which can be expressed by

(16)

where D and R are the numbers of neighboring pixels and the radius, respectively. o and c stand for the center position and its neighborhood positions, respectively. riu2 means rotation invariant uniform pattern. ς(.) is the function of consistency measure calculated by the number of bit leaps, defined as

(17)

where ρ(.) is the threshold function, and it can be calculated by

(18)

Afterward, the histogram of can be calculated as

(19)

(20)

where Z is the total number of pixels in the , u is the possible LBP pattern.

The histogram features are denoted as F_h together and regarded as the third group of structure-related features.

Color-related features

As described in section 3.1, color information is very important for a high-quality MEF image. Since the RGB color space is highly correlated with each other, it is not proper for feature extraction. Using unrelated color space for feature extraction can eliminate the redundancy between different features, so as to perceive visual stimuli more effectively. Therefore, the RGB color space is converted into YUV color space. The obtained Y, U, and V color channels are utilized for the respective feature extraction.

For the Y color channel, the mean, standard deviation, and skewness are calculated as the first, second, and third-order moment statistics. As a result, the first group of color-related features is formed, denoted by F_o.

Besides the F_o, the natural scene statistic model is also used to extract features in the U and V color channels for representing the variation of color information. Specifically, U and V channel maps are processed by the local mean subtraction and divisive normalization, and then the mean subtracted contrast normalized (MSCN) coefficients are calculated, which can be given by

(21)

where k ∈ {U,V}, are the MSCN coefficients of k at the position of (x, y). μ(i, j) and δ(i, j) represent the local mean and standard deviation of k(i, j), respectively.

The color information of different MEF images will have different MSCN coefficients distributions, which can be caught by the generalized Gaussian distribution (GGD). The GGD can be expressed by

(22)

where , α and λ² are the shape and variance of the Gaussian distribution, respectively, is the Gamma function.

Finally, the α and λ² of U and V color channel maps constitute the second group of color-related features, denoted by F_n.

Quality regression

With the extracted energy-related features F_c, F_e and F_t, structure-related features F_w, F_m and F_h, and color-related features F_o, F_n, the whole feature vector can be denoted by F_all = {F_c, F_e, F_t, F_w, F_m, F_h, F_o, F_n}. Then, random forest (RF) is adopted to learn the mapping relationship from the whole feature vector F_all to the subjective rating. The RF model can be expressed by

(23)

where is the mapping function by RF, and Q is the predicted quality score by the trained RF model.

Experimental settings

1) Dataset: The proposed method is tested on an open MEF image dataset [40], the details of the dataset are shown in Table 1. The dataset is consisting of 136 MEF images, which were obtained from the MEF algorithms, i.e., local energy weighted linear combination, global energy weighted linear combination, Li12 [9], Raman09 [8], ShutaoLi12 [11], Mertens07 [7], Gu12 [10], ShutaoLi13 [12].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The details of the MEF image database [40].

https://doi.org/10.1371/journal.pone.0283096.t001

2) Performance criteria: To validate the performance of the proposed method, three common indicators are used including the Person linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC) and Root mean squared error (RMSE). For these indicators, PLCC is utilized for prediction accuracy evaluation, SROCC calculates the prediction monotonicity, and RMSE represents the prediction error. For an excellent IQA method, the PLCC and SROCC are close to 1, and RMSE is close to 0. They can be represented as

(24)

(25)

(26)

where Q_i and M_i are the predicted quality score and the MOS value of the i-th MEF image, while and are the mean values of Q_i and M_i, respectively. l_i represents the rank difference of i-th MEF image between the objective and subjective quality assessments. K represents the number of MEF images.

3) Evaluation protocol: In the experiments, k-fold cross-validation is employed to test the performance of the proposed method. Since the MEF dataset can be divided into 17 subsets based on the number of the source MEF images, k is 17, 16 subsets are used for quality prediction model training, and the remaining subset is used to test. Finally, the mean performance is reported through the 17-cross verification.

Overall performance comparison

To validate the performance of the proposed method, comparative experiments are implemented between the proposed method and 13 NR-IQA methods for ordinary and tone-mapped images on the above-mentioned MEF dataset. Among these compared methods, 10 of which are designed for ordinary images, namely DIIVINE [16], BLIND-II [17], BRISQUE [18], CurveletQA [19], GradLog [21], ConstrastQA [20], GLBP [22], OG [23], NIQMC [25], and SCORER [26]. The remaining 3 methods are designed for tone-mapped images, involving BTMQI [31], HIGRADE-1 [32], HIGRADE-2 [32]. The PLCC, SROCC and RMSE results of such comparative methods and the proposed method are shown in Table 2. Note that the best performance values are highlighted in bold.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The overall performance comparison results.

https://doi.org/10.1371/journal.pone.0283096.t002

From Table 2, it can be drawn that, the methods designed for ordinary images show poor performance. Since they mainly describe the distortion representation of ordinary images, the features extracted by these methods cannot match the special distortion of MEF images. It is obvious that they are not proper to predict the quality of MEF images. Among them, the GradLog method exhibits relatively better performance, which may attribute to the consideration of structure information based on the gradient and Gaussian response. It can well represent the structure loss in the process of MEF, and the PLCC and SROCC values achieve the level of 0.631 and 0.567, respectively. In addition, the performance of the methods designed for tone-mapped images reaches higher PLCC and SROCC values than that of the methods designed for ordinary images. The main reason is that the MEF images and tone-mapped images both have the problem of underexposure and overexposure. Finally, the performance of our proposed method is superior to other methods analyzed above. The proposed method makes use of the idea of energy layer and structure layer decomposed from MEF image based on the joint bilateral filter to percept the detail and structure information and so on. Besides, color information is also considered. In summary, the energy-related, structure-related, and color-related features can supplement each other.

To illustrate the performance comparison results more intuitively, Fig 4 gives out the scatter plots of the proposed method as well as the competing methods. As can be found, the proposed method has superior convergence than the other comparative methods and the points can be fitted by the logistic function better. So, it is obvious that the proposed method achieves better performance than other comparative methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Scatter plots for the objective predicted scores and MOS values of sixteen NR-IQA methods.

(a) DIIVINE [16]; (b) BLINDS_II [17]; (c) BRISQUE [18]; (d) CurveletQA [19]; (e) GradLog [21]; (f) ContrastQA [20]; (g) GLBP [22]; (h) OG [23]; (i) NIQMC [25]; (j) SCORER [26]; (k) BTMQI [31]; (l) HIGRADE-1 [32]; (m) HIGRADE-2 [32]; (n) Proposed.

https://doi.org/10.1371/journal.pone.0283096.g004

Ablation study

Through the above overall performance comparison, the proposed method has been validated to have superior performance to other methods for ordinary and tone-mapped images. However, different features constitute the whole feature vector, and their own contribution should be tested. The results are given out in Table 3, including the individual features and the combinations of different features. Among the performances of different features, it can be drawn that, the energy-related features, the structure-related features and the color-related features contribute equally, and the combination of them maximizes performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance comparison of different feature combinations.

https://doi.org/10.1371/journal.pone.0283096.t003

Even so, the combination of multiple classes of features may cause redundancy. Therefore, feature selection is utilized based on the RF to determine the final feature vector, and the results are listed in Table 4. It can be clearly found that the 15-dimension feature vector selected from the whole extracted features obtained the best performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance comparison with features of different dimensions.

https://doi.org/10.1371/journal.pone.0283096.t004