2.1 Histogram equalization-based methods

Inspired by Histogram equalization (HE), a large number of improved methods based on HE have been proposed. For example, Pizer et al. [5] proposed an contrast limited adaptive histogram equalization (CLAHE), which restricts each sub-histogram. CLAHE improves the image contrast and suppresses noises, but it first divides the image into multiple block regions, which result in block artifacts in the reconstructed image. To solve the block artifact phenomenon, Srinivasan et al. [19] proposed a histogram equalization using local region stretching, which divides the image into multiple areas according to the gray level, and then histogram equalization is performed for each gray level area. Ibrahim et al. [6] proposed dynamic histogram equalization with preserved luminance, which has a good effect on some locally underexposed images by calculating the global average luminance. However, this method is greatly limited when the image luminance is extremely dark. Although histogram equalization-based methods and their improvement methods are simple and convenient, they usually suffer from image distortion and noises problems.

2.2 Retinex-based methods

Retinex is a theory that imitates human vision, and it is widely used in image enhancement tasks. In recent years, a large number of Retinex-based methods have been proposed. For example, SSR [7], MSR [8] and MSRCR [9], they are pioneering methods. Wang et al. [10] proposed a method to improve the contrast and naturalness of images, but the method still produces severe noises. Cai et al. [11] proposed joint intrinsic-extrinsic prior model (JieP) to protect structural integrity through shape prior. Guo et al. [20] proposed LIME to accelerate the optimization of the illumination map, which greatly reduces the computations. Xu et al. [21] proposed a new detail-preserving variational enhancement model that uses the l1 prior to constraining the illumination map. Retinex-based methods are usually better for image color fidelity, but these methods are poorly adaptable and also prone to artifacts, strong noises, and overexposure. In addition, it is difficult to estimate the illumination map of extremely low-light images with heavy noises.

2.3 Learning-based methods

In recent years, learning-based methods achieved great progress. LLNet [12] was proposed as an early end-to-end method for low-light image enhancement. Wang et al. [13] proposed a globally aware network (GLAD), which incorporates a pyramid structure to extract image features. MIRNet [14] combines a residual structure and an attention mechanism to reconstruct the correct luminance and color. Some methods combine Retinex theory, for example. Wei et al. [15] proposed RetinexNet, which designed a decomposition net and an enhancement net to decompose and process images. Inspired by Ref [15], Zhang et al. [16] proposed KinD, which designed a light adjustment block to flexibly adjust the luminance. Wang et al. [22] proposed DeepUPE, which combines Retinex and bilateral filtering. Liu et al. [23] proposed RAUNA, which designed a decomposition network with explicit and implicit priors, and taken into account both global and local brightness in the augmented network. Some methods try to learn enhancement results from unpaired data, such as Jiang et al. [24] proposed a degradation-to-refinement generation network (DRGN). Li et al. [17] proposed zero-reference depth curve estimation (Zero-DCE++) for low light image enhancement. Risheng et al. [18] proposed Retinex-inspired unrolling with cooperative prior architecture search (RUAS), which first searches the network architecture through the training set, and then finds the low-light prior structure from the compact search space. These methods further liberate the restriction on the dataset, but they are not stable in training. Compared with traditional methods, learning-based methid combined with Retinex theory can better estimate the illumination and reflection maps and have better enhancement performance, but the noises suppression of these methods is still inadequate. Compared with depth Retinex-based methods, the end-to-end methods are more effective in suppressing noises. The end-to-end methods usually have low complexity and has great potential for assisting high-level vision tasks and applying on mobile devices. Recently, Li et al. [25] proposed a luminance-aware pyramid network (LPNet). LPNet is a lightweight end-to-end method, and it has strong competitiveness. LPNet designed a multi-scale contrast feature block for feature extraction and a luminance loss for guiding the model to learn luminance information. However, LPNet uses too many split operations, resulting in insufficient information exchange between channels and the reconstructed image lacks texture. In addition, due to the inconsistency of pyramid structure, the shallow features and deep features of the pyramid structure cannot be fully utilized. This causes color deviations in the reconstructed images by LPNet.

In our work, we propose an efficient adaptive feature aggregation network (EAANet). EAANet contains two important blocks, MFAB and AFAB. MFAB utilizes asymmetric convolution and dual attention mechanisms to fully extract features and capture critical features. Compared with LPNet, MFAB is more efficient to extract features. AFAB overcomes the inconsistency of pyramid structure and enhances the communication between shallow fine-grained information and deep semantic information. We describe the proposed EAANet in Section 3.

3.1 Multi-scale feature aggregation block (MFAB)

Some methods designed novel structures to extract abundant features. For example, MSRN [32] uses convolutional layers with different kernels. LPNet [25] splits features along the channel dimension and then passes through different numbers of convolutional layers, respectively. However, the feature extraction capability of these methods is inefficient. Convolutional layer with large kernel generates a large number of parameters, and the split operation can hinder the information interaction between channels and degrade the feature representation. We proposed the MFAB to better extract and utilize the features. As shown in Fig 2, the feature from the previous layer is divided into two branches by two 1 × 1 convolutional layer, respectively. Each branch has only half the number of channels. One branch passes through a 3 × 3 convolutional layer, and then waits for a concatenation with another branch. The other branch can be roughly divided into two parts: An information aggregation block (IAB) and a double attention block (DAB).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

(a) The structure of our proposed MFAB. MFAB contains IAB and DAB. (b) The structure of ACB. ACB transforms to standard convolution in the inference stage.

https://doi.org/10.1371/journal.pone.0272398.g002

Learning-based methods are often necessary to design complex structures to obtain rich feature information. ACNet proposes an efficient method for capturing feature information, it try to enhances the standard convolution kernel using asymmetric convolution. Inspired by ACNet [34], we designed an asymmetric convolution block (ACB), which contains 2 asymmetric convolutions and 1 standard convolution. The outputs of the asymmetric convolution and the standard convolution are combined by an element summing operation and Leaky Relu activation is added. For the IAB, the features pass through three ACB. During the training phase, asymmetric convolution block contains asymmetric convolution and standard convolution, the expression can be defined as Eq (8): (8) where U is the feature of the previous layer. K is convolution kernel. δ is Leaky Relu activation. During the inference phase, we transform the asymmetric convolution to standard convolution to reduce the number of parameters of the network while maintaining performance, and the operation can be defined as Eq (9): (9) where ⊕ is the element-wise addition of the kernel parameters. Receptive field has a great influence on feature extraction, and the receptive fields of the feature output by each ACB are different. We concatenate the output of each asymmetric convolution block to fuse the feature information of different receptive fields.

Low-light images have a lot of noises and the extracted features are often redundant. Attention mechanisms are used for high-level vision tasks such as image classification, image recognition, etc. It can promote models to focus more on valuable features and fade out useless features. Several methods [25, 35, 36] have demonstrated that the attention mechanism helps low-level visual models to learn the correct texture and suppress noises. Therefore, we try to use the attention mechanism to reduce feature redundancy. Specifically, we designed parallel spatial and channel attention mechanisms. Color degradation is a challenge for low-light image enhancement. Inspired by Ref [37], we added standard deviation (STD) based on global average pooling to the channel attention branch, as shown in Fig 3. The standard deviation is calculated as the degree of dispersion of individual pixels from the mean value of the image. By adding standard deviation, the model can be promoted to pay more attention to edge information, so the color of images can be improved. For the standard deviation, we define X = [x₁, x₂, …., x_c] as input, whose shape is C × H × W, and the expression can be defined as Eq (10): (10) where ε_c is the standard deviation of the c-th element of the output. STD can improves the color of images. In order to reduce the computational complexity, existing channel attention mechanisms usually perform channel dimensionality reduction, and this operation lead to the result that the model cannot effectively capture dependencies between channels and lose some important contextual information. Inspired by Ref [38], we use 1-dimensional convolution to avoid dimensional reduction. The output of the global average pool is first element-wise summed with ε_c, and then a one-dimensional convolution is used to avoid channel reduction, and the one-dimensional convolution allows sufficient information interaction between channels. The features are normalized by the Sigmoid activation function and finally the original features are rescaled by the element multiplication operation. In the spatial attention branch, we do average pooling and maximum pooling for the spatial dimensions, respectively. The output of both pooling ways are concatenated and then the valuable features are learned from the spatial dimension by convolutional layers. Finally, sigmod activation and element multiplication operations are performed to rescale the features. The outputs of spatial attention and channel attenion are concatenated and pass through a convolution layer to adjust the channels. Our dual attention block can be defined as Eqs (11)–(13): (11) (12) (13) where F_ch and F_sp is channel and spatial attention. F_DAB is our dual attention block. Conv7 denotes a 7 × 7 convolutional layer. GAP is global average pooling. GMP is global max pooling. STD is the standard deviation. Finally, the two branches are concatenated and a residual structure is added to stabilize the gradient propagation. In the spatial attention branch, the convolution kernel size is set as 7. In the channel attention branch, the convolution kernel of the 1-dimensional convolution is set as 3. Due to the combination of asymmetric convolution and 1D convolution, MFAB is very lightweight and also has excellent performance to efficiently extract valuable features and fade out redundant features. The reconstructed images have vivid colors and rich textures.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The structure of our proposed dual attention.

Conv1d is a 1-Dimension convolution. We set the kernel of Conv1d as 3. ⊗ is element-wise multiplication operation.

https://doi.org/10.1371/journal.pone.0272398.g003

3.2 Adaptive feature aggregation block (AFAB)

Existing methods often use a pyramid structure to exploit multi-scale features. The pyramid structure incorporates deep semantic information into the shallow layer through concatenation and element-wise summation operation to guide the model to learn finer features. But in practice, the inconsistency of the pyramid structure hinders the information interaction between shallow fine-grained features and deep semantic information. Liu et al. [39] proposed adaptive spatial feature fusion (ASFF) to address the effect of feature scale inconsistency on single-shot detectors. ASFF assigns weights to features to suppress feature conflicts between different scales, thus improving the scale invariance of features. Yi et al. [40] proposed an adaptive feature selection module (AFSM) for fusing neighboring scale features to improve the image dehazing effect. However, both ASFF and AFSM reduce the channel dimension so as to reduce the number of parameters, which leads to the underutilization of features. AFSM incorporates a channel attention mechanism, but the module only adaptively selects features at neighboring scales, ignoring the association of features at non-neighboring scales. We designed an AFAB, it receives features from each layer of the pyramid structure and adaptively rescales the features at each layer by the multi-branch attention structure. In addition, AFAB combines multi-branch attention structure with 1D convolution to enhance the channel information interaction and fully utilize the valuable information. As shown in Fig 4, L₁ is . L₂ and L₃ are obtained by up-sampling the outputs of and . The inputs of the three branches are fused by element-wise summation. Then the fused features generate global information by global average pooling. Conv1d means 1-dimensional convolution. The softmax activation values of the three branches are multiplied by the original features on each branch and then fused by element-wise summation operation. The operation of AFAB can be defined as Eqs (14) and (15): (14) (15) where S_N denotes the output of the three branches of AFAB. ⊗ denotes element-wise multiplication operation. The AFAB can efficiently establish the dependencies between depth-layers semantic information and shallow fine-grained information, making full use of features at different scales. In addition, the block is lightweight and only adds a small number of parameters. The pseudo code of our proposed method is shown in Algorithm 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The structure of our proposed AFAB.

We set the kernel of Conv1d as 3.

https://doi.org/10.1371/journal.pone.0272398.g004

Algorithm 1: Efficient Adaptive Feature Aggregation Network for Low-light Image Enhancement

 Input: Low light /Normal light sample set .

 Output: Enhanced image .

 Initialization: Learning rate η, batch size m, parameter θ.

1 for k = 1, 2, 3 …… N do

2  I₁ is extracted by Eq (1);

3  Extract the I_n by Eq (2), n = 2,3;

4  F_ACB is calculated by Eq (8);

5  ε_c is calculated by Eq (10), and F_DAB is calculated by Eq (13);

6  Extract the by Eq (4);

7  Extract the by Eq (5), n = 1,2;

8  F_AFAB is calculated by Eq (15);

9  Calculate by Eq (6) to update enhanced image ;

10  Update parameter θ with adam optimizer;

11  Update Learning rate η;

12 end

13 Asymmetric convolution kernels are fused by Eq (9);

14 Obtain final enhanced image .

3.3 Loss function

We designed a loss function, which consists of two parts: Content loss and perceptual loss.

Content Loss: L2 loss tends to produce overly smooth images. The images generated by L1 loss cannot effectively suppress noises. SSIM loss measures the similarity of the output to ground-truth by brightness, contrast, and structure. The images generated by SSIM loss contain less noises and are of good quality. Therefore, we use SSIM loss as content loss. Content loss can be defined as Eqs (16) and (17): (16) (17) where Loss_c is content loss. x, y are the generated image and the target image respectively. μ_x, μ_y are the average values of x and y respectively. , are the variances of x and y respectively. σ_xy is the covariance of x and y.C₁, C₂ are constants to avoid dividing by zero.

Perceptual loss: Johnson et al. [41] proposed a perceptual loss function to optimize visual effects, which is based on the VGG model. We introduce this loss function to improve the image visual quality. We use a pretrained VGG19 model, and the model weights are obtained by training on the ImageNet dataset [42]. Perceptual loss can be defined as Eq (18): (18) where Loss_p is the total loss. C_j, H_j, and W_j are the number of channels, height, and width, respectively. ϕ_i is the output of layer m of the VGG19 network. When we use features from different layers of VGG, the optimization effect on the model will be different, we use the output of the last convolution of the features layer to calculate the mapping differences. The two losses have a weight ratio of 1.

4.1 Datasets

In experiments, we use the datasets LOL [15] and MIT5K [43]. The LOL dataset is the first paired dataset collected in real scenes, containing 485 train pairs and 15 test pairs. We follow the previous experience and extract 35 images from the training set for evaluating. The MIT5K dataset is often used by researchers in low-light image enhancement field. ln order to make a fair comparison with other methods, we use the Expert C-retouched image as the ground truth. We use 4500 pairs for training and the remaining 500 pairs for evaluating and testing. For the MIT5K dataset, we only consider image enhancement under RGB. The dataset can be made by using Lightroom software, setting the length as 500. The image sizes from the LOL dataset and the MIT5K dataset are 600 × 400 and 500 × 333, respectively.

4.2 Implementation details

We train our network on NVidia GTX 1080ti GPU by using the pytorch framework. The network is trained for 150 epochs. The optimizer uses the Adam optimizer, and the Adam optimizer parameters use the framework default values. We use the cosine annealing strategy and set the initial learning rate as 2 × 10⁻⁴. The learning rate threshold is set as 2 × 10⁻⁶. To enhance the robustness of the network, we use random rotation, mirroring for data augmentation. We set the batch size as 16, the patch size as 96 × 96, and set the channel as 32.

4.3 Comparison with typical methods

We compared some recent work on the LOL dataset, including Xu’s method [21], RetinexNet [15], GLAD [13], KinD [16], LPNet [25], Zero-DCE++ [17], RUAS [18], RAUNA [23], DRGN [24]. For the KinD method, we chose an exposure of 5 as the benchmark. Similarly, on the MIT5K dataset, we compared with some recent work including Xu’s method, White-Box [44], DPE [45], DeepUPE [22], LPNet, Zero-DCE++, RUAS, RAUNA, DRGN. The code and pre-trained models for the above methods were provided by the original authors. For Zero-DCE++, RUAS and DRGN, since the training datasets are different, we retrain the model on the two datasets to do a fair comparison. In this subsection, we compare three aspects of quantitative comparison, efficiency comparison, and visual comparison.

4.3.1 Quantitative comparison.

We use 2 metrics, PSNR and SSIM, for quantitative comparison. PSNR is a commonly used image objective evaluation index, the higher the PSNR value, the higher the image quality. SSIM measures the image quality in terms of luminance, contrast and structure, and the SSIM value is closer to 1, the higher the image quality. Tables 1 and 2 reflect the results of the quantitative comparison of the methods on the LOL and MIT5K datasets, where the values of PSNR and SSIM are the average of the test sets. We can see that the PSNR and SSIM of Xu’s method, Zero-DCE++, RUAS are not good on both datasets. RAUNA has good PSNR, but SSIM is not well. It can be clearly seen that our method achieved the best PSNR and SSIM on both datasets, which indicates that our method has the best quality of the images. Our method can effectively learn the mapping relationship between low luminance and normal luminance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Quantitative comparison on LOL dataset.

https://doi.org/10.1371/journal.pone.0272398.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Quantitative comparison on MIT5K dataset.

https://doi.org/10.1371/journal.pone.0272398.t002

4.3.3 Visual comparison.

We show the visual comparison of some state-of-the-art methods on LOL and MIT5K datasets respectively. The visual comparison results on the LOL dataset are shown in Figs 5 and 6. We can see the luminance of the images reconstructed by Xu’s method are low. The images generated by GLAD and RetinexNet have a lot of noises. The images produced by KinD are too smooth and has some color distortion. LPNet and DRGN lack details. In addition, LPNet cannot handle the mirror area well. The enhanced images based on Zero-DCE++ and RUAS have a lot of noises and severe color distortion. The images recovered by RAUNA have a small amount of noises. The images generated by our method have rich colors and details while suppressing noises. The visual comparison results on the MIT5K dataset are shown in Figs 7 and 8. It can be seen that the recovered images by White-Box have overexposure. The images reconstructed by DPE and LPNet have low luminance. There are noises and color deviation in the images reconstructed by DeepUPE, RAUNA and DRGN. The enhanced images based on Zero-DCE++ and RUAS have severe distortion. The images generated by our method have better visual perception.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Visual results of different methods on LOL dataset for image number 55.

(a) Input. (b) Xu’s method. (c) RetinexNet. (d) GLAD. (e) KinD. (f) LPNet. (g) Zero-DCE++. (h) RUAS. (i) RAUNA. (j) DRGN. (k) Ours. (l) GT.

https://doi.org/10.1371/journal.pone.0272398.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Visual results of different methods on LOL dataset for image number 111.

(a) Input. (b) Xu’s method. (c) RetinexNet. (d) GLAD. (e) KinD. (f) LPNet. (g) Zero-DCE++. (h) RUAS. (i) RAUNA. (j) DRGN. (k) Ours. (l) GT.

https://doi.org/10.1371/journal.pone.0272398.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Visual results of different methods on MIT5K dataset for image number 4501.

(a) Input. (b) Xu’s method. (c) White-Box. (d) DPE. (e) DeepUPE. (f) LPNet. (g) Zero-DCE++. (h) RUAS. (i) RAUNA. (j) DRGN. (k) Ours. (l) GT.

https://doi.org/10.1371/journal.pone.0272398.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Visual results of different methods on MIT5K dataset for image number 4832.

(a) Input. (b) Xu’s method. (c) White-Box. (d) DPE. (e) DeepUPE. (f) LPNet. (g) Zero-DCE++. (h) RUAS. (i) RAUNA. (j) DRGN. (k) Ours. (l) GT.

https://doi.org/10.1371/journal.pone.0272398.g008

The above visual comparison results reflect the advantages of our method. On two datasets, the enhanced images of our method are more realistic and natural.

5.1 Effectiveness of multi-scale feature aggregation block

We investigate the effect of MFAB, as shown in Table 3. Case 4 is the proposed method EAANet. Case 3 removes MFAB and we can see that the value of PSNR and SSIM for the model have decreased. As shown in Fig 10 (c), the visual results of the model without MFAB are obviously bad. It shows that MFAB has a positive impact on the model. MFAB contains IAB and DAB. We conduct a series of ablation studies to verify the effectiveness of each component.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Study on EAANet.

Pyramid, AFAB and MFAB are pyramid structure, adaptive feature aggregation block and multi-scale feature aggregation block, respectively.

https://doi.org/10.1371/journal.pone.0272398.t003

5.1.1 Effectiveness of information aggregation block (IAB).

As shown in Table 4, we use Case 3 as the benchmark data. To verify the effectiveness of IAB, we use three 3 × 3 convolutions instead of IAB. Each convolution is followed by an Leaky Relu activation. The results are shown in Case 1 of Table 4. Compared with Case 3, the PSNR and SSIM of Case 1 are significantly lower, which shows the effectiveness of IAB. In addition, we also conducted an ablation study to investigate the effect of ACB on the model, Case 2 use asymmetric convolution instead of standard convolution. We can clearly see that the PSNR and SSIM of Case 2 are higher than that of Case 1. This adequately illustrates the effectiveness of asymmetric convolution. Asymmetric convolution can extract richer features. In the inference phase, asymmetric convolution can be transformed into standard convolution to reduce the computations while maintaining the model performance. The PSNR and SSIM of Case 2 are lower than that of Case 3. This also proves the effectiveness of the aggregation structure. IAB fuses the features from different receptive fields to ensure the maximum utilization of features. We further compare with the feature extraction structure of some classical methods, as shown in Fig 9. The structure in Fig 9(a) is from LPNet. The structure in Fig 9(b) is from MSRN. We replace the IAB with the above structures for comparison experiments respectively, and the results are shown in Table 5. It can be seen that Case 3 achieves the best results. Case 1 is lightweight, but the excessive split operation limits the interaction of channels and leads to the underutilization of features. Case 2 achieves good results, but generates more parameters and computations. The above experiments illustrate that IAB is efficient.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Feature extraction structure of different methods.

(a) Split+Conv3. (b) Conv3+Conv5. (c) IAB.

https://doi.org/10.1371/journal.pone.0272398.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Study on Information Aggregation Block (IAB).

ACB, Aggregation are asymmetric convolutional block, aggregation structure respectively.

https://doi.org/10.1371/journal.pone.0272398.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Study on Information Aggregation Block (IAB).

ACB, Aggregation are asymmetric convolutional block, aggregation structure respectively.

https://doi.org/10.1371/journal.pone.0272398.t005

5.1.2 Effectiveness of dual attention block (DAB).

We investigated the effectiveness of DAB, as shown in Table 6, and we used Case 4 as the baseline data. Compared with Case 4, the PSNR and SSIM are significantly lower in Case 1, demonstrating that the DAB has a positive effect. To further illustrate the advantages of dual attention block, we investigated the effectiveness of each component for the dual attention block. Case 2 uses only channel attention (CA). Case 3 adds spatial attention (SA) to CA. We can see that both CA and SA can improve the model. Case 4 is the proposed DAB. We can see that Case 4 has the best performance, which proves the effectiveness of STD. As shown in Fig 10(h) has the best visual effect.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Visual results of models with different components on LOL dataset.

(a) Input. (b) w\o pyramid. (c) w\o MFAB. (d) w\o AFAB. (e) w\o DAB. (f) CA. (g) CA+SA. (h) Ours. (i) GT.

https://doi.org/10.1371/journal.pone.0272398.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Study on Dual Attention Block (DAB).

CA,SA,STD are channel attention, spatial attention, and standard deviation respectively.

https://doi.org/10.1371/journal.pone.0272398.t006

The above study verifies the effectiveness of IAB and DAB. IAB can extract features efficiently, while DAB can reduce redundant features.

5.2 Study on the number of multi-scale feature aggregation blocks

In this part, we explored the effect of the number of multi-scale feature aggregation blocks (MFABs). In general, deeper network has more parameters and stronger fitting ability. We provided the results of PSNR, SSIM, parameters, computations and the running time by using different numbers of MFABs, as shown in Table 7. When the number of MFABs gradually increases, the model can obtain better PSNR and SSIM, but the parameters, computations and running time of the model also rise. When the number of MFABs is more than 4, the rising trend of PSNR becomes slow and SSIM appears to decrease, while the parameters and computations of the model increase greatly. In terms of lightweight and efficiency, we set the number of MFABs as 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Study on the number of Multi-scale Feature Aggregation Blocks (MFABs).

https://doi.org/10.1371/journal.pone.0272398.t007

5.3 Effectiveness of network structure

In this part, we performed a series of ablation experiments on network structure, pyramid structure and AFAB. We explored the impact of the pyramid structure, as shown in Table 3. Case 1 removes the pyramid structure and AFAB, retains only the shallow branches, and doubles the number of channels to equal the number of parameters in Case 4 for a fair comparison. We can see that the PSNR and SSIM of Case 1 are significantly lower due to the lack of a pyramid structure. As shown in Fig 10, the visual results of the model without the pyramid structure are significantly worse. The experimental and visual results show that the pyramid structure is effective.

We investigated the effect of AFAB and the results are shown in Table 3. Case 2 removes AFAB, and it can be seen that the PSNR and SSIM of Case 2 are significantly lower. We give the corresponding visual effects, as shown in Fig 10, the images generated by the model using AFAB have better contrast and color. AFAB solves the inconsistency of the pyramid and enables the deep semantic information to better guide the shallow fine-grained information to learn finer textures and colors. To illustrate the advantages of the proposed AFAB, we further compare it with ASFF [39] and AFSM [40]. Since AFSM only performs adaptive fusion of neighboring scale features, we improved it to receive features from each layer of the pyramid. As shown in Table 8, we can see that AFAB obtains the best value of PSNR and SSIM. This indicates that AFAB can better adaptively fuse the features of different scales.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Study on Adaptive Feature Aggregation Block (AFAB).

https://doi.org/10.1371/journal.pone.0272398.t008

In summary, all parts of the model are effective, and our model achieves good performance with the synergy of each component.