Unsupervised water scene dehazing network using multiple scattering model

Shunmin An; Xixia Huang; Linling Wang; Zhangjing Zheng; Le Wang

doi:10.1371/journal.pone.0253214

Abstract

In water scenes, where hazy images are subject to multiple scattering and where ideal data sets are difficult to collect, many dehazing methods are not as effective as they could be. Therefore, an unsupervised water scene dehazing network using atmospheric multiple scattering model is proposed. Unlike previous image dehazing methods, our method uses the unsupervised neural network and the atmospheric multiple scattering model and solves the problem of difficult acquisition of ideal datasets and the effect of multiple scattering on the image. In our method, in order to embed the atmospheric multiple scattering model into the unsupervised dehazing network, the unsupervised dehazing network uses four branches to estimate the scene radiation layer, transmission map layer, blur kernel layer and atmospheric light layer, the hazy image is then synthesized from the four output layers, minimizing the input hazy image and the output hazy image, where the output scene radiation layer is the final dehazing image. In addition, we constructed unsupervised loss functions which applicable to image dehazing by prior knowledge, i.e., color attenuation energy loss and dark channel loss. The method has a wide range of applications, with haze being thick and variable in marine, river and lake scenes, the method can be used to assist ship vision for target detection or forward road recognition in hazy conditions. Through extensive experiments on synthetic and real-world images, the proposed method is able to recover the details, structure and texture of the water image better than five advanced dehazing methods.

Citation: An S, Huang X, Wang L, Zheng Z, Wang L (2021) Unsupervised water scene dehazing network using multiple scattering model. PLoS ONE 16(6): e0253214. https://doi.org/10.1371/journal.pone.0253214

Editor: Thippa Reddy Gadekallu, Vellore Institute of Technology: VIT University, INDIA

Received: April 29, 2021; Accepted: May 29, 2021; Published: June 28, 2021

Copyright: © 2021 An et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the manuscript and its Supporting information files.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Haze is a common atmospheric phenomenon that in water conditions such as oceans, rivers, lakes, etc. In these conditions, the frequent occurrence of haze often causes the captured images to lose detailed information, and affects lots of visual tasks in water conditions, such as target detection and object classification. So far, although many methods of image dehazing have been proposed and achieved desired effects [1–6], the dehazing algorithms for water conditions have not been paid much attention to. Unlike land conditions, there is a lot of dense haze in water conditions and it is difficult to collect ideal data sets. Some image dehazing methods suitable for land conditions will result in halo and blur when used in water conditions. There are two reasons for this phenomenon. On the one hand, the haze in water conditions is dense, and multiple scattering has a great influence on the image. But the image dehazing algorithms of land conditions only consider single scattering. So, the blur phenomenon will appear when use the atmospheric single scattering model to remove haze in water conditions. On the other hand, it is difficult to collect ideal datasets in water conditions, and the model trained with the datasets of land conditions will cause halo phenomenon when applied to water conditions. The main contributions of our method of image dehazing in water conditions are as follows: the unsupervised network and atmospheric multiple scattering model are adopted in this method, which solves the problem of the difficulty of collecting ideal datasets and the influence of multiple scattering on the image. In addition, this method has a wide range of application prospects. The changing environment in the real hazy condition will reduce the effect of dehazing of the pre-trained model. But the unsupervised method and multiple scattering model have been adopted in our method, which could be adapted to the changeable real-world haze conditions well, for example: in the ocean condition, the haze is thick and changeable, so this method can assist the visual analysis of shipping channel. In the following, we will introduce the existing image dehazing methods and our proposed method.

Physical models of imaging in hazy conditions can be divided into atmospheric single scattering model [7] and atmospheric multiple scattering model [8]. Among them, Wang et al. [8] and He et al. [9] used the atmospheric multiple scattering model for dehazing, which achieved better effect compared with the atmospheric single scattering model. According to the atmospheric scattering model proposed by Narasimhan and Nayar [7, 10], it can be seen that the atmospheric single scattering model is a degenerate form of the atmospheric multiple scattering model, and the atmospheric multiple scattering model takes into account the interference caused by multiple scattering to the image, while the atmospheric single scattering model only considers the case of single scattering. Therefore, multiple scattering is widespread in dense haze conditions, and the method of dehazing using atmospheric single scattering model often fails to achieve the desired effect. At present, image dehazing methods can be roughly divided into two categories: the prior-based method and the learning-based method. It should be pointed out that the two methods use the atmospheric single scattering model as the physical model to the most extent. The prior-based method manually extracts the features of the hazy image through observation and hypothesis [1, 3, 4, 11, 12]. For example, He et al. [11] proposed a method based on dark channel prior knowledge statistics to solve the problem of image dehazing, and got haze-free images by this prior. But almost all the methods of image dehazing that based on the prior rely too much on prior knowledge, which result in unsatisfactory dehazing effects in certain changing conditions. In order to reduce the dependence on prior knowledge, the learning-based method has attracted extensive attention. Unlike the prior-based method, the parameters of the atmospheric single scattering model are learned from the training data set in the learning-based method [2, 13–20]. For example, Cai et al. [14] proposed an end-to-end method of image dehazing for the problem of image dehazing that relies too much on prior knowledge. Cai et al. [14] automatically extracted the characteristics of haze in hazy images and successfully estimated the transmission map through neural network, and finally achieved dehazing results. But almost all learning-based methods of image dehazing are carried out through supervision and training. Therefore, if the gap between the training set and the real condition is too large, its effect is still not ideal. But the unsupervised image enhancement method only focuses on to the information of the input image itself and does not require a large number of training sets to train the network model, it has attracted widespread attention from researchers in recent years. At the same time, a variety of image enhancement methods based on unsupervised learning have been proposed [5, 21–23]. In summary, in the prior-based image dehazing methods, there are too much reliance on the prior knowledge, resulting in suboptimal dehazing in certain changing scenes. In learning-based image dehazing methods, ideal datasets are difficult to collect, so network models trained on synthetic datasets are not ideal for application to real-world scenes. Image dehazing has been little studied in unsupervised image enhancement methods, and the above methods do not consider the effect of multiple scattering on the image.

We noticed that the haze in the water condition is dense and it is difficult to collect he ideal dataset. Previous methods of image dehazing dehaze through the atmospheric single scattering model and synthetic dataset, so the dehazing effect is not ideal. Unlike previous dehazing methods, we proposed a reconstructed atmospheric multiple scattering model and an unsupervised dehazing network for dehazing in water condition, which solved the problem of the effect of multiple scattering on images and the difficulty of collecting ideal data. Specifically, the reconstructed atmospheric multiple scattering model simply views the convolution of the blur kernel and the haze-free image as Hadamard product. The unsupervised dehazing network uses four branches (J-Net, T-Net, K-Net, and K-estimation module) and decomposes the atmospheric multiple scattering model into four constituent layers, i.e., scene radiation layer, transmission map layer, blur kernel layer, and atmospheric light layer, which makes the atmospheric multiple scattering model be embedded in the unsupervised network well. It should be noted that there are three unsupervised branches and one prior-based branch in the unsupervised dehazing network. The three unsupervised branches and one prior-based branch are used to estimate the scene radiation layer, the transport map layer, the blur kernel layer and the atmospheric light layer respectively. In addition, we have constructed unsupervised losses based on prior knowledge, including color attenuation energy loss and dark channel loss, which are used to constrain sub-networks. Finally, the unsupervised dehazing network reconstructs the four layers of output into the hazy image in a self-supervised manner, and minimizes the reconstruction loss of the reconstructed hazy image and the input hazy image. In this process, the output scene radiation layer is the haze-free image.

To sum up, the main work of the proposed method is in three aspects:

For the effect of multiple scattering on the image in the water condition, we have proposed to dehaze through atmospheric multiple scattering model in water condition. Specifically, the convolution of the blur kernel and the clean image is simply viewed as Hadamard product.
For the problem that it is difficult to collect datasets in the water condition, we have proposed to dehaze through an unsupervised dehazing network for water conditions. Specifically, the unsupervised dehazing network consists of three unsupervised branches and one prior-based branch, which are used to estimate the scene radiation layer, the transmission map layer, the blur kernel layer and the atmospheric light layer. The whole network synthesizes hazy images in a self-supervised manner, and the final output scene radiation layer is a haze-free image. The unsupervised dehazing network can only focus on the features of the real-world image itself and avoid intensive operations on the dataset.
In order to constrain the output of the branch of unsupervised network, we have proposed an unsupervised loss function. Specifically, the colour attenuation energy loss function and the dark channel loss are constructed based on the colour attenuation prior [12] and the dark channel prior respectively, which are used to constrain the J-Net and the T-Net.

The structure of this paper can be divided into the following four sections: The first section explains the problems about dehazing in water conditions, the advantages and disadvantages of existing methods of image dehazing and our contribution. In the second section, we provide an overview of the relevant work, mainly including the atmospheric scattering physical model, prior-based method, learning-based method and unsupervised image enhancement method. In the third section, we analyze and explain the proposed method from the reconstructed atmospheric multiple scattering model and the unsupervised dehazing network. It should be pointed out that in the unsupervised dehazing network, the realization methods of the four branches (J-Net, T-Net, K-Net and A-estimation module) are introduced respectively. In the fourth section, we demonstrate the advantages of the proposed method through qualitative and quantitative experiments.

Related work

In this part, firstly, the physical model of atmospheric scattering was introduced, which includes the atmospheric single scattering model and the atmospheric multiple scattering model. Secondly, the prior-based dehazing methods and the learning-based dehazing methods were introduced respectively. Finally, the application of unsupervised neural network in image enhancement was introduced.

Atmospheric scattering model

Atmospheric scattering physical models [10] can be divided into atmospheric single scattering model [7] and atmospheric multiple scattering model [8]. Among them, Wang et al. [8] and He et al. [9] carried out dehazing work through the atmospheric multiple scattering model and achieved better dehazing effects than using the atmospheric single scattering model. According to the atmospheric scattering model proposed by Narasimhan and Nayar [7, 10], the atmospheric single scattering model is a degenerate form of the atmospheric multiple scattering model, and the atmospheric multiple scattering model takes into account the interference caused by multiple scatterings to the image, while the atmospheric single scattering model only considers the case of single scattering.

Atmospheric single scattering model.

Fig 1 shows the process of atmospheric single scattering, and the atmospheric single scattering model can be expressed as: (1)

Download:

Fig 1. The imaging process using the atmospheric single scattering model, which ignores the multiple scattering of light in the atmosphere, but only considers the single scattering of light in the atmosphere.

https://doi.org/10.1371/journal.pone.0253214.g001

In this equation, the hazy image is represented by I(x); The haze-free image is represented by J(x); A represents the global atmospheric light, and t(x) is the transmission map, which can be expressed in the following ways: (2) here β is the scattering coefficient; d(x) is the distance between the target and the imaging device.

Atmospheric multiple scattering model.

Fig 2 shows the process of atmospheric multiple scattering [8], and the atmospheric multiple scattering model can be expressed as: (3) Here h_A is the convolution matrix, and it is the presence of the convolution matrix that causes the blurring of the image. Many researchers solve the convolution matrix through filtering or Monte Carlo methods [24–26], but these methods are too complicated. For simplicity of representation, we call the convolution matrix as the blur kernel and denote it by k in the following. Here * is the convolution operation. Under the influence of multiple scattering, different scene points produce aliasing on the imaging plane and cause the result of imaging to be blurred.

Download:

Fig 2. The imaging process using the atmospheric multiple scattering model.

Different from the atmospheric single scattering model, the atmospheric multiple scattering model takes into account the multiple scattering of light in the atmosphere and can describe the interaction between the reflected light from different sites.

https://doi.org/10.1371/journal.pone.0253214.g002

The prior-based method

The prior-based dehazing method processes hazy images through the observation and statistics of hazy images and manually extracts the prior. Many prior-based methods have been proposed based on different observations and assumptions [1, 3, 4, 11, 12]. For example, He et al. [11] proposed a statistics method based on the prior knowledge of dark channel for image dehazing. This method thinks that there are some pixels in the local area of haze-free image, and at least one of the colour channels’ value is very small or even close to zero. Based on this observation, it performs dehazing processing on the input hazy image. Although the dark channel prior method can effectively solve the problem of image dehazing in outdoor conditions, the overall result of dehazing is dark and the sky area is prone to distortion. Zhu et al. [12] proposed a statistical method based on the prior knowledge of colour attenuation for the degradation of dehazing result. This method think that the depth of a haze-free image is positively correlated with the difference between its brightness and saturation. Based on this observation, this method models the depth of the scene and successfully restores haze-free image. In addition, non-local colour prior (NCP) [3] and haze-line prior (HLP) [4] have also been proposed based on observations and assumptions. Although these prior-based methods have achieved good results in many scenes, the priors through manually extracted may be inconsistent with the information of environmental, and the dehazing effect cannot always achieve the desired results.

The learning-based method

Deep learning methods are currently used to solve many problems [27–30]. Unlike the prior-based method, the learning-based method adopts a data-driven approach and try to directly estimate the intermediate transmission map, the atmospheric light map and the final dehazing result. In order to avoid relying too much on prior information, many researchers completed the image dehazing task through convolutional neural network instead of manually extracting the prior. Therefore, some image dehazing methods through neural networks have been proposed [2, 13–20]. For example, Cai et al. [14] proposed an end-to-end image dehazing method for the problem of image dehazing that relies too much on prior knowledge. They automatically extract the features of haze in hazy images and successfully estimated the transmission map through neural network, and finally realized image dehazing. Although the convolutional neural network can automatically extract the characteristics of haze in the hazy image and successfully estimate the transmission map, the recovered dehazing results are prone to network artifacts because of block training during the training process. Aiming at the problem of estimation of atmospheric light, Zhang et al. [16] proposed to estimate the transmission map and atmospheric light simultaneously. This method can estimate the transmission map and atmospheric light at the same time by integrating the atmospheric scattering model into the neural network model through convolutional neural network and successfully recovered the haze-free image. Although the transmission map and atmospheric light can be estimated at the same time through the convolutional neural network, the recovered dehazing result still has a certain amount of haze, and the effect is general under dense haze. It should be pointed out that although learning-based image dehazing methods are widely used, their results of dehazing are sub-optimal because of using atmospheric single scattering model and data-driven model.

Unsupervised image enhancement method

Since the above shortcomings of learning-based image enhancement methods, unsupervised image enhancement methods have been proposed [5, 21–23]. For instance, in response to the problems with data-driven learning methods, Ulyanov et al. [22] proposed a deep image prior method (DIP). This method can restore the input degraded picture to an undegraded picture by using an untrained neural network and an early stopping strategy. Although this method can achieve image enhancement, the parameters that can be estimated by the neural network are limited and it is difficult to determine the early stopping time, which will cause a direct impact on the final result. In response to the problem of prior parameter estimation of image depth, Irani et al. [23] proposed a new method of unsupervised image enhancement. They solved complex image enhancement problems by coupling multiple DIPs and using the idea of layer separation, such as image segmentation, watermark removal, etc. Although this method can solve complex image enhancement problems through multiple DIPs, it still uses the early stopping strategy when fits the network model, which will not be conducive to the estimation of the optimal result. Li et al. [5] proposed a new unsupervised image dehazing method for the problem with the early stopping strategy. They inputted the hazy image into the unsupervised network and constrained unsupervised network by minimizing the loss of input hazy image and output synthetized hazy image. Although this method can achieve unsupervised dehazing, the final output is still sub-optimal because of using the atmospheric single scattering model. This method is hereafter referred to as YOLY [5].

Although the proposed method and the YOLY both use unsupervised method for image dehazing, they are quite different. Firstly, in principle, YOLY is not a dehazing method specially designed for water conditions. The proposed method solves the dehazing problem in water conditions by using atmospheric multiple scattering model, while YOLY uses atmospheric single scattering model. But in the water condition, the dehazing result by using the atmospheric single scattering model is suboptimal. Secondly, according to the characteristics of hazy waters, the prior knowledge is fused with the neural network, while YOLY only uses the neural network to estimate the information of each layer. In addition, the structure of network and loss function are also different. In terms of the structure of network, the proposed method designed four branches to estimate the information of different layers according to the reconstructed atmospheric multiple scattering model, while YOLY only contains three branches. In terms of loss function, unsupervised loss function was proposed for clean image estimation network and transmission image estimation network in the proposed method, while YOLY proposed unsupervised loss function for clean image estimation network and atmospheric light estimation network.

Proposed method

In this section, the unsupervised dehazing network for water conditions by using the multi-scattering model will be illustrated. Specifically, the reconstructed atmospheric multiple scattering model and unsupervised dehazing network will be introduced.

The reconstructed atmospheric multiple scattering model

In water conditions, many methods by using atmospheric single scattering model are prone to halo and blur because of multiple scattering in the dehazing results. According to the description in literature [8], because the reflected light enters the imaging device at different angles, it will form a dispersion spot caused by the multiple scattering and the clear image will become hazy. As shown in Fig 3, the image becomes blur caused by dispersion spot. Therefore, according to Eq (3) of atmospheric multiple scattering model, it can be deduced as follows: (4)

Download:

Fig 3. The dispersion spot generated by multiple scattering.

Due to the influence of multiple scattering, the original clear image will be blurred, and the dispersion spot will be formed in local area.

https://doi.org/10.1371/journal.pone.0253214.g003

Therefore, it is a challenge to obtain blur kernel and make deconvolution operation. Many super-resolution methods have carried out targeted research on this problem, enlightenments can be obtained from some researches. For example, Gu et al. [31] proposed an image super-resolution method for the calculation of blur kernel and deconvolution. This method simply viewed the convolution of the feature map of the low-resolution image and the blur kernel as Hadamard product in the neural network, and finally restored the high-resolution image. In hazy images, blurring is widespread due to the influence of multiple scattering, and the halo and blur of the image are caused by the convolution of the blur kernel and the clean image. Therefore, according to Eq (4), the convolution of blur kernel and haze-free image can be simply viewed as a per pixel product. Therefore, the reconstructed atmospheric multiple scattering model can be expressed in the following way: (5)

Unsupervised dehazing network

In this section, a new unsupervised dehazing network will be introduced, the network structure is shown in Fig 4. The following sections will introduce the four branches of the unsupervised dehazing network and explain the constructed loss functions.

Download:

Fig 4. The proposed unsupervised dehazing network.

In short, the unsupervised dehazing network consists of three unsupervised branches and a prior-based branch. The input hazy image is decomposed into four different layers by four branches, and then the four layers will be synthesized into hazy image according to the reconstructed atmospheric multiple scattering model.

https://doi.org/10.1371/journal.pone.0253214.g004

J-Net.

In the J-Net branch depicted in Fig 5, the U-shaped network [32] can increase the depth of the network and improve its learning ability. Specifically, J-Net consists of the convolution layer, the down-sampling layer, the up-sampling layer, the Instance Normalization layer [33] and the LeakyReLU activation function. In order to constrain the J-Net branch and generate haze-free image, the colour attenuation prior [12] is used to construct the unsupervised loss function, which is called the colour attenuation energy loss: (6)

Download:

Fig 5. Clean image estimation branch, color attenuation energy loss is used for the constraint of network, the output of the network is a clean image with three-channel.

https://doi.org/10.1371/journal.pone.0253214.g005

In this equation, the brightness and saturation of the haze-free image are represented by V(f_J(X)) and S(f_J(X)) respectively. According to the colour attenuation prior [12], the depth of the image is positively correlated with the difference between brightness and saturation in the haze-free image. Therefore, as the output of J-Net, the difference between the brightness value and saturation of f_J(X) should be as close as possible. So that, J-Net can be constrained by L2 regularization of the difference between image’s brightness and saturation.

T-Net.

In the T-Net branch depicted in Fig 6, the structure of T-Net is similar to the structure of J-Net. In order to constrain the generation of the transmission map of T-Net branch, the dark channel loss is constructed according to dark channel prior [11], and the total variation loss [34] is used to preserve the structure and details.

Download:

Fig 6. Transmission map estimation branch, dark channel loss and total variation loss are used to constrain the network whose output is a single channel transmission map.

https://doi.org/10.1371/journal.pone.0253214.g006

The dark channel loss has been incorporated the dark channel prior method. Specifically, first, the method of solving the transmission map through dark channel prior is used to solve the rough transmission map, and then do L2 regularization between the transmission map of the output of T-Net and the rough transmission map. Obtaining the dark channel of the haze-free image is the first condition. The dark channel can be expressed in the following way: (7)

According to the dark channel prior (8) in the Eq (7), x and y are the pixel coordinates, J^c represents the c-th color channel of the haze-free image, Ω(x) is the local area centered on x, J^dark represents the dark channel of haze-free image. In addition, atmospheric light also needs to be required according to the dark channel. In the dark channel of the hazy image, the pixel with the largest pixel value in the top 0.1% of the image is selected to correspond to the hazy image, and the pixel with the highest pixel value is selected as atmospheric light. Then, according to the dark channel prior, after two minimum filtering, the equation can be obtained, equation “(see (9))”, (9) where is the rough transmission map, and A^c is the atmospheric light solved by the dark channel prior. According to Eqs (7) and (8), the following equation can be obtained: (10)

By substituting Eq (10) into Eq (9) for sorting, the rough transmission map can be expressed by the following equation: (11) where ω is the correction factor, and the default is taken as ω = 0.95. According to the above process, the rough transmission map can be estimated by the method of dark channel prior [11], and then the rough transmission map and the transmission map predicted by T-Net can be L2-regularized to constrain the T-Net branch, which is called the dark channel loss. The dark channel loss function can be expressed in the following way: (12) the f_T(X) represents transmission map predicted by T-Net, represents the rough transmission map estimated by dark channel prior [11]. In addition, in the total variation loss [34], L1 regularization is aim to constrain the gradient, which can preserve structure and details: (13) where ∇_h and ∇_v represent the horizontal and vertical differential operation matrix respectively.

K-Net.

In the K-Net branch, since the clean image, the transmission map and the blur kernel are subject to the input image, the similar network structures are used for J-Net, T-Net and K-Net. But the difference in K-Net is that it does not employ an explicit loss, but only guides optimization in a self-supervised manner.

A-estimation module.

In the water condition, dense haze is a common phenomenon, and the changes of atmospheric light layer are relatively gentle. Therefore, the method of solving the atmospheric light layer through dark channel prior is used to solve the atmospheric light layer. That is, in the dark channel map, the pixel with the largest pixel value in the top 0.1% is selected to correspond to the hazy image, and the pixel with the highest pixel value is selected as atmospheric light layer.

After the four layers of output are obtained, the network needs to reconstruct the above four-layer output into a hazy image in a self-supervised manner and minimize the reconstruction loss function between the reconstructed hazy image and the input hazy image. The reconstruction loss function is composed of mean square error and structural similarity loss [35]. Therefore, the detailed information of the image and the structure and texture information of the image can be considered at the same time. The structural similarity loss can be expressed in the following way: (14) where X and Y represent input hazy image and output hazy image respectively, μ_X and μ_Y represent the mean values of image X and Y respectively, σ_X and σ_Y represent the standard deviations of X and Y respectively, σ_XY represents the covariance of image X and Y, C₁ and C₂ are constants. The mean square error can be expressed as follows: (15)

Therefore, reconstruction loss can be expressed as follows: (16) here α and β are the positive weights of each loss function.

In summary, the total loss of this model can be expressed as follows: (17) here γ, δ and ε are the positive weights of each loss function.

Experiments

In order to verify the effectiveness of the proposed method, comparative experiments have been carried out on the synthetic hazy image and the real hazy image, and the proposed method has been compared with the related dehazing method. The results show that the proposed method performs well on both synthetic and real images, especially in real water conditions, and the dehazing effect of the proposed method is superior to other dehazing methods. In the experiment, ten synthetic hazy images in water condition selected from the RESIDE data set were used for quantitative analysis, and ten real-world hazy images in water condition collected from the Internet were used for qualitative analysis. The synthetic image and the real image datasets can be found at: https://doi.org/10.17605/OSF.IO/Y46W5. The network was trained on a PC with Nvidia GeForce RTX 2060 GPU.

Quantitative results on synthetic images

In the comparative experiment of synthetic images, ten synthetic hazy images in water condition selected from the RESIDE dataset were used for qualitative analysis, and the proposed method has been compared with five methods of dehazing. The five dehazing methods included priori-based dehazing methods and learning-based dehazing methods. The five dehazing methods are as follows: DCP [11], CAP [12], DehazeNet [14], AOD-Net [15], and GCAN [2]. To evaluate the comparison results from an objective perspective, PSNR [36] and SSIM [35] were set as objective evaluation metrics [37].

According to Table 1, the proposed method is superior to the five dehazing methods in metric of PSNR and SSIM. Specifically, compared with the best prior-based method, the proposed method is 0.7 and 0.02 higher on PSNR and SSIM respectively. And compared with the best learning-based method, the proposed method is 1.19 and 0.01 higher on PSNR and SSIM respectively. This shows that the dehazing results recovered by the proposed method have a high degree of detail reduction and a complete structure. In addition, the best visual effect is shown as Fig 7, especially in the borders and detail areas of the images. For example, DCP [11] cannot restore the colour and texture information of the sky area well. For AOD-NET [15], it can be seen that there is still some haze in the image of dehazing results. However, the proposed method not only dehazes to the greatest extent, but also retains the colour and texture information. Therefore, the proposed method has excellent quantitatively and qualitatively results on the synthetic hazy image in water condition.

Download:

Fig 7. Qualitative comparison of different methods on synthetic images.

From the left to the right column (i.e., Fig 7(a)–7(g)), the input hazy image, DCP [1], CAP [2], DehazNet [4], AOD-Net [5], GCAN [10], and our result).

https://doi.org/10.1371/journal.pone.0253214.g007

Download:

Table 1. The average results of SSIM and PSNR on the synthetic images.

https://doi.org/10.1371/journal.pone.0253214.t001

Qualitative visual results on real-world images

In order to show that the proposed method still has a good dehazing effect in the real-world scene, five dehazing methods also were used in comparison experiment with the real image. And the five dehazing methods are as follows: DCP [11], CAP [12], Dehazenet [14], AOD-Net [15], and GCAN [2].

According to Fig 8, it can be seen that the DCP [11] results in a dark haze and color distortion in the sky area and at the edges of the image, which makes the visual effect of the dehazing result poor. For CAP [12], AOD-Net [15] and DehazeNet [14], their dehazing results are not only still unclear, but also blurred, and the dehazing effect is not obvious in the region with far depth of field. For GCAN [2], its dehazing result is brighter in visual effect, which compensates the information in the depth of field of the dehazing result. But its dehazing result has serious colour distortion and a layer of halo at the edge of the image. For the proposed method, it not only restores the colour of original image well, but also has a more thorough dehazing result. On the whole, the colour of the dehazing result is more realistic, without overexposure and distortion. In terms of details, the result of dehazing can reflect the details and edge information of the image well, without blurring and halo. Therefore, the proposed unsupervised dehazing network has a good dehazing effect on synthetic hazy image in water and real hazy image in water, and has excellent results in both quantitative and qualitative aspects.

Download:

Fig 8. Quantitative comparison of different methods on real-world images.

From the left to the right column (i.e., Fig 8(a)–8(g)), the input hazy image, DCP [1], CAP [2], DehazNet [4], AOD-Net [5], GCAN [10], and our result).

https://doi.org/10.1371/journal.pone.0253214.g008

Conclusion

In this paper, an unsupervised dehazing network for water condition by using multiple scattering model was proposed. Compared with previous dehazing methods, the proposed network has three advantages: 1) Atmospheric multiple scattering model is used to dehaze in water conditions, which can effectively avoid the influence of multiple scattering on the image. 2) The unsupervised neural network is used for image dehazing. It divides the hazy image into scene radiation layer, transmission layer, blur kernel layer and atmospheric light layer, and does not use any information other than the content of input image. 3) In the unsupervised dehazing network, prior knowledge is fused into the neural network. Specifically, an unsupervised loss function incorporating prior knowledge is constructed in the network. And according to the characteristics of the water condition, the atmospheric light layer can be obtained by using prior knowledge. Compared with the learning-based methods, the proposed method does not require intensive operations on the data set. In addition, extensive experiments were conducted on synthetic images and real images, and the proposed method was compared with five state-of-the-art dehazing methods. Through quantitative and qualitative analysis, it is concluded that the proposed method has better dehazing effect than previous dehazing methods. In subsequent work, we will attempt to combine atmospheric multiple scattering model with semi-supervised learning to process real-world hazy images while ensuring real-time performance.

Supporting information

S1 File. Hazy images in water area.

https://doi.org/10.1371/journal.pone.0253214.s001

(ZIP)

References

1. Xie B, Guo F, Cai Z. Improved Single Image Dehazing Using Dark Channel Prior and Multi-scale Retinex. 2010 International Conference on Intelligent System Design and Engineering Application. 2010: 848–851.
2. Chen D, He M, Fan Q, Liao J, Zhang L, Hou D, et al. Gated context aggregation network for image dehazing and deraining. 2019 IEEE Winter Conf Appl Comput Vis. 2019: 1375–1383.
3. Berman D, Avidan S, editors. Non-local image dehazing. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1674–1682.
4. Berman D, Treibitz T, Avidan S. Air-light estimation using haze-lines. 2017 IEEE International Conference on Computational Photography (ICCP). 2017: 1–9.
5. Li B, Gou Y, Gu S, Liu JZ, Zhou JT, Peng X. You only look yourself: Unsupervised and untrained single image dehazing neural network. Int J Comput Vis. 2021;129(5): 1754–1767.
- View Article
- Google Scholar
6. Li B, Ren W, Fu D, Tao D, Feng D, Zeng W, et al. Benchmarking single-image dehazing and beyond. IEEE Trans Image Process. 2018;28(1): 492–505. pmid:30176593.
- View Article
- PubMed/NCBI
- Google Scholar
7. Nayar SK, Narasimhan SG. Vision in bad weather. Proc IEEE Int Conf Comput Vis. 1999;2: 820–827.
- View Article
- Google Scholar
8. Wang R, Li R, Sun H. Haze removal based on multiple scattering model with superpixel algorithm. Signal Processing. 2016;127: 24–36.
- View Article
- Google Scholar
9. He R, Wang Z, Fan Y, Feng DD. Multiple scattering model based single image dehazing. 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA). 2013: 733–737.
10. McCartney EJ. Optics of the atmosphere: scattering by molecules and particles. New York. 1976.
11. He K, Sun J, Tang X. Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell. 2011;33(12): 2341–2353. pmid:20820075.
- View Article
- PubMed/NCBI
- Google Scholar
12. Zhu Q, Mai J, Shao L. A fast single image haze removal algorithm using color attenuation prior. IEEE Trans Image Process. 2015;24(11): 3522–3533. pmid:26099141.
- View Article
- PubMed/NCBI
- Google Scholar
13. Ren W, Liu S, Zhang H, Pan J, Cao X, Yang M-H. Single image dehazing via multi-scale convolutional neural networks. Comput Vis ECCV. 2016: 154–169.
- View Article
- Google Scholar
14. Cai B, Xu X, Jia K, Qing C, Tao D. Dehazenet: An end-to-end system for single image haze removal. IEEE Trans Image Process. 2016;25(11): 5187–5198. pmid:28873058.
- View Article
- PubMed/NCBI
- Google Scholar
15. Li B, Peng X, Wang Z, Xu J, Feng D. Aod-net: All-in-one dehazing network. Proc IEEE Int Conf Comput Vis. 2017: 4770–4778.
16. Zhang H, Patel VM. Densely connected pyramid dehazing network. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3194–3203.
17. Ren W, Ma L, Zhang J, Pan J, Cao X, Liu W, et al. Gated fusion network for single image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 3253–3261.
18. Li R, Pan J, Li Z, Tang J. Single image dehazing via conditional generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8202–8211.
19. Engin D, Genç A, Kemal Ekenel H. Cycle-dehaze: Enhanced cyclegan for single image dehazing. Conf Comput Vis Pattern Recognit Workshops. 2018: 825–833.
20. Li L, Dong Y, Ren W, Pan J, Gao C, Sang N, et al. Semi-supervised image dehazing. IEEE Trans Image Process. 2019;29: 2766–2779. pmid:31751272.
- View Article
- PubMed/NCBI
- Google Scholar
21. Sidorov O, Yngve Hardeberg J. Deep hyperspectral prior: Single-image denoising, inpainting, super-resolution. IEEE Int Conf Comput Vis Workshops. 2019: 0–0.
22. Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9446–9454.
23. Gandelsman Y, Shocher A, Irani M. "Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 11026–11035.
24. Metari S, Deschenes F, editors. A new convolution kernel for atmospheric point spread function applied to computer vision. 2007 IEEE 11th international conference on computer vision. 2007: 1–8.
25. Reinersman PN, Carder KL. Monte Carlo simulation of the atmospheric point-spread function with an application to correction for the adjacency effect. Applied optics. 1995;34(21): 4453–4471. pmid:21052279.
- View Article
- PubMed/NCBI
- Google Scholar
26. Semenov AA, Moshkov AV, Pozhidayev VN, Barducci A, Marcoionni P, Pippi I. Estimation of normalized atmospheric point spread function and restoration of remotely sensed images. IEEE Trans Geosci Remote Sens. 2011;49(7): 2623–2634.
- View Article
- Google Scholar
27. Gadekallu TR, Alazab M, Kaluri R, Maddikunta PKR, Bhattacharya S, Lakshmanna K, et al. Hand gesture classification using a novel CNN-crow search algorithm. Complex & Intelligent Systems. 2021: 1–14.
- View Article
- Google Scholar
28. Bhattacharya S, Maddikunta PKR, Pham Q-V, Gadekallu TR, Chowdhary CL, Alazab M, et al. Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey. Sustain Cities Soc. 2021;65: 102589. pmid:33169099.
- View Article
- PubMed/NCBI
- Google Scholar
29. Gadekallu TR, Rajput DS, Reddy MPK, Lakshmanna K, Bhattacharya S, Singh S, et al. A novel PCA–whale optimization-based deep neural network model for classification of tomato plant diseases using GPU. J Real Time Image Process. 2020.
- View Article
- Google Scholar
30. Rajput DS, Basha SM, Xin Q, Gadekallu TR, Kaluri R, Lakshmanna K, et al. Providing diagnosis on diabetes using cloud computing environment to the people living in rural areas of India. J Ambient Intell Humaniz Comput. 2021: 1–12.
- View Article
- Google Scholar
31. Gu J, Lu H, Zuo W, Dong C. Blind super-resolution with iterative kernel correction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1604–1613.
32. R. Olaf, F. Philipp., B. Thomas, “U-net: Convolutional networksfor biomedical image segmentation”, In Int. Conf. Medi-cal Image. Computing and Computer-Assisted Intervention. (MICCAI), 2015, pp. 234–241.
33. Xu Z, Yang X, Li X, Sun X. The effectiveness of instance normalization: a strong baseline for single image dehazing. arXiv preprint arXiv:180503305. 2018.
34. Li Y, Santosa F. A computational algorithm for minimizing total variation in image restoration. IEEE Trans Image Process. 1996;5(6): 987–95. pmid:18285186.
- View Article
- PubMed/NCBI
- Google Scholar
35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13(4): 600–612. pmid:15376593.
- View Article
- PubMed/NCBI
- Google Scholar
36. Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment. Electronics letters. 2008;44(13): 800–801.
- View Article
- Google Scholar
37. Wang Z., Li Q. Information Content Weighting for Perceptual Image Quality Assessment. IEEE Trans Image Process. 2011;20(5):1185–1198. pmid:21078577.
- View Article
- PubMed/NCBI
- Google Scholar

[ref1] 1. Xie B, Guo F, Cai Z. Improved Single Image Dehazing Using Dark Channel Prior and Multi-scale Retinex. 2010 International Conference on Intelligent System Design and Engineering Application. 2010: 848–851.

[ref2] 2. Chen D, He M, Fan Q, Liao J, Zhang L, Hou D, et al. Gated context aggregation network for image dehazing and deraining. 2019 IEEE Winter Conf Appl Comput Vis. 2019: 1375–1383.

[ref3] 3. Berman D, Avidan S, editors. Non-local image dehazing. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1674–1682.

[ref4] 4. Berman D, Treibitz T, Avidan S. Air-light estimation using haze-lines. 2017 IEEE International Conference on Computational Photography (ICCP). 2017: 1–9.

[ref5] 5. Li B, Gou Y, Gu S, Liu JZ, Zhou JT, Peng X. You only look yourself: Unsupervised and untrained single image dehazing neural network. Int J Comput Vis. 2021;129(5): 1754–1767.
View Article
Google Scholar

[6] View Article

[7] Google Scholar

[ref6] 6. Li B, Ren W, Fu D, Tao D, Feng D, Zeng W, et al. Benchmarking single-image dehazing and beyond. IEEE Trans Image Process. 2018;28(1): 492–505. pmid:30176593.
View Article
PubMed/NCBI
Google Scholar

[9] View Article

[10] PubMed/NCBI

[11] Google Scholar

[ref7] 7. Nayar SK, Narasimhan SG. Vision in bad weather. Proc IEEE Int Conf Comput Vis. 1999;2: 820–827.
View Article
Google Scholar

[13] View Article

[14] Google Scholar

[ref8] 8. Wang R, Li R, Sun H. Haze removal based on multiple scattering model with superpixel algorithm. Signal Processing. 2016;127: 24–36.
View Article
Google Scholar

[16] View Article

[17] Google Scholar

[ref9] 9. He R, Wang Z, Fan Y, Feng DD. Multiple scattering model based single image dehazing. 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA). 2013: 733–737.

[ref10] 10. McCartney EJ. Optics of the atmosphere: scattering by molecules and particles. New York. 1976.

[ref11] 11. He K, Sun J, Tang X. Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell. 2011;33(12): 2341–2353. pmid:20820075.
View Article
PubMed/NCBI
Google Scholar

[21] View Article

[22] PubMed/NCBI

[23] Google Scholar

[ref12] 12. Zhu Q, Mai J, Shao L. A fast single image haze removal algorithm using color attenuation prior. IEEE Trans Image Process. 2015;24(11): 3522–3533. pmid:26099141.
View Article
PubMed/NCBI
Google Scholar

[25] View Article

[26] PubMed/NCBI

[27] Google Scholar

[ref13] 13. Ren W, Liu S, Zhang H, Pan J, Cao X, Yang M-H. Single image dehazing via multi-scale convolutional neural networks. Comput Vis ECCV. 2016: 154–169.
View Article
Google Scholar

[29] View Article

[30] Google Scholar

[ref14] 14. Cai B, Xu X, Jia K, Qing C, Tao D. Dehazenet: An end-to-end system for single image haze removal. IEEE Trans Image Process. 2016;25(11): 5187–5198. pmid:28873058.
View Article
PubMed/NCBI
Google Scholar

[32] View Article

[33] PubMed/NCBI

[34] Google Scholar

[ref15] 15. Li B, Peng X, Wang Z, Xu J, Feng D. Aod-net: All-in-one dehazing network. Proc IEEE Int Conf Comput Vis. 2017: 4770–4778.

[ref16] 16. Zhang H, Patel VM. Densely connected pyramid dehazing network. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 3194–3203.

[ref17] 17. Ren W, Ma L, Zhang J, Pan J, Cao X, Liu W, et al. Gated fusion network for single image dehazing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 3253–3261.

[ref18] 18. Li R, Pan J, Li Z, Tang J. Single image dehazing via conditional generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 8202–8211.

[ref19] 19. Engin D, Genç A, Kemal Ekenel H. Cycle-dehaze: Enhanced cyclegan for single image dehazing. Conf Comput Vis Pattern Recognit Workshops. 2018: 825–833.

[ref20] 20. Li L, Dong Y, Ren W, Pan J, Gao C, Sang N, et al. Semi-supervised image dehazing. IEEE Trans Image Process. 2019;29: 2766–2779. pmid:31751272.
View Article
PubMed/NCBI
Google Scholar

[41] View Article

[42] PubMed/NCBI

[43] Google Scholar

[ref21] 21. Sidorov O, Yngve Hardeberg J. Deep hyperspectral prior: Single-image denoising, inpainting, super-resolution. IEEE Int Conf Comput Vis Workshops. 2019: 0–0.

[ref22] 22. Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9446–9454.

[ref23] 23. Gandelsman Y, Shocher A, Irani M. "Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 11026–11035.

[ref24] 24. Metari S, Deschenes F, editors. A new convolution kernel for atmospheric point spread function applied to computer vision. 2007 IEEE 11th international conference on computer vision. 2007: 1–8.

[ref25] 25. Reinersman PN, Carder KL. Monte Carlo simulation of the atmospheric point-spread function with an application to correction for the adjacency effect. Applied optics. 1995;34(21): 4453–4471. pmid:21052279.
View Article
PubMed/NCBI
Google Scholar

[49] View Article

[50] PubMed/NCBI

[51] Google Scholar

[ref26] 26. Semenov AA, Moshkov AV, Pozhidayev VN, Barducci A, Marcoionni P, Pippi I. Estimation of normalized atmospheric point spread function and restoration of remotely sensed images. IEEE Trans Geosci Remote Sens. 2011;49(7): 2623–2634.
View Article
Google Scholar

[53] View Article

[54] Google Scholar

[ref27] 27. Gadekallu TR, Alazab M, Kaluri R, Maddikunta PKR, Bhattacharya S, Lakshmanna K, et al. Hand gesture classification using a novel CNN-crow search algorithm. Complex & Intelligent Systems. 2021: 1–14.
View Article
Google Scholar

[56] View Article

[57] Google Scholar

[ref28] 28. Bhattacharya S, Maddikunta PKR, Pham Q-V, Gadekallu TR, Chowdhary CL, Alazab M, et al. Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey. Sustain Cities Soc. 2021;65: 102589. pmid:33169099.
View Article
PubMed/NCBI
Google Scholar

[59] View Article

[60] PubMed/NCBI

[61] Google Scholar

[ref29] 29. Gadekallu TR, Rajput DS, Reddy MPK, Lakshmanna K, Bhattacharya S, Singh S, et al. A novel PCA–whale optimization-based deep neural network model for classification of tomato plant diseases using GPU. J Real Time Image Process. 2020.
View Article
Google Scholar

[63] View Article

[64] Google Scholar

[ref30] 30. Rajput DS, Basha SM, Xin Q, Gadekallu TR, Kaluri R, Lakshmanna K, et al. Providing diagnosis on diabetes using cloud computing environment to the people living in rural areas of India. J Ambient Intell Humaniz Comput. 2021: 1–12.
View Article
Google Scholar

[66] View Article

[67] Google Scholar

[ref31] 31. Gu J, Lu H, Zuo W, Dong C. Blind super-resolution with iterative kernel correction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1604–1613.

[ref32] 32. R. Olaf, F. Philipp., B. Thomas, “U-net: Convolutional networksfor biomedical image segmentation”, In Int. Conf. Medi-cal Image. Computing and Computer-Assisted Intervention. (MICCAI), 2015, pp. 234–241.

[ref33] 33. Xu Z, Yang X, Li X, Sun X. The effectiveness of instance normalization: a strong baseline for single image dehazing. arXiv preprint arXiv:180503305. 2018.

[ref34] 34. Li Y, Santosa F. A computational algorithm for minimizing total variation in image restoration. IEEE Trans Image Process. 1996;5(6): 987–95. pmid:18285186.
View Article
PubMed/NCBI
Google Scholar

[72] View Article

[73] PubMed/NCBI

[74] Google Scholar

[ref35] 35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13(4): 600–612. pmid:15376593.
View Article
PubMed/NCBI
Google Scholar

[76] View Article

[77] PubMed/NCBI

[78] Google Scholar

[ref36] 36. Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment. Electronics letters. 2008;44(13): 800–801.
View Article
Google Scholar

[80] View Article

[81] Google Scholar

[ref37] 37. Wang Z., Li Q. Information Content Weighting for Perceptual Image Quality Assessment. IEEE Trans Image Process. 2011;20(5):1185–1198. pmid:21078577.
View Article
PubMed/NCBI
Google Scholar

[83] View Article

[84] PubMed/NCBI

[85] Google Scholar

Figures

Abstract

Introduction

Related work

Atmospheric scattering model

Atmospheric single scattering model.

Atmospheric multiple scattering model.

The prior-based method

The learning-based method

Unsupervised image enhancement method

Proposed method

The reconstructed atmospheric multiple scattering model

Unsupervised dehazing network

J-Net.

T-Net.

K-Net.

A-estimation module.

Experiments

Quantitative results on synthetic images

Qualitative visual results on real-world images

Conclusion

Supporting information

S1 File. Hazy images in water area.

References