2.1 Smooth wavelet decomposition of images

First, for the acquired infrared night-vision image, and then implement the smooth wavelet decomposition of layer gives high frequency components (horizontal, vertical and diagonal directions) and one low frequency component.

During the process, the high-pass filter is set to , the low-pass filter is , combined with the stationary wavelet conversion principle [13–14], the wavelet component of the infrared night vision image is decomposed into a low frequency component and a high frequency component , , in the horizontal, vertical and diagonal directions, the process is shown in formula (1):

(1)

In formula (1), represents the image translation coefficient, represents the scale factor.

2.2 Image edges as well as noise markers

After the image is decomposed by wavelet, the magnitude of the edge increases with the increase of scale, while the magnitude of the noise decreases rapidly with the increase of scale, so it is more beneficial to utilize the characteristics of the edge and the noise exhibited between adjacent scales to protect the edge while suppressing the noise. Calculate the correlation value of wavelet coefficients of neighboring scales, if the correlation is large, the pixel at that position is labeled as edge, otherwise the pixel is labeled as noise, the steps are as follows:

Step 1: For the high frequency component , , obtained after smooth wavelet decomposition of the infrared night vision image, for which the correlation values of the expansion coefficients are calculated, and applying normalization to the correlation values, the process is shown in formula (2):

(2)

Step 2: For different pixel points , when , indicating that the pixel point is an edge pixel, for which edge labeling needs to be implemented, and set and to 0.

Step 3: Thereby calculate the unlabeled edge pixel point energy for the layer; During the process, when the calculated pixel point energy (noise variance), then you need to go back to step 1 and restart the calculation, otherwise you need to calculate the non-zero standard deviation of the value in ; If , it means that the pixel point is marked as an edge point.

Consistency verification is performed on the tagged pixels, i.e., if the center pixel in a region is tagged as a noise point (edge point) and most of the other pixel points in its neighborhood are tagged as edge points (noise points), then the center pixel is also tagged as an edge point (noise point). In the paper, a 3 × 3 size window is used for consistency detection. The wavelet component of the largest scale layer after image decomposition can effectively suppress the fine texture and noise, but the edge is not easy to locate, so the wavelet coefficient correlation method is not used in this layer to mark the edge and noise, but the edge points marked in the smaller scale layers are merged to mark the edges of the high-frequency component of the layer, and the unmarked points are treated as noise.

2.3 Neighborhood-based shrinkage of wavelet coefficients

After the pixel labeling is completed, a wavelet coefficient shrinkage algorithm that fully takes into account the nature of the pixel neighborhood is used to reduce the visual interference of the image.

During the wavelet shrinkage process [15–16], the wavelet coefficients of the decomposed infrared night vision image is set to be , thus carrying out wavelet coefficient shrinkage, the process is shown in formula (3).

(3)

In formula (3), represents the image noise variance, represents the set of neighborhood points of the pixel center , represents the size of the image, represents the wavelet coefficient value, represents the wavelet coefficient contraction result, represents the contraction factor, represents the scale factor, represents the scale factor.

When the noise variance is large, the corresponding wavelet coefficient value of the noise may also be large, and the corresponding wavelet coefficient value becomes large after the contraction, thus generating isolated dark and bright spots, but the noise in the sub-large-scale high-frequency component is basically removed, so the following method is proposed to remove the isolated spots:

1. (1). For the wavelet coefficients in each of the three directions at the sub-large scale, calculate the mean of the absolute values of the coefficients after contraction;
2. (2). Set as a labeling matrix with size equal to the size of the high-frequency component image, if a pixel in the sub-massive high-frequency component, the absolute value of the wavelet coefficients is less than , then set to 1;
3. (3). For each pixel point of the minimum-scale high-frequency component, if is equal to 1, set the gray value of that pixel to 0.

Finally, the above method can eliminate the bright spots and dark spots in the smallest scale wavelet components, and complete the noise removal process of infrared night vision images.

3.1 Illumination estimation methods for infrared night vision images

Illumination estimation using bilateral filtering. Bilateral filtering is a very useful filtering technique that can be used in various aspects of image processing and graphics, and is a filtering technique with edge preservation. Its output value is not only related to the blank position of the surrounding pixels, but also related to their luminance difference, which is formally defined as shown in formula (5).

(5)

In formula (5), represents the set of image pixels, represents bilateral filtering output results for pixel point , , represents the Gaussian function, the and represents the spatial domain value of the point as well as the luminance domain value for the pixel space , the represents the normalization factor.

In the specific implementation process, if the bilateral filter is realized only according to the original definition, the computational efficiency is very low. For this reason, the calculation speed of bilateral filtering is improved by using the method based on the layering of gray values, which has a better effect on the calculation speed of filtering in larger space scales.

During the process, a 3D grid is used to represent the 2D gray scale image, the first two dimensions of the grid correspond to the positions of the image pixels, and the third dimension corresponds to the brightness of the image. After the definition, the bilateral filter can be calculated by the following steps:

1. (1). First, the infrared night vision image is regarded as a two-dimensional image, and the vector grid is initialized, so that it satisfies the conditions of formula (6).

(6)

And based on the above conditions, Gaussian filtering is applied to the network and the result is shown in formula (7):

(7)

In formula (7), represents the airspace parameter, represents the luminance field parameter, represents a Gaussian function in three dimensions.

1. (2). In the post-processing grid, set the final result for position is , which completes the bilateral filtering calculation .

The illumination image is obtained after the illumination estimation process on the input image.A part of the pixels at both ends of the histogram of the illumination image is truncated by the histogram interception method and the remaining pixels are compressed to range, which were then corrected using a modified Gamma correction method. When calibrating, the original image pixels are denoted by , the control parameters are described as . Finally, the final illumination image is computed using the linear lazar.

3.2 Reflective image enhancement

After obtaining the illumination image, the reflection image can be obtained by doing the difference operation between the original image and the illumination image in the logarithmic domain. The reflection image contains the detailed information of the image, so it is very important to enhance it, and the Sigmoid function is used to enhance , the specific process is shown in formula (8):

(8)

In formula (8), represents the brightness of the reflected image, represents the control parameter. Where, since the reflected image brightness is in the number field of the image, there will be negative cases from time to time, and the larger the calculation result of , the steeper the luminance mapping curve of the reflection image is, and the more significant the enhancement of the reflection image is.

Combining the above steps, the preliminary enhancement process of infrared night image is obtained as follows:

Step 1: First, based on the infrared night vision image, obtain the image brightness image , and logarithmically, using bilateral filtering as well as gray value layering to obtain the illumination image ;

Step 2: Calculate the difference of the luminance image and illuminated image , obtain the reflection image ;

Step 3: Use the histogram to intercept both ends of the illuminance image and calibrate it to determine the illuminance estimate.

Step 4: Using formula (8) for reflection image enhancement , and add it to the illumination image and get a new image , complete the initial visual enhancement of infrared night vision images.

4.1 Multi-level feature extraction

Due to the overall information obscurity of infrared night vision images [19], the conventional convolutional receptive field is limited by the size of the convolutional kernel, which is unable to capture the long-distance-dependent information and impedes the extraction of neighborhood features. The pixel attention module is improved by combining the advantages of depth-separable convolution and cavity convolution, and the sensory field is effectively enlarged. In this way, a pixel-based perception is used to extract multilevel features from infrared night vision images, as shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Pixel perception module structure.

https://doi.org/10.1371/journal.pone.0330349.g002

According to the analysis of Fig 2, the input infrared night vision image is sent to the sampling module using anti aliasing sampling method. The module generates shallow feature maps through different downsampling multiples and uses skip connection mechanism internally to alleviate the gradient dispersion problem that may be caused by downsampling, ensuring stable transmission of feature information; Subsequently, the shallow feature maps generated by the sampling module enter the subsequent convolution operation process in sequence. Firstly, channel information is integrated through 1 * 1 point convolution at a lower computational cost to provide effective feature representations for subsequent operations. Then, through 5 * 5 depth separable convolution (which splits conventional convolution into deep convolution and point by point convolution, utilizing the sparse connection of wide neighborhood pixel information by large kernel convolution properties to initially expand the receptive field and capture a wider range of local features without significantly increasing parameters), and finally through 5 * 5 hole convolution (which further expands the receptive field without increasing the number of parameters, captures long-distance dependency information, and extracts richer features), point convolution, 5 * 5 depth separable convolution, and 5 * 5 hole convolution are performed. The result is a multiplication operation at the end, Effectively integrating feature information extracted by different convolution methods to obtain the final output of the pixel perception module, this module generates multi-scale feature maps of different levels (Level 1, Level 2, Level 3) through this multi form convolution operation and feature fusion method, enhances pixel information expression, and improves the network’s ability and adaptability to extract features from infrared night vision images.

It can be seen that the pixel perception module contains the sampling module and the pixel attention sub-module, both of which sequentially carry out the multilevel feature map generation and the expansion of the sensory field. Firstly, the input image flow to individual sampling modules to generate shallow features corresponding to downsampling multiples. The sampling method is anti-aliased, and the sampling module utilizes hopping connections within the module to mitigate gradient dispersion.

The feature maps are sequentially subjected to point convolution, 5*5 depth separable convolution and cavity convolution, and the information between channels is integrated by using the low computational cost of linear operation, while the pixel information in a wide range of neighborhoods is sparsely connected by the nature of large kernel convolution, so as to expand the sensory field without increasing the parameters dramatically. Finally, the results of multi-form convolution operation are combined with at the end to obtain the module output . Given the input , the process is shown in formula (9):

(9)

In formula (9), represents sawtooth sampling. represents the sampling multiplier. represents the 1*1 convolution, represents the 3*3 convolution, , represents a 5*5 depth separable convolution as well as a 5*5 null convolution.

4.2 Multi-level infrared night vision image feature reconstruction

Due to the redundancy of useless information and the weakening of the characterization ability of effective information in infrared night vision images, the interaction of feature information within the image is limited, which is not conducive to the reconstruction of image details. In order to filter the redundant information and establish the long-term dependence between local and global features, an efficient multi-head transposed attention module is introduced to model the global information and realize the feature reconstruction. The module consists of a multihead transposed attention layer (MDTA) and a gated feedforward network (GDFN).

The MDTA structure in the module is shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Multi-head transposition attention layer (MDTA) structure.

https://doi.org/10.1371/journal.pone.0330349.g003

Analysis of Fig 3 shows that, given a tensor input normalized by the layers of the normalization is . By aggregating the image context information between the channels through 1*1 convolution, followed by encoding the spatial context information between the channels through 3*3 depth-separated convolution, both of which utilize the complementary advantages of convolution operations to strengthen the local spatial characterization capability, and thus generate queries , index as well as the key value . Finally, applying the dot product operation to reshape the projection of and , and interactions generate a transposed attention map with size , implicitly modeling the global relationship between image cables, implicitly model the global relationship between the image cords. The MDTA module flow is shown in formula (10):

(10)

In formula (10), and represents the input and output feature images, represents the decomposition result of the input of the original image , represents the point product size parameter controlling and .

The structure of GDFN in the module is shown in Fig 4. GDFN is improved according to the conventional feed-forward network, which mainly introduces an additional gating mechanism in the conventional feed-forward network and replaces the conventional convolution with the depth-separable convolution.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Gated Feedforward Network (GDFN) Structure.

https://doi.org/10.1371/journal.pone.0330349.g004

The gating mechanism [20] in Fig 4 is represented as the product of two parallel paths of linear transformation layers, one of which invokes the CELU activation function to realize the control information flow. The similarity with MDTA is that GDFN also uses depth-separable convolutional encoding of neighborhood pixel position information to optimize the linkage of spatial context information, which helps to recover the image structure. Given a tensor input of , the GDFN training process is shown in formula (11):

(11)

In formula (11), represents the GELU function, represents layer standardization. represents the element-by-element cumulative multiplication symbol.

The difference with MDTA is that GDFN achieves the purpose of information interaction and complementarity by controlling the flow of information at each level in the branch, while MDTA utilizes parallel structure and multiple forms of convolution to enrich the contextual information.

4.3 Adaptive cross-level feature fusion

The commonly used feature fusion methods for multi-scale networks are mainly splicing and summing, which are characterized by simple and efficient operation, but the operation limits the feature expression ability of the network, which is not conducive to the improvement of model performance. In order to fully utilize the high quality features extracted from the network, an adaptive feature fusion module (SKFF) is introduced into the network framework of this paper, which utilizes the self-attention mechanism to nonlinearly fuse the features of different levels, and the structure of the module is shown in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Structure of Adaptive Feature Fusion Module (SKFF).

https://doi.org/10.1371/journal.pone.0330349.g005

Analyzing Fig 5 shows that SKFF can be regarded as two stages. The first stage is the feature aggregation stage: Firstly, the tensors are merged sequentially using splicing, reshaping and accumulation operations, and during the process, set ; Second, the features are aggregated from the spatial dimension using global average pooling get the channel information . Finally, point-by-point convolution is used to strengthen the feature correlation and generate the tensor , where the 1/8th of convolution is . The second stage is the weighted fusion stage; The tensor is transmitted in two forward directions. The feature descriptors and . are generated by two forward propagation and 1*1 convolution respectively; Followed by the computation of the attention weights , using the softmax function. Finally and are multiplied with the attention weights, respectively, and summing at the end to get the module output. The specific flow is shown in formula (12).

(12)

The adaptive feature fusion module (SKFF) plays a key role in the network framework of this article. It uses self attention mechanism to nonlinearly fuse different hierarchical features, avoiding the limitations of traditional feature fusion methods such as concatenation and summation, which limit the network’s feature expression ability and are not conducive to improving model performance due to their simple operation. Consider this module as two stages of feature aggregation and weighted fusion. Firstly, features are aggregated and correlated through operations such as concatenation, reshaping, accumulation, global average pooling, and point by point convolution. Then, feature descriptors are generated through 1*1 convolution. The softmax function is used to calculate attention weights and weight the sum to obtain the output. This design conforms to the cutting-edge concept of multi-scale feature fusion and can dynamically weight features at different levels to improve the effectiveness of feature fusion.

4.4 The loss function

The Charbonnier loss function is widely used in the field of image restoration due to its own operational boundaries and convergence properties. The Charbonnier loss function is formulated as shown in formula (13).

(13)

In formula (13). , represents loss function processing of infrared night vision image labels as well as samples, represents a constant.

However, the Charbonnier loss function only calculates the error between pixels and does not take the global information into account, so the problem of excessive smoothing usually occurs. In order to further enhance the realism of the high-frequency details of the image while recovering the image, an edge loss function is attached to the Charbonnier loss function for constraining the high-frequency components between labels and samples, as shown in formula (14).

(14)

In formula (14), , represent the edge features extracted by the Laplacian operator versus , .

The combination of the above loss functions has good efficacy in the image recovery task, but has limited ability to recover implicit information such as image structure. In order to further improve the visual effect of the image, the structural similarity loss function is introduced, and auxiliary for model optimized in the directions of contrast and structure. In summary, the combination of the three is constructed into a joint loss function as shown in formula (15):

(15)

According to the loss function to complete the infrared night vision image feature enhancement output results, to realize the infrared night vision image feature enhancement processing.

5.1 Experimental setup

During the experiment, an open field monitoring point was selected as the test site, and an InGaAs uncooled short-wave infrared camera was used to collect infrared night-vision images at different locations of the site. The collected infrared night vision images were integrated to create a collection of test samples for experimental use. At the same time, Ubuntu 20.04 LTS operating system, Intel Core i7-10700K CPU, Python 3.8 programming language, and OpenCV image processing library were selected to test the image enhancement effect. Based on the test sample set, five images are selected as test samples to test the effectiveness of the image enhancement method. The specific test samples are shown in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Infrared night vision image test sample.

https://doi.org/10.1371/journal.pone.0330349.g006

In Fig 6, there is no issue of inaccurate focusing in samples 1 and 4. Sample 1: Due to the object being close to the camera and in a low light environment in the shooting scene, the contour edges of the object exhibit a certain degree of blur, which is a characteristic of shortwave infrared imaging under complex lighting conditions and not a focusing error; In sample 4, the overall contrast of the image is low and there is some noise interference, making the image appear unclear. However, after analyzing the camera shooting parameters and imaging principles, this performance is also caused by infrared imaging characteristics and environmental factors, rather than inaccurate focusing.

The relevant parameter settings during the experiment are shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Relevant parameters during the experimental process.

https://doi.org/10.1371/journal.pone.0330349.t001

In this experiment, a detailed hyperparameter analysis was conducted on the proposed infrared night vision image enhancement algorithm based on cross level feature fusion. The learning rate of the cross level feature fusion enhanced network is set to 0.001, which can balance the convergence speed and stability well during the training process; Set the batch size to 32, which ensures training efficiency while also allowing the model to fully learn the diversity of data; The number of training rounds is set to 50, and after multiple experimental verifications, the model can achieve good convergence results under this number of rounds. Setting the number of smooth wavelet decomposition layers to 4 can effectively extract feature information from images at different scales. The spatial parameter of bilateral filtering is set to 15, and the brightness parameter is set to 0.1. By adjusting these two parameters, the image edge details are well preserved while denoising. The reasonable setting of these hyperparameters plays a key role in achieving good performance of the algorithm in infrared night vision image enhancement tasks.

In this experiment, in order to ensure the fairness and effectiveness of the comparative testing, the optimal experimental parameters that each comparative method can achieve in the experimental environment were selected through multiple experiments and parameter adjustments for the four comparative methods under the conditions of this experiment, in order to verify the practical feasibility of the proposed method in the process of enhancing infrared night vision images.

5.2 Analysis of test results

We conducted infrared night vision image enhancement in the test samples and tested the image denoising effect of the proposed method. The test results are shown in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Comparison of proposed method before and after denoising.

https://doi.org/10.1371/journal.pone.0330349.g007

From the analysis of Fig 7, it can be seen that the proposed method for enhancing infrared night vision images is more conducive to suppressing noise while protecting edges and labeling pixel edges, as it utilizes the features exhibited by edges and noise between adjacent scales during infrared night vision image denoising. Therefore, this method preserves the detailed information within the image completely during infrared night vision image denoising. This proves that the proposed method is effective in enhancing infrared night vision images.

After completing the detection of image denoising effect, the proposed method, comparison method 1, comparison method 2, comparison method 3 and comparison method 4 were used to continue the infrared night vision image enhancement process, and the peak signal-to-noise ratio and structural similarity were selected as the performance test indexes of image enhancement to test the actual enhancement performance of the image enhancement process of the different methods, the specific peak signal-to-noise ratio, the peak signal-to-noise ratio, the peak signal-to-noise ratio, the peak signal-to-noise ratio, the peak signal-to-noise ratio, the peak signal-to-noise ratio, the peak signal-to-noise ratio, and the acquisition process for the structure similarity ratio and structural similarity is shown in formula (16):

(16)

In formula (16), , represents constant. , represents the contrast variance of the image and image . , represents the mean luminance variance. represents the covariance between and . represents the mean square error between the original image and the processed image. represents the maximum possible pixel value in the image.

The higher the PSNR as well as SSIM values during image enhancement, the higher the structural similarity between the processed image and the original image, i.e., the better the image quality. The specific test results are shown in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Test results of actual infrared night vision image enhancement performance using different methods.

https://doi.org/10.1371/journal.pone.0330349.t002

Analyzing Table 2, it can be seen that the peak signal-to-noise ratio and the structural similarity of the detected infrared night vision image after infrared night vision image enhancement by the proposed method are the best among the five methods. This is mainly due to the fact that the proposed method uses bilateral filtering algorithm to further enhance the illumination of the feature-enhanced infrared night-vision image during the image enhancement, and thus the proposed method is effective in infrared night-vision image enhancement.

Based on the above test results, the proposed method, comparison method 1, comparison method 2, comparison method 3 and comparison method 4 are used to carry out infrared night vision image enhancement, and the actual enhancement effect of different methods is tested, and the test results are shown in Fig 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Actual image enhancement effect test results of different methods.

https://doi.org/10.1371/journal.pone.0330349.g008

Analyzing Fig 8, we can see that, when carrying out infrared image enhancement, comparing method 1 due to the uniform redistribution of low-frequency images, increasing the resource consumption of the algorithm, so the method in the infrared image enhancement, algorithmic performance is poor; comparing method 2 due to the joint image pyramid fusion features, feature parameters of the mismatch between the algorithms, resulting in the use of the algorithm to increase the energy consumption, so the method in the infrared image enhancement, algorithmic performance Comparison method 3 has poor enhancement effect in infrared image enhancement due to the existence of multi-degree fusion when fusing the input high-frequency and low-frequency subgraphs using wavelet image fusion; Comparison method 4 has poor enhancement effect in infrared image enhancement due to the fact that a part of the foreground target image information is segmented into the background image when region segmentation is performed; Because the bilateral filtering algorithm is used to implement further illumination enhancement to the feature-enhanced infrared night vision image during image enhancement, the proposed method has good enhancement effect when infrared night vision image enhancement is carried out.

To comprehensively evaluate the effectiveness of the infrared night vision image enhancement algorithm based on cross level feature fusion proposed in this article, public datasets FLIR and KAIST were added in the ablation experiment, which contain rich infrared night vision images and can more comprehensively evaluate the performance of the proposed method. The setting only uses denoising and preliminary enhancement steps, does not include cross level feature fusion network as the basic method, adds denoising and preliminary enhancement steps to the basic method as denoising+preliminary enhancement, only uses cross level feature fusion network for image enhancement as a cross level feature fusion network, and includes denoising, preliminary enhancement, and cross level feature fusion network as the complete algorithm. Meanwhile, citing time consumption as an evaluation metric. The experimental results are shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Results of ablation experiments on different datasets.

https://doi.org/10.1371/journal.pone.0330349.t003

According to the analysis in Table 3, it can be seen that the complete method has the shortest time consumption on the FLIR and KAIST datasets, which are 50ms and 55ms respectively. This is because in the implementation process of the complete method, the bilateral filtering and other operations were optimized based on grayscale value layering, effectively improving the calculation speed. Especially at large spatial scales, this optimization has a more significant effect on improving the filtering calculation speed, making the overall processing time shorter even if the complete method contains more complex steps. The time consumption of the denoising+preliminary enhancement method is reduced compared to the basic method, indicating a certain optimization synergy between the denoising processing and preliminary enhancement steps in implementation. When used alone, the cross level feature fusion network consumes relatively less time, indicating that the network structure itself has certain advantages in computational efficiency. Overall, the experimental results not only validate the effectiveness of the algorithm proposed in this paper in terms of performance, but also indirectly reflect the positive effect of the grayscale value based hierarchical optimization strategy on improving the processing speed of the algorithm.