Multi-focus image fusion algorithm based on focus detection in spatial and NSCT domain

Hongmei Wang

doi:10.1371/journal.pone.0204225

Abstract

Multi-focus image fusion is an effective approach to obtain the all-in-focus image. Focus detection is the key issue of multi-focus image fusion. Aiming at the shortcoming of spatial domain and transform domain algorithms for multi-focus image fusion, a novel multi-focus image fusion algorithm is proposed by combing focus detection in spatial domain and non-subsampled contourlet transform (NSCT) domain. At first, the focused pixels are detected by the sum-modified-Laplacian algorithm in spatial domain. At the same time, the focus detection method is proposed in NSCT domain, namely by MPCNN and voting fusion methods for high-frequency subbands of NSCT. Then, the morphological operation is utilized to correct the focus detection results in spatial domain and NSCT domain. At last, synthesis of detection results is implemented and the fused image can be obtained. Experimental results verified that the proposed algorithm outperformed some state-of-the-art fusion algorithms in terms of both subjective observation and objective evaluations.

Citation: Wang H (2018) Multi-focus image fusion algorithm based on focus detection in spatial and NSCT domain. PLoS ONE 13(9): e0204225. https://doi.org/10.1371/journal.pone.0204225

Editor: A Lenin Fred, Mar Ephraem College of Engineering & Technology, INDIA

Received: January 8, 2018; Accepted: August 13, 2018; Published: September 20, 2018

Copyright: © 2018 Hongmei Wang. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: This work is jointly supported by the National Natural Science Foundation of China (No. 61401366, 61771400, http://www.nsfc.gov.cn/).

Competing interests: "The authors have declared that no competing interests exist."

1 Introduction

Image fusion is one of the most important research subjects in image processing. Nowadays, most research focus on pixel-level image fusion. Pixel-level image fusion is achieved by combing two or more images coming from different or same image sensors so as to get a new image which contains more information than any of the original images and is more favorable for the follow-up working, such as target recognition, image understanding and so on [1].

It is well-known that the camera is limited in focus. An effective way to obtain an all-in-focus image is by image fusion [2]. The key issue in multi-focus image fusion is to detect the focused regions of each source image correctly. Varieties of focus measures have been developed, such as spatial frequency (SF), sum-modified-Laplacian (SML), and Tenenbaum gradient (Tenengrad) [3]. To improve the precision of focus detection results, Zhang [4] presented a novel focus measure method based on graph-based visual saliency algorithm, the watershed and morphological methods.

Basically, there are two types of methods for multi-focus image fusion. One is the spatial domain-based methods, which select pixels or regions from focused parts in the spatial domain to compose the fused images. The above focus measures have been widely applied in spatial domain-based fusion algorithms. Chen [5] proposed a multi-focus image fusion method based on edge model and multi-matting. The edge model and a traditional block-based focus measure are combined to estimate focus maps. In literature [6], orientation information motivated pulse coupled neural network was applied to obtain the initial decision map, and then the mathematical morphology was employed to modify the decision map. SF and the improved alternate SML is applied to build the initial tri-state map in [7] for multi-focus image fusion. Also, inspired by the mechanism of visual attention in humans, saliency detection model is proposed to detect the most noticeable and attractive region in a scene [8]. A novel multi-focus image fusion method based on the regional saliency is proposed in literature [4], in which the focused region of the source image is merged into the fused image as much as possible. Experiments demonstrated that the proposed approach can accurately extract the focused region and is superior to traditional methods in subjective and objective evaluations.

Another type of image fusion methods is the transform domain-based methods [9–11]. Compared with traditional multi-scale transform, MGA (multi-scale geometric analysis) transform can take full advantage of the geometric regularity of image intrinsic structures and obtain the asymptotic optimal representation, so the MGA transform-based image fusion can get better results and attracted more attention. Ridgelet transform, Curvelet transform, Contourlet transform, Shearlet transform [12] and Non-subsampled Contourlet transform (NSCT) [13] have been widely explored in image fusion, especially NSCT [14–17].

The transform domain-based image fusion algorithms are usually composed of three steps: image decomposition, fusion of the low-frequency subband and high-frequency subbands, image reconstruction. The critical element of transform-based image fusion algorithms is the design of fusion rules for subbands. Average or weighted average method is the most commonly used one for the low- frequency subband fusion. For the high-frequency subbands, the most popular fusion rule is to select the coefficients of subbands with larger absolute values. As a result, these rules do not take any consideration of the surrounding pixels. In order to tackle such a problem of traditional rules, some novel fusion rules were proposed for the low-frequency subband and high-frequency subbands [14]. In [17], the modified fusion rule is proposed for low-frequency subband based on SML. Also, a new high-frequency fusion rule based on local Log-Gabor energy is designed. Gao [18] presented a novel multi-focus image fusion algorithm based on non-subsampled Shearlet transform.

Sparse decomposition can represent the salient information of an image by building the relationship between features and sparse coefficients [19, 20]. Most of the sparse representation (SR)-based image fusion methods also belong to the transform domain-based techniques. Unlike the traditional multi-scale transforms that presume the basis functions, SR learns an over-complete dictionary from a set of training images and is proven to be more comprehensive and effective to extract the structure information of the source image. As a result, SR-based algorithm can get fused image with higher quality than traditional multi-scale transform-based algorithms [21, 22]. However, most of the SR-based image fusion algorithms have high computational complexity because of the increased time consumed during the sparse coding.

The above focus measures can be regarded as features. To overcome the shortcoming of artificial feature extraction, deep learning (DL) [23] has been applied in image fusion in recent years. In [24], Liu applied Siamese network to finish image fusion. Experimental results demonstrated that the proposed method can obtain state-of-the-art fusion performance in terms of both visual quality and objective assessment. Deshmukh [25] applied deep believe network (DBN) to obtain the feature vectors of input images. Mean of feature vectors are calculated and multiplied with input source images to obtain the fused image. At the same time, the all convolutional neural network (ACNN) is applied in multi-focus image fusion in literature [26]. Multi-focus image fusion algorithms based on deep learning network involve two key contents: establishment of giant image database and time-consuming training of the network.

No matter the multi-focus image fusion algorithm is implemented in spatial domain or in transform domain, each algorithm has its merit and shortcoming. Spatial domain-based image fusion algorithms have advantageous ability over transform domain-based methods in alleviating blurring effect and eliminating undesirable artifacts [18]. However, the spatial domain-based methods often suffer from block effect and erroneous results at the focused border regions. Although transform domain, especially the high-frequency subbands, can describe the salient features more effectively, transform domain-based fusion algorithms suffer from blurring effect because the results are usually obtained by image reconstruction which modifies the original image information to a certain extent. Inspired by these properties, a novel image fusion scheme by combining spatial information and transformation information is proposed in this paper. The contributions are summarized as follows:

A focus detection method which combines MPCNN and voting theory is proposed for high-frequency subbands of NSCT.
The focus detection results in spatial domain and NSCT domain are combined together to get more reliable fused image.
Modified postprocessing method for focus map is proposed to get high quality fused image.

The remainder of this paper is organized as follows. Brief reviews of NSCT theory and MPCNN are introduced in section 2. In section 3, the proposed algorithm based on focus measure in spatial domain and NSCT domain is described detailed. Experimental results and performance analysis are given in section 4. Finally, the conclusions are presented and future work is detailed in section 5.

2 Preliminaries

2.1 Non-subsampled contourlet transform

NSCT is a kind of multi-scale and multi-direction computation framework of the discrete images. Compared with other past MGA tools like DWT, NSCT has many important properties [13]:

The shift-invariance property thoroughly overcomes the Gibbs effects.
The size of subbands is identical, so it is not necessary to require the size of source image is multiples of 2.
Identical sizes are very convenient for us to devise the fusion rules for subbands. Refer to reference [13] for the detail theory of NSCT. Fig 1 shows the ‘zoneplate’ image and its NSCT decomposition results. The decomposition level of NSCT is set to 3. The directional number of each level is 2¹, 2² and 2², respectively.

Download:

Fig 1. ‘zoneplate’ image and its NSCT decomposition results.

(a) Original image, (b) Low-frequency subband, (c)-(d) 2-directional high-frequency subbands of the first layer, (e)-(h) 4-directional high-frequency subbands of the second layer, (i)-(l) 4-directional high-frequency subbands of the third layer.

https://doi.org/10.1371/journal.pone.0204225.g001

2.2 Memristive pulse coupled neural network

Pulse coupled neural network (PCNN) is a biologically inspired neural network based on the work by Eckhorn [27]. It has been proved that PCNN is very suitable for image processing such as image segmentation, image enhancement, pattern recognition and image fusion [14–16].

Through analyzing the universal phenomenon in biological nerve system and combining with the Eckhorn PCNN, in literature [28], the memristive pulse coupled neural network (MPCNN) is proposed and proved to be more effective than standard PCNN model. The model of MPCNN is given as follows. (1) (2) (3) (4) (5) (6) Where I_ij is the gray value of the corresponding pixel at position (i,j) for input image I; L_ij, F_ij and U_ij are the link input signal, external input and internal behavior, respectively; T_ij and Y_ij are threshold and output of the neuron, respectively. M_ij and R are memristor and resistor, respectively. M, W and β are parameters of MPCNN [28].

3 The proposed fusion algorithm

The proposed fusion algorithm includes the following four steps.

The focus detection in spatial domain is obtained by the SML, and the focus map Flag_S in spatial domain can be obtained.
The multi-focus images are decomposed by NSCT, the low-frequency subband and high-frequency subbands are obtained. The focus detection result (Flag_T) in NSCT domain is acquired by MPCNN and voting strategy for high- frequency subbands.
Synthesizing the focus map Flag_S in spatial domain and the focus map Flag_T in NSCT domain, the final focus map Flag can be obtained.
The fused image F can be obtained by source images and focus map Flag.

The detail description is given in section (3.1)-(3.4), respectively. The flowchart of the proposed algorithm is given in Fig 2.

Download:

Fig 2. Flowchart of the proposed algorithm.

https://doi.org/10.1371/journal.pone.0204225.g002

3.1 Focus detection in spatial domain

In literature [3], the authors demonstrated that SML can provide better performance than other focus measures, such as Variance, EOG, EOL, SF, and so on. In this paper, SML is applied as a focus measure to detect the focused region in spatial domain.

Suppose I_A and I_B are two multi-focus images to be fused, respectively. The SML of I_A and I_B can be abbreviated as SML_A and SML_B, respectively. The focus detection result is obtained by following formula.

(7)

Fig 3 gives focus detection results (focus maps) by SML for low-frequency subbands of NSCT and original source images. Fig 3(A1), 3(B1), 3(A2) and 3(B2) are multi-focus images, respectively. Fig 3(C1) and 3(C2) are focus detection results by SML in NSCT domain, namely for low-frequency subbands of NSCT with decomposition level is 2. Fig 3(D1) and 3(D2) are focus maps by SML for low-frequency subbands of NSCT with decomposition level is 3. Fig 3(E1) and 3(E2) are focus maps by SML in spatial domain, namely using multi-focus source images. From Fig 3, we can find that SML for source images is superior to low-frequency subbands of NSCT.

Download:

Fig 3. Focus maps by SML for source images and low-frequency subbands.

(a1)-(b1), (a2)-(b2) Multi-focus images, (c1)-(c2) Focus maps by SML for low-frequency subbands of NSCT with decomposition level set to 2, (d1)-(d2) Focus maps by SML for low-frequency subbands of NSCT with decomposition level set to 3, (e1)-(e2) Focus maps by SML for source images.

https://doi.org/10.1371/journal.pone.0204225.g003

3.2 Focus detection in NSCT domain

Low—frequency subband and high-frequency subbands can be obtained by NSCT transform. From section 3.1, we can find that focus detection result in spatial domain is better than low-frequency subband. High-frequency subbands contain the salient feature of image and are applied to obtain the focus detection result in NSCT domain as a result. Usually, salient features of multi-focus images correspond to lager absolute value of high-frequency subbands coefficients. Therefore, the most popular fusion rule is to select the coefficients with larger absolute values. The shortcoming of this rule is obvious in that it does not take any consideration of the surrounding pixels. In recent years, PCNN is proved to be effective in the fusion of high-frequency subbands and utilized frequently [14]. In this paper, an improved PCNN model, namely MPCNN is utilized as the tool to detect the focused region for high-frequency subbands. Also voting theory is introduced to get more precise detection result.

Supposing and are high-frequency subbands coefficients at level j, direction k, position (x,y) of source image A and B, respectively. Let (I = A or B) denotes sum of absolute value of at level j and all direction k (k = 1, 2… N, N is the total directional number at level j), namely, (8) (9)

The process of focus detection by high-frequency subbands in NSCT domain is given as follows.

Algorithm: MPCNN and voting-based focus detection for high-frequency subbands of NSCT

1. Initialize parameters, matrices of MPCNN and set the maximum iterative number.

2. For each level of high-frequency of multi-focus images A and B, do

Set total firing map ; feed and into MPCNN as input I_ij;

For each iteration, do

Execute MPCNN formulas (1)–(6), obtain firing output Y^A[n] and Y^B[n] of the network;

Add up Y^A[n] and Y^B[n] with and , respectively (10) (11)

End for

Compare and , Calculate the focus detection result for level j as follow (12)

End for

3. Apply voting theory to focus detection result acquired from all levels and obtain the final focus detection result in NSCT domain, labeled Flag_T.

3.3 Postprocessing of focus detection results

Although many focus measures have been utilized into the multi-focus image fusion, such as energy, RMSE, SF and SML utilized in this paper. However, determination by above focus measure is insufficient to identify all the focused pixels. In [18], morphological operation is employed to correct the focus detection results. Experimental results show morphological operation alone is not enough for our focus detection results. In this paper, the morphological operation and inconsistent processing are combined together to correct the focus detection results. The specific steps are given as follows.

Step 1: Execute morphological operation on Flag_s and Flag_T as follows, respectively. (13) (14) (15) (16) Where Flag_X can be Flag_s or Flag_T. Areasize is the size of region in Flag_X to be removed.

Step 2: Inconsistent check for Flag_s and Flag_T, the result is labeled Flag_I.

(17)

Step 3: Processing of inconsistent focused regions.

From left-top to right-bottom, for all the pixels satisfying Flag_I(i,j) = 1, choose a region of Flag_s and Flag_T centering with (i,j), abbreviated as R_s and R_T. Carry out the following formula. (18) Where std is standard deviation.

Though above process, the final focus detection result, labeled Flag, can be obtained.

(19)

Fig 4 shows the total focus detection results for ‘lab’ images. Fig 4(A) is the near focused image and Fig 4(B) is the far focused image. Fig 4(C) is the focus detection result by SML in spatial domain, Fig 4(D) is the morphological operation on Fig 4(C). Fig 4(E) is the focus detection result by MPCNN and voting algorithm in NSCT domain, Fig 4(F) is the morphological operation on Fig 4(E). Fig 4(G) shows the consistent regions (the white and black labeled regions) and inconsistent regions (the blue labeled regions) for Fig 4(D) and Fig 4(F). Fig 4(H) is the final focus map.

Download:

Fig 4. Multi-focus images and focus maps.

(a)-(b) Multi-focus source images, (c) Focus map by SML in spatial domain, (d) Morphological operation result of (c), (e) Focus map by MPCNN and voting in NSCT domain, (f) Morphological operation result of (d), (g) Consistent and inconsistent regions for (d) and (f), (h) Final focus map.

https://doi.org/10.1371/journal.pone.0204225.g004

From Fig 4(D) and 4(F), we can find decision maps in spatial and NSCT domain are complementary. This is advantageous to obtain accurate focus detection result and higher quality fused image.

3.4 Acquirement of the fused image

Supposing A, B and F are two source images and fused image, respectively. Flag is the focus map. The fused image can be obtained by the following formula.

(20)

The pseudo codes of the proposed algorithm are given as follows.

Input: Multi-focus source images A and B

Output: Fused image F

Begin:

Compute SML for source images A and B, and represented as SML_A and SML_B, respectively;

Compare SML_A and SML_B, and obtain spatial-domain focus map Flag_S;

Set the parameters of NSCT transform, decompose image A and image B by NSCT, respectively;

Compute sum of absolute value and for high frequency subbands;

Set the parameters of MPCNN;

Compute and compare outputs of MPCNN for and ;

Apply voting theory to obtain the focus map Flag_T in transform domain;

Execute morphological operation on Flag_S and Flag_T, respectively;

Inconsistent check for Flag_S and Flag_T;

Processing for inconsistent regions and obtain final focus map Flag;

Compute the fused image F by Flag and source images.

End.

4 Experimental results and discussion

In this section, experimental results are given to evaluate the effectiveness of the proposed algorithm. Four databases are applied in ours experiments. (1) The Petrović database [29] which contains 50 pairs of images including aerial images, outdoor images and indoor images. (2) The multi-focus images database [30] which contains 10 pairs of multi-focus images. (3) Lytro multi-focus database [22] which contains 20 pairs of color multi-focus images with size 520×520. (4) The artificial database which is produced by adding Gaussian blur to part of the original images with different standard derivations and Gaussian filter with different size.

Fig 5 shows some multi-focus images and the corresponding focus maps by ours proposed algorithm. Fig 5(A1) and 5(B1) are ‘pepsi’ images, Fig 5(C1) is the focus map. Fig 5(A2) and 5(B2) are ‘balloon’ images, Fig 5(C2) is the focus map. Fig 5(A3) and 5(B3) are ‘desk’ images, Fig 5(C3) is the focus map. Fig 5(A4) and 5(B4) are ‘lytro-20’ images, Fig 5(C4) is the focus map. Fig 5(A5) and 5(B5) are ‘peppers’ images, Fig 5(C5) is the focus map.

Download:

Fig 5. Multi-focus source images and final focus maps.

From top to bottom are multi-focus images and focus maps, respectively.

https://doi.org/10.1371/journal.pone.0204225.g005

The fusion results obtained from the proposed algorithm are compared with other state-of-the-art image fusion algorithms, including: image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled Contourlet transform domain (NSCT-SF-PCNN) [14], multi-scale weighted gradient-based fusion for multi-focus image (MWGF) [10], the guided filter fusion algorithm (GFF) [11], multi-focus image fusion with sparse representation (SR) [19] and the CNN-based multi-focus image fusion method [24]. The codes of the NSCT-SF-PCNN algorithm are available on the author’s homepage [31]. The codes of the MWGF-based method are available on [32]. The codes of the GFF-based method are available on homepage [33]. The codes of the SR-based method and CNN-based fusion method are provided by [34]. In ours algorithm, the parameters of MPCNN are same as literature [28]. The pyramidal and directional filter for NSCT is 'maxflat' and 'dmaxflat7', respectively.

Fig 6 presents the experimental results for ‘desk’ images (480×640). Fig 6(A)–6(F) are the results by NSCT-SF-PCNN method, MWGF method, GFF method, SR method, CNN method and ours algorithm, respectively. To show the effectiveness of ours algorithm further, the residual images between the fused images and source image are given in Fig 6(G)–6(L), respectively. The residual image are obtained by subtracting the Fig 5(B3) from each fused image. If the focus regions are selected correctly, the corresponding value of left part pixels of residual image should be zero.

Download:

Fig 6. Fused images and residual images by different methods for ‘desk’ images.

(a)-(f) Fused images by different methods, (g)-(l) Residual images by (a)-(f) with Fig 5(B3).

https://doi.org/10.1371/journal.pone.0204225.g006

The fusion results of the NSCT-SF-PCNN-based method, MWGF-based method and GFF-based method contain some undesirable artifacts on the fused image, especially for the NSCT-SF-PCNN-based method. This can be demonstrated by the residual images obviously. The SR-based fusion method performs well relatively, but its image quality is still not very high. The CNN method and ours method can select the focused regions most exactly because almost all the pixels in left part of residual images have zero value.

As we know, the fused image can be evaluated by subjective observation and objective evaluation. Objective evaluation also plays important role in image fusion. In this paper, average gradient (AG), standard deviation (STD), mutual information (MI) and edge information preservation value (Q^AB/F) are applied as objective evaluation indexes [35–36]. Usually, the higher value of AG, STD, MI and Q^AB/F, the better quality of the fused image. The objective evaluation results for ‘desk’ images are shown in Table 1.

Download:

Table 1. Objective assessment of different fusion algorithms for ‘desk’ images.

https://doi.org/10.1371/journal.pone.0204225.t001

From Table 1, we can find that the proposed algorithm get the highest value of AG, MI and Q^AB/F. The value of STD is a little lower than MWGF-based algorithm. This also demonstrates that the proposed algorithm is the best one compared with other algorithms.

Next, image fusion results with all-in-focus image as reference are given to demonstrate the effectiveness of the proposed algorithm further. The all-in-focus image ‘peppers’ (384×512) comes from Matlab toolbox. The multi-focus source images are produced by artificial way and shown in Fig 5(A5) and Fig 5(B5). Fig 7(A)–7(F) are the results by NSCT-SF-PCNN-based method, MWGF-based method, GFF-based method, SR-based method, CNN-based method and ours method, respectively. Fig 7(G)–7(L) are residual images by all-in-focus ‘peppers’ image with Fig 7(A)–7(F), respectively. If the focused regions are selected correctly, the value of pixels of residual images should be zero.

Download:

Fig 7. Fused images and residual images by different methods for ‘peppers’ images.

(a)-(f) Fused images by different methods, (g)-(l) Residual images by (a)-(f) with all-in-focus image.

https://doi.org/10.1371/journal.pone.0204225.g007

From Fig 7, we can find that most pixels of the residual image by ours algorithm are zeroes. Lots of pixels have non-zero value in Fig 7(G) and Fig 7(H), respectively. Also, some pixels in Fig 7(I), Fig 7(J) and Fig 7(K) have non-zero value. These results also demonstrate that ours algorithm can select the focused region more correctly.

Furthermore, we compare ours algorithm with state-of-the-art literatures for ‘pepsi’ images in Fig 8. Fig 8(A) is the MI of fused image by different algorithms. Fig 8(B) is the Q^AB/F of fused image by different algorithms. In this figure, black block represent the result by algorithm of literature [14]; red block is the result by algorithm of literature [15]; green block is the result by algorithm of literature [16]; blue block is the result by algorithm of literature [17] and the last cyan block is the result by ours algorithm.

Download:

Fig 8. MI and Q^AB/F for ‘pepsi’ multi-focus images.

https://doi.org/10.1371/journal.pone.0204225.g008

From Fig 8, we can find that the proposed algorithm can obtain higher MI than the algorithms in literature [14–16], and a litter lower than the algorithm in literature [17]. At the same time, Q^AB/F by our algorithm is higher than the algorithms in literature [14–17]. These results also demonstrated that the proposed algorithm is effective for multi-focus image fusion.

At last, the computational performance of the proposed algorithm is evaluated in this section. Table 2 lists the time consuming on the ‘desk’ images of the six algorithms. Because NSCT-SF-PCNN and ours algorithm applied NSCT decomposition which is very time consuming, the efficiency of NSCT-SF-PCNN and ours algorithm are lower. SR-based method takes the longest time because of the sparse coding during the fusion. CNN-based method are also low efficient because of the convolution and pooling operation. In the six algorithms, MWGF-based method is relatively time efficient. GFF-based method costs shortest time and is the most efficient one as a result.

Download:

Table 2. Time consuming (s) of the six algorithms on ‘desk’ images.

https://doi.org/10.1371/journal.pone.0204225.t002

5 Conclusions and future work

Pixel-level image fusion has been a very important topic in multi-sensor image fusion. Aiming at obtain an all-in-focus image, a novel multi-focus image fusion algorithm is proposed which combine the property of spatial domain and NSCT domain. In spatial domain, SML is utilized to detect the focused regions. In NSCT domain, MPCNN and voting method are combined together to obtain the focus detection result. At last, by synthesizing the decision maps in spatial domain and NSCT domain, the focus detection result can be obtained and employed to get the fused image. Plentiful experiments are carried through to verify the effectiveness of the proposed algorithm. Experimental results demonstrate that the proposed algorithm outperforms the NSCT-SF-PCNN-based fusion method, MWGF-based fusion algorithm, GFF-based algorithm and some other state-of-the-art image fusion methods in terms of both visual quality and objective evaluation. Although the proposed algorithm can get better fusion result, there is a limit to real-time application. In the future research, we will focus on simplifying the model to improve the efficiency of the algorithm.

Acknowledgments

The author would like to thank the reviewers and the editors for their valuable comments and helpful suggestions.

References

1. Blum R S, Liu Z. Multi-Sensor Image Fusion and Its Applications. Taylor & Francis, 2006.
2. Aslantas V, Toprak A N. A pixel based multi-focus image fusion method. Optics Communications. 2014; 332:350–358.
- View Article
- Google Scholar
3. Huang W, Jing Z L. Evaluation of focus measures in multi-focus image fusion. Pattern Recognition Letters. 2007; 28:493–500.
- View Article
- Google Scholar
4. Zhang B H, Lu X Q, Pei H Q, Liu H, Zhao Y, Zhou W T. Multi-focus image fusion algorithm based on focused region extraction. Neurocomputing. 2017; 174: 733–748.
- View Article
- Google Scholar
5. Chen Y B, Guan J W, Cham W K. Robust Multi-Focus Image Fusion Using Edge Model and Multi-Matting. IEEE Transactions on Image Processing. 2018; 27(3): 1526–1541. pmid:29990190
- View Article
- PubMed/NCBI
- Google Scholar
6. Du C B, Gao S. Multi-focus image fusion algorithm based on pulse coupled neural networks and modified decision map. Optik. 2018; 157:1003–1015.
- View Article
- Google Scholar
7. Rahman M A, Liu S,Wong C Y, Lin S C F,Liu S C,Kwok N M. Multi-focal image fusion using degree of focus and Fuzzy logic. Digital Signal Processing. 2017; 60:1–19.
- View Article
- Google Scholar
8. Han J G, Pauwels E J, Zeeuw P. Fast saliency-aware multi-modality image fusion. Neurocomputing. 2013; 111: 70–80.
- View Article
- Google Scholar
9. Zhang Y, Bai X Z, Wang T. Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure. Information Fusion. 2017; 35: 81–101.
- View Article
- Google Scholar
10. Zhou Z Q, Li S, Wang B. Multi-scale weighted gradient-based fusion for multi-focus images. Information Fusion. 2014; 20: 60–72.
- View Article
- Google Scholar
11. Li S T, Kang X D, Hu J W. Image fusion with guided filtering. IEEE Transactions on Image processing. 2013; 22(7): 2864–2875. pmid:23372084
- View Article
- PubMed/NCBI
- Google Scholar
12. Lim W Q. The Discrete Shearlet Transform: A New Directional Transform and Compactly Supported Shearlet Frames. IEEE Transaction on Image Process. 2010; 19(5):1166–1180. pmid:20106737
- View Article
- PubMed/NCBI
- Google Scholar
13. Cunha A L, Zhou J P, Do M N. The nonsubsampled contourlet transform: theory, design, and applications. IEEE Transactions on Image Processing. 2006; 15(10):3089–3101. pmid:17022272
- View Article
- PubMed/NCBI
- Google Scholar
14. Qu X B, Yan J W, Xiao H Z, Zhu Z Q. Image Fusion Algorithm Based on Spatial Frequency-Motivated Pulse Coupled Neural Networks in Nonsubsampled Contourlet Transform Domain. Acta Automatic Sinica. 2008; 34:1508–1514.
- View Article
- Google Scholar
15. Yin M, Pang J Y, Wei Y, Duan P H. Image Fusion Algorithm Based on Nonsubsampled Dual-tree Contourlet Transform and Compressive Sensing Pulse Coupled Neural Network. Journal of Computer-Aided Design & Computer Graphics.2016; 28:411–419.
- View Article
- Google Scholar
16. Wang J, Li Q, Jia Z H, Kasabov N, Yang J. A novel multi-focus image fusion method using PCNN in nonsubsampled Contourlet transform domain. Optik. 2015; 126: 2508–2511.
- View Article
- Google Scholar
17. Yang Y, Tong S, Huang S Y, Lin P. Multifocus image fusion based on NSCT and Focused Area Detection. IEEE Sensors. 2015; 2824–2835.
- View Article
- Google Scholar
18. Gao G R, Xu L P, Feng D Z. Multi-focus image fusion based on non-subsampled shearlet transform. IET Image Processing. 2013; 7: 633–639.
- View Article
- Google Scholar
19. Yang B, Li S. Multifocus image fusion and restoration with sparse representation. IEEE Transactions on Instrumentation and Measurement. 2010; 59: 884–892.
- View Article
- Google Scholar
20. Zhang B H, Lu X Q, Pei H Q, Liu Y X, Zhou W T, Jiao D D. Multi-focus image fusion based on spare decomposition and background detection. Digital Signal Processing. 2016; 58: 50–63.
- View Article
- Google Scholar
21. Zhang Q, Liu Y, Blum R S, Han J G, Tao D C. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review. Information Fusion. 2018; 40:57–75.
- View Article
- Google Scholar
22. Nejati M, Samavi S, Shirani S. Multi-focus Image Fusion Using Dictionary-Based Sparse Representation. Information Fusion. 2015; 25:72–84.
- View Article
- Google Scholar
23. Jiao L C, Yang S Y, Liu F, Wang Z G, Feng Z X. Seventy Years beyond Neural Networks: Retrospect and Prospect. Chinese Journal of Computers. 2016; 39(8):1697–1716.
- View Article
- Google Scholar
24. Liu Y, Chen X, Peng H, Wang Z F. Multi-focus image fusion with a deep convolutional neural network. Information Fusion. 2017; 36:191–207.
- View Article
- Google Scholar
25. Deshmukh V, Khaparde A, Shaikh S. Multi-focus image fusion using deep belief network. Information and Communication Technology for Intelligent Systems, ICTIS 2017. 2018; 83: 233–241.
- View Article
- Google Scholar
26. Du C B, Gao S. Multi-focus image fusion with the all convolutional neural network. Optoelectronics Letters. 2018; 14(1): 0071–0075.
- View Article
- Google Scholar
27. Johnson J L, Padgett M L. PCNN models and applications. IEEE Transactions on Neural Networks. 1999; 10:480–498. pmid:18252547
- View Article
- PubMed/NCBI
- Google Scholar
28. Zhu S, Wang L D, Duan S K. Memristive pulse coupled neural network with applications in medical image processing. Neurocomputing. 2017; 227:149–157.
- View Article
- Google Scholar
29. Petrović V. Subjective tests for image fusion evaluation and objective validation. Information Fusion. 2007; 8(2): 208–216.
- View Article
- Google Scholar
30. http://www.imagefusion.org.
31. http://quxiaobo.8866.org/project/NSCT_SF_PCNN/NSCT-SF-PCNN-ImageFusion-Toolbox.zip.
32. https://www.researchgate.net/publication/307415978_MATLAB_Code_of_Our_Multi-focus_Image_Fusion_Algorithm_MWGF.
33. http://xudongkang.weebly.com/index.html.
34. http://home.ustc.edu.cn/~liuyu1.
35. Wang Q, Yu D R, Shen Y. An overview of image fusion metrics. International Instrumentation and Measurement Technology Conference. 2009; 5:5–7.
- View Article
- Google Scholar
36. Xydeas C S, Petrović V. Objective image fusion performance measure. Electronics Letters. 2000; 4:308–309.
- View Article
- Google Scholar

[ref1] 1. Blum R S, Liu Z. Multi-Sensor Image Fusion and Its Applications. Taylor & Francis, 2006.

[ref2] 2. Aslantas V, Toprak A N. A pixel based multi-focus image fusion method. Optics Communications. 2014; 332:350–358.
View Article
Google Scholar

[3] View Article

[4] Google Scholar

[ref3] 3. Huang W, Jing Z L. Evaluation of focus measures in multi-focus image fusion. Pattern Recognition Letters. 2007; 28:493–500.
View Article
Google Scholar

[6] View Article

[7] Google Scholar

[ref4] 4. Zhang B H, Lu X Q, Pei H Q, Liu H, Zhao Y, Zhou W T. Multi-focus image fusion algorithm based on focused region extraction. Neurocomputing. 2017; 174: 733–748.
View Article
Google Scholar

[9] View Article

[10] Google Scholar

[ref5] 5. Chen Y B, Guan J W, Cham W K. Robust Multi-Focus Image Fusion Using Edge Model and Multi-Matting. IEEE Transactions on Image Processing. 2018; 27(3): 1526–1541. pmid:29990190
View Article
PubMed/NCBI
Google Scholar

[12] View Article

[13] PubMed/NCBI

[14] Google Scholar

[ref6] 6. Du C B, Gao S. Multi-focus image fusion algorithm based on pulse coupled neural networks and modified decision map. Optik. 2018; 157:1003–1015.
View Article
Google Scholar

[16] View Article

[17] Google Scholar

[ref7] 7. Rahman M A, Liu S,Wong C Y, Lin S C F,Liu S C,Kwok N M. Multi-focal image fusion using degree of focus and Fuzzy logic. Digital Signal Processing. 2017; 60:1–19.
View Article
Google Scholar

[19] View Article

[20] Google Scholar

[ref8] 8. Han J G, Pauwels E J, Zeeuw P. Fast saliency-aware multi-modality image fusion. Neurocomputing. 2013; 111: 70–80.
View Article
Google Scholar

[22] View Article

[23] Google Scholar

[ref9] 9. Zhang Y, Bai X Z, Wang T. Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure. Information Fusion. 2017; 35: 81–101.
View Article
Google Scholar

[25] View Article

[26] Google Scholar

[ref10] 10. Zhou Z Q, Li S, Wang B. Multi-scale weighted gradient-based fusion for multi-focus images. Information Fusion. 2014; 20: 60–72.
View Article
Google Scholar

[28] View Article

[29] Google Scholar

[ref11] 11. Li S T, Kang X D, Hu J W. Image fusion with guided filtering. IEEE Transactions on Image processing. 2013; 22(7): 2864–2875. pmid:23372084
View Article
PubMed/NCBI
Google Scholar

[31] View Article

[32] PubMed/NCBI

[33] Google Scholar

[ref12] 12. Lim W Q. The Discrete Shearlet Transform: A New Directional Transform and Compactly Supported Shearlet Frames. IEEE Transaction on Image Process. 2010; 19(5):1166–1180. pmid:20106737
View Article
PubMed/NCBI
Google Scholar

[35] View Article

[36] PubMed/NCBI

[37] Google Scholar

[ref13] 13. Cunha A L, Zhou J P, Do M N. The nonsubsampled contourlet transform: theory, design, and applications. IEEE Transactions on Image Processing. 2006; 15(10):3089–3101. pmid:17022272
View Article
PubMed/NCBI
Google Scholar

[39] View Article

[40] PubMed/NCBI

[41] Google Scholar

[ref14] 14. Qu X B, Yan J W, Xiao H Z, Zhu Z Q. Image Fusion Algorithm Based on Spatial Frequency-Motivated Pulse Coupled Neural Networks in Nonsubsampled Contourlet Transform Domain. Acta Automatic Sinica. 2008; 34:1508–1514.
View Article
Google Scholar

[43] View Article

[44] Google Scholar

[ref15] 15. Yin M, Pang J Y, Wei Y, Duan P H. Image Fusion Algorithm Based on Nonsubsampled Dual-tree Contourlet Transform and Compressive Sensing Pulse Coupled Neural Network. Journal of Computer-Aided Design & Computer Graphics.2016; 28:411–419.
View Article
Google Scholar

[46] View Article

[47] Google Scholar

[ref16] 16. Wang J, Li Q, Jia Z H, Kasabov N, Yang J. A novel multi-focus image fusion method using PCNN in nonsubsampled Contourlet transform domain. Optik. 2015; 126: 2508–2511.
View Article
Google Scholar

[49] View Article

[50] Google Scholar

[ref17] 17. Yang Y, Tong S, Huang S Y, Lin P. Multifocus image fusion based on NSCT and Focused Area Detection. IEEE Sensors. 2015; 2824–2835.
View Article
Google Scholar

[52] View Article

[53] Google Scholar

[ref18] 18. Gao G R, Xu L P, Feng D Z. Multi-focus image fusion based on non-subsampled shearlet transform. IET Image Processing. 2013; 7: 633–639.
View Article
Google Scholar

[55] View Article

[56] Google Scholar

[ref19] 19. Yang B, Li S. Multifocus image fusion and restoration with sparse representation. IEEE Transactions on Instrumentation and Measurement. 2010; 59: 884–892.
View Article
Google Scholar

[58] View Article

[59] Google Scholar

[ref20] 20. Zhang B H, Lu X Q, Pei H Q, Liu Y X, Zhou W T, Jiao D D. Multi-focus image fusion based on spare decomposition and background detection. Digital Signal Processing. 2016; 58: 50–63.
View Article
Google Scholar

[61] View Article

[62] Google Scholar

[ref21] 21. Zhang Q, Liu Y, Blum R S, Han J G, Tao D C. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review. Information Fusion. 2018; 40:57–75.
View Article
Google Scholar

[64] View Article

[65] Google Scholar

[ref22] 22. Nejati M, Samavi S, Shirani S. Multi-focus Image Fusion Using Dictionary-Based Sparse Representation. Information Fusion. 2015; 25:72–84.
View Article
Google Scholar

[67] View Article

[68] Google Scholar

[ref23] 23. Jiao L C, Yang S Y, Liu F, Wang Z G, Feng Z X. Seventy Years beyond Neural Networks: Retrospect and Prospect. Chinese Journal of Computers. 2016; 39(8):1697–1716.
View Article
Google Scholar

[70] View Article

[71] Google Scholar

[ref24] 24. Liu Y, Chen X, Peng H, Wang Z F. Multi-focus image fusion with a deep convolutional neural network. Information Fusion. 2017; 36:191–207.
View Article
Google Scholar

[73] View Article

[74] Google Scholar

[ref25] 25. Deshmukh V, Khaparde A, Shaikh S. Multi-focus image fusion using deep belief network. Information and Communication Technology for Intelligent Systems, ICTIS 2017. 2018; 83: 233–241.
View Article
Google Scholar

[76] View Article

[77] Google Scholar

[ref26] 26. Du C B, Gao S. Multi-focus image fusion with the all convolutional neural network. Optoelectronics Letters. 2018; 14(1): 0071–0075.
View Article
Google Scholar

[79] View Article

[80] Google Scholar

[ref27] 27. Johnson J L, Padgett M L. PCNN models and applications. IEEE Transactions on Neural Networks. 1999; 10:480–498. pmid:18252547
View Article
PubMed/NCBI
Google Scholar

[82] View Article

[83] PubMed/NCBI

[84] Google Scholar

[ref28] 28. Zhu S, Wang L D, Duan S K. Memristive pulse coupled neural network with applications in medical image processing. Neurocomputing. 2017; 227:149–157.
View Article
Google Scholar

[86] View Article

[87] Google Scholar

[ref29] 29. Petrović V. Subjective tests for image fusion evaluation and objective validation. Information Fusion. 2007; 8(2): 208–216.
View Article
Google Scholar

[89] View Article

[90] Google Scholar

[ref30] 30. http://www.imagefusion.org.

[ref31] 31. http://quxiaobo.8866.org/project/NSCT_SF_PCNN/NSCT-SF-PCNN-ImageFusion-Toolbox.zip.

[ref32] 32. https://www.researchgate.net/publication/307415978_MATLAB_Code_of_Our_Multi-focus_Image_Fusion_Algorithm_MWGF.

[ref33] 33. http://xudongkang.weebly.com/index.html.

[ref34] 34. http://home.ustc.edu.cn/~liuyu1.

[ref35] 35. Wang Q, Yu D R, Shen Y. An overview of image fusion metrics. International Instrumentation and Measurement Technology Conference. 2009; 5:5–7.
View Article
Google Scholar

[97] View Article

[98] Google Scholar

[ref36] 36. Xydeas C S, Petrović V. Objective image fusion performance measure. Electronics Letters. 2000; 4:308–309.
View Article
Google Scholar

[100] View Article

[101] Google Scholar

Figures

Abstract

1 Introduction

2 Preliminaries

2.1 Non-subsampled contourlet transform

2.2 Memristive pulse coupled neural network

3 The proposed fusion algorithm

3.1 Focus detection in spatial domain

3.2 Focus detection in NSCT domain

3.3 Postprocessing of focus detection results

3.4 Acquirement of the fused image

4 Experimental results and discussion

5 Conclusions and future work

Acknowledgments

References