https://orcid.org/0000-0001-9309-8550
Figures
Abstract
In the field of oncology imaging, the fusion of magnetic resonance imaging (MRI) and positron emission tomography (PET) modalities is crucial for enhancing diagnostic capabilities. This article introduces a novel fusion method that leverages the strengths of both modalities to overcome limitations associated with functional information in MRI and the spatial resolution in PET scans. Our approach integrates the Laplacian pyramid for extracting high and low-frequency components, along with empirical mode decomposition and phase congruency to preserve crucial structural details in the fused image. Additionally, a rolling guidance filter is employed to mitigate edge detail loss. Through extensive comparative experiments on multi-focus and multi-modal image datasets, our method consistently outperforms existing techniques in terms of visualization, objective metrics, and computational efficiency. The proposed fusion method demonstrates superior performance, establishing it as a compelling alternative for oncology imaging applications.
Citation: Suryanarayana G, Nimmagadda SM, Nageswara Rao S, Y. Mahnashi AM, Kondamuri SR, Hussain Ahmadini AA, et al. (2025) Enhanced MRI-PET fusion using Laplacian pyramid and empirical mode decomposition for improved oncology imaging. PLoS One 20(5): e0322443. https://doi.org/10.1371/journal.pone.0322443
Editor: Liang-Jian Deng, University of Electronic Science and Technology of China, CHINA
Received: September 13, 2024; Accepted: March 21, 2025; Published: May 19, 2025
Copyright: © 2025 Suryanarayana et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are available from the article.
Funding: This work is funded by the Deanship of Graduate Studies and Scientific Research, Jazan University, Saudi Arabia, through project number: RG24-S042.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
In oncology imaging, the combination of magnetic resonance imaging (MRI) and positron emission tomography (PET) plays a pivotal role in detecting conditions such as breast cancer and brain tumors. While MRI offers high-resolution anatomical details, it often lacks depth in functional information. Conversely, PET provides functional information but with lower spatial resolution, posing challenges for accurate disease diagnosis [1]. To address these limitations and harness the complementary strengths of both modalities, image fusion techniques have been widely explored. Image fusion involves combining information from multiple source images to create a single composite image that integrates the best features of each modality [2]. Various fusion methods can be categorized into spatial domain-based, transform domain-based, and learning-based techniques.
Spatial domain methods directly manipulate the individual pixels within the input image to achieve the desired result [3]. This method does not require sophisticated mathematical procedures or frequency domain conversions. The most popular techniques for spatial fusion include weighted average, principal component analysis, and simple fusion methods [4]. While these techniques are user-friendly, they often yield fused images with notable distortion and reduced effectiveness.
Transform domain fusion involves the conversion of input images from the spatial domain to the frequency domain using various transform methods such as hybrid transforms, Stationary Wavelet Transform (SWT), Discrete Cosine Transform (DCT), and Discrete Wavelet Transform (DWT) [4–8]. This process entails partitioning the input image into numerous sub-band images through low-pass and high-pass filtering [9]. Transform domain fusion methods typically result in improved peak signal-to-noise ratios (PSNR) and reduced mean square errors (MSE). However, they are often attributed to translational invariance, which can potentially lead to distorted edges in the output image [10].
Learning-based techniques utilize machine learning or deep learning algorithms to automatically learn the fusion process from a set of training data [11–14]. Convolutional Neural Networks (CNNs) are commonly used for learning-based fusion tasks [15,16]. These networks learn hierarchical representations of input images and fuse them at different levels of abstraction to generate the fused image. Learning-based fusion techniques have shown promising results in various applications and can adapt to different types of input data [17,18]. They are particularly useful when the fusion problem is complex and traditional techniques may not be sufficient.
However, learning-based techniques often require large datasets, which may not always be practical in medical scenarios. Additionally, these methods tend to be computationally intensive compared to spatial and transform domain techniques. Therefore, in this work, we propose a novel MRI-PET fusion method using a combination of Laplacian pyramid (LP) and empirical mode decomposition (EMD). This approach aims to extract and integrate high and low-frequency components effectively while preserving crucial structural details within the fused image. Our method achieves computational balance by applying phase congruency (PC) and rolling guidance filter (RGF) exclusively on the intrinsic mode functions of high-frequency components. Unlike existing methods, our approach strikes a balance between computational efficiency and information preservation, making it suitable for medical image fusion tasks.
Section 2 provides an overview of the preliminary techniques utilized in this study, including the RGF and LP. The proposed fusion method is detailed in Section 3, where we outline our approach for combining these techniques to achieve effective image fusion. In Section 4, we validate our proposed method through both qualitative and quantitative analyses, evaluating its performance against established benchmarks. Finally, Section 5 concludes the paper, summarizing key findings.
2. Preliminaries
This section provides an overview of the RGF and LP techniques used in this work.
2.1. Rolling guidance filter
RGF is a pivotal element within our proposed method for fusing MRI and PET images. Its primary function is to counteract the loss of edge details that often occurs during the fusion process [19]. Unlike conventional filtering techniques, the RGF dynamically adjusts to the local structures present in the images, thereby preserving crucial edge information. This adaptability contributes significantly to achieving higher peak signal-to-noise ratios (PSNR) and lower mean square errors (MSE) compared to previous fusion methodologies especially in transform domain. By retaining edge details, the RGF substantially improves the overall quality of the fused images, thereby addressing the challenges posed by the low spatial resolution of PET and the limited functional information provided by MRI [20].
RGF integrates essential components such as the Gaussian filter (GF) and the joint bilateral filter (JBF). The block diagram of RGF shown in Fig 1 illustrates the input passing through a GF followed by stages of JBFs. The output of GF is denoted by
, while the output of
th stage JBF is denoted by
. These filters [21] are implemented in the spatial domain using moving window operation as
In the equation, denotes the index pixel coordinates, while
represents the neighborhood surrounding the
th pixel.
is the pixel intensity at
in the input image, serving as the initial data to be processed by the RGF.
represents the filtered output pixel value at
. The weight assigned to the
th input pixel in the
th output pixel is denoted by
. For Gaussian filter, the weight is determined by
From scale space theory, the GF smoothens details, with the scale parameter dependent on the degree of high-contrast edges intended to be preserved. However, the GF not only removes small-scaled structures but also blurs large-scaled structures. To recover these blurred high-contrast edges, the Gaussian output is iteratively filtered using JBF, with weights determined by the equation
Here, the first iteration starts with . JBF considers both Euclidean distance and radiometric differences, recovering high-contrast edges by adjusting weights according to neighbouring pixels. Additionally, the degree of sharpness increases with each stage of filtering due to the cascading effect of these filters.
Indeed, and
play crucial roles in controlling the spatial and range weights, respectively, within the JBF. These parameters determine the extent of spatial and intensity variations considered during the filtering process. By utilizing the input image and previous Gaussian functions, the JBF effectively generates images that retain Gaussian edge information while simultaneously preserving structural details and minimizing artifacts. This approach ensures that the filtering process maintains edge sharpness and overall image quality.
2.2. Laplacian pyramid
The LP, pioneered by Burt et al [22], marks a significant milestone in image processing and computer vision. This hierarchical image representation method is renowned for its efficiency and adaptability across a range of image analysis tasks. Unlike conventional pyramidal structures, the LP doesn’t store down-sampled versions of the image. Instead, it innovatively decomposes the original image into a sequence of successively down-sampled and band-pass-filtered sub-images, each capturing distinct frequency components. This approach enables the extraction of intricate details while preserving computational resources. One of its key strengths lies in providing a multiresolution representation of the image. Each level of the pyramid corresponds to a specific frequency band, facilitating the independent analysis of various features [23]. This hierarchical structure proves invaluable in scenarios demanding detailed examination of specific frequency components.
The decomposition of input image is depicted in Fig 2, where
and
represent high frequency sub-band images, while
denotes the low frequency sub-band image. The LP is widely employed in image compression, excelling at capturing and preserving visual details while reducing overall data size, making it invaluable for storage or bandwidth-constrained applications. Additionally, it is highly effective in edge detection tasks, emphasizing fine features crucial for feature identification in computer vision applications. Moreover, the LP is pivotal in image blending, seamlessly integrating different images at various levels to produce a blended output that retains essential details from each source image. This capability is particularly useful in graphics and visual effects, where smooth transitions are paramount. Supporting progressive image reconstruction, the LP enables obtaining approximations of the original image at different resolutions, catering to scenarios requiring both quick overviews and detailed refinement [24]. Overall, it stands as a cornerstone in image processing and computer vision, offering a compact, multiresolution representation that continues to shape advancements in the field.
By integrating RGF and LP, the proposed fusion approach not only leverages the strengths of multi-scale decomposition and selective smoothing but also ensures that both high and low-frequency components are optimally processed.
3. Proposed method
In this section, we delve into the detailed processes of PC, EMD, and the LP, considering an input MRI image and a PET image
. The block diagram of proposed method is depicted in Fig 3.
Laplacian Pyramid- Application of LP to the input images generates sub-band images at both high and low frequencies. For image , this results in
where ,
represent sub-band images at high frequencies, and
denotes the low-frequency sub-band image. Similarly, for image B
where represent sub-band images at high frequencies, and
denotes the low-frequency sub-band image.
Empirical mode decomposition- Applying EMD to the high-frequency sub-band images generates residue as well as intrinsic mode functions (IMFs). In this context, the residue exhibits low frequency, while the IMFs demonstrate high frequency.
Equation (6) represents the decomposition of into IMFs
and residual function
using EMD. In equation (7),
, and
are the IMFs and residue of
. Similarly,
, and
corresponds to
, while
, and
corresponds to
. Following decomposition, all these IMFs undergo edge extraction using Gaussian and JBF operations.
Phase congruency- Leveraging the phase congruence of Fourier components, PC is capable of highlighting small discontinuities in images with varying lighting and contrast, thereby aiding in a deeper understanding of the image’s context and content. This approach detects subtle variations in image intensity and highlights anomalous values. This fusion process of IMFs based on PC can be described as
Here are the fused IMFs corresponding to
and
respectively.
are the respective fused IMFs corresponding to
and
.
Mean fusion- Mean fusion (MF) is applied to the residues and
to yield
, and similarly to
and
to produce
. The low-frequency sub-band images
corresponding to MRI and PET images are also subjected to mean fusion resulting in
Finally, the resultant fused image F is produced by combining the fused IMFs and
along with the residuals
and the low-frequency sub-band
The fused image exhibits enhanced structural detail preservation and edge clarity, offering a robust solution for advancing diagnostic precision and efficacy in oncology imaging.
4. Experiments and results
This section outlines the fusion process applied to diverse datasets, evaluating algorithmic performance using key metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), mutual information (MI), average gradient (AG), and entropy fusion (EN), quality of edge information (QABF), loss of edge information (LABF), and noise or artifacts added during the fusion (NABF). The sum of QABF, LABF, and NABF should be unity. A comparative analysis was conducted to assess the effectiveness of our algorithm against alternatives, including structure-aware (STAW) [12], nonsubsampled contourlet transform (NSCT) [5], Laplacian re-decomposition (LRD) [6], CNN [13], EMD [7], inter-modality attention (IMA) [14], dual-tree complex wavelet transform (DTCWT) [8], correlation-driven feature decomposition fusion, FCDFusion [25], CDDFuse [26], and CMTFusion [27]. All experiments and output visualizations were exclusively performed in MATLAB 2021 on a laptop equipped with an Intel Core i4 CPU and 4.0 GB RAM. We used two datasets to evaluate different fusion methods. Dataset-1 comprises 10 MRI-PET test image pairs [28], while dataset-2 contains 269 image pairs [29]. From dataset-2, 245 pairs were used to train the learning-based models, and the remaining 24 image pairs were used for evaluation.
Fig 4 presents the test MRI and PET images [28] used for evaluation of various fusion methods. The fusion results of the proposed technique and other existing methods on MRI-PET testset-1 are displayed in Fig 5. The initial two images depict the MRI-PET source images, followed by the fusion results generated by EMD, CNN, LRD, NSCT, DTCWT, STAW, IMA, FCDFusion, CCDFuse, CMTFusion and our proposed technique. While most techniques excel in capturing structural details from the MRI, distinctions arise in edge extraction quality. Notably, EMD, CNN, LRD, NSCT, DTCWT, STAW, and IMA exhibit significant edge distortion in the fusion results, with varying degrees of local detail loss in the image. Although EMD and LRD successfully preserve edge information, they tend to overly enhance spatial structure in the MRI, leading to some original information loss. In contrast, our proposed fusion method excels in both fidelity and visual quality. It achieves higher definition in local details of the fused image while maintaining overall structural integrity.
Similarly, Figs 6–8 present fusion results of various techniques on MRI-PET testset-2, testset-3, and testset-4 respectively, providing a comparative analysis of the effectiveness of various fusion methods in capturing and preserving important information from the input datasets.
Table 1 presents a comparison of various image fusion methods based on different evaluation metrics for image pairs taken from [28]. Ideally, EN, PSNR, MI, and AG should have high values, indicating better image quality, information retention, and sharpness. QABF and SSIM should be close to 1, reflecting excellent structural and edge quality. Conversely, LABF and NABF should be close to zero, indicating minimal loss of edge information and artifacts introduced during the fusion process.
The results in Table 1 demonstrate that the proposed technique achieves superior performance compared to learning-based methods such as CNN, FCDFusion, and CCDFuse in terms of PSNR, which reflects its ability to preserve image quality with minimal distortion. While CMTFusion achieves a marginally higher PSNR, the proposed technique outperforms it in all other critical metrics except EN. This indicates that the proposed method not only maintains high-quality image fidelity but also excels in preserving structural details (QABF, LABF, NABF, and SSIM) and enhancing image sharpness (AG). To better demonstrate its effectiveness, we have separately averaged the “higher-value-better” and “lower-value-better” quality indices across all methods and included these values in Table 1. The results show that the proposed method exceeds the average performance in 6 out of 8 quality metrics, underscoring its robustness for MRI-PET image fusion.
The fusion results of various methods on two sample image pairs taken from [29] are displayed in Figs 9 and 10 respectively. Table 2 presents the comparison of evaluation results. The proposed technique performs consistently well across multiple metrics, demonstrating its effectiveness in MRI-PET fusion. It achieves a PSNR of 32.298, which is comparable to methods like NSCT and EMD. Although CMTFusion achieves the highest PSNR, the proposed method significantly outperforms CMTFusion in almost all other parameters, such as QABF, SSIM, and AG, making it a better overall choice.
Compared to learning-based methods like CNN, FCDFusion, and CDDFuse, the proposed technique shows superior performance in key metrics. It has a higher MI than CNN and better SSIM than FCDFusion and CNN, indicating its ability to retain critical information and structural details. While CDDFuse slightly outperforms the proposed technique in SSIM and AG, the proposed method provides a balanced trade-off across all metrics, including strong structural preservation and competitive sharpness. The proposed method outperforms the dataset-2 average in 6 out of 8 quality metrics, further highlighting its robustness and effectiveness.
When comparing the results from Table 1 and Table 2, the proposed technique consistently delivers balanced and robust performance across diverse datasets. While other methods may excel in specific metrics or datasets, the proposed technique ensures reliable fusion quality by maintaining a strong overall performance. This consistency across both datasets reinforces its suitability for MRI-PET fusion in practical medical applications.
Table 3 presents the run times (in seconds) of various methods averaged over 34 images, providing insights into the computational efficiency of each technique. The proposed method achieves an exceptionally low run time of 0.020 seconds, making it significantly faster than most other methods listed. Overall, the proposed technique not only delivers high-quality fusion results but also ensures fast processing times, making it an ideal choice for real-time applications where both performance and speed are critical.
To assess the effectiveness of the proposed method, we conducted an ablation study, summarized in Table 4. The study was divided into four phases to evaluate the contribution of each component. In the first phase, we used only the LP and mean fusion as a baseline, focusing on the multi-resolution decomposition of images. The second phase incorporated EMD with mean fusion, enabling finer decomposition of high-frequency components into IMFs and residues. The third phase replaced mean fusion with PC for fusing IMFs, emphasizing subtle intensity variations and structural details in high-frequency components. Finally, the fourth phase combined all components—EMD, PC, and the RGF. The RGF significantly enhanced edge preservation and minimized artifacts, resulting in the most robust fusion performance. This phase represents the full implementation of the proposed method.
5. Conclusion
In conclusion, the proposed MRI-PET fusion technique utilizing LP and EMD marks a significant advancement in oncology imaging. By effectively merging the high-resolution anatomical data from MRI with the functional insights from PET scans, this method enhances the diagnostic capabilities far beyond what is possible with traditional fusion techniques. Our comprehensive evaluation demonstrates that the method not only preserves essential structural and edge details but also significantly improves objective metrics such as PSNR, MI, and SSIM compared to existing methods. This approach sets a new standard for high-quality, efficient medical image processing, potentially transforming diagnostic procedures in oncology by providing clearer, more informative images that facilitate better patient outcomes.
Acknowledgments
The authors gratefully acknowledge the funding of the Deanship of Graduate Studies and Scientific Research, Jazan University, Saudi Arabia, through project number: RG24-S042.
References
- 1. Song J, Zheng J, Li P, Lu X, Zhu G, Shen P. An Effective Multimodal Image Fusion Method Using MRI and PET for Alzheimer’s Disease Diagnosis. Front Digit Health. 2021;3:637386. pmid:34713109
- 2. Wong TZ, Turkington TG, Hawk TC, Coleman RE. PET and brain tumor image fusion. Cancer J. 2004;10(4):234–42. pmid:15383204
- 3. Zhao L, Zhang Y, Dong L, Zheng F. Infrared and visible image fusion algorithm based on spatial domain and image features. PLoS One. 2022;17(12):e0278055. pmid:36584047
- 4. Kaur H, Koundal D, Kadyan V. Image Fusion Techniques: A Survey. Arch Comput Methods Eng. 2021;28(7):4425–47. pmid:33519179
- 5. Peng H, Li B, Yang Q, Wang J. Multi-focus image fusion approach based on CNP systems in NSCT domain. Computer Vision and Image Understanding. 2021;210:103228.
- 6. Li X, Guo X, Han P, Wang X, Li H, Luo T. Laplacian Redecomposition for Multimodal Medical Image Fusion. IEEE Trans Instrum Meas. 2020;69(9):6880–90.
- 7. Xie Q, Hu J, Wang X, Zhang D, Qin H. Novel and fast EMD-based image fusion via morphological filter. Vis Comput. 2022;39(9):4249–65.
- 8. Wei S, Ke W. A Multi-Focus Image Fusion Algorithm with DT-CWT. 2007 International Conference on Computational Intelligence and Security (CIS 2007). 2007:147–51.
- 9. Zhijun Wang, Ziou D, Armenakis C, Li D, Qingquan Li. A comparative analysis of image fusion methods. IEEE Trans Geosci Remote Sensing. 2005;43(6):1391–402.
- 10. Desale RP, Verma SV. Study and analysis of PCA, DCT & DWT based image fusion techniques. 2013 International Conference on Signal Processing , Image Processing & Pattern Recognition. 2013:66–9.
- 11. Li Y, Zhao J, Lv Z, Li J. Medical image fusion method by deep learning. International Journal of Cognitive Computing in Engineering. 2021;2:21–9.
- 12. Chen Q, Liu Y. Structure-Aware Feature Fusion for Unsupervised Domain Adaptation. AAAI. 2020;34(07):10567–74.
- 13. Liu Y, Chen X, Cheng J, Peng H. A medical image fusion method based on convolutional neural networks. 2017 20th International Conference on Information Fusion (Fusion). 2017:1–7.
- 14. Gao P, Jiang Z, You H, Lu P, Hoi SCH, Wang X, et al. Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019:6632–41.
- 15. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, et al. Recent advances in convolutional neural networks. Pattern Recognition. 2018;77:354–77.
- 16. Li Z, Liu F, Yang W, Peng S, Zhou J. A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Trans Neural Netw Learn Syst. 2022;33(12):6999–7019. pmid:34111009
- 17. Zhang H, Le Z, Shao Z, Xu H, Ma J. MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Information Fusion. 2021;66:40–53.
- 18. Hazirbas C, Ma L, Domokos C, Cremers D. FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture. 2017. pp. 213–228.
- 19. Zhang Q, Shen X, Xu L, Jia J. Rolling Guidance Filter. 2014. pp. 815–830.
- 20. Chen J, Zhang L, Lu L, Li Q, Hu M, Yang X. A novel medical image fusion method based on Rolling Guidance Filtering. Internet of Things. 2021;14:100172.
- 21. Suryanarayana G, Dhuli R. Edge preserving super-resolution algorithm using multi-stage cascaded joint bilateral filter. Int J Model Simul Sci Comput. 2017;08(01):1750003.
- 22. Burt P, Adelson E. The Laplacian Pyramid as a Compact Image Code. IEEE Trans Commun. 1983;31(4):532–40.
- 23. Lai W-S, Huang J-B, Ahuja N, Yang M-H. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017:5835–43.
- 24. Ghiasi G, Fowlkes CC. Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation. 2016. pp. 519–534.
- 25. Li H, Fu Y. FCDFusion: a Fast, Low Color Deviation Method for Fusing Visible and Infrared Image Pairs. 2024 [cited 13 Dec 2024]. Available: https://arxiv.org/abs/2408.01080v1
- 26. Zhao Z, Bai H, Zhang J, Zhang Y, Xu S, Lin Z, et al. CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023:5906–16.
- 27. Park S, Vien AG, Lee C. Cross-Modal Transformers for Infrared and Visible Image Fusion. IEEE Trans Circuits Syst Video Technol. 2024;34(2):770–85.
- 28. Gu X. Havard Medical Image Fusion Datasets CT-MRI PET-MRI SPECT-MRI. GitHub. [cited 12 Sep 2024] https://github.com/xianming-gu/Havard-Medical-Image-Fusion-Datasets
- 29. Gu X, Wang L, Deng Z, Cao Y, Huang X, Zhu Y. Adaptive spatial and frequency experts fusion network for medical image fusion. Biomedical Signal Processing and Control. 2024;96:106478.