Pyramid-based image empirical mode decomposition for the fusion of multispectral and panchromatic images
© Teo and Lau; licensee Springer. 2012
Received: 1 April 2011
Accepted: 9 January 2012
Published: 9 January 2012
Image fusion is a fundamental technique for integrating high-resolution panchromatic images and low-resolution multispectral (MS) images. Fused images may enhance image interpretation. Empirical mode decomposition (EMD) is an effective method of decomposing non-stationary signals into a set of intrinsic mode functions (IMFs). Hence, the characteristics of EMD may apply to image fusion techniques. This study proposes a novel image fusion method using a pyramid-based EMD. To improve computational time, the pyramid-based EMD extracts the IMF from the reduced layer. Next, EMD-based image fusion decomposes the panchromatic and MS images into IMFs. The high-frequency IMF of the MS image is subsequently replaced by the high-frequency IMF of the panchromatic image. Finally, the fused image is reconstructed from the mixed IMFs. Two experiments with different sensors were conducted to validate the fused results of the proposed method. The experimental results indicate that the proposed method is effective and promising regarding both visual effects and quantitative analysis.
The development of earth resources' satellites is mainly focus on improving spatial and spectral resolutions . As the spatial and spectral information are the two critical factors for enriching the capability of image interpretation, fusion of high spatial and high spectral images may increase the usability of satellite images. Most remote sensing applications, such as image interpretation and feature extraction, require both spatial and spectral information; therefore, the demands for fusing high-resolution multispectral (MS) images are increasing.
Currently, most optical sensors are capable of acquiring high spatial resolution panchromatic (Pan) and low spatial resolution MS bands simultaneously; for example, QuickBird, IKONOS, and SPOT series. Due to the technological constraints and costs, the spatial resolution of panchromatic images is better than the spatial resolution of MS images in an optical sensor. To overcome this problem, image fusion techniques (also called color fusion, pan sharpen, or resolution merge) are widely used to obtain a fused image with both high spatial and high spectral information.
The approaches of image fusion may be categorized into three types : projection-substitution, relative spectral contribution, and ARSIS (Amélioration de la Résolution Spatiale par Injection de Structures). Intensity-Hue-Saturation (IHS)  transform is one of the famous fusion algorithms using the projection-substitution method. This method interpolates MS image into the spatial resolution of a panchromatic image and converts the MS image according to intensity, hue, and saturation bands. The intensity of the MS image is then replaced with a high-spatial panchromatic image and reversed to red, green, and blue bands. However, this method is limited to three-band images.
The projection-substitution method also includes principle component analysis (PCA) , independent component analysis (ICA) , as well as other method. The PCA converts an MS image into several components based on eigen vectors and values. A high spatial panchromatic image replaces the first component of MS image with a large variance and performs the inverse PCA. The image fusion process is similar to the IHS method. Though this method is not constrained by the number of bands, significant color distortion may result.
The relative spectral contribution method utilizes the linear combination of bands to fuse panchromatic and MS images. Brovey transformation  is one of the well-known approaches in this category. The fused image is based on a linear combination of panchromatic and MS images.
The ARSIS is a multi-scale fusion approach, which improves spatial resolution by structural injection. This approach is widely used in image fusion because the advantage of multi-scale analysis may improve the fusion results. The multi-scale approach includes the Wavelet transform , empirical mode decomposition (EMD) , parameterized logarithmic image processing , as well as other methods. The Wavelet approach transforms the original images into several high and low frequency layers before replacing the high frequency of MS image with those that are from panchromatic image. Then, an inverse Wavelet transform is selected to construct the mixed layers for image fusion. A more detailed comparison among fusion methods is discussed in [10, 11].
The main difference between the Wavelet and EMD fusion approaches is depended on decomposition. The EMD method is an empirical method, which decomposes a nonlinear and non-stationary signal into a series of intrinsic mode functions (IMFs) . It is obtained from the signal by an algorithm called the "sifting process" produces a signal that obtains these properties. The EMD method is widely used in one-dimensional signal processing as well as in two-dimensional image processing. Wavelet decomposition is related to the predefine Wavelet basis while the EMD is a non-parametric data-driven process that is not required to predetermine the basis during decomposition. The EMD fusion approach is similar to the Wavelet fusion approach, in that it replaces the high frequency of MS images with those that are from panchromatic image.
The EMD can be applied in many image processing applications such as noise reduction [13, 14], texture analysis , image compression , image zooming , and feature extraction [18, 19]. Because the algorithm of image fusion via EMD is not yet mature, a small number of studies have reported on image fusion using EMD. Hariharan et al.  combined the visual and thermal images using the EMD method. First, the two-dimensional image is vectorized into a one-dimensional vector to fulfill the one-dimensional EMD decomposition. A set of weights are then multiplied by the number of IMFs. Finally, the weighted IMFs are combined to reconstruct the fused image. From the visual aspect, the experimental results show that the EMD method is better than Wavelet and PCA method. Liu et al.  used a bidimensional EMD method in image fusion; the results demonstrate that the EMD method may preserve both spatial and spectral information. The authors also indicated that the two-dimensional EMD is a highly time-consuming process.
Wang et al.  integrated QuickBird panchromatic and MS images using the EMD method. The row-column decomposition is selected to decompose the image in rows and columns separately using a one-dimensional EMD decomposition. The quantity evaluation demonstrates that the EMD algorithm may provide more favorable results when compared with either the IHS or Brovey method. Chen et al.  combined the Wavelet and EMD in the fusion of QuickBird satellite images. A similar row-column decomposition process is applied in the fusion process. The experiment also substantiates the promising result of the EMD fusion method.
EMD was originally developed to decompose one-dimensional data. Most EMD-based fusion methods use row-column decomposition schemes rather than two-dimensional decomposition. Because the image is two-dimensional, a two-dimensional EMD is more appropriate for image data processing. However, two-dimensional EMD decomposition has seldom been discussed in image fusion.
The sifting process of two-dimensional EMD is interactive, and involves three main steps: (1) determining the extreme points; (2) interpolating the extreme points for the mean envelope; and (3) subtracting the signal using the mean envelope. Determining the extreme points and the interpolation in two-dimensional space is considerably time consuming. Therefore, a new method to improve computation performance is necessary.
The objective of this study is to establish an image fusion method using a pyramid-based EMD. The proposed method reduces the spatial resolution of the original image during the sifting process. First, the proposed method determines and interpolates the extreme points of the reduced image. Then the results are expanded to obtain the mean envelope with identical dimensions to the original image.
The proposed method comprises three main steps: (1) the decomposition of panchromatic and MS images using pyramid-based EMD; (2) image fusion using the mixed IMFs of panchromatic and MS images; and (3) quality assessment of the fused image. The test data include SPOT images of a forest area and QuickBird images of a suburban area. The quality assessment considers two distinct aspects: the visual and quantifiable. Fusion results of the modified IHS, PCA, and wavelet methods are also provided for comparison.
This study establishes a novel image fusion method using a pyramid-based EMD. The proposed method can improve the computational performance of two-dimensional EMD in image fusion, and can also be applied to EMD-based image fusion. The major contribution of this study is the improvement of the computational performance of two-dimensional EMD using image pyramids. The proposed method extracts the mean envelope of the coarse image, and resamples the mean envelope to equal the original size during the sifting process. The benefits of the proposed method are reduced computation time for extreme point extraction and interpolation.
This article is organized as follows. Section 2 presents the proposed pyramid-based EMD fusion method. Section 3 shows the experimental results from using different image fusion methods. This study also compares and discusses one- and two-dimensional EMD in image fusion. Finally, a conclusion is presented in Section 4.
2. The proposed scheme
This section introduces the basic ideas and procedures of one-dimensional EMD and row-column EMD. One-dimensional EMD can be extended to two-dimensional EMD before determining the technical details of pyramid-based two-dimensional EMD. This section describes EMD-based image fusion in the final part.
2.1. One-dimensional EMD
EMD is used to decompose signals into limited IMFs. An IMF is defined as a function in which the number of extreme points and the number of zero crossings are the same or differ by one . The IMFs are obtained through an iterative process called the sifting process. A brief description of the sifting process is shown below.
Step 1. Determine the local maxima and minima of the current input signal h(i, j)(t), where i is the number of the IMF and j is the number of iteration. In the first iteration, h(1,1)(t) is the original time series signal X(t).
Step 2. Compute the upper and lower envelopes u(i, j)(t) and 1(i, j)(t) by interpolating the local minimum and maximum using the cubic splines interpolation.
2.2. Row-column EMD
EMD was originally developed to manage one-dimensional data. To apply this method to two-dimensional data, a row-column EMD  is proposed based on one-dimensional EMD. The purpose of row-column EMD is to perform EMD on the rows and columns. This method determines and interpolates the extreme points of the one-dimensional space. The row-column EMD process is briefly described below.
2.3. Pyramid-based EMD
This study proposed pyramid-based EMD to avoid the striping effect of row-column EMD. Two-dimensional EMD determines and interpolates the extreme points of a two-dimensional space rather than one-dimensional space. The main difference between pyramid-based and row-column EMD is the generation of a mean envelope. The additional image pyramid improves the computation performance of two-dimensional EMD. The process of pyramid-based two-dimensional EMD is described below.
Step 1. Reduce the input image from h(i, j)(p, q) to h(i, j)(p g ,q g ) using Gaussian image pyramid , where i is the number of the IMF; j is the number of the iteration and g is the number of pyramid layer. In the first iteration, h(1,1)(p g ,q g ) is the original reduced image X(p g ,q g ). The reduced scale is related to the smoothness of the input image and EMD computation time.
Step 4. Expand the mean envelope to the original image size m(i, j)(p, q).
Step 5. Subtract the h(i, j)(p, q) by the mean envelope to obtain the sifting result h(i, j+1)(p, q), as shown in (5). If m(i, j)(p, q) < ε, then h(i, j+1)(p, q) is IMF i (p, q). Subtract the original image by this IMF i (p, q) to obtain residual r i (p, q). If m(i, j)(p, q) > ε, then h(i, j+1)(p, q) is treated as the input data and Step 1 is repeated. The procedure will be terminated when r i (p, q) < ε.
2.4. EMD-based image fusion
Because only the high-frequency IMF was changed from a panchromatic to a MS image, the remaining IMFs will not affect the image fusion results. Thus, the decomposition process can be simplified. This study only decomposes the image into two IMFs, high- and low-frequency, for image fusion. The EMD image-fusion process is described as follows: For data preprocessing, the panchromatic and MS images are registered into the same system. Next, the MS image is resampled to match the size of the panchromatic image. Then, the method proposed by this study uses EMD to decompose the two images into several IMFs and a residual. The first IMF of the panchromatic image replaces the first IMF of the MS image. Finally, the fused image is obtained by reconstructing the mixed IMFs of the MS image. The reconstruction process combines the mixed IMFs and residuals, as shown in Equation 6.
3. Experimental results
Related information of test data
Gray level (bits)
Test area (m*m)
Spatial resolution (m)
Spatial resolution (m)
G, R, NIR
G, R, NIR
The quality assessment includes the visual and quality aspects. Regarding the visual aspect, the fused and the original MS images are visually compared. Both row-column and pyramid-based EMD are applied during image fusion to enable a comparison. In addition, this study employed the commercial software ERDAS Imagine 2010 to fuse the images using different methods, including modified IHS , PCA, and Wavelet. These images were then compared with the image fused using the EMD method.
The experiment required establishing a number of parameters. Because the purpose of EMD is image fusion, the image was only decomposed into two components: high-frequency and a remainder layer. The stopping criterion is 99% or a mean envelope less than 2 pixels. The image pyramid scales are reduced layers 1 and 2. The experiment results are discussed in the following section. The window of openness is 5-15 pixels in different iterations. Both the threshold of the minimal points for positive openness and threshold of the maximum points for negative openness were less than 75 degrees.
3.1. Quality evaluation of the fused image
The quality assessment considers both the visual and quantifiable aspects, and refers to both spatial and spectral qualities. In other words, the fusion method should improve the spatial resolution and preserve spectral content. Several indices are selected to evaluate the quality of a fused image. The experiment compares the fused image with the original MS image to ensure spectral fidelity. The three spectral indices are RMSE , ERGAS , and the correlation coefficient. Spatial index is the entropy of an image.
3.1.1. Root mean square error (RMSE)
where Bias is the difference between mean value of MS and fused images, SDD is the standard deviation of difference between MS and fused images.
3.1.2. Erreur relative globale adimensionnelle de synthèse (ERGAS)
where h and l are the resolution of PAN and MSI, respectively. N is the number of spectral band (B i ). M is the mean value of each spectral band.
3.1.3. Correlation coefficient
where C is the coefficient of correlation, F(i, j) and M(i, j) are the gray value of the fused and MS images, respectively. μ F is the mean of fused image, μ M is the mean of MS image, and m and n are the image sizes.
where E is the Entropy, P k is the probability of gray value k in the image.
3.2. Case I
Statistical information of SPOT image.
Pyramid EMD Reduce 1
Pyramid EMD Reduce 2
3.3. Case II
Statistical information of QuickBird image
Pyramid EMD Reduce 1
Pyramid EMD Reduce 2
This article proposes an EMD-based image fusion method using image pyramids. The proposed method uses image pyramids to improve the computation performance of two-dimensional EMD. An openness strategy during the extraction of extreme points of two-dimensional EMD is also proposed. This experimental study uses SPOT and QuickBird images in distinct areas to evaluate the proposed method, and compare the results with other fusion approaches. The experimental results demonstrate an improvement of pyramid-based two-dimensional EMD. Used during image decomposition, this method may overcome the linear effect of row-column EMD. In addition, the proposed method is sensor-independent but can be applied to the integration of heterogeneous sensors, such as optical and radar images.
This study was supported in part by the Industrial Technology Research Institute of Taiwan and National Science Council of Taiwan under Project NSC 99-2221-E-009-131. The author would like to thank the Center for Space and Remote Sensing Research at National Central University in Taiwan for providing the test data sets.
- Li Z, Chen J, Baltsavias E: Advances in Photogrammetry, Remote Sensing and Spatial Information Sciences: 2008 ISPRS Congress Book. Taylar & Francis Group; 2008.Google Scholar
- Wald L, Ranchin T, Mangolini M: Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images. Photogram Eng Remote Sens 1997, 63(6):691-699.Google Scholar
- Carper WJ, Lillesand TM, Kiefer RW: The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data. Photogram Eng Remote Sens 1990, 56(4):459-467.Google Scholar
- Chavez JPS, Sides SC, Anderson JA: Comparison of three difference methods to merge multiresolution and multispectral data: landsat TM and SPOT panchromatic. Photogram Eng Remote Sens 1991, 57(3):295-303.Google Scholar
- Zhang G, Wang L, Zhang H: A fusion algorithm of high spatial and spectral resolution images based on ICA. Int Arch Photogram Remote Sens Spatial Inf Sci 2008, XXXVII(B7):1295-1300.Google Scholar
- Pohl C, van Genderan JL: Multisensor image fusion in remote sensing: concepts, methods and application. Int J Remote Sens 1998, 19: 823-854.View ArticleGoogle Scholar
- Jorge N, Xavier O, Octavi F, Albert P, Vicenc P, Roman A: Multiresolution-based imaged fusion with additive wavelet decomposition. IEEE Trans Geosci Remote Sens 1999, 37(3):1204-1211.View ArticleGoogle Scholar
- Wang J, Zhang J, Liu Z: EMD Based multi-scale model for high resolution image fusion. Geospatial Inf Sci 2008, 11(1):31-37.View ArticleGoogle Scholar
- Nercessian SC, Panettar KA, Agaian SS: Multiresolution decomposition schemes using the parameterized logarithmic image processing model with application to image fusion. EURASIP J Adv Signal Process 2011, 2011: 17. (Article ID 515084)View ArticleGoogle Scholar
- Thomas C, Ranchin T, Wald L, Chaunussot J: Synthesis of multispectral images to high spatial resolution: a critical review of fusion methods based on remote sensing physics. IEEE Trans Geosci Remote Sens 2088, 46(5):1301-1312.View ArticleGoogle Scholar
- Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM: Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Trans Geosci Remote Sens 2007, 45(10):3012-3021.View ArticleGoogle Scholar
- Huang NE, Shen Z, Long SR, Wu ML, Shih HH, Zheng Q, Yen NC, Tung CC, Liu HH: The empirical mode decomposition and the Hilbert spectrum for nonlinear and nonstationary time series analysis. Proc R Soc Lond 1998, A 454: 903-995.MathSciNetView ArticleMATHGoogle Scholar
- Han C, Guo H, Wang C, Fan D: A novel method to reduce speckle in SAR images. Int J Remote Sens 2002, 23(23):5091-5101.Google Scholar
- Bernini MB, Federico A, Kaufmann GH: Noise reduction in digital speckle pattern interferometry using bidimensional emplirical mode decompostion. Appl Opt 2008, 47(14):2592-2598.View ArticleGoogle Scholar
- Nunes JC, Bouaoune Y, Delechelle E, Niang O, Bunel Ph: Image analysis by bidimensional emirical mode decopostion. Image Vis Comput 2003, 21: 1019-1026.View ArticleMATHGoogle Scholar
- Linderhed A: Image Empirical Mode Decomposition: a new tool for image processing. In The First International Conference on The Advances of Hilbert-Huang Transform and It's Application. Jhung-Li, Taiwan; 2006.Google Scholar
- Tian Y, Huang Y, Li Y: Image zooming method using 2D EMD technique. Proceeding of IEEE the 6th World congress on Intellligent Control and Automation 2006, 2: 10036-10040.View ArticleGoogle Scholar
- Ayenu-Prah A, Attoh-Okine N: Evaluating pavement cracks with bidimensional empirical mode decomposition. EURASIP J Adv Signal Process 2008, 2008: 7. (Article ID 861701)View ArticleMATHGoogle Scholar
- Khan JF, Barner K, Adhami R: Feature point detection utilizing the empirical mode decomposition. EURASIP J Adv Signal Process 2008, 2008: 13. (Article ID 287061)View ArticleMATHGoogle Scholar
- Hariharan H, Gribok A, Abidi M, Koschan A: Image fusion and enhancement via empirical mode decomposition. J Pattern Recogn Res 2006, 1(1):16-32.View ArticleGoogle Scholar
- Liu Z, Song P, Zhang J, Wang J: Bidimensional empirical mode decomposition for the fusion of multispectral and panchromatic images. Int J Remote Sens 2007, 28(18):4081-4093.View ArticleGoogle Scholar
- Chen S, Su y, Zhang R, Tian J: Fusing remote sensing images using a trous wavelet transform and empirical mode decomposition. Pattern Recogn Lett 2008, 29: 330-342.View ArticleGoogle Scholar
- Richards JA, Jia X: Remote Sensing Digital Image Analysis: An Introduction. 3rd edition. Springer; 1999.View ArticleGoogle Scholar
- Yokoyama R, Shirasawa M, Pike R: Visualizing topograhpy by openness: a new application of image processing to digital elevation methods. Photogram Eng Remote Sens 2002, 68(3):257-265.Google Scholar
- Nunes JC, Delechelle E: Empirical mode decompostion: applications on signal and image processing. Adv Adapt Data Anal 2008, 1(1):125-175.View ArticleGoogle Scholar
- Siddiqui Y: The modified IHS method for fusing satellite imagery. In Proceedings of ASPRS Annual Conference. Reno Nevada, USA; 2006.Google Scholar
- Otazu X, Gonzáles-Audícana M, Fors O, Nunez J: Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods. IEEE Trans Geosci Remote Sens 2005, 43(10):2379-2385.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.