Multifocus image fusion scheme based on feature contrast in the lifting stationary wavelet domain

Abstract

For fusion of multifocus images, a novel image fusion method based on multiscale products in lifting stationary wavelet (LSWT) domain is proposed in this article. In order to avoid the influence of noise and select the coefficients of the fused image properly, different subband coefficients employ different selection principles. For choosing the low frequency subband coefficients, a new modified energy of Laplacian (EOL) is proposed and used as the focus measure to select the coefficients from the clear parts of the low frequency subband images; when choosing the high frequency subband coefficients, a novel feature contrast measurement of the multiscale products is proposed, which is proved to be more suitable for fusion of multifocus images than the traditional contrast measurement, and used to select coefficients from the sharpness parts of the high frequency subbands. Experimental results demonstrate that the proposed fusion approach outperforms the traditional discrete wavelet transform (DWT)-based, LSWT-based and LSWT-traditional-contrast-(LSWT-Tra-Con)-based image fusion methods, even though the source images are corrupted by a Gaussian noise, in terms of both visual quality and objective evaluation.

1. Introduction

In applications of digital cameras, when a lens focuses on a subject at a certain distance, all subjects at that distance are sharply focused. Subjects not at the same distance are out of focus and theoretically are not sharp. It is often not possible to get an image that contains all relevant objects in focus. One way to overcome this problem is image fusion, in which one can acquire a series of pictures with different focus settings and fuse them to produce an image with extended depth of field [1–3]. During the fusion process, all the important visual information found in the input images must be transferred into the fused image without introduction of artifacts. In addition, the fusion algorithm should be reliable and robust to imperfections such as noise or mis-registration [4–6].

During the last decade, a number of techniques for image fusion have been proposed. A simple image fusion method consists in taking the average of the source images pixel by pixel. However, along with simplicity comes several undesired side effects including reduced contrast. In recent years, many researchers have recognized the multiscale transforms (MST) are very useful for image fusion, and various MST-based fusion methods have been proposed [7–11]. In MST domain, the discrete wavelet transform (DWT) becomes the most popular and important multiscale decompositions method in image fusion. Compared with the Laplacian pyramid transform, the DWT has been found to have some advantages such as: (1) The DWT cannot only possess localization but also provide directional information, while the pyramid representation fails to introduce any spatial orientation selectivity into the decomposition process [9]. So DWT can represent the underlying information of the source images more efficiently. This advantage would make the fused image more accurate. (2) No blocking artifacts, which often occur in Laplacian pyramid-fused images, can be observed in the DWT-based fused images. (3) DWT-based fusion has better signal-to-noise ratios than Laplacian-based fusion [12]. (4) DWT-based fusion images can improve the perception over pyramid-based fusion images. More advantages of the DWT over Laplacian pyramid scheme can be seen in [9, 12].

However, the DWT has its own disadvantages. It needs a great deal of convolution calculations, and it either consumes much time or occupies memory resources, which impedes its real-time application. Relative to the DWT, the lifting wavelet transform (LWT) [13] can overcome its shortcomings. Unfortunately, the original LWT and DWT lack shift-invariance and cause pseudo-Gibbs phenomena around singularities [14], which will reduce the resultant image quality. Thus, new lifting stationary wavelet transform (LSWT) [15], as a fully shift-invariant form of LWT, can be introduced and used as the MST method in this article.

Except for the LSWT discussed in the above paragraph, the nonsubsampled contourlet transform (NSCT) [16], which also possesses the shift-invariant, is another important MST method in image fusion field. Compared with the LSWT, the NSCT is built upon non-subsampled pyramids and nonsubsampled directional filter banks [16]. In NSCT, the non-subsampled pyramids are first used to achieve the multi-scale decomposition, and then the nonsubsampled directional filter banks are employed to achieve the direction decomposition. The number of direction decomposition at each level can be different, which is much more flexible than the three directions in wavelet. So it can get better fusion results than the LSWT. However, the NSCT is more time consuming than the LSWT because of its multi-direction and complexity, which impede its real-time application greatly. By considering both the fusion results and computing complexity, in our proposed method, the LSWT is used as the MST method.

In image fusion algorithm in MST domain, one of the most important things for improving fusion quality is the selection of fusion rules, which influences the performance of fusion algorithm remarkably. According to physiological and psychological research, the human vision system (HVS) is highly sensitive to the local image contrast level. To meet this requirement, Toet and Ruyven [17] developed the local luminance contrast in their research in contrast pyramid (CP). In the local luminance contrast, the contrast level is measured by measuring the ratio of the high frequency component of image to the local luminance of the background.

Based on the idea of [17], many different forms of contrast measurement have been proposed and successfully used in image fusion [18, 19]. However, in these contrast measurements, the value (or absolute value) of a single pixel of the high frequency subband in MST domain is directly used as the strength value of the high frequency component. In fact, the value (or absolute value) of a single pixel of the high frequency subband is very limited in determining which pixel is from the clear part of the sub-images. So, a pure use of a single pixel as the high frequency component in the local contrast measurements is not ideal. In addition, almost all the MST-based image fusion algorithms do not consider the noise influence. In many practical applications, additive Gaussian noise, which is characterized by adding to each image pixel a value from a zero-mean Gaussian distribution, can be systematically introduced into image during acquisition. This noise may cause miscalculation of sharpness values, which in turn, degrade the performance of image fusion. To be useful in real process operation, the fusion algorithm should provide pleasing fusion performance for the clean image fusion; meanwhile it should be reliable and robust to imperfections such as noise.

It is well known that there exist dependencies between wavelet coefficients. If a wavelet coefficient produced by a true signal is of large magnitude at a finer scale, its parents at coarser scales are likely to be large as well. However, for those coefficients caused by noise, the magnitudes will decay rapidly along the scales. So, multiplying the adjacent wavelet scales, namely multiscale products (MSP), can sharpen the important structures while weakening noise [20, 21]. Therefore, multiscale products can distinguish edge structures from noise more effectively.

To make up for the aforementioned deficiencies of the traditional MST-based image fusion methods, we present a new multifocus image fusion scheme which incorporates the merits of interscale dependencies into the image fusion field. In this method, after decomposing the original images using the LSWT, we use a new modified energy of Laplacian, which can reflect features of the edges of the low frequency subimage in LSWT domain, as the focus measure to select the coefficients of the fused image; when choosing the high frequency subband coefficients, a novel local neighborhood feature contrast of the multiscale products, which can effectively represent the salient features and sharp boundaries of image, is developed and used as the measurement to select coefficients from the clear parts of source images. The experimental results show that the proposed method does well in fusion of multifocus images no matter they are clean or not, and outperforms typical wavelet-based, LSWT-based, NSCT-based and LSWT typical contrast-based fusion algorithms in terms of objective criteria and visual appearance.

The article is organized as follows. In Sections 2 and 3, the theory of LSWT and multiscale products are introduced respectively in detail; Section 4 describes the image fusion algorithm using LSWT and multiscale products. Section 5 compares the performance of the new algorithm with the performance of other conventional fusion techniques applied to sequences of multifocus test images. Finally, in Section 6 we conclude the article with a short summary.

2. Lifting stationary wavelet transform

2.1. Lifting wavelet transform

Lifting wavelet transform (LWT), proposed by Sweldens [22], is a new wavelet construction method using the lifting scheme in time domain. The main feature of the LWT is that it provides an entirely spatial domain interpretation of the transform, as opposed to the traditional frequency domain based constructions. It abandons the Fourier transform as design tool for wavelets, and wavelets are no longer defined as translates and dilates of one fixed function. Compared with the classical wavelet transform, the LWT requires less computation and memory, and can produce integer-to-integer wavelet transform. It is always perfectly reconstructed no matter how the prediction operator and update operator are designed. Moreover, it possesses several advantages, including possibility of adaptive and nonlinear design, in place calculations, and so on [13, 22, 23]. The decomposition stage of LWT consists of three steps: split, prediction and update.

In the split step, the original signal (or approximate signal) a _l at level l is split into even samples and odd samples

$$a_{l+1}=a_l\left(2i\right),d_{l+1}=a_l\left(2i+1\right)$$

(1)

In the prediction step, we apply a prediction operator P on a_l+1to predict d_l+1. The resultant prediction error d_l+1is regarded as the detail signal of a _l

$$d_{l+1}\left(i\right)=d_{l+1}\left(i\right)-\sum _{r=-M/2+1}^{M/2}{p}_r{a}_{l+1}\left(i+r\right)$$

(2)

where p _r is one of the coefficient of P and M is the length of prediction coefficients.

In the update step, an update of even samples a_l+1is accomplished by using an update operator U to detail signal d_l+1and adding the result to a_l+1, the resultant a_l+1can be regarded as the approximation signal of a _l .

$$a_{l+1}\left(i\right)=a_{l+1}\left(i\right)+\sum _{j=-N/2+1}^{N/2}{u}_j{d}_{l+1}\left(i+j-1\right)$$

(3)

where u _j is the coefficient of U and N is the length of update coefficients.

Let a _l be the input signal for lifting scheme, the detail and approximation signals at the lower resolution level can be obtained by iterating of the above three steps on the output a.

The inverse LWT can be performed by reversing the prediction and update operators and changing each '+' into '-' and vice versa. The complete expression of the reconstruction of LWT is shown in Equations (4)-(6). Figure 1 depicts the structure of LWT. The computational costs of the forward and inverse transform are exactly the same. The prediction operator P and update operator U can be designed by the interpolation subdivision method introduced in [23]. Choosing different P and U is equivalent to choosing different biorthogonal wavelet filters [24].

Figure 1

Structure of LWT, both analysis side (left part) and synthesis side (right part).

$$a_{l+1}\left(i\right)=a_{l+1}\left(i\right)-\sum _{j=-N/2+1}^{N/2}{u}_j{d}_{l+1}\left(i+j-1\right)$$

(4)

$$d_{l+1}\left(i\right)=d_{l+1}\left(i\right)+\sum _{r=-M/2+1}^{M/2}{p}_r{a}_{l+1}\left(i+r\right)$$

(5)

$$a_l\left(2i\right)=a_{l+1},a_l\left(2i+1\right)=d_{l+1}$$

(6)

2.2. Lifting stationary wavelet transform

In the LWT, the shift-invariance is not ensured because there exists the split step and the length of approximation signal and detail signal decreases. However, the shift-invariance is desirable in many image applications such as image enhancement, image denoising and image fusion. In order to obtain the LSWT which possesses the shift-invariance, the method of literature [15] is adopted in this article.

In the LSWT, the split step is discarded. Assuming P^land U^lrepresent the prediction and update operator of the lifting stationary wavelet at level l, respectively. The initial prediction operator P⁰ and initial update operator U⁰ can be obtained once M and N are determined, where P⁰ = {p _m }, m = 0,1,..., M - 1; U⁰ = {u _n }, n = 0,1,..., N - 1. The coefficients of P^land U^lare designed by padding P⁰ and U⁰ with zeros [15]. The prediction coefficients and update coefficients at level l in the LSWT are expressed as follows:

$$p_i^l=p_0^0,\underset{2^l-1}{\underbrace{0\dots 0}},p_1^0,\underset{2^l-1}{\underbrace{0\dots 0}},p_2^0,\dots ,p_{M-2}^0,\underset{2^l-1}{\underbrace{0\dots 0}},p_{M-1,}^0$$

(7)

$$u_j^l=u_0^0,\underset{2^l-1}{\underbrace{0\dots 0}},u_1^0,\underset{2^l-1}{\underbrace{0\dots 0}},u_2^0,\dots ,u_{N-2}^0,\underset{2^l-1}{\underbrace{0\dots 0}},u_{N-1}^0,$$

(8)

The decomposition results of an approximation signal a _l at level l via lifting stationary wavelet are expressed by following equations.

$$d_{l+1}=a_l-P^{l+1}{a}_l,a_{l+1}=a_l+U^{l+1}{d}_{l+1}$$

(9)

where d_l+1and a_l+1are detail signal and approximation signal of a _l at level l + 1.

The reconstruction procedure of LSWT is directly achieved from its forward transform, which is expressed as below.

$$a_l=\frac{1}{2}\left(a_{l+1}-U^{l+1}{d}_{l+1}+d_{l+1}+P^{l+1}\left(a_{l+1}-U^{l+1}{d}_{l+1}\right)\right)$$

(10)

The forward and inverse transform of LSWT is shown in Figure 2.

Figure 2

The forward and inverse transform of LSWT.

Compared with the DWT, LSWT do not downsample and upsample the highpass and the lowpass coefficients during the decomposition and reconstruction of the image. So, the LSWT not only retains the perfect properties of the LWT, but also possess the shift-invariance. When LSWT is introduced into image fusion, more information for fusion can be obtained. In addition, the size of different sub-images is identical, so it is easy to find the relationship among different subbands, which is beneficial for designing fusion rules [25]. Therefore, the LSWT is more suitable for image fusion.

3. Multiscale products of LSWT

In MST-based image fusion algorithms, almost all the schemes design the fusion rule, namely, selection principles for high frequency subband coefficients (simplified into 'fre-coefs' in figures in this article) based on the wavelet coefficients directly. It is worth noting that much of the noise is also related to high frequencies. As a result, the fused images obtained by these methods are more noisy than the source images. It is well known that there exist dependencies between wavelet coefficients: if a coefficient at a coarser scale has small magnitude, its descendant coefficients at finer scales are likely to be small and vice versa. If two adjacent wavelet subbands are multiplied it can amplify the significant features and dilute noise [21, 26].

Suppose f(x) is a one-dimensional (1-D) discrete signal, we define the multiscale products of W _l f as

$$P_{l}f\left(x\right)=\prod _{k=-k_1}^{k_2}{W}_{l+k}f\left(x\right)$$

(11)

where k₁ < l and k₂ ≤ L - l are nonnegative integers if we use L to denote the max level, W _l f (x) denotes the LSWT of signal f (x) at scales l and position x.

The support of an isolated edge will increase by a factor of two across scale and the neighboring edges will interfere with each other at coarse scale. So in practice it is sufficient to implement the multiplication at two adjacent scales [20]. If we let k₁ = 0 and k₂ = 1, then we calculate the LSWT scale products as

$$P_{l}f\left(x\right)=W_{l}f\left(x\right)\cdot W_{l+1}f\left(x\right)$$

(12)

According to [7] and Equation (12), for 2D image f, the multiscale products at the l th scale, d th direction and location (x, y) can be defined as

$$P_l^d\left(x,y\right)=W_l^{d}f\left(x,y\right)\cdot W_{l+1}^{d}f\left(x,y\right)$$

(13)

where d = 1, 2, 3 denote the horizontal, vertical and diagonal directions.

To demonstrate the merits of multiscale products of LSWT, in Figure 3, the LSWT and multiscale products of a noisy test image (f₁ = g_l + δ) are illustrated, respectively. Though the LSWT coefficients of the original signal g₁ are immersed into noise at fine scales, they are enhanced in the scale products P _l f. The significant features of g₁ are more distinguishable in P _l f than in W _l f. So we can conclude that the multiscale production of LSWT can amplify the significant features and dilute noise.

Figure 3

The noisy test image 'Pepper' (variance δ = 0.01), the HL subimages of LSWT and the multiscale products at the first three scales: (a) noisy image 'Pepper'; (b)-(e) HL1, HL2, HL3, HL4 high frequency sub-images of 'Pepper'; (f)-(h) multiscale products at the first three scales.

4. The proposed fusion algorithm

A good image fusion algorithm should preserve all the salient features of the source images and introduce as less artifacts or inconsistency as possible. In addition, the fusion algorithm should be reliable and robust to imperfections such as noise. In this article, we develop a novel multifocus image fusion scheme to incorporate the merits of interscale dependencies of LSWT into the image fusion technique. Two adjacent wavelet subbands are multiplied to amplify the significant features and dilute noise. In contrast to the conventional MST-based image fusion schemes, we design the fusion rule of the high frequency subbands based on the multiscale products instead of the wavelet coefficients. So our proposed image fusion method can be fairly resistant to the noise because the multiscale products can distinguish edge structures from noise effectively.

Apart from the LSWT and multiscale products in the above section, fusion rules, namely, selection principles for different subband coefficients are another important component in our proposed fusion method. The following study presented in this article is concerned with a design of novel fusion rules for the low frequency subband coefficients and the high frequency subband coefficients. Throughout this study, it has been assumed that the images studied have been appropriately pre-registered, so that corresponding features can coincide pixel to pixel [27]. To simplify the discussion, we assume the fusion process is to generate a composite image F from a pair of source images denoted by A and B. The general procedure of the proposed LSWT-MSP-based fusion algorithm is illustrated in Figure 4 and implemented as

Figure 4

Schematic diagram of LSWT-MSP-based fusion algorithm.

• (1) Decompose the registered source images A and B, respectively, into one low frequency subimage and a series of high frequency subimages via LSWT.

• (2) Select fusion coefficients for the low frequency subimage and each high frequency subimage from A and B according to fusion rules.

• (3) Reconstruct the original image based on the new fused coefficients of subimages by taking an inverse LSWT transform, then the fused image F is obtained.

4.1. Selection of lowpass subband coefficients

As the coefficients in the coarsest scale subband represent the approximation component of the source image, the simplest way is to use the conventional averaging method to produce the composite coefficients. However, this will reduce the fused image contrast. To improve the fused image quality, a clarity measure should be defined to determine whether a coefficient of the low frequency subband is in focus or out of focus.

For multifocus image fusion, many typical focus measurements, e.g. variance, energy of image gradient (EOG), spatial frequency (SF), and energy of Laplacian (EOL) of the image, are compared in literature [28]. They all measure the variation of pixels. Pixels with greater values of these measurements, when source images are compared with each other, are considered from the focus parts. According to literatures [28, 29] we know that EOL can provide a better performance than SF and EOG for fusion multifocus images. In this article, we use a new improved energy of image Laplacian (IEOL) as the focus measure to select coefficients from the clear parts of the source images.

The complete original expression of the energy of Laplacian (EOL) of the image f is shown in Equation (14):

$$\mathsf{\text{EOL}}=\sum _x\sum _y{\left(f_{xx}+f_{yy}\right)}^2$$

(14)

where

$$\begin{array}{ll}\hfill f_{xx}+f_{yy}& =-f\left(x-1,y-1\right)-4f\left(x-1,y\right)-f\left(x-1,y+1\right)-4f\left(x,y-1\right)+20f\left(x,y\right)\\ -4f\left(x,y+1\right)-f\left(x+1,y-1\right)-4f\left(x+1,y\right)-f\left(x+1,y+1\right)\end{array}$$

(15)

In Equation (15), the f (x, y) is the gray value of pixel at position (x, y) of image f ⋅ f _xx + f _yy represents image gradient obtained by Laplacian operator [-1,-4,-1; -4, 20, -4; -1, -4, -1].

However, the second derivatives in different directions may have different signs which cause one sign to cancel the other. This phenomenon may occur frequently in the textured images. In order to avoid the problem, and maintain robustness of the algorithm in the face of adverse effects that may occur in image fusion. We will use an improved EOL (IEOL) as the clarity measure to select coefficients from the clean parts of source images.

The improved sum of Laplacian (ISL) and the improved energy of Laplacian (IEOL) of image f are computed as:

$$\begin{array}{c}\text{ISL(}x,y\text{)}\text{=}\left|8f(x,y)-4f(x-1,y)-4f(x+1,y)\right|\\ +\left|8f(x,y)-4f(x,y-1)-4f(x,y+1)\right|\\ +\left|2f(x,y)-f(x-1,y+1)-f(x+1,y-1)\right|\\ +\left|2f(x,y)-f(x-1,y-1)-f(x+1,y+1)\right|\end{array}$$

(16)

$$\mathsf{\text{IEOL}}\left(x,y\right)=\sum _a\sum _b{W}_l\left(a,b\right){\left(ISL\left(x+a,y+b\right)\right)}^2$$

(17)

where W _l is a template which size is relatively small, and must satisfy the normalization rule Σ _a Σ _b W _l (a, b) = 1. For the low frequency subband, it contains low frequency information. In order to match the information of the LSWT neighborhood of low frequency subband, the values of center and center neighborhood of the template should have little change between each other [30]. In this article, the template size is 3 × 3. In order to highlight the center pixel of the window, a weighted template is used, which is given as:

$$W_l\left(a,b\right)=\frac{1}{15}\left[\begin{array}{ccc}1\hfill & \hfill 2\hfill & \hfill 1\hfill \\ 2\hfill & \hfill 3\hfill & \hfill 2\hfill \\ 1\hfill & \hfill 2\hfill & \hfill 1\hfill \end{array}\right]$$

The IEOL can be used as clarity measure to determine which coefficient is in focus. Suppose the source images A and B are decomposed using LSWT, $f_l^L\left(x,y\right)$ and $f_F^L\left(x,y\right)$ denote the low frequency coefficients of the source image L (K = A, B) and the fused image F which are located at (x, y) in the L th decomposition level, respectively. The ${\mathsf{\text{IEOL}}}_K^L\left(x,y\right)$ denotes the IEOL measurement of $f_K^L\left(x,y\right)$. The proposed IEOL-based fusion rule can be described as follows:

$$f_F^L\left(x,y\right)=\left\{\begin{array}{c}{f}_A^L\left(x,y\right)if:{\mathsf{\text{IEOL}}}_A^L\left(x,y\right)>{\mathsf{\text{IEOL}}}_B^L\left(x,y\right)\hfill \\ f_B^L\left(x,y\right)if:{\mathsf{\text{IEOL}}}_A^L\left(x,y\right)\le {\mathsf{\text{IEOL}}}_B^L\left(x,y\right)\hfill \end{array}\right.$$

(18)

It means that coefficients with maximum IEOL measurement are selected as the coefficient of the fused image when subbands are compared in the LSWT domain. For simplicity, we name this fusion rule as 'IEOL-max' rule in this article.

4.2. Selection of bandpass subband coefficients

The coefficients in the high frequency subbands represent the detailed component of the source image. In traditional multiresolution fusion algorithms, such as [9, 31, 32], the multiresolution coefficients with larger absolute value are considered as sharp brightness changes or salient features in the corresponding source image, such as the edges, contours, and region boundaries, and so on. Thus, for the high frequency subbands coefficients, the most commonly used selection principle is the 'absolute-maximum-choosing' scheme (simplified and named 'Coef-abs-max') without taking any consideration of lowpass subband coefficients, that is, all the information in the lowpass subband is neglected.

Furthermore, in many practical applications, images are distorted by noise during the acquisition or transmission process. But almost all the traditional MST-based image fusion algorithms are designed to transfer the high frequency information from the input images to the fused image. It is worth noting that much of the image noise is also related to the high frequencies and may cause miscalculation of sharpness value. As a result, the fused images obtained by these methods are more noisy than the source images, and the performances are degraded. To make up for the deficiencies of traditional MST-based image algorithms, in our proposed method, after decomposing the original images using LSWT, we design a new image fusion rule based on multiscale products.

As we know, the HVS is highly sensitive to the local image contrast level. To meet this requirement, Toet and Ruyven developed the local luminance contrast in their research in CP [17]. It is defined as

$$R=\frac{L^{\prime }-L_{B^{\prime }}}{L_{B^{\prime }}}=\frac{\Delta L}{L_{B^{\prime }}}$$

(19)

where L' denotes the local gray level, L _B' is the local brightness of the background and corresponds to the low frequency component. Therefore, ΔL can be taken as the high frequency component.

Based on the above idea, many different forms of contrast measurement have been proposed in MST domain and provide better performance than the 'Coef-abs-max' scheme [18, 19, 25]. However, in those contrast measurements, the value (or absolute value) of a single pixel of the high frequency sub-image, namely the coefficient of the high frequency subband when the source image is decomposed by the MST, is used as ΔL. In fact, the value (or absolute value) of a signal pixel is very limited in determining which pixel is from the clear part of the sub-image. So, a pure use the value (or absolute value) of a single pixel as the high frequency component is not effective enough. We believe it will be more reasonable to employ feature of the high frequency subband, rather than the value (or absolute value) of pixel, as ΔL in the contrast measurement in Equation (19).

Like the sharpness measure, the ISL, shown in Equation (16), can effectively represent the salient features and sharp boundaries of an image. Pixels with larger values of ISL, when the source images are compared with each other, are more possible in focus. That means the ISL can successfully determine which pixel is in the focus. Therefore, it is reasonable to utilize ISL as one type of feature of the high frequency subband to represent ΔL in contrast measurement.

If we use ISL^d,l(x, y) (l = 1, 2,...,L) to denote the ISL located at (x, y) in the d th direction (d = 1, 2, 3) and l th scale. The feature contrast R^d,l(x, y) is defined as

$$R^{d,l}\left(x,y\right)=\frac{\mathsf{\text{IS}}{\mathsf{\text{L}}}^{d,l}\left(x,y\right)}{f^l\left(x,y\right)}$$

(20)

where f ^l(x, y) denotes the low frequency coefficients located at (x, y) in the l th scale. In order to improve the robustness of the contrast to the noise of the low frequency subband, the feature contrast can be modified as

$$S^{d,l}\left(x,y\right)=\frac{\mathsf{\text{IS}}{\mathsf{\text{L}}}^{d,l}\left(x,y\right)}{{\bar{f}}^l\left(x,y\right)}$$

(21)

where

$${\bar{f}}^l\left(x,y\right)=\frac{1}{m\times n}\sum _m\sum _n{f}^l\left(x+m,y+n\right)$$

(22)

In Equation (22) the local area size m × n may be 3 × 3 or 5 × 5. In practice, to reduce the computation complexity and the influence of low frequency subband noise, f^l(x, y) can be substituted with the coarsest lowpass subband image f^L(x, y).

To further conform to the characteristics of HVS, the feature contrast must be improved using the 'local-based' idea, thus a local neighborhood-based feature contrast is proposed in this article. It can be represented as

$$S{R}^{d,l}\left(x,y\right)=\sum _a\sum _b{W}_h\left(a,b\right)S^{d,l}\left(x+a,y+b\right)$$

(23)

where W _h is a template of size 3 × 3. For the high frequency subband, it contains high frequency information. In order to match the information of the LSWT neighborhood of high frequency subband, the values of center and center neighborhood of the template should have relative large change between each other [30]. In this article, a weighted template based on city-block distance is used, which is

$$W_h\left(a,b\right)=\frac{1}{16}\left[\begin{array}{ccc}1\hfill & \hfill 2\hfill & \hfill 1\hfill \\ 2\hfill & \hfill 4\hfill & \hfill 2\hfill \\ 1\hfill & \hfill 2\hfill & \hfill 1\hfill \end{array}\right]$$

In order to make up for the deficiencies of the traditional MST-based image fusion algorithm, which cannot restrain the noise influence, a new image fusion scheme is proposed in this article. In this fusion method we incorporate the merits of interscale dependencies, which can amplify the significant features, dilute noise and distinguish edge structures from noise more effectively, into the multifocus image fusion technique. In contrast to the traditional MST-based fusion methods, we design the fusion rule of the high frequency subbands based on the multiscale products instead of the wavelet coefficients. According to the formulae (23), the local feature contrast of multiscale products can be defined as

$$\mathsf{\text{MS}}{\mathsf{\text{R}}}^{d,l}\left(x,y\right)=\sum _a\sum _b{W}_h\left(a,b\right)\mathsf{\text{MP}}{\mathsf{\text{S}}}^{d,l}\left(x+a,y+b\right)$$

(24)

where

$$\mathsf{\text{MP}}{\mathsf{\text{S}}}^{d,l}\left(x,y\right)=\frac{\mathsf{\text{PS}}{\mathsf{\text{L}}}^{d,l}\left(x,y\right)}{{\bar{f}}^l\left(x,y\right)}$$

(25)

$$\begin{array}{ll}\hfill \mathsf{\text{PS}}{\mathsf{\text{L}}}^{d,l}\left(x,y\right)& =\left|8P^{d,l}f\left(x,y\right)-4P^{d,l}f\left(x-1,y\right)-4P^{d,l}f\left(x+1,y\right)\right|\\ +\left|8P^{d,l}f\left(x,y\right)-4P^{d,l}f\left(x,y-1\right)-4P^{d,l}f\left(x,y+1\right)\right|\\ +\left|2P^{d,l}f\left(x,y\right)-P^{d,l}f\left(x-1,y+1\right)-P^{d,l}f\left(x+1,y-1\right)\right|\\ +\left|2P^{d,l}f\left(x,y\right)-P^{d,l}f\left(x-1,y-1\right)-P^{d,l}f\left(x+1,y+1\right)\right|\end{array}$$

(26)

where PSL^d,l(x, y) denotes the ISL of multiscale products located at (x, y) in l th scale and d th direction; P^d,lf(x, y) and MPS^d,l(x, y) are the corresponding multiscale products and the feature contrast.

Therefore, the proposed selection principle for the high frequency subband coefficients can be described as follows:

$$f_F^{d,l}\left(x,y\right)=\left\{\begin{array}{c}{f}_A^{d,l}\left(x,y\right)if:{\mathsf{\text{MSR}}}_A^{d,l}\left(x,y\right)>{\mathsf{\text{MSR}}}_B^{d,l}\left(x,y\right)\hfill \\ f_B^{d,l}\left(x,y\right)if:{\mathsf{\text{MSR}}}_A^{d,l}\left(x,y\right)\le {\mathsf{\text{MSR}}}_B^{d,l}\left(x,y\right)\hfill \end{array}\right.$$

(27)

The local feature contrast of multiscale products cannot only effectively represent the salient features and sharp boundaries of image, but also effectively avoid the noise influence. A large value of the feature contrast means more high frequency information. So the proposed fusion scheme can extract more useful detail information from source images and inject them into the fused image. For simplicity, we name this fusion rule as 'MSP-con-max' in this article.

5. Experimental results and analysis

To evaluate the performance of the proposed fusion method, several experimental results are presented in this section. Experiments are performed on four sets of 256-level images: clean 'pepsi' (of size 512 × 512), clean 'flower' (of size 384 × 512), clean 'barb' (of size 512 × 512) and noisy 'pepsi' (of size 512 × 512). All of them are registered perfectly and shown in Figure 5a-f,h-j, respectively.

Figure 5

Source images for multifocus image fusion. (a) and (b), (c) and (d), are the multifocus clean image pairs; (h)-(j) are the originals with blur at the right, left and middle, respectively, and (g) is the reference image of (h)-(j); (e) and (f) are the multifocus noisy image pairs, which are partly defocused and partly in good-focus.

In order to show the advantages of the new image fusion method, we establish three steps to demonstrate that the proposed image fusion method outperforms other methods. First, 'MSP-con-max' is compared with 'Coef-abs-max', the 'Traditional-contrast-max' ('Tra-con-max'), and the proposed 'Feature-contrast-max' ('Fea-con-max'), which is designed according to Equation (23), to demonstrate the performance of the 'MSP-con-max' rule. For the 'Tra-con-max', the absolute value of a single pixel of the high frequency subband is used as ΔL in the contrast measurement. Second, the proposed image fusion algorithm is compared with DWT-simple-based method (Method 1), LSWT-simple-based method (Method 2), and NSCT-simple-based method (Method 3), in all of which the low frequency subband coefficients and the high frequency subband coefficients are simply merged by the 'averaging' scheme and the 'Coef-abs-max' scheme, respectively. For comparison purposes, the proposed algorithm is also compared with other four fusion algorithms (namely Methods 4-7). In Methods 4 and 5, LSWT is used as the MST method, and the 'IEOL-max' fusion rule is employed to merge the low frequency subband coefficients. For fusion of the high frequency subband coefficients, the 'Coef-abs-max' and 'Tra-con-max' fusion rules, are respectively used in Methods 4 and 5. For Method 6, the fusion rules of [7], which have been deigned based on the feature of the multiscale products and pulse coupled neural network (PCNN) [7], are respectively used to merge the low and high frequency subband LSWT coefficients (We name the method as 'LSWT-PCNN'). In this method, the PCNN is a model based on the cats primary visual cortex. It is characterized by the global coupling and pulse synchronization of neurons and has been proven suitable for image processing [33]. In Method 7, the NSCT is used as the MST method, and our proposed 'IEOL-max' and 'MSP-con-max' are, respectively, employed to fuse the low and high frequency subbands coefficients (We name it as 'NSCT-MSP-Con'). For multiscale scale products of NSCT, it can be defined just like Equation (13).

In all of these methods, the 'db5' and 'db53' wavelets, together with a decomposition level of 3 are used in DWT-based and LSWT-based methods (including Methods 2, 4, 5, 6 and our proposed method), respectively. Three decomposition levels are also used in the NSCT-based method (including NSCT-simple and NSCT-MSP-Con). All of these methods are used to fuse the multifocus clean images. Third, multifocus noisy images, as shown in Figure 5e,f, are fused by above different methods.

5.1. Contrast-based fusion rule in LSWT domain

In this section, we will show the performance of 'Fea-con-max' and 'MSP-con-max' fusion rules. In order to demonstrate the advantages of the new fusion rule, 'MSP-con-max' and 'Fea-con-max' are compared with'Tra-con-max' and 'Coef-abs-max' on high frequency subbands in LSWT domain.

Figure 6a-d shows the high frequency sub-images of the labeled region of Figure 5a,b,e,f in LSWT domain. One can see that the values of coefficients in clear part are greater than those of blurry part, even though the source image is in a noisy environment. That is why typical 'Coef-abs-max' is used in MST-based fusion algorithms.

Figure 6

Comparison of Coef-abs-max, Tra-con-max, Fea-con-max, and MSP-con-max rules. (a), (b) and (c), (d) are the high frequency subbands of the labeled parts of Figure 5a,b and Figure 5e,f; (e), (f) and (g), (h) are the multiscale products of the Figure 6a-d, respectively; (i)-(l) are decision maps of Coef-abs-max, Tra-con-max, Fea-con-max, and MSP-con-max rules in fusion (a) and (b); (m)-(p) are decision maps of Coef-abs-max, Tra-con-max, Fea-con-max, and MSP-con-max rules in fusion (c) and (d).

Figure 6e-h shows the multiscale products of Figure 6a-d, respectively. From Figure 6g,h, we can find that the multiscale products of LSWT can distinguish edge structures from noise effectively. Figure 6i-l are the decision maps, in which the coefficients selected from the image in Figure 6b are represented by white color, whereas the coefficients from Figure 6a are represented by black color. Since labeled part of Figure 6b is clearer than that of Figure 6a, the optimal decision map should be in white color in the whole decision map, which means all coefficients should be selected from Figure 6b. However, the decision maps of 'Coef-abs-max' rule and 'Tra-con-max' rule, shown in Figure 6i,j, indicate that these rules do not select the coefficients from the clear part completely even though 'Tra-con-max' shows better performance than 'Coef-abs-max'. Figure 6k,l indicates that the proposed feature contrast is more reasonable than the traditional contrast. It is also proven that applying feature such as ISL to the contrast is more reasonable than the absolute value of a single pixel.

Figure 6m-p shows the decision maps, in which the white color indicates that coefficients are selected from Figure 6d, otherwise selected from Figure 6c. From these figures we can see that the proposed 'MSP-con-max' rule do well in fusion of the multifocus noisy images. All of these demonstrate that the proposed fusion rule cannot only select the coefficients of the fused image properly but also restrain the influence of noise effectively.

The results of objective assessment are shown in Figure 7a,b. Figure 7a denotes the performance of different fusion rules in fusion of the multifocus clean image, and Figure 7b presents the performance of the different fusion rules in fusion of the multifocus noisy image. In Figure 7a, 'From a' and 'From b' denote the number of pixels that come from Figure 6a,b, respectively. Obviously, the proposed method is superior to others because the number of pixels that come from Figure 6b is the largest. As a result, the fused image is closer to the good-focus source image when using our proposed fusion rule, compared to using 'Coef-abs-max' rule, and 'Tra-con-max' rule, when the source images are noise-free. From Figure 7b, the same conclusion can be drawn that the proposed 'MSP-con-max' fusion rule outperforms the traditional fusion rules, when the source images are in a noisy environment.

Figure 7

Performance of different fusion rules: (a) performance of different fusion rules when the source images are clean; (b) performance of different fusion rules when the source images are in a noisy environment.

5.2. Fusion of clean multifocus images

In this section, the experiments are performed on three pairs of multifocus clean images, which are shown in Figure 5a-d,h-j, respectively. All the experiments are implemented in Matlab7.01 and on AMD Athlon(tm) 2.4 GHz with 2 G RAM. For further comparison, besides visual observation, two objective criteria are used to compare the fusion results. The first criterion is the mutual information (MI) [34]. It is a metric defined as the sum of mutual information between each input image and the fused image. The second criterion is Q^AB/F[35] metric, proposed by Xydeas and Petovic, which considers the amount of edge information transferred from the input images to the fused image. This method uses a Sobel edge detector to calculate strength and orientation information at each pixel in both source and the fused images. For both criteria, the larger the value, the better is the fusion result.

The first experiment is performed on the 'pepsi' multifocus clean images which have been registered perfectly. Figure 8 illustrates the fusion results obtained by the above mentioned eight different methods (including the proposed method). For a clearer comparison, the difference images between the fused images, which are fused results using Methods 1-7 and our proposed method, and the source image in Figure 5b are given in Figure 8i-p. To make better comparisons, Figure 8q-x illustrates parts of the labeled regions of Figure 8i-p. For the focused regions, the difference between the source image and the fused image should be zero. So the lower residue features in the difference image mean the better the fusion method transfers information of the source images to fused image. Focusing on the images which are shown in Figure 8q-s, one can obviously find that the fused images obtained by the LSWT method (Method 2) and the NSCT method (Method 3) are clearer than the DWT fused result. It is proven that shift-invariant methods such as LSWT and NSCT can overcome the pseudo-Gibbs phenomena successfully and improve the quality of the fused image around edges. Figure 8t indicates that the proposed fusion rule of the low frequency subband is more reasonable and useful in fusion multifocus clean images when compared with the 'averaging' fusion scheme. From Figure 8u, we can find that the 'Tra-con-max' fusion scheme does not extract almost all the useful information of the source images and nor transfer it to the fused image. However, Figure 8v,x shows that fused images attained by our proposed method and Methods 6 and 7 are with better visual quality. Almost all of the useful information of the source images has been transferred to the fused images, and meantime, fewer artifacts are introduced during the fusion process. All of these demonstrate the proposed feature contrast is more reasonable and useful than the traditional contrast.

Figure 8

The 'Pepsi' multifocus image fusion results: (a)-(h) fused images using Methods 1-7 and the proposed method, respectively; (i)-(p) difference images between Figures 5b and 8a-h; (q)-(x) are the parts of the labeled regions of Figure 8i-p.

In order to further evaluate the fusion performance, the second experiment is performed on another set of multifocus clean images, which are also registered perfectly and shown in Figure 5c,d. The resultant fused images are shown in Figure 9a-h. Again, for clearer comparison, the difference images between the fused images, which are fused results using Methods 1-7 and our proposed method, and the source images which are shown in Figure 5d are given in Figure 9i-p. Parts of the labeled regions of Figure 9i-p are extracted and put into Figure 9q-x. Figure 9q-x indicates that the proposed method and NSCT-MSP-Con-based method can extract almost all the good-focalized parts of source images and preserve the detailed information better than the other methods. Moreover the proposed method provides similar performance compared with NSCT-MSP-Con-based method, even though the NSCT is more suitable for image fusion, i.e., because the fusion rules, which are designed in this article, are every effective and can extract almost all the useful information of the source images and transfer it to the fused image, no matter the image fusion method is in the LSWT domain or in NSCT domain.

Figure 9

The 'Flower' multifocus image fusion results: (a)-(h) fused images using Methods 1-7 and the proposed method, respectively; (i)-(p) difference images between Figures 5d and 9a-g; (q)-(x) are the parts of the labeled regions of Figure 9i-p.

Three source images with different blur regions, as shown in Figure 5h-j, are used to evaluate the fusion performance in the third experiment. To make better comparisons, the difference images between the fused images and the reference image, which are shown in Figure 5g, are given in Figure 10i-p. For clearer comparison, the labeled parts of Figure 10i-p are extracted and shown in Figure 10q-x. From Figure 10q-x, the same conclusion can be drawn that the proposed method outperforms others methods.

Figure 10

There 'Barb' images fusion results: (a)-(h) fused images using Methods 1-7 and the proposed method, respectively; (i)-(p) difference images between Figures 5g and 10a-h; (q)-(x) are the parts of the labeled regions of Figure 10i-p.

Furthermore, the values of objective criteria on mutual information (MI), Q^AB/Fand the execution time of Figures 8a-h, 9a-h, and 10a-h are listed in Tables 1, 2, and 3, respectively. We observe that the fused images produced by NSCT-simple-based method are slightly better than the LSWT-simple fusion results, and all of them outperform the DWT approach in terms of MI and Q^AB/F. However, the NSCT is time consuming, which impedes its realtime application. As the modified version of LWT, LSWT consumes more time than DWT, that is because LSWT possesses the shift-invariance, and needs to process more dates of the image during the fusion process.

Table 1 Performance of different fusion methods on precessing Figure 5a,b

Table 2 Performance of different fusion methods on precessing Figure 5c,d

Table 3 Performance of different fusion methods on precessing Figure 5h-j

Form Tables 1, 2, and 3, we find that the NSCT-MSP-Con-based method provides similar performance compared with our proposed method. However, the NSCT-MSP-Con is more time consuming than our proposed method, because of its multi-direction and complexity. By considering both the fusing results and computing complexity, we utilize LSWT as the MST method in our proposed algorithm. For the LSWT-PCNN-based method, it is also more time consuming than the proposed method, because the PCNN neuron is very complex and it needs iterative operation to obtain pleasing fusion results. Moreover, the number of parameters of each neuron which need to be adjusted is large and they affect each other greatly. In image processing with PCNN, people often assign the same values to the corresponding parameters of each neuron. They are all chosen with experiments or experiences. For the visual system of eyes, it is impossible that all the parameters of neurons have the same value. They should be related to the situation of the neuron cell. All of these disadvantages significantly compromise the performance of LSWT-PCNN. Relative to LSWT-PCNN, our proposed method not only considers the property of HVS, which is highly sensitive to the local image contrast level, but also possesses some advantages such as simple calculation, high efficiency, etc. So our proposed method can provide better performance than LSWT-PCNN.

The values of Tables 1, 2, and 3 demonstrate that the proposed image fusion algorithm significantly outperforms other approaches (except NSCT-MSP-Con-based method) in terms of MI and Q^AB/F. Moreover, since reference (everywhere in focus) image of Figure 5h-j, as shown in Figure 5g, is available, performance comparison of different methods can be made using root mean square error (RMSE). The values of RMSE between Figure 11a-h and Figure 5g are given in Table 3. From Table 3, we can find that the objective evaluation results of RMSE coincide with the MI and Q^AB/Fevaluation results very well.

Figure 11

The noisy 'pepsi' multifocus noisy image fusion results: (a)-(h) fused images using Methods 1-7 and our proposed method, respectively. (i)-(p) are the parts of the corresponding regions of (a)-(h).

5.3. Fusion of noisy multifocus images

In order to evaluate the performance of the proposed method in a noisy environment, the input multifocus images 'pepsi', as shown in Figure 5e,f, have been additionally corrupted by Gaussian noise with deviation δ = 0.01.

In following experiment, since reference (everywhere-in-focus) images of the scenes under analysis are not available, performance comparison of the proposed method cannot be made using RMSE based metrics for this kind of image. Therefore, image fusion performance evaluation measures which do not require the availability of an ideal image have to be employed. For comparison, besides visual observation, objective criteria on MI and Q^AB/Fare used to evaluate how much information of the multifocus clean images, which are shown in Figure 5a,b, is contained in the fused images. However, the objective criteria on MI and Q^AB/Fcannot evaluate the performance of these fusion methods in terms of the input/output noise transmission. For further comparison, the improvement in terms of peak signal to noise ratio (PSNR), proposed by Loza et al. [6], is adopted to measure the noise change between the fused image and source noisy image. Let ${\sigma }_{n,f}^2$ denotes the noise variance in the fused output, the improvement in terms of PSNR is formulated as:

$$\Delta \mathsf{\text{PSNR}}=10\left(\text{log}\frac{255}{{\sigma }_{n,f}^2}-\text{log}\frac{255}{{\sigma }_n^2}\right)=10\text{log}\frac{{\sigma }_n^2}{{\sigma }_{n,f}^2}$$

(28)

For the criteria of ΔPSNR, the larger the value, the less noise of fused image is introduced from the original noisy image, and the better is the fusion result.

Figure 11a-h illustrates the fusion results obtained by the above different methods. For a clearer comparison, Figure 11i-p illustrates the parts of the fusion results. By looking at the image examples shown in Figure 11i-l, one can find that the edges information of the fused images, which are fused results using Methods 1-4, respectively, are immersed into noise. That is because all these fusion methods are designed to transfer the high frequency information from the input images into the fused image. It is worth noting that much of the image noise is also related to high frequencies. As a result, the fused images obtained by these methods are more noisy than the source images. From Figure 11m, we can see that the edges of Figure 11m are not clearer than Figure 11n-p, because the noise of the source images causes miscalculation of the contrast values. Therefore, in the presence of noise, the performance of Methods 1-5 may not be as good as those in the noiseless environments. Figure 11n indicates that the Method 6 can reduce the noise level to some extent, but the edges information of the fused image is not clearer compared with Figure 11o,p, which are fused by the Method 7 and our proposed algorithm.

Furthermore, Table 4 gives the quantitative results of the Figure 11. From Table 4, we observe that different fusion methods appear to provide different image fusion performance and the proposed scheme outperforms the other seven image fusion algorithms in terms of larger MI, and Q^AB/Fquality. And the values of ΔPSNR indicate that the proposed fusion rule of the high frequency subband is more reliable, robust and stable than other fusion rules.

Table 4 Performance of different fusion methods on precessing Figure 5e,f

6. Conclusion

In this article, a new multifocus image fusion algorithm based on feature contrast of multiscale products is proposed in LSWT domain. In the proposed algorithm, a novel feature contrast of multiscale products, which stands for edge features in high frequency sub-images in LSWT domain, is developed and used as the fusion scheme of the high frequency subbands. Three pairs of clean multifocus images and one pair of noisy multifocus images are used to test the performance of the proposed image fusion method, respectively. The experimental results demonstrate that the proposed method outperforms the DWT-simple-based method, the LSWT-simple-based method, LSWT-Traditional-Contrast-based method, the LSWT-PCNN-based method and the NSCT-simple-based method in terms of both visual quality and objective evaluation, even though the source images are in a noisy environment. In the future, we will do more research on the fusion of the noisy images, in order to carry out denoising and fusion of noisy source images simultaneously. And that will become the new trends to develop in image fusion field in the future.