Image fusion research based on the Haar-like multi-scale analysis

Abstract

In view of the serious color and definition distortion in the process of the traditional image fusion, this study proposes a Haar-like multi-scale analysis model, in which Haar wavelet has been modified and used for the medical image fusion to obtain even better results. Firstly, when the improved Haar wavelet basis function is translated, inner product and down-sampled with each band of the original image, the band is decomposed into four sub-images containing one low-frequency subdomain and three high-frequency subdomains. Secondly, the different fusion rules are applied in the low-frequency domain and the high-frequency domains to get the low-frequency sub-image and the high-frequency sub-images in each band. The four new sub-frequency domains are inverse-decomposed to reconstruct each new band. The study configures and synthesizes these new bands to produce a fusion image. Lastly, the two groups of the medical images are used for experimental simulation. The Experimental results are analyzed and compared with those of other fusion methods. It can be found the fusion method proposed in the study obtain the superior effects in the spatial definition and the color depth feature, especially in color criteria such as OP, SpD, CR and SSIM, comparing with the other methods.

1 Introduction

Image fusion is an image enhancement technology, whose goal is to synthesize and process multi-source images of the same scene from different types of sensors, so as to generate result images with richer information and more robust performance that can better assist the completion of the next task [1,2,3,4]. Image fusion technology has been widely used in the fields of remote sensing engineering and medical engineering since its birth [5, 6]. At the same time, the research of image fusion algorithm has also attracted the attention of more and more scholars. In general, the image fusion algorithm research focuses on the following aspects: the spatial domain fusion method such as PCA, Brovey and morphological traditional fusion method, the fusion method based on artificial intelligence such as neural network, deep learning, the multi-scale analysis fusion method such as wavelet analysis and super wavelet analysis, and integrated the fusion method of the above three method [7, 8]. Image fusion strategies based on multi-scale analysis have been an enduring topic [9,10,11,12,13,14,15,16,17,18].

In the field of information fusion based on multi-scale analysis, the image fusion based on wavelet analysis and ultra-wavelet analysis is its typical application. As early as the mid-1980s, the wavelet packet analysis theory began to provide a new mathematical tool for image fusion [9,10,11]. Pajares et al. [9] detailed the early development of image fusion methods based on wavelet analysis. Krishn et al. [10] proposed that medical image fusion based on wavelet transformation and PCA transformation was used to assist medical diagnosis. In order to extract spectral and spatial information in the source images, the original medical image was decomposed into multiple sub-frequency domains by two-dimensional discrete wavelet, and the PCA theory was used to facilitate the improvement of spatial definition information. Zhu et al. [11] applied wavelet packet analysis to the IHS space and applied PCA theory to the fusion rules of high-frequency and low-frequency domain of wavelet packet decomposition to obtain good fusion effect. In the late 1990s, the emergence of ultra-wavelet analysis provided further support for the development of image fusion [12,13,14,15,16,17,18,19,20]. Ultra-wavelet analysis is the inheritance and development of wavelet analysis. On the basis of wavelet analysis, many methods and theories of ultra-wavelet analysis are derived, such as Contourlet analysis, Curvelet analysis, Shearlet analysis and so on [12,13,14,15,16,17,18]. Haithem et al. [12] discussed the application and development of various multi-scale transformation methods such as Pyramid, Wavelet, Ridgelet, Curvelet, Contourlet and Shearlet waves in the field of medical image fusion. In 2010, Yang et al. fused multiple groups of images by using contourlet wavelet packet transformation [13]. In 2017, Bao et al. [14] performed remote sensing image fusion based on Shearlet transformation combined with DS evidence theory. In 2018, Wu et al. [15] performed remote sensing image fusion under the double transformation of PCA transformation and Curvelet transformation, and obtained good results. In 2019, Zhu et al. used phase consistency and Laplace energy method as the fusion rules in the high domains and the low-frequency domain of NSCT decomposition, respectively, to fuse multimodal medical images to obtain satisfactory results [16].

In a broad sense, the concept of multi-scale analysis is not limited to the above wavelets analysis and ultra-wavelet analysis. In addition, there are also image fusion strategies based on multi-scale analysis or approximate multi-scale analysis, such as Laplace pyramid [17], guide filter [18, 19], multi-stage edge protection filter [20], multi-level potential low-rank representation mdlatlrr [21], significant multi-scale [22], empirical wavelet [23], and frame set transformation [24]. Because the concept of multi-scale analysis has an open extension, more scholars are keen on imaging research based on the multi-scale analysis. Literature [18] designed a multi-type images fusion method based on the image decomposition model with multi-level guided filtering and combining salient feature extraction and deep learning fusion strategy. Chen et al. [22] presented a multi-scale decomposition fusion method based on visual salience and Gaussian smoothing filter to decompose the original image into salient layer, detail layer, and base layer, in which nonlinear functions were used to calculate the weight coefficient fusion in salient layer and the phase consistency fusion rule was used for detail layer, respectively.

Some existing image fusion methods, such as PCA, super wavelet analysis, often have serious color distortion in the process of image fusion. For this problem, this paper proposes an improved Haar-like multi-scale transform for image fusion combined with the idea of multi-scale analysis. The three channels of a source image are converted to multi-scale transformation domain and decomposed as the low and the high-frequency bands by the Haar-like multi-scale analysis. The new three channels can be produced when merging the coefficients of the low and the high-frequency bands. Then they are performed by the inverse Haar-like multi-scale analysis and constructed as the fusion image. The principle of the proposed Haar-like multi-scale transform displayed as follows. The study improves the Haar wavelet presented in 1909 to be suitable for the multi-scale decomposition and reconstruction of an image and thus forms a modal of image fusion which gets one low-frequency band and three high-frequency bands after a single decomposition of the image. This research has three contributions described below. Firstly, it bases on the Haar wavelet idea and extends the filter length in convolution summation operation from 2 to 4. It uses a low-pass smoothing filter sequence [1, 1, 1, 1]/4 and a high-pass decomposition filter sequence [1, − 1, 1, − 1]/4. After the low-pass filter and the high-pass filter are convolved and dropped 2 sampling with source image, respectively, one low-frequency subimage and three high-frequency sub-images can be obtained. The original image is obtained after performing the reconstruction inverse operation. Secondly, it constructs a Haar-like multi-scale analysis model and applies this Haar-like analysis model to image fusion situations. Lastly, the low-frequency subdomain applies the fusion rule based on L2 norm and the three high-frequency subdomains apply the fusion rule based on matrix eigenvalue. The study has been organized as five parts. Part One introduces the subject of the thesis; Part Two denotes the relative theoretical representation of the thesis; Part Three describes the proposed methods of the thesis; Part Four declares the discussion and conversation of the simulation experiments; Part Five offers the conclusion of the thesis.

2 Multi-scale analysis theory

Multi-scale analysis of signals, also known as multi-resolution analysis provides a way to look at problems from multiple angles that contains both coarse scale and fine scale. One can see the whole picture of things through the coarse scale and the details of things through the fine scale. One can choose suitable multiple-scale patterns according to different needs. Only by combining multiple scales can the observers see both the whole and the parts and grasps the spatial distribution characteristics of the target object as much as possible [25,26,27,28]. The theory of multi-resolution analysis was proposed by US. Mallat in 1988 to study image processing problems. The famous Mallat algorithm comes from this. Image wavelet transform and super-wavelet transform are typical of multi-resolution analysis applications. A series of image components with different resolutions based on tower decomposition replaces the original image at a fixed scale. A wavelet decomposition of the signal is to make the original signal decomposed by row convolution and down sampling operation and column convolution and down sampling operation. One low-frequency sub-signal and three high-frequency sub-signals are obtained through the row filter operation and the column filter operation. The order of the two operations does not affect the decomposition results. The low-frequency sub-signal is decomposed again to obtain a second stage decomposition of the low-frequency sub-signal and three high-frequency sub-signals. The two layers of such signals get a low-frequency sub-signal and six high-frequency sub-signals. The low-frequency sub-signal and the high-frequency sub-signal are decomposed and reconstructed to obtain the original signal. The image is a typical two-dimensional signal. The wavelet decomposition and reconstruction of the images make it the opportunity to interpret the image from a deep perspective and study the image features and the interconnection between the features from multiple angles. In this process, there is no information loss and redundancy, which also increases the accuracy of image feature description. The multi-scale analysis strategy includes other solutions besides the wavelet transform series [17,18,19,20,21,22,23,24]. Multi-resolution analysis of images has always been widely used in image research ranging from image fusion, segmentation, to image feature extraction, image enhancement, and image compression encoding.

3 Methods

3.1 The Haar-like multi-resolution analysis

The original image ${I}_{A}$ has the size of $M\times N$. Low-pass decomposition filter sequence is defined as ${R}_{L}=[\mathrm{1,1},\mathrm{1,1}]/\sqrt{2}$ and high-pass decomposition filter sequence is defined as ${R}_{H}=[1,-\mathrm{1,1},-1]/\sqrt{2}$. It uses ${R}_{L}$ and ${R}_{H}$ to decompose the raw image ${I}_{A}$ by row and generate a row-transformed matrix ${I}_{B}$, which includes one low-frequency band and one high-frequency band. The row-decomposition process can be defined as Eq. (1)

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{B}(x,y)={(({I}_{A}(i,j+1),{I}_{A}(i,j+2),{I}_{A}(i,j+3),{I}_{A}(i,j+4))*{R}_{L})}_{4} , i=x \,and\, j=2*(y-1),\\ {I}_{B}(x,midc+y)={(({I}_{A}(i,j+1),{I}_{A}(i,j+2),{I}_{A}(i,j+3),{I}_{A}(i,j+4))*{R}_{H})}_{4} . \end{array}\right.\end{array}\end{array}$$

(1)

${I}_{A}(i,j)$ is the $(i,j)$th element in the original image ${I}_{A}$; ${I}_{B}(x,y)$ is the $(x,y)$th element in the row filter image ${I}_{B}$; $midc=INT(N/2)$ where $INT(\mu )$ defines the variable $\mu$. It uses ${R}_{L}$ and ${R}_{H}$ to decompose the transition image ${I}_{B}$ by column and generate a column-transformed matrix ${I}_{C}$ again. The final matrix includes one low-frequency band and three high-frequency bands. The column-decomposition process can be defined as Eq. (2)

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{C}(x,y)={(({I}_{B}(i+1,y),{I}_{B}(i+2,y),{I}_{B}(i+3,y),{I}_{B}(i+1,y))*{R}_{L})}_{4} , i=2*(x-1)\, and\, j=y,\\ {I}_{C}(x+midr,y)={(({I}_{B}(i+1,y),{I}_{B}(i+2,y),{I}_{B}(i+3,y),{I}_{B}(i+1,y))*{R}_{H})}_{4} . \end{array}\right.\end{array}\end{array}$$

(2)

The present multi-scale model uses a system of linear equations to reconstruct the original image. Matrix ${I}_{B}$ is constructed from matrix ${I}_{C}$ from Eq. (3)

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{B}(1,y)+{I}_{B}(2,y)+{I}_{B}(3,y)+{I}_{B}(4,y)=sqrt(2)*{I}_{C}(1,y), \\ {I}_{B}(1,y)-{I}_{B}(2,y)+{I}_{B}(3,y)-{I}_{B}(4,y)=sqrt(2)*{I}_{C}(midr+1,y), \\ {I}_{B}(3,y)+{I}_{B}(4,y)+{I}_{B}(5,y)+{I}_{B}(6,y)=sqrt(2)*{I}_{C}(2,y), \\ {I}_{B}(3,y)-{I}_{B}(4,y)+{I}_{B}(5,y)-{I}_{B}(6,y)=sqrt(2)*{I}_{C}(midr+2,y), \\ ...\\ \alpha *{I}_{B}(1,y)+\alpha *{I}_{B}(2,y)+{I}_{B}(m-1,y)+{I}_{B}(m,y)=sqrt(2)*{I}_{C}(midr,y), \\ \alpha *{I}_{B}(1,y)-\alpha *{I}_{B}(2,y)+{I}_{B}(m-1,y)-{I}_{B}(m,y)=sqrt(2)*{I}_{C}(midr+midr,y).\end{array}\right.\end{array}\end{array}$$

(3)

where the extension coefficient $\alpha$ is a positive integer and $1\le y\le n$. Matrix ${I}_{A}$ is constructed from matrix ${I}_{B}$ from Eq. (4)

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{A}(x,1)+{I}_{A}(x,2)+{I}_{A}(x,3)+{I}_{A}(x,4)=4*{S}_{row}(x,1), \\ {I}_{A}(x,1)-{I}_{A}(x,2)+{I}_{A}(x,3)-{I}_{A}(x,4)=4*{S}_{row}(x,midc+1), \\ {I}_{A}(x,3)+{I}_{A}(x,4)+{I}_{A}(x,5)+{I}_{A}(x,6)=4*{S}_{row}(x,2), \\ {I}_{A}(x,3)-{I}_{A}(x,4)+{I}_{A}(x,5)-{I}_{A}(x,6)=4*{S}_{row}(x,midc+2), \\ ...\\ \beta *{I}_{A}(x,1)+\beta *{I}_{A}(x,2)+{I}_{A}(x,n-1)+{I}_{A}(x,n)=4*{S}_{row}(x,midc), \\ \beta *{I}_{A}(x,1)-\beta *{I}_{A}(x,2)+{I}_{A}(x,n-1)-{I}_{A}(x,n)=4*{S}_{row}(x,midc+midc).\end{array}\right.\end{array}\end{array}$$

(4)

In Eq. (4) $1\le x\le m$ and the extension coefficient β is a positive integer. The original image ${I}_{A}$ of $m\times n$ size can be produced by solving Eq. (4).

The original image ${I}_{A}$ is low-pass filtered and down-sampled and then high-pass filtered and down-sampled row by row, which can form a row transformation matrix ${I}_{B}$. The image ${I}_{B}$ is low-pass filtered and down-sampled and then high-pass filtered and down-sampled column by column, which can form a column transformation matrix ${I}_{C}$. The matrix ${I}_{C}$ involves the four parts that point to one approximate part and three detail parts representing the horizontal, vertical and diagonal direction. Inverse decomposition of these 4 parts can be reconstructed to obtain the original signal. Figure 1 shows the first-order Haar-like multi-resolution decomposition and reconstruction of an image signal.

Fig. 1

First-order Haar-like multi-scale decomposition and reconstruction of an image: a the original image; b the four sub-images which involving an approximate part and three detail parts pointing to the horizontal, the vertical and the diagonal direction after the first-level Haar-like multi-scale decomposition

3.2 The image fusion rules

The three channels r, g and b of the source images ${I}_{P}$ and ${I}_{Q}$ are subjected to multi-scale decomposition of level 1 by the Haar-like multi-resolution analysis, respectively. The low-frequency and the high-frequency subdomains are obtained from the three channels of each image.

3.2.1 The image fusion rule in low-frequency field

The $\kappa (\kappa =r,g or b)$ band of the images ${I}_{P}$ and ${I}_{Q}$ are decomposed at level 1 by the Haar-like multi-scale analysis, respectively, to produce the low-frequency sub-images ${I}_{P\kappa }^{a}$ and${I}_{Q\kappa }^{a}$. The ${L}_{2}$ norm of the matrix can be used for feature extraction in image processing, which is used for solving the square root of the sum of squares of all elements in the matrix to extract luminance information. Suppose the size of image${I}_{P\kappa }^{a}$, ${I}_{Q\kappa }^{a}$ and $I$ have the size of $M$ row $N$ column. The idea of the low-frequency fusion rule $LowfrqR({I}_{P\kappa }^{a} , {I}_{Q\kappa }^{a})$ is as follows.

Firstly, a series of 3 × 3 matrices centering on the $(i,j)$th point of the image $I$ are calculated by using the sliding window mode in order. A matrix of 3 × 3 is written as

$$A=\left[\begin{array}{lll}I(i-1,j-1)& I(i-1,j)& I(i-1,j+1)\\ I(i,j-1) & I(i,j) & I(i,j+1) \\ I(i+1,j-1)& I(i+1,j)& I(i+1,j+1)\end{array} \right]$$

(5)

The ${L}_{2}$ norm of the matrix of Eq. (5) is

$${||A||}_{2}=\sqrt{\begin{array}{l}{(I(i-1,j-1))}^{2}+{(I(i-1,j))}^{2}+{(I(i-1,j+1))}^{2}+ \\ {(I(i,j-1))}^{2}+{(I(i,j))}^{2}+{(I(i,j+1))}^{2}+ \\ {(I(i+1,j-1))}^{2}+{(I(i+1,j))}^{2}+{(I(i+1,j+1))}^{2}) .\end{array}}$$

(6)

Depending on Eq. (6), there are two pairs of $M\times N$ ${L}_{2}$ norms of the matrices of 3 × 3 size produced from the sub-images ${I}_{P\kappa }^{a}$ and${I}_{Q\kappa }^{a}$, respectively, by using the sliding window mode. The $M\times N$ ${L}_{2}$ norms from the sub-image ${I}_{P\kappa }^{a}$ are arranged as the matrix ${M}_{P\kappa }^{L2}$ whereas the $M\times N$ ${L}_{2}$ norms from the sub-image ${I}_{Q\kappa }^{a}$ are arranged as the matrix ${M}_{Q\kappa }^{L2}$.

Secondly, the spatial frequency is often used to represent the spatial clarity information of an image. The spatial frequency value ${SF}_{P\kappa }$ is computed from the matrix ${M}_{P\kappa }^{L2}$ and the spatial frequency value ${SF}_{Q\kappa }$ is computed from the matrix ${M}_{Q\kappa }^{L2}$ according to Eq. (7) written as

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{SF}_{h}=({\sum }_{i=1,j=1}^{M,N}{(I(i+1,j)-I(i,j))}^{2})/((M-1)*(N-1)), \\ {SF}_{v}=({\sum }_{i=1,j=1}^{M,N}{(I(i,j+1)-I(i,j))}^{2})/((M-1)*(N-1)),\\ {SF}_{src}=\sqrt{{SF}_{h}+{SF}_{v}} . \end{array}\right.\end{array}\end{array}$$

(7)

${SF}_{h}$ in Eq. (7) points to the mean of the squares sum of difference of the adjacent elements of the matrix $I$ in horizontal direction, and ${SF}_{v}$ in Eq. (7) points to that in vertical direction. The ${SF}_{src}$ is the square root of the sum of the ${SF}_{h}$ and the ${SF}_{v}$. Here ${SF}_{src}$ is mentioned to ${SF}_{P\kappa }$ or ${SF}_{Q\kappa }$.

Thirdly, the 0 and 1 binary graph ${M}_{bin}$ is generated according to Eq. (8), in which the low-frequency ratio $T1$ is equal to 10.

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{M}_{bin}(i,j)=1\quad if {M}_{P\kappa }^{L2} (i,j)>T1*{SF}_{P\kappa }\, and\, {M}_{Q\kappa }^{L2} (i,j)>T1*{SF}_{Q\kappa } , \\ {M}_{bin}(i,j)=0\quad otherwise . \end{array}\right.\end{array}\end{array}$$

(8)

Lastly, according to the value of the $(i,j)$th pixel of the binary graph ${M}_{bin}$, it selects the appropriate data source for the $(i,j)$th pixel of the fusion image, which can be defined as Eq. (9)

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{F\kappa }^{a} (i,j)={I}_{P\kappa }^{a} (i,j) \quad if {M}_{bin}(i,j)=1, \\ {I}_{F\kappa }^{a} (i,j)={I}_{Q\kappa }^{a} (i,j) \quad if {M}_{bin}(i,j)=0 .\end{array}\right.\end{array}\end{array}$$

(9)

When ${M}_{bin}(i,j)$ is 1, ${I}_{F\kappa }^{a} (i,j)$ chooses the value from the low-frequency subdomain ${I}_{P\kappa }^{a} (i,j)$. When ${M}_{bin}(i,j)$ is 0, ${I}_{F\kappa }^{a} (i,j)$ chooses the value from the low-frequency subdomain ${I}_{Q\kappa }^{a} (i,j)$.

3.2.2 The image fusion rule in high-frequency field

There are three high-frequency sub-images ${I}_{P\kappa }^{h}$ and${I}_{Q\kappa }^{h}$, ${I}_{P\kappa }^{v}$ and ${I}_{Q\kappa }^{v}$, ${I}_{P\kappa }^{d}$ and ${I}_{Q\kappa }^{d}$ derived from the r band, the g band and the b band of the images ${I}_{P}$ and ${I}_{Q}$ by the decomposition of the first layer by the Haar-like multi-scale analysis, respectively. For a linear system with small perturbations, the state variables of the system will change very little after a spatial step if the eigenvalue of its matrix is close to or equal to zero. The $M\times N$ square sums based on the eigenvalue of the matrix of 3 × 3 size arise in turn from the image$I$, which are centering on the $(i,j)$ th pixel of the image $I$ by using the sliding window mode of Eq. (5). If this square sum is small and falls within a certain range, then it means that the change between one point and another point in the image is not very variable, and the edge features are not obvious. However, it can be considered as edge features for cases with large changes. The high-frequency fusion rule $HighfrqR({I}_{P\kappa }^{\gamma } , {I}_{Q\kappa }^{\gamma })$ is described below, where $\gamma$ is equal to a, h, or v.

The research makes Eq. (5) as $B$ and the corresponding eigenvalue of $B$ as ${\lambda }_{i}(1\le i\le 3)$ according to the eigenequation of 3-order phalanx $Bx=\lambda x$. The sum of the squares of these 3 eigenvalues is ${({\lambda }_{1})}^{2}$+${({\lambda }_{2})}^{2}$+${({\lambda }_{3})}^{2}$. The algorithm based on square sum of eigenequation is applied to the three pairs of high-frequency subdomains ${I}_{P\kappa }^{h}$ and ${I}_{Q\kappa }^{h}$, ${I}_{P\kappa }^{v}$ and ${I}_{Q\kappa }^{v}$, ${I}_{P\kappa }^{d}$ and ${I}_{Q\kappa }^{d}$ to get the three couples of the corresponding eigenequation matrices ${M}_{P\kappa }^{h}$ and ${M}_{Q\kappa }^{h}$, ${M}_{P\kappa }^{v}$ and ${M}_{Q\kappa }^{v}$, ${M}_{P\kappa }^{d}$ and ${M}_{Q\kappa }^{d}$.

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{F\kappa }^{\gamma } (i,j)=\beta * {I}_{P\kappa }^{\gamma } (i,j)\quad if {M}_{P\kappa }^{\gamma } (i,j)<T2\, and\, {M}_{Q\kappa }^{\gamma } (i,j)<T2, \\ {I}_{F\kappa }^{\gamma } (i,j)={I}_{Q\kappa }^{\gamma } (i,j)\quad otherwise . \end{array}\right.\end{array}\end{array}$$

(10)

where the lift ratio $\beta$ ranges from 1.1 to 1.9 and the high-frequency ratio $T2$ is 0.001 in the study.

Equation (10) is used to configure the appropriate data sources for the high-frequency domain of the fused image ${I}_{F\kappa }^{\gamma }$. ${I}_{F\kappa }^{\gamma } (i,j)$) chooses the value from the product of $\beta$ and the high-frequency subdomain ${I}_{P\kappa }^{\gamma } (i,j)$ when ${M}_{P\kappa }^{\gamma } (i,j)<T2$ and ${M}_{Q\kappa }^{\gamma } (i,j)<T2$, otherwise ${I}_{F\kappa }^{\gamma } (i,j)$) chooses the value from the high-frequency subdomain ${I}_{Q\kappa }^{\gamma } (i,j)$.

3.3 The image fusion algorithm based on the Haar-like multi-scale analysis

The Haar-like multi-scale analysis strategy in Sect. 3.1 is applied to the image fusion of two images of the same scene, which is described in Fig. 2. The proposed image fusion scheme can be divided into the following steps.

Fig. 2

Image fusion model based on the proposed Haar-like multi-scale analysis

Step One. Each channel is decomposed into one sub-image of low-frequency and three sub-images of high-frequency by the $HaarLdec()$ method, which is denoted as Eq. (11)

$$\begin{array}{l}\begin{array}{l}{[I}_{\omega \kappa }^{a}, { I}_{\omega \kappa }^{h}, { I}_{\omega \kappa }^{v}, { I}_{\omega \kappa }^{d}] =HaarLdec({I}_{\omega \kappa })\end{array}\end{array}$$

(11)

where $\omega$ means the image source pointing to the image $P$ or the image$Q$; $\kappa$ means the r channel, the g channel or the b channel of an image. ${I}_{Pr}, {I}_{Pg}$ and ${I}_{Pb}$ represent the r channel, g channel and b channel of the original image ${I}_{P}$; and ${I}_{Qr}, {I}_{Qg}$ and ${I}_{Qb}$ represent the r channel, g channel and b channel of the original image${I}_{Q}$. ${I}_{\omega \kappa }^{a}, { I}_{\omega \kappa }^{h}, { I}_{\omega \kappa }^{v}$ and ${I}_{\omega \kappa }^{d}$ represent the approximate sub-image, the horizontal sub-image, the vertical sub-image and the diagonal sub-image when the $\kappa$ channel of the image $\omega$ is decomposed at the first level.

Step Two. It makes the fusion result be an image ${I}_{F}$. Its r-channel, g-channel and b-channel will carry out the corresponding fusion rules. The low-frequency fusion rule of the approximate sub-image ${I}_{F\kappa }^{a}$ is corresponding to the $LowfrqR()$ method in Eq. (12). The frequency band coefficients are derived from the images ${I}_{P}$ and ${I}_{Q}$. The high-frequency fusion rules of the horizontal sub-image ${I}_{F\kappa }^{h}$, the vertical sub-image ${I}_{F\kappa }^{v}$ and the diagonal sub-image ${I}_{F\kappa }^{d}$ are all corresponding to the $HighfrqR($) method in Eq. (12). The three high-frequency bands coefficients are derived from the image ${I}_{P}$ multiplied by the lift coefficient [29] and image ${I}_{Q}$.

$$\begin{array}{l}\begin{array}{l}\left\{\begin{array}{l}{I}_{F\kappa }^{a} =LowfrqR({I}_{P\kappa }^{a} , {I}_{Q\kappa }^{a}),\\ {I}_{F\kappa }^{h} =HighfrqR({I}_{P\kappa }^{h} , {I}_{Q\kappa }^{h}),\\ {I}_{F\kappa }^{v} =HighfrqR({I}_{P\kappa }^{v} , {I}_{Q\kappa }^{v}),\\ {I}_{F\kappa }^{d} =HighfrqR({I}_{P\kappa }^{d} , {I}_{Q\kappa }^{d}).\end{array}\right.\end{array}\end{array}$$

(12)

Step Three. The research generates and associates the three new channels to obtain the fusion image. Replying on Eq. (13), the three channels are reconstructed separately by the inverse Haar-like multi-scale transformation, i.e. the $HaarLrec()$ method.

$$\begin{array}{l}\begin{array}{l}{I}_{F\kappa }=HaarLrec({ I}_{F\kappa }^{a}, { I}_{F\kappa }^{h}, { I}_{F\kappa }^{v}, { I}_{F\kappa }^{d})\end{array}\end{array}$$

(13)

The image ${I}_{Fr}$ is reconstructed from ${I}_{Fr}^{a}, { I}_{Fr}^{h}, { I}_{Fr}^{v}$ and ${I}_{Fr}^{d}$ after the reverse Haar-like multi-scale decomposition; the image ${I}_{Fg}$ is reconstructed from ${I}_{Fg}^{a}, { I}_{Fg}^{h}, { I}_{Fg}^{v}$ and ${I}_{Fg}^{d}$ after the reverse Haar-like multi-scale decomposition and the image ${I}_{Fb}$ is reconstructed from ${I}_{Fb}^{a}, { I}_{Fb}^{h}, { I}_{Fb}^{v}$ and ${I}_{Fb}^{d}$ after the reverse Haar-like multi-scale decomposition. The channel image ${I}_{Fr}$, the channel image ${I}_{Fg}$, and the channel image ${I}_{Fb}$, are associated to get fused image ${I}_{F}$.

It can be found the following views from the above multi-scale tower shape decomposition and reconstruction process by using the Haar-like multi-scale. The multi-scale analysis algorithm proposed in this article analyzes the image from the perspective of the frequency domain, and the image can be divided into the low-frequency and high-frequency parts. The low-frequency part reflects the body information of the image such as the outline of the object and the basic composition area. It can mostly extract sufficient color information derived from the source image ${I}_{Q}$. The high-frequency part reflects the detail information of the image such as the textures and edges of the objects. It can mostly extract sufficient boundary information derived from the source image ${I}_{P}$.

4 Simulation experiments

4.1 Experimental materials and qualitative analysis

To identify the universality of the fusion method proposed, this research provides two sets of medical images as experimental materials. The first set of medicine images is a pair of computed tomography (CT) image and positron emission computed tomography (PET) image, which describe the scene of the kidney organs of a patient. The second set of medical images is a pair of CT image and PET image showing the female pelvic tissue scene. These two sets of source images are presented in Fig. 3. Figure 3a, b represents the CT image and the PET image of the first dataset, respectively. Figure 3c, d represents the CT image and the PET image of the second medicine dataset, respectively. Figures 4 and 5 show the fused images corresponding to these two sets of source images. Eleven comparative fusion methods are used, which are principal component analysis (PCA), hue saturation vision (HSV), wavelet packet transformation (WPT), curvelet, contourlet, nonsubsampled contourlet transformation (NSCT), HSV + WPT, and HSV + NSCT. HSV + WPT means the WPT fusion method based on the HSV space; HSV + NSCT means the NSCT fusion method based on the HSV space. Additionally, there are three fusion methods from Refs. [30,31,32] which are named as DEMEF [30], NDFA [31], and PODFA [32].

Fig. 3

Original images for groups ranging from the first to the second: a the CT image from the first group, b the PET image from the first group, c the CT image from the second group, d the PET image from the second group

Fig. 4

Images fused from Group 1: a image fused by PCA method, b image fused by HSV method, c image fused by WPT method, d image fused by Curvelet method, e image fused by Contourlet method, f image fused by NSCT method, g image fused by method using HSV and WPT, h image fused by method using HSV and NSCT, i image fused by DEMEF method, j image fused by NDFA method, k image fused by PODFA method, l image fused by Proposed method

Fig. 5

Images fused from Group 2: a image fused by PCA method, b image fused by HSV method, c image fused by WPT method, d image fused by Curvelet method, e image fused by Contourlet method, f image fused by NSCT method, g image fused by method using HSV and WPT, h image fused by method using HSV and NSCT, i image fused by DEMEF method, j image fused by NDFA method, k image fused by PODFA method, l image fused by Proposed method

There are twelve images fused displaying in Fig. 4 for the first group of data and in Fig. 5 for the second group of data, respectively. Figure 4a is corresponding to PCA fusion method for the ovarian cancer image; Fig. 4b is corresponding to HSV fusion method for the ovarian cancer image; Fig. 4c is corresponding to WPT fusion method for the ovarian cancer image; Fig. 4d is corresponding to Curvelet fusion method for the ovarian cancer image; Fig. 4e is corresponding to Contourlet fusion method for the ovarian cancer image; Fig. 4f is corresponding to NSCT fusion method for the ovarian cancer image; Fig. 4g is corresponding to HSV + WPT fusion method for the kidney organs image; Fig. 4h is corresponding to HSV + NSCT fusion method for the kidney organs image; Fig. 4i is corresponding to the DEMEF fusion method for the ovarian cancer image; Fig. 4j is corresponding to the NDFA fusion method for the ovarian cancer image; Fig. 4k is corresponding to the PODFA fusion method for the ovarian cancer image; Fig. 4l is corresponding to the fusion method represented for the ovarian cancer image. Figure 5a is corresponding to PCA fusion method for the female pelvic image; Fig. 5b is corresponding to HSV fusion method for the female pelvic image; Fig. 5c is corresponding to WPT fusion method for the female pelvic image; Fig. 5d is corresponding to Curvelet fusion method for the female pelvic image; Fig. 5e is corresponding to Contourlet fusion method for the female pelvic image; Fig. 5f is corresponding to NSCT fusion method for the female pelvic image; Fig. 5g is corresponding to HSV + WPT fusion method for the female pelvic image; Fig. 5h is corresponding to HSV + NSCT fusion method for the female pelvic image; Fig. 5i is corresponding to the DEMEF fusion method for the female pelvic image; Fig. 5j is corresponding to the NDFA fusion method for the female pelvic image; Fig. 5k is corresponding to the PODFA fusion method for the female pelvic image; Fig. 5l is corresponding to the fusion method represented for the female pelvic image.

Qualitative evaluation of the fusion effect of these two sets of images. Through Fig. 4a–l, the texture sharpness of the fusion images is compared and analyzed. Figure 4a has distinct lines, but compared with the PET source image, its spectral distortion is relatively large. Figure 4b has the lowest texture sharpness and the largest spectral distortion. The fusion effects of Fig. 4d–g are very similar to Fig. 4h. In this set of images, they are clearly drawn but close to the PET source image. Figure 4c looks similar to Fig. 4l. They have the highest texture sharpness and are closest to the PET source image in spectral terms in Fig. 4. Obviously, Fig. 4i–k look similar and they have the poor spectrum effects. The second set of pelvic images are shown in Fig. 5a–l. The texture clarity from Fig. 5a of PCA is obvious, and so is Fig. 5b of HSV. They have the most distorted colors compared to the other seven images in Fig. 5. The visual sense of the three images involving Fig. 5d, e, g are quite similar. They have achieved a medium level in both spatial information and color information in the second group of fusion images. Compared to the other multiple-scale fusion images in Fig. 5, the spatial clarity of the fusion Fig. 5g of HSV + WPT and the fusion Fig. 5h of HSV + NSCT is not prominent enough. But their color distortion is relatively low. The scenes the three images of Fig. 4i–k all have a serious spectrum loss. In terms of clarity and color information, the fusion effect of Fig. 5c by using wavelet packet is close to that of Fig. 5l by using the proposed fusion method. They not only have high definition, but also have high color depth information, especially Fig. 5l closest to Fig. 3d of the PET source image.

4.2 Quantization criteria

The quantitative evaluation has be done in a consistent way. A group of basic quantitative indicators such as average gradient (AG), space frequency (SF), standard derivation (StD), mean value (MV), overall performance (OP), mutual information (MI), spectrum distort (SpD), correlation ratio (CR) and structural similarity (SSIM), has been widely used in the image fusion area [33,34,35]. AG criterion indicates the space definition of an image, and the quality of an image is proportion to its value. SF also reflects the spatial definition information of the fused image in which the larger its value indicates, the better the definition is. MV criterion indicates the gray mean value of an image, which represents the color effect of the fusion image. When its MV value get closer to that of the source color image, the better its spectral fusion is. StD criterion indicates the degree of dispersion between the pixel value and the mean, and the larger the standard deviation is, the better the quality of the image is.

OP criterion indicates the rationality of the corresponding method [34]. OP = Σ_κ(AG_κ − SpD_κ)/3 = AG-SpD, where κ = r, g, b. When the OP value is positive, the comprehensive performance evaluation combining other reasonable index values and its method should be recommended. MI criterion indicates the relationship between the fused image and the source images. SpD criterion indicates how much color component is lost from the original color image for the fused image. CR criterion indicates how many source color features of the color image are acquired from the fusion image. The larger its value is, the more color information the fusion image obtains from the source color image. SSIM criterion indicates the similarity between the fused image and the source color image. The larger its value is, the higher their similarity is, which means the more structured information is obtained from the source color image. The index values of the AG, SF, StD and MV from the two groups of original images are shown in Table 1. The performances merged by using the various methods for the first and the second set of data are shown in Table 2 and 3.

Table 1 The assessment parameters of source images

Table 2 Performance of the different fusion methods on processing Fig. 4 for Group 1

Table 3 Performance of the different fusion methods on processing Fig. 5 for Group 2

4.3 Quantization evaluation

By observing the index values of the twelve fusion images of the first set of the kidney organs data, it is easy to find the following aspects. The WPT fusion method yielded the best spatial resolution which indicates an average gradient value of 10.8598 and a spatial frequency value of 22.9335 whereas the fusion method proposed in the study gets the second spatial resolution which indicates an average gradient value of 9.2437 and a spatial frequency value of 20.2221. The OP value of the fusion method proposed is 2.7240, being the first OP value of various methods, whereas the OP value of WPT is 2.1412, being the second OP value of various methods. The MI value of the HSV fusion method is 4.1398, being the first MI value of various methods, whereas the MI value of the fusion method proposed is 3.2787, being the third MI value of various methods. Finally, please look at the spectral fusion feature which corresponds to the three criteria: SpD, CR, and SSIM. The color features of the fusion method proposed have the best results, with the lowest spectral distortion SpD value of 6.2316, the maximum correlation coefficient value of 0.9684 and the maximum SSIM value of 0.9687. The fusion image performance of the other seven methods is far less than the above two methods for both spatial features and color features.

Similarly, the various index values of the second set of the twelve pelvic fusion images can be concluded as follows. According to Table 3, compared with the other 8 fusion methods, the fusion effect of the fusion method proposed is absolutely dominant. On one hand, the AG value of Fig. 5l is 7.5683, seconding only to the AG value of 7.6579 in Fig. 5c; the SF value of Fig. 5l is 18.6898, seconding only to the SF value of 18.9065 in Fig. 5a. In contrast, Fig. 5a has the most serious color loss, while Fig. 5c is nowhere behind Fig. 5l in terms of the other six indicators except AG. On the other hand, Fig. 5l of the fusion method proposed tops the list in terms of these criteria, such as OP, MI, SpD, CR and SSIM. The OP value of 2.7240, the MI value of 4.1673, the SpD value of 4.8443, the CR value of 0.9882 and the SSIM value of 0.9882.

PCA and HSV all belong to the spatial domain fusion methods. This type of fusion methods easily leads to severe color loss in the fusion images, especially for the kidney organs image. The remaining six conventional methods are all fusion methods based on multi-scale transformation domains. WPT uses a 2-layer wavelet packet to decompose the original image. Its wavelet packet basis function uses reverse bi-orthogonal wavelet instead of Haar wavelet, in which the former works better than the latter. Compared with other traditional methods, the fusion results are the best in respect of spatial characteristics and color depth. HSV + WPT performs a 2-level wavelet packet decomposition based on the inter-conversion of the RGB space and the HSV space for the image and its wavelet packet basis function also uses reverse bi-orthogonal wavelet. The fusion method is slightly less effective than the WPT method. The fusion effects of these three super-wavelet methods including Curvelet, Contourlet and NSCT are progressive, and they are not very different from each other. All three methods have close parameter values and are at intermediate levels among the eleven conventional methods. HSV + NSCT is the transformation by using NSCT based on the inter-conversion between the RGB space and the HSV space for the image. The fusion effect is somewhere between the Curvelet method and the HSV + WPT method.

In terms of spatial resolution and color resolution, the overall fusion performances of these 9 methods can be divided into five grades from general to excellent. The fusion effect of the HSV method and the PCA method is at the 5th level; the fusion effect of the Curvelet method, the Contourlet method and the NSCT method is at the 4th level; the fusion effect of the HSV + NSCT method and the HSV + WPT method is at the 3th level; the fusion effect of the WPT method is at the 2th level; the fusion effect of the proposed method is admirable, which is at the 1th level.

5 Conclusions

This study proposes a Haar-like multi-scale transformation method based on multi-scale analysis and Haar wavelet theory, which is applied for two kinds of image fusion occasions involving the kidney organs image fusion and the female pelvic image fusion, and achieves good results. From the fusion experiments of two sets of images, the represented fusion method gets better performance in both spatial performance and color characteristics, especially in terms of color effects. The kidney organs fusion image can maximize the extraction of spectral information of the source PET image, while the female pelvic fusion image can retain as much color depth information in the source color PET image as possible. The kidney organs images are usually fused for a specific application. It is common to merge the three-band high resolution CT image with the multi-band low resolution PET image. The goal is to generate a high-quality image that combines high-resolution features of CT image and spectral features of PET image to facilitate various analysis and processing. Although ordinary black and white CT images are able to highly distinguish bone from soft tissue, they have low measurable ability between different soft tissues. By merging CT image and PET image, ordinary CT and PET images are upgraded to color spectral medical images. It can effectively reduce the difference of the imaging contrast. Full fusion of spatial information and spectral information in source images can provide reliable guarantee for the further processing of unmixing, segmentation and classification of the medicine images. It is a more delicate, effective, accurate and rich performances of the image information inside the human body and the lesion area, which exceeds the traditional CT with PET images in quality. The improvement of the diagnosis rate and the further cognition of the disease provide the color medical image, especially for the diagnosis of tumor diseases.