 Research
 Open access
 Published:
Image fusion research based on the Haarlike multiscale analysis
EURASIP Journal on Advances in Signal Processing volume 2024, Article number: 23 (2024)
Abstract
In view of the serious color and definition distortion in the process of the traditional image fusion, this study proposes a Haarlike multiscale analysis model, in which Haar wavelet has been modified and used for the medical image fusion to obtain even better results. Firstly, when the improved Haar wavelet basis function is translated, inner product and downsampled with each band of the original image, the band is decomposed into four subimages containing one lowfrequency subdomain and three highfrequency subdomains. Secondly, the different fusion rules are applied in the lowfrequency domain and the highfrequency domains to get the lowfrequency subimage and the highfrequency subimages in each band. The four new subfrequency domains are inversedecomposed to reconstruct each new band. The study configures and synthesizes these new bands to produce a fusion image. Lastly, the two groups of the medical images are used for experimental simulation. The Experimental results are analyzed and compared with those of other fusion methods. It can be found the fusion method proposed in the study obtain the superior effects in the spatial definition and the color depth feature, especially in color criteria such as OP, SpD, CR and SSIM, comparing with the other methods.
1 Introduction
Image fusion is an image enhancement technology, whose goal is to synthesize and process multisource images of the same scene from different types of sensors, so as to generate result images with richer information and more robust performance that can better assist the completion of the next task [1,2,3,4]. Image fusion technology has been widely used in the fields of remote sensing engineering and medical engineering since its birth [5, 6]. At the same time, the research of image fusion algorithm has also attracted the attention of more and more scholars. In general, the image fusion algorithm research focuses on the following aspects: the spatial domain fusion method such as PCA, Brovey and morphological traditional fusion method, the fusion method based on artificial intelligence such as neural network, deep learning, the multiscale analysis fusion method such as wavelet analysis and super wavelet analysis, and integrated the fusion method of the above three method [7, 8]. Image fusion strategies based on multiscale analysis have been an enduring topic [9,10,11,12,13,14,15,16,17,18].
In the field of information fusion based on multiscale analysis, the image fusion based on wavelet analysis and ultrawavelet analysis is its typical application. As early as the mid1980s, the wavelet packet analysis theory began to provide a new mathematical tool for image fusion [9,10,11]. Pajares et al. [9] detailed the early development of image fusion methods based on wavelet analysis. Krishn et al. [10] proposed that medical image fusion based on wavelet transformation and PCA transformation was used to assist medical diagnosis. In order to extract spectral and spatial information in the source images, the original medical image was decomposed into multiple subfrequency domains by twodimensional discrete wavelet, and the PCA theory was used to facilitate the improvement of spatial definition information. Zhu et al. [11] applied wavelet packet analysis to the IHS space and applied PCA theory to the fusion rules of highfrequency and lowfrequency domain of wavelet packet decomposition to obtain good fusion effect. In the late 1990s, the emergence of ultrawavelet analysis provided further support for the development of image fusion [12,13,14,15,16,17,18,19,20]. Ultrawavelet analysis is the inheritance and development of wavelet analysis. On the basis of wavelet analysis, many methods and theories of ultrawavelet analysis are derived, such as Contourlet analysis, Curvelet analysis, Shearlet analysis and so on [12,13,14,15,16,17,18]. Haithem et al. [12] discussed the application and development of various multiscale transformation methods such as Pyramid, Wavelet, Ridgelet, Curvelet, Contourlet and Shearlet waves in the field of medical image fusion. In 2010, Yang et al. fused multiple groups of images by using contourlet wavelet packet transformation [13]. In 2017, Bao et al. [14] performed remote sensing image fusion based on Shearlet transformation combined with DS evidence theory. In 2018, Wu et al. [15] performed remote sensing image fusion under the double transformation of PCA transformation and Curvelet transformation, and obtained good results. In 2019, Zhu et al. used phase consistency and Laplace energy method as the fusion rules in the high domains and the lowfrequency domain of NSCT decomposition, respectively, to fuse multimodal medical images to obtain satisfactory results [16].
In a broad sense, the concept of multiscale analysis is not limited to the above wavelets analysis and ultrawavelet analysis. In addition, there are also image fusion strategies based on multiscale analysis or approximate multiscale analysis, such as Laplace pyramid [17], guide filter [18, 19], multistage edge protection filter [20], multilevel potential lowrank representation mdlatlrr [21], significant multiscale [22], empirical wavelet [23], and frame set transformation [24]. Because the concept of multiscale analysis has an open extension, more scholars are keen on imaging research based on the multiscale analysis. Literature [18] designed a multitype images fusion method based on the image decomposition model with multilevel guided filtering and combining salient feature extraction and deep learning fusion strategy. Chen et al. [22] presented a multiscale decomposition fusion method based on visual salience and Gaussian smoothing filter to decompose the original image into salient layer, detail layer, and base layer, in which nonlinear functions were used to calculate the weight coefficient fusion in salient layer and the phase consistency fusion rule was used for detail layer, respectively.
Some existing image fusion methods, such as PCA, super wavelet analysis, often have serious color distortion in the process of image fusion. For this problem, this paper proposes an improved Haarlike multiscale transform for image fusion combined with the idea of multiscale analysis. The three channels of a source image are converted to multiscale transformation domain and decomposed as the low and the highfrequency bands by the Haarlike multiscale analysis. The new three channels can be produced when merging the coefficients of the low and the highfrequency bands. Then they are performed by the inverse Haarlike multiscale analysis and constructed as the fusion image. The principle of the proposed Haarlike multiscale transform displayed as follows. The study improves the Haar wavelet presented in 1909 to be suitable for the multiscale decomposition and reconstruction of an image and thus forms a modal of image fusion which gets one lowfrequency band and three highfrequency bands after a single decomposition of the image. This research has three contributions described below. Firstly, it bases on the Haar wavelet idea and extends the filter length in convolution summation operation from 2 to 4. It uses a lowpass smoothing filter sequence [1, 1, 1, 1]/4 and a highpass decomposition filter sequence [1, − 1, 1, − 1]/4. After the lowpass filter and the highpass filter are convolved and dropped 2 sampling with source image, respectively, one lowfrequency subimage and three highfrequency subimages can be obtained. The original image is obtained after performing the reconstruction inverse operation. Secondly, it constructs a Haarlike multiscale analysis model and applies this Haarlike analysis model to image fusion situations. Lastly, the lowfrequency subdomain applies the fusion rule based on L2 norm and the three highfrequency subdomains apply the fusion rule based on matrix eigenvalue. The study has been organized as five parts. Part One introduces the subject of the thesis; Part Two denotes the relative theoretical representation of the thesis; Part Three describes the proposed methods of the thesis; Part Four declares the discussion and conversation of the simulation experiments; Part Five offers the conclusion of the thesis.
2 Multiscale analysis theory
Multiscale analysis of signals, also known as multiresolution analysis provides a way to look at problems from multiple angles that contains both coarse scale and fine scale. One can see the whole picture of things through the coarse scale and the details of things through the fine scale. One can choose suitable multiplescale patterns according to different needs. Only by combining multiple scales can the observers see both the whole and the parts and grasps the spatial distribution characteristics of the target object as much as possible [25,26,27,28]. The theory of multiresolution analysis was proposed by US. Mallat in 1988 to study image processing problems. The famous Mallat algorithm comes from this. Image wavelet transform and superwavelet transform are typical of multiresolution analysis applications. A series of image components with different resolutions based on tower decomposition replaces the original image at a fixed scale. A wavelet decomposition of the signal is to make the original signal decomposed by row convolution and down sampling operation and column convolution and down sampling operation. One lowfrequency subsignal and three highfrequency subsignals are obtained through the row filter operation and the column filter operation. The order of the two operations does not affect the decomposition results. The lowfrequency subsignal is decomposed again to obtain a second stage decomposition of the lowfrequency subsignal and three highfrequency subsignals. The two layers of such signals get a lowfrequency subsignal and six highfrequency subsignals. The lowfrequency subsignal and the highfrequency subsignal are decomposed and reconstructed to obtain the original signal. The image is a typical twodimensional signal. The wavelet decomposition and reconstruction of the images make it the opportunity to interpret the image from a deep perspective and study the image features and the interconnection between the features from multiple angles. In this process, there is no information loss and redundancy, which also increases the accuracy of image feature description. The multiscale analysis strategy includes other solutions besides the wavelet transform series [17,18,19,20,21,22,23,24]. Multiresolution analysis of images has always been widely used in image research ranging from image fusion, segmentation, to image feature extraction, image enhancement, and image compression encoding.
3 Methods
3.1 The Haarlike multiresolution analysis
The original image \({I}_{A}\) has the size of \(M\times N\). Lowpass decomposition filter sequence is defined as \({R}_{L}=[\mathrm{1,1},\mathrm{1,1}]/\sqrt{2}\) and highpass decomposition filter sequence is defined as \({R}_{H}=[1,\mathrm{1,1},1]/\sqrt{2}\). It uses \({R}_{L}\) and \({R}_{H}\) to decompose the raw image \({I}_{A}\) by row and generate a rowtransformed matrix \({I}_{B}\), which includes one lowfrequency band and one highfrequency band. The rowdecomposition process can be defined as Eq. (1)
\({I}_{A}(i,j)\) is the \((i,j)\)th element in the original image \({I}_{A}\); \({I}_{B}(x,y)\) is the \((x,y)\)th element in the row filter image \({I}_{B}\); \(midc=INT(N/2)\) where \(INT(\mu )\) defines the variable \(\mu\). It uses \({R}_{L}\) and \({R}_{H}\) to decompose the transition image \({I}_{B}\) by column and generate a columntransformed matrix \({I}_{C}\) again. The final matrix includes one lowfrequency band and three highfrequency bands. The columndecomposition process can be defined as Eq. (2)
The present multiscale model uses a system of linear equations to reconstruct the original image. Matrix \({I}_{B}\) is constructed from matrix \({I}_{C}\) from Eq. (3)
where the extension coefficient \(\alpha\) is a positive integer and \(1\le y\le n\). Matrix \({I}_{A}\) is constructed from matrix \({I}_{B}\) from Eq. (4)
In Eq. (4) \(1\le x\le m\) and the extension coefficient β is a positive integer. The original image \({I}_{A}\) of \(m\times n\) size can be produced by solving Eq. (4).
The original image \({I}_{A}\) is lowpass filtered and downsampled and then highpass filtered and downsampled row by row, which can form a row transformation matrix \({I}_{B}\). The image \({I}_{B}\) is lowpass filtered and downsampled and then highpass filtered and downsampled column by column, which can form a column transformation matrix \({I}_{C}\). The matrix \({I}_{C}\) involves the four parts that point to one approximate part and three detail parts representing the horizontal, vertical and diagonal direction. Inverse decomposition of these 4 parts can be reconstructed to obtain the original signal. Figure 1 shows the firstorder Haarlike multiresolution decomposition and reconstruction of an image signal.
3.2 The image fusion rules
The three channels r, g and b of the source images \({I}_{P}\) and \({I}_{Q}\) are subjected to multiscale decomposition of level 1 by the Haarlike multiresolution analysis, respectively. The lowfrequency and the highfrequency subdomains are obtained from the three channels of each image.
3.2.1 The image fusion rule in lowfrequency field
The \(\kappa (\kappa =r,g or b)\) band of the images \({I}_{P}\) and \({I}_{Q}\) are decomposed at level 1 by the Haarlike multiscale analysis, respectively, to produce the lowfrequency subimages \({I}_{P\kappa }^{a}\) and\({I}_{Q\kappa }^{a}\). The \({L}_{2}\) norm of the matrix can be used for feature extraction in image processing, which is used for solving the square root of the sum of squares of all elements in the matrix to extract luminance information. Suppose the size of image\({I}_{P\kappa }^{a}\), \({I}_{Q\kappa }^{a}\) and \(I\) have the size of \(M\) row \(N\) column. The idea of the lowfrequency fusion rule \(LowfrqR({I}_{P\kappa }^{a} , {I}_{Q\kappa }^{a})\) is as follows.
Firstly, a series of 3 × 3 matrices centering on the \((i,j)\)th point of the image \(I\) are calculated by using the sliding window mode in order. A matrix of 3 × 3 is written as
The \({L}_{2}\) norm of the matrix of Eq. (5) is
Depending on Eq. (6), there are two pairs of \(M\times N\) \({L}_{2}\) norms of the matrices of 3 × 3 size produced from the subimages \({I}_{P\kappa }^{a}\) and\({I}_{Q\kappa }^{a}\), respectively, by using the sliding window mode. The \(M\times N\) \({L}_{2}\) norms from the subimage \({I}_{P\kappa }^{a}\) are arranged as the matrix \({M}_{P\kappa }^{L2}\) whereas the \(M\times N\) \({L}_{2}\) norms from the subimage \({I}_{Q\kappa }^{a}\) are arranged as the matrix \({M}_{Q\kappa }^{L2}\).
Secondly, the spatial frequency is often used to represent the spatial clarity information of an image. The spatial frequency value \({SF}_{P\kappa }\) is computed from the matrix \({M}_{P\kappa }^{L2}\) and the spatial frequency value \({SF}_{Q\kappa }\) is computed from the matrix \({M}_{Q\kappa }^{L2}\) according to Eq. (7) written as
\({SF}_{h}\) in Eq. (7) points to the mean of the squares sum of difference of the adjacent elements of the matrix \(I\) in horizontal direction, and \({SF}_{v}\) in Eq. (7) points to that in vertical direction. The \({SF}_{src}\) is the square root of the sum of the \({SF}_{h}\) and the \({SF}_{v}\). Here \({SF}_{src}\) is mentioned to \({SF}_{P\kappa }\) or \({SF}_{Q\kappa }\).
Thirdly, the 0 and 1 binary graph \({M}_{bin}\) is generated according to Eq. (8), in which the lowfrequency ratio \(T1\) is equal to 10.
Lastly, according to the value of the \((i,j)\)th pixel of the binary graph \({M}_{bin}\), it selects the appropriate data source for the \((i,j)\)th pixel of the fusion image, which can be defined as Eq. (9)
When \({M}_{bin}(i,j)\) is 1, \({I}_{F\kappa }^{a} (i,j)\) chooses the value from the lowfrequency subdomain \({I}_{P\kappa }^{a} (i,j)\). When \({M}_{bin}(i,j)\) is 0, \({I}_{F\kappa }^{a} (i,j)\) chooses the value from the lowfrequency subdomain \({I}_{Q\kappa }^{a} (i,j)\).
3.2.2 The image fusion rule in highfrequency field
There are three highfrequency subimages \({I}_{P\kappa }^{h}\) and\({I}_{Q\kappa }^{h}\), \({I}_{P\kappa }^{v}\) and \({I}_{Q\kappa }^{v}\), \({I}_{P\kappa }^{d}\) and \({I}_{Q\kappa }^{d}\) derived from the r band, the g band and the b band of the images \({I}_{P}\) and \({I}_{Q}\) by the decomposition of the first layer by the Haarlike multiscale analysis, respectively. For a linear system with small perturbations, the state variables of the system will change very little after a spatial step if the eigenvalue of its matrix is close to or equal to zero. The \(M\times N\) square sums based on the eigenvalue of the matrix of 3 × 3 size arise in turn from the image\(I\), which are centering on the \((i,j)\) th pixel of the image \(I\) by using the sliding window mode of Eq. (5). If this square sum is small and falls within a certain range, then it means that the change between one point and another point in the image is not very variable, and the edge features are not obvious. However, it can be considered as edge features for cases with large changes. The highfrequency fusion rule \(HighfrqR({I}_{P\kappa }^{\gamma } , {I}_{Q\kappa }^{\gamma })\) is described below, where \(\gamma\) is equal to a, h, or v.
The research makes Eq. (5) as \(B\) and the corresponding eigenvalue of \(B\) as \({\lambda }_{i}(1\le i\le 3)\) according to the eigenequation of 3order phalanx \(Bx=\lambda x\). The sum of the squares of these 3 eigenvalues is \({({\lambda }_{1})}^{2}\)+\({({\lambda }_{2})}^{2}\)+\({({\lambda }_{3})}^{2}\). The algorithm based on square sum of eigenequation is applied to the three pairs of highfrequency subdomains \({I}_{P\kappa }^{h}\) and \({I}_{Q\kappa }^{h}\), \({I}_{P\kappa }^{v}\) and \({I}_{Q\kappa }^{v}\), \({I}_{P\kappa }^{d}\) and \({I}_{Q\kappa }^{d}\) to get the three couples of the corresponding eigenequation matrices \({M}_{P\kappa }^{h}\) and \({M}_{Q\kappa }^{h}\), \({M}_{P\kappa }^{v}\) and \({M}_{Q\kappa }^{v}\), \({M}_{P\kappa }^{d}\) and \({M}_{Q\kappa }^{d}\).
where the lift ratio \(\beta\) ranges from 1.1 to 1.9 and the highfrequency ratio \(T2\) is 0.001 in the study.
Equation (10) is used to configure the appropriate data sources for the highfrequency domain of the fused image \({I}_{F\kappa }^{\gamma }\). \({I}_{F\kappa }^{\gamma } (i,j)\)) chooses the value from the product of \(\beta\) and the highfrequency subdomain \({I}_{P\kappa }^{\gamma } (i,j)\) when \({M}_{P\kappa }^{\gamma } (i,j)<T2\) and \({M}_{Q\kappa }^{\gamma } (i,j)<T2\), otherwise \({I}_{F\kappa }^{\gamma } (i,j)\)) chooses the value from the highfrequency subdomain \({I}_{Q\kappa }^{\gamma } (i,j)\).
3.3 The image fusion algorithm based on the Haarlike multiscale analysis
The Haarlike multiscale analysis strategy in Sect. 3.1 is applied to the image fusion of two images of the same scene, which is described in Fig. 2. The proposed image fusion scheme can be divided into the following steps.
Step One. Each channel is decomposed into one subimage of lowfrequency and three subimages of highfrequency by the \(HaarLdec()\) method, which is denoted as Eq. (11)
where \(\omega\) means the image source pointing to the image \(P\) or the image\(Q\); \(\kappa\) means the r channel, the g channel or the b channel of an image. \({I}_{Pr}, {I}_{Pg}\) and \({I}_{Pb}\) represent the r channel, g channel and b channel of the original image \({I}_{P}\); and \({I}_{Qr}, {I}_{Qg}\) and \({I}_{Qb}\) represent the r channel, g channel and b channel of the original image\({I}_{Q}\). \({I}_{\omega \kappa }^{a}, { I}_{\omega \kappa }^{h}, { I}_{\omega \kappa }^{v}\) and \({I}_{\omega \kappa }^{d}\) represent the approximate subimage, the horizontal subimage, the vertical subimage and the diagonal subimage when the \(\kappa\) channel of the image \(\omega\) is decomposed at the first level.
Step Two. It makes the fusion result be an image \({I}_{F}\). Its rchannel, gchannel and bchannel will carry out the corresponding fusion rules. The lowfrequency fusion rule of the approximate subimage \({I}_{F\kappa }^{a}\) is corresponding to the \(LowfrqR()\) method in Eq. (12). The frequency band coefficients are derived from the images \({I}_{P}\) and \({I}_{Q}\). The highfrequency fusion rules of the horizontal subimage \({I}_{F\kappa }^{h}\), the vertical subimage \({I}_{F\kappa }^{v}\) and the diagonal subimage \({I}_{F\kappa }^{d}\) are all corresponding to the \(HighfrqR(\)) method in Eq. (12). The three highfrequency bands coefficients are derived from the image \({I}_{P}\) multiplied by the lift coefficient [29] and image \({I}_{Q}\).
Step Three. The research generates and associates the three new channels to obtain the fusion image. Replying on Eq. (13), the three channels are reconstructed separately by the inverse Haarlike multiscale transformation, i.e. the \(HaarLrec()\) method.
The image \({I}_{Fr}\) is reconstructed from \({I}_{Fr}^{a}, { I}_{Fr}^{h}, { I}_{Fr}^{v}\) and \({I}_{Fr}^{d}\) after the reverse Haarlike multiscale decomposition; the image \({I}_{Fg}\) is reconstructed from \({I}_{Fg}^{a}, { I}_{Fg}^{h}, { I}_{Fg}^{v}\) and \({I}_{Fg}^{d}\) after the reverse Haarlike multiscale decomposition and the image \({I}_{Fb}\) is reconstructed from \({I}_{Fb}^{a}, { I}_{Fb}^{h}, { I}_{Fb}^{v}\) and \({I}_{Fb}^{d}\) after the reverse Haarlike multiscale decomposition. The channel image \({I}_{Fr}\), the channel image \({I}_{Fg}\), and the channel image \({I}_{Fb}\), are associated to get fused image \({I}_{F}\).
It can be found the following views from the above multiscale tower shape decomposition and reconstruction process by using the Haarlike multiscale. The multiscale analysis algorithm proposed in this article analyzes the image from the perspective of the frequency domain, and the image can be divided into the lowfrequency and highfrequency parts. The lowfrequency part reflects the body information of the image such as the outline of the object and the basic composition area. It can mostly extract sufficient color information derived from the source image \({I}_{Q}\). The highfrequency part reflects the detail information of the image such as the textures and edges of the objects. It can mostly extract sufficient boundary information derived from the source image \({I}_{P}\).
4 Simulation experiments
4.1 Experimental materials and qualitative analysis
To identify the universality of the fusion method proposed, this research provides two sets of medical images as experimental materials. The first set of medicine images is a pair of computed tomography (CT) image and positron emission computed tomography (PET) image, which describe the scene of the kidney organs of a patient. The second set of medical images is a pair of CT image and PET image showing the female pelvic tissue scene. These two sets of source images are presented in Fig. 3. Figure 3a, b represents the CT image and the PET image of the first dataset, respectively. Figure 3c, d represents the CT image and the PET image of the second medicine dataset, respectively. Figures 4 and 5 show the fused images corresponding to these two sets of source images. Eleven comparative fusion methods are used, which are principal component analysis (PCA), hue saturation vision (HSV), wavelet packet transformation (WPT), curvelet, contourlet, nonsubsampled contourlet transformation (NSCT), HSV + WPT, and HSV + NSCT. HSV + WPT means the WPT fusion method based on the HSV space; HSV + NSCT means the NSCT fusion method based on the HSV space. Additionally, there are three fusion methods from Refs. [30,31,32] which are named as DEMEF [30], NDFA [31], and PODFA [32].
There are twelve images fused displaying in Fig. 4 for the first group of data and in Fig. 5 for the second group of data, respectively. Figure 4a is corresponding to PCA fusion method for the ovarian cancer image; Fig. 4b is corresponding to HSV fusion method for the ovarian cancer image; Fig. 4c is corresponding to WPT fusion method for the ovarian cancer image; Fig. 4d is corresponding to Curvelet fusion method for the ovarian cancer image; Fig. 4e is corresponding to Contourlet fusion method for the ovarian cancer image; Fig. 4f is corresponding to NSCT fusion method for the ovarian cancer image; Fig. 4g is corresponding to HSV + WPT fusion method for the kidney organs image; Fig. 4h is corresponding to HSV + NSCT fusion method for the kidney organs image; Fig. 4i is corresponding to the DEMEF fusion method for the ovarian cancer image; Fig. 4j is corresponding to the NDFA fusion method for the ovarian cancer image; Fig. 4k is corresponding to the PODFA fusion method for the ovarian cancer image; Fig. 4l is corresponding to the fusion method represented for the ovarian cancer image. Figure 5a is corresponding to PCA fusion method for the female pelvic image; Fig. 5b is corresponding to HSV fusion method for the female pelvic image; Fig. 5c is corresponding to WPT fusion method for the female pelvic image; Fig. 5d is corresponding to Curvelet fusion method for the female pelvic image; Fig. 5e is corresponding to Contourlet fusion method for the female pelvic image; Fig. 5f is corresponding to NSCT fusion method for the female pelvic image; Fig. 5g is corresponding to HSV + WPT fusion method for the female pelvic image; Fig. 5h is corresponding to HSV + NSCT fusion method for the female pelvic image; Fig. 5i is corresponding to the DEMEF fusion method for the female pelvic image; Fig. 5j is corresponding to the NDFA fusion method for the female pelvic image; Fig. 5k is corresponding to the PODFA fusion method for the female pelvic image; Fig. 5l is corresponding to the fusion method represented for the female pelvic image.
Qualitative evaluation of the fusion effect of these two sets of images. Through Fig. 4a–l, the texture sharpness of the fusion images is compared and analyzed. Figure 4a has distinct lines, but compared with the PET source image, its spectral distortion is relatively large. Figure 4b has the lowest texture sharpness and the largest spectral distortion. The fusion effects of Fig. 4d–g are very similar to Fig. 4h. In this set of images, they are clearly drawn but close to the PET source image. Figure 4c looks similar to Fig. 4l. They have the highest texture sharpness and are closest to the PET source image in spectral terms in Fig. 4. Obviously, Fig. 4i–k look similar and they have the poor spectrum effects. The second set of pelvic images are shown in Fig. 5a–l. The texture clarity from Fig. 5a of PCA is obvious, and so is Fig. 5b of HSV. They have the most distorted colors compared to the other seven images in Fig. 5. The visual sense of the three images involving Fig. 5d, e, g are quite similar. They have achieved a medium level in both spatial information and color information in the second group of fusion images. Compared to the other multiplescale fusion images in Fig. 5, the spatial clarity of the fusion Fig. 5g of HSV + WPT and the fusion Fig. 5h of HSV + NSCT is not prominent enough. But their color distortion is relatively low. The scenes the three images of Fig. 4i–k all have a serious spectrum loss. In terms of clarity and color information, the fusion effect of Fig. 5c by using wavelet packet is close to that of Fig. 5l by using the proposed fusion method. They not only have high definition, but also have high color depth information, especially Fig. 5l closest to Fig. 3d of the PET source image.
4.2 Quantization criteria
The quantitative evaluation has be done in a consistent way. A group of basic quantitative indicators such as average gradient (AG), space frequency (SF), standard derivation (StD), mean value (MV), overall performance (OP), mutual information (MI), spectrum distort (SpD), correlation ratio (CR) and structural similarity (SSIM), has been widely used in the image fusion area [33,34,35]. AG criterion indicates the space definition of an image, and the quality of an image is proportion to its value. SF also reflects the spatial definition information of the fused image in which the larger its value indicates, the better the definition is. MV criterion indicates the gray mean value of an image, which represents the color effect of the fusion image. When its MV value get closer to that of the source color image, the better its spectral fusion is. StD criterion indicates the degree of dispersion between the pixel value and the mean, and the larger the standard deviation is, the better the quality of the image is.
OP criterion indicates the rationality of the corresponding method [34]. OP = Σ_{κ}(AG_{κ} − SpD_{κ})/3 = AGSpD, where κ = r, g, b. When the OP value is positive, the comprehensive performance evaluation combining other reasonable index values and its method should be recommended. MI criterion indicates the relationship between the fused image and the source images. SpD criterion indicates how much color component is lost from the original color image for the fused image. CR criterion indicates how many source color features of the color image are acquired from the fusion image. The larger its value is, the more color information the fusion image obtains from the source color image. SSIM criterion indicates the similarity between the fused image and the source color image. The larger its value is, the higher their similarity is, which means the more structured information is obtained from the source color image. The index values of the AG, SF, StD and MV from the two groups of original images are shown in Table 1. The performances merged by using the various methods for the first and the second set of data are shown in Table 2 and 3.
4.3 Quantization evaluation
By observing the index values of the twelve fusion images of the first set of the kidney organs data, it is easy to find the following aspects. The WPT fusion method yielded the best spatial resolution which indicates an average gradient value of 10.8598 and a spatial frequency value of 22.9335 whereas the fusion method proposed in the study gets the second spatial resolution which indicates an average gradient value of 9.2437 and a spatial frequency value of 20.2221. The OP value of the fusion method proposed is 2.7240, being the first OP value of various methods, whereas the OP value of WPT is 2.1412, being the second OP value of various methods. The MI value of the HSV fusion method is 4.1398, being the first MI value of various methods, whereas the MI value of the fusion method proposed is 3.2787, being the third MI value of various methods. Finally, please look at the spectral fusion feature which corresponds to the three criteria: SpD, CR, and SSIM. The color features of the fusion method proposed have the best results, with the lowest spectral distortion SpD value of 6.2316, the maximum correlation coefficient value of 0.9684 and the maximum SSIM value of 0.9687. The fusion image performance of the other seven methods is far less than the above two methods for both spatial features and color features.
Similarly, the various index values of the second set of the twelve pelvic fusion images can be concluded as follows. According to Table 3, compared with the other 8 fusion methods, the fusion effect of the fusion method proposed is absolutely dominant. On one hand, the AG value of Fig. 5l is 7.5683, seconding only to the AG value of 7.6579 in Fig. 5c; the SF value of Fig. 5l is 18.6898, seconding only to the SF value of 18.9065 in Fig. 5a. In contrast, Fig. 5a has the most serious color loss, while Fig. 5c is nowhere behind Fig. 5l in terms of the other six indicators except AG. On the other hand, Fig. 5l of the fusion method proposed tops the list in terms of these criteria, such as OP, MI, SpD, CR and SSIM. The OP value of 2.7240, the MI value of 4.1673, the SpD value of 4.8443, the CR value of 0.9882 and the SSIM value of 0.9882.
PCA and HSV all belong to the spatial domain fusion methods. This type of fusion methods easily leads to severe color loss in the fusion images, especially for the kidney organs image. The remaining six conventional methods are all fusion methods based on multiscale transformation domains. WPT uses a 2layer wavelet packet to decompose the original image. Its wavelet packet basis function uses reverse biorthogonal wavelet instead of Haar wavelet, in which the former works better than the latter. Compared with other traditional methods, the fusion results are the best in respect of spatial characteristics and color depth. HSV + WPT performs a 2level wavelet packet decomposition based on the interconversion of the RGB space and the HSV space for the image and its wavelet packet basis function also uses reverse biorthogonal wavelet. The fusion method is slightly less effective than the WPT method. The fusion effects of these three superwavelet methods including Curvelet, Contourlet and NSCT are progressive, and they are not very different from each other. All three methods have close parameter values and are at intermediate levels among the eleven conventional methods. HSV + NSCT is the transformation by using NSCT based on the interconversion between the RGB space and the HSV space for the image. The fusion effect is somewhere between the Curvelet method and the HSV + WPT method.
In terms of spatial resolution and color resolution, the overall fusion performances of these 9 methods can be divided into five grades from general to excellent. The fusion effect of the HSV method and the PCA method is at the 5th level; the fusion effect of the Curvelet method, the Contourlet method and the NSCT method is at the 4th level; the fusion effect of the HSV + NSCT method and the HSV + WPT method is at the 3th level; the fusion effect of the WPT method is at the 2th level; the fusion effect of the proposed method is admirable, which is at the 1th level.
5 Conclusions
This study proposes a Haarlike multiscale transformation method based on multiscale analysis and Haar wavelet theory, which is applied for two kinds of image fusion occasions involving the kidney organs image fusion and the female pelvic image fusion, and achieves good results. From the fusion experiments of two sets of images, the represented fusion method gets better performance in both spatial performance and color characteristics, especially in terms of color effects. The kidney organs fusion image can maximize the extraction of spectral information of the source PET image, while the female pelvic fusion image can retain as much color depth information in the source color PET image as possible. The kidney organs images are usually fused for a specific application. It is common to merge the threeband high resolution CT image with the multiband low resolution PET image. The goal is to generate a highquality image that combines highresolution features of CT image and spectral features of PET image to facilitate various analysis and processing. Although ordinary black and white CT images are able to highly distinguish bone from soft tissue, they have low measurable ability between different soft tissues. By merging CT image and PET image, ordinary CT and PET images are upgraded to color spectral medical images. It can effectively reduce the difference of the imaging contrast. Full fusion of spatial information and spectral information in source images can provide reliable guarantee for the further processing of unmixing, segmentation and classification of the medicine images. It is a more delicate, effective, accurate and rich performances of the image information inside the human body and the lesion area, which exceeds the traditional CT with PET images in quality. The improvement of the diagnosis rate and the further cognition of the disease provide the color medical image, especially for the diagnosis of tumor diseases.
Data availability
The data is used to support the findings of this study are available from the corresponding author upon request.
Abbreviations
 PCA:

Principal component analysis
 IHS:

Intensity hue saturation
 CT:

Computed tomography
 PET:

Positron emission computed tomography
 HSV:

Hue saturation vision
 WPT:

Wavelet packet transformation
 NSCT:

Nonsubsampled contourlet transformation
 AG:

Average gradient
 SF:

Space frequency
 StD:

Standard derivation
 MV:

Mean value
 OP:

Overall performance
 MI:

Mutual information
 SpD:

Spectrum distort
 CR:

Correlation ratio
 SSIM:

Structural similarity
References
X. Jin, D. Zhou, S. Yao, R. Nie, Multifocus image fusion method using SPCNN optimized by particle swarm optimization. Soft. Comput. 22, 6395–6407 (2018)
T. Zhou, Qi. Li, Lu. Huiling, Q. Cheng, X. Zhang, GAN review: models and medical image fusion applications. Inf. Fusion 91, 134–148 (2023)
H. Kaur, D. Koundal, V. Kadyan, Image fusion techniques: a survey. Arch. Comput. Methods Eng. 28, 1–23 (2021)
Y. Quan, Y. Tong, W. Feng, G. Dauphin, W. Huang, W. Zhu, M. Xing, Relative total variation structure analysisbased fusion method for hyperspectral and LiDAR data classification. Remote Sens. 13(1143), 1–29 (2021)
B. Ajisha, A.K. Nisa, Survey on remote sensing image fusion. AIP Conf. Proc. 2222, 030027 (2020)
Q. Zhu, H. Li, H. Ye, Z. Zhang, R. Wang, Z. Fan, D. Zhang, Incomplete multimodal brain image fusion for epilepsy classification. Inf. Sci. 582, 316–333 (2022)
Q. Cao, Multimodal Medical Image Fusion Algorithm and Its Application in Radiotherapy. A Doctoral Dissertation Submitted to Southeast University (2021)
R. Zhu, Research on Some Problems of Multimodal Medical Image Fusion. A Doctoral Dissertation Submitted to JiLin University (2021)
G. Pajares, J.M. de la Cruz, A waveletbased image fusion tutorial. Pattern Recognit. 37, 1855–1872 (2004)
A. Krishn, V. Bhateja, A. Sahu, Medical image fusion using combination of PCA and wavelet analysis, in 2014 International Conference on Advances in Computing,Communications and Informatics (ICACCI) (2014), pp. 986–991
X. Zhu, W. Bao, Investigation of remote sensing image fusion strategy applying PCA to wavelet packet analysis based on IHS transform. J. Indian Soc. Remote Sens. 47(3), 413–425 (2019)
H. Hermessi, O. Mourali, E. Zagrouba, Multimodal medical image fusion review theoretical background and recent advances. Signal Process. 183, 2–27 (2021)
S.Y. Yang, M. Wang, L.C. Jiao, R.X. Wu, Z.X. Wang, Image fusion based on a new contourlet packet. Inf. Fusion 11, 78–84 (2010)
W. Bao, W. Wang, Y. Zhu, Pleiades satellite remote sensing image fusion algorithm based on shearlet transform. J. Indian Soc. Remote Sens. 46, 1–11 (2017)
Wu. Zhiliang, Y. Huang, K. Zhang, Remote sensing image fusion method based on PCA and curvelet transform. J. Indian Soc. Remote Sens. 46(5), 687–695 (2018)
Z. Zhu, M. Zheng, G. Qi, Di. Wang, Y. Xiang, A phase congruency and local Laplacian energy based multimodality medical image fusion method in NSCT domain. IEEE Access 27, 20811–20823 (2019)
J. Du, W. Li, B. Xiao et al., Union Laplacian pyramid with multiple features for medical image fusion. Neuro Comput. 194, 326–339 (2016)
Yu. Tian, W. Yang, Ji. Wang, Image fusion using a multilevel image decomposition and fusion method. Appl. Opt. 60(24), 7466–7479 (2021)
Z. Wang, Y. Ma, Z. Cui, Medical Image fusion based on guided filtering and sparse representation. J. Univ. Electron. Sci. Technol. China 51(2), 264–273 (2022)
W. Tan, W. Thitøn, P. Xiang, H. Zhou, Multimodal brain image fusion based on multilevel edgepreserving filtering. Biomed. Signal Process. Control 64, 102280 (2021)
G. Li, Y. Lin, Qu. Xingda, An infrared and visible image fusion method based on multiscale transformation and norm optimization. Inf. Fusion 71, 109–129 (2021)
J. Chen, K. Wu, Z. Cheng et al., A saliencybased multiscale approach for infrared and visible image fusion. Signal Process. 182(107936), 1–12 (2021)
S. Polinati, D.P. Bavirisetti, K.N. Rajesh, R. Dhuli, Multimodal medical image fusion based on contentbased decomposition and PCAsigmoid. Curr. Med. Imaging 18(5), 546–562 (2022)
W. Kong, Q. Miao, R. Liu, Y. Lei, J. Cui, Q. Xie, Multimodal medical image fusion using gradient domain guided filter random walk and side window filtering in framelet domain. Inf. Sci. 585, 418–440 (2022)
E. Guariglia, S. Silvestrov, Fractionalwavelet analysis of positive definite distributions and wavelets on D′(C), in Engineering Mathematics II, ed by R. Silvestrov (Springer, 2016), pp. 337–353
F. Pedroso, A. Furlan, R.C. Conteras, L.G. Caobianco, J.S. Neto, R. Guido, CWT x DWT x DTWT x SDTWT: clarifying terminologies and roles of different types of wavelet transforms. Int. J. Wavelets Multiresolut. Inf. Process. (2020). https://doi.org/10.1142/S0219691320300017
E. Guariglia, R.C. Guido, Chebyshev wavelet analysis. J. Funct. Spaces 2022(1), 5542054 (2022)
R.C. Guido, Wavelets behind the scenes: practical aspects, insights, and perspectives. Phys. Rep. 985, 1–23 (2022)
W. Bao, X. Zhu, A novel remote sensing image fusion approach research based on HSV space and Biorthogonal wavelet packet. J. Indian Soc. Remote Sens. 43(3), 467–473 (2015)
Q. Wang, W. Chen, X. Wu, Z. Li, Detailenhanced multiscale exposure fusion in YUV color space. IEEE Trans. Circuits Syst. Video Technol. 30(8), 2418–2429 (2020). https://doi.org/10.1109/TCSVT.2019.2919310
M. Jana, S. Basu, A. Das, NSCTDCT based Fourier analysis for fusion of multimodal images, in 2021 IEEE 8th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON), Dehradun, India (2021), pp. 1–6. https://doi.org/10.1109/UPCON52273.2021.9667618
M. Jana, S. Basu, A. Das, Fusion of multimodal images using parametrically optimized PCNN and DCT based Fourier analysis, in 2022 IEEE Delhi Section Conference (DELCON), New Delhi, India (2022), pp. 1–7.https://doi.org/10.1109/DELCON54057.2022.9753411
Z. Wang, A.C. Bovik, H.R. Sheikh et al., Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
S. Daneshvar, H. Ghassemian, MRI and PET image fusion by combining IHS and retinainspired models. Inf. Fusion 11, 114–123 (2010)
R. Zhu, X. Li, X. Zhang, J. Wang, HID The hybrid image decomposition model for MRI and CT fusion. J. Biomed. Health Informat. 26(2), 727–739 (2022)
Acknowledgements
The authors would like to acknowledge all the participants for their contributions to this research.
Funding
This work is sponsored by Natural Science Foundation of Xinjiang Uygur Autonomous Region (Grant No. 2022D01C425), the Postgraduate Course Scientific Research Project of Xinjiang University (Grant No. XJDX2022YALK11).
Author information
Authors and Affiliations
Contributions
All the authors contribute equally in data collection, processing, experiments, and article writing. The authors read and approved the final manuscript.
Author's information
Xiaoliang Zhu is currently an associate professor at Xinjian University. Zhu received the PhD degree from Ningxia University in Computational Mathematics in 2020 and qualified as a senior engineer in 2014. Zhu graduated with a master's degree in North Minzu University in 2015. The research interest focuses on digital image process, Artificial Intelligence, and numerical solution of the partial differential equations.
Mengke Wen graduated from Xinjiang Medical University for a bachelor's degree in 2013 and a master's degree in 2015, respectively. In 2015, he worked at the Cancer Hospital affiliated with Xinjiang Medical University, with specializing in gynecological oncology.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Zhu, X., Wen, M. Image fusion research based on the Haarlike multiscale analysis. EURASIP J. Adv. Signal Process. 2024, 23 (2024). https://doi.org/10.1186/s13634024011182
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13634024011182