Multi-modal image matching based on local frequency information

Image matching that aims to find the corresponding features or image patches between two images of the same scene is often a fundamental issue in computer vision. It has been widely used in vision navigation [1], target recognition and tracking [2], super-resolution [3], 3-D reconstruction [4], pattern recognition [5], medical image processing [6], etc. In this paper, we focus on the issue of matching for multi-modal (or multi-sensor) images that differ in relation to the type of visual sensor. There are many important issues that make multi-modal image matching a very challenging problem [7]. First, multi-modal images are captured using different visual sensors (e.g. SAR, optical, infrared, etc.) at different time. Second, images with different modalities are normally mapped to different intensity values. This makes it difficult to measure similarity based on their intensity values since the same content may be represented by different intensity values. The problem is further complicated by the fact that various intrinsic and extrinsic sensing conditions may lead to image non-homogeneity. Finally, the disparity between the intensity values of multi-modal images can lead to coincidental local intensity matches between non-corresponding content, which may make the algorithm difficult to search the correct solution. Hence, the focuses of multi-modal image matching reside in illumination (contrast and brightness) invariant representations, common structure extraction from varying conditions and robust similarity measure.

The existing approaches for multi-modal image matching can be generally classified as feature-based and region-based. Feature-based matching utilizes extracted features to establish correspondence. Interest points [8, 9], edges [10], etc. are often used as the local features because of their robustness in extraction and matching. In [8], Scale Invariant Feature Transform (SIFT) and cluster reward algorithm (CRA) [11] are used to match multi-modal remote sensing images. The SIFT operator is first adopted to extract feature points and perform coarse match, and then the CRA similarity measure is used to achieve accurate correspondence. In [10], Yong et al. propose the algorithm for multi-source image matching based on information entropy which comprehensively considers of the intensity information and the edge direction information. For feature-based methods two requirements must be satisfied: (i) features are extracted robustly and (ii) feature correspondences are established reliably. Failure to meet either of them will cause this type of method to fail. In contrast to feature-based methods, region-based methods make use of the whole image content to establish correspondence. While most approaches use features for image matching, there is also a significant amount of work on region-based matching. In [12], local phase-coherence representation is constructed for multi-modal image matching. This representation has some merits that make it a promising candidate for handling situations where non-homogeneous image contrast exists: (i) it is relatively insensitive to the level of signal energy; (ii) it depends on the structures in the image and can emphasize the edges and ridges at the same time; and (iii) it has a good localization in the spatial domain. In [13], M. Irani et al. present an energy-image representation based on directional-derivative filters. A set of filters, oriented in the horizontal, vertical, and the two diagonal directions, are applied to the raw image, and then the derivative image is squared to get an “energy” image. Thus, the directional information is preserved in this energy representation. This approach, however, requires explicit directional filters and explicit filtering with Gaussian functions to create a pyramid. In addition, mutual information that has been commonly used and showed great promise in medical image processing is often adopted as the similarity measure for multi-modal image matching since it is insensitive to variation of intensities and doesn’t require knowledge of the relationship (joint intensity distribution) of the two different modalities [14, 15]. The main merit of region-based method is their ability of resistance against noise and texture distortions since abundant information can be adopted by using a relatively large template, and thus providing a high matching accuracy.

In this paper, we bring forward a local frequency information-based matching frame for multi-modal images. It takes advantage of the merits of both MLPA and FSPC by using the CAS, and can be used to match images captured by similar as well as different types of sensors at different time.

The visual system of human can reliably recognize the same object/scene under widely varying conditions. If the illumination of a scene is changed by several orders of magnitude, our interpretation for it can keep unchanged largely. Thus, in the image matching the main form of invariance is invariance to illumination, this is particularly important for multi-modal images where non-homogeneous contrast and brightness variation frequently occur. In this work, the local frequency information is used to construct image representations namely FSPC and MLPA, which are both dimensionless and invariant to non-homogeneous illumination variation and contrast reversal, for multi-modal image matching.

2.2. Local frequency representations

In the search for invariant quantities in multi-modal images, the proposed approach is to take advantage of information from the frequency domain, rather than spatial domain. Let I denote the signal, and LG _n,θ ^e and LG _n,θ ^o denote the even-symmetric and odd-symmetric component of Log-Gabor function at the scale n and orientation θ. The response vector formed by the responses of each quadrature pair of filters can be expressed as

$\left[e_{n,\theta }\left(x\right),o_{n,\theta }\left(x\right)\right]=\left[I\left(x\right)*L{G}_{n,\theta }^e,I\left(x\right)*L{G}_{n,\theta }^o\right]$

(4)

The values e _n,θ (x) and o _n,θ (x) can be regarded as real and imaginary parts of complex valued frequency component. The amplitude of the response vector at the scale n and orientation θ is given by

$A_{n,\theta }\left(x\right)=\sqrt{e_{n,\theta }{\left(x\right)}^2+o_{n,\theta }{\left(x\right)}^2}$

(5)

and the phase is given by

${\varphi }_{n,\theta }\left(x\right)=atan2\left(o_{n,\theta }\left(x\right),e_{n,\theta }\left(x\right)\right)$

(6)

At each location x of a signal, we will have an array of these response vectors (each vector corresponds to one scale and orientation of filter). The response vectors form the basis of the proposed representations. The MLPA can be calculated as follow:

$\begin{array}{l}\mathit{MLPA}\left(x\right)=\{\begin{array}{l}\frac{atan2\left(F\left(x\right),H\left(x\right)\right)}{\pi }\times 255,\hfill \\ \frac{\pi +atan2\left(F\left(x\right),H\left(x\right)\right)}{\pi }\times 255,\hfill \end{array}\hfill \\ \mathit{if}atan2\left(F\left(x\right),H\left(x\right)\right)\ge 0;\\ \mathit{if}atan2\left(F\left(x\right),H\left(x\right)\right)<0.\end{array}$

(7)

where F(x) and H(x) can be calculated by summing the even and odd filter convolutions:

$F\left(x\right)={\displaystyle \sum _{\theta }{\displaystyle \sum _n{e}_{n,\theta }}}\left(x\right)$

(8)

$H\left(x\right)={\displaystyle \sum _{\theta }{\displaystyle \sum _n{o}_{n,\theta }}}\left(x\right)$

(9)

Contrast-reversal that may occur between the multi-modal images (e.g. Figure 1b) is eliminated by transferring the orientation of the mean local frequency vector [F(x), H(x)] that locates at the third/fourth quadrant (where αtan2(F(x),H(x))<0) to the first/second quadrant (where αtan2(F(x),H(x))≥0). Each value of MLPA, which is independent of the overall energy of the signal, is a measure of mean local phase angle. Hence, all MLPA maps have the same units, and are invariant to both scale and offset illumination changes (e.g. Figures 2 and 3). The main goal of MLPA is to eliminate the variation of intensity values between corresponding pixels of multi-modal image pair by using the phase information of local frequency. For a sophisticated matching algorithm, an outlier rejection mechanism is normally necessary since in many situations there are more “outliers” (non-common scene) than “inliers” (common scene) between multi-modal images. However, only by MLPA one cannot identify those inliers and eliminate the influence of the outliers. Hence, in this work the FSPC that aims to capture the common scene information while suppressing the illumination- and sensor-dependent information is developed by using the amplitude information of local frequency.

For multi-modal images, the signals are correlated primarily in high-frequency information, while correlation between the signals tends to degrade with the reduction of high-frequency information [13]. This is because high-frequency information (e.g. edge, contour, corner, junction, etc.) normally corresponds to the physical structure that is common to images with different modalities. On the other hand, low-frequency information depends heavily on the illumination and the photometric and physical imaging properties of sensors, and these are substantially different in multi-modal images. To capture the common physical structure, the high-pass filters (e.g. Sobel, Prewitt, Laplacian, etc.) that are working in spatial domain are reasonably adopted [10, 13]. Those methods are straightforward and quite fast to compute. However, they normally depend on the intensity gradient information which highly relates with local image contrast, and therefore the non-homogeneous variation of contrast may degrade the performance of algorithm.

Working in frequency domain, phase congruency theory postulates that the structural information can be perceived at points where the local frequency components are maximally in phase, rather than assumes it is a point of maximal intensity gradient. The measure of phase congruency at a point x in a signal proposed by Morrone et al. in [18, 19] can be expressed as

$P{C}_1=\frac{E\left(x\right)}{{\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }}}}=\frac{\sqrt{F^2\left(x\right)+H^2\left(x\right)}}{{\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }}}}$

(10)

where E(x) denotes the energy that is the magnitude of a vector sum. As we can see in the definition formula, phase congruency is the ratio of the energy E(x) to the overall length taken by the local frequency components in reaching the end point. If all the local frequency components are in phase, all the response vectors would be aligned and the value of phase congruency, PC₁, would be a maximum of 1. If there is no coherent of phase, the value of PC₁ falls to a minimum of 0. Phase congruency is a quantity that is independent of the overall magnitude of the signal making it invariant to variation of image brightness and contrast.

Clearly, phase congruency is only significant if it occurs over a wide range of frequencies (phase congruency over many spectrums is more significant than phase congruency over narrow spectrums). Thus, as a measure of feature significance, phase congruency should be weighted by the frequency spread. To address the problem of the conventional phase congruency [18, 19], we present a novel FSPC by using a weighing function that devalues the phase congruency at locations where the spread of filter responses is narrow. A measure of frequency spread can be defined as

$s\left(x\right)=\frac{1}{\sqrt{N}}\left(\frac{{\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }\left(x\right)}}}{\sqrt{{\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }^2\left(x\right)}}}+\epsilon }\right)$

(11)

where N denotes the total number of filters, and ε is used for avoiding division by zero and discounting the result when both ${\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }\left(x\right)}}$ and $\sqrt{{\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }^2\left(x\right)}}}$ are very small. The value of spread function, s(x), varies between 0 and 1. If the distribution of filter responses is uniform over all spectrums, s(x), reaches its maximum value of 1. The frequency spread weighing function can be constructed by applying a hyperbolic tangent function to the filter response spread value,

$W\left(x\right)=\frac{1}{2}\left(1+tanh\left(\frac{\lambda }{2}\left(s\left(x\right)-c\right)\right)\right)$

(12)

where c is the “cut-off” value, below which the value of phase congruency will be penalized, and λ is a gain factor that controls the sharpness of c. Thus, the definition of FSPC can be given as

$\mathit{FSPC}\left(x\right)=\frac{W\left(x\right)E\left(x\right)}{{\displaystyle \sum _{\theta }{\displaystyle \sum _n{A}_{n,\theta }\left(x\right)+\epsilon }}}\times 255$

(13)

Weighting by frequency spread has benefit of reducing those ill-conditioned responses that have the low frequency spread, as well as improving the localization accuracy of features, especially the smoothed features whose responses are normally uniform [20]. In addition, the noise resistance is also improved to some extent since the responses of noise are normally skewed to the high frequency end, and therefore have the relatively narrow frequency spectrums.

Having obtained the local frequency representations, we then use them to perform matching operations. As we can see from the definitions of local frequency representations, MLPA primarily represents the phase information of local frequency, whereas FSPC mainly utilizes the amplitude information, which means MLPA and FSPC can be compensated each other to some extent since information independence. Hence, by using some proper fusion scheme that makes best use of the merits of MLPA and FSPC, one can achieve better matching performance. For example, the only use of MLPA may induce errors particularly in the texture-less image regions where FSPC normally has quite small value since the lack of significant features. In addition, it may be difficult to distinguish between two search windows that have similar MLPA but different FSPC.

where d = − |MLPA ₁ − MLPA ₂|, $c=\frac{1}{2}\left(\mathit{FSP}{C}_1+\mathit{FSP}{C}_2\right)$. d is the similarity component that reflects how well the two signals resemble each other, and c is the confidence component that reflects the confidence that a match is correct.

MLPA with low FSPC is normally less reliable than those with high FSPC. Therefore, it is important to give more confidence to the higher FSPC. In fact, the confidence component is the mean value of the two FSPCs, so the confidence highly relates with the significance of signals and will be given a larger value when both signals are significant. Hence, CAS ₀ is normally given a relatively large value when the two pixels are both similar and significant and a relatively small value when they are not.

This measure returns 0 when the matching windows are identical. The denominator, C, is in fact related to the confidence component. For a same value of similarity D, the definition of CAS indicates a similarity is larger as the associated confidence components are high. It is apparent that CAS is invariant for the global linear illumination transformations: I→αI+b.

To achieve robust multi-modal image match, we first present two image representations—FSPC and MLPA based on the Log-Gabor wavelet transformation, and then design the CAS that combines confidence and similarity by using the information of FSPC and MLPA to find the correspondence. The proposed method has three main merits: (1) both MLPA and FSPC keep invariant for non-homogeneous illumination (contrast, brightness) variation and contrast reversal that frequently occur between multi-modal images; (2) FSPC can effectively capture the common scene structural information while suppressing the non-common sensor-dependent properties; (3) As the confidence factor, the structural information extracted by FSPC can be allocated more weighting softly by CAS. In addition, the proposed method is threshold-free, and therefore can retain as much image detail information as possible to resist noise influence and scene distortions between images. Experiments using numerous real and synthetic images demonstrate that our method can match multi-modal images robustly. Through comparison experiments, we also demonstrate the advantage of our method over the conventional methods. In the future, we plan to introduce the geometric transformation into our matching frame, and extend our method to image alignment.