Recent multiscale AM-FM methods in emerging applications in medical imaging

Abstract

Amplitude-modulation frequency-modulation (AM-FM) decompositions represent images using spatially-varying sinusoidal waves and their spatially-varying amplitudes. AM-FM decompositions use different scales and bandpass filters to extract the wide range of instantaneous frequencies and instantaneous amplitude components that may be present in an image. In the past few years, as the understanding of its theory advanced, AM-FM decompositions have been applied in a series of medical imaging problems ranging from ultrasound to retinal image analysis, yielding excellent results. This article summarizes the theory of AM-FM decompositions and related medical imaging applications.

1. Introduction

In the field of computer aided detection and diagnostics (CAD), recent advances in image processing techniques have brought a wide array of applications into the field. Many existing CAD methods rely on fixed basis functions based on wavelet decompositions [1] and Gabor filters [2]. Amplitude-Modulation Frequency-Modulation (AM-FM) methods [3–6] represent an emerging technique that shows great promise in this area.

Multidimensional AM-FM models and methods provide us with powerful, image and video decompositions that can effectively describe non-stationary content. They represent an extension to standard Fourier analysis, allowing both the amplitude and the phase functions to vary spatially over the support of the image, following changes in local texture and brightness.

To explain some of the advantages of AM-FM methods, we begin with the basic AM-FM model. In the 2D model, we expand an input image I(x, y) into a sum of AM-FM harmonics using:

$$I\left(x,y\right)=\sum _{n=1}^M{a}_n\left(x,y\right)\text{cos}{\phi }_n\left(x,y\right),$$

(1)

where a _n (x, y) denote slowly-varying instantaneous amplitude (IA) functions, φ _n (x, y) denote the instantaneous phase (IP) components, and n = 1, 2,..., M indexes the different AM-FM harmonics. In (1), the n th AM-FM harmonic is represented by a _n (x, y) cos φ _n (x, y).

With each phase function, the instantaneous frequency (IF) vector field is defined by $\nabla {\phi }_n\left(x,y\right)$. Here, the AM-FM demodulation problem is defined as one of determining the IA, IP, and IF functions for any given input image.

AM-FM decompositions provide physically meaningful texture measurements. Significant texture variations are captured in the frequency components. For single component cases, IF vectors are orthogonal to equi-intensity lines of an image, while the IF magnitude provides a measure of local frequency content. In (1), by using AM-FM components from different scales, we can produce IF vectors from different scales, at a pixel-level resolution [4, 5].

Since AM-FM texture features are provided at a pixel-level resolution, AM-FM models can be used to segment texture images that are difficult to model with the standard brightness-based methods [7]. On the other hand, using just histograms of the IF and IA, we can design effective content-based image retrieval systems using very short image feature vectors [8, 9].

In summary, the advantages of AM-FM methods include [10]: (i) they provide a large number of physically meaningful texture features, over multiple scales, at a pixel-level resolution, (ii) the image can be reconstructed from the AM-FM decompositions, (iii) based on the target application, different AM-FM decompositions using different frequency coverage can be designed, and (iv) very robust methods for AM-FM demodulation have been recently developed (see some recent examples in [4]).

AM-FM decompositions can be extended to represent videos using:

$$I\left(x,y,t\right)=\sum _{n=1}^M{a}_n\left(x,y,t\right)\text{cos}{\phi }_n\left(x,y,t\right),$$

(2)

where each AM-FM function has been extended to be a function of both space and time. The original phase-based modeling approach was provided in [11] and was recently extended and improved in [12, 13].

AM-FM decompositions can also be used to reconstruct the input images, allowing us to evaluate their effectiveness on different parts of the image. For continuous-space image decompositions, AM-FM reconstruction examples can be found in [14], while [4, 5] give several recent, robust multiscale examples for both images and videos. AM-FM transform examples were shown in [15], while multidimensional orthogonal FM transforms were demonstrated in [16].

An early example of the use of frequency-domain filtering to target a particular application can be found in the fingerprint examples in [17]. More recently, [18] provided a tree-growth application, where inter-ring spacing was used to design filterbanks that cover a specific part of the spectrum, so as to recover tree ring and tree growth structure from very noisy image inputs. For general images, Gabor filterbank approaches were investigated in [3]. Similarly, for general images and videos, dyadic, multiscale decompositions were introduced in [4].

A summary of AM-FM methods is provided in Section 2. Results are presented for several medical imaging applications in Section 3. Finally, conclusions and future work are presented in Section 4.

2. Methods

2.1. Multidimensional AM-FM demodulation methods

Recent interest in the development of multidimensional AM-FM methods can be traced to early work on speech signal models by Jim Kaiser. In early work, Dr. Kaiser developed on algorithms for estimating the energy of ID signals in [19–21]. This led to the development of the ID and 2D energy separation algorithm (ESA) as described in [22–24]. Maragos et al. continued with the ID AM-FM work previously mentioned for ID application in [22, 23, 25, 26]. Early work on multidimensional energy separation methods appeared in [27]. AM-FM demodulation based on Gabor wavelets appeared in [28, 29]. Early research on the use of multidimensional energy operators continued in [24, 30]. In what follows, we begin with an introduction to multidimensional energy operators.

In 2D, image energy is estimated using the Teager-Kaiser operator given by [30]:

$$\Psi \left\{I\right\}\left(x,y\right)\equiv {∥\nabla I\left(x,y\right)∥}^2-I\left(x,y\right){\nabla }^{2}I\left(x,y\right),$$

(3)

where ∇² = ∂²/∂x² + ∂²/∂y² denotes the Laplacian operator. The IA and IF estimates are obtained using:

$${\widehat{\phi }}_x\left(x,y\right)\approx \sqrt{\frac{\Psi \left\{\partial I/\partial x\right\}}{\Psi \left\{I\right\}}},$$

(4)

$$\left|a\left(x,y\right)\right|\approx \frac{\Psi \left\{I\right\}}{\sqrt{\Psi \left\{\partial I/\partial x\right\}+\Psi \left\{\partial I/\partial y\right\}}},$$

(5)

where ${\widehat{\phi }}_y$, is estimated by replacing the x-derivative by the y-derivative in (4). To eliminate sign ambiguities, the Teager-Kaiser operator can select the candidate IF for which the image gives the largest projection (e.g., see [31]). Recently, Kokkinos et al. showed a related accurate demodulation method using energy operators in [32] by computing all necessary derivatives by convolving with derivatives of Gabor filters, as opposed to using finite differences. The IA estimates are corrected by dividing by the magnitude response of the Gabor filter at the estimated IF (see [32] for details).

Analytic image methods for AM-FM demodulation are based on providing a Hilbert-based extension of the ID Hilbert-based demodulation approach. Here, the basic idea is to simply apply the ID Hilbert operator along the rows (or the columns). The fundamental advantage of this approach is that it preserves the 2D phase and magnitude spectra of the 2D input image. In fact, implementation involves taking the 2D FFT of the input image, removing spectral frequency with a negative row-frequency component, multiplying the result by 2, and taking the inverse 2D FFT. Given the conjugate symmetry of 2D images, the removal of two frequency quadrants does not result in the loss of any spectral information. Furthermore, it can be shown that for single-component AM-FM signals, this can lead to exact demodulation. In practice though, we replace derivatives by finite differences. We will further elaborate on this method in Section 2.2. For early work on this approach we refer to Havlicek's dissertation [3]. Havlicek et al. [3, 33–41] presented the first results for TV-dimensional signals using the quasi-eigenfunction approximation (QEA) method (see Section 2.2).

Both ESA and Hilbert-based methods share the use of a filterbank prior to AM-FM demodulation. The basic idea is to use a filterbank to be able to separate out among different AM-FM components. AM-FM demodulation is then applied at the output of each channel filter. Here, it is important to note that in the event that two AM-FM components fall within the same channel filter, the filterbank approach will not allow us to separate them. Here, new algebraic approaches should be considered.

The filterbank generates AM-FM demodulation outputs for each channel. Both ESA and Hilbert-based methods select estimates from a dominant component. For ESA, the dominant component is selected based on an energy criterion. In QEA, the dominant component is often selected based on the maximum IA estimate. Here, please note that a single channel is selected over the entire filterbank. In a multiscale approach, instead of selecting dominant components over the entire filterbank, we select the dominant channel from a collection of channels. The basic idea is to define scales based on the frequency magnitude. The most popular approach is to define low, medium, and high frequency scales (see examples in [4]).

Another robust approach for computing AM-FM estimates based on a quasi-local method was developed in [42–44] for 1D signals. This methodology was extended to digital images in [5, 45]. Furthermore, in [5], we have a comparison of ESA, Hilbert-based, and the quasi-local methods for a variety of 2D AM-FM signals. From the comparisons, we note that the choice of the filterbank can have a dramatic effect on the estimation. Flat passbands tend to help with the IA estimation. On the other hand, Gabor-based filterbanks are easy to design and implement and they can perform very well on IF estimation.

Related with the Hilbert based approach, Larkin et al. [46, 47] introduced the phase quadrature transform by isotropically extending the Hilbert transform in 2D. Here, instead of applying the Hilbert transform along the rows or the columns, the spiral-phase quadrature transform is applied along all directions. It is equivalent to applying the Hilbert transform along each radial direction. Here, a single component AM-FM signal is convolved with

$$s\left(x,y\right)=\frac{i\left(x+iy\right)}{2\pi {\left(x^2+y^2\right)}^{3/2}}=\frac{i\text{exp}\left(i\theta \right)}{2\pi r^2},$$

(6)

with a frequency response given by

$$S\left(u,v\right)=\frac{u+iv}{\sqrt{u^2+v^2}}=\text{exp}\left[i\varphi \left(u,v\right)\right].$$

(7)

Thus, the quadrature-phase transform does not alter the magnitude of the AM-FM signal. On the other hand, the same cannot be said about the phase. The phase information is not longer preserved. By examining (7), we can see that this is especially problematic for high frequencies (H).

Alternatively, Felsberg and Sommer proposed an n D generalization of the 1D analytic signal based on the Riesz transform, which is used instead of the Hilbert transform [48, 49]. This is termed the monogenic signal in [48]. Felsberg and Sommer introduced the following filters in the frequency domain [50]:

$$\begin{array}{c}{H}_1\left(u_1,u_2\right)=j\frac{u_1}{u_1^2+u_2^2}\\ H_2\left(u_1,u_2\right)=j\frac{u_2}{u_1^2+u_2^2}.\end{array}$$

with spatial impulse responses given by

$$\begin{array}{c}{h}_1\left(x,y\right)=\frac{-x}{2\pi {\left(x^2+y^2\right)}^{1.5}}\\ h_2\left(x,y\right)=\frac{-y}{2\pi {\left(x^2+y^2\right)}^{1.5}}.\end{array}$$

The local amplitude of the filter response corresponds to a quantitative measure of a structure (including the contrast) and the local phase corresponds to a qualitative measure of a structure (step, peak, etc.) [50]. The monogenic signal f _M is then defined as a 3D vector formed by the signal f(x, y) with its Riesz transform ${\mathbf{f}}_{ℛ}=\left(\mathbf{h}*f\right)$ with (h) = (h, h₂), using

$${\mathbf{f}}_M\left(x,y\right)=\left(f,\mathbf{h}*f\right)\left(x,f\right).$$

(8)

The phase and magnitude of the monogenic signal f _M are then taken as the phase and magnitude of the AM-FM signal. This is also extended to multiple scales in the multiresolution framework of [51], and into scale-space in [49]. Other related work on image demodulation based on the Riesz transform extension has been reported by Mellor, Noble, Hahn, Felsberg, Sommer and collaborators in [48, 52–57]. In (8), it is important to note that while the input signal is 2D, the generated monogenic signal is actually 3D. This was done to extend the ID analytic properties to 2D. On the other hand, it is also clear that these 2D convolutions will also alter the phase of the 2D input AM-FM signal. In fact, as can be seen in the fingerprint example of Figure eleven of [51], the estimated amplitudes contains the ridges. In contrast, in the ESA fingerprint examples of [6], the ridges are modeled as a Frequency-Modulation process. This is a fundamental difference in the different approaches considered here.

Similar to AM-FM methods, we mention the work by Knutsson et al. for representing local structures on phase using tensors [58–60]. Furthermore, the complex Wavelet transform provides an extension to the discrete wavelet transform that is related to the 2D Hilbert-space extension [61–65].

2.2. Multiscale AM-FM methods

In this section, we provide more details on the use of multiple-scales in AM-FM demodulation. Many of the concepts introduced in this section are shared by the AM-FM demodulation methods described in Section 2.1.

We consider multiscale AM-FM representations of images given by

$$I\left(x,y\right)\simeq \sum _{n=1}^M{a}_n\left(x,y\right)\text{cos}{\phi }_n\left(x,y\right),$$

(9)

where n = 1, 2, ..., M denote different scales [4, 5]. In (9), a continuous image I(x, y) is a function of a vector of spatial coordinates (x, y). A collection of M different scales are used to model essential signal modulation structure. The amplitude functions a _n (·) are always assumed to be positive.

AM-FM models non-stationary image content in terms of its amplitude and phase functions [6]. The aim is to let the frequency-modulated (FM) components cos φ _n (·) capture fast-changing spatial variability in the image intensity. For each phase function φ _n (·) we define the IF, ∇φ _n (·). in terms of the gradient:

$$\nabla {\phi }_n\left(x,y\right)=\left(\frac{\partial {\phi }_n}{\partial x}\left(x,y\right),\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)\right).$$

(10)

The IF vector ∇φ _n (·) can vary continuously over the spatial domain of the input signal.

We generalize the concept of scale by considering AM-FM demodulation over a collection of bandpass filters. This is clearly depicted in Figure 1. For real-valued images, we only need two quadrants as shown in Figure 2. Note that by applying the ID Hilbert operator along the columns, the upper two frequency quadrants will be set to zero. On the other hand, if the monogenic signal is computed here, we will keep the entire frequency spectrum and generate a 3D signal as given in (8).

Figure 1

Block diagram of the 2D multiscale AM-FM demodulation. Dominant AM-FM components are selected over different image scales. The bandpass filter selector (upper left) is used to define the bandpass filters that correspond to each scale. The dominant AM-FM component is selected using the maximum amplitude at every pixel (see lowest block).

Figure 2

Four-scale filterbank used. (a) Complete frequency spectrum of the filterbank. (b) Zoom on the low frequency bandpass filters.

The effect of either operation is to generate a complex-valued signal estimates of the form: a(x, y) exp (jφ(x, y)), where the IA and phase functions will hopefully approximate the input AM-FM components. Then, a collection of bandpass filters is used to isolate the individual AM-FM components [3].

The basic assumption here is that different AM-FM components will be picked up by different bandpass filters at the same image region. In other words, given any local image region the assumption is that the corresponding AM-FM components will be separated by the use of different filters in the interbank.

In discrete terms (see Figure 1), given the digital input image I(k₁, k₂) (where k₁ and k₂ are the discrete versions of x and y, respectively), the application of an analytic extension generates an "analytic signal" of the form: I _AS (k₁, k₂). This 2D extension is processed through a collection of bandpass filters (to be described in the Section 2.3) within the desired scale. Each processing block will produce the IA, the IP, and the IF in both x and y directions. At the output of each block, the IA and the IP are estimated by simply taking the magnitude and the phase of the output.

In dominant component analysis (DCA), for each pixel approach, the best AM-FM demodulation estimates are selected from the bandpass filter that produces the largest IA estimate. For energy-based approaches, the dominant channel is selected based on the maximum energy estimate. Hence, the algorithm adaptively selects the estimates from the bandpass filter with the maximum response. This approach does not assume spatial continuity, and allows the model to quickly adapt to singularities in the input signal (both high and low frequency changes).

A robust method for estimating the IF was produced by Murray et al. [4, 5]. Here, for estimating the first IF component for low-frequencies, four estimates are generated using:

$${\widehat{\phi }}_1\left(k_1,k_2\right)=\frac{1}{n_1}\text{arccos}\left(\frac{{Ī}_{AS}\left(k_1+n_1,k_2\right)+{Ī}_{AS}\left(k_1-n_1,k_2\right)}{2I_{AS}\left(k_1,k_2\right)}\right),$$

(11)

where ${Ī}_{AS}\left(k_1,k_2\right)={Î}_{AS}\left(k_1,k_2\right)/\left|{Î}_{AS}\left(k_1,k_2\right)\right|$, and n₁ represents a variable displacement that varies from 1 to 4. Variable spacing, local linear phase (VS-LLP) produces the most accurate of the four estimates by considering the condition number of the arccos(·) function [4, 5]. Similarly, the approach is repeated for the second direction. It turns out that VS-LLP reduces to the earlier method of QEA for the special case when n₁ = n₂ = 1 (see [3] for QEA).

2.3. Filterbank design

For computing the AM-FM estimates, the AM-FM components need to be isolated from (9) using a multiscale filterbank. Here, the basic idea is to isolate different AM-FM components over different frequencies (see [66] for details). Figure 2 (a) depicts the frequency support for a dyadic filterbank decomposition.

In Figure 2, the low pass filter (LPF, with label number 1) has frequency support in [-π/16, π/16] radians for both the x and y directions. The filters in the highest frequencies (filters from 2 to 7 in Figure 2), have a bandwidth of π/2 for both x and y directions. The bandwidth is decreased by a factor of 0.5 for each added scale. In Figure 2 (b), a closeup that shows the low frequency filters is provided.

The scales are provided using a collection of bandpass filters as: (i) LPF, (ii) very low frequencies (VL), (iii) low frequencies (L), (iv) medium frequencies (M) and (v) H as given in [4], When the structures are well defined in sizes, the measurements can be related with specific instantaneous wavelengths described in number of pixels [67, 68].

For example, for classifying lung opacities in pneumoconiosis (to be fully described in Section 3.3), the correspondence between opacity size and instantaneous frequencies using eleven different combinations of scales (CoS) is shown in Table 1. The pneumoconiosis images that were analyzed in the study have a resolution of 300 dpi. This resolution means that along 0.5 mm we have 6 pixel samples. From the ILO standard [69], the rounded opacities labeled as p, q and r, are in the sizes up to 10 mm. With the knowledge of the sizes, the lengths can be related with the corresponding frequencies. These relationships are summarized in Table 2. Based on the frequency ranges for each rounded opacity type, the cut-off frequencies of the bandpass filters are set up to fit the opacities in them by each scale. Then, a four-scale filter-bank is designed such that each scale will be related with the frequency ranges described in Table 2.

Table 1 CoS used for computing the dominant AM-FMfeature parameters

Table 2 Relationship among ILO standard grades for rounded opacities and the size in mm, size in pixels and range in frequency

2.4. Post-processing methods

2.4.1. Histogram features

AM-FM features can be summarized using appropriately chosen histograms. Here, we provide a summary of how this can be accomplished.

For each of the 11 CoS (see Table 1), a 96-bin feature vector that contains the AM-FM histograms is created. Each AM-FM histogram has the information of (32-bin each): (i) the IA, (ii) the IF magnitude, and (iii) the IF angle. Thus, each image produces 11 different feature vectors, each one corresponding to one of the 11 different CoS.

Since the AM-FM estimates of neighboring pixels could be affected by noisy estimates, a 9 × 9 median filter is often applied to the output of the IA and IF estimates (see [4, 5, 14]). We refer to [70] for methods that can be used to adaptively set the median filter window size.

Due to the ambiguity of the IF vectors (coming from cos φ(k₁, k₂) = cos(-φ(k₁, k₂))), the IF estimates are mapped to two frequency quadrants:

• Instantaneous frequencies $\frac{\partial {\phi }_n}{\partial x}\left(x,y\right)\ge 0$ and $\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)\ge 0$ are kept as they are.

• Instantaneous frequencies $\frac{\partial {\phi }_n}{\partial x}\left(x,y\right)\ge 0$ and $\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)<0$ are kept as they are.

• Instantaneous frequencies $\frac{\partial {\phi }_n}{{\partial }_x}\left(x,y\right)<0$ and $\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)<0$ are mapped to $\left|\frac{\partial {\phi }_n}{\partial x}\left(x,y\right)\right|$ and $\left|\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)\right|$.

• Instantaneous frequencies $\frac{\partial {\phi }_n}{{\partial }_x}\left(x,y\right)<0$ and $\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)\ge 0$ are mapped to $\left|\frac{\partial {\phi }_n}{\partial x}\left(x,y\right)\right|$ and $-\left|\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)\right|$.

Then, the IF histograms are computed.

For the IF angle, given by arctan $\left(\frac{\partial {\phi }_n}{\partial y}\left(x,y\right)/\frac{\partial {\phi }_n}{\partial x}\left(x,y\right)\right)$, the histogram is centered around the maximum value using:

[h] ← histogram [IF angle using l bins] {Where h is a column vector.}

H ← [hhh]. {Triplicate version of h.}

i ← Location of the maximum value of h.

h ←H_{l+i-l/2+l:l+i+l/2}.

As last step, all the computed histograms (IA, IF magnitude, and IF angle) are normalized such that the cumulative sum of each one is equal to 1. Thus, the bins will now correspond to the probability density function (PDF) estimates.

2.5. Partial least squares classification of AM-FM features

The histograms of AM-FM features can be used in classification applications. Here, we introduce a promising classifier based on Partial least squares.

Partial least squares (PLS, [71]) is a linear regression method. Here, classification is expressed as a regression problem is given by

$$y=X\beta +\epsilon ,$$

(12)

where y is a n × 1 vector of the classification variables, X is a n × p matrix of the extracted AM-FM features, β is a p × 1 vector of regression weights, and ε is a n × 1 vector of residuals. The least squares solution to estimating β is given by the normal equations β = (X^TX)^-l (X^Ty).

In most of medical imaging applications, there are much more features than images (p <n), and AM-FM features in X are highly correlated. Thus, X^TX canbe singular or nearly singular and a unique solution to the normal equations will not exist. PLS reduces X to a lower dimensional subspace (k ≪ p). The first step is to factor X as X = TL, where T is an orthogonal n × k matrix of T-scores and L is a k × p matrix of factor loadings. The T-scores matrix are used to find a threshold for classification as outlined in [72],

2.6. Morphological segmentation

AM-FM features can also be used for image segmentation. A simple Bayesian classification scheme based on the IA and IF magnitude was demonstrated in [7].

The segmented AM-FM regions can be very noisy. Segmentation region denoising can be achieved via the use of an alternating sequential filter (ASF) as given in [73]. The noise in the segmented image is assumed to be present due to small structural components that do not belong to the abnormal regions. Under this assumption, it can be shown that the optimal morphological filter will be an ASF defined by:

$$M\left(\left(\left(\left(\left(\left(\left(I_S\circ B\right)•B\right)•2B\right)•2B\right)\dots \right)\circ nB\right)•nB\right),$$

(13)

where I _s is the binary segmented image, B is the structural element; nB is the result of n dilations of B by itself; ○ the standard open operation and • the standard close operation.

The value of n needs to be chosen so that the smaller, noisy structures are removed, while the larger structures are preserved. Once the order of the morphological filter is calculated, the segmented image is denoised using (13).

3. Applications in medical imaging

We provide a selected list of AM-FM biomedical imaging applications in Table 3 (see [10] for an earlier version of this table). Here, note that there is a number of medical applications based on energy operators. More recently, medical imaging applications have benefited from the use of multiscale methods summarized in Section 2.2.

Table 3 Select list of AM-FM applications in Biomedical Imaging

An interesting application of AM-FM models in motion estimation was presented by Fleet and Jepson [11]. In [11], Fleet and Jepson use a 3D Gabor interbank to estimate optical flow motion at pixel-level resolution. Their approach motivated the later application of a harmonic phase (HARP) method in [74].

In one of the earlier applications in electron microcopy, a simple Bayesian method using the IA and the IF was used, in conjunction with morphological filtering to provide segmentations of different abnormalities over 26 images [7]. Elshinawy et al. [75, 76] demonstrated the reconstruction of breast cancer images using AM-FM components.

Boudraa et al. [77] introduced a new cross-energy operator and used the operator to demonstrate the functional segmentation of dynamic nuclear images. Maragos et al. provide an important application for ultrasound spectroscopy in [78], where AM-FM models are used for improving Doppler ultrasound resolution. Vector-valued based AM-FM demodulation is given in [79] by Alexandratou et al. Multiscale AM-FM methods were applied to chest radiographs [67, 68], ultrasound images of the carotid artery [80–83], image retrieval in ophthalmology by Acton et al. [84], retinal image classification [8, 85, 86], and electron microscopy [7].

We also have some related 1D medical signal applications by different research groups. Relevant 1D AM-FM methods appear in [87–90]. Medical applications include the classification of surface electromyographic signals in [80], and the analysis of brain rhythms in electroencephalograms [91].

More general (non-medical) AM-FM applications in tracking include the work reported by Prakash et al. [92, 93] and Mould et al. [94]. Recent AM-FM texture analysis research is also reported by Kokkinos et al. [95] and Tay et al. [96]. Recently, Belaid et al. [97] presented an ultrasound image segmentation method based on the monogenic signal and quadrature filters [49].

In the rest of this section, we provide recent examples from the application of multiscale AM-FM decompositions to biomedical imaging. While the applications are focused on the particular models described in Section 2.2, it is important to note that new applications are expected, especially from methods based on multidimensional energy operators and methods based on the monogenic signal.

3.1. Optical flow motion estimation

One of the earliest applications of AM-FM methods comes from the application of phase-based method in estimating optical-flow motion. Here, we use the term "optical-flow motion" to differentiate it from actual object motion. We begin with a formulation of optical flow motion and proceed to describe the phase-based approach.

Let the image intensity in a digital video sequence be denoted by I(x, y, t). After a sufficiently small time interval Δt, the intensity at (x, y) will move to a point (x + Δx, y + Δy). Here, we are assuming that the image intensity is preserved: I(x + Δx, y + Δy, t + Δt) = I(x, y, t).

A Taylor-series expansion at starting point (x, y) gives:

$$I\left(x+\Delta x,y+\Delta y,t+\Delta t\right)=I\left(x,y,t\right)+\Delta x\cdot \frac{\partial I}{\partial x}+\Delta y\cdot \frac{\partial I}{\partial y}+\Delta t\cdot \frac{\partial I}{\partial t}+h.o.t.$$

where h.o.t refers to higher-order terms. Next, if we ignore the higher order terms, divide by Δt, and then take the limit as Δt → 0, we get the Optical flow constraint equation (OFCE):

$$\frac{\partial I}{\partial x}\cdot \frac{\partial x}{\partial t}+\frac{\partial I}{\partial y}\cdot \frac{\partial y}{\partial t}+\frac{\partial I}{\partial t}=0.$$

To formulate the motion problem, define the motion velocities u and v using: u(x, y, t) = ∂x/∂t and v(x, y, t) = ∂y/∂t. This leads to a new form of the OFCE given by:

$$I_{x}u+I_{y}v+I_t=0.$$

(14)

In (14), at every pixel, we have a single equation in two unknowns. This forces us to add additional constraints. For example, a continuity constraint requires that the solution keeps the spatial integral of $u_x^2+u_y^2+v_x^2+v_y^2$ small over the solution small. With this additional constraint we end up with two equations for two unknowns per pixel.

Fleet and Jepson [11] proposed the use of a single-component AM-FM model for modeling digital video based on

$$R\left(x,y,t\right)=a\left(x,y,t\right)\text{exp}\left(j\phi \left(x,y,t\right)\right).$$

(15)

For estimating the AM-FM components, Fleet and Jepson used a 3D Gabor interbank. Then they estimate the IF using

$$\nabla \phi \left(x,y,t\right)=\frac{Im\left[R*\left(x,y,t\right)\nabla R\left(x,y,t\right)\right]}{a^2\left(x,y,t\right)},$$

(16)

where * denotes the complex conjugate, and ∇ refers to the spatial-gradient. The IF is then used to estimate the velocity components that are along the direction of the the IF using

$${\left(u,v\right)}_{\text{projected on IF}}=\left(\frac{-{\phi }_t\left(x,y,t\right)}{∥\nabla \phi \left(x,y,t\right)∥}\right)\left(\frac{\nabla \phi \left(x,y\right)}{∥\nabla \phi \left(x,y,t\right)∥}\right),$$

(17)

where the first parenthesis gives the velocity magnitude, while the second parenthesis provides the unit vector defined in the direction of the IF. To provide for a method of identifying accurate motion estimation, Fleet and Jepson require that the estimated IF is within the range of the estimating filter. This is expressed as

$$∥\nabla \phi \left(x,y,t\right)-\left(k_i,w_i\right)∥\le \tau {\sigma }_k$$

(18)

where (k _i , w _i ) is the peak tuning frequency of the i th filter, σ _k is the standard deviation of the filter's amplitude spectrum, and τ is a threshold used to reject unreliable estimates of instantaneous frequencies. Similarly, for the amplitude, they require that the local signal amplitude must be as large as the average local amplitude, and at least 5% of the largest response amplitude across all the filters at that frame. When either one of these two conditions is not met, the method does not provide velocity estimates.

To recover the full velocity components, Fleet and Jepson apply a local linear fit to the projected velocities. This is accomplished by fitting a local linear model over 5 × 5 neighborhoods:

$$(u(x,y),v(x,y))=({\alpha }_0+{\alpha }_{1}x+{\alpha }_{2}y,{\beta }_0+{\beta }_{1}x+{\beta }_{2}y),$$

(19)

where the local velocity estimate at the central pixel is given by (α₀, β₀).

This study represents an important early application of an AM-FM model to motion estimation. It provided the motivation to the recent work in estimating cardiac motion from MRI (see HARP method in [74]). We expect to see the emergence of new, multiscale AM-FM methods that would be applied to motion estimation from medical video.

3.2. Retinal image analysis

Diabetic retinopathy (DR) and age-related macular degeneration (AMD) are two retinal diseases that present particular characteristics (lesions) on retinal photographs which can be used in early detection and/or classification of the disease. DR and AMD affect individuals in their most productive years of life. Early detection that leads to prevention of vision loss, alone, will lead to significant decrease in risk of early vision loss.

According to the National Eye Institute, DR is one of the leading causes of blindness among working-age Americans, while macular degeneration is a leading cause of blindness among older Americans [98]. It has been shown that regular comprehensive eye exams and timely treatment can lead to improved outcomes and reduced loss of vision. However, to screen the tens of millions at risk for DR would tax the health-care system beyond capacity. Results of an automatic DR screening system based on AM-FM exhibit strong promise in addressing this problem.

AMD is the most common cause of visual loss in the United States and is a growing public health problem. One third of Americans will develop AMD in their lifetimes. To detect AMD, retinal images are graded using the age-related eye disease study (AREDS) protocol for human grading. AM-FM has been used to develop an automated system for characterizing pathological features on these images.

3.2.1. Diabetic retinopathy

Multiscale AM-FM decompositions have been used to characterize retinal images with DR. and to screen out patients with different levels of the disease. Figure 3 shows an image of a retina with DR and one of its IA image.

Figure 3

Example of a retinal image with DR. Note the exudates shown with a white square. (a) Original Retinal Image. (b) IA using CoS 4. (c) Thresholded Image of (b) showing large IA values. A valid range for the pixels between [15-30%] of the maximum value was used for determining the thresholded image.

The types of lesions that need to be detected by the system are shown in Figure 4. Agurto et al. [9, 85, 99] presented the extraction of AM-FM features for classification of DR images using 376 images from the MESSIDOR database [100].

Figure 4

Lesions typical of a DR retinal image. A) Microaneurysm B) Exudates C) Hemorrahages D) Neovascularization.

These 376 images were processed with AM-FM and were divided in two groups for training and testing purposes. Table 4 shows the number of images per risk in each group. The most severe level of DR is grade 3, in which 15 or more microaneurysms or 5 or more hemorrhages are present. For grade 2, the number of microaneurysms in the retina is between 5 and 15 and for grade 1 the retinal images contain between 1 and 5 microaneurysms. The grade 0 corresponds to diabetic patients with no apparent DR in the image. It is important to mention that the images not only contain red lesions such as microaneurysms and hemorrhages, but also contain bright lesions such as exudates and cotton-wool spots.

Table 4 Images correctly classified per risk

After all the images were processed with AM-FM, they were divided in regions of 100 × 100 pixels avoiding the optic disc. Feature vectors were calculated using the moments in each region: mean, standard deviation, skewness, and kurtosis. Finally, the information per image is the input for the PLS algorithm were the algorithm is trained to classify normal images vs abnormal images.

The results obtained with this approach show an area under the ROC curve of 0.84 with best sensitivity/specificity of 92%/54%. Table 4 (last column) shows the number of images correctly classified per risk level [9]. On this table we can see a large percentage of the high-risk patients are being classified correctly.

3.2.2. Age-related macular degeneration

Barriga et al. [86] have developed a system based on AM-FM to generate multiscale features for classifying pathological structures, such as drusen, on a retinal image. Drusen are subreti-nal deposits characteristic of AMD. Figure 5 shows an AMD patient with soft and hard drusen present (white spots on the fundus). The standard high-resolution AREDS images were downloaded from the Wisconsin Fundus Photo Reading Center's website [101]. A certified ophthalmic technician (reader) selected the retinal features. For these studies, only areas that contain retinal background, vessels, soft drusen, and hard drusen were used. After the reader selected the features, regions of interest (ROI) of 40 × 40 pixels were extracted from the image.

Figure 5

Retinal image from AMD patient from the AREDS standard database.

After calculating the AM-FM histograms, principal components analysis (PCA) was applied to the matrix containing histograms of the 120 ROIs (30 for each retinal structure) for three different AM-FM features: IA, IF magnitude (|IF|), and IF angle. Finally, the Mahalanobis distances between the different retinal structures based on PCA decomposition were calculated. These distances are shown in Table 5.

Table 5 Mahalanobis distance between retinal features: Retinal Background (RB), Hard Drusen (DRH), Soft Drusen (DRS), and Vessels

The numbers in Table 5 represent the standard deviations separating the histograms of the retinal structures. The entries under hard drusen (DRH) and soft drusen (DRS) show that they are significantly differentiated from other structures in the retina. A distance of 3 standard deviations represents a classification accuracy of more than 90%. The most interesting entry is the Mahalanobis distance between the two drusen types (2.8 standard deviations). Though still considered a high classification rate (85%), this demonstrates the challenge not only to the algorithm, but for the grader in unequivocally assigning drusen to one class or another.

3.3. Pneumoconiosis in chest radiographs

The chest radiograph is an essential tool used in the screening, surveillance, and diagnosis of dust-related respiratory illness resulting from silica and coal dust, asbestos, and a variety of other dusts that can lead to disease. The standardized method used by the international labor organization (ILO, [69]) for interpreting the chest radiograph for inorganic dust-induced diseases or pneumoconioses has been widely utilized for the past five decades.

Two limitations of the current ILO system are the intra- and inter-interpreter variability and the time consuming process of interpreting large numbers of radiographs taken for screening and surveillance programs. Murray et al. [67, 68] have developed a technique based on AM-FM that detects the level of pneumoconioses based on the nodule formation on the lungs.

Currently, the US remains 6th in the world for pneumoconiosis and interstitial lung disease (ILD) or pulmonary interstitial fibrosis as large numbers of workers continue to be exposed to dust in their work environment. Coal workers' pneumoconiosis (CWP) is identified by a specific pattern of X-Ray abnormalities and a history of exposure to coal dust. The chest radiograph is the single most useful tool for clinically evaluating both occupationally related and non-occupational chronic lung diseases. The chest radiograph is an essential tool used in the screening, surveillance, and diagnosis of dust-related respiratory illness resulting from silica and coal dust, asbestos, and a variety of other dusts that can lead to disease.

An AM-FM method for grading chest radiographs according to the ILO standards is presented in [67, 68] by Murray et al. Related work, by different research groups, can be found in [102–108]. First, a logarithmic transformation to the images is applied to improve the contrast of the X-Rays [109]. Then, the AM-FM estimates of each X-Ray image is encoded using their histograms as described in the Section 2.4.1 using the tuned bandpass filters and scales showed in Tables 1 and 2. The final classification was computed using the linear regression method PLS (described in Section 2.5).

The results obtained by the system were excellent (area under the ROC curve, AUC = 1.0). However, the significance of the results should be debated based on the small database used. The authors plan as future work the testing of the system using a much bigger database. The authors produced classification results that were significantly better than those presented in [102, 103, 107].

3.4. Carotid artery ultrasound images

Cardiovascular disease (CVD) is the third leading cause of death and adult disability in the industrial world after heart attack and cancer. Of all the deaths caused by CVD among adults aged 20 and older, an estimated 6 millions are attributed to coronary heart disease and to stroke, with atherosclerosis as the underlying cause [80–82, 110].

A method based on image analysis of ultrasound images of carotid plaques that can differentiate between the stable plaques that tend to remain asymptomatic and the unstable ones that eventually produce symptoms has the potential to refine the basis for surgery and spare some patients from an unnecessary costly operation which itself carries a risk of stroke.

AM-FM methods are being applied for characterizing and analyzing plaques in ultrasound images. Christodoulou et al. present in [81] their investigations for the AM-FM characterization of carotid plaques in ultrasound images. In [82], Loizou et al. present how to use AM-FM features for describing atherosclerotic plaque features. In what follows, we described the basics of their approaches.

The extraction of features characterizing efficiently the structure of ultrasound carotid plaques is important for the identification of individuals with asymptomatic carotid stenosis at risk of stroke. Christodoloulou et al. present how to characterize the carotid plaques using AM-FM features in [81]. They use as basic descriptors the information about the AM-FM estimates similar as in the Section 2.4.1: the IA, IF magnitude and IF angle histograms. In addition to the AM-FM features, the authors compute Spatial Gray Level Dependence Matrices (SGLDM) computing the following texture measures: (i) angular second moment, (ii) contrast, (iii) correlation, (iv) sum of squares: variance, (v) inverse difference moment, (vi) sum average, (vii) sum variance, (viii) sum entropy, (ix) entropy, (x) difference variance, (xi) difference entropy and (xii) information measures of correlation. Also, the Gray Level Difference Statistics (GLDS) are computed: (i) contrast, (ii) angular second moment, (iii) entropy and (iv) mean. The final step applied by the authors corresponds to the statistical k-nearest neighbor (KNN) classifier implemented for different values of k (k = 1, 3, 5, 7, 9, 11, 13). The authors applied their method to a database of 274 carotid plaque ultrasound images divided as 137 symptomatic and 137 asymptomatic. Using the AM-FM estimates, the authors got a classification success up to 71.5% when the three AM-FM estimates and k = 5 were used. This result was better than using the textures features SGLDM and GLDS that gave 68.2%.

Loizou et al. [82] present in an AM-FM analysis as an application for investigating the intima media complex (IMC), media layer (ML) and intima layer (IL) of the common carotid artery (CCA). This study represents the first study for the IMC, IL and ML. Clinically, the intima-media thickness (IMT) is used as a validated measure for the assessment of atherosclerosis, that causes enlargement of the arteries and thickening of the artery walls. The authors use the same AM-FM histograms as in Section 2.4.1 but computed using only horizontal-oriented bandpass filters. Thus, the authors use similar scales like those described in Table 1 but using only the filters numbered as (see Figure 2a): 1 (LPF), 2, 5 (in the H scale), 8, 11 (in the M scale) and 14 and 17 (in the L scale). The reason for the selection of these horizontal-oriented filters is related with the horizontally elongated nature of the atherosclerotic structures. As the last step for the statistical analysis, the authors use the Mann-Whitney rank sum test in order to identify is there are significant differences (SD) or not (NS) between the extracted AM-FM features. The authors use p < 0.05 for significant differences and comparison between different age groups.

Their study, performed on 100 ultrasound images, reveals that the IA of the media layer decreases with age. The authors state that the decreasing of the IA is maybe related to the reduction in calcified, stable plaque components and an increase in stroke risk with age. In terms of the IF, the median of this AM-FM estimate of the media layer increases suggesting fragmentation of solid, large plaque components that also increases the risk of stroke.

4. Conclusions and future work

We have provided a summary of recent AM-FM applications in medical imaging. In the coming years, we expect that there will be a variety of new applications. Here, it is important to note that medical imaging applications come from a variety of AM-FM methods and models.

The advantages of considering AM-FM models come from their ability to describe image structures over different frequency scales. For example, in retinal images, structures such as drusen and microaneurysms have a well-defined shape that get captured by the higher-frequency scale AM-FM filters. On the other hand, vessels and hemorrhages are better represented in the lower frequencies. In chest radiographs, nodules can also be well represented using the AM-FM features. The combination of feature extraction and classification using PLS has produced robust systems for the analysis of chest radiographs, where an automatic grading system for pneumoconiosis has been demonstrated on a limited-size database. For DR, a screening system has been developed and it is currently being tested on an extensive database for FDA approval.

Future work on AM-FM models will undoubtedly yield new methods and new applications. We expect to see new applications from at-least three separate approaches: (i) applications from AM-FM methods associated with multidimensional energy operators, (ii) applications from multiscale AM-FM decompositions based on Hilbert-based methods, (iii) applications associated with methods associated with the monogenic signal, and (iv) applications from new AM-FM methods that are currently being developed.