- Open Access
Complementary tensor-driven image coherence diffusion for oriented structure enhancement
EURASIP Journal on Advances in Signal Processing volume 2011, Article number: 70 (2011)
Oriented structure enhancement plays important role in computer vision tasks, where the diffusion is encouraged along the preferred direction instead of perpendicular to it. By analyzing the differential geometric property of the oriented structure, a complementary tensor is proposed by combining the first and the second-order structure tensors as complementary descriptors, which can precisely analyze not only the step edges, but also the weak edges such as narrow peak or ridge-like structures. Complementary diffusion tensor is constructed from the new structure tensor, which steers coherence diffusion for oriented structure enhancement. Furthermore, fast algorithm based on additive operator splitting scheme is used for numerical solution, which is much faster than usual approach. The experimental results on several images are provided. Experiments show that the image diffusion process steered by the new complementary tensor can strengthen the oriented structures and also close the interrupted lines as well. Both strong and weak edges are enhanced while noise is removed. Our approach is very promising and could be applied to many other images.
Image enhancement is an important preprocessing step that removes noise while preserving semantically important structures such as edges and oriented structures. This may give great help for simplifying subsequent image analysis like segmentation and understanding. In recent years, nonlinear PDE-based diffusion for image enhancement has attracted much attention for its adaptive behavior in a purely data-driven way that is flexible enough to cope with the rich image structures .
Image diffusion by nonlinear partial differential equation (PDE) approach was first introduced by Perona and Malik , followed by their study, numerous nonlinear diffusion filters have been proposed [3–8]. However, most of them use a scalar diffusivity thus the diffusion flux is along gradient direction, which may blur edges. Besides that, such isotropic diffusion cannot preserve the oriented structures precisely [1, 7].
With the increasing appearance of oriented structures in many computer vision and image processing problems, for instance pulmonary vessels, trabecular bone, fingerprint, paintings, wood texture, etc. Many attempts on oriented patterns enhancement have been made [9–12]. PDE-based coherence-enhancing diffusion has proved to be an attractive tool, which is an anisotropic diffusion steered by the diffusion tensor that allows a more flexible orientation-dependent filtering [10, 11].
There are two important steps for coherence-enhancing diffusion: first analyze coherent structures and second enhance it. Structure tensor is an important descriptor for coherence analyzing, whose eigenvectors and eigenvalues can provide coherent direction and measurement, respectively. From the structure tensor, the diffusion tensor is constructed such that its eigenvalues encourage the diffusion along the coherent direction than across to it, which steers the coherent diffusion.
Precise coherence analysis is crucial to the diffusion behavior. The original structure tensor was proposed by Cottete and Germain , i.e., the tensor product of the gradient: , where u σ = G σ *u is the slightly smooth version image by convolving u with Gauss kernel G σ . The eigenvalues of J0 provide coherent measurement and one of the eigenvectors provides the coherent direction. However, the simple structure tensor fails in analyzing corners or parallel structures. To solve this problem, Weickert [10, 11] proposed the structure tensor: . The eigenvalues of Jρ measure the variation of the gray values within a window size of order ρ. This structure tensor is useful to analyze strong edges, corners, and T-junctions. Brox and Weickert  proposed a nonlinear structure tensor by diffusing J0 under the image gradient field ∇u, which may bring much computational cost. However, the above structure tensors are essentially linear smoothing of J0, and use only the local average of the first-order differential information. Many image features such as narrow peaks, ridge-like edges could not accurately be described by J ρ for the gradient is close to zero on these structures.
By analyzing the first- and second-order directional derivatives of the different edges, we propose a new structure tensor which is a desirable descriptor for analyzing not only strong edges, but also weak edges such as narrow peaks or ridge-like structures. The proposed structure tensor combines the first- and the second-order structure tensors as complementary descriptors that play important roles on detecting different kind of edges. The complementary diffusion tensor is constructed from the new structure tensors, which can steer the coherence diffusion controlled by a switch parameter. The diffusion can preserve not only strong edges, but also weak edges precisely while removing the noise. Furthermore, numerical implementation is solved by the additive operator splitting (AOS) scheme, resulting in fast convergence rate.
The remainder of the article is organized as follows. In Section 2, mathematical background of the coherence diffusion is outlined. In Section 3, edge structure tensor is proposed by combing the first- and the second-order structure tensors as complementary pairs. In Section 4, a complementary diffusion tensor is constructed from the new structure tensor. In Section 5, efficient numerical implementation of the diffusion PDE by fast AOS scheme is provided, and in Section 6, experimental results are provided and compared with that of the other methods. Finally, in Section 7, conclusions are reported.
2. Coherence diffusion
In image processing, the anisotropic diffusion process is equivalent to minimizing the following energy variational:
where f denotes the original image defined on the image domain Ω and u the diffusion image. β is a small positive. ψ(s2 ) denotes a penalizing function that is differential in its argument and convex in s.
From variational method, we can derive the following Euler Lagrange equations of anisotropic diffusion :
The detailed derivative can be found in .
It , then Equation 2 is the steady-state solution of the following PDE with the reflecting boundary condition and the original f as the initial condition:
where D(·) is the diffusion tensor and is a positive definite symmetric matrix. In the coherence diffusion case, it is desirable to rotate the diffusion tensor such that the diffusion is encouraged along the orientation of interesting features instead of perpendicular to it. The desired oriented direction is called coherent direction along which the diffusion performs more sufficient.
Designing diffusion tensor D is crucial to the diffusion process. To steer the coherence diffusion, the differential geometry property of oriented structure must be analyzed so as to detect the oriented structure and the coherent direction precisely.
3. Edge structure and its differential geometric property
The edge in image is defined as the position where the gray value changed suddenly. This kind of edge always refers to either step edge or the strong edge. The strong edge can easily be detected from the extreme of its first derivative. However, there are many oriented structures such as narrow peaks, ridge-like, or flow-like patterns in images. However, their first derivative is zero. We called them weak edges. These patterns play very important role as the strong edge did. Therefore, finding out the method for detecting and enhancing these oriented structure is also in great need.
Let us denote the intensity image by u(x, y). The directional derivative of U at point (x, y) in the direction, α = (cosθ, sinθ) T is denoted by u α (x, y). It is defined as:
The second-orof u along direction α is denoted by , and it follows that
where H is the Hessian matrix of u.
Figure 1 shows the Edge structure and its differential geometric property. Figure 1a, b shows the step edges and its first- and second-order derivatives. We could see that its first-order derivative can reach its extreme. Therefore, the strong edge can be detected by its first derivative.
In the following, we could see that the geometry property of the strong edge can by analyzed by the structure tensor: , where uσ = Gσ *u is the slightly smooth version image by convolving u with Gauss kernel G σ . We call J0 the first-order structure tensor.
From linear algebra, we can derive that the two eigenvalues of J0 are , and the corresponding eigenvectors are , , where denotes the vector that is perpendicular to ∇u σ . We know that the direction of ∇u σ is perpendicular to the edge. Therefore, we can conclude that the coherent direction is , because the desired diffusion direction is along the edge but not across ro it.
Figure 1c, d shows the weak edge, whose first-order directional differential is zero. However, we could see that the ridge-like structure has the extremum on its second-order directional differential. Let μ1, μ2 (μ1 ≥ μ2) denote the eigenvalues of H and e1, e2 the corresponding eigenvectors. From Rayleigh's quotient , we can derive that
So, the eigenvalues of the Hessian matrix are exactly the two extreme of and the corresponding eigenvectors are the directions along which the second directional derivative reaches its extreme.
Since the Hessian matrix can describe the second-order structure of the local intensity variations along the eigenvectors, we call it second-order structure tensor and denote it as J H = H σ , where H σ is the Hessian matrix of u σ .
From Figure 1c, d, we can see that when there are dark (bright) narrow long structures in the image, reaches its maximum (minimum) along eigenvectors e1 (e2). As a result, the coherent direction is along e2 (e1), respectively.
From above, we can conclude that the first- and the second-order structure tensors play complementary role in detecting different kinds of edges. Let us use J ρ = G ρ *J0, where G ρ is the Gauss kernel, to replace J0. The first-order structure tensor J ρ is useful to analyze strong edges such as step edge, corners, and T-junctions, while fails in detecting weak edges precisely. However, the second-order structure tensor J H can capture weak edges as narrow peaks and ridge-like structures while fails in detecting strong edges. Therefore, the two structure tensors J ρ and J H can be complementary to each other and provide reliable coherence estimation on different structures.
4. Complementary diffusion tensor
A natural way to construct diffusion tensor D is from structure tensor J such that D has the same eigenvectors as J and its eigenvalues prefer the diffusion along the coherent direction than across to it.
Let μ1, μ2 (μ 1 ≥ μ2) denote the eigenvalues of J and e1, e2 the corresponding eigenvectors. The coherent direction estimated from J is denoted by e(co) and the orthogonal direction by .
When μ1 >> μ2, it corresponds to anisotropic-oriented structure. When μ1 ≈μ2, it corresponds to isotropic structures. The coherence measurement is given by K = (μ1 - μ2)2. To encourage coherent diffusion, the eigenvalues of D can be chosen as follows [10, 11]:
where λ(co)and are the diffusivity along the direction e(co)and , respectively. γ > 0 serves as a threshold parameter. We can see that λ(co)is an increasing function with respect to the coherence measurement K. when (μ1 - μ2)2 > > γ, λ(co) ≈1. Otherwise, it leads to λ(co) ≈c, where c∈(0, 1) is small positive parameter that guarantees that D is positive definite. It means that the more coherent the structure is, the more diffusion along the coherent direction e(co)is. Whereas the diffusion is not preferred at the direction of as the diffusivity is very small there.
Therefore, D can be obtained by
where is the eigenvector matrix.
A switch parameter T is introduced to control the roles of the two structure tensors in detecting different coherent structures. When , J ρ is available to detect strong edges. Whereas , J H is more reliable to detect weak edges.
Let D ρ , D H denote the diffusion tensor constructed from J ρ and J H , respectively. Therefore, the diffusion tensor is given by
when , the coherent direction of D is always along ; Otherwise, it is along that is the coherent direction estimated from J H . It is noted that the coherent direction estimated from J H can be divided into the following two cases. When there are dark curvilinear structures in the bright background, the coherent direction is along e2. Otherwise there are bright curvilinear structures in the dark background, the coherent direction is along e 1 .
Substitute (8) into Equation 3, we can obtain the following partial differential diffusion equation:
Equation 9 states that the diffusion tensor can steer a complementary coherence diffusion controlled by the parameter T, where the two diffusion process are both coherent and can strength the step edge, corners as well as narrow peaks, ridge-like structures while smoothing the interior of the image.
5. Numerical implementation by parallel AOS algorithm
The numerical solution of the diffusion equation (9) can be implemented by the semi-implicit discretization and AOS scheme [10, 18]. It is a parallel algorithm and has fast algorithm for inverse matrix. Therefore, it is 10 times faster than usual numerical method.
As the diffusion tensor D is a positive definite symmetric matrix, let , then
To facilitate the expression, we denote , then we obtain
Using central differential discretization, we obtain:
Using central differential discretization, we obtain
Let L ij denote a central difference approximation to the operator . Then
where m is the dimension of the image. In our case, m = 2.
The discretization of Equation 9 is given by the finite difference scheme:
Represent the whole image of size [M,N] as a vector of size [MN,1]. In this vector-matrix notation and use the semi-implicit discretization, we obtain
Let , and denote ,
It is noted that L ll is tri-diagonal and diagonally dominant, and use AOS scheme , we can obtain
AOS scheme is based on the fact that Equation 21 has the same first-order Taylor expansion in ττ as that of (20). In this way, the complex matrix inverse problem in (20) can be simplified to the matrix inverse in (21). The central difference approximation of L ll , (l = 1, 2) guarantees that it is diagonally dominant tri-diagonal. Therefore, the diffusion equation (21) converts to solving a diagonally dominant tri-diagonal system of linear equation, which can be easily solved by Thomas algorithm . By the AOS algorithm, the numerical implementation is 10 times faster than the usual numerical method.
The AOS algorithm of the PDE (9) is given as follows
Input U = Uk
Calculate structure tensor J for each point and calculate the corresponding eigenvalues and eigenvectors;
Construct diffusion tensor D using (6) and (7);
For l = 1:m
Calculate by Thomas algorithm.
Output U = Uk+1
Selection of parameters is important for the diffusion process and the detailed discussion can be found in . In the following experiments, we choose β = 0.1, c = 0.001, and γ the 90% quantile of the histogram for (μ1-μ2)2. In the semi-implicit discretization case, the recommendation of time step τ is not more than 5. In the experiments, we set τ = 2.5. The iteration time is related to the spatial scale . In another word, the bigger the iteration time is, the more the diffusion is closer to its steady state.
Parameter T is crucial to switch the two diffusion processes. If T is chosen too small, some narrow long structures cannot be captured precisely. However if T is selected too large, some gradient information may lose and computational cost increases as more pixels are involved in second-order differential computing. Because setting of T is to recognize the weak edge where the gradient is very small. When T is setting to be x% quantile of the histogram for , it means that x% of the gradients are smaller than T. The small portion of x% has small gradients that are the potential weak edge and the second-order directional differential needs to be calculated there. Experiments show that setting T to be 5-10% quantile of the histogram for is sufficient for wide categories of images. In the experiments, we set T to be 10% quantile of . The noise scale σ and integration scale ρ are given for each case.
We design three groups of experiments as follows:
Complementary diffusion for image coherence-enhancing. The diffusion equation is give by (9).
Weickert's coherence-enhancing diffusion . It corresponds to the case that D ≡ D ρ .
Isotropic diffusion by P-M equation for image enhancement . It corresponds to the case that D is a scalar diffusivity. where λ is the contrast parameter that can be chosen as the 90% quantile of the histogram for .
The numerical solutions that are implemented by the AOS scheme are described in Section 5. When D is scalar diffusivity, it can be considered as the special case that D is single element matrix. Then , where L ii is the central difference approximation to the operator . In this case, Vk = Uk .
The experimental results are shown in Figures 2, 3, and 4. The first columns are the original images. The second columns are the results by the proposed complementary diffusion tensor-driven approach. The third columns are the results by Weickert's coherence-diffusion method. The fourth columns are the results by P-M-diffusion equation.
Figure 2 is a noisy tree texture image, where the growth ring is corresponding to ridge-like peaks. From Figure 2b, we can see that the growth ring of tree is preserved precisely and some interrupted line is closed as well. The growth ring in Figure 2b is more straight and smooth than that of Figure 2c. Moreover, the noise is removed better by our approach. Figure 2c shows that the noise is magnified somewhere and some flow-like artifacts appear in the interior area. In Figure 2d, the edge is blurred and the growth ring could not be seen clearly.
Figure 3 is a noisy wood texture image, where texture lines are corresponding to narrow peaks. Figure 3b shows that the important image features as line structures are preserved perfectly whereas Figure 3c severely blurs and twists some lines, and some flow-like artifacts appear in the interior area in Figure 3c. Figure 3d blurs the edge and also could not remove the scattered noise.
Figure 4 is a zebra texture image, where the stripe corresponds to ridge-like structures. Figure 4b precisely preserve these structures while removing the noise and the enhanced stripe is smooth and natural. Figure 4c, however, shows some heterogeneous in the interior of the stripe and deforms the edge in some degree. Figure 4d could not obtain the oriented structure and fails in removing the noise.
Experiments show that by our approach the diffusion is preferred along the desired coherent direction for the image features that are precisely described by the proposed complementary structure tensor. Therefore, important features are preserved and enhanced while the noise is removed very well. However, Weickert's approach deforms some narrow peak and ridge-like-oriented structures because the local average of the first-order structure tensor cannot recognize these patterns precisely . As to the isotropic diffusion by P-M equation, it cannot close the interrupted line structures and also blurs the edge . The proposed switch diffusion is robust and reliable for enhancing-oriented structures.
In this article, complementary diffusion tensor-driven method for image coherence enhancement was proposed. A new structure tensor combing the first- and the second-order directional differential information were proposed, which can capture not only strong edges but also narrow peak and ridge-like structures precisely. The two structure tensors play important roles in different diffusion stage controlled by a switch parameter, which can provide precise coherence estimation on different structures. A complementary diffusion was steered by the diffusion tensor constructed from the new structure tensor, which can preserve many important edges, corners, T-junctions as well as narrow peaks and ridge-like structures while removing the noise. Furthermore, fast parallel AOS algorithm was applied to numerical implementation that is very efficient. Experiments by our approach were provided and compared with that of other methods, which were very promising. Our approach is reliable and could be applied to oriented structure coherence enhancement.
Weickert J: A Review of Nonlinear Diffusion Filtering. Scale-Space Theory in Computer Vision. Volume 1252. Springer, Berlin; 1997:3-28.
Perona P, Malik J: Scale-space and edge detection using anisotropic diffusion. IEEE Trans Mach Intell 1990,12(7):629-639. 10.1109/34.56205
Alvarez L, Lions P-L, Morel J-M: Image selective smoothing and edge detection by nonlinear diffusion. II SIAM J Numer Anal 1992, 29: 845-866. 10.1137/0729052
Sapiro G, Tannenbaum A: Affine invariant scale-space. Int J Comput Vis 1993, 11: 25-44.
Rudin L, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. In Modeliz Mat Traitement d' Images. INRIA; 1992:149-179.
Osher S, Rudin LI: Feature-oriented image enhancement using shock filters. SIAM J Numer Anal 1990, 27: 919-940. 10.1137/0727053
Salinas HM, Fernández D Cabrera: Comparison of PDE-based nonlinear diffusion approaches for image enhancement and denoising in optical coherence tomography. IEEE Trans Med Imag 2007,26(6):761-771.
Fang D, Nanning Z, Jianru X: Image smoothing and sharpening based on nonlinear diffusion equation. Signal Process 2008,88(11):2850-2855. 10.1016/j.sigpro.2008.05.008
Chen Y, Levine SE: Image recovery via diffusion tensor and time-delay regularization. J Vis Commun Image Rep 2002, 13: 156-175. 10.1006/jvci.2001.0497
Weickert J: Coherence-enhancing diffusion filtering. Int J Comput Vis 1999,31(2-3):111-127.
Weickert J: Coherence-enhancing diffusion of colour images. Image Vis Comput 1999, (17):201-212.
Bakalexis SA, Boutalis YS, Mertzios BG: Edge detection and image segmentation based on nonlinear anisotropic diffusion. 14th International Conference on Digital Signal Processing 2002 2002, 2: 1203-1206.
Cottet G-H, Germain L: Image processing through reaction combined with nonlinear diffusion. Math Comput 1993, 61: 659-673. 10.1090/S0025-5718-1993-1195422-2
Brox T, Weickert J: In Nonlinear Matrix Diffusion For Optic Flow Estimation. Edited by: Van Gool L. 2002, 446-453. DAGM 2002, LNCS 2449
Weickert J: Nonlinear diffusion filtering. In Handbook on Computer Vision and Applications. Volume 2. Edited by: Jähne B, Haußecker H, Geißler P. Signal Processing and Pattern Recognition; 1999.
Hongmei Z: Study on medical image segmentation based on variational method. PhD, Xi an JiaoTong University. 2004.
Kreyszig E: Differential Geometry. Dover Publications; 1991.
Weickert J, Romeny BM ter Haar, Viergever MA: Efficient and reliable schemes for nonlinear diffusion filtering. IEEE Trans Image Process 1998,7(3):398-410. 10.1109/83.661190
Schwarz HR: Numerische Mathematik. Teubner, Stuttgart, Germany; 1988.
This study was supported by the National Basic Research Program 973 under Grant Nos. 2010CB732603 and 2011CB707903, and the National Natural Science Foundation of China under Grant (No. 60801057).
The authors declare that they have no competing interests.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Hong-mei, Z., Ming-xi, W. & Zheng-zhong, B. Complementary tensor-driven image coherence diffusion for oriented structure enhancement. EURASIP J. Adv. Signal Process. 2011, 70 (2011). https://doi.org/10.1186/1687-6180-2011-70
- tensor driven
- image diffusion
- structure tensor
- diffusion tensor
- second-order directional derivative
- AOS scheme