Introducing oriented Laplacian diffusion into a variational decomposition model
- Reza Shahidi^{1}Email authorView ORCID ID profile and
- Cecilia Moloney^{1}
https://doi.org/10.1186/s13634-016-0415-2
© The Author(s) 2016
Received: 2 June 2016
Accepted: 28 October 2016
Published: 10 November 2016
Abstract
The decomposition model proposed by Osher, Solé and Vese in 2003 (the OSV model) is known for its good denoising performance. This performance has been found to be due to its higher weighting of lower image frequencies in the H ^{−1}-norm modeling the noise component in the model. However, the OSV model tends to also move high-frequency texture into this noise component. Diffusion with an oriented Laplacian for oriented texture is introduced in this paper, in lieu of the usual Laplacian operator used to solve the OSV model, thereby significantly reducing the presence of such texture in the noise component. Results obtained from the proposed oriented Laplacian model for test images with oriented texture are given, and compared to those from the OSV model as well as the Mean Curvature model (MCM). In general, the proposed oriented Laplacian model yields higher signal-to-noise ratios and visually superior denoising results than either the OSV or the MCM models. We also compare the proposed method to a non-local means model and find that although the proposed method generally yields slightly lower signal-to-noise ratios, it generally gives results of better perceptual visual quality.
Keywords
1 Introduction
1.1 Image decomposition
According to the definition of Meyer [1], image decomposition is the process of splitting an image into components of different priorities, usually so that the original image is roughly equal to the sum of these components. Often the original image is denoted by f, which is considered to approximately equal the sum of two components, u and v, i.e. f≈u+v. In Meyer’s 2001 monograph [1], decomposition was deemed to be important for image coding and transmission. In addition to the problem of decomposition being interesting and important in its own right [2], decomposition has also proven to be useful for texture discrimination [3], image denoising [4], image inpainting [5] and image registration [6]. In this paper, the focus is on the application of decomposition to the problem of the denoising of images with oriented texture.
1.2 Meyer’s decomposition framework
Meyer’s decomposition framework models the various components of the decomposition as having small norms in different Banach spaces, i.e. complete normed vector spaces. In practical implementations of this framework [3], the energy method is used, in which an energy is defined over the image being decomposed and its components, and is subsequently minimized using the Calculus of Variations.
where the original image is f and f≈u+v. The first term \(\left (\int _{\Omega }|\nabla u|dxdy\right)\) is included to produce a bounded variation (piecewise-smooth) image u upon energy minimization, while the second term is a fidelity term, which ensures that the final u component is close to the initial image f. Although the minimization of E _{ TV }(u) preserves sharp edges, it destroys fine structure such as texture. However, Eq. 1 has been used successfully for the denoising and deblurring of images of bounded variation.
over all g _{1} and g _{2} such that \(v=\text {div}({\vec {g}})\) where \({\vec {g}}=(g_{1},g_{2})\).
1.3 The Osher-Solé-Vese decomposition model
with adiabatic boundary conditions. This equation was found by approximating \(\vec {g}\) as having no divergence-free part, and approximating the L ^{ ∞ } part of the ∗-norm in Eq. 2 with the square of the L ^{2}-norm of the same function. The resulting energy functional includes an inverse Laplacian of f−u; this was eliminated in [4] by showing that under some rather relaxed conditions, found in the second Remark of that paper, the gradient descent solution of an Euler-Lagrange equation u _{ t }=−E OSV′(u), where E OSV′(u) is the first variation of E _{ OSV } with respect to u, converges to the same solution as the equation u _{ t }=Δ E OSV′(u). Thus no inverse Laplacian appears in Eq. 4. Additionally, the auxiliary functions g _{1} and g _{2} of Eq. 2 are no longer involved in the PDE of Eq. 4, as they disappear in the derivation.
2 Incorporating oriented diffusion into the OSV model
2.1 Motivation and initial derivation
Observe that Eq. 5 consists of two terms, one with the negative Laplacian of the curvature of the level lines of u; the other a fidelity term, which makes sure that the evolved image does not stray too far away from the original image f. Equation 5 has been shown to perform well for denoising [4]. Since the original image f is noisy, and the second term keeps the evolved image close to this noisy image, the denoising itself must be due to the first term; i.e. the negative Laplacian term.
There are also two conductivity coefficients c _{ ζ } and c _{ η } which determine how much diffusion there is in the isophote and gradient directions respectively.
The expressions for the two directional derivatives in Eqs. 7 and 8 can be substituted into Eq. 6. Recall that the function K represents the curvature of the cartoon component u.
In our experiments for this paper, better results were obtained in practice by using the normalized isophote orientation estimates of the original image f instead of those of the curvature image K. An initial explanation for the improvement in results is the sensitivity to noise of the formulae in Eqs. 7 and 8.
We now explain how estimates of the normalized isophote and gradient directions are extracted from the original noisy image using the linear structure tensor (Section 2.2.1), as well as how this image is separated into oriented and non-oriented regions (Section 2.2.2). This separation by region is important to decide whether anisotropic or isotropic diffusion should be performed at each image pixel, and the directional estimates are important to determine which directional diffusion parameters are used for anisotropic diffusion at each pixel in the oriented region. After this explanation in Section 2.2, we return to the derivation of the proposed model in Section 2.3 using these extracted gradient and isophote orientation estimates.
2.2 Orientation and coherence calculation
The proposed algorithm depends on the noise-resistant computation of orientation at each pixel in the image, in addition to the detection of the image regions that are coherent, or roughly consisting of one orientation. Two methods were considered in our experiments by which such calculations could be performed and be robust to noise. Both use the structure tensor, or the outer product of the image gradient with itself. The first is called the linear structure tensor, which corresponds to a linear diffusion or Gaussian blurring of the structure tensor elements, and the second is called the nonlinear structure tensor, an extension of the linear structure tensor which corresponds to nonlinear diffusion of the structure tensor entries. Only the linear structure tensor is described here, since it was found to be more efficient computationally than the nonlinear structure tensor, and gave results of similar quality. The linear structure tensor is now described, along with methods of determining the orientation coherence [9]. If not otherwise specified, it is assumed the orientation refers to the gradient orientation, or the orientation in which the image changes the most in intensity.
2.2.1 Linear structure tensor
2.2.2 Orientation coherence and oriented region determination
The threshold used for the coherence of the orientations of f is dependent on the image. Call this parameter coher _{ thresh }; we set coher _{ thresh }=15 for all experiments in this paper.
Now define the oriented region to be Ω _{ O }, the union of pixels in the image where there is a dominant orientation in f. Similarly, we define the non-oriented region to be Ω _{ NO }, the union of pixels in the image where there is no such dominant orientation.
2.3 The oriented Laplacian Osher-Solé-Vese model
which is subsequently substituted in the oriented Laplacian expression in Eq. 6. Note that the denominators in Eqs. 7 and 8 do not appear since we are dealing with the normalized isophote estimates, as opposed to Eqs. 7 and 8, where the partial derivatives of K are unnormalized.
where (ζ _{ x },ζ _{ y })=(f _{ i s o,x },f _{ i s o,y }) is a unit vector defined for each pixel in the image pointing in the isophote direction.
Within the oriented region Ω _{ O }, Eq. 14 is used, whereas within the non-oriented region Ω _{ NO }, the ordinary evolution equation for Osher-Solé-Vese decomposition, as in Eq. 4 is utilized. Each equation and region involves a weighting parameter λ. This parameter becomes λ _{ O } in Ω _{ O }, and λ _{ NO } in Ω _{ NO }. In practice, each model is evolved over the entire image, but after convergence restricted to either Ω _{ O } or Ω _{ NO }, as initially defined by the coherence threshold. This avoids any noticeable artifacts at the boundaries of the regions by using the image itself to effectively pad each region for its model of diffusion along internal interfaces.
We call the decomposition model defined by the PDE of Eq. 14, the Oriented Laplacian Osher-Solé-Vese (OLOSV) decomposition model.
3 Denoising by mean curvature motion
A similar denoising model to the OLOSV model developed in Eq. 14 above is the mean curvature motion (MCM) denoising model. Both models are based on the idea of denoising edges, only along and not across them. However, it will be seen that MCM denoising does not lend itself readily to a dual norm formulation, and thus cannot avail of the better quality results obtained from such a formulation.
with u _{ ζ ζ } the second-order directional derivative of u in the isophote direction.
derived by setting (ζ _{ x },ζ _{ y })=(f _{ i s o,x },f _{ i s o,y }) with this isophote direction measured in a noise-resistant manner, as in Section 2.2.1. We found from our experiments that using these image isophotes gave better denoising results.
3.1 Explicit timestepping
Then Eq. 14 (for OLOSV) and Eq. 16 (for MCM) are each separately solved via explicit timestepping (with the time step chosen by experimentation to be Δ t=0.002).
The above two equations are derived from Eqs. 14 and 4, respectively, by assuming we have reached equilibrium, so that u _{ t }=0, and then multiplying each side by u−f and integrating over the image domain Ω.
4 Numerical implementation
Then, similar to [14], \(f+\frac {1}{\lambda _{NO}}\Delta \text {div}p^{n}\) converges to the u component of the decomposition in Ω _{ NO }, the non-oriented section of the image, which implies that \(-\frac {1}{\lambda _{NO}}\Delta \text {div}p^{n}\) converges to the component v=f−u in Ω _{ NO }. Similarly, in the oriented section Ω _{ O } of the image, \(f+\frac {1}{\lambda _{O}}\Delta _{\zeta \zeta }\text {div}p^{n}\) converges to the u component of the decomposition and thus \(-\frac {1}{\lambda _{O}}\Delta _{\zeta \zeta }\text {div}p^{n}\) converges to the component v=f−u in Ω _{ O }.
When used in conjunction with the OLOSV model, such an implicit numerical method gives rise to a new denoising method, which we call the Implicit Oriented Laplacian Osher-Solé-Vese, or IOLOSV method.
5 TV-Hilbert space formulation
The proposed IOLOSV model presented in this paper differs from the general TV-Hilbert model in [14] and the TV-Gabor model in [15], in that, in the proposed model, the operator K is spatially varying, whereas in the two other models, the operator K is constant throughout the entire image, and not data-dependent.
where \(<f,Kg>_{L^{2}} = \sum _{i,j} f_{i,j}(Kg)_{i,j}\). Then the Sobolev space H ^{−1} used in the model of Osher, Solé and Vese fits into the TV-Hilbert space framework, with the operator K=−Δ ^{−1} as the negative of the inverse Laplacian operator.
The proposed IOLOSV method can also be made to fit into the TV-Hilbert space decomposition framework, with the operator K spatially varying and equal to \(K = -\Delta _{\zeta \zeta }^{-1}\), the negative of the spatially varying second-order directional derivative (in the direction of the image isophote). We denote the Hilbert space thus obtained as \(H^{-1}_{\zeta }\) in order to emphasize its link with both the Sobolev space H ^{−1} and its dependence on the spatially varying isophote direction ζ(x,y).
5.1 Calculation of iteration-dependent fidelity parameter
where \(C_{\mathcal {H}}\) is a constant that only depends on the Hilbert space \(\mathcal {H}\) over which we are taking the norm of the noise image n. Its calculation will be described in the next subsection.
In both equations, the noise component v is defined to be equal to f−u, where f is the original noise-contaminated image and u is the obtained cartoon component.
In the above equation, \(||v_{n}||_{\mathcal {H}}\) is the norm of v _{ n } in the Hilbert space \(\mathcal {H}\); the calculation of this norm is detailed below in Section 5.3. Then v is calculated using this new value of λ for the next iteration. The procedure is implemented separately in Ω _{ O }, where \(\mathcal {H} = H^{-1}_{\zeta }\), and in Ω _{ NO }, where \(\mathcal {H} = H^{-1}\). Moreover, the constant \(\alpha _{\mathcal {H}}\) is chosen separately in each of Ω _{ O } and Ω _{ NO }, with the selected values being 0.6 and 0.98, respectively, based on experimental findings.
For Ω _{ O } we slightly alter the calculation of the numerator of Eq. 26 to ensure enough noise is extracted from oriented parts of the image. We calculate the L ^{2}-norm of the noise in the non-oriented region Ω _{ NO } and then compute the expected L ^{2}-norm of the noise in the oriented region Ω _{ O } by balancing the value of this norm in Ω _{ O } and the sizes of both regions, by assuming that the noise power is spatially invariant across both regions. Finally, in the denominator of Eq. 26, we use the maximum of the actual measured L ^{2}-norm of the noise in Ω _{ O } and 0.7 times the expected L ^{2}-norm of this noise in this region, based on that in Ω _{ NO }.
Execution of the proposed algorithm is stopped when the numerator and denominator of the multiplicative factor in Eq. 26 become close to one another. More specifically, for the non-oriented region Ω _{ NO }, iterations are stopped once the numerator becomes greater than 0.95 times the denominator, while for the oriented region Ω _{ O } we stop iterating once the numerator becomes greater than 0.99 of the denominator.
5.2 Calculation of constant \(C_{\mathcal {H}}\)
5.3 Calculation of \(\mathcal {H}\)-norms
both of these squared-norm values being used in the dynamic update of the fidelity parameter λ, for their respective Hilbert spaces.
6 Non-local means
To set this paper within the forefront of research on image denoising, we also compare the proposed OLOSV algorithm with non-local means, a recent addition to the field. Non-local means denoising, introduced in [16] by Buades et al. is known to give good denoising results. It does so by comparing image patches from different parts of an image and using all or many of these patches to improve denoising performance by performing a weighted average on these patches based on their similarity in pixel intensities.
where ω(x,y) is a weighting function measuring the similarity between the image patches centred at pixels x and y.
7 Test images
Among the test images used in this paper are several well-known images with significant oriented texture, i.e. barbara which is 512 × 512 pixels, fingerprint which is 256 × 240 pixels, and bridges which is 1024 × 683 pixels, each with bitdepth of 8 bits.
Results obtained for the three test images barbara, bridges and fingerprint are presented in the next section; these results demonstrate that in general, denoised images with higher signal-to-noise ratios are obtained with IOLOSV over both the MCM and OSV algorithms, and the results are visually superior.
8 Experimental results
8.1 Experimental method
To each of the three original test images, additive white Gaussian noise (AWGN) was added with a standard deviation σ=15.
Then each of the four local iterative denoising algorithms (OSV, OLOSV, IOLOSV and MCM) was run on the noisy test images to obtain a denoised image; the signal-to-noise ratio is then calculated against the noise-free original image. We also ran the non-local means (NLM) algorithm on all three images. For non-local means, 5 × 5 pixel image patches were used to calculate the weight function measuring similarity between patches centred at each pixel in 11 × 11 neighborhoods.
Of the four iterative algorithms, only the proposed OLOSV model with implicit timestepping (IOLOSV) had a well-defined stopping condition, whereas the other 3 algorithms were run until a peak in the calculated signal-to-noise ratio (SNR) was detected. This procedure could actually artificially inflate the SNRs of the other 3 algorithms with respect to IOLOSV, since in practice, the SNR is not easily calculated for an image being denoised without having the denoised image at our disposal.
As already mentioned in the description of the proposed IOLOSV method in Section 5.1, the values of the fidelity parameter λ for the oriented and non-oriented regions of the image are updated dynamically for each of these regions. We stop the denoising process for the IOLOSV method separately for each of the oriented and non-oriented image regions, based on when the dynamic value of λ for each of these regions stabilizes.
8.2 Results and discussion
Final SNRs for local OSV denoising vs. OLOSV, MCM, IOLOSV and NLM denoising on test images
Denoising method | |||||
---|---|---|---|---|---|
Image | OSV | OLOSV | MCM | IOLOSV | NLM |
barbara | 14.219 | 15.101 | 15.716 | 15.845 | 17.313 |
fingerprint | 10.990 | 10.994 | 11.963 | 11.939 | 11.930 |
bridges | 15.991 | 16.143 | 16.214 | 16.423 | 16.834 |
9 Conclusions
The OLOSV decomposition scheme, in both its explicit and implicit forms of implementation, has been found to be extremely good at denoising oriented texture, and is a substantial improvement over the model of Osher, Solé and Vese on which it is based. It also has some other good properties.
For example, one good property of OLOSV is that it can be generalized to images where there are up to two dominant isophote orientations at a point. These orientations can be calculated using the multiple orientation framework in [17]. In regions where there are two orientations present, the model’s evolution equation consists of the sum of two oriented Laplacians with isophote directions ζ _{1} and ζ _{2} and gradient directions η _{1} and η _{2}.
Additionally, it was found from the experiments in this paper that cartoon edges were much less visible in the noise components v of the OLOSV and IOLOSV results when compared with the OSV noise components (e.g. compare Fig. 5 d to Fig. 6 b, d). This is because a directional diffusion was performed close to these cartoon edges (if they were calculated as being in the coherent regions of the image), and along the direction of the edges, and not perpendicular to them. Unfortunately, there tends to be some slight visible smearing of the noise close to the edges, but this could be reduced by only including pixels on cartoon edges in the region in the image computed to have a coherent orientation, and not those around the edges. The detection of such cartoon edges could be implemented with a Total-Variation-like diffusion, similar to [18].
- 1.
Mathematically OLOSV is a novel nonlinear diffusion flow based on the Laplacian of the level-set curvature of the cartoon component, instead of the Laplacian of the image itself, and
- 2.
OLOSV is based on a model for image decomposition, and thus variations of it could potentially be used for other applications of image decomposition, e.g. inpainting.
- 1.
The proposed OLOSV model fits into the Hilbert space formulation of [14], and therefore admits an implicit timestepping implementation, which we called IOLOSV in this paper, as opposed to mean curvature motion which does not. This means that the proposed method is far more efficient; this is confirmed by our experiments, that required only on the order of ten iterations for convergence, as opposed to several hundred for mean curvature motion.
- 2.
The proposed IOLOSV method admits an easy-to-define stopping criterion which allows automatic cessation of execution of the algorithm when the SNR is at or close to its peak value. Such a stopping criterion is not as easily defined for mean curvature motion, necessitating human intervention to determine when the image has in fact been denoised.
Finally, the OLOSV decomposition model can be extended quite easily to colour images by using the colour total variation model of Chan and Blomgren [20]. This remains as future work.
Declarations
Acknowledgements
The authors would like to acknowledge the partial support of this work by a Natural Sciences and Engineering Research Council (NSERC) of Canada Discovery Grant to Cecilia Moloney.
Competing interests
The authors declare that they have no competing interests.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
- Y Meyer, Oscillating Patterns in Image Processing and Nonlinear Evolution Equations: The Fifteenth Dean Jacqueline B. Lewis Memorial Lectures. Am. Math. Soc (2001).Google Scholar
- R Shahidi, C Moloney, Decorrelating the structure and texture components of a variational decomposition model. IEEE Trans. Image Process. 18(2), 299–309 (2009).MathSciNetView ArticleGoogle Scholar
- LA Vese, SJ Osher, Modeling textures with total variation minimization and oscillating patterns in image processing. J. Sci.Comput. 19(1-3) (2003).Google Scholar
- SJ Osher, A Solé, LA Vese, Image decomposition and restoration using total variation minimization and the H ^{−1} norm. Multiscale Model. Simul. SIAM Interdiscip. J. 1(3), 349–370 (2003).MathSciNetView ArticleMATHGoogle Scholar
- M Bertalmio, G Sapiro, V Caselles, C Ballester, in Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. Image inpainting (ACM Press/Addison-Wesley Publishing Co.New York, 2000), pp. 417–424. doi:http://dx.doi.org/10.1145/344779.344972. http://dx.doi.org/10.1145/344779.344972.Google Scholar
- D Paquin, D Levy, E Schreibmann, L Xing, Multiscale image registration. Math. Biosci. Eng. 3:, 389–418 (2006).MathSciNetView ArticleMATHGoogle Scholar
- L Rudin, S Osher, E Fatemi, Nonlinear total variation based noise removal algorithms. Phys. D. 60:, 259–268 (1992).MathSciNetView ArticleMATHGoogle Scholar
- D Tschumperlé, Pde’s based regularization of multivalued images and applications. PhD thesis, Université de Nice - Sophia Antipolis (2002).Google Scholar
- J Weickert, Coherence-enhancing diffusion filtering. Int. J. Comp. Vis. 31(2-3), 111–127 (1999).View ArticleGoogle Scholar
- L Florack, R Duits, G Jongbloed, MC van Lieshout, L Davies, Mathematical Methods for Signal and Image Analysis and Representation (Springer, 2012). https://books.google.ca/books?id=oby5nrg72iQC. Accessed 22 Aug 2016.
- R Terebes, O Lavialle, P Baylou, M Borda, Mixed anisotropic diffusion. ICPR. 3:, 760–763 (2002). doi: doi:http://dx.doi.org/10.1109/ICPR.2002.1048102.
- A Chambolle, An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20:, 89–97 (2004).MathSciNetView ArticleGoogle Scholar
- J-F Aujol, A Chambolle, Dual norms and image decomposition models. International Journal of Computer Vision. 63(1), 85–104 (2005).MathSciNetView ArticleGoogle Scholar
- J-F Aujol, G Gilboa, Constrained and snr-based solutions for tv-hilbert space image denoising. J. Math. Imaging Vis. 26(1-2), 217–237 (2006). doi:http://dx.doi.org/10.1007/s10851-006-7801-6.
- J-F Aujol, G Gilboa, T Chan, S Osher, in Proceedings of Variational, Geometric, and Level Set Methods in Computer Vision 2005, LNCS Vol. 3752. Structure-Texture decomposition by a TV-Gabor model (Springer Berlin HeidelbergBerlin, 2005), pp. 85–96.View ArticleGoogle Scholar
- A Buades, B Coll, J-M Morel, in Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2. A Non-Local Algorithm for Image Denoising (IEEE Computer SocietyWashington, 2005), pp. 60–65. doi:http://dx.doi.org/10.1109/CVPR.2005.38. http://dx.doi.org/10.1109/CVPR.2005.38.Google Scholar
- T Aach, C Mota, I Stuke, M Muhlich, E Barth, Analysis of superimposed oriented patterns. IEEE Trans. Image Process. 15(12), 3690–3700 (2006).MathSciNetView ArticleGoogle Scholar
- R Shahidi, C Moloney, in Proceedings of the International Conference on Image Processing. Variational Textured Image Decomposition with Improved Edge Segregation (IEEEAtlanta, 2006).Google Scholar
- F Catte, PL Lions, JM Morel, T Coll, Image selective smoothing and edge detection by nonlinear diffusion. SIAM J. Numer. Anal.29(1), 182–193 (1992).MathSciNetView ArticleMATHGoogle Scholar
- P Blomgren, TF Chan, Color TV: total variation methods for restoration of vector-valued images. IEEE Trans. Image Process. 7(3), 304–309 (1998).View ArticleGoogle Scholar