- Open Access
An improved method for color image editing
EURASIP Journal on Advances in Signal Processing volume 2011, Article number: 98 (2011)
In this article, an improved approach for editing color images has been presented. In this approach, using the Poisson equation, a guided vector field is created by employing source and target images within a selected region at the first step. Next, the guided vector is used in generating the result image. Most of the existing techniques in the literature perform image editing without taking color information into account. However, without utilizing color information, image geometry cannot be created properly in some cases which may result in unsatisfactory results. Unlike the existing methods, the proposed study utilizes all the information contained in each of the color channels in computing gradient norm and performing inpainting process. The test results show that the suggested technique generates satisfactory results in editing color images.
Digital image processing operations are related with global changes including image correction, filtering, colorization, or local changes in a selected region where altering processes take place. Commercial or artistic photomontages take the local changes in images into account. Along with the technologic improvement, some softwares, such as Adobe© Photoshop©, have been released for image editing. However, professional experience is required to be able to use those kinds of softwares efficiently, and editing photos by the indicated software is tiresome. In addition, the regions altered using those tools may include some visible artifacts.
Digital image editing methods based on the Poisson equation have frequently been used in recent researches [1–12]. Perez et al.  presented an image editing approach based on the Poisson equation with Dirichlet boundary conditions. However, using this method, color inconsistency may occur in edited regions of images. Sun et al.  proposed an image matting algorithm with the Poisson equation. However, because of a long processing time, the method is not practically usable. Chuan et al.  improved the method that was suggested by Perez et al. to overcome the color inconsistency problem. But, the experiments show that the improved method is still too complex. Leventhal and Sibley  suggested an alpha interpolation technique to remove brightly colored artifacts caused by mixed seamless image cloning in edited images. Jia et al.  presented a method, called drag-and-drop pasting, which computes an optimized boundary condition automatically by employing a new objective function. However, the method is compared not with mixed seamless image cloning but with only seamless image cloning approach proposed in . Georgiev [6–8] developed a new method that is invariant to relighting and handles seamlessly illumination change including adaptation and perceptual correctness of results. The method processes the image by considering its surrounding texture contrast as well. Fattal et al.  presented a method to render high dynamic range images on a monitor using a Poisson equation. Shen et al.  suggested a method whose outputs are generated from the gradient maps by employing a Poisson equation. Dizdaroğlu and İkibaş  introduced a color image editing method with the Poisson equation, on which this study was built. Yang et al.  proposed a variational model, a distance-enhanced random walks algorithm and a multi-resolution framework based on the Poisson image editing method. Although many methods were summarized related with the research area, most of those methods are complex or may cause artifacts because of independent implementation of each color channel, or using only the lightness channel.
In this article, we present a digital image editing method that utilizes all the information contained in each of the color channels based on the Poisson equation. The test results show that the proposed method generates output images in a seamless manner with no blurring artifacts.
2. Image editing
Let f: Ω→Rn and f: Ω→R be color (n = 3) and grayscale images, respectively, and they are defined on domain of Ω→R2 . f i : Ω→R stands for the image channel i of f(1 ≤ i ≤ n): ∀p = (x, y)∈Ω. The proposed method will be explained in detail in the following sections.
The Poisson image editing is basically a process of obtaining a new result image f using the source image g and target image f*. In this method, the guided vector field v is first created using both the images g and f*. After that, the method reconstructs the image f with Dirichlet boundary conditions so that the gradient of f is closest in the L 2 -norm to v over the region Γ, while satisfying f = f* over ∂Γ [1, 4]. Therefore, the edited image region includes the features of both images g and f*, and matches the rest of the image in a seamless manner (see Figure 1).
A basic approach for editing process is to minimize the variation of the image f over the region Γ by estimating the gradient norm ||∇f||, which is also known as Tihkonov regularization :
In Equation 2, the image gradient denoted by ∇f is the first-order derivative of the grayscale image f with respect to its spatial coordinates p = (x, y):
In order to represent magnitudes of the grayscale image f and its maximum variation directions, a vector ∇f: Γ→R2 is created. Scalar and point-wise measurement of the image variations are given by the gradient norm ||∇f||, which is used in image analysis in many cases:
Here, f x and f y are the first-order derivatives of the image f in x and y directions, respectively. These can be calculated using Taylor's formula. Figuring out the function f minimizing the functional E(f), requires complex processing operations. A necessary condition is given by the Euler-Lagrange equations, which must be confirmed by f to reach a minimum of E(f):
The calculation of and according to the standard differentiation rules is as follows:
Accordingly, is found as a solution to Equation 5. Constant 2 may be removed from the equation for simplicity.
A classic iterative method called gradient descent is used for solving the partial differential equation (PDE) in Equation 1. As a matter of fact, Equation 1 can be seen as the gradient of the functional E(f). A local minimizer fmin of E(f) can be found by starting from finitial and then following the opposite direction of the gradient:
Solving the equation:
is found. Here, Δ is the Laplace operator, and also Taylor's formula is used in finding second-order derivatives of f xx and f yy . Equation 7 is called as diffusion or heat equation.
The regularization PDE shown in Equation 8 is compatible with the approach expressed above:
where dt represents adapting time step.
As clearly followed above, the image is gradually blurred in an isotropic way during the PDE evolution. Here, isotropic smoothing acts as a low-pass filter suppressing high frequencies in image of f. Unfortunately, since image edges and noise are both high frequency signals, the edges are quickly blurred as seen in Figures 2 and 3c. Therefore, the interpolation operation based on a guided vector field could generate a better result .
A function, minimizing the difference between the gradient ∇f of image f and guided vector field v over the region Γ, should be found. The equation given below can be used for this operation:
The result is found when the above equation is solved by the Poisson equation with Dirichlet boundary conditions as follows:
Here, is the divergence of v = (u,v). This is a fundamental equation used in digital image editing.
The guided vector field v can directly be obtained from the source image g. In this case, the interpolation operation is conducted considering the equation as follows:
Equation 12 is obtained for seamless image cloning, if Equation 10 is reorganized accordingly:
Therefore, the source image is seamlessly cloned to the target image without blurring artifacts within the user-selected region as seen in Figure 3d. If the source image is directly copied to the destination image, some visual annoying effects may reveal in the result image as shown in Figure 3b.
Seamless image cloning method may result in blurring artifacts in the image editing region if the user-selected boundary intersects with prominent structures in the target image as seen in Figure 4c. To solve this problem, another technique is introduced in . At each point of the region Γ, if stronger variations on any of the source image g or the target image f* is to remain the same, the following vector field can be used for mixed seamless image cloning as seen in Figure 4e.
Another approach for image editing is arranging local illumination changes . In this method, the guided vector field can be defined as follows to arrange the illumination changes in any region of the image as shown in Figure 5a,b.
with α = 0.2 times the average gradient norm of f* over Γ and, β = 0.2.
Also, the gradient ∇f* of the target image can be passed through a sparse sieve . Therefore, it is retained only the most prominent features on the image, which is called texture flattening. This is performed by applying on Equation 15:
where M is a binary mask obtained using an edge detection approach as depicted in Figure 6a,d.
An efficient method presented in [6–8] utilizes not only pixels values but also texture contrasts while editing the selected region seamlessly as illustrated in Figure 7. In this figure, we cannot perceive pixel values precisely because of the fact that a variable background encloses a rectangle which is a constant color. Here, whereas pixel gradient is zero, perceived, or covariant gradient is not zero. Consequently, the following equation is invariant relighting and considers illumination changes [6–8] as seen in Figure 8a,g.
This approach, covariant Laplace equation, can be considered as a Poisson equation with a modified Δg term. Actually, the above equation can be seen as equal to by calculating simple differentiation. The equation is solved in three main steps for covariant inpainting which fills the image holes by propagating linear structures of the source region into the target region by means of diffusion process utilizing the Poisson equation approach [6, 8].
These steps are
Dividing the result image f by the source image g.
Solving the PDE in the user-selected region.
Multiplying the result by the source image g.
3. The proposed method
The method studied in  utilizes each color channel of the image in the Poisson equation independently. In other words, classical scalar diffusion PDEs on each channel f i of the initial image finitial is used while processing color images in many applications. However, since each image channel f i evolves independently with different smoothing geometries, this approach is useless because of linear structures of the image that have different directions for each vector components f i . Even if these directions are relatively close to each others, the diffusion process will not behave in a coherent way and a high risk of vector components blending (that will blur the edges) may occur . In addition to this, the exact geometric structure of the image may not be obtained if all the information contained in each of the color channels is considered. Consequently, the image structure may inaccurately diffuse to the target region. Figure 9 shows the lightness channel component of the test image in CIE-Lab color space. Examining the image carefully, it is seen that the lightness channel of the image was not able to be obtained properly. In this case, as can clearly be seen in Figure 9b, insufficient results may be obtained when the image is edited based on gradient norm of the lightness channel. Therefore, image editing should be done by considering the effects of color channels on each other.
Before explaining the proposed color image editing in details, it is worth to mention the essentials of inpainting color images. In recent years, many notable approaches have been developed for image inpainting, or completion [14, 15, 17–22]. Our proposed method employs trace-based PDE method in which the local geometry of the color image can be calculated based on all the information contained in each of the color channels as follows:
J is defined as follows for a color image f = (R,G,B):
The positive eigenvalues λ+/- and the orthogonal eigenvectors θ+/- of J are formulized as follows:
More consistent geometry is obtained, provided that Jσ = J*Gσ is smoothed by a Gaussian filter. Here, Jσ is a good estimator of the local geometry of f at point p. Its spectral elements give the vector-valued variations (by the eigenvalues λ+, λ-, of Jσ) at the same time and the orientations (edges) of the local image structures (by the eigenvectors θ+⊥θ- of Jσ).
Tschumperlé and Deriche  suggested designing a particular field T: Γ→P(2) of diffusion tensors to define the specification of the local smoothing method for the regularization process. Apparently, it should be noticed that T, depending on the local geometry Jσ of f, is thus defined from the spectral elements λ+, λ-, and θ+, θ- of Jσ:
The strengths of smoothing along θ+and θ- are set by two functions denoted by s-/+: R2 →R, where the types of applications determine s- and s+. Sample functions for image denoising are proposed in :
Here, the goal of smoothing operation can be listed as follows:
The pixels on image edges are smoothed along θ- with a strength inversely relative to the vector edge strength (anisotropic smoothing).
The pixels on homogeneous regions are smoothed along all possible directions (isotropic smoothing).
Tschumperlé and Deriche  present a regularization-PDE-based approach to agree with the local smoothing geometry T based on a trace operator:
where H i is the Hessian matrix of f i :
In this study, H i is a symmetric matrix since the images are regular ones and .
As Tschumperlé and Deriche  have shown, Equation 24 can be viewed as a local filtering approach with oriented and normalized Gaussian kernels. Here, a small convolution is locally applied around each point p with a 2D Gaussian mask oriented by the tensor T(p):
As a matter of fact, a link exists between anisotropic diffusion PDE and classical filtering techniques:
The regularization PDE shown in Equation 27 is compatible with all local geometric properties expressed above:
with the diffusion tensor .
Here, in our seamless image cloning method, each image channel f i evolves independently with different smoothing structures as depicted in Figure 11. However, unlike the method proposed in , we apply a cutting approach on pixel values in the processed image and a linear normalization method on the result image to obtain a better visual result as seen in Figure 12e.
Equation 13 employed in mixed seamless cloning method can be rewritten as below by considering all the information contained in each of the color channels:
In our method, the color image gradient norm is used to obtain the image structure more accurately as shown in Figure 10b.
The mentioned methods in [6, 8, 9] were applied only on the lightness channel of the image to arrange local illumination changes and to flatten texture. Therefore, we modify the arrangement of local illumination changes and texture flattening methods as stated in Equations 31 and 32, respectively.
where M is the binary image obtained by employing ||∇f*(p)||.
Covariant inpainting method presented in [6, 8] completes the user-selected region in a seamless manner, which takes into account both pixel values and texture contrast of the surrounding area of the edited region as seen Figure 8f,g. However, in the method diffusion process is not coherently employed for all the color image channels f i . Therefore, we extend the method in [6, 8] for color image editing to diffuse the image structure accurately to the target region as depicted in Figures 13f and 14b. Namely, to solve Equation 33 for inpainting step, we use a trace-based method , which is explained above.
Therefore, we calculate only the gradient norm by taking the interaction among the channels into account in order to edit color images using the proposed methods. However, diffusion process in these methods, except for our covariant inpainting approach, has been done separately for each color channel.
4. Experimental results
The proposed method is compared with the previous approaches suggested in [1, 5–8, 12, 15, 18–21] by utilizing seamless and mixed seamless image cloning, arrangement of local lightness variation, texture flatting, multi-resolution cloning considering the color fidelity, covariant inpainting, image completion, curvature-preserving PDE-based inpainting, combined PDE and texture synthesis, and variational inpainting approaches. The methods are tested on color images containing RGB color space and some images in referenced papers were utilized in these performed tests. The proposed editing operation is performed on only the selected region marked by the user.
The test results of mixed seamless image cloning using only the lightness channel and our method are shown in Figure 15. Whereas our method is able to preserve the image structures in the result image, the method considering the lightness channel fails to protect image edges because of using only the lightness channel of the images while obtaining image gradient norms, as seen in Figure 15c,d.
The test results of seamless image cloning method suggested in  and our seamless image cloning approach are depicted in Figure 12. Both methods give almost the same results. But, the result generated by our method has slightly more vibrant colors.
The test results of seamless and mixed seamless image cloning suggested in  and our proposed method are given in Figure 4. Both approaches developed for seamless editing method cause blurring on some parts of the edited region of the target image. However, mixed seamless image cloning method does not cause any blurriness on the result image. As seen in Figure 4f, there is almost no color inconsistency on the result image generated by the proposed method since all the information contained in each of the color channels is used in editing the image. This situation can be clearly observed on variation of color of fume in the result image.
The test results for seamless image cloning proposed in  and mixed seamless image cloning in our study are given in Figure 16. The former method performs complex operations to find an optimized boundary condition. The method uses a shortest closed-path algorithm which is designed to search for the location of the boundary. Its additional computational complexity is O(MN), where M and N stand for the number of pixels in the band of the boundary. Also, the method employs a blended guidance field to incorporate the object's alpha matte to preserve the object's fractional boundary properly. Unlike that method, our proposed method is simple, and it is as effective as the former one. However, in Figure 16g, sea waves can be realized through the man's body because of the fact that they have prominent structures against to his body. This problem may be solved using a modified Poisson matting method. The additional time complexity of our method over the mixed seamless cloning method in  to compute the color image gradient norm is O((n - 1)L), where n is the channel number of color images, and L is the number of pixels in the edited region.
Other test results for image cloning are given in Figure 17. The existing seamless image cloning method may diffuse prominent global coloring of the object toward the background as seen in Figure 17d. The method proposed in  takes the color fidelity term into account to avoid the problem observed in Figure 17e. Our seamless and mixed seamless image cloning methods generate acceptable results; however, some artifacts may come into existence in the neighborhood of the user-selected region. Also, dominant structures of some grasses can be seen through the squirrel's body in our mixed seamless image cloning method as shown in Figure 17g.
Another experiment illustrates how to conceal balustrade. The test results generated by the proposed method are given in Figure 18e,f. Figure 18b-d shows the results of methods proposed in [19, 15, 18], respectively. Unlike other methods, our seamless image cloning method and the approach proposed in  produces visually plausible results on the brick wall and trees as seen in Figure 18d,e. However, curvature-preserving PDE-based method in  produces too much blurring effects, as depicted in Figure 18c and our mixed seamless image cloning method fails to remove the object from the image, because the gradient of the object is more dominant against the user-selected patch as seen in Figure 18f.
The results obtained by applying local lightness arrangement methods are given in Figures 5 and 19. As observed in Figures 5a-c and Figure 19a-c, the edited region is almost flat and the gradient norms in these regions are low. For this reason, both the methods suggested in  and our proposed approach produce acceptable results in terms of visual quality.
The results of texture flattening methods are given in Figure 6. It is seen that our method properly flattens the selected region on the test image.
The results for removing a scratch from a shadow region are depicted in Figure 8. Both covariant cloning methods in [6–8] and our covariant inpainting approach not only correct the lighting but also reconstruct texture in a seamless manner as shown in Figure 8g,h. However, constrained PDE-based method in  and Poisson cloning approach in [6, 8] generate visually annoying effects as seen in Figure 8e,f.
The test results for concealing a blotch on a film frame are given in Figure 20. Our covariant inpainting method generates a visually plausible result while the image completion method in  produces some blocking artifacts on the result image. However, the method in  works fully automatically, which is not the case for our covariant inpainting approach. Actually, direct cloning and our seamless image cloning methods give good results for this film frame as well, as seen in Figure 20b,c. However, our mixed seamless image cloning method fails to generate the efficient result because of the dominant structure of the object, which the blotch cannot be concealed as shown in Figure 20d.
The results of other tests on defect removal are shown in Figure 13. Our covariant inpainting method, our seamless image cloning approach, and the image completion method in  produce better results compared with other methods as seen in Figure 13c,e,f. However, direct cloning, our mixed seamless image cloning methods give visually annoying effects. We are also able to compare our covariant inpainting method with the method presented in [6, 8] since it was applied to a grayscale form of the original image. Compared to the method in [6, 8], our covariant inpainting approach generates less blurring artifacts and linear structures of the given image is propagated better, as shown in Figure 14a,b.
Lastly, another test result is given for object removal. Figure 21 shows the result of constrained PDE-based algorithm in , variational inpainting method in , and our covariant inpainting approach. Unlike other methods, our method gives a good result; however, two different patches must be selected by the user because of not being able to remove the object from the image efficiently.
The methods were implemented in Microsoft Visual C++ 2005 by employing CImg Library. The program was run on a PC with Pentium 2.20 GHz processor and 2 GB RAM. The average required time depends typically on selected regions as well as image editing methods. The processing time for our covariant inpainting method shown in Figure 13f is about 5 s for seven iterations.
In this article, a method is presented for editing color images by effectively using all the information contained in each of the color channels. In this method, gradient norm and inpainting process are utilized by considering the effects of color channels on each other. This approach minimizes the color inconsistency and thus the selected region in the source image is cloned to the target image seamlessly.
As a future task, automatic region selection using moment invariants may be developed so that the proposed method is improved to be able to generate a faster output. In addition to this, the method will be extended to remove defects such as blotches from old motion pictures since there are proper patches in neighbor frames to retouch the current one .
Perez P, Gangnet M, Blake A: Poisson image editing. ACM Trans Graph 2003,22(3):313-318. 10.1145/882262.882269
Sun J, Jia J, Tang CK, Shum HY: Poisson matting. ACM Trans Graph 2004,23(3):315-321. 10.1145/1015706.1015721
Chuan Q, Shuozhong W, Xinpeng Z: Image editing without color inconsistency using modified poisson equation. International Conference on Intelligent Information Hiding and Multimedia Signal Processing 2008.
Leventhal D, Sibley PG: Poisson image editing extended. International Conference on Computer Graphics and Interactive Techniques, ACM SIGGRAPH 2006.
Jia J, Sun J, Tang C, Shum H: Drag-and-drop pasting. International Conference on Computer Graphics and Interactive Techniques, ACM SIGGRAPH 2006.
Georgiev T: Covariant derivatives and vision. ECCV 2006, 9th European Conference on Computer Vision, Graz, Austria 2006.
Georgiev T: Photoshop healing brush: a tool for seamless cloning. Workshop on Applications of Computer Vision in conjunction with ECCV 2004, Prague 2004, 1-8.
Georgiev T: Image reconstruction invariant to relighting. EUROGRAPHICS 2005.
Fattal R, Lischinski D, Werman M: Gradient domain high dynamic range compression. Proceedings of ACM SIGGRAPH 2002 2002, 249-256.
Shen J, Jin X, Zhou C, Wang CCL: Gradient based image completion by solving the Poisson equation. Comput Graph 2007,31(1):119-126. 10.1016/j.cag.2006.10.004
Dizdaroğlu B, İkibaş C: A seamless image editing technique using color information. IPCV'09: International Conf. on Image Processing, Computer Vision & Pattern Recognition, Las Vegas, USA 2009.
Yang W, Zheng J, Cai J, Rahardja S, Chen CW: Natural and seamless image composition with color control. IEEE Trans Image Process 2009,18(11):2584-2592.
Tikhonov AN: Regularization of incorrectly posed problems. Soviet Math Dokl 1963, 4: 1624-1627.
Tschumperlé D, Deriche R: Vector-valued image regularization with PDE's: a common framework for different applications. IEEE Trans Pattern Anal Mach Intell 2005,27(4):506-517.
Tschumperlé D: Fast anisotropic smoothing of multi-valued images using curvature-preserving PDE's. Int J Comput Vis 2006,68(1):65-82. 10.1007/s11263-006-5631-z
Dizdaroğlu B: Solving heat flow equation for image regularization. In The 1st International Symposium on Computing in Science & Engineering (ISCSE). Aydin, Turkey; 2010:372-377.
Tschumperlé D: PDE's based regularization of multivalued images and applications. PhD thesis. Université de Nice-Sophia Antipolis; 2002.
Dizdaroğlu B: An image completion method using decomposition. EURASIP J Adv Signal Process 2011. Article ID 831724, 15
Harald G: Combined PDE and texture synthesis approach to inpainting. Proc ECCV 2004, 2: 214-224.
Barnes C, Shechtman E, Finkelstein A, Goldman DB: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph 2009,28(3):24:1-24:11.
Bugeau A, Bertalmío M, Caselles V, Sapiro G: A comprehensive framework for image inpainting. IEEE Trans Image Process 2010,19(10):2634-2645.
Dizdaroğlu B, Gangal A: A spatiotemporal algorithm for detection and restoration of defects in old color films. Lecturer Notes in Computer Sciences 2007, 4678: 509-520. 10.1007/978-3-540-74607-2_46
Tschumperlé D: The C++ template image processing library, The CImg Library.[http://cimg.sourceforge.net/]
6. Competing interests
The authors declare that they have no competing interests.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Dizdaroğlu, B., İkibaş, C. An improved method for color image editing. EURASIP J. Adv. Signal Process. 2011, 98 (2011). https://doi.org/10.1186/1687-6180-2011-98
- Poisson equation
- guided interpolation
- editing color images
- inpainting color images
- calculation of gradient of color images
- seamless and mixed seamless image cloning
- arranging local illumination changes
- texture flattening
- covariant inpainting