Region Adaptive Color Demosaicing Algorithm Using Color Constancy
© ChangWon Kim et al. 2010
Received: 7 October 2009
Accepted: 2 February 2010
Published: 13 April 2010
This paper proposes a novel way of combining color demosaicing and the auto white balance (AWB) method, which are important parts of image processing. Performance of the AWB is generally affected by demosaicing results because most AWB algorithms are performed posterior to color demosaicing. In this paper, in order to increase the performance and efficiency of the AWB algorithm, the color constancy problem is examined during the color demosaicing step. Initial estimates of the directional luminance and chrominance values are defined for estimating edge direction and calculating the AWB gain. In order to prevent color failure in conventional edge-based AWB methods, we propose a modified edge-based AWB method that used a predefined achromatic region. The estimation of edge direction is performed region adaptively by using the local statistics of the initial estimates of the luminance and chrominance information. Simulated and real Bayer color filter array (CFA) data are used to evaluate the performance of the proposed method. When compared to conventional methods, the proposed method shows significant improvements in terms of visual and numerical criteria.
In order to render images captured with a single chip image sensor as a viewable image, an image processing pipeline is required. The most important parts of this image processing pipeline are demosaicing and the automatic white balance (AWB). Since only one color component is available at each pixel, the other two missing color components have to be estimated from the neighboring pixels. This process is referred to as CFA demosaicing or CFA interpolation. The color constancy property of the human visual system allows the perceived color to remain relatively constant at different color temperatures . This capability is required for cameras to generate natural-looking images that match with human perception. The goal of the AWB method is to emulate human color constancy. This is normally achieved by adjusting the image so that it looks as if it were taken under a canonical light (usually daylight).
In recent years, there have been investigations into more sophisticated demosaicing algorithms. Based on the assumption of smooth hue transition, demosaicing is performed using a ratio model which assumes that the ratio between luminance and chrominance at the same position is constant in the neighborhood . Instead of using color ratios, many methods also make use of interchannel color differences which assume that the difference between luminance and chrominance is smooth in small region [4–7]. Since human visual systems are sensitive to the edges in images, edge-directed demosaicing method chooses the interpolation direction to avoid interpolating across edges, instead interpolating along any edges in the image [8–11]. Instead of choosing a certain interpolation direction, the edge indicator function is used in [12–17]. The edge indicator functions in several directions are defined as measures of edge information and a missing pixel is determined as a weighted sum of its neighbors. Color demosaicing is performed by reconstruction approaches [18–21]. Demosaiced image is obtained by deriving Minimum Mean Square Error (MMSE) estimator . Regularization approaches are proposed in  and the color channels are reconstructed using the projections onto convex sets (POCSs) technique . In , the demosaicing problem is formulated as a Bayesian estimation problem. Another recent demosaicing approach, which is referred as decision-based demosaicing algorithm, divided the demosaicing procedure into an interpolation stage and decision stage [22–27]. In the interpolation stage, horizontally and vertically interpolated images are produced, respectively. In the decision stage, soft-decision or hard-decision methods are employed for choosing the pixels interpolated in the direction with fewer artifacts. The Fisher discriminant is used as the decision criterion . Homogeneity map is used to improve the reasonability of the criterion . The second order Taylor series are used to produce directionally interpolated images in the interpolation stage . Demosaicing error is minimized by the directional linear minimum mean square-error estimation technique . In , Chung and Chan presented an adaptive demosaicing algorithm by using the variances of color differences along horizontal and vertical edge directions. However, in the method proposed by Tsai and Song , the decision stage is performed before the interpolation stage.
Various algorithms, such as gray world (GW), perfect reflectors (Max- ), gamut mapping, and color by correlation, have been proposed to maintain the color constancy of an image under different light sources. A good comparison of these algorithms can be found in [28, 29]. The gray world algorithm assumes that the average of the surface reflectance of a typical scene is achromatic [30, 31]. The perfect reflectors algorithm is a simple and fast color constancy algorithm which estimates the light source color from the maximum response of the different color channels . The gamut mapping algorithm is based on the observation that only a limited set of values can be observed under a given illuminant [2, 33]. The basic idea of color by correlation is to precompute a correlation matrix which describes the extent to which proposed illuminants are compatible with the occurrence of image chromaticities [34, 35].
For most AWB algorithms, , , and color components at each pixel are required; thus AWB algorithms are performed after color demosaicing. Therefore, the performance of the AWB is mainly affected by the demosaicing results. In order to increase the performance and efficiency of the AWB algorithms, the color constancy problem can be treated in the color demosaicing step. Color demosaicing that considers color constancy has several advantages. Firstly, computational complexity is reduced. Second, image quality is improved because the AWB method can be performed using original Bayer data which is not degraded by the color demosaicing process. During the color demosaicing process, color information in the fine detail region can be degraded and this introduces false color artifacts. These artifacts influence the AWB gain and also color artifacts can be emphasized by the AWB process. In order to use these advantages and avoid problems, a novel color demosaicing algorithm which uses color constancy is proposed in this paper. For an initial estimate of proposed algorithm, the channel is directionally interpolated using Taylor series approximation and the chrominance channel is calculated using the concept of spectral and spatial correlation (SSC) . An edge-based AWB algorithm is performed using initial estimates instead of using a full color image. The AWB gain is obtained by using a predefined achromatic region and initial estimates in the edge region to prevent color failure situations when more than one uniform object existed in the image. In order to improve the performance and the computational complexity, the region is classified into flat, edge, and pattern regions at each pixel. Based on these preclassified region, the color demosaicing process with color constancy is performed to reduce the number of interpolation errors using the AWB gain and local statistics of the initial estimates.
The rest of this paper is organized as follows. Section 2 provides the motivation for combining color demosaicing and AWB methods by posing the problem. The initial estimate values of and the chrominance channel are defined and the AWB gain is calculated. A detailed explanation of the proposed method and its theoretical improvements are presented. Section 3 presents experimental results of simulated and real CFA data and some comparisons with other algorithms. The paper is concluded in Section 4.
2. A Joint Color Demosaicing Method and the AWB Algorithm
2.1. Initial Estimate of Color Demosaicing and Color Constancy
where represents either or . The superscripts , , , and mean Top, Bottom, Left, and Right direction. is a initial estimate of value at the or position calculated toward top direction. In the case of superscripts , , and the values are calculated toward bottom, left, and right direction, respectively.
where the superscripts , , , and mean Top, Bottom, Left, and Right direction. is a initial estimate of value at the or position calculated toward top direction. In the case of superscripts , , and the values are calculated toward bottom, left, and right direction, respectively. From (1) and (3), initial estimates of the luminance and chrominance values are obtained. Using these values, color constancy gains are calculated and the optimal demosaicing values are determined. The details of obtaining the color constancy gains are explained in the next section.
2.2. Obtaining Color Constancy Gains
For the AWB algorithms, a modified version of the edge-based method is used in this paper. Although the edge-based method is simple, the color constancy accuracy of the method is reasonable , and the average edge difference in the scene is assumed to be achromatic. To prevent color failure when more than one uniform object existed in an image, a predefined achromatic region is used in this paper.
For the edge-based AWB method, edge point detection is required before the AWB process. The region is classified into flat, edge, and pattern regions in the Bayer CFA for more accurate edge direction decisions and for improving the computational efficiency in the color demosaicing process. In order to perform the proposed method, instead of using full color images, Bayer CFA data is used to classify the types of region at the and pixel locations.
Although the effects of dominant color can be alleviated by using predefined achromatic regions, the dominant color problem may still exist, for instance, for texture images because pattern edge regions with similar colors are regarded as edge points . In cases like this, the AWB method only based on the edge detection may perform worse than the GW method. Also, color artifacts can be introduced in the pattern region. Due to the these regions, only normal edge regions are used as edge points for the AWB method in this paper.
where represents an edge pixel, as defined in (9), and represents the total number of edge pixels included in . The obtained color constancy gain is used for the and channels while the channel demosaicing at the and pixel locations.
2.3. Green Channel Demosaicing
In a Bayer CFA, the green plane is sampled at a rate twice as high as in the red and blue planes. Thus, the amount of aliasing in the green plane tends to be less than that in the red and blue planes. The green plane possesses the most spatial information of the image to be interpolated and has a great influence on the visual quality of the image. In most cases, the AWB method alters only the and channels. The channel is kept unchanged because the wavelength of the green color band is close to the peak of the human luminance frequency response. For these reasons, green plane interpolation at the and channels should be performed first.
2.4. Red and Blue Plane Interpolation
where . In this case, is calculated directly, using fully interpolated channel values. is the same value as the one in the green plane, except in a different pixel location. The interpolation of the values at the location and the interpolation of the values are performed in a similar way as interpolation of the value at the location.
3. Experimental Results
Experiments were also performed on 12-bit sensor data taken from a Micron 2-Megapixel image sensor (MT9D111) in an indoor environment with different lighting conditions and in an outdoor environment. The measurement of the performance of the proposed algorithm was divided into two stages. The first stage is the color demosaicing performance and the second stage is the color constancy ability. In the first stage, for comparison, six conventional algorithms, including the methods of Pei and Tam , Hamilton and Adams , Lu and Tan , Wu and Zhang , Li and Randhawa , and Tsai and Song , were implemented. In the second stage, the proposed algorithms were compared with the results of GW, Max-RGB, Shades of Gray, and Gray edge .
PSNR comparison of color demosaicing algorithms.
NCD comparison of color demosaicing algorithms.
In Figures 12 and 13, the results of the simulated Kodak1 and Kodak 19 images (in which detailed and fine structured regions appear) can be seen. From this visual comparison, it is clear that most conventional methods suffered from zipper effects along the edges. The methods proposed by Wu and Zhang, Li and Randhawa, and Tsai and Song showed good results but they still produce more color artifacts than the proposed method. These experimental results explain that the proposed method performs satisfactorily not only in textured regions but also in normal edge regions.
The computational complexity comparison.
Edge based AWB
Average running time (sec)
A method of recovering white balanced and full color images from color sampled data was presented in this paper. In order to avoid the problem of treating these methods separately and increasing the computational efficiency, a simultaneous color demosaicing and AWB scheme was proposed. Initial estimates were calculated for AWB weight and color demosaicing by using second-order Taylor series approximation and an SSC assumption. The gray edge assumption was used to achieve color constancy and a predefined achromatic region was used to avoid dominant color problem. Region adaptive color demosaicing was performed to improve the performance and the computational complexity. The experiments verified that the proposed method effectively suppressed color artifacts while preserving the details, texture, and fine structures in the images and showed well-white-balanced images. The results of the proposed algorithm indicate that it outperformed conventional algorithms in both quantitative and qualitative criteria. Moreover, the proposed algorithm was more computationally efficient than when the AWB method and color demosaicing were treated separately. The future research in this area include a new approach to combine other color demosaicing and AWB methods and experiments with other sensors to improve the algorithm.
This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MEST) through the Biometrics Engineering Research Center (BERC) at Yonsei University (2009-0062990) and by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2009-0079024).
- Bayer BE: Color imaging array. US patent 3 971 065, July 1976Google Scholar
- Forsyth D: A novel algorithm for color constancy. International Journal of Computer Vision 1990, 5(1):5-36. 10.1007/BF00056770MathSciNetView ArticleGoogle Scholar
- Cok DR: Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal. US patent 4 642 678, Febraury 1987Google Scholar
- Adams JE Jr.: Interactions between color plane interpolation and other image processing functions in electronic photography. Cameras and Systems for Electronic Photography and Scientific Imaging, February 1995, San Jose, Calif, USA, Proceedings of SPIE 2416: 144-151.View ArticleGoogle Scholar
- Adams JE, Hamilton JF Jr.: Adaptive color plane interpolation in single color electronic camera. US patent 5 506 619, April 1996Google Scholar
- Adams JE Jr.: Design of practical color filter array interpolation algorithms for digital cameras. Real-Time Imaging II, February 1997, San Jose, Calif, USA, Proceedings of SPIE 3028: 117-125.View ArticleGoogle Scholar
- Pei S-C, Tam I-K: Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 2003, 13(6):503-513. 10.1109/TCSVT.2003.813422View ArticleGoogle Scholar
- Hibbard RH: Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients. US patent 5 382 976, January 1995Google Scholar
- Laroche CA, Prescott MA: Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients. US patent 5 373 322, December 1994Google Scholar
- Hamilton JF Jr., Adams JE: Adaptive color plane interpolation in single sensor color electronic camera. US patent 5 629 734, May 1997Google Scholar
- Adams JE Jr., Hamilton JF Jr.: Adaptive color plane interpolation in single sensor color electronic camera. US patent 5 652 621, July 1997Google Scholar
- Kimmel R: Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 1999, 8(9):1221-1228. 10.1109/83.784434View ArticleGoogle Scholar
- Hur BS, Kang MG: Edge-adaptive color interpolation algorithm for progressive scan charge-coupled device image sensors. SPIE Optical Engineering 2001, 40(12):2698-2708.View ArticleGoogle Scholar
- Park SW, Kang MG: Color interpolation with variable color ratio considering cross-channel correlation. SPIE Optical Enginerring 2004, 43(1):34-43.MathSciNetGoogle Scholar
- Lu W, Tan Y: Color fiter array demosaicking: new method and performance measures. IEEE Transactions on Image Processing 2003, 12(10):1194-1210. 10.1109/TIP.2003.816004View ArticleGoogle Scholar
- Ramanath R, Snyder WE: Adaptive demosaicking. Journal of Electronic Imaging 2003, 12(4):633-642. 10.1117/1.1606459View ArticleGoogle Scholar
- Kim C, Kang MG: Noise insensitive high resolution color interpolation scheme considering cross-channel correlation. SPIE Optical Engineering 2005, 44(12):-15.Google Scholar
- Trussel H, Hartwig R: Mathematics for demosaicking. IEEE Transactions on Image Processing 2002, 11: 485-492. 10.1109/TIP.2002.999681MathSciNetView ArticleGoogle Scholar
- Keren D, Osadchy M: Restoring subsampled color images. Machine Vision Applied 1999, 11(4):197-202. 10.1007/s001380050102View ArticleGoogle Scholar
- Gunturk BK, Altunbasak Y, Mersereau RM: Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 2002, 11(9):997-1013. 10.1109/TIP.2002.801121View ArticleGoogle Scholar
- Mukherjee J, Parthasarathi R, Goyal S: Markov random field processing for color demosaicing. Pattern Recognition Letters 2001, 22(3-4):339-351. 10.1016/S0167-8655(00)00129-XView ArticleMATHGoogle Scholar
- Wu XL, Zhang N: Primary-consistent soft-decision color demosaicking for digital cameras (patent pending). IEEE Transactions Processing 2004, 13(9):1263-1274. 10.1109/TIP.2004.832920View ArticleGoogle Scholar
- Hirakawa K, Parks TW: Adaptive homogeneity-directed demosaicing algorithm. IEEE Transactions on Image Processing 2005, 14(3):360-368.View ArticleGoogle Scholar
- Li JSJ, Randhawa S: High order extrapolation using taylor series for color filter array demosaicing. Proceedings of the International Conference on Image Analysis and Recognition (ICIAR '05), September 2005, Toronto, Canada, Lecture Notes in Computer Science 3656: 703-711.View ArticleGoogle Scholar
- Zhang L, Wu X: Color demosaicking via directional linear minimum mean square-error estimation. IEEE Transactions on Image Processing 2005, 14(12):2167-2178.View ArticleGoogle Scholar
- Chung K-H, Chan Y-H: Color demosaicing using variance of color differences. IEEE Transactions on Image Processing 2006, 15(10):2944-2955.View ArticleGoogle Scholar
- Tsai C-Y, Song K-T: Heterogeneity-projection hard-decision color interpolation using spectral-spatial correlation. IEEE Transactions on Image Processing 2007, 16(1):78-91.MathSciNetView ArticleGoogle Scholar
- Barnard K, Cardei V, Funt B: A comparison of computational color constancy algorithms—part I: methodology and experiments with synthesized data. IEEE Transactions on Image Processing 2002, 11(9):972-984. 10.1109/TIP.2002.802531View ArticleGoogle Scholar
- Barnard K, Martin L, Coath A, Funt B: A comparison of computational color constancy algorithms—part II: experiments with image data. IEEE Transactions on Image Processing 2002, 11(9):985-996. 10.1109/TIP.2002.802529View ArticleGoogle Scholar
- van de Weijer J, Gevers T, Gijsenij A: Edge-based color constancy. IEEE Transactions on Image Processing 2007, 16(9):2207-2214.MathSciNetView ArticleGoogle Scholar
- Lin J: An automatic white balance method based on edge detection. Proceedings of the 10th IEEE International Symposium on Consumer Electronics (ISCE '06), 2006 1-4.Google Scholar
- Land E, McCann J: Lightness and retinex theory. Journal of the Optical Society of America A 1971, 61(1):1-11. 10.1364/JOSA.61.000001View ArticleGoogle Scholar
- Finlayson GD, Hordley SD, Tastl I: Gamut constrained illuminant estimation. International Journal of Computer Vision 2006, 67(1):93-109. 10.1007/s11263-006-4100-zView ArticleGoogle Scholar
- Finlayson GD, Hubel PH, Hordley S: Color by correlation. Proceedings of the 5th IS&T/SID Color Imaging Conference: Color Science, Systems and Applications, November 1997, Scottsdale, Ariz, USA 6-11.Google Scholar
- Finlayson GD, Hordley SD, Hubel PM: Color by correlation: a simple, unifying framework for color constancy. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001, 23(11):1209-1221. 10.1109/34.969113View ArticleGoogle Scholar
- Khriji L, Cheikh FA, Gabbouj M: High-resolution digital resampling using vector rational fiters. SPIE Optical Engineering 1999, 38(5):893-901.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.