Bayer patterned high dynamic range image reconstruction using adaptive weighting function

Abstract

It is not easy to acquire a desired high dynamic range (HDR) image directly from a camera due to the limited dynamic range of most image sensors. Therefore, generally, a post-process called HDR image reconstruction is used, which reconstructs an HDR image from a set of differently exposed images to overcome the limited dynamic range. However, conventional HDR image reconstruction methods suffer from noise factors and ghost artifacts. This is due to the fact that the input images taken with a short exposure time contain much noise in the dark regions, which contributes to increased noise in the corresponding dark regions of the reconstructed HDR image. Furthermore, since input images are acquired at different times, the images contain different motion information, which results in ghost artifacts. In this paper, we propose an HDR image reconstruction method which reduces the impact of the noise factors and prevents ghost artifacts. To reduce the influence of the noise factors, the weighting function, which determines the contribution of a certain input image to the reconstructed HDR image, is designed to adapt to the exposure time and local motions. Furthermore, the weighting function is designed to exclude ghosting regions by considering the differences of the luminance and the chrominance values between several input images. Unlike conventional methods, which generally work on a color image processed by the image processing module (IPM), the proposed method works directly on the Bayer raw image. This allows for a linear camera response function and also improves the efficiency in hardware implementation. Experimental results show that the proposed method can reconstruct high-quality Bayer patterned HDR images while being robust against ghost artifacts and noise factors.

1 Introduction

Image capturing devices like digital cameras and camcorders have recently improved remarkably. However, image sensors such as charge-coupled devices (CCD) and complementary metal-oxide semiconductors (CMOS) in these imaging devices can still only capture a limited dynamic range. As a result, when a captured scene contains a dynamic range above the given limitation, a loss of information is inevitable even if the exposure is adjusted according to the brightness of the scene. Thus, many methods based on signal processing have been proposed to reproduce scenes with a high dynamic range (HDR).

To obtain HDR images, many HDR imaging approaches utilize low dynamic range (LDR) images with different exposures [1–11]. Most of these approaches first convert the pixel values of input images into radiance values by using the camera response function (CRF), where the CRF refers to the function that maps the radiance values of a given scene to the pixel values in the captured image, and the radiance refers to the physical quantity of light energy on each element on the sensor array. Next, the radiance values of the input images are combined into a single HDR image using weighting functions based on the reliability of the input data.

In early studies, conventional approaches were proposed to estimate the CRF from multiple LDR images. These CRF estimation approaches can be categorized as parametric [1, 2] and non-parametric approaches [3, 4]. In parametric approaches, Mann and Picard [1] presented a variety of parametric forms for CRF estimation. Mitsunaga and Nayar [2] used a high-order polynomial function to estimate the CRF. On the other hand, in terms of non-parametric approaches, Debevec and Malik [3] estimated the CRF using an objective function with a smoothness constraint. Pal et al. [4] used a Bayesian network consisting of a probabilistic model for an imaging function and a generative model for smooth functions.

In recent years, some techniques have been proposed to prevent artifacts in HDR images caused by moving objects [5–10]. If local motion occurs in a scene while the LDR images are being captured, a ghost artifact appears in the HDR image. Most of the ghost artifact-preventing techniques first detect the local motion by using ghost artifact measurement and then combine the LDR images without the ghost artifact regions.

All these approaches, however, combine LDR images without considering the influence of noise factors. Dark regions in an LDR image taken with short exposure time contain relatively more noise factors because the regions are under-exposed or noisy. Thus, the noise level is increased in the corresponding dark regions of the reconstructed HDR image. These approaches also utilize RGB images processed by the image processing module (IPM). The IPM is a processor which converts Bayer raw data into an RGB image suitable to the human visual system [12–14]. Using RGB images processed by the IPM makes the CRF estimation inaccurate since the IPM includes several adaptive nonlinear sub-modules, e.g., dynamic range compression, noise reduction [15], and color correction [16] algorithms. An inaccurate CRF leads to poor HDR imaging performance. Moreover, the use of RGB images increases the hardware complexity since the IPM has to be performed several times before the HDR image reconstruction, as shown in Figure 1a.

Figure 1

Comparison of hardware complexity when using (a) conventional methods and (b) the proposed method.

In this paper, we introduce a new approach which performs HDR image reconstruction on the Bayer raw images before the IPM [17], as shown in Figure 1b. The proposed method can be widely used in applications such as real-time HDR video cameras because of the reduction of hardware complexity. The CRF estimation is simpler and more accurate than conventional methods because the CRF before the IPM is linear. The proposed method considers the noise and the ghost artifact problems. For this purpose, a new weighting function is proposed to combine the Bayer patterned LDR (BP-LDR) images. The proposed weighting function is designed so that each of the BP-LDR images independently covers each corresponding region according to the radiance value in order to reduce the influence of the noise. The regions covered by each BP-LDR image are determined by the exposure of each BP-LDR image and the existence of local motion. This avoids using the short-exposure BP-LDR image to reconstruct the dark regions in the Bayer patterned HDR (BP-HDR) image. The weighting function also detects the local motion in the Bayer pattern and excludes ghosting regions. To detect the local motion, the luminance and the chrominance values are directly calculated in the Bayer pattern, and the differences of these values are utilized. When the proposed method is compared with conventional methods, the detection performance is improved since an accurate CRF is employed.

The rest of this paper is organized as follows: In Section 2, the proposed BP-HDR image reconstruction approach is described in detail. The properties of the CRF are discussed and analyzed in Section 2.1. Section 2.2 describes the design process of the adaptive weighting function for BP-HDR image reconstruction. In Section 3, experimental results of various test images are presented, and the paper is concluded in Section 4.

2 Proposed BP-HDR image reconstruction

2.1 Properties of the camera response function

In general, the radiance value I^R which passes through the lens of the camera is converted into a pixel value I by the image sensor and the IPM part. The CRF is a function that relates the radiance value to the pixel value and is expressed as

$$I=f\left(I^R·\Delta t\right)$$

(1)

where Δ t represents the exposure time.

CCD and CMOS image sensors are widely used in image acquisition systems. Both of these sensors utilize the same kind of element called photodiode [18], which generates a current proportional to the light energy. The photodiode offers a high level of resistance when there is no light falling on it. On the other hand, the resistance of the photodiode is reduced and the current increases linearly with the light energy when light falls on it. That is, the response function of the photodiode is linear with respect to the radiance value. An example under the condition with 6500 k fluorescent lighting using a CMOS sensor with Bayer pattern is shown in Figure 2. In the plot of the pixel value versus the intensity of the radiance, each of these response functions appear to be linear.

Figure 2

Camera response function of the RGB channels in the Bayer raw data.

Although the relationship between the light energy and the image sensor output is linear, the CRF is usually non-linear due to the IPM. In the IPM, the linear response function is intentionally converted into a non-linear function to produce an image attractive to the human visual system. The response function generated by the IPM is often designed to mimic the nonlinearity of film, where the film response function is designed to produce attractive images [19, 20]. Moreover, the response function generated by the IPM also varies for every pixel due to spatially adaptive processing modules for image enhancement. Thus, it is not appropriate to estimate the CRF with images acquired from the IPM’s output. Therefore, we use the CRF before the IPM and apply it to the BP-LDR images to reconstruct the BP-HDR image.

In the BP-LDR images, the CRF is expressed linearly as f(x)=α x+β as shown in Figure 2. Here, α represents the slope of the CRF corresponding to the sensitivity of the RGB channels. That is, α differs according to the color channel of the Bayer pattern. However, there is no need to estimate α because the auto white balance sub-module of IPM adjusts the slopes of the RGB channels to be equal [14]. Therefore, f(·) can be approximated as f(x)=x+β. β represents the black level of an image sensor which can be simply estimated as the average of the optical black region located on the boundary of the image sensor [21].

2.2 Proposed design method of the weighting function for BP-HDR image reconstruction

The proposed method combines a set of differently exposed Bayer raw images to obtain a single BP-HDR image. The BP-LDR images are combined with different weights determined by the weighting function. The weighting function consists of the data reliability term and the ghost artifact reduction term, and it utilizes the linear property of the CRF. The data reliability term determines how much a certain BP-LDR should be reflected in the BP-HDR image. The data reliability term is designed to exclude the influence of noise factors. The ghost artifact reduction term excludes ghosting regions from the reconstruction process. The reconstructed BP-HDR image ($Ê$) is calculated as

$$Ê(i,j)=\frac{\sum _{n=1}^N{W}_n\left(i,j\right)·f^{-1}\left(I_n\left(i,j\right)\right)/\Delta t_n}{\sum _{n=1}^N{W}_n\left(i,j\right)}$$

(2)

where W _n represents the weighting function corresponding to the n-th BP-LDR image, I _n (i,j) represents the given pixel value at the position (i,j) of the n-th BP-LDR image, Δ t _n represents the exposure time of I _n , and N is the number of BP-LDR images. For convenience, we arranged I₁,I₂,….,I _n ,…,I _N such that Δ t₁>Δ t₂>….>Δ t _n >….>Δ t _N . Furthermore, f represents the CRF explained in the previous section, and thus f^-1(I _n )/Δ t _n represents the radiance value of I _n . Therefore, (2) can be regarded as an equation which combines the radiance values of the BP-LDR into the BP-HDR image.

Figure 3 shows a block diagram of the proposed method regarding the estimation of the proposed weighting function. In order to estimate the weighting function W _n , first, the weight for the data reliability ($W_n^r$) and the weight for the ghost artifact reduction ($W_n^l$) have to be estimated. The data reliability weight $W_n^r$ is further modified to ${Ŵ}_n^r$ to reflect the change of the data reliability according to the ghosting regions in the other LDR images. Then, finally, ${Ŵ}_n^r$ and $W_n^l$ are combined to obtain the final weighting function W _n :

$$W_n(i,j)={Ŵ}_n^r(i,j)·W_n^l(i,j),$$

(3)

Figure 3

Block diagram for the proposed weighting function.

The details are described in the following sections.

2.2.1 The weight for data reliability

Conventional methods use a symmetric weighting function that decreases with the distance from the center of the pixel value range [3, 11]. This is based on the fact that the pixel values near the center are the most reliable. If considering only the influence of the noise, the higher part of the pixel value range seems to be reliable since the shot noise in general image sensor has a Poisson distribution. However, since the higher part is close to the saturation value and since post-processing methods such as the gamma correction make the higher part more close to the saturation value, a smaller weight should be assigned to the higher part to prevent the use of the saturated pixel values in the reconstruction process. Figure 4a shows two examples of weighting functions that are generally used. These weighting functions are converted into the functions in the radiance domain, as shown in Figure 4b. Using the function values in the radiance domain, the ratio of how much the BP-LDR images are combined into the BP-HDR image is calculated. For example, the radiance value L₁ in Figure 4b has a weight of about 0.6 (point a) in the middle exposure BP-LDR image, while the weights in the other BP-LDR images are about 0.2 (point b). This means the weight for the middle exposure is about three times larger than the weights for the other exposures at L₁. In general, it can be observed that a BP-LDR image captured with long exposure has a large weight in the low-radiance region, while a BP-LDR image captured with short exposure has a large weight in the high-radiance region. However, a major problem is that the short-exposure BP-LDR image has a considerable weight in the very low radiance region. Using the short-exposure BP-LDR image in the low-radiance region is improper since it can be under-exposed or noisy in this region.

Figure 4

Two examples of conventional weighting functions. (a) In the pixel value domain and (b) in the radiance domain.

To overcome the abovementioned problems, we use a designing method for the reliability function. The designing method consists of two steps as listed in Algorithm ??. The first step is to design a desired weighting function (${\tilde{W}}_n^r$) in the radiance domain. The second step is the conversion of the designed weighting function in the radiance domain into that in the pixel value domain.

In the first step, the design of ${\tilde{W}}_n^r$ has to be done so that for a certain n, ${\tilde{W}}_n^r$ covers only a certain region in the radiance domain, as shown in Figure 5b. That is, we design the weighting function so that the short-exposure BP-LDR image has an effect only in the high-radiance region, while the long-exposure BP-LDR image has an effect only in the low-radiance region. This reduces the amount of noise in the reconstructed BP-HDR image, since the short-exposure BP-LDR image, which normally contains noise in the dark regions, is not used to reconstruct the low-radiance region in the BP-HDR image.

Figure 5

Proposed weighting function for reliability of the data. (a) In the pixel value domain and (b) in the radiance domain.

To make the design simple, the following constraints are used:

• 1. The two adjacent functions ${\tilde{W}}_n^r$ and ${\tilde{W}}_{n+1}^r$ intersect at the same point as in conventional methods.

• 2. All of the functions ${\tilde{W}}_n$ have the same slopes at the transition region.

By obeying the first constraint, the most reliable BP-LDR image for a certain radiance range becomes consistent with that in conventional methods. However, since the conventional weighting functions are designed to adapt to the RGB images processed by the image processing module such as the gamma correction, they are improper for the BP-LDR images. Therefore, we modify the conventional weighting functions to adapt to the BP-LDR images. That is, the maximum value position (ρ) in the weighting functions is changed from the center of the pixel value range to a more reliable position as will be later mentioned in Section 3. By obeying the second constraint, the change of the two weights between different exposure LDR images becomes close to linear. This can prevent artifacts which can be possibly generated due to a nonlinear change in the weight ratio. To reduce the overlap between adjacent weighting functions while obeying the above mentioned constraints, the slopes at the transition region have to be increased.

The proposed weighting functions are calculated starting with that corresponding to the longest exposure image. As shown in Figure 5b, ${\tilde{W}}_1^r$ for the longest exposure BP-LDR image I₁ is asymmetric with respect to point ${\mu }_1^H$ as

$$\begin{array}{l}{\tilde{W}}_1^r\left(I_1^R\right)=\left\{\begin{array}{l}1,\text{if}{I}_1^R<{\mu }_1^H\hfill \\ exp\left(-C·{\left(I_1^R-{\mu }_1^H\right)}^2\right),\text{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(4)

where

$${\mu }_1^H=\frac{f^{-1}\left(\rho \right)}{\Delta t_1}.$$

(5)

Here, C represents the parameter that controls the slope of the function ${\tilde{W}}_1^r$.

Obeying the second constraint, C should be the same for all of ${\tilde{W}}_n^r$. For $I_1^R<{\mu }_1^H$, ${\tilde{W}}_1^r$ has the value 1 since I₁ is the most reliable in this range. For $I_1^R\ge {\mu }_1^H$, ${\tilde{W}}_1^r$ is the Gaussian function with mean ${\mu }_1^H$. C is determined by the function value δ at the intersection point γ₁ of ${\tilde{W}}_1^r$ and ${\tilde{W}}_2^r$, i.e., $\delta ={\tilde{W}}_1^r\left({\gamma }_1\right)={\tilde{W}}_2^r\left({\gamma }_1\right)$. It is calculated from (4) as

$$C=-\frac{log\left(\delta \right)}{{\left({\gamma }_1-{\mu }_1^H\right)}^2}.$$

(6)

As described above, the intersection point γ _n of ${\tilde{W}}_n^r$ and ${\tilde{W}}_{n+1}^r$ is the same as in conventional methods. The procedure of calculating γ _n is described in the Appendix. The result is

$${\gamma }_n=\frac{\rho (I_{max}-\beta )-\beta (I_{max}-\rho )}{\Delta t_n\rho +\Delta t_{n+1}(I_{max}-\rho )},$$

(7)

where I_max represents the maximum value of the BP-LDR images (4,095 for 12-bit images). ${\tilde{W}}_n^r$ for n=2,…,N-1 can be calculated by the following equation:

$$\begin{array}{l}{\tilde{W}}_n^r\left(I_n^R\right)=\left\{\begin{array}{l}exp\left(-C·{\left(I_n^R-{\mu }_n^L\right)}^2\right),\text{if}{I}_n^R<{\mu }_n^L\hfill \\ 1,\text{if}{\mu }_n^L<I_n^R<{\mu }_n^H\hfill \\ exp\left(-C·{\left(I_n^R-{\mu }_n^H\right)}^2\right),\text{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(8)

where ${\mu }_n^L$ and ${\mu }_n^H$ are the mean values of the Gaussian functions applied to the low part and the high part in ${\tilde{W}}_n^r$, respectively. Since ${\tilde{W}}_{n-1}^r$ and ${\tilde{W}}_n^r$ intersect at the point γ_n-1 and are symmetric around γ_n-1, ${\mu }_n^L$ is obtained from ${\mu }_{n-1}^H$:

$${\mu }_n^L=2{\gamma }_{n-1}-{\mu }_{n-1}^H.$$

(9)

Likewise, ${\mu }_n^H$ is calculated from (8) as

$$\begin{array}{ll}exp\left(-C·{({\gamma }_n-{\mu }_n^H)}^2\right)\hfill & =\delta \hfill \\ {\gamma }_n-{\mu }_n^H\hfill & =\sqrt{-\frac{log\left(\delta \right)}{C}}\hfill \\ {\mu }_n^H\hfill & ={\gamma }_n-\sqrt{-\frac{log\left(\delta \right)}{C}}.\hfill \end{array}$$

(10)

The various coefficients to design the proposed weighting function ${\tilde{W}}_n^r$ are marked in Figure 6. The weighting function ${\tilde{W}}_n^r$ corresponding to the shortest exposure image I _N is asymmetric with respect to ${\mu }_n^L$:

$$\begin{array}{l}{\tilde{W}}_N^r\left(I_N^R\right)=\left\{\begin{array}{l}exp\left(-C·{\left(I_N^R-{\mu }_N^L\right)}^2\right),\text{if}{I}_N^R<{\mu }_N^L\hfill \\ 1,\text{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(11)

Figure 6

Illustration of the data reliability weighting function${\tilde{W}}_n^r$. The coefficients to design the function are included in this figure.

For $I_N^R<{\mu }_N^L$, ${\tilde{W}}_n^r$ is a Gaussian function with mean ${\mu }_n^L$. For $I_N^R\ge {\mu }_N^L$, $W_n^r$ becomes 1 since I _N is the most reliable in this range.

After the desired weighting function is designed by the abovementioned step, it has to be converted to a weighting function in the pixel value domain. This is done by step 2 in Algorithm ??. By step 2, ${\tilde{W}}_1^r$, ${\tilde{W}}_n^r$, and ${\tilde{W}}_n^r$ in (4), (8), and (11) are converted into $W_1^r$, $W_n^r$, and $W_n^r$, respectively, as

$$\begin{array}{ll}{W}_1^r\left(I_1\right)\hfill & =\left\{\begin{array}{l}1,\text{if}{I}_1<f\left({\mu }_1^H\Delta t_1\right)\hfill \\ exp\left(-C·{\left(\frac{f^{-1}\left(I_1\right)}{\Delta t_1}-{\mu }_1^H\right)}^2\right),\text{otherwise},\hfill \end{array}\right.\end{array}$$

(12)

$$\begin{array}{ll}{W}_n^r\left(I_n\right)\hfill & =\left\{\begin{array}{l}exp\left(-C·{\left(\frac{f^{-1}\left(I_n\right)}{\Delta t_n}-{\mu }_n^L\right)}^2\right),\text{if}{I}_n<f\left({\mu }_n^L\Delta t_n\right)\hfill \\ 1,\text{if}f\left({\mu }_n^L\Delta t_n\right)\le I_n<f\left({\mu }_n^H\Delta t_n\right)\hfill \\ exp\left(-C·{\left(\frac{f^{-1}\left(I_n\right)}{\Delta t_n}-{\mu }_n^H\right)}^2\right),\text{otherwise},\hfill \end{array}\right.\end{array}$$

(13)

$$\begin{array}{ll}{W}_N^r\left(I_N\right)\hfill & =\left\{\begin{array}{l}exp\left(-C·{\left(\frac{f^{-1}\left(I_N\right)}{\Delta t_N}-{\mu }_N^L\right)}^2\right),\text{if}{I}_N<f\left({\mu }_N^L\Delta t_N\right)\hfill \\ 1,\text{otherwise}.\hfill \end{array}\right.\end{array}$$

(14)

The weighting function $W_n^r$ obtained in this section is further modified to consider local motion by the method described in Section 2.2.3.

2.2.2 The weight for ghost artifact reduction

In general, any movement introduces ghost artifacts when input images with different exposures are sequentially acquired. Figure 7 shows the effect of these ghost artifacts. Figure 7a shows three input images with different exposures, and Figure 7b shows the HDR image result using a conventional Gaussian weight [11]. The movement of the people while the input images were being captured caused the ghost artifacts of the HDR image. The artifacts are shown in the close-up image (Figure 7c). To prevent the ghost artifacts, the region where local motion occurs has to be excluded from the reconstruction process.

Figure 7

An example of the ghost artifact in an HDR image. (a) Input images, (b) the result containing ghost artifact, and (c) close-up of the red box in (b).

The proposed method uses the weighting function to exclude the ghost artifacts. Small weights are assigned to the pixels where ghost artifacts occur to exclude the ghosting regions from the reconstruction process. Before calculating the weighting function, the image with the fewest saturated and dark pixels is selected as the reference image. From the correspondence between the reference image and the other BP-LDR images, the weighting function $W_n^l$ is decomposed into the switching (s _n ) and the weighting ($W_n^l$) components as

$$W_n^l\left(i,j\right)=s_n\left(i,j\right)·w_n^l\left(i,j\right).$$

(15)

The switching component s _n is defined based on the fundamental assumption that the pixel value captured at a long exposure should be larger than that at a short exposure. Therefore, if the pixel value at a short exposure is larger than at a long exposure, this means that one of the two pixel values is wrong and should be excluded from the reconstruction process. Therefore, s _n becomes

$$s_n\left(i,j\right)=\left\{\begin{array}{ll}1,\hfill & \text{if}(i,j)\in M_n\text{or}n=n_0\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\right.$$

(16)

where n₀ denotes the reference image and M _n represents the region in I _n which satisfies the abovementioned fundamental assumption. Even though the switching component is not used, the weighting component $W_n^l$ assigns small weights to these regions. However, the switching component is effective to reduce ghost artifacts because these regions are completely excluded from the reconstruction process by the switching component.

The weighting component $W_n^l$ is determined by the differences of the luminance and chrominance values between the reference image and the other BP-LDR images. The regions with large differences can be regarded as regions where local motion occur, and thus small weights have to be assigned to these regions. On the other hand, large weights have to be assigned to regions with small differences. The above description is applied to $W_n^l$, and it is assigned to the pixels as

$$\begin{array}{ll}{w}_n^l\left(i,j\right)=\hfill & exp\left(-C_Y·D_n^Y\left(i,j\right)-C_R·D_n^R\left(i,j\right)\right.\hfill \\ \left.-C_B·D_n^B\left(i,j\right)\right).\hfill \end{array}$$

(17)

Here, $D_n^Y$ represents the difference of the luminance values between $I_{n_0}$ and I _n . $D_n^X$, where X denotes the red (R) or blue (B) channel, represents the difference of the chrominance values between $I_{n_0}$ and I _n . The parameters C _Y , C _R , and C _B are chosen to balance between $D_n^Y$, $D_n^R$, and $D_n^B$, respectively.

To obtain $D_n^Y$ and $D_n^X$, first, the BP-LDR images are adjusted to the same exposure setting because the exposures of the BP-LDR images differ. The procedure to adjust the exposure can be expressed in terms of the formula below:

$$I_{n_1\to n_2}=min\left\{f\left(f^{-1}\left(I_{n_1}\right)\times \frac{\Delta t_{n_2}}{\Delta t_{n_1}}\right),I_{max}\right\}$$

(18)

where $I_{n_1\to n_2}$ represents the image $I_{n_1}$ of which the exposure time changes from $\Delta t_{n_1}$ to $\Delta t_{n_2}$, and min{a,b} represents the minimum value between a and b. From the exposure adjustment in (18), the exposure time of $I_{n_1}$ changes from $\Delta t_{n_1}$ to $\Delta t_{n_2}$ in the radiance domain, and then it is converted in the pixel value domain. After adjusting the exposures, $D_n^Y$ is calculated as

$$D_n^Y\left(i,j\right)=\frac{\left|{\tilde{Y}}_{n_0}\left(i,j\right)-{\tilde{Y}}_n\left(i,j\right)\right|}{max\left\{{\tilde{Y}}_{n_0}\left(i,j\right),{\tilde{Y}}_n\left(i,j\right)\right\}}$$

(19)

where

$${\tilde{Y}}_n\left(i,j\right)=\sum _{\left(p,q\right)\in S}{G}_{\sigma }\left(p,q\right)·Y_n\left(i+p,j+q\right).$$

(20)

Here, max{a,b} represents the maximum value between a and b, Y _n represents the luminance component of I _n , G _σ (·) represents the corresponding truncated Gaussian kernel with variance σ², and S represents an index set of a support, i.e., a rectangular windowed search range. The normalization of $\left|{\tilde{Y}}_{n_0}\left(i,j\right)-{\tilde{Y}}_n\left(i,j\right)\right|$ to $D_n^Y\left(i,j\right)$ is to compensate for the difference according to the level of luminance. The luminance component Y _n is calculated in the Bayer pattern from neighboring pixels as

$$\begin{array}{ll}{Y}_n\left(i,j\right)\hfill & =\frac{1}{4}·I_n(i,j)+\frac{1}{8}·\left(I_n(i-1,j)+I_n(i,j-1)\right.\hfill \\ \left.+I_n(i+1,j)+I_n(i,j+1)\right)\hfill \\ +\frac{1}{16}·\left(I_n(i-1,j-1)+I_n(i-1,j+1)\right.\hfill \\ \left.+I_n(i+1,j-1)+I_n(i+1,j+1)\right).\hfill \end{array}$$

(21)

By applying (21), Y _n contains 50% green, 25% red, and 25% blue colors regardless of the position. $D_n^X$ is also calculated after adjusting the exposures as follows:

$$D_n^X\left(i,j\right)=\frac{\left|{\tilde{K}}_{n_0}^X\left(i,j\right)-{\tilde{K}}_n^X\left(i,j\right)\right|}{max\left\{{\tilde{Y}}_{n_0}^X\left(i,j\right),{\tilde{Y}}_n^X\left(i,j\right)\right\}}$$

(22)

where ${\tilde{K}}_n^X$ is calculated by

$$\begin{array}{ll}{\tilde{K}}_n^X\left(i,j\right)=\hfill & \sum _{\left(p,q\right)\in S}{G}_{\sigma }\left(p,q\right)·\left(Y_n\left(i+p,j+q\right)\right.\\ \left.-X_n\left(i+p,j+q\right)\right).\hfill \end{array}$$

(23)

Here, X _n represents the pixel value of the X channel in I _n , which is calculated by the bilinear interpolation depending on the position.

Figure 8 shows the results of obtaining $D_n^Y$, $D_n^R$, and $D_n^B$ between the reference image (middle exposure) and the other two BP-LDR images. The regions where $D_n^Y$, $D_n^R$, and $D_n^B$ are large are regarded as the local motion regions. In Figure 8, it can be seen that the local motion is detected effectively. Although it is difficult to detect the local motion in the dark region of the short-exposure BP-LDR image due to the influence of the noise, there is no problem since the short-exposure BP-LDR image is rarely used to reconstruct dark regions of the scene.

Figure 8

Ghost artifact regions calculated from the captured BP-LDR images.

2.2.3 The weight for data reliability considering local motion

If the weighting function $W_n^r$ obtained in the previous section as ${Ŵ}_n^r$ in (3) is used, this will cause a problem: a certain local motion can cause artifacts. For easy understanding, let us consider a situation in which two LDR images have different exposures. Suppose that the image with shorter exposure is the reference image. Figure 9 illustrates this situation. Here, the local motion region is shown in the longer exposure image I₁ (marked by light gray in Figure 9c), and the black region is shown in the shorter exposure image I₂ (marked by dark gray in Figure 9c). When calculating W _n for n=1,2 by (3) using $W_n^r$, they have weighting values W₁≈0 and W₂≈0, respectively. This is due to the fact that the local motion region in I₁ leads to $W_1^l\approx 0$, and the black region in I₂ leads to $W_2^r\approx 0$. As a result, an artifact appears since no LDR images are utilized in the intersection of these two regions (marked by black in Figure 9c). To avoid the artifact, the weighting function $W_n^r$ has to be modified.

Figure 9

An example of a situation in which the weighting functions W _n for all n become 0. (a) A long-exposure image I₁, (b) a short-exposure image I₂ (reference image), and (c) a weight image (light gray: W₁≈0, dark gray: W₂≈0, black: W₁,W₂≈0).

The modified weighting function ${Ŵ}_n^r$ should increase its weight if a local motion occurs in the opponent BP-LDR image to compensate for the small weighting value of the local motion region. For example, again, regarding the case where the image I₁ is obtained with the longest exposure, I₂ with the second longest, I₃ with the third, and so on for I_n=4,5,…,N. In this case, first, the weighting function ${Ŵ}_1^r$ for the longest exposure BP-LDR image I₁ is composed of $W_1^r$ and a partial weighting of $W_2^r$ to compensate for the local motion in I₂. In detail, when a local motion is detected in a certain region in I₂, a partial weight of $W_2^r$ is added to ${Ŵ}_1^r$ for this region. The modified weight ${Ŵ}_1^r$ does not include the partial weights of $W_{n=3,\dots ,N}^r$, since $W_{n=3,\dots ,N}^r$ are almost 0 in the radiance value range of I₁. Therefore, ${Ŵ}_1^r$ can be expressed as follows:

$$\begin{array}{ll}{Ŵ}_1^r\left(i,j\right)=\hfill & W_1^r\left(I_1\left(i,j\right)\right)\hfill \\ +W^s\left(I_1\left(i,j\right)\right)·\left(1-W_2^l\left(i,j\right)\right)·W_2^r\left(I_{1\to 2}\left(i,j\right)\right).\hfill \end{array}$$

(24)

The weight $W_2^l$ is the weight defined in (15) for the case of image I₂. It has a small value if the likelihood of a local motion in I₂ is large. If $W_2^l$ has a small value, the weight $W_2^r$ is much reflected in the weight ${Ŵ}_1^r$. The image I_1→2 is used to compute $W_2^r$, where I_1→2 is calculated from (18). Here, W^s(·) represents a simple hat function [7] as shown in Figure 10, which prevents the use of saturated values in I _n :

$$W^s\left(I_n\right)=1-{\left(2·\frac{I_n}{I_{max}}-1\right)}^{12}.$$

(25)

Figure 10

The hat function W ^s to avoid using the saturation level.

Next, we consider the case for ${Ŵ}_2^r$. The weight ${Ŵ}_2^r$ includes the weighting values $W_2^r$ and $W_3^r$ in the same manner as explained in the abovementioned case. Furthermore, the weight $W_1^r$ is additionally included in ${Ŵ}_2^r$. This is due to the fact that the radiance range of I₂ includes the radiance range of I₁, which is again due to the fact that I₂ is obtained with a shorter exposure than I₁. Therefore, using I₂ to compensate for the local motion in I₁ and I₃, ${Ŵ}_2^r$ becomes

$$\begin{array}{ll}{Ŵ}_2^r\left(i,j\right)=& W_2^r\left(I_2\left(i,j\right)\right)\\ +W^s\left(I_2\left(i,j\right)\right)·\left(1-W_3^l\left(i,j\right)\right)·W_3^r\left(I_{2\to 3}\left(i,j\right)\right)\\ +\left(1-W_1^l\left(i,j\right)\right)·W_1^r\left(I_{2\to 1}\left(i,j\right)\right).\end{array}$$

(26)

Here, the weight W^s is not required for the third term in the right hand side of (26), since the radiance value range captured by I₁ corresponds to the low-radiance part of I₂.

Similarly, ${Ŵ}_3^r$ contains the weights $W_3^r$, $W_1^r$, $W_2^r$, and $W_4^r$:

$$\begin{array}{ll}{Ŵ}_3^r\left(i,j\right)=& W_3^r\left(I_3\left(i,j\right)\right)\\ +W^s\left(I_3\left(i,j\right)\right)·\left(1-W_4^l\left(i,j\right)\right)·W_4^r\left(I_{3\to 4}\left(i,j\right)\right)\\ +\left(1-W_2^l\left(i,j\right)\right)·W_2^r\left(I_{3\to 2}\left(i,j\right)\right)\\ +\left(1-W_2^l\left(i,j\right)\right)·\left(1-W_1^l\left(i,j\right)\right)·W_1^r\left(I_{3\to 1}\left(i,j\right)\right).\end{array}$$

(27)

The fourth term compensates for the case if local motion is detected in both I₁ and I₂. If local motion occurs only in I₂, then the third term in (27) would be enough. Otherwise, the case that local motion occurs only in I₁ is already compensated for by the third term in (26).

Generalizing the case for n (except for n = 1 and n = N), ${Ŵ}_n^r$ can be represented as

$$\begin{array}{ll}{Ŵ}_n^r\left(i,j\right)=& W_n^r\left(I_n\left(i,j\right)\right)\\ +W^s\left(I_n\left(i,j\right)\right)·\left(1-W_{n+1}^l\left(i,j\right)\right)\\ ·W_{n+1}^r\left(I_{n\to n+1}\left(i,j\right)\right)\\ +\sum _{k=1}^{n-1}\left(\left(\prod _{p=1}^k\left(1-W_{n-p}^l\left(i,j\right)\right)\right)\right.\\ \left.·W_{n-k}^r\left(I_{n\to n-k}\left(i,j\right)\right)\right).\end{array}$$

(28)

Finally, the weighting function ${Ŵ}_n^r$ is used as the data reliability term in (3).

3 Experimental results

The performance of the proposed algorithm was tested with several BP-LDR images, which were captured with a CMOS sensor at three different exposures (Δ t₁=t, Δ t₂=t/4, and Δ t₃=t/16). The BP-LDR images have a pixel value range of 0≤intensity≤4,095 (12-bit). The 12-bit pixel value range is widely used for digital cameras. Later, the 12-bit range is compressed to 8-bit RGB data by the IPM.

With the proposed method, several parameters were set empirically and tested with various images to obtain the best results. The parameter ρ in (5) was set to $\frac{2}{3}{I}_{max}$. The parameter δ in (6), which determines the degree of the overlap between the data reliability weights, was set to 0.25. The parameters C _Y , C _R and C _B in (17) were set to 20, 10, and 10, respectively. The kernel size S and the variance σ² in (20) and (23) were set to 5×5 and 4, respectively. No pre-processes were performed on the input BP-LDR image, but pre-processes such as bad pixel correction [22] and Gr-Gb imbalance correction [23] can improve the HDR result according to the quality of the imaging sensor. For better visualization, we showed the results in RGB images rather than in Bayer patterned images. All the input BP-LDR images were post-processed by the edge-preserving color interpolation [24], white balancing, color correction, and gamma correction. For the resulting BP-HDR images, an additional tone-mapping algorithm [25] was used to compress the dynamic range, which visualizes the HDR image information on a low dynamic range display.

We compared the performance with respect to the influence of noise at dark regions and ghost artifacts with three conventional methods. The first conventional method (CM1) uses the weighted summation using the Gaussian weighting function without considering ghost artifact reduction in [11]. The second (CM2) and third (CM3) method are commercial software programs that are widely used to obtain an HDR image with ghost artifact reduction in [26, 27], respectively. In CM2 and CM3, the parameters associated with ghost artifact removal were set to the highest level. For CM2 and CM3, the BP-LDR images were preprocessed by the same edge-preserving color interpolation, white balancing, and gamma correction algorithms which are used for the visualization.

First, we performed an experiment when there was no local motion. Figure 11 shows our HDR result applied to a scene containing both indoor and outdoor environments. As can be seen in Figure 11a, the limited dynamic range of the imaging sensor revealed saturation and black regions in the LDR images. In the short-exposure LDR image, the pixels in the bright regions avoided saturation, but the details in the dark regions disappeared. On the other hand, in the long exposure LDR image, the dark regions, e.g., the part under the desk, became visible, but the bright regions became saturated. In comparison, with the proposed method, the fine details became visible in both the bright and the dark regions, as can be seen in Figure 11b. Figure 11c visualizes the values of the weighting functions W_n=1,2,3 in colors. The R, G, and B channels were assigned to W₁, W₂, and W₃, respectively. For example, a red region represents the fact that the value of W₁ is dominant in that region. As a result, in the reconstruction of the BP-HDR image, the long-exposure BP-LDR image has a dominant effect on the region under the desk, while the short-exposure BP-LDR image has a dominant effect on the sky region.

Figure 11

Experimental results of the captured scene with window. (a) The three LDR images, (b) the result of the proposed method, and (c) the weight image obtained by W _n .

Figure 12 shows the results of the three CMs and the proposed method applied to a scene containing very dark regions around the test chart. Figure 12a shows the ‘chart’ images with different exposures. Figure 12b,c,d,f shows the results of the CMs and the proposed method, respectively. To confirm the effect of noise, the lower left corners of the results were magnified as shown in Figure 13. The proposed results were less affected by the noise compared with the result of the CMs. Especially, with the CM2 and the CM3, more noise was observed in the dark regions compared with the others. This is due to the fact that the weight for the long-exposure BP-LDR image decreased in the process of excluding ghosting regions. The long-exposure BP-LDR image appears less noisy in the dark regions than the other images. However, the proposed method provides better performance than the CM1 although our method considered ghost artifacts unlike the CM1. Figure 14a,b shows the weight images of Figure 12b,e, which correspond to the weighting functions used in the CM1 and the proposed method, respectively. As can be observed, with the CM1, the surrounding dark region revealed similar small weights for all the BP-LDR images. In comparison, with the proposed method, W _n assigned a dominant weight for the long-exposure BP-LDR image in the dark region, which made the dark region visible.

Figure 12

Experimental results of the captured scene with chart. (a) The three LDR images, (b) CM1, (c) CM2, (d) CM3, and (e) the proposed method.

Figure 13

Close-up comparison of Figure12. (a) CM1, (b) CM2, (c) CM3, and (d) the proposed method.

Figure 14

The weight images of Figure12. (a) CM1 and (b) the proposed method.

Figure 15 shows the results for a scene with a desk. The scene contained a very dark region under the desk as shown in Figure 15a. In Figure 15, it appears that the noise under the desk was reduced effectively in the results of the CM2 and the proposed method, while the CM1 and the CM3 did not reduce the influence of noise and even generated some artifacts in the light source. However, with the CM2, the details in both the bright and dark regions were disappeared when compared with the other results. To evaluate the performance with respect to the influence of noise, the standard deviation (σ) and the coefficient of variation (CV) were used as objective performance criteria. The CV is defined as the ratio of the standard deviation σ to the mean μ, i.e., σ/μ. The CV is useful for comparison when the means in the result images are different from each other. Table 1 presents the standard deviations and the CVs of the proposed method and the CMs, which are calculated in the homogeneous dark regions of Figures 13 and 15. As described in Table 1, the PM recorded a smaller standard deviation and CV values than the CMs. From Table 1, it is clear that the proposed method outperforms the CMs in numerical values.In the second experiment, we performed experiments on captured scenes that included object movements. Figure 16 shows the results for a scene with leaves blowing in the wind. Figure 16b shows the ghost artifacts around the leaves since CM1 cannot consider local motion. The CM2 cannot remove the ghost artifacts effectively, as can be seen in Figure 16c. Figure 16d,e shows the results when using the CM3 and the proposed method, respectively. Both results show almost no ghost artifacts when compared with the results of the CM1 and the CM2. As a result, the CM3 and the proposed method are able to prevent ghost artifacts from small local motion like blowing leaves.Figure 17 shows the results for a scene with moving people. The captured input images are shown in Figure 7a. As can be seen, the motion regions are extremely large. The ghost artifacts are observed with all of the conventional methods. However, the proposed method introduces no ghost artifacts. Figure 18 shows a close-up of Figure 17. In Figure 18a, false color artifacts are observed in bright regions. This is due to the fact that saturated pixels in the longest exposure BP-LDR image are used in the reconstruction process. The reconstructed radiance regions affected by these saturated pixels become falsely achromatic. Meanwhile, the BP-HDR image performs post-processing such as white balancing since the colors in the Bayer raw images may be unbalanced. As a result, false colors may occur in the achromatic regions. Moreover, as can be seen in Figure 18, conventional methods cannot detect the person wearing black pants in front of the black desk because their brightness values are similar. In comparison, the proposed method prevents ghost artifacts and produces a high-quality BP-HDR image, as shown in Figure 18d.We presented the results of the proposed method together with the input images in Figure 19. Figure 19 demonstrates that the proposed method provides BP-HDR images without any artifacts.

Figure 15

Experimental results of the captured scene with desk. (a) The three LDR images, (b) CM1, (c) CM2, (d) CM3, and (e) the proposed method.

Table 1 Comparison of experimental results in quantitative terms

Figure 16

Experimental results of the captured scene with leaves blowing in the wind. (a) The three LDR images, (b) the result of CM1, (c) the result of CM2, (d) the result of CM3, and (e) the result of the proposed method.

Figure 17

Experimental results of the captured scene with moving people. (a) The result of CM1, (b) the result of CM2, (c) the result of CM3, and (d) the result of the proposed method.

Figure 18

Close-up comparison of Figure17. (a) The result of CM1, (b) the result of CM2, (c) the result of CM3, and (d) the result of the proposed method.

Figure 19

Results of the proposed BP-HDR image reconstruction method with various test images.

4 Conclusion

In this paper, we have proposed a BP-HDR (Bayer patterned high dynamic range) image reconstruction algorithm from multiple BP-LDR (Bayer patterned low dynamic range) images. Unlike conventional methods, the proposed method works on the Bayer raw image. This allows for a linear CRF and also improves the efficiency in hardware implementation. The proposed method aims to deal with the noise and the ghost artifact problems. For this aim, a new weighting function is proposed to be designed so that each of the BP-LDR images independently covers its corresponding region according to the radiance value. Furthermore, the weighting function is designed to detect the local motion in the Bayer pattern and to exclude ghosting regions. As a result, the proposed method weakens the influence of noise in the short-exposure BP-LDR image and prevents ghost artifacts. Experimental results show that the proposed method produces a high-quality BP-HDR image while being robust against ghost artifacts and noise factors, even when there exists excessive local motion.

\thelikesection Appendix

\thelikesubsection Procedure for calculating intersection point γ _n

The intersection point γ _n of ${\tilde{W}}_n^r$ and ${\tilde{W}}_{n+1}^r$ is determined as the same point obtained from the conventional weighting function. We use the simple weighting function to calculate γ _n as

$$W_n^c\left(x\right)=\left\{\begin{array}{l}\frac{1}{\rho }·x,\text{if}x<\rho \hfill \\ \frac{-1}{I_{max}-\rho }(x-\rho )+1,\text{otherwise},\hfill \end{array}\right.$$

(29)

The weighting function $W_n^c$ is shown in the top part of Figure 4. The falling part of $W_n^c$ and the rising part of $W_{n+1}^c$ intersect in the radiance domain. The procedure for calculating the intersection point γ _n is shown below:

$$\begin{array}{ll}\frac{-1}{I_{max}-\rho }\left(f\left({\gamma }_n\Delta t_n\right)-\rho \right)+1\hfill & =\frac{1}{\rho }·f\left({\gamma }_n\Delta t_{n+1}\right)\hfill \\ \frac{1}{I_{max}-\rho }·f\left({\gamma }_n\Delta t_n\right)+\frac{1}{\rho }·f\left({\gamma }_n\Delta t_{n+1}\right)\hfill & =\frac{\rho }{I_{max}-\rho }+1\hfill \\ \frac{1}{I_{max}-\rho }·({\gamma }_n\Delta t_n+\beta )+\frac{1}{\rho }·({\gamma }_n\Delta t_{n+1}+\beta )\hfill & =\frac{\rho }{I_{max}-\rho }+1\hfill \\ {\gamma }_n\left(\frac{\Delta t_n}{I_{max}-\rho }+\frac{\Delta t_{n+1}}{\rho }\right)+\frac{\beta }{\rho }+\frac{\beta }{I_{max}-\rho }\hfill & =\frac{\rho }{I_{max}-\rho }+1\hfill \\ {\gamma }_n\hfill & =\frac{\frac{\rho -\beta }{I_{max}-\rho }-\frac{\beta }{\rho }+1}{\frac{\Delta t_n}{I_{max}-\rho }+\frac{\Delta t_{n+1}}{\rho }}\hfill \\ {\gamma }_n\hfill & =\frac{\rho (I_{max}-\beta )-\beta (I_{max}-\rho )}{\Delta t_n\rho +\Delta t_{n+1}(I_{max}-\rho )}.\hfill \end{array}$$

(30)