Analysis and processing of pixel binning for color image sensor

Abstract

Pixel binning refers to the concept of combining the electrical charges of neighboring pixels together to form a superpixel. The main benefit of this technique is that the combined charges would overcome the read noise at the sacrifice of spatial resolution. Binning in color image sensors results in superpixel Bayer pattern data, and subsequent demosaicking yields the final, lower resolution, less noisy image. It is common knowledge among the practitioners and camera manufacturers, however, that binning introduces severe artifacts. The in-depth analysis in this article proves that these artifacts are far worse than the ones stemming from loss of resolution or demosaicking, and therefore it cannot be eliminated simply by increasing the sensor resolution. By accurately characterizing the sensor data that has been binned, we propose a post-capture binning data processing solution that succeeds in suppressing noise and preserving image details. We verify experimentally that the proposed method outperforms the existing alternatives by a substantial margin.

1 Introduction

Recent progress on digital camera technology has had extraordinary impact on numerous electronic industries, including mobile phones, security, vehicle, bioengineering, and computer vision systems. In many applications, sensor resolution has exceeded the optical resolution, meaning that the additional hardware complexity to increase pixel density would not necessarily result in large image quality gains. The significant improvement in sensor sensitivity has allowed cameras to operate in lighting conditions that were unthinkable with film cameras.

Despite increased sensitivity, however, noise remains a serious problem in modern image sensors. Available technologies for reducing noise in hardware include backside illuminated architecture [1, 2], color filters with higher transmittance [3, 4], and pixel binning [5–7]. Processing techniques at our disposal include image denoising [8–10], joint denoising and demosaicking [11–14], image deblurring [15, 16] (long shutter to compensate for light), and single-shot high dynamic range imaging [17].

The goal of this article is to provide a comprehensive characterization of the pixel binning for color image sensors, and propose post-capture signal processing steps aimed at eliminating the binning artifacts. Binning refers to the concept of combining the electrical charges of neighboring pixels together to form a superpixel. The combined signal will then be amplified by a source follower and converted into digital values by an analog-to-digital converter. The main benefit of this technique is that the combined charges would overcome the read noise, even if the individual pixel values are small. The improved noise performance comes at the price of spatial resolution loss, however. Binning in color image sensors is complicated by the presence of color filter array (CFA). Data are typically obtained via a single CCD or CMOS sensor with a CFA spatial subsampling procedure, a physical construction whereby each pixel location measures only a single color. Figures 1a,b show the most well known CFA scheme called the Bayer pattern, which involves red, green, and blue filters. To maintain the fidelity of color, binning in color image sensors are performed by combining neighboring pixels with the same color filter. As evidenced by the two well known binning configurations shown in Figures 1a,b, the resultant superpixel form a Bayer pattern, as shown in Figure 1c. The subsequent demosaicking algorithm—the process of interpolating to recover the full RGB representation of the image from the CFA subsampled sensor data—yields the final, lower resolution, less noisy image.

Figure 1

Commonly used binning schemes. Binning refers to the concept of combining the electrical charges of neighboring pixels together to form a superpixel. (a–b) The numbers over the high resolution Bayer pattern indicate which pixels are combined together. (c) The resultant superpixel Bayer pattern, where the numbers indicate the relative locations of the combined pixels (for [7] and [5]).

However, it is a common knowledge among the practitioners and camera manufacturers that binning introduces pixelization artifacts. An example is shown in Figure 2. As will be made clear in the sequel, these artifacts differ from the ones stemming from loss of resolution, and therefore it cannot be eliminated simply by increasing the sensor resolution. In-depth analysis of the sampling scheme implied by the binning proves that gross mismatch between binning and demosaicking results is at fault for the severe pixelization. Hence the right way to correct this problem is to design a binning-aware demosaicking algorithm. The proposed method still draws from the established demosaicking principles, but with profound differences in the way spatially high frequency components are handled. To the best of the knowledge of the authors, this is the first major article to examine pixel binning problem in color image sensors from the signal processing perspective, and to provide post-capture processing solution to correct for the pixelization artifacts.

Figure 2

Binning vs. no binning. Compared to no binning, binning succeeds in reducing noise. However, the pixelization and zippering artifacts deteriorate the image quality. (a) Reconstruction from full resolution CFA; (b) reconstruction from Kodak PIXELUX scheme of Figure 1a; (c) reconstruction from PhaseOne scheme of Figure 1b.

The remainder of this article is organized as follows. We begin by briefly reviewing CFA sampling and demosaicking in Section 2. Section 3. provides a rigorous analysis of binning. A novel binning-aware demosaicking technique is developed in Section 4. We experimentally verify its effectiveness in 5. before making concluding remarks in Section 6.

2 Background

2.1 CFA sampling

Thanks to the seminal work of [18] and further investigations by [19–21], CFA sampling is well characterized and understood. The key insight is the two dimensional Fourier analysis of CFA sampled sensor data, which reveals that the signal is preserved by an efficient space-color representation. Specifically, let $\mathit{x}:{\mathbb{Z}}^2\to {\mathbb{R}}^3$, where x(n)=x _r (n),x _g (n),x _b (n)]^T correspond to the RGB tri-stimulus value at location $\mathit{n}\in {\mathbb{Z}}^2$. Then the CFA subsampled data has the following form:

$$\begin{array}{l}y\left(\mathit{n}\right)=\mathit{c}{\left(\mathit{n}\right)}^T\mathit{x}\left(\mathit{n}\right)\\ =\mathit{c}{\left(\mathit{n}\right)}^T\left[\begin{array}{c}110\\ 100\\ 101\end{array}\right]\left[\begin{array}{c}010\\ 1-10\\ 0-11\end{array}\right]\mathit{x}\left(\mathit{n}\right)\\ =\left[1c_{\alpha }\left(\mathit{n}\right)c_{\beta }\left(\mathit{n}\right)\right]\left[\begin{array}{c}{x}_g\left(\mathit{n}\right)\\ x_{\alpha }\left(\mathit{n}\right)\\ x_{\beta }\left(\mathit{n}\right)\end{array}\right],\end{array}$$

(1)

where $\mathit{c}:{\mathbb{Z}}^2\to {[0,1]}^3$ denotes the translucency of CFA at location n. The advantage to the representation is that the difference images x _α =x _r −x _g and x _β =x _b −x _g enjoy rapid spectral decay and can serve as a proxy for chrominance. On the other hand, the “baseband” green image x _g can be taken to approximate luminance. As our eventual image recovery task will be to approximate the true color image triple x(n) from acquired sensor data y(n), note that recovering either representations ({x _r x _g x _b }or {x _g x _α x _β }) are equivalent. Moreover, the representation of (1) allows us to re-cast the pure-color sampling structure in terms of sampling structures c _α and c _β associated with the difference channels x _α and x _β . For more extensive investigation on the bandlimitedness assumptions of {x _g x _α x _β }, see [18–20].

Denote by the uppercase letters the discrete space Fourier transforms and $\mathit{\omega }={({\omega }_1,{\omega }_2)}^T\in {\{\mathbb{R}/(2\Pi \left)\right\}}^2$ ($\mathbb{R}/\left(2\Pi \right)$ denotes the quotient group of $\mathbb{R}$ by the subgroup $2\mathrm{\Pi \mathbb{Z}}$) the two dimensional Fourier index. Then the Fourier analysis of CFA is:

$$\begin{array}{l}{C}_{\alpha }\left(\mathit{\omega }\right)=\sum _{\mathit{\lambda }\in \Pi {\mathbb{Z}}_2^2}\frac{\delta (\mathit{\omega }-\mathit{\lambda })}{4},\\ C_{\beta }\left(\mathit{\omega }\right)=\sum _{\mathit{\lambda }\in \Pi {\mathbb{Z}}_2^2}{e}^{j\left\{{\mathit{\lambda }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}\frac{\delta (\mathit{\omega }-\mathit{\lambda })}{4},\end{array}$$

where δ(·) is the Dirac delta function, and ${\mathbb{Z}}_2$ denotes the cyclic group of order 2. Note that the phase shift term in C _β arises due to the relative position of blue pixels relative to the red (the origin is assumed to be on a red pixel). The corresponding Fourier analysis of the sensor data y takes the following form:

$$\begin{array}{l}Y\left(\mathit{\omega }\right)=X_g\left(\mathit{\omega }\right)+C_{\alpha }\left(\mathit{\omega }\right)\star X_{\alpha }\left(\mathit{\omega }\right)+C_{\beta }\left(\mathit{\omega }\right)\star X_{\beta }\left(\mathit{\omega }\right)\\ =X_g\left(\mathit{\omega }\right)+\sum _{\mathit{\lambda }\in \Pi {\mathbb{Z}}_2^2}\frac{X_{\alpha }(\mathit{\omega }-\mathit{\lambda })+e^{j\left\{{\mathit{\lambda }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{\beta }(\mathit{\omega }-\mathit{\lambda })}{4},\hfill \end{array}$$

(2)

where ⋆denotes convolution. The Fourier support of the resultant sensor signal is shown in Figure 3.

Figure 3

Idealized spectral support of a color image acquired under the Bayer pattern. In each figure, the horizontal and vertical axes span [−Π,Π)²of Fourier index, and the DC is located at the center of the figure. Solid lines indicate the baseband signals, while replicated spectra with the dashed lines arises as a result of CFA sampling. Black and red lines correspond to the support of luminance and chrominance images, respectively. Alias-inducing chrominance replications are shown with (a) Radially symmetric luminance, (b) vertical feature luminance, (c) horizontal feature luminance.

2.2 Demosaicking

Most demosaicking algorithms described in the literature make use (either implicitly or explicitly) of correlation structure in the spatial frequency domain, often in the form of local sparsity or directional filtering [14, 19, 21–23]. As noted in our earlier discussion, the set of carrier frequencies induced by c _α and c _β include Π,0]^T and [0,Π^T, locations that are particularly susceptible to aliasing by horizontal and vertical edges. Figures 3b,c indicates these scenarios, respectively; it may be seen that in contrast to the radially symmetric baseband spectrum of Figure 3a, chrominance–luminance aliasing occurs along one of either the horizontal or vertical axes. However, successful reconstruction can still occur if a noncorrupted copy of this chrominance information is recovered, thereby explaining the popularity of (nonlinear) directional filtering steps [19, 21–23]. We can, therefore, view the CFA design problem as one of spatial-frequency multiplexing, and the CFA demosaicking problem as one of demultiplexing to recover subcarriers, with spectral aliasing given the interpretation of “cross talk” [19].

In order to carry out this demultiplexing, signal-adaptive demosaicking methods take the scenarios of Figure 3a–c into account. Typically, this is carried out by first filtering in both horizontal and vertical directions to yield reconstructions ${\widehat{\mathit{x}}}_h$ and ${\widehat{\mathit{x}}}_v$, respectively. Taking their convex combination to yield the final result:

$$\begin{array}{l}{\widehat{\mathit{x}}}_{\tau }\left(\mathit{n}\right)=\tau \left(\mathit{n}\right){\widehat{\mathit{x}}}_h\left(\mathit{n}\right)+(1-\tau (\mathit{n}\left)\right){\widehat{\mathit{x}}}_v\left(\mathit{n}\right),\end{array}$$

(3)

where τ∈[0,1] is a set of weights. Based on models of a “natural image” behavior, various policies for determining the appropriate weights have been developed [14, 19, 21–23]. For example, the weight combination should maximize the homogeneity $u_{\widehat{x}}\left(\mathit{n}\right)$—defined as a percentage of pixels in the neighborhood of n(denoted η(n)) that are similar to x(n)[22]:

$$\begin{array}{l}{u}_{\widehat{x}}\left(\mathit{n}\right)=\frac{\#\{\mathit{m}\in \eta (\mathit{n}):d(\widehat{\mathit{x}}\left(\mathit{n}\right),\widehat{\mathit{x}}\left(\mathit{m}\right))<\epsilon \}}{\#\left\{\eta \right(\mathit{n}\left)\right\}}\end{array}$$

(4)

where d(·,·) is some distance metric and εis a tolerance parameter.

3 Analysis of binning

Let us rigorously analyze the effects that binning has on the acquired sensor data. We begin in Section 3.1 with a brief review of the signal-to-noise ratio (SNR) gains that binning is expected to improve [24]—the main motivation behind binning. An in-depth analysis in Section 3.2 will prove that a combination of binning and demosaicking results in a loss of resolution that is far worse than commonly believed. Section 3.3 offers an alternative perspective that paves a path towards recovering artifact-free images.

3.1 Signal measurement uncertainty

There are at least three types of noise that contribute to the overall error. “Shot noise” is due to the stochasticity of the photon arrival process, and it is well modeled by Poisson distribution. The dark current stemming from in-circuit electron excitation results in “thermal noise,” whose power is proportional to the exposure time. Finally, the source follower and analog-to-digital converter introduce the homoscedastic noise that is known as the “read noise.” The overall SNR of captured image is well modeled by:

$${\text{SNR}}_{\text{pix}}:=20\underset{10}{log}(Q·t·y)-10\underset{10}{log}(Q·t·y+D·t+N),$$

(5)

where t is the exposure time, Q is the quantum efficiency constant, D is the dark current constant, and N is the read noise power.

Owing to the fact that the image sensor resolution exceeds the optical resolution in many applications, binning is an attractive way to trade off the excess spatial resolution for gains in SNR. It is instructive first to consider summing M pixel values digitally, post-acquisition. The signal y is boosted M-fold while the noise power increases M times, resulting in an overall 10 log₁₀ (M)dB gain:

$$\begin{array}{c}{\text{SNR}}_{\text{sum}}:=20{log}_{10}(M·Q·t·y)\hfill \\ -10{log}_{10}(M·Q·t·y+M·D·t+M·N)\hfill \\ ={\text{SNR}}_{\text{pix}}+10{log}_{10}\left(M\right)\ge {\text{SNR}}_{\text{pix}}.\hfill \end{array}$$

(6)

Combining electrical charges of neighboring pixels to form a superpixel in hardware offers advantages over simply summing pixels digitally. The main difference is that when the electrical charges are combined before source follower and analog-to-digital converter, the uncertainty due to read noise remains constant. The corresponding SNR is:

$$\begin{array}{l}{\text{SNR}}_{\text{bin}}=20{log}_{10}(M·Q·t·y)\\ -10{log}_{10}(M·Q·t·y+M·D·t+N)\hfill \\ \ge {\text{SNR}}_{\text{sum}}.\hfill \end{array}$$

(7)

As illustrated by the example in Figure 4, the differences between SNR_bin and SNR_sum are more noticeable when the signal intensity y becomes small and read noise N become dominant—meaning that binning is most effective in the low light ranges.

Figure 4

SNR as a function of signal intensity. Here, M=4, Q=0.70, t=1/100s, D=0.1electrons/pixel/second, and N=10electrons rms/pixel [24]. See (5-7).

3.2 Binning “sampling”

Due to the fact that binning combines M electric charges of neighboring pixels, each pixel cannot be shared by more than one superpixel. Moreover, the charges can be combined by summation only (i.e. no fractional combinations). As such, the options for binning schemes are fairly limited. Furthermore, the superpixels produced by pixel binning in color image sensors form a Bayer pattern that requires the additional step of demosaicking to recover the full color low resolution image. We will show that superpixel Bayer pattern suffers from many problems that the pixel-level Bayer pattern does not, leading to the conclusion that combining pixel binning and demosaicking is the wrong approach.

Consider Kodak PIXELUX, the most widely used binning scheme illustrated in Figure 1a,c [7]. It combines four neighboring pixel values together to form one superpixel. This process of combining neighboring pixels to form a single superpixel is equivalent to applying a convolution operator followed by downsampling:

• filtering: let h_bindenote the filter coefficients

  $$\begin{array}{l}{h}_{\text{bin}}\left(\mathit{n}\right)=\Delta \left(\mathit{n}-\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)+\Delta \left(\mathit{n}-\left(\genfrac{}{}{0ex}{}{1}{-1}\right)\right)\\ +\Delta \left(\mathit{n}-\left(\genfrac{}{}{0ex}{}{-1}{1}\right)\right)+\Delta \left(\mathit{n}-\left(\genfrac{}{}{0ex}{}{-1}{-1}\right)\right),\hfill \end{array}$$

  (8)

  where Δ(·)denotes the Kronecker delta function. Then the charge summation in PIXELUX is

  $$\begin{array}{l}{y}_{\text{bin}}\left(\mathit{n}\right)=y\left(\mathit{n}\right)\star h_{\text{bin}}\left(\mathit{n}\right).\end{array}$$

• downsampling: to yield the superpixel Bayer pattern data s, do

  $$\begin{array}{c}s\left(2\mathit{n}\right)=y_{\text{bin}}\left(4\mathit{n}\right)\\ s\left(2\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)=y_{\text{bin}}\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)\\ s\left(2\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)=y_{\text{bin}}\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)\\ s\left(2\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)=y_{\text{bin}}\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right).\end{array}$$

  (9)

Note that downsampling implied by (9) is non-uniform—the spatial relationships between samples are changed by the different relative shifts applied to each super pixels (contrast this to (11) below). The Fourier transform of s is (derived in Appendix Appendix 2: Proof of Fourier representation of binning sampling):

$$\begin{array}{c}S\left(\mathit{\omega }\right)\approx \sum _{\mathit{\lambda }\in \Pi {\mathbb{Z}}_2^2}\left(X_{\alpha }\left(\frac{\mathit{\omega }-\mathit{\lambda }}{2}\right)\right.\hfill \\ \left.+\underset{\text{unwanted filter}}{\underbrace{e^{j{\left(\frac{\mathit{\omega }}{2}\right)}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)}}}{e}^{j{\left(\frac{\mathit{\lambda }}{2}\right)}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)}{X}_{\beta }\left(\frac{\mathit{\omega }-\mathit{\lambda }}{2}\right)\right)\hfill \\ +\underset{\text{unwanted filter}}{\underbrace{\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j{\left(\frac{\mathit{\omega }}{2}\right)}^T\theta }{H}_{\text{bin}}\left(\frac{\mathit{\omega }}{2}\right)}{16}}}{X}_g\left(\frac{\mathit{\omega }}{2}\right)\hfill \\ +\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2\setminus \left(\genfrac{}{}{0ex}{}{0}{0}\right)}\underset{\text{antialias filter}}{\underbrace{\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j{\left(\frac{\mathit{\omega }}{2}+\mathit{\lambda }\right)}^T\theta }{H}_{\text{bin}}\left(\frac{\mathit{\omega }}{2}-\mathit{\lambda }\right)}{16}}}\underset{\text{aliasing}}{\underbrace{X_g\left(\frac{\mathit{\omega }}{2}-\mathit{\lambda }\right)}}.\hfill \end{array}$$

(10)

The corresponding Fourier support of S(ω) is shown in Figure 5. Note that the unwanted filter will boost X _g to 16 at the DC. The approximate relation above is admitted by the bandlimitedness assumptions of X _α and X _β :

$$\begin{array}{l}{H}_{\text{bin}}\left(\mathit{\omega }\right)X_{\alpha }(\mathit{\omega }-\mathit{\lambda })\approx 4X_{\alpha }(\mathit{\omega }-\mathit{\lambda })\\ H_{\text{bin}}\left(\mathit{\omega }\right)X_{\beta }(\mathit{\omega }-\mathit{\lambda })\approx 4X_{\beta }(\mathit{\omega }-\mathit{\lambda }).\end{array}$$

Figure 5

Idealized spectral support of binning sampled data S ( ω )in (10), corresponding to Figure 1. As before, solid lines indicate the baseband signals, while spectra with the dashed lines arises as a result of CFA sampling. Black and red lines correspond to the support of luminance and chrominance images, respectively. The blue box represents the original sampling rate.

The main advantage of binning in (9) over (2) is that the signal strength of the baseband X _g and the chrominance components X _α and X _β are boosted by four times—consistent with the SNR analysis in the previous section. As evidenced by Figure 5a, the Fourier support of (9) closely resembles the Bayer pattern of Figure 3a. Superpixel Bayer pattern data in (10) is far from an ideal Bayer pattern representation of the true image x(n) we hope to recover from s(n), however. One distortion we see is the unwanted filtering term $\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{e}^{j{\mathit{\omega }}^T\mathit{\theta }/2}$ that degrades the baseband luminance/green signal X _g (ω). Another complication is that the antialiasing is only partially effective, allowing aliasing to corrupt the baseband X _g (ω) near $\mathit{\omega }=\pm {[0,\frac{\Pi }{4}]}^T,\pm {[\frac{\Pi }{4},0]}^T,\pm {[\frac{\Pi }{4},\frac{\Pi }{4}]}^T,\pm {[\frac{\Pi }{4},\frac{-\Pi }{4}]}^T$.

Contrary to the popular belief that Kodak PIXELUX binning results in 2×2reduction in resolution, the main conclusion we draw from (9) is that the “Nyquist rate” of this binning scheme is Π/4due to high risk of aliasing—implying that the actual resolution loss is 4×4, far worse than the presumed 2×2. Even if this Nyquist rate did not cause problems (e.g. increase sensor resolution), s does not escape the unwanted filtering term in (9)—this cannot be eliminated simply by increasing sensor resolution. Hence when a demosaicking algorithm is applied to the superpixel Bayer pattern data s, what is expected is a filtered and aliased image that we have already seen in Figure 2.

3.2 Binning “subsampling”

Below, we offer an alternative perspective to the analysis of Section 3.2. The analytical results contained herein will provide the basis for the proposed binning-aware demosaicking algorithm. Continuing with the analysis of PIXELUX, consider Figure 6a which displays data equivalent to the superpixels of Figure 1c. The superpixels are placed at the center of the four averaged pixels, denoting the implied superpixel positions. Other locations are given 0 value. This data can be represented by applying a convolution operator followed by subsampling, as follows:

Figure 6

Binning subsampling is an alternative interpretation to the binning sampling in Figure 1. (a) Subsampled data t(n)in (11) equivalent to the superpixel Bayer pattern of Figure 1c. (b) Idealized spectral support of binning subsampled data T(ω)in (12). The baseband signal X _g is free of aliasing in the shaded region. As before, solid lines indicate the baseband signals, while spectra with the dashed lines arises as a result of CFA sampling. Black and red lines correspond to the support of luminance and chrominance images, respectively.

• filtering: The charge summation in PIXELUX is

  $$\begin{array}{l}{y}_{\text{bin}}\left(\mathit{n}\right)=y\left(\mathit{n}\right)\star h_{\text{bin}}\left(\mathit{n}\right).\end{array}$$

• subsampling: to yield the binning subsampling data t, do

  $$\begin{array}{c}t\left(4\mathit{n}\right)=y_{\text{bin}}\left(4\mathit{n}\right)\\ t\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)=y_{\text{bin}}\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)\\ t\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)=y_{\text{bin}}\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)\\ t\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)=y_{\text{bin}}\left(4\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)\\ t\left(\mathit{n}\right)=0\text{otherwise.}\end{array}$$

  (11)

With arithmetic, the Fourier transform of t is deduced to:

$$\begin{array}{c}T\left(\mathit{\omega }\right)\approx \sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\frac{X_{\alpha }(\mathit{\omega }-\mathit{\lambda })+e^{j\left\{{\mathit{\lambda }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{\beta }(\mathit{\omega }-\mathit{\lambda })}{4}\\ +\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}{H}_{\text{bin}}(\mathit{\omega }-\mathit{\lambda })X_g(\mathit{\omega }-\mathit{\lambda })}{16}.\end{array}$$

(12)

Note that the summation over λ suggests 16 modulations. However, except $\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_3^2$, other λ results in $\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{e}^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}$ is 0, as shown in Figure 7. The support of this transform is illustrated in Figure 6b.^a As evidenced by this figure, the modulated baseband signal components X _g (ω−λ)overlap each other almost entirely—that is, they are aliased. However, the shaded regions of Figure 6b are still free of aliasing. Indeed, this uncorrupted portion of the Fourier support is the key to post-binning processing that is the subject of next section.

Figure 7

Fourier transform of$\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{e}^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}{M}_{\text{bin}}\left(\mathit{\omega }\right)$.

4 Binning-aware demosaicking

Motivated by the analysis of pixel binning subsampling in (12), we now present a novel binning-aware demosaicking aimed at recovering full-color image xwithout introducing binning artifacts. We accomplish this in three stages.

Step 1: Chrominance estimation

Drawing parallels to [19], we assume that local image features are either vertically or horizontally oriented (approximately). If this assumption holds, certain subsets of the modulated chrominances in (11) are assumed to be alias-free conditional under the vertically or horizontally oriented image features—this is illustrated in Figures 8a. For example, assuming horizontal feature, an amplitude demodulation images x _α and x _β :

$$\begin{array}{c}\left[\begin{array}{c}{\widehat{x}}_{\alpha ,h}\\ {\widehat{x}}_{\beta ,h}\end{array}\right]=\underset{\text{from (12)}}{\underbrace{{\left[\begin{array}{c}11\\ 1-1\\ 1-j\\ 1j\end{array}\right]}^{†}}}\underset{\text{demodulation}}{\underbrace{\left[\begin{array}{c}{h}_0\left(\mathit{n}\right)\star \left(t\left(\mathit{n}\right)·e^{j\left\{{\mathit{n}}^T\left(\genfrac{}{}{0ex}{}{\Pi }{\Pi }\right)\right\}}\right)\\ h_0\left(\mathit{n}\right)\star \left(t\left(\mathit{n}\right)·e^{j\left\{{\mathit{n}}^T\left(\genfrac{}{}{0ex}{}{0}{\Pi }\right)\right\}}\right)\\ h_0\left(\mathit{n}\right)\star \left(t\left(\mathit{n}\right)·e^{j\left\{{\mathit{n}}^T\left(\genfrac{}{}{0ex}{}{-\Pi /2}{\Pi }\right)\right\}}\right)\\ h_0\left(\mathit{n}\right)\star \left(t\left(\mathit{n}\right)·e^{j\left\{{\mathit{n}}^T\left(\genfrac{}{}{0ex}{}{\Pi /2}{\Pi }\right)\right\}}\right)\end{array}\right],}}\end{array}$$

(13)

Figure 8

Idealized spectral support of binning subsampled data, at various stages of binning-aware demosaicking. Shaded regions denote filter support. See text.

where (·)^† denotes a pseudo inverse matrix and h₀ is a lowpass filter whose passbands matches the support of X _α and X _β . The reconstruction of vertically oriented image feature (denoted ${\widehat{x}}_{\alpha ,v},{\widehat{x}}_{\beta ,v}$) is same as (13) but with 90° rotation.

Step 2: Luminance filtering

Once the ${\widehat{x}}_{\alpha ,h}$ and ${\widehat{x}}_{\beta ,h}$ are recovered, we compute the green image ${\widehat{x}}_{g,h}$. Subtracting

$$\sum _{\mathit{m}\in 4{\mathbb{Z}}^2}{\widehat{x}}_{\alpha ,h}\left(\mathit{m}\right)\Delta (\mathit{m}-\mathit{n})+{\widehat{x}}_{\beta ,h}(\mathit{m}+\left(\genfrac{}{}{0ex}{}{1}{1}\right))\Delta (\mathit{m}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)-\mathit{n})$$

from subsampled binning data t(n) results a Fourier transform that is comprised only of x _g : (from (12))

$$\begin{array}{l}{T}_g\left(\mathit{\omega }\right)=\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}{H}_{\text{bin}}(\mathit{\omega }-\mathit{\lambda })X_g(\mathit{\omega }-\mathit{\lambda })}{16}.\end{array}$$

(14)

This is illustrated in Figure 8b. To reconstruct the green image ${\widehat{x}}_{g,h}$ from the unaliased (shaded in Figure 8b) portions of t _g , we carry out a standard demodulation, as follows:

$$\begin{array}{l}{\widehat{x}}_{g,h}=\underset{\text{isolate signal}}{\underbrace{h_2\left(\mathit{n}\right)\star \left\{\stackrel{\text{modulation}}{\overbrace{f\left(\mathit{n}\right)·\left\{\underset{\text{isolate unaliased}}{\underbrace{h_1\left(\mathit{n}\right)\star t_g\left(\mathit{n}\right)}}\right\}}}\right\},}}\end{array}$$

where h₁and h₂ are lowpass and highpass filters, respectively; and f is a sum of sinusoids intended for modulation, as follows:

$$\begin{array}{l}{H}_1\left(\mathit{\omega }\right)=\left\{\begin{array}{ll}1\hfill & \text{if}\left|{\omega }_1\right|>\frac{\Pi }{2}\text{and}\left|{\omega }_2\right|>\frac{\Pi }{2}\\ 0& \text{else}\end{array}\right.\hfill \\ H_2\left(\mathit{\omega }\right)=\left\{\begin{array}{ll}1\hfill & \text{if}\left|{\omega }_1\right|<\frac{\Pi }{2}\text{and}\left|{\omega }_2\right|<\frac{\Pi }{2}\\ 0\hfill & \text{else}\hfill \end{array}\right.\hfill \\ F\left(\mathit{\omega }\right)=\sum _{\mathit{\lambda }\in \left\{\left(\genfrac{}{}{0ex}{}{\pm \Pi }{\pm \Pi }\right)/4\right\}}\frac{\delta \left(\mathit{\omega }+\mathit{\lambda }\right)}{\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{e}^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}}.\hfill \end{array}$$

(15)

As illustrated in Figure 8c,d, the modulation by f(n)not only shifts the spectrums, but also creates additional aliasing copies. Hence, the filter h₂is needed to attenuate them. The same procedure can be used to find the green image ${\widehat{x}}_{g,v}$ based on ${\widehat{x}}_{\alpha ,v}$ and ${\widehat{x}}_{\beta ,v}$.

Step 3: Directional selection

Once ${\widehat{\mathit{x}}}_h=\{{\widehat{x}}_{g,h},{\widehat{x}}_{\alpha ,h},{\widehat{x}}_{\beta ,h}\}$ and ${\widehat{\mathit{x}}}_v=\{{\widehat{x}}_{g,v},{\widehat{x}}_{\alpha ,v},{\widehat{x}}_{\beta ,v}\}$ are found, they must be combined to yield the final estimate, ${\widehat{\mathit{x}}}_t=\{{\widehat{x}}_g,{\widehat{x}}_{\alpha },{\widehat{x}}_{\beta }\}$ via the convex combination (3). As already mentioned, the directional selection variable τ has received considerable attention in research and many techniques are available. However, these studies often lack analysis under noise—although binning reduces noise considerably, most directional selection variables are nevertheless sensitive to random perturbations.

To address the problem of directional selection under noise, we modified the τcriteria used in the popular adaptive homogeneity directed (AHD) demosaicking method as follows:

$$\begin{array}{l}\widehat{\tau }\left(\mathit{n}\right)=arg\underset{\tau \in [0,1]}{max}{u}_{{\widehat{x}}_{\tau }}\left(\mathit{n}\right)\end{array}$$

(16)

$$\begin{array}{l}{\widehat{\mathit{x}}}_t\left(\mathit{n}\right)={\widehat{\mathit{x}}}_{\widehat{\tau }\left(\mathit{n}\right)}\left(\mathit{n}\right),\end{array}$$

(17)

where ${\widehat{\mathit{x}}}_{\tau }$ and $u_{\widehat{x}}$ are as defined in (3) and (4), respectively. Contrast this to the original AHD formulation which selected either ${\widehat{\mathit{x}}}_h$ or ${\widehat{\mathit{x}}}_v$ (i.e. τ∈{0,1} instead of τ∈[0,1]) as the final output ${\widehat{\mathit{x}}}_t$. The modified strategy of (16) behaves similarly to the original AHD near the edges of an image, but encourages averaging in the flat regions of the image. It was found empirically to be far more robust to directional selection under noise.

5 Experimental validation

5.1 Setup

The proposed binning-aware demosaicking ${\widehat{\mathit{x}}}_t\left(\mathit{n}\right)$ in (16) is compared to four available alternatives (${\widehat{\mathit{x}}}_s\left(\mathit{n}\right)$, ${\widehat{\mathit{x}}}_p\left(\mathit{n}\right)$, ${\widehat{\mathit{x}}}_y\left(\mathit{n}\right)$, and ${\widehat{\mathit{x}}}_{y^{\prime }}\left(\mathit{n}\right)$). The first is a state-of-the-art demosaicking method [19] applied to superpixels s(n)(i.e. output from PIXELUX binning):

$$\begin{array}{l}{\widehat{\mathit{x}}}_s\left(\mathit{n}\right)=demosaicking\left(s\right(\mathit{n}\left)\right).\end{array}$$

The second is the same demosaicking method [19] applied to PhaseOne binning superpixels p(n):

$$\begin{array}{l}{\widehat{\mathit{x}}}_p\left(\mathit{n}\right)=demosaicking\left(p\right(\mathit{n}\left)\right).\end{array}$$

The third is the application of the same demosaicking method [19] to a full resolution CFA y(n)(i.e. without binning):

$$\begin{array}{l}{\widehat{\mathit{x}}}_y\left(\mathit{n}\right)=demosaicking\left(y\right(\mathit{n}\left)\right).\end{array}$$

The fourth is a simulation of a lower resolution sensor. Let x^′(n) denote the downsampled version of the ideal lowpassed (antialiased) image:

$$\begin{array}{l}{\mathit{x}}^{\prime }\left(\mathit{n}\right)=\{h_2\star \mathit{x}\}\left(2\mathit{n}\right).\end{array}$$

(18)

The CFA subsampled data captured by this lower resolution sensor is then

$$\begin{array}{l}{y}^{\prime }\left(\mathit{n}\right)=\mathit{c}{\left(\mathit{n}\right)}^T{\mathit{x}}^{\prime }\left(\mathit{n}\right),\end{array}$$

where $\mathit{c}:{\mathbb{Z}}^2\to {[0,1]}^3$ is same the translucency of CFA used in (1). The application of the same demosaicking method [19] to lower resolution CFA y^′(n) is:

$$\begin{array}{l}{\widehat{\mathit{x}}}_{y^{\prime }}\left(\mathit{n}\right)=demosaicking\left(y^{\prime }\right(\mathit{n}\left)\right).\end{array}$$

The output images from the proposed method (${\widehat{\mathit{x}}}_t$) and the full resolution demosaicking (${\widehat{\mathit{x}}}_y$) have the same size as the original image x. On the other hand, the conventional binning processing are based on superpixel sampling, so the pixel density of ${\widehat{\mathit{x}}}_s$ and ${\widehat{\mathit{x}}}_p$ is just a quarter of the original image (same is true also for ${\widehat{\mathit{x}}}_{y^{\prime }}$). Hence when we compare all results (Figures 9, 10, 11, 12, 13; Table 1), we downsample ${\widehat{\mathit{x}}}_t$ and ${\widehat{\mathit{x}}}_{y^{\prime }}$ by 2×2 (in the same manner as (18)) such that all results have the same pixel density as the lower resolution image x^′.

Figure 9

Example of images used in experiment (zoomed).

Figure 10

Reconstructed images with various noise levels. Demosaicking method used for comparison is that of [19]. Here, LR ≜low resolution, DS ≜downsample, PO ≜PhaseOne [5], K ≜Kodak PIXELUX [7].

Figure 11

Reconstructed images with various noise levels. Demosaicking method used for comparison is that of [19]. Here, LR ≜low resolution, DS ≜downsample, PO ≜PhaseOne [5], K ≜Kodak PIXELUX [7].

Figure 12

Reconstructed images with various noise levels. Demosaicking method used for comparison is that of [19]. Here, LR ≜low resolution, DS ≜downsample, PO ≜PhaseOne [5], K ≜Kodak PIXELUX [7].

Figure 13

PSNR (red/green/blue pixels are combined) of each methods, Kodak and PhaseOne refers to the binning methods of[7] and [5], respectively. Demosaicking method used for comparison is that of [19].

Table 1 Reconstruction performance in PSNR with various noise levels

The linear images used in this simulation study are a part of the collection of [25, 26], examples of which are shown in Figure 9. Numerical scores in Table 1 and Figure 13 were obtained by averaging performance over 84 images. Noise is simulated by adding pseudorandom white Gaussian noise to the CFA data y(n), the superpixel CFA data s(n) and p(n), and the lower resolution CFA data y^′(n). In the experiments, the 12 bit image data in [25, 26] were renormalized to ranges 0–1—meaning noise standard deviation σ _n =0.01 correspond to standard deviation of 40.96 in a 12 bit camera processing pipeline, etc. Considering the noise models in (57), one may ask if such a simplified noise model is appropriate. As evidenced by the analysis in (7), however, the difference between SNR and SNR_sumis M (the number of pixels combined together); and the difference between SNR_sum and SNR_bin is the read noise power N. Hence the SNR gains in binning is attributed only to the signal-independent portion of the noise, and not on the signal dependent portion. Furthermore, the read noise dominates in the low light regime. Hence simulated additive white Gaussian noise suffices for experimental verification. The binning subsample signal t(n) represents the same data as s(n) and is computed by upsampling s(n)(insert zeros where necessary).

5.2 Results

Example outputs from four different methods (${\widehat{\mathit{x}}}_{y^{\prime }},{\widehat{\mathit{x}}}_y,{\widehat{\mathit{x}}}_s,{\widehat{\mathit{x}}}_p,{\widehat{\mathit{x}}}_t$) are shown in Figures 10, 11 and 12. As expected, demosaicking applied to a full resolution CFA (${\widehat{\mathit{x}}}_y$) has a noisy appearance due to low SNR of individual pixels. However, edges and image features are clearly defined even after downsampling thanks to the full resolution description. Demosaicking applied to superpixel CFAs (${\widehat{\mathit{x}}}_s,{\widehat{\mathit{x}}}_p$), on the other hand, yields the opposite qualities—the noise is significantly reduced owing to high SNR of binning, but the image suffers from severe artifacts stemming from aliasing in (10). More specifically, the aliasing in Kodak PIXELUX binning manifests itself as a pixelization artifact, while PhaseOne binning results in zippering artifacts. However, one may argue that the aliasing artifacts in ${\widehat{\mathit{x}}}_p$ become less bothersome at the highest level of noise because the zippering and noise become less distinguishable. By contrast, the proposed binning-aware demosaicking method (${\widehat{\mathit{x}}}_t$) succeeds in suppressing noise while preserving the image features. Of particular interest is the comparison between ${\widehat{\mathit{x}}}_s$ and ${\widehat{\mathit{x}}}_t$, since they both use Kodak PIXELUX binning but the proposed method yield drastically improved outcomes. Overall, the proposed method has better visual quality than ${\widehat{\mathit{x}}}_s$ and ${\widehat{\mathit{x}}}_p$ for σ _n <0.03; but proposed has a slightly noisier appearance at the highest level of noise (σ _n =0.03). Finally, the output from the low resolution camera ${\widehat{\mathit{x}}}_{y^{\prime }}$ is both robust to noise and aliasing. This is expected, as lower resolution CFA data y^′(n) does not share the problems that superpixel CFAs p(n),s(n),t(n) have. However, ${\widehat{\mathit{x}}}_y$ has superior reconstruction over ${\widehat{\mathit{x}}}_{y^{\prime }}$ without noise (σ _n =0). Figure 12 shows an example where none of the reconstruction methods produced a satisfactory output (except for ${\widehat{\mathit{x}}}_y$ under no noise).

The performance is evaluated also in terms of peak SNR, using the downsampled version of the ideal lowpassed (antialiased) image x^′in (18) as their reference. The results are summarized in Table 1. When there is no noise (σ _n =0), ordinary demosaicking reconstruction ${\widehat{\mathit{x}}}_y$ and lower resolution sensor ${\widehat{\mathit{x}}}_{y^{\prime }}$ yields the best results, as expected. However, the proposed ${\widehat{\mathit{x}}}_t$ is a very close third, yielding comparably satisfactory results. Binning result ${\widehat{\mathit{x}}}_s$ is worst by far due to binning artifacts.

When noise is taken into consideration, the quality of ${\widehat{\mathit{x}}}_y$ suffers greatly as expected. Even with noise variance as little as σ _n =0.005, the performance of ${\widehat{\mathit{x}}}_y$ deteriorates significantly, while performance of ${\widehat{\mathit{x}}}_s$, ${\widehat{\mathit{x}}}_p$, ${\widehat{\mathit{x}}}_t$, and ${\widehat{\mathit{x}}}_{y^{\prime }}$ in terms of PSNR are far less sensitive to noise. With moderate noise levels (σ _n <0.03) the proposed binning-aware demosaicking clearly outperforms the artifact-plagued demosaicking of superpixels. With the largest noise level considered (σ _n =0.03), PSNR performances of ${\widehat{\mathit{x}}}_s$, ${\widehat{\mathit{x}}}_p$, and ${\widehat{\mathit{x}}}_t$ are closer to each other because deteriorations in output images are dominated by noise (rather than by artifacts).

The analysis in Figures 10, 11, 12, 13 and Table 1 sheds a light on the decades-old debate about resolution versus noise. On one hand, the lower resolution sensor delivers consistent performance under noise (${\widehat{\mathit{x}}}_{y^{\prime }}$). However, Figure 11 shows that under no noise, extra sensor resolution is still desirable. Consider Figure 13. The comparison between green (low resolution) and red (high resolution) curves is consitent with the image quality of Figures 10 and 11. With the availability of pixel binning, we would compare the green curve with the “max function” over the red and blue (binning) curves in Figure 13. Hence one can think of binning as a way to narrow the gap between the red and green curves in noise, without making sacrifices to the advantages of higher spatial resolution.

6 Conclusion

In this article, we proved via a rigorous analysis of binning sampling that Kodak PIXELUX binning scheme results in 4×4reduction in image resolution—contrary to the popular belief that binning of four pixels should result in 2×2reduction in resolution. We proposed a binning-aware demosaicking algorithm based on the Fourier analysis of binning subsampling to combine unaliased copies of the Fourier spectra together via the demodulation. The resultant method succeeds in reconstructing the color image with only 2×2 resolution loss—or increasing the resolution by 2×2over the traditional approach of applying demosaicking to superpixels. The binning-aware demosaicking also succeeds in suppressing noise and preserving image details. We verified experimentally that the binning-aware demosaicking outperforms the alternatives.

Appendix 1: Proof of Fourier Representation of binning subsampling

We provide the proof for Equation (12). Let H_binbe the Fourier transform of (8). Then the combination of charges can be represented as:

$$\begin{array}{l}{X}_{r_{\text{bin}}}\left(\mathit{\omega }\right)\triangleq X_r\left(\mathit{\omega }\right)H_{\text{bin}}\left(\mathit{\omega }\right)\\ X_{g_{\text{bin}}}\left(\mathit{\omega }\right)\triangleq X_g\left(\mathit{\omega }\right)H_{\text{bin}}\left(\mathit{\omega }\right)\\ X_{b_{\text{bin}}}\left(\mathit{\omega }\right)\triangleq X_b\left(\mathit{\omega }\right)H_{\text{bin}}\left(\mathit{\omega }\right).\end{array}$$

Due to band limitedness of X _α and X _β , the following approximation hold:

$$\begin{array}{cc}{X}_{{\alpha }_{\text{bin}}}\left(\mathit{\omega }\right)\triangleq X_{r_{\text{bin}}}\left(\mathit{\omega }\right)-X_{g_{\text{bin}}}\left(\mathit{\omega }\right)& =H_{\text{bin}}\left(\mathit{\omega }\right)X_{\alpha }\left(\mathit{\omega }\right)\approx 4X_{\alpha }\left(\mathit{\omega }\right)\\ X_{{\beta }_{\text{bin}}}\left(\mathit{\omega }\right)\triangleq X_{b_{\text{bin}}}\left(\mathit{\omega }\right)-X_{g_{\text{bin}}}\left(\mathit{\omega }\right)& =H_{\text{bin}}\left(\mathit{\omega }\right)X_{\beta }\left(\mathit{\omega }\right)\approx 4X_{\beta }\left(\mathit{\omega }\right).\end{array}$$

(19)

where we used the fact that H_bin(0)=4.

Define $m_{\text{bin}}\left(\mathit{n}\right)=\sum _{\mathit{\theta }\in {\mathbb{Z}}_4^2}\Delta (\mathit{n}-\mathit{\theta })$, as illustrated in Figure 14. The binning subsampling data t(n)refers to the concept of combining the electrial charges of four neighboring pixels together to form a superpixel. The process is illustrated in Figure 6a. Mathmatically, t(n) can be written as:

$$\begin{array}{l}t\left(\mathit{n}\right)=m_{\text{bin}}\left(\mathit{n}\right)x_{r_{\text{bin}}}\left(\mathit{n}\right)+m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}\right)\hfill \\ +m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}\right)+m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)x_{b_{\text{bin}}}\left(\mathit{n}\right)\hfill \\ =m_{\text{bin}}\left(\mathit{n}\right)x_{{\alpha }_{\text{bin}}}\left(\mathit{n}\right)+m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)x_{{\beta }_{\text{bin}}}\left(\mathit{n}\right)\hfill \\ +\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{m}_{\text{bin}}(\mathit{n}+\theta )x_{g_{\text{bin}}}\left(\mathit{n}\right).\hfill \end{array}$$

Figure 14

Binning sampling filter m _bin ( n ).

In the Fourier domain, t(n)can be expressed as

$$\begin{array}{l}T\left(\mathit{\omega }\right)=M_{\text{bin}}\left(\mathit{\omega }\right)\star X_{{\alpha }_{\text{bin}}}\left(\mathit{\omega }\right)+\left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{M}_{\text{bin}}\right(\mathit{\omega }\left)\right)\star X_{{\beta }_{\text{bin}}}\left(\mathit{\omega }\right)\\ +\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{e}^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}{M}_{\text{bin}}\left(\mathit{\omega }\right)\star X_{g_{\text{bin}}}\left(\mathit{\omega }\right)\end{array}$$

where

$$\begin{array}{l}{M}_{\text{bin}}\left(\mathit{\omega }\right)=\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\frac{\delta (\mathit{\omega }-\mathit{\lambda })}{16}.\end{array}$$

With arithmetic and approximation of (19), the Fourier transform of t(n)simplifies to:

$$\begin{array}{l}T\left(\mathit{\omega }\right)\approx \sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\frac{X_{\alpha }(\mathit{\omega }-\mathit{\lambda })+e^{j\left\{{\mathit{\lambda }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{\beta }(\mathit{\omega }-\mathit{\lambda })}{4}\\ +\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}{H}_{\text{bin}}(\mathit{\omega }-\mathit{\lambda })X_g(\mathit{\omega }-\mathit{\lambda })}{16}.\end{array}$$

The Fourier support of T(ω) is illustrated in Figure 6b. Note that the summation over λsuggests that binning subsampling will result in 16 modulations. However, $\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}{e}^{j\left\{{\mathit{\lambda }}^T\mathit{\theta }\right\}}$ is 0 for many values of λ, as shown in Figure 7. As a result, there are only nine actual modulations.

Appendix 2: Proof of Fourier representation of binning sampling

We provide the proof for Equation (10). The binning sampling data s(n)refers to the concept of combining the electrial charges of four neighboring pixels together to form a superpixel Bayer pattern. The process is illustrated in Figures 1a,c. Similar to binning subsampling (see Appendix Appendix 1: Proof of Fourier Representation of binning subsampling, binning sampling s(n)has the following representation (it is mathmatically convenint to consider $s\left(\frac{\mathit{n}}{2}\right)$ for n even, rather than s(n)directly);

$$\begin{array}{l}s\left(\frac{\mathit{n}}{2}\right)=m_{\text{bin}}\left(\mathit{n}\right)x_{r_{\text{bin}}}\left(\mathit{n}\right)+m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{2}{0}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)\hfill \\ +m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{2}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)\hfill \\ +m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{2}{2}\right)\right)x_{b_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)\hfill \\ =m_{\text{bin}}\left(\mathit{n}\right)x_{{\alpha }_{\text{bin}}}\left(\mathit{n}\right)+m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{2}{2}\right)\right)x_{{\beta }_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right)\hfill \\ +m_{\text{bin}}\left(\mathit{n}\right)x_{g_{\text{bin}}}\left(\mathit{n}\right)+m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{2}{0}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right)\hfill \\ +m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{2}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right)\hfill \\ +m_{\text{bin}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{2}{2}\right)\right)x_{g_{\text{bin}}}\left(\mathit{n}+\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right).\hfill \end{array}$$

In Fourier domain,

$$\begin{array}{l}S\left(2\mathit{\omega }\right)=M_{\text{bin}}\left(\mathit{\omega }\right)\ast X_{{\alpha }_{\text{bin}}}\left(\mathit{\omega }\right)+\left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{2}{2}\right)\right\}}{M}_{\text{bin}}\left(\mathit{\omega }\right)\right)\\ \ast \left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{{\beta }_{\text{bin}}}\left(\mathit{\omega }\right)\right)+M_{\text{bin}}\left(\mathit{\omega }\right)\ast X_{g_{\text{bin}}}\left(\mathit{\omega }\right)\\ +\left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{2}{0}\right)\right\}}{M}_{\text{bin}}\left(\mathit{\omega }\right)\right)\ast \left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{1}{0}\right)\right\}}{X}_{g_{\text{bin}}}\left(\mathit{\omega }\right)\right)\\ +\left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{0}{2}\right)\right\}}{M}_{\text{bin}}\left(\mathit{\omega }\right)\right)\ast \left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{0}{1}\right)\right\}}{X}_{g_{\text{bin}}}\left(\mathit{\omega }\right)\right)\\ +\left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{2}{2}\right)\right\}}{M}_{\text{bin}}\left(\mathit{\omega }\right)\right)\ast \left(e^{j\left\{{\mathit{\omega }}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{g_{\text{bin}}}\left(\mathit{\omega }\right)\right)\\ =\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\frac{X_{{\alpha }_{\text{bin}}}(\mathit{\omega }-\mathit{\lambda })+e^{j\left\{{(\mathit{\omega }+\mathit{\lambda })}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{{\beta }_{\text{bin}}}(\mathit{\omega }-\mathit{\lambda })}{16}\\ +\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2}\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j\{{(\mathit{\omega }+\mathit{\lambda })}^T\theta }{X}_{g_{\text{bin}}}(\mathit{\omega }-\mathit{\lambda })}{16}.\end{array}$$

Separating the $X_{g_{\text{bin}}}(\mathit{\omega }-\mathit{\lambda })$ to two parts, λ=$\left(\genfrac{}{}{0ex}{}{0}{0}\right)$and λ≠$\left(\genfrac{}{}{0ex}{}{0}{0}\right)$ and downsampling (2ω↦ω), we have

$$\begin{array}{l}S\left(\mathit{\omega }\right)=\sum _{\mathit{\lambda }\in \Pi {\mathbb{Z}}_2^2}\frac{\left(X_{{\alpha }_{\text{bin}}}\left(\frac{\mathit{\omega }-\mathit{\lambda }}{2}\right)+e^{j\left\{{\left(\frac{\mathit{\omega }+\mathit{\lambda }}{2}\right)}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)\right\}}{X}_{{\beta }_{\text{bin}}}\left(\frac{\mathit{\omega }-\mathit{\lambda }}{2}\right)\right)}{4}\\ +\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j{\left(\frac{\mathit{\omega }}{2}\right)}^T\theta }{X}_{g_{\text{bin}}}\left(\frac{\mathit{\omega }}{2}\right)}{16}\\ +\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2\setminus \left(\genfrac{}{}{0ex}{}{0}{0}\right)}\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j\{{(\frac{\mathit{\omega }}{2}+\mathit{\lambda })}^T\theta }{X}_{g_{\text{bin}}}(\frac{\mathit{\omega }}{2}-\mathit{\lambda })}{16},\end{array}$$

where the 1/4 term on $X_{{\alpha }_{\text{bin}}}$ and $X_{{\beta }_{\text{bin}}}$ comes from exchanging ${\mathbb{Z}}_4^2$ with ${\mathbb{Z}}_2^2$. With arithmetic and approximation of (19), the Fourier transform of s(n)simplifies to:

$$\begin{array}{l}S\left(\mathit{\omega }\right)\approx \sum _{\mathit{\lambda }\in \Pi {\mathbb{Z}}_2^2}\left(X_{\alpha }\left(\frac{\mathit{\omega }-\mathit{\lambda }}{2}\right)\right.\hfill \\ \left.+\underset{\text{unwanted filter}}{\underbrace{e^{j{\left(\frac{\mathit{\omega }}{2}\right)}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)}}}{e}^{j{\left(\frac{\mathit{\lambda }}{2}\right)}^T\left(\genfrac{}{}{0ex}{}{1}{1}\right)}{X}_{\beta }\left(\frac{\mathit{\omega }-\mathit{\lambda }}{2}\right)\right)\hfill \\ +\underset{\text{unwanted filter}}{\underbrace{\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j{\left(\frac{\mathit{\omega }}{2}\right)}^T\theta }{H}_{\text{bin}}\left(\frac{\mathit{\omega }}{2}\right)}{16}}}{X}_g\left(\frac{\mathit{\omega }}{2}\right)\hfill \\ +\sum _{\mathit{\lambda }\in \frac{\Pi }{2}{\mathbb{Z}}_4^2\setminus \left(\genfrac{}{}{0ex}{}{0}{0}\right)}\underset{\text{antialias filter}}{\underbrace{\sum _{\mathit{\theta }\in {\mathbb{Z}}_2^2}\frac{e^{j{\left(\frac{\mathit{\omega }}{2}+\mathit{\lambda }\right)}^T\theta }{H}_{\text{bin}}\left(\frac{\mathit{\omega }}{2}-\mathit{\lambda }\right)}{16}}}\underset{\text{aliasing}}{\underbrace{X_g\left(\frac{\mathit{\omega }}{2}-\mathit{\lambda }\right)}}\hfill \end{array}$$

7 Endnote

^aFilter h_bin is a combination of highpass and lowpass. However, binning takes advantage of the fact that the sensor resolution exceeds optical resolution, meaning h_bin is taken to be a lowpass/antialiasing filter on x _g .