Layer-based sparse representation of multiview images

Abstact

This article presents a novel method to obtain a sparse representation of multiview images. The method is based on the fact that multiview data is composed of epipolar-plane image lines which are highly redundant. We extend this principle to obtain the layer-based representation, which partitions a multiview image dataset into redundant regions (which we call layers) each related to a constant depth in the observed scene. The layers are extracted using a general segmentation framework which takes into account the camera setup and occlusion constraints. To obtain a sparse representation, the extracted layers are further decomposed using a multidimensional discrete wavelet transform (DWT), first across the view domain followed by a two-dimensional (2D) DWT applied to the image dimensions. We modify the viewpoint DWT to take into account occlusions and scene depth variations. Simulation results based on nonlinear approximation show that the sparsity of our representation is superior to the multi-dimensional DWT without disparity compensation. In addition we demonstrate that the constant depth model of the representation can be used to synthesise novel viewpoints for immersive viewing applications and also de-noise multiview images.

1 Introduction

The notion of sparsity, namely the idea that the essential information contained in a signal can be represented with a small number of significant components, is widespread in signal processing and data analysis in general. Sparse signal representations are at the heart of many successful signal processing applications, such as signal compression and de-noising. In the case of images, successful new representations have been developed on the assumption that the data is well modelled by smooth regions separated by edges or regular contours. Besides wavelets, which have been successful for image compression [1], other examples of dictionaries that provide sparse image representations are curvelets [2], contourlets [3], ridgelets [4], directionlets [5], bandlets [6, 7] and complex wavelets [8, 9]. We refer the reader to a recent overview article [10] for a more comprehensive review on the theory of sparse signal representation.

In parallel and somewhat independently to these developments, there has been a growing interest in the capture and processing of multiview images. The popularity of this approach has been driven by the advent of novel exciting applications such as immersive communication [11] or free-viewpoint and three-dimensional (3D) TV [12]. At the heart of these applications is the idea that a novel arbitrary photorealistic view of a real scene can be obtained by proper interpolation of existing views. The problem of synthesising a novel image from a set of multiview images is known as image-based rendering (IBR) [13].

Multiview data sets are inherently multi-dimensional. In the most general case multiview images can be parameterised using a single 7D function called the plenoptic function [14]. The dimensions, however, can be reduced by making some simplifying assumptions as discussed in the next section. In particular, the assumption that a camera can move only along two directions leads to the 4D light field parameterisation [15]. If the camera moves only along a straight line the 3D epipolar-plane image (EPI) volume is obtained. We will discuss and use these two parameterisations throughout the article. Intuitively, in the case of a multi-view image array which captures the same scene from different locations, a significantly more sparse representation can be obtained than the independent analysis of each image. When dealing with multiview images, however, the data model must take into account appearing (disocclusions) and disappearing (occlusions) objects. This nonlinear property means that finding a sparse representation is inherently more difficult than in the two-dimensional (2D) case. For this reason, in this article we propose a hybrid method to obtain a sparse representation of multiview images. The fundamental component of the algorithm is the layer-based representation. In many situations, it is possible to divide the observed scene into a small number of depth layers that are parallel to the direction of camera motion. The layer-based representation partitions the multiview images into a set of layers each related to a constant depth in the observed scene. See also Figure 1 for a visual example of the partition. We present a novel method to extract these regions, which takes into account the structure of multiview data to achieve accurate results. In the case of the 4D light field, the sparse representation of the data is then obtained by taking a 4D discrete wavelet transform (DWT) of each depth layer. First we take a view compensated DWT along the two view directions, then the 2D separable spatial DWT is taken. This new representation is more effective than a standard separable DWT and this is shown using nonlinear approximation results. In addition, we present IBR and de-noising applications based on the extracted layers.

Figure 1

Animal Farm layer-based representation [34]. The dataset can be divided into a set of volumes where each one is related to a constant depth in the scene. Observe that the layer contours at each viewpoint remain constant, unless there is an intersection with another layer which is modelled by a smaller depth.

The article is organised as follows. Next we review the structure of multiview data, discuss the layer-based representation and present a high-level overview of our proposed method. In Section 3 we present the layer extraction algorithm. The multi-dimensional DWT is discussed in Section 4. We finally evaluate the proposed sparse representation in Section 5 and conclude in Section 6.

2 Multiview data structure

We start by introducing the plenoptic function and the structure of multiview data. In addition we present a layer-based representation that exploits the multiview structure to partition the data into volumes each related to a constant depth in the scene.

2.1 Plenoptic function

In the IBR framework, multiview images form samples of a multi-dimensional structure called the plenoptic function [14]. Introduced by Adelson and Bergen, this function parameterises each light ray with a 3D point in space (V _x , V _y , V _z ) and its direction of arrival (θ,ϕ). Two further variables λ and t are used to specify the wavelength and time, respectively. In total the plenoptic function is therefore seven dimensional:

$$I=P_7\left(V_x,V_y,V_z,\theta ,\varphi ,\lambda ,t\right),$$

(1)

where I corresponds to the light ray intensity.

In practise, however, it is not feasible to store, transmit or capture the 7D function. A number of simplifications are therefore applied to reduce its dimensionality. Firstly, it is common to drop the λ parameter and instead deal with either the monochromatic intensity or the red, green, blue (RGB) channels separately. Secondly, the light rays can be recorded at a specific moment in time, thus dropping the t parameter. This simplification can for example be applied when viewing a stationary scene. The resulting object is a 5D function.

A popular parameterisation of the plenoptic function, known as the light field [15] defines each light ray by its intersection with a camera plane and a focal plane:

$$I=P_4\left(V_x,V_y,x,y\right),$$

(2)

where as illustrated in Figure 2, (V _x , V _y ) and (x, y) correspond to the coordinates of the camera and the focal plane, respectively. Observe that the dataset can be analysed as a 2D array of images, where each image is formed by the light rays which pass through a specific point on the camera plane. In Figure 3 we illustrate an example of a light field with 16-camera locations. The camera positions are evenly spaced on a 2D grid (V _x , V _y ).

Figure 2

Light field parameterisation. Each light ray is defined by its intersection with a camera plane (V _x , V _y ) and a focal plane (x, y) [36].

Figure 3

Captured light field [34]. Dataset can be analysed as a 2D array of images.

The light field can be further simplified by setting the 2D camera plane to a line. This is also known as the EPI volume [16]:

$$I=P_3\left(V_x,x,y\right).$$

(3)

In comparison to the light field, the EPI is easier to visualise and in the following sections we use it to present a number of concepts. All of the properties are however easily generalised to the light field. Next, we review the EPI and light field structure and present the layer-based representation.

2.2 EPI and light field structure

In this section we show that an EPI volume and a light field are structured datasets. By structure we mean that the fundamental component of multiview images are lines along which the intensity of the pixels is constant. This concept is shown in Figure 4c. This illustration is obtained by stacking an array of images into a volume and taking a cross section through the dataset. It can be clearly observed that pixels are redundant along lines of varying gradients. These pixels along which the intensity of the volume is constant are also known as an EPI line.

Figure 4

EPI volume structure. (a) Camera setup. The sampling camera moves along a straight line; the direction of the camera is perpendicular to the camera location line. (b) Each point in space maps to a line in the EPI volume. Observe that the blue object is closer to the focal plane and therefore occludes the red object. It can be shown using (4) and (5) that a data sample (x, y, V _x ) can be mapped onto a different viewpoint ${V_x}^{\prime }$ with spatial coordinates $x^{\prime }=x-\frac{f\left(V_x^{\prime }-V_x\right)}{Z}$ and y' = y. (c) Shows a cross section of the EPI volume. This figure is obtained by stacking a 1D array of images into a volume and taking a cross section of the dataset. Two EPI lines which correspond to two points in space are illustrated.

In order to demonstrate why the fundamental component of multiview images are EPI lines, consider the setup in Figure 4a. Here we show a simplified version of the scene: the horizontal axis corresponds to the camera location line; the line parallel to it defines the focal plane of each camera^a; and the vertical axis defines the depth of the scene. The curved line thus corresponds to the surface of the object.

Given this setup consider a point in space with coordinates (X, Y, Z). Assuming a Lam-bertian scene^b this point will appear in each of the images with coordinates

$$x=\frac{fX}{Z}-\frac{f{V}_x}{Z},$$

(4)

$$y=\frac{fY}{Z},$$

(5)

where f is the focal length. As illustrated in Figure 4b, x is linearly related to the camera location V _x . The rate of change in the pixel location, also known as the disparity $\Delta p=\frac{f}{Z}$, is inversely related to the depth of the object. Thus, a point in space maps to a line in the EPI volume.

However, the above analysis does not take into account occlusions. Clearly when two lines intersect, the EPI line corresponding to a smaller depth (larger disparity) will occlude all the EPI lines which are related to a larger depth (smaller disparity) in the scene. This principle is illustrated in Figure 4c.

The above concepts can also be extended to the light field, where the camera is allowed to move along two dimensions (V _x , V _y ). In this case, a point (X, Y, Z) maps onto a 2D plane as

$$\left(\begin{array}{c}X\hfill \\ Y\hfill \\ Z\hfill \end{array}\right)\to \left(\begin{array}{c}x\hfill \\ y\hfill \\ V_x\hfill \\ V_y\hfill \end{array}\right)=\left(\begin{array}{c}\left(X-V_x\right)f/Z\hfill \\ \left(Y-V_y\right)f/Z\hfill \\ V_x\hfill \\ V_y\hfill \end{array}\right).$$

(6)

2.3 Layer-based representation

The layer-based representation is an extension of the EPI line concept. The representation partitions the multiview data into homogenous regions, where each layer is a collection of EPI lines modelled by a constant depth plane. An example of a layer-based representation is shown in Figure 1.

Consider a set of EPI lines modelled by a constant disparity Δp _k as shown in Figure 5a. We define the layer carved out by the EPI lines with ${\mathscr{H}}_k$ and the boundary which delimits the region with Γ _k Assuming there are no occlusions, observe that using (4) and (5), the boundary Γ _k can be defined by a contour on one of the viewpoints projected to the remaining frames. More specifically, if we define the contour γ _k (s) = [x (s), y (s)] to be the boundary on the viewpoint (V _x = 0), we obtain the relationship

Figure 5

Comparison between two layers ${\mathscr{H}}_{k-1}$, ${\mathscr{H}}_k$ and their intersection. The layers are ordered in terms of depth (i.e., ${\mathscr{H}}_{k-1}$ corresponds to a smaller depth than ${\mathscr{H}}_k$). (a) A set of EPI lines related to constant disparity $\Delta p_k$. The collection of EPI lines carve out a layer ${\mathscr{H}}_k$. Observe that the complete segmentation of the layer can be defined by a boundary on one viewpoint projected to the remaining frames. (b) ${\mathscr{H}}_{k-1}$ modelled by a constant disparity $\Delta p_{k-1}$. (c) When the two layers intersect, ${\mathscr{H}}_{k-1}$ will occlude ${\mathscr{H}}_k$ as it is modelled by a smaller depth. We define the visible volumes with ${\mathscr{H}}_{k-1}^{\mathcal{V}}$ and ${\mathscr{H}}_k^{\mathcal{V}}$.

$${\Gamma }_k\left(s,V_x\right)=\left(\begin{array}{c}x\left(s\right)-\Delta p_k{V}_x\hfill \\ y\left(s\right)\hfill \\ V_x\hfill \end{array}\right),$$

(7)

where s parameterises the contour γ _k (s).

In order to take into account occlusions, we can use the same principles as in the case of EPI lines; a layer will be occluded when it intersects with other layers which are related to a smaller depth in the scene. We illustrate this in Figure 5, which shows that when two layers intersect we obtain their visible representations^c ${\mathscr{H}}_{k-1}^{\mathcal{V}}$ and ${\mathscr{H}}_k^{\mathcal{V}}$. In this example the layers are ordered in terms of increasing depth (i.e., ${\mathscr{H}}_k$ corresponds to a larger depth than ${\mathscr{H}}_{k-1}$). In general, the visible regions of a layer can be defined as

$${\mathscr{H}}_k^{\mathcal{V}}={\mathscr{H}}_k\bigcap \overline{\left(\bigcup _{j=1}^{k-1}{\mathscr{H}}_j\right)}.$$

(8)

We illustrate the layers ${\mathscr{H}}_k$ and ${\mathscr{H}}_k^{\mathcal{V}}$ from the Animal Farm dataset in Figure 6.

Figure 6

Layer from the animal farm dataset. (a) The unoccluded layer ${\mathscr{H}}_k$ can be defined using the contour γ _k (s) on one viewpoint projected to the remaining frames. The 2D contour is denoted by the red curve on the first image. (b) Occluded layer ${\mathscr{H}}_k^{\mathcal{V}}$ can be inferred by removing the regions which intersect with other layers related to a smaller depth.

There are a number of advantages to segmenting a multiview dataset into layers. Firstly, each layer is highly redundant in the direction of the disparity Δp. This is due to the fact that each layer consists of EPI lines modelled by a constant depth. Secondly, any occluded regions are explicitly defined by the representation. These regions correspond to artificial boundaries, and their specific locations can be used to design a transform which takes them into account. Thirdly, the boundary of each layer can be efficiently defined by a contour on one viewpoint γ (s) and its disparity Δp. This is important if a compression algorithm based on the sparse representation is to be implemented, where the segmentation of each layer must also be transmitted.

2.4 Sparse representation method high-level overview

We use the above analysis to develop a new method that provides sparse representations of multiview images. The method is outlined in Figure 7. The first step of the method is to obtain a layer-based representation. As highlighted in Section 2.3 each layer is modelled by a constant depth plane and a contour on one of the image viewpoints. To extract these layers, we use a variational framework where the general segmentation results are modified to include the camera setup and the occlusion constraints.

Figure 7

High-level block diagram. The data is initially segmented into layers where each volume is related to a constant depth in the scene. The obtained layers are then decomposed using a 4D DWT along the viewpoint and spatial dimensions. Additionally, we illustrate the obtained transform coefficients at each stage of the method.

In the following step we decompose the layers using a 4D DWT applied in a separable fashion across the viewpoint and the spatial dimensions. We modify the viewpoint transform to include disparity compensation and also efficiently deal with disoccluded regions. Additionally, the transform is implemented using the lifting scheme [17] to reduce the complexity and maintain invertibility.

In the following sections we describe the layer extraction and 4D DWT stages in more detail.

3 Layer-based segmentation

Data segmentation is the first stage of the proposed method. Here we introduce our segmentation algorithm which achieves accurate results by taking into account the structure of multiview data. We introduce the method by first describing a general segmentation problem and then showing how that solution can be adapted to extract layers from a light field dataset.

3.1 General region-based data segmentation

Consider a general segmentation problem shown in Figure 8. The aim is to partition an m-dimensional dataset $\mathcal{D}\subset {ℝ}^m$ into subsets $\mathscr{H}$ and $\bar{\mathscr{H}}$ where the boundary which delimits the two regions is defined by Γ (σ) with σ ε ℝ^m-1. This type of problem can be solved using an optimisation framework, where the boundary is obtained by minimising an objective function J:

Figure 8

An m dimensional dataset $\mathcal{D}$ is partitioned into $\mathscr{H}$ and $\bar{\mathscr{H}}$. The boundary is a closed curve defined by Γ [34] and is an (m-1)-dimensional object.

$$\Gamma =\text{arg min}\left\{J\left(\Gamma \right)\right\}.$$

(9)

The cost function in (9) can be defined using either a boundary or region-based approach. The boundary methods evaluate the cost only on Γ and, hence, they are influenced by local data properties and easily affected by noise. In contrast, the region-based methods evaluate the cost function over a complete region and are therefore more robust. A typical region-based cost function [18] can be defined as:

$$J\left(\Gamma \right)=\underset{\mathscr{H}}{\int }d\left(\mathbf{x},\mathscr{H}\right)d\mathbf{x}+\underset{\bar{\mathscr{H}}}{\int }d\left(\mathbf{x},\bar{\mathscr{H}}\right)d\mathbf{x}+\underset{\Gamma }{\int }\eta d\sigma ,$$

(10)

where the descriptor d (·) measures the homogeneity of each region and x ε ℝ^m. The descriptor can be designed such that when x belongs to the region $\mathscr{H}$, d(x,$\mathscr{H}$) tends to zero and vice versa. Note also that (10) has an additional regularisation term η, which acts to minimise the length of the boundary.

The optimisation problem defined in (10) cannot be solved directly for Γ. An iterative solution can, however, be obtained by making the boundary a function of an evolution parameter τ. Consider modeling the boundary using a partial differential equation (PDE), also known as an active contour [19]:

$$\frac{\partial \Gamma \left(\sigma ,\tau \right)}{\partial \tau }=\mathbf{v}\left(\sigma ,\tau \right)=F\left(\sigma ,\tau \right),\mathbf{n}\left(\sigma ,\tau \right),$$

(11)

where v is a velocity vector, which can be expressed in terms of a scalar force F acting in the outward normal direction n to the boundary. The velocity vector v can be evaluated in terms of the descriptor d(·) by differentiating (10) with respect to τ. Applying the Eulerian framework [18], the derivative can be expressed in terms of boundary integrals:

$$\frac{\partial J\left(\Gamma \left(\tau \right)\right)}{\partial \tau }=\underset{\Gamma \left(\tau \right)}{\int }\left[d\left(\mathbf{x},\mathscr{H}\right)-d\left(\mathbf{x},\bar{\mathscr{H}}\right)+\eta \kappa \left(\mathbf{x}\right)\right]\left(\mathbf{v}\cdot \mathbf{n}\right)d\sigma ,$$

(12)

where κ is the curvature of the boundary Γ and · denotes the dot product. Observe that v and n correspond to the velocity and the normal vectors in (11), respectively.

The velocity vector, which evolves Γ in the steepest descent direction can hence be deduced using the Cauchy-Schwarz inequality as:

$$\mathbf{v}=\left[d\left(\mathbf{x},\bar{\mathscr{H}}\right)-d\left(\mathbf{x},\mathscr{H}\right)-\eta \kappa \left(\mathbf{x}\right)\right]\mathbf{n}.$$

(13)

The above framework is also known as 'competition-based' segmentation. This is clear from (13), where a point on the boundary will experience a positive force if it belongs to the region $\mathscr{H}$ and vice versa, hence evolving the contour in the correct direction. In conclusion, the general segmentation problem can be solved by modeling the boundary Γ as a PDE and evolving the contour in the direction of the velocity vector v.

3.2 Multiview image segmentation

In the case of a light field, the goal is to extract N layers, where each volume is modelled by a constant depth Z _k or the associated disparity Δp _k . In the context of the previous section, this is equivalent to segmenting the data into 4D layers {${\mathscr{H}}_1,...,{\mathscr{H}}_N$}, where the boundary of each layer is defined by {Γ₁,..., Γ _N } (the background volume ${\mathscr{H}}_N$ is assigned the residual regions which do not belong to any other layer).

In this setup, ${\mathscr{H}}_k$ corresponds to a layer which is defined by a contour on one viewpoint and a disparity as outlined in Section 2.3. However, due to occlusions the complete layer will not be visible in the dataset. Therefore, we define the cost function^d in terms of the visible regions ${\mathscr{H}}_k^{\mathcal{V}}$, and this leads to the following:

$$\underset{\left\{{\Gamma }_1,\dots ,{\Gamma }_N,\Delta p_1,\dots ,\Delta p_N\right\}}{\text{min}}\left(\sum _{k=1}^N\underset{{\mathscr{H}}_k^{\mathcal{V}}}{\int }{d}_k\left(\mathbf{x},\Delta p_k\right)d\mathbf{x}\right),$$

(14)

where x = [x, y, V _x , V _y ] ^Tand ${\mathscr{H}}_k^{\mathcal{V}}$ correspond to the visible regions of each layer.

Recall that under our assumptions, the intensity along each EPI line is constant. We therefore choose the descriptor d _k (x, Δp _k )

$$d_k\left(\mathbf{x},\Delta p_k\right)={\left[I\left(\mathbf{x}\right)-\mu \left(\mathbf{x},\Delta p_k\right)\right]}^2,$$

(15)

where μ (x, Δp _k ) is the mean of the EPI line which passes through a point x and has a disparity Δp _k .

The aim of the segmentation is then to obtain the layer boundaries Γ _k and the disparity values Δp _k for k = 1,..., N by minimising (14). Observe however that (14) has a large number of unknown variables. In order to minimise the function, we consider the problem of layer evolution and disparity estimation separately and then show how the problem is iteratively solved in Section 3.2.4.

Assuming the layer disparities are known, the minimisation can be simplified to

$$\underset{\left\{{\Gamma }_1,\dots ,{\Gamma }_N\right\}}{\text{min}}\left(\sum _{k=1}^N\underset{{\mathscr{H}}_k^{\mathcal{V}}}{\int }{d}_k\left(\mathbf{x},\Delta p_k\right)d\mathbf{x}\right).$$

(16)

One way to minimise (16) is to evolve iteratively the boundary of each layer. For example assuming there are three volumes ${\mathscr{H}}_1$, ${\mathscr{H}}_2$ and ${\mathscr{H}}_3$ and we choose to evolve the boundary of the first one, the energy function can be expressed as

$$J_1=\underset{{\mathscr{H}}_1^{\mathcal{V}}}{\int }{d}_1\left(\mathbf{x},\Delta p_1\right)d\mathbf{x}+\underset{\int \overline{H_1^{\mathcal{V}}}{d}_1^{\text{out}}\left(\mathbf{x}\right)d\mathbf{x}}{\underbrace{\underset{{\mathscr{H}}_2^{\mathcal{V}}}{\int }{d}_2\left(\mathbf{x},\Delta p_2\right)d\mathbf{x}+\underset{{\mathscr{H}}_3^{\mathcal{V}}}{\int }{d}_3\left(\mathbf{x},\Delta p_3\right)d\mathbf{x}}}$$

(17)

$$=\underset{{\mathscr{H}}_1^{\mathcal{V}}}{\int }{d}_1\left(\mathbf{x},\Delta p_1\right)d\mathbf{x}+\underset{\overline{{\mathscr{H}}_1^{\mathcal{V}}}}{\int }{d}_1^{\text{out}}\left(\mathbf{x}\right)d\mathbf{x},$$

(18)

where $d_1^{\text{out}}\left(\mathbf{x}\right)=d_i\left(\mathbf{x},\Delta p_i\right)$ when $\mathbf{x}\in {\mathscr{H}}_i^{\mathcal{V}}$ for i = 2, 3.

In general, when evolving the k-th layer, the cost function can be simplified to

$$J_k=\underset{{\mathscr{H}}_k^{\mathcal{V}}}{\int }{d}_k\left(\mathbf{x},\Delta p_k\right)d\mathbf{x}+\underset{\overline{{\mathscr{H}}_k^{\mathcal{V}}}}{\int }{d}_k^{\text{out}}\left(\mathbf{x}\right)d\mathbf{x}\text{.}$$

(19)

A possible solution would then be to evaluate the 4D velocity vector of the boundary corresponding to ${\mathscr{H}}_k^{\mathcal{V}}$ This approach, however, would not explicitly take into account the structure of multiview data in the minimisation. In the following we show how (19) is solved by imposing the camera setup and the occlusion constraints.

3.2.1 Imposing camera setup and occlusion constraints

Recall that the background layer corresponds to the object with the largest depth (smallest disparity). If the boundary of this layer increases, it will automatically be occluded by the remaining layers in the dataset. Therefore, the structure of the visible layers will remain unchanged, and hence the cost must also remain the same. When evolving the k-th layer, we model this by using the following indicator function:

$${ℐ}_k\left(\mathbf{x}\right)=\left\{\begin{array}{c}0,\text{if }\mathbf{x}\in \left(\bigcup _{j=1}^{k-1}{\mathscr{H}}_j\right)\\ 1,\text{otherwise},\end{array}\right.$$

(20)

where the layers {${\mathscr{H}}_1,...,{\mathscr{H}}_N$} are ordered in terms of increasing depth. Incorporating this into (18) allows the cost to be expressed in terms of ${\mathscr{H}}_k$ as follows:

$$J_k=\underset{{\mathscr{H}}_k}{\int }{d}_k\left(\mathbf{x},\Delta p_k\right){ℐ}_k\left(\mathbf{x}\right)d\mathbf{x}+\underset{\overline{{\mathscr{H}}_k}}{\int }{d}_1^{\text{out}}\left(\mathbf{x}\right){ℐ}_k\left(\mathbf{x}\right)d\mathbf{x}.$$

(21)

Observe that the integration bounds ${\mathscr{H}}_k$ now correctly correspond to the layer boundary Γ _k . This, therefore, allows the derivative of the cost to be defined as

$$\frac{d{J}_k}{d\tau }=\underset{{\Gamma }_k}{\int }\left[d_k\left(\mathbf{x},\Delta p_k\right){ℐ}_k\left(\mathbf{x}\right)-d_k^{\text{out}}\left(\mathbf{x}\right){ℐ}_k\left(\mathbf{x}\right)\right]\left({\mathbf{v}}_{{\Gamma }_k}\bullet {\text{n}}_{{\Gamma }_k}\right)d\sigma .$$

(22)

Additionally, recall that using the camera setup constraint, the boundary Γ _k can be parameterised by a 2D contour γ _k (s) = [x (s),y(s)] on the image viewpoint (V _x = 0). Substituting this into (22) we obtain

$$\frac{d{J}_k}{d\tau }=\underset{{\gamma }_k}{\int }\left[D_k\left(s,\Delta p_k\right)-D_k^{\text{out}}\left(s\right)\right]\left({\mathbf{v}}_{{\gamma }_k}\bullet {\mathbf{n}}_{{\gamma }_k}\right)ds,$$

(23)

where v_γkand n_γknow correspond to the velocity and the outward normal vector of the 2D boundary^e, respectively. In addition, the new objective functions D _k (⋅) and $D_k^{\text{out}}\left(\cdot \right)$ are defined as

$$D_k\left(s,\Delta p_k\right)=\iint d_k\left(\mathbf{x},\Delta p_k\right){ℐ}_k\left(\mathbf{x}\right)d{V}_{x}d{V}_y$$

(24)

$$D_k^{\text{out}}\left(s\right)=\iint d_k^{\text{out}}\left(\mathbf{x}\right){ℐ}_k\left(\mathbf{x}\right)d{V}_{x}d{V}_y,$$

(25)

where

$$\mathbf{x}=\left[\begin{array}{c}x\left(s\right)-\Delta p_k{V}_x\hfill \\ y\left(s\right)-\Delta p_k{V}_y\hfill \\ V_x\hfill \\ V_y\hfill \end{array}\right].$$

Note that the new descriptors $D_k^{\text{out}}\left(\cdot \right)$ and D _k (·) are simply the descriptors $d_k^{\text{out}}\left(\cdot \right)$ and d _k (·) integrated over the viewpoint dimensions.

The velocity vector which reduces the cost in the direction of steepest descent can therefore be chosen as^f

$${\mathbf{v}}_{\gamma k}=\left[D_k^{\text{out}}\left(s\right)-D_k\left(s,\Delta p_k\right)\right]{\mathbf{n}}_{\gamma k}.$$

(26)

There are two main advantages in simplifying the evolution from a 4D to a 2D contour. First, the approach ensures that the layer boundary remains consistent across the views. Secondly, the complexity is reduced from evolving a 4D hypersurface to a 2D contour. We show a comparison between an unconstrained and constrained boundary evolution in Figure 9. Observe that by imposing the camera setup and occlusion constraints in Figure 9b we obtain a segmentation which is consistent with the EPI structure. In conclusion, (26) defines a velocity vector, which evolves the layer boundary γ _k (s) towards the desired segmentation for each layer.

Figure 9

2D EPI volume cross section showing unconstrained and constrained boundary evolution. [34]. (a) Unconstrained boundary evolution. (b) Constrained boundary evolution. The segmentation is defined using a contour γ(s) on image viewpoint (V _x = 0) and a disparity Δp.

3.2.2 Disparity and number of layers estimation

In the previous section we presented an approach to derive the velocity vector for each layer. However, the knowledge of the disparities is required in order to evaluate the correct evolution. We evaluate these parameters by assuming the 2D layer contours {γ_l...,γ _N } are constant. In this case, the objective function can be simplified to:

$$\underset{\left\{\Delta p_1,\dots ,\Delta p_N\right\}}{\text{min}}\sum _{k=1}^N\underset{{\mathscr{H}}_k^{\mathcal{V}}}{\int }{d}_k\left(\mathbf{x},\Delta p_k\right)d\mathbf{x}.$$

(27)

In contrast to the optimisation of the layer contours, this problem is significantly simpler. A solution can be obtained in an iterative approach by estimating the disparity of each layer assuming the remaining disparities are constant. Each parameter is then estimated using the MATLAB nonlinear optimisation toolbox.

In addition, observe that we require the knowledge of the number of layers N. In our approach we initialise this value using a stereo match algorithm [20]. Alternatively, one could estimate the number of layers using the spectral properties of the light field [21] as proposed in [22].

3.2.3 Level-set method for the boundary evolution

We have demonstrated an approach to derive the velocity vector for each boundary. We then implement the evolution of the active contours using the level-set method [23].

This method, instead of evolving directly the 2D boundary, implicitly models the curve using a higher dimensional surface z = ϕ(x, y, τ). The original boundary is then defined as the zero-level of the new function

$$\begin{array}{c}\gamma \left(s,\tau \right)=\text{arg}\left\{\varphi \left(x,y,\tau \right)\right\}\\ \text{such that }\varphi \left(x,y,\tau \right)=0,\end{array}$$

(28)

where s parameterises the (x, y) coordinates on the 2D boundary.

The evolution equation of the surface can then be derived as follows. First, by implicitly differentiating ϕ (γ (s, τ), τ) = 0 with respect to τ we obtain

$$\frac{\partial \varphi }{\partial \tau }+\frac{\partial \varphi }{\partial x}\frac{\partial x}{\partial \tau }+\frac{\partial \varphi }{\partial y}\frac{\partial y}{\partial \tau }=0\iff \frac{\partial \varphi }{\partial \tau }+\nabla \varphi \left(\gamma \left(s,\tau \right),\tau \right)\cdot \frac{\partial \gamma }{\partial \tau }=0,$$

(29)

where ∇ is the gradient operator. Second, observe that the normal to the surface ϕ (x, y, τ) evaluated on the boundary γ (s, τ) corresponds to the outward normal vector of the boundary n _γ . This implies that

$$\frac{\nabla \varphi }{\left|\nabla \varphi \right|}={\mathbf{n}}_{\gamma }.$$

(30)

Combining (29), (30) with original boundary model $\frac{\partial \gamma }{\partial \tau }=F{\mathbf{n}}_{\gamma }$ we obtain the level-set evolution equation [23]

$$\frac{\partial \varphi \left(x,y,\tau \right)}{\partial \tau }=-F\left(x,y\right)\left|\nabla \varphi \left(x,y,\tau \right)\right|.$$

(31)

There are two main advantages to using the level-set method. First, the surface implicitly models any topological changes of the boundary. Second, unlike other parameterisation schemes, the approach does not suffer instability issues since (31) is evaluated on a fixed cartesian grid.

The evolution of the level-set method does however have a drawback in terms of increased complexity. To evolve the surface, the velocity vector must be evaluated at every position on the grid. In our approach, we deal with this problem by using the narrowband implementation [24], where only a region around the boundary is evolved instead of the complete surface.

3.2.4 Layer segmentation algorithm overview

An overview of the complete layer extraction algorithm is shown in Algorithm 1. First, the 2D contours and the disparity of each layer are initialised using a stereo matching algorithm [20]. The algorithm evaluates the disparity of each layer and then iteratively evolves the boundaries^g using the proposed velocity vector in (26). This process continues for a certain number of iterations or until the change in the overall cost is below a predefined threshold.

An example of the extracted layers using the method outlined in Algorithm 1 is shown in Figure 1. In addition, in Figure 10 we show a comparison between an initialised layer-boundary using the stereo matching algorithm and the final layer contour.

Figure 10

Comparison between an initialised and output layer contour. (a) Tsukuba dataset. (b) Initialised layer contour using a stereo matching algorithm [20]. (c) Layer contour after running Algorithm 1. The extraction algorithm improves the accuracy of the layer-based representation.

To obtain a sparse representation, the obtained layers are decomposed using a 4D DWT as explained in the following section.

4 Data decomposition

In this stage, the redundancy of the texture in each layer shown in Figure 1 is reduced using a multi-dimensional wavelet transform. In the following, we present the inter-view and the spatial transforms in more detail.

Algorithm 1 Layer extraction algorithm

STEP 1: Initialise the 2D boundary of each layer {γ₁, γ₂,..., γ _N } using a stereo matching algorithm (Algorithm [20] in our implementation).

STEP 2: Estimate the disparity of each layer {Δ p₁, Δp₂,... Δp _N } by minimising the squared error along the EPI lines.

STEP 3: Reorder the layers in terms of increasing depth.

STEP 4: Iteratively evolve the layer boundaries assuming the remaining layers are constant:

for k = 1 to N-1 do

Evaluate the velocity vector v_γkof the k-th layer.

Evolve the boundary γ _k according to the velocity vector.

end for

STEP 5: Return to STEP 2 or exit algorithm if the change in the cost (14) is below a predefined threshold.

4.1 Inter-view 2D DWT

We implement the inter-view 2D DWT on each layer in two steps: first by applying a 1D disparity compensated DWT across the row images (V _y ) followed by the column images (V _x ) as illustrated in Figure 11. The process is iterated on the low-pass components to obtain a multiresolution decomposition.

Figure 11

Inter-view 2D DWT is implemented in a separable approach by filtering the image rows followed by the image columns by using the 1D disparity compensated DWT. The red arrow shows the direction of the 1D DWT. (a) Extracted layer: 2 × 2 light field. (b) Transform coefficients following 1D disparity compensated DWT across each row. (c) Transform coefficients following 1D disparity compensated DWT across each column. Note that the background has been labeled grey and is outside the boundary of the layer.

In our implementation of the 1D DWT we use the disparity compensated Haar transform. This is motivated by the fact that the light field intensity along the EPI lines is constant. Therefore, a wavelet with one vanishing moment is enough to obtain a sparse representation. It is applied by modifying the standard lifting equations [17] and including a warping operator $\mathcal{W}$ as follows:

$$\begin{array}{c}{ℒ}_o\left[n\right]=\frac{P_o\left[n\right]-\mathcal{W}\left\{P_e\left[n\right]\right\}}{2},\\ {ℒ}_e\left[n\right]=P_e\left[n\right]+\mathcal{W}\left\{{ℒ}_o\left[n\right]\right\},\end{array}$$

(32)

where, P _o [n] and P _e [n] represent 2D images with spatial coordinates (x, y) located at odd (2n + 1) and even (2n) camera locations, respectively. Following (32), ${ℒ}_e\left[n\right]$ contains the 2D low-pass subband and ${ℒ}_o\left[n\right]$ the high-pass subband. Assuming that $\mathcal{W}$ is invertible and the images are spatially continuous, the above transform can be shown to be equivalent to the standard DWT applied along the motion trajectories [25].

In both the prediction and update steps in (32), the warping operator $\mathcal{W}$ is chosen to maximise the inter-image correlation. This is achieved by using a projective operation that maps one image onto the same viewpoint as its odd/even complement in the lifting step. Using (4) and the fact that the layers are modelled by a constant disparity, we define the warping operation from viewpoint n₁ to n₂ along the V _x dimension as:

$${\mathcal{W}}_{n_1\to n_2}\left\{P\left[n_1\right]\right\}\left(x,y\right)=P\left[n_1\right]\left(x+\Delta p\left(n_2-n_1\right),y\right),$$

(33)

where Δp is the layer disparity.

Note that in the case of an occlusion, the DWT leads to filtering across an artificial boundary and, thus, results in a reduced sparsity efficiency. To prevent this, we use the concept proposed in [26] to create a shape-adaptive transform in the view domain. The transform in (32) is modified whenever a pixel at an even or odd location is occluded such that

$${ℒ}_e\left[n\right]=\left\{\begin{array}{c}{P}_e\left[n\right],\text{occlusion at }2n+1\\ \hat{\mathcal{W}}\left\{P_o\left[n\right]\right\},\text{occlusion at }2n\end{array}\right.,$$

(34)

and the high pass coefficient in ${ℒ}_o\left[n\right]$ is set to zero. In (34), the warping operator $\hat{\mathcal{W}}$ is set to an integer pixel precision to ensure invertibility and is set to be the ceiling of the disparity in (33).

4.2 Spatial shape-adaptive 2D DWT

Following the inter-view transform we reduce the intra-view redundancy by using a 2D DWT. However, prior to applying the 2D DWT on each image, we recombine the transform coefficients into a single layer. This is done to increase the number of decompositions which can be applied by the spatial transform. A comparison between the original and recombined layers is illustrated in Figure 12. Note that due to occlusions and the way in which the inter-view transform is implemented, two or more layers may overlap in each subband. In this case, we apply a separate spatial transform to the overlapped pixels.

Figure 12

Merging of the inter-view transform coefficients. (a) Tsukuba transform coefficients following the inter-view transform. Each of the three transformed layers is composed of one low-pass subband and three high frequency images. (b) Recombined layers. The view subbands from each layer are grouped into a single image to increase the number of decompositions that can be applied by the spatial transform. In each subband two or more layers may overlap. We apply a separate shape-adaptive 2D DWT to the overlapped pixels.

Note that the overlapped pixels are commonly bounded by an irregular (non-rectangular) shape. For that reason, the standard 2D DWT applied to the entire spatial domain is inefficient due to the boundary effect. We therefore use the shape-adaptive DWT [26] within arbitrarily shaped objects. The method reduces the magnitude of the high pass coefficients by symmetrically extending the texture whenever the wavelet filter is crossing the boundary. The 2D DWT is built as a separable transform with linear-phase symmetric wavelet filters (9/7 or 5/3 [27]), which, together with the symmetric signal extensions, leads to critically sampled transform subbands.

5 Evaluation

In this section we evaluate the performance of the proposed sparse representation using its nonlinear approximation properties. In addition, we demonstrate de-noising and IBR applications based on the proposed decomposition.

5.1 Nonlinear approximation

We evaluate the sparseness of the representation using its N-term nonlinear approximation. To implement the nonlinear approximation, we keep the N largest coefficients in the transform domain, reconstruct the data and evaluate the data fidelity in terms of PSNR.

Our results show that the proposed layer-based representation offers superior approximation properties compared to a typical multi-dimensional DWT^h. We demonstrate this in Figure 13 on three datasets: Tsukuba light field [272 × 368 × 4 × 4], Teddy EPI [368 × 352 × 4] and Doll EPI [368 × 352 × 4] (all from [28]), which vary in terms of scene complexity, number of images and spatial resolution. We show that in each case our approach achieves a sparser representation across the complete range of retained coefficients, with PSNR gains of up to 7 dB on the Tsukuba light field. The Tsukuba light field has a larger PSNR improvement than the respective Teddy and Doll EPI volumes due to the additional viewing dimension. This means that there exists more redundant information and this is fully exploited by our representation. We also show that the PSNR curves correspond to a subjective improvement in Figure 14.

Figure 13

N-term nonlinear approximation of the layer-based representation in comparison to a standard multi-dimensional DWT. The percentage of retained coefficients is evaluated as $\frac{100N_r}{N_d}$, where N _r is the total number of retained coefficients and N _d is the total number of pixels in the dataset. (a) Tsukuba light field. (b) Teddy EPI. (c) Doll EPI.

Figure 14

Nonlinear approximation subjective evaluation. (a) Sparse layer-based representation (PSNR 29.62dB) with 1.18% of coefficients retained. (b) Standard 3D DWT (PSNR 26.6dB) with 1.20% of coefficients retained. (c) Original Teddy EPI dataset.

We note that the nonlinear approximation metric is also a good indicator of the compression capability of the representation. In practice, the issue of compression is more complicated due to the additional problem of encoding the locations of the significant coefficients and also to the rate allocation. These issues are beyond the scope of this paper, however, we refer the reader to [29] where these problems are addressed and a complete multiview image compression method is presented.

5.2 De-noising

Here we demonstrate de-noising results based on the proposed sparse representation in the presence of additive white Gaussian noise (AWGN). We implement the de-noising by soft thresholding the wavelet coefficients in each subband. For each subband, the threshold is chosen by minimising the Stein's Unbiased Risk Estimate (SURE) of the mean squared error (MSE) [30].

Note that the aim of this section is not to compare the results to the state-of-the-art in multiview de-noising techniques but to demonstrate that the sparse representation can be used for de-noising applications. In Figure 15 we compare our algorithm to the competitive SURE-LET OWT de-noising method [31] applied to each image independently. In this setup, we assume the noise is added to the extracted layers. Analysing the Tsukuba light field, Teddy EPI and Doll EPI datasets, our approach corresponds to a PSNR improvement of up to 2 dB. The light field has the most significant gain due to a sparser representation, which results from the larger number of images in the dataset.

Figure 15

De-noising comparison between the proposed sparse representation and the SURE-LET OWT method [31]. In the proposed approach, the de-noising is implemented by soft thresholding the transform coefficients in each subband. The threshold step-size is chosen by minimising the SURE estimate of the MSE. (a) Tsukuba light field. (b) Teddy EPI. (c) Doll EPI.

The subjective results are illustrated in Figure 16 and these clearly show that the proposed sparse representation attains more visually pleasing results than the SURE-LET OWT method.

Figure 16

De-noising subjective evaluation. (a) Tsukuba light field corrupted with AWGN (PSNR 18.6 dB). (b) De-noised dataset using proposed sparse representation (PSNR 28.65 dB). (c) De-noised dataset using SURE-LET OWT applied to each image independently (PSNR 27.16 dB).

5.3 Image-based rendering

In this section we present viewpoint interpolation results based on the layer-based representation shown in Figure 1.

To render an image at an arbitrary viewpoint, we linearly interpolate the closest available images. Recall that the data pixels are highly correlated in the direction of the disparity. We take this into account by modifying the support of the rendering kernel according to the disparity of each layer. Additionally, we modify the interpolation in the presence of occlusions to further improve the results. In this case, only pixels that belong to the layer are used in the rendering process.

In order to obtain an objective evaluation, we use the leave-one-out approach. In this case the images located at odd camera viewpoint locations are removed and synthesised using the scene modelⁱ.

We compare our results to a state-of-the-art stereo matching algorithm [32] and an EPI tubes extraction method [33]. These methods specify the structure of the EPI lines, and the interpolation is implemented using the same approach as in the proposed algorithm.

In Table 1 we show a comparison on four datasets: Dwarves EPI [555 × 695 × 7] [28], Lobby EPI [800 × 800 × 5], Desk light field [500 × 500 × 4 × 4] and Animal Farm EPI [235 × 625 × 32 (last three from [34]). Observe that the layer-based representation achieves an improved SNR in comparison to both the stereo matching and the EPI tubes extraction method on all datasets.

Table 1 Image-based rendering evaluation

In addition to the quantitative evaluation, we show a subjective comparison in Figure 17. This shows that interpolation using the proposed layer-based representation achieves significantly improved results. We note that using our method we obtain artifact free and photo-realistic images. In comparison, aliasing artifacts are present in the stereo matching and EPI tube extraction methods. This is due to incorrect compensation of the interpolation kernel, which stems from inaccurate depth correspondence in the scene.

Figure 17

IBR subjective evaluation. (a-c) Desk light field interpolation using (a) proposed layer-based representation, (b) stereo matching algorithm [32] and (c) EPI tubes extraction method [33]. (d-f) Dwarves EPI interpolated images in the same order. Note that the layer extraction algorithm has been implemented on the grayscale images. The obtained layers were then used to interpolate each of the RGB channels to show colour results.

6 Conclusion

We presented a novel method to obtain a sparse representation of multiview images. The fundamental component of the algorithm is the layer-based representation, which partitions the multiview images into a set of layers each related to a constant depth in the scene. We presented a novel method to obtain the layer-based representation using a general segmentation framework which takes into account the structure of multiview data to achieve accurate results. The obtained layers are then decomposed using a 4D DWT applied in a separable approach, first across the camera viewpoint and then the image dimensions. We modify the viewpoint transform to efficiently deal with occlusions and depth variations. Simulation results based on nonlinear approximation have shown that the sparsity of our representation is superior to a multi-dimensional DWT with the same decomposition structure without disparity compensation. In addition, we have shown that the proposed representation can be used to efficiently synthesise novel viewpoints for IBR applications and also de-noise multiview images in the presence of AWGN.

Endnotes

^aEach camera in the setup is modelled by the pinhole model [35]. ^bLight ray intensity is constant when an object is observed from a different angle. ^cBy visible regions we mean the EPI line segments which are present in the EPI volume. ^dWe have not included the regularisation terms for the sake of clarity. ^eIt can be shown that given a fronto-parallel depth plane, the inner product of v_γk· n_γkis equal to v_Γk· n_Γk. ^fNote that in practise we also include a regularisation term to constrain the evolution according to the curvature of the boundary. ^gNote that the background layer ${\mathscr{H}}_N^{\mathcal{V}}$ is automatically assigned all of the regions which do not belong to the remaining layers and is therefore not evolved. ^hThis multi-dimensional DWT has the same decomposition structure as our method, however no disparity compensation. iThe extracted layers are obtained using the dataset with the removed images.