Methods for depth-map filtering in view-plus-depth 3D video representation

2.1 Properties of depth maps

Depth map is gray-scale image which encodes the distance to the given scene pixels for a certain perspective. The depth is usually aligned with and ac-companies the color view of the same scene [9].

Single view plus depth is usually a more efficient representation of a 3D scene than two-channel stereo. It directly encodes geometrical information contained otherwise in the disparity between the two views thus providing scalability and possibility to render multiple views for displays with different sizes [1]. Structure-wise, the depth image is piecewise smooth (as representing gradual change of depth within objects) with delineated, sharp discontinuities at object boundaries. Normally, it contains no textures. This structure should be taken into account when designing compression or filtering algorithms.

Having a depth map given explicitly along with color texture, a virtual view for a desired camera position can be synthesized using DIBR [2]. The given depth map is first inversely-transformed to provide the absolute distance and hence the world 3D coordinates of the scene points. These points are projected then onto a virtual camera plane to obtain a synthesized view. The technique can encounter problems with dis-occluded pixels, non-integer pixel shifts, and partly absent background textures, which problems have to be addressed in order to successfully apply it [1].

The quality of the depth image is a key factor for successful rendering of virtual views. Distortions in the depth channel may generate wrong objects contours or shapes in the rendered images (see, for example, Figure 1d,e) and consequently hamper the visual user experience manifested in headache and eye-strain, caused by wrong contours of familiar objects. At the capture stage, depth maps might be not well aligned with the corresponding objects. Holes and wrongly estimated depth points (outliers) might also exist. At the compression stage, depth maps might suffer from blocky artifacts if compressed by contemporary methods such as H.264 [10]. When accompanying video sequences, the consistency of successive depth maps in the sequence is an issue. Time-inconsistent depth sequences might cause flickering in the synthesized views as well as other 3D-specific artifacts [11].

At the capture stage, depth can be precisely estimated in floating-point hight resolution, however, for compression and transmission it is usually converted to integer values (e.g., in 256 gray-scale gradations). Therefore, the depth range and resolution have to be properly maintained by suitable scaling, shifting, and quantizing, where all these transformations have to be invertible.

Depth quantization is normally done in linear or logarithmic scale. The latter approach allows better preservation of geometry details for closer objects, while higher geometry degradation is tolerated for objects at longer distances. This effect corresponds to the parallax-based human stereo-vision, where the binocular depth cue losses its importance for more distanced objects and is more important and dominant for closer objects. The same property can be achieved if transmitting linearly quantized inverse depth maps. This type of depth representation basically corresponds to binocular disparity (also known as horizontal parallax), including again necessary modifications, such as scaling, shifting, and quantizing.

2.2 Depth map filtering problem formulation

This section formally formulates the problem of filtering of depth maps and specifies the notations used hereafter. Consider an individual color video frame in YUV (YCbCr) or RGB color space y(x) = [y ^Y (x), y ^U (x), y ^V (x)] or y(x) = [y ^R (x),y ^G (x),y ^B (x)], together with the associated per-pixel depth z(x), where x= [x₁, x₂] is a spatial variable, x∈ X, X being the image domain.

A new, virtual view η(x) = [η ^Y (x), η ^U (x), η ^V (x)] can be synthesized out of the given (reference) color frame and depth by DIBR, applying projective geometry and knowledge about the reference view camera, as discussed in Section 2.1 [2]. The synthesized view is composed of two parts, η= η _v + η _o , where η _v denotes the visible pixels from the position of the virtual view camera and η_o denotes the pixels of occluded areas. The corresponding domains are denoted by X _v and X _o correspondingly, X _v ⊂ X, X _o = X\X _v .

where C = Y, U, V or R, G, B. Both degradations are modeled as independent white Gaussian processes: ${\epsilon }^C\left(\cdot \right)~N\left(0,{\sigma }_C^2\right),\epsilon \left(\cdot \right)~N\left(0,{\sigma }^2\right)$. Note that the variance of color signal noise (${\sigma }_C^2$) differs from the one of the depth signal noise (σ²).

If degraded depth and reference view are used in DIBR, the result will be a lower-quality synthesized view $\stackrel{⌣}{\eta }$. Unnatural discontinuities, e.g., blocking artifacts, in the degraded depth image cause geometrical distortions and distorted object boundaries in the rendered view. The goal of the filtering of degraded depth maps is to mitigate the degradation effects (caused by e.g., quantization or imperfect depth estimation) in the depth image domain, i.e., to obtain a refined depth image estimate $ẑ$, which would be closer to the original, error-free depth, and would improve the quality of the rendered view.

2.3 Depth map quality measures

Measuring the quality of depth maps has to take into account that depth maps are type of imagery which are not visualized per-se, but through rendered views.

In our study, we consider two types of measures:

Measures for the first group have the advantage of being simple, while measures from the second group are closer to subjective perception of depth. For both of these groups we suggest and test new measures.

Percentage of bad pixels

Bad pixels percentage metric is defined in [12] to measure directly the performance of stereo-matching algorithms.

$BAD=\frac{100}{M}\sum _x\left(\left|z\left(x\right)-ẑ\left(x\right)\right|>{\Delta }_d\right),$

where $ẑ$ is the computed depth, z is the true depth and Δ _d is a threshold value, (usually equal to 1). Figure 2 shows thresholding results for some highly compressed depth maps. We include this metric to our experiments in an attempt to check its applicability for comparing post-filtering methods. For this metric, lower value means better quality.

Depth consistency

Analysing the BAD metric, one can notice that the thresholding imposed there, does not emphasize the importance of small or big differences. It is equally important, when the error is just a quantum above the threshold and when it is quite high.

In a case of depth degraded by compression artifacts, almost all pixels are quantized thus changing their original values and therefore causing the BAD metric to show very low quality while the quality of the rendered views will not be that bad.

Starting from the idea that the perceptual quality of rendered view will depend more on the amount of geometrical distortions than on the number of bad depth pixels, we suggest to give preference to areas where the change between ground truth depth and compressed depth is more abrupt. Such changes are expected to cause perceptually high geometrical distortions.

Consider the gradient of the difference between true depth and approximated depth $\nabla \xi =\nabla \left(z-ẑ\right)$. By depth consistency we denote the percentage of pixels, having magnitude of that gradient higher than a pre-specified threshold.

$CONSIST=\frac{100}{N}\sum \left({∥\nabla \xi ∥}_2>{\sigma }_{consist}\right).$

(5)

The measure favors non-smooth areas in the restored depth considered as main source of geometrical distortion, as illustrated in Figure 3.

3.1 LPA approach

The anisotropic LPA is a pixel-wise method for adaptive signal estimation in noisy conditions [16]. For each pixel of the image, local sectorial neighborhood is constructed. Sectors are fitted for different directions. In the simplest case, instead of sectors, 1D directional estimates of four (by 90 degrees) or eight (by 45 degrees) different directions can be used. The length of each sector, denoted as scale, is adjusted to meet the compromise between the exact polynomial model (low bias) and sufficient smoothing (low variance). A statistical criterion, denoted as intersection of confidence intervals (ICI) rule is used to find this compromise [18, 19], i.e., the optimal scale for each direction. These optimal scales in each direction determine an anisotropic star-shape neighborhood for every point of the image well adapted to the structure of the image. This neighborhood has been successfully utilized for shape-adaptive transform-based color image de-noising and de-blurring [14].

After finding optimal scales h₊(x) for each direction at pixel x₀ a star shape neighborhood ${\Omega }_{x_0}$ is formed, as illustrated in Figure 4a. There is a clear evidence that there is a relation between the adaptive neighborhoods and the (distorted) depth, as examplified in Figure 4b. Adaptive neighborhoods are formed for every pixel in the image domain X. Once adaptive neighborhoods are found, one must find some modeling for depth channel before utilizing this structural information.

Color-driven LPA-ICI

Luminance channel of the color image is usually considered as the most informative channel for processing and also as the most distinguishable by the human visual system. That is why many image filtering mechanisms use color transformation to extract luminance and then process it in different way to compare with chrominance channels. This also may be explained by the fact that luminance is usually the less noisy component and thus it is most reliable. Nevertheless, for some color processing tasks pixel differentiation based only on luminance channel is not appropriate due to some colors may have the same luminance whereas they have different visual appearance.

Our hypothesis is that a color difference signal will better differentiate color pixels, as illustrated in Figure 5. L₂ norm is used to form a color difference map around pixel x₀:

$C_x^{x_0}=\sqrt{{\left(Y_{x_0}-Y_x\right)}^2+{\left(U_{x_0}-U_x\right)}^2+{\left(V_{x_0}-V_x\right)}^2},$

(14)

where x₀ is the currently processing pixel, and $x\in {\Omega }_{x_0}$.

The color difference map is used as a source of structural information, i.e., the LPA-ICI procedure is run over this map instead over the luminance channel. Differences are illustrated in Figure 5. In our implementation, we calculate color-difference only for those pixels of the neighborhood which participate in 1D directional convolutions. Additional computational cost of such implementation is about 10% of the overall LPA-ICI procedure.

For all mentioned LPA-ICI based strategies the main adjusting parameter, capable to set proper smoothing for varying depth degradation parameter (e.g., varying QP in coding) is the parameter Γ.

3.1.1 Comparison of LPA-ICI approaches

The performance of different versions of the LPA-ICI approach are compared in Figure 6. The 'normalized RMSE' (equation 6) and 'depth consistency' (equation 5) metrics have been computed and averaged over a set of test images. The parameters of the filters were empirically optimized with 'depth consistency' (equation 5) as a cost measure. As it can be seen, the color-driven LPA-ICI approach with plane fitting and encapsulated aggregation is the best performing approach, while also having the most stable and consistent results. Because of the superior performance of color-driven LPA-ICI, we use it in the experiments from now on. All experiments and comparisons involving LPA-ICI presented in the following sections refer to the optimized color-driven LPA-ICI implementation.

3.2 Bilateral filter

where ${\omega }_a\left(t\right)=e^{-\frac{t}{{\gamma }_a}},\left(a=s,c\right)$ and u∈ Ω _x are neighborhood pixels of point x. This design allows for relatively fast implementation by storing all possible color and distance weights as look-up tables. Parameters γ _s , γ _c and processing window size Ω _x are adjustable parameters of the filter. Figure 7 illustrates the filtering. The color channel (Figure 7a) provides the color difference information, with respect to the processed pixel position (Figure 7b). It is further weighted by spatial Gaussian filter to determine the weights of pixels from the depth map taking part in estimating the current (central) pixel (Figure 7f).

3.3 Spatial-depth super resolution approach

where L _η denotes tunable search range.

The bilateral filtering, defined as in Equation 15 enforces an assumption of piecewise smoothness. The procedure is illustrated in Figures 8 and 9. The approach resembles the local depth estimation idea, where a volumetric cost volume is further aggregated with bilateral filter.

After the bilateral filtering is applied to the cost volume, the depth is refined and true depth discontinuities might be completely recovered.

In our implementation of the filter, we have suggested two simplifications:

The main tunable parameters of the filter are the parameters of the bilateral filter γ _d and γ _c . As long as the processing time of the filter still remains extremely high, we do not perform optimization of this filter directly, but assume that the optimal parameters γ _d = f _d (QP) and γ _c = f _c (QP) found for the direct bilateral filter are optimal or nearly optimal for this filter as well.

3.4 Practical implementation of the super resolution filtering

Furthermore, the computation cost is reduced by assuming that not all depth hypotheses are applicable for the current pixel. A safe assumption is that only depths within the range d ∈ [d_min, d_max] where d_min = min(z(u)), d_max = max(z(u)), u ∈ Ω _x have to be checked.

Additionally, depth range is scaled with the purpose to further reduce the number of hypothesizes. This step is especially efficient for certain types of distortions such as compression (blocky) artifacts. For compressed depth maps, the depth range appears to be sparse due to the quantization effect.

Figure 10 illustrates histograms of depth values before and after compression so to confirm the use of rescaled search range of depth hypotheses. This modification speeds up the procedure and relies on the subsequent quadratic interpolation to find the true minimum. A pseudo-code of the suggested procedure in Equation 20 is given in following listing.

Require: C, the color image; D, the depth image; X, a spatial image domain

for all x ∈ X do

   $d_{\text{min}}=\underset{u\in {\Omega }_x}{\text{min}}{D}_u,d_{\text{max}}=\underset{u\in {\Omega }_x}{\text{max}}{D}_u$

   if d_max - d_min < γ_thr then

        ${\widehat{D}}_x=D_x$

   else

        $F\left(x,u\right)=\frac{∥C_u-C_x∥∥u-x∥}{{\sum }_u∥C_u-C_x∥∥u-x∥}\left\{\mathsf{\text{bilateral}}\mathsf{\text{weights}}\right\}$

      S_best ← S_max {S_max is maximum reachable value for S}

      for d = ⌊d_min⌋ to [d_max] do

         S← 0

         for all u∈ Ω _x do

            E ← min{(d - D _u )², ηL}

            S ← S + F ( x, u ) * E

         end for

         if S < S_best then

            S_best ← S

            d_best ← d

         end if

      end for

      ${\widehat{D}}_x=d_{\mathsf{\text{best}}}$

   end if

end for

The memory foot-print required by our implementation is significantly lower than the one imposed by a direct implementation. A straightforward implementation would require a large memory buffer to store the complete cost volume in order to process it pixel-by-pixel and avoid computing (the same) color weights across different slices. In the proposed implementation, two memory buffers with relatively low sizes are required: a memory buffer which is equal to the processing window size to store current color weights, and a buffer to store the cost values for the current pixel along the 'd' dimension. In case of multi-thread (parallelized) implementation, these memory buffers are multiplied by the number of processing threads. More information about platform-specific optimization of the proposed algorithm is given in [20].

Figure 11 illustrates the performance in terms of speed. The figure shows experiments with different implementations of the filtering procedure. The 'Teddy' dataset has been processed (see also Figure 12 for reference). All filter versions have been implemented in C and then compiled into MEX files to be run within Matlab environment. The experiments have been run on a 1.3 GHz Pentium Dual-Core processor with 1 Gb of RAM under MS Windows XP operating system. In the figure, the vertical axis shows the execution time in seconds and the horizontal line shows the number of slices processed (i.e., the depth dynamic range assumed). The dotted curve shows single-pass bilateral filtering. It does not depend on the dynamic range, but on the window size, thus it is a constant in the figure. The red line shows the computational time for the original approach implemented as a three step procedure for the full dynamic range. Naturally, it is a linear function with respect to the slices to be filtered. Our implementation (blue curve) applying a reduced dynamic range is also linearly depending on the number of slices, but with dramatically reduced steepness.

4.1 Experimental setting

In our experiments, we consider depth maps degraded by compression. Thus degradation is characterized by the quantization parameter (QP). For better comparison of selected approaches, we present two types of experiments. In the first set of experiments, we compare the performance of all depth filtering algorithms assuming the true color channel is given (it has been also used in the optimization of the tunable parameters). This shows ideal filtering performance, while in practice it cannot be achieved due to the fact that the color data is also degraded by e.g., compression.

In the second set of experiments, we compare the effect of depth filtering in the case of mild quantization of the color channel. General assumption is that color data is transmitted with backward compatibility in mind, and hence most of the bandwidth is occupied by the color channel. Depth maps in this scenario are heavily compressed, to consume not more than 10-20% of the total bit budget [21, 22].

We consider the case where both y and z are to be coded as H.264 intra frames with some QPs, which leads to their quantized versions y _q and z _q . The effect of quantization of DCT coefficients has been studied thoroughly in the literature and corresponding models have been suggested [23]. Following the degradation model in Section 2.2, we assume quantization noise terms added to the color channels and the depth channel considered as independent white Gaussian processes: ${\epsilon }^C\left(\cdot \right)~N\left(0,{\sigma }_C^2\right),\epsilon \left(\cdot \right)~N\left(0,{\sigma }^2\right)$. While this modeling is simple, it has proven quite effective for mitigating the blocking artifacts arising from quantization of transform coefficients [14]. In particular, it allows for establishing a direct link between the QP and the quantization noise variance to be used for tuning deblocking filtering algorithms [14].

Training and test datasets for our experiments (see Figure 12) were taken from Middlebury Evaluation Testbench [12, 24, 25]. In our case, we cannot tolerate holes and unknown areas in the depth datasets, since they produce fake discontinuities and unnatural artifacts after compression. We semi-manually processed 6 images to fill holes and to make their width and height be multiples of 16.

4.2 Visual comparison results

Figures 13 and 14 present depth images paired with consecutive rendered frames (no occlusion filling is applied). This approach helps to illustrate artifacts in the depth channel as well as their effect on the rendered images.

As it is seen in the top row (a), rendering with true depth, results in sharp and straight object contours, as well as in continuous shapes of occlusion holes. For such holes, a suitable occlusion filling approach will produce good estimate.

Row (b) shows unprocessed depth after strong compression (H.264 with QP = 51) frame and its rendering capability. Objects edges are particularly affected by block distortions.

With respect to occlusion filling, the methods behave as follows.

Among all filtering results, the latter one contains occlusions which are most similar to the occlusions of the original depth rendering result. Visually, super-resolution depth approach is considered to be the best. The numerically estimated results for all presented approaches are presented in following section.

4.3 Numerical results for ideal color channel

Figure 15 summarizes averaged results over the three test datasets: 'venus', 'sawtooth', and 'teddy'.

X-axis on all the plots represents varying QP parameters of the H.264 Intra coding, while each Y -axis shows a particular metric. On the most of the metric plots it is visible that there is no need to apply any kind of filtering before QP reaches some critical value. Before that value, the quality of the compressed depth is high enough, so no filtering could improve it.

The group of structurally-constrained methods clearly outperforms the simple methods working on the depth image only. The two PSNR-based metrics and the BAD metric seem to be less reliable in characterizing the performance of the methods. The three remained measures, namely depth consistency, discontinuity falses and gradient-normalized RMSE perform in a consistent manner. While Normalized RMSE is perhaps the measure closest to the subjective perception, we favor also the other two measures of this group as they are relatively simple and do not require calculation of the warped (rendered) image.

4.4 Numerical results for compressed color channel

So far, we have been working with uncompressed color channel. It has been involved in the optimizations and comparisons. Our aim was to characterize the pure influence of the depth restoration only.

In practice, when 'color-plus-depth' frame is compressed and then transmitted over a channel, the color frame is also compressed with a pre-specified bit-rate, aiming at maximizing visual quality of the video. Transmission of 'color plus depth' stream has also to be constrained within a given bit-budget. Thus, receiver-side device has to cope with compressed color and compressed depth.

In the second experiment, we assume mild quantization of the color image, e.g., by QP = 30. For our test imagery, the first depth QP corresponds to about 10% of the total bit-rate. 'Depth consistency', 'Discontinuity falses' and 'PSNR of rendered channel' are calculated for different depth maps: compressed, post-filtered with LPA-ICI filtering approach, post-processed with the bilateral filter and post-filtered with our implementation of the super-resolution approach. The resulting numbers are averaged over three dataset images. Visual results of hypothesis filtering are presented in Figure 16 which shows comparison between higly-compressed depth filtered with compressed color (second row) and same, filtered with ideal color (last row). The numerical results are given in Figures 17, 18, and 19. Cases with post-processed depth are marked with color. One can see that the depth postprocessing clearly makes a difference allowing to use stronger quantization of the depth channel and still to achieve good quality.