Distributed transform coding via source-splitting

Abstract

Transform coding (TC) is one of the best known practical methods for quantizing high-dimensional vectors. In this article, a practical approach to distributed TC of jointly Gaussian vectors is presented. This approach, referred to as source-split distributed transform coding (SP-DTC), can be used to easily implement two terminal transform codes for any given rate-pair. The main idea is to apply source-splitting using orthogonal-transforms, so that only Wyner-Ziv (WZ) quantizers are required for compression of transform coefficients. This approach however requires optimizing the bit allocation among dependent sets of WZ quantizers. In order to solve this problem, a low-complexity tree-search algorithm based on analytical models for transform coefficient quantization is developed. A rate-distortion (RD) analysis of SP-DTCs for jointly Gaussian sources is presented, which indicates that these codes can significantly outperform the practical alternative of independent TC of each source, whenever there is a strong correlation between the sources. For practical implementation of SP-DTCs, the idea of using conditional entropy constrained (CEC) quantizers followed by Slepian-Wolf coding is explored. Experimental results obtained with SP-DTC designs based on both CEC scalar quantizers and CEC trellis-coded quantizers demonstrate that actual implementations of SP-DTCs can achieve RD performance close to the analytically predicted limits.

1 Introduction

Many new applications such as multi-camera imaging systems rely on networks of distributed wireless sensors to acquire signals in the form of high-dimensional vectors [1]. In such situations, an encoder in each sensor quantizes a vector of observation variables (without exchanging any information with other sensors) and transmits its output to a central processor which jointly decodes all the sources. The strong statistical dependencies among the signals observed by different sensors can be exploited in the decoder to reduce the transmission bit-rate of each sensor. This problem, in general, is referred to as distributed (or multiterminal) vector quantization (VQ). The design of a distributed VQ for a large number of source variables is a difficult task. A practically simpler, yet very effective approach to quantizing a large number of correlated variables by using a bank of single variable quantizers is transform coding (TC) [2–4]. Clearly, TC can be used for distributed VQ when separately observed vectors have both inter-vector and intra-vector statistical dependencies, a situation typical in applications such as camera networks. Most of the previous work [5–7] studies Wyner-Ziv (WZ) transform coding (WZ-TC), which is a special case of more general multiterminal transform coding (MT-TC) [8]. In WZ-TC, a single source is quantized given that the decoder has access to side information about the source.

Information-theoretic studies of distributed transform coding (DTC) can be found in [5, 8]. In [8], the optimal linear transform for Gaussian WZ-TC under the mean square-error (MSE) criterion is shown to be the conditional Karhunen-Loéve transform (CKLT), which is a natural extension of the result in [2]. This result is based on the assumption that each transform coefficient is compressed by a rate-distortion (RD) optimal WZ quantizer and hence describes the optimal performance theoretically attainable (OPTA) in Gaussian WZ-TC. However, the optimal solution to the more general MT-TC problem remains unsolved, even for the Gaussian case. In [8], an iterative descent algorithm for determining the OPTA of Gaussian MT-TC problem is given. It is shown that, while this algorithm [referred to as the distributed KLT (DKLT)] always converges to a solution, the final solution is not necessarily the global optimum. In any case, the practical implementation of distributed quantizers implied by the DKLT remains an open problem. In [5] WZ-TC based on high-rate scalar quantization and ideal Slepian-Wolf (SW) coding [9] is studied. In particular, it is shown that, for jointly Gaussian vectors, CKLT followed by uniform scalar quantization is asymptotically optimal, a natural extension of the result in [10] for entropy-coded quantization at high-rate. More importantly, the bit-allocations and quantizer-step sizes found in [5] can be used for practical design of WZ-transform codes as long as high-rate approximations hold. However, we note that, even when scalar quantizers are used, achieving good performance with this approach still requires the use of a subsequent block-based SW coding method (e.g., Turbo codes or LDPC codes). Other previous studies on WZ-TC can be found in [6, 7]. However, they rely on WZ scalar quantization of transform coefficients. Such methods are therefore most suitable for applications requiring low coding delay as their performance is strictly inferior to block-based quantization.

In contrast to WZ-TC, we consider in this article the practical design of two-terminal transform codes for jointly Gaussian vectors in which arbitrary transmission rates can be assigned to each terminal. Our approach is based on the idea of source-splitting[11, 12] to convert the two-terminal TC problem into two WZ-TC problems. Since transform codes quantize linear projections, we perform source splitting in terms of optimal linear approximations, i.e., a linear approximation of one source is provided as decoder side-information for the other source. The proposed source-split DTC (SP-DTC) approach only requires the design of two-sets of WZ quantizers sequentially, and avoids having to iteratively optimize two sets of WZ quantizers to each other as in [8]. However, this approach requires the solution of a bit allocation problem involving dependent WZ quantizers. To solve this problem for Gaussian sources, we propose an efficient tree-search algorithm, which can be used to find the a good SP-DTC under different models for quantization of transform coefficients. When used with the RD-optimal WZ quantization model [8], this algorithm can potentially locate the optimal SP-DTC for Gaussian sources. In practice, with constraints imposed on tree-search complexity, the algorithm yields a near-optimal solution. We refer to the optimal solution to the Gaussian problem as the source-split DKLT (SP-DKLT). Using this algorithm, we numerically compute the rate-region achievable with a SP-DKLT code for two examples of jointly Gaussian vector sources. This study shows that, when there is sufficient inter-source correlation, optimal SP-DKLT codes can achieve substantially better performance than independent transform codes for the two sources. However, we find that the rates achievable with SP-DKLT codes are strictly inside the optimal achievable rate-region predicted by the DKLT algorithm of [8]. In order to approach the performance predicted by the optimal SP-DKLT in practice, block WZ quantization of transform coefficients is required. For implementation of block WZ quantizers, we consider the use of trellis-coded quantization (TCQ) followed by SW coding. This two stage approach is known to achieve the RD function of Gaussian WZ coding [13]. In order practically implement this approach, we introduce the idea of designing conditional entropy constrained TCQ (CEC-TCQ) based on analytically found bit-allocations. We present experimental results to demonstrate that practical implementations of SP-DTCs for Gaussian sources can closely approach the performance limits indicated by the optimal SP-DKLT. On the other hand when SW coded high-rate scalar quantization model [5] is assumed for encoding transform coefficients, the tree-search algorithm proposed in this article can also be used to find asymptotically good SP-DTC codes for scalar quantization based implementations. These codes can be readily implemented using CEC scalar quantizers (CEC-SQ) as demonstrated by experimental studies presented in this article. In our experimental study, we also investigate the design of good SP-DTCs based on widely used discrete cosine transform (DCT).

This article is organized as follows. Section 2 presents a review of WZ-TC of Gaussian vectors and motivates the particular approach introduced in this article. Section 3 presents the idea of SP-DTC and develops the tree-search algorithm for finding the optimal transforms and the bit-allocation for SP-DKLT codes. Section 4 computes the achievable rate region of SP-DKLT codes for two example Gaussian source models, and presents experimental results obtained by designing SP-DTCs based on both KLT and DCT. Finally, some concluding remarks are given in Section 5.

Notation: As usual, bold letters denote vectors and matrices, upper case denotes random variables, and lower case denotes realizations. Σ _X denotes the auto-covariance matrix of the vector X. Σ _XY and Σ _X|Y , respectively denote the joint covariance matrix of (X, Y) and the conditional covariance matrix of X given Y. The eigenvalues λ₁ , . . ., λ _M of a M × M covariance matrix are always indexed such that λ₁≥ λ₂ . . . ≥ λ _M , and the corresponding KLT matrix has the structure \mathbf{T}=\left({\mathbf{u}}_1^T,\dots ,{\mathbf{u}}_M^T\right), where u _m is the eigenvector associated λ _m .

2 WZ-TC of Gaussian vectors

Consider encoding of a Gaussian vector \mathbf{X}\in {ℝ}^{M_1} using B bits per vector, given that the decoder has access to a jointly Gaussian vector \mathbf{Y}\in {ℝ}^{M_2}. Assume that both vectors have zero mean, and let the auto-covariance matrix of X be Σ _X = E{XX^T}. In WZ-TC, a linear transform is first applied to X and each component of the transform coefficient vector U = T^TX is separately compressed by a WZ quantizer, considering Y as decoder side-information, where T is a M₁ × M₁ unitary matrix. Let Û be the quantized value of U. The decoder then estimates the source vector based on Û and Y. We wish to find the optimal transform and the allocation of B bits among M₁ transform coefficients, which minimize the quantization MSE E\left\{{∥\mathbf{X}-\widehat{\mathbf{X}}∥}^2\right\}, where \widehat{\mathbf{X}}=E\left\{\mathbf{X}|\widehat{\mathbf{U}},\mathbf{Y}\right\} is the optimal estimate (at the decoder) of the source vector. The solution of this problem requires an analytical model for coefficient quantization. To this end, [8] considers RD optimal WZ quantization (RD-WZQ) model, the solution based on which is appropriate for practical block quantization techniques such as TCQ. On the other hand [5] considers SW-coded high-rate scalar quantization (SWCHRSQ) model.

Let the eigenvalues of the conditional covariance matrix Σ_X|Y= E{XX^T|Y} be \mathit{\lambda }={\left({\lambda }_1,\dots ,{\lambda }_{M_1}\right)}^T. The CKLT of X given Y is defined as θ= T^TX, where T is M₁ × M₁ unitary matrix such that Σ _X|Y = TΛT^T, where \mathbf{\Lambda }=\mathsf{\text{diag}}\left({\lambda }_1,{\lambda }_2,...,{\lambda }_{M_1}\right)[8]. It is easy to verify that E{θθ^T|Y} = Λ, i.e., the components of θ are conditionally uncorrelated, given Y. For convenience, define

d^{*}\left(\mathit{\lambda },B,N\right)\triangleq {\left(\prod _{m=1}^N{\lambda }_m\right)}^{\frac{1}{N}}{2}^{-\frac{2B}{N}},

where N ≤ M₁ is a positive integer.

2.1 RD-WZQ model

When RD-WZQ model is used, the quantization MSE for U _m is given by [14]

d_m^{\mathsf{\text{RD}}-\mathsf{\text{WZQ}}}={\lambda }_m{2}^{-2r_m},

(1)

where λ _m = var(U _m | Y) and var(·|·) denotes the conditional variance. The optimal solution to the WZ-TC problem under RD-WZQ model is given by the following theorem.

Theorem 1 Given jointly Gaussian X and Y as defined above, and a total bit budget of B bits, if each transform coefficient U _m , m = 1, . . ., M₁, where U = T^TX, is quantized by an RD optimal WZ quantizer which uses Y as decoder side-information, then the transform T which minimizes E{∥\mathbf{X}-\widehat{\mathbf{X}}∥}^2 is the CKLT of X given Y, and the number of bits allocated to quantizing U _m is

{\rho }_m\left(\mathit{\lambda },B,N\right)=\left\{\begin{array}{cc}\frac{1}{2}{log}_2\left[\frac{{\lambda }_m}{d^{*}\left(\mathit{\lambda },B,N\right)}\right]\hfill & m=1,\dots ,N\hfill \\ 0\hfill & m=N+1,\dots ,M\hfill \end{array}\right.

(2)

where N ≤ M is the largest integer for which λ _m ≥ d^∗ (λ, B, N), m = 1, . . ., N. The quantization MSE of the mth coefficient is

\begin{array}{c}{d}_m={\lambda }_m{2}^{-2{\rho }_m}\hfill \\ =\left\{\begin{array}{cc}{d}^{*}\left(\mathit{\lambda },B,N\right),\hfill & m=1,\dots ,N\hfill \\ {\lambda }_m,\hfill & m=N+1,\dots ,M\hfill \end{array}\right.\hfill \end{array}

(3)

and the overall MSE is

E\left\{{∥\mathbf{X}-\widehat{\mathbf{X}}∥}^2\right\}=N{d}^{*}\left(\mathit{\lambda },B,N\right)+\sum _{m=N+1}^{M_1}{\lambda }_m.

(4)

Proof 1 Directly follows from [[8], Section III-B].

Note that RD-WZQ model implies infinite-dimensional VQ of each coefficient and hence the above MSE is the OPTA in the Gaussian WZ-TC problem.

2.2 SWC-HRSQ model

The asymptotically (in rate) optimal solution to the WZ-TC problem under SWC-HRSQ model is given by the following theorem.

Theorem 2 Let X, Y, and B be as in Theorem 1. If each transform coefficient U _m , m = 1, . . ., M₁ , where U = T^TX, is quantized by a high-rate scalar quantizer and the quantizer output is encoded by a SW code which uses Y as decoder side-information, then the transform T which asymptotically minimizesE{∥\mathbf{X}-\widehat{\mathbf{X}}∥}^2is the CKLT of X given Y and the bit allocation is given by (2). Furthermore, the asymptotically optimal quantizer for U _m , m = 1, . . ., N is a uniform quantizer with step-size \mathrm{\Delta }=\sqrt{\left(2\pi e\right)d^{*}\left(\mathit{\lambda },B,N\right)}. The resulting quantization MSE is

E\left\{{∥\mathbf{X}-\widehat{\mathbf{X}}∥}^2\right\}=\left(\frac{\pi e}{6}\right)N{d}^{*}\left(\mathit{\lambda },B,N\right)+\sum _{m=N+1}^{M_1}{\lambda }_m.

(5)

Proof 2 See Section "Proof of Theorem 2" in Appendix 1.

2.3 Sufficiency of scalar side-information

The WZ quantizers with vector-valued decoder side-information as considered in Theorem 1 are difficult to design in practice. However, the following theorem establishes that when CKLT is used and RD-WZQ model applies for quantization of coefficients, a linear transformation of the side-information vector can be used to convert the vector side-information problem into an equivalent scalar side-information problem. Furthermore, [[5], Section 6] shows that this result applies in asymptotic sense to the SWC-HRSQ model as well.

Theorem 3 Let the mean-zero vectors\mathbf{X}\in {ℝ}^{M_1}and\mathbf{Y}\in {ℝ}^{M_2}be jointly Gaussian, and let T be the CKLT of X given Y. Suppose that transform coefficients U _m , m = 1, . . ., M₁, where U = T^TX, are each compressed by an RD optimal WZ quantizer relative to decoder side-information Y. Then, the minimum MSE (MMSE) estimate ũ _m ( y ) = E{U _m | y} of U _m given Y = y is a sufficient statistic for decoder side-information for quantizing U _m .

Proof 3 See Section "Proof of Theorem 3" in Appendix 1.

Wyner-Ziv transform coding is a special case of more general MT-TC where two or more terminals apply TC to their respective inputs and transmit the quantized outputs to a single decoder which exploits the inter-source correlation to jointly reconstruct all the sources. In this case, the problem is to optimally allocate a given bit budget among all the terminals such that the total MSE is minimized. However, the closed-from solution to this problem appears difficult, due to the inter-dependence of the encoders in different terminals. An iterative descent algorithm is given in [8] for solving the Gaussian MT-TC problem. Given a total bit-budget, the bit-rate of the system is incremented by a small amount in each iteration, and the optimal WZ-TC for each terminal is determined by fixing the encoders of all other terminals and considering their outputs as decoder-side information. The solution that gives the MMSE is accepted and the iterations are repeated until the total bit-budget is exhausted. While this algorithm, referred to as the DKLT algorithm, is guaranteed to converge to at least a locally optimal solution, there is no tractable way to implement the quantizers implied by the final solution since it is not practical to optimize a set of near-optimal WZ quantizers in each iteration of this algorithm. Note also that, DKLT requires joint decoding of two vector sources.

3 Source-splitting based distributed TC

In general, designing a multi-terminal VQ is more difficult than designing a WZ-VQ, due to the mutual dependence among the encoders. However, one could use WZ-VQs to realize a multi-terminal VQ by using source-splitting [12]. It is known that in the quadratic-Gaussian case, source-splitting can be used to realize any rate-pair in the achievable rate-region by only using ideal WZ-VQs which correspond to the corner-points of the achievable rate-region [[12], Section V-C], [15]. While the same optimality properties cannot be claimed for source-splitting by linear transforms, the aforementioned observation still provides us the motivation to take a similar approach in practically realizing the DTCs which can operate at arbitrary rates, by using only WZ quantizers.

A block diagram of the SP-DTC system is shown in Figure 1. Let the total number of bits available for encoding two jointly Gaussian vectors {\mathbf{X}}_1\in {ℝ}^{M_1} and {\mathbf{X}}_2\in {ℝ}^{M_2} be B bits. The terminal 1 performs source splitting by providing an approximation {\mathbf{Y}}_1^{\prime }\in {ℝ}^{N_1^{\prime }} of X₁ at the rate bits/vector as decoder side-information for WZ coding of terminal 2, where N_1^{\prime }\le M_1. In a TC framework, the goal is to provide the best (in MMSE sense) linear approximation of X₁ as the decoder side-information. Therefore, {\mathbf{Y}}_1^{\prime } is the B_1^{\prime }-bit approximation of a linear projection {\mathbf{U}}_1^{\prime }={\mathbf{T}}_1^{\prime T}{\mathbf{X}}_1, where {\mathbf{T}}_1^{\prime } is a M₁ × M₁ unitary matrix. Given the side-information {\mathbf{Y}}_1^{\prime } at the decoder, the terminal 2 quantizes a linear projection {\mathbf{U}}_2={\mathbf{T}}_2^T{\mathbf{X}}_2 of X₂ using bits/vector, where T₂ is a M₂ × M₂ unitary matrix. Let {\mathbf{Y}}_2\in {ℝ}^{N_2}, be the quantized value of U₂, where N₂≤ M₂. Then, given the quantized linear projections of both X₁ and X₂ available at the decoder, the terminal 1 quantizes the a linear projection {\mathbf{U}}_1^{″}={\mathbf{T}}_1^{″T}{\mathbf{X}}_1 of X₁ using B_1^{″}=B-B_1^{\prime }-B_2 bits/vector, where {\mathbf{T}}_1^{″} is a M₁ × M₁ unitary matrix. Let {\mathbf{Y}}_1^{″}\in {ℝ}^{N_1^{″}} be the quantized value of {\mathbf{U}}_1^{″}, where N_1^{″}\le M_1. In the receiver, each source vector is reconstructed by a WZ decoder. The MMSE optimal reconstructions for X₁ and X₂ are, respectively given by {\widehat{\mathbf{X}}}_1=E\left\{{\mathbf{X}}_1|{\mathbf{Y}}_1^{″},\mathbf{V}\right\} and {\widehat{\mathbf{X}}}_2=E\left\{{\mathbf{X}}_2|\mathbf{V}\right\}, where \mathbf{V}={\left({\mathbf{Y}}_1^{\prime }{\mathbf{Y}}_2\right)}^T. The total transmission rate for source X₁ is thus B_1=B_1^{\prime }+B_1^{″} bits/vector. The rates used by terminals 1 and 2 in bits/sample are given by R₁ = B₁/M₁ and R₂ = B₂/M₂, respectively.

Figure 1

The proposed source-split transform coding (SP-DTC) system for two-terminal distributed quantization of two correlated vectors.

Let {\mathbf{U}}_1^{\prime }={\left(U_{1,1}^{\prime },\dots ,U_{1,M_1}^{\prime }\right)}^T, {\mathbf{U}}_1^{″}={\left(U_{1,1}^{″},\dots ,U_{1,M_1}^{″}\right)}^T and {\mathbf{U}}_2={\left(U_{2,1},\dots ,U_{2,M_2}\right)}^T. Also, let the bit-rates allocated to quantizing these transform coefficients be {\mathbf{r}}_1^{\prime }={\left(r_{1,1}^{\prime },\dots ,r_{1,M_1}^{\prime }\right)}^T, {\mathbf{r}}_1^{″}={\left(r_{1,1}^{″},\dots ,r_{1,M_1}^{″}\right)}^T, and {\mathbf{r}}_2={\left(r_{2,1},\dots ,r_{2,M_2}\right)}^T respectively, and define \mathbf{r}={\left({\mathbf{r}}_1^{\prime },{\mathbf{r}}_1^{″},{\mathbf{r}}_2\right)}^T. Given a total of B bits for encoding both X₁ and X₂, the design of a SP-DTC involves determining the values of the transforms {\mathbf{T}}_1^{\prime }, {\mathbf{T}}_1^{″}, and {\mathbf{T}}_2^{″}, and a bit allocation among the transform coefficients \mathbf{U}={\left({\mathbf{U}}_1^{\prime },{\mathbf{U}}_1^{″},{\mathbf{U}}_2\right)}^T such that the total MSE

D\left(\mathbf{r}\right)=E\left\{{∥{\mathbf{X}}_1-{\widehat{\mathbf{X}}}_1∥}^2+{∥{\mathbf{X}}_2-{\widehat{\mathbf{X}}}_2∥}^2\right\}

(6)

is minimized. By writing the quantization MSEs of U_{1,i}^{″} and U_2,jas d_{1,i}^{″}\left(r_{1,i}^{″},{\mathbf{r}}_1^{\prime },{\mathbf{r}}_2\right), and d_{2,j}\left(r_{2,j},{\mathbf{r}}_1^{\prime }\right), respectively, i = 1, . . ., M₁, j = 1, . . ., M₂, the total MSE can be expressed as

D\left(\mathbf{r}\right)=\sum _{i=1}^{M_1}{d}_{1,i}^{″}\left(r_{1,i}^{″},{\mathbf{r}}_1^{\prime },{\mathbf{r}}_2\right)+\sum _{j=1}^{M_2}{d}_{2,j}\left(r_{2,j},{\mathbf{r}}_1^{\prime }\right).

(7)

The bit allocation problem can now be stated as follows:

Given a total bit-budget of B bits

\underset{\mathbf{r}}{min}D\left(\mathbf{r}\right)

(8)

subject to

\begin{array}{c}\sum _{m=1}^{2M_1+M_2}{r}_m\le B,\hfill \\ r_m\ge 0,m=1,\dots ,2M_1+M_2,\hfill \end{array}

where \mathbf{r}={\left(r_1,\dots ,r_{2M_1+M_2}\right)}^T. The explicit solution of this minimization problem is unfortunately intractable due to the inter-dependence of the three transform codes involved.

However, an explicit solution can be found for a variant of this problem obtained by fixing B_1^{\prime }, B_1^{″} and B₂, so that the number of bits allocated to each transform code is fixed and it is only required to optimize the bit allocation among the quantizers within each transform code. For simplicity, we refer to this problem as the constrained bit-allocation problem. In the following, an explicit solution to this problem is derived. Based on the result, we then present a tree-search algorithm to solve the unconstrained problem (8). Under both RD-WZQ and SWC-HRSQ models, the optimal transforms for Gaussian sources are CKLTs. Therefore, we refer to the solution to problem (8) as the SP-DKLT.

3.1 Solution to the constrained bit-allocation problem

3.1.1 RD optimal quantization

Let B_1^{\prime }, B_1^{″}, and B₂ be fixed in Figure 1 and let the coefficient quantization be represented by the RD-WZQ model (for components of {\mathbf{U}}_1^{\prime }, this reduces to the non-distributed RD-optimal quantization). From Theorem 1, it follows that the MMSE optimal transform (in the sense of providing the best linear approximation as decoder side information for terminal 2) {\mathbf{T}}_1^{\prime } is the KLT of X₁. Let the eigenvalues of {\sum }_{X_1} be {\mathit{\lambda }}_1^{\prime }=\left({\lambda }_{1,1}^{\prime },\dots ,{\lambda }_{1,M_1}^{\prime }\right). Then, using (2), the optimal bit allocation for U_{1,m}^{\prime } can be given by

r_{1,m}^{\prime }={\rho }_m\left({\mathit{\lambda }}_1^{\prime },B_1^{\prime },N_1^{\prime }\right),m=1,\dots ,M_1.

(9)

for some N_1^{\prime }\le M_1. Let the quantized value of U_{1,m}^{\prime } be {\mathit{Û}}_{1,m}^{\prime }. We note that E{U}_{1,m}^{\prime 2}={\lambda }_{1,m}^{\prime } for m = 1, . . ., M₁ and E{\left(U_{1,m}^{\prime }-{\mathit{Û}}_{1,m}^{\prime }\right)}^2=d^{*}\left({\mathit{\lambda }}_1^{\prime },B_1^{\prime },N_1^{\prime }\right) for m=1,\dots ,N_1^{\prime }. In RD theoretic sense, the quantized value (up to a scaling factor) of the mean-zero Gaussian variable U_{1,m}^{\prime } can be given by Y_{1,m}^{\prime }=U_{1,m}^{\prime }+Z_{1,m}^{\prime }, m=1,\dots ,N_1^{\prime }, where Z_{1,m}^{\prime } is a mean-zero Gaussian random-variable independent of U_{1,m}^{\prime } such that

E{Z}_{1,m}^{\prime 2}=\frac{{\lambda }_{1,m}^{\prime }{d}^{*}\left({\mathit{\lambda }}_1^{\prime },B_1^{\prime },N_1^{\prime }\right)}{{\lambda }_{1,m}^{\prime }-d^{*}\left({\mathit{\lambda }}_1^{\prime },B_1^{\prime },N_1^{\prime }\right)},

(10)

see [[16], Section 10.3.2]. Therefore, we can write

{\mathbf{Y}}_1^{\prime }={\mathbf{K}}_1^{\prime \left(N_1^{\prime }\right)T}{\mathbf{X}}_1+{\mathbf{Z}}_1^{\prime },

(11)

where {\mathbf{K}}_1^{\prime \left(N_1^{\prime }\right)} denotes the M_1\times N_1^{\prime } matrix consisting of first N_1^{\prime } columns of {\mathbf{K}}_1^{\prime }. The covariance matrix of "quantization noise" vector {\mathbf{Z}}_1^{\prime }={\left(Z_{1,1}^{\prime },\dots ,Z_{1,N_1^{\prime }}^{\prime }\right)}^T is given by {\sum }_{Z_1^{\prime }}=\mathsf{\text{diag}}\left(Z_{1,1}^{\prime },\dots ,Z_{1,N_1^{\prime }}^{\prime }\right). According to (11), {\mathbf{Y}}_1^{\prime } and X₁ are jointly Gaussian, and it follows that

{\sum }_{Y_1^{\prime }}={\mathbf{K}}_1^{\left(N_1^{\prime }\right)T}{\sum }_{X_1}{\mathbf{K}}_1^{\left(N_1^{\prime }\right)}+{\sum }_{Z_1^{\prime }}.

(12)

Furthermore, X₂ and {\mathbf{Y}}_1^{\prime } are jointly Gaussian with the conditional covariance matrix

{\sum }_{X_2|Y_1^{\prime }}={\sum }_{X_2}-{\sum }_{X_1{X}_2}^T{\mathbf{K}}_1^{\left(N_1^{\prime }\right)}{\sum }_{Y_1^{\prime }}^{-1}{\mathbf{K}}_1^{\left(N_1^{\prime }\right)T}{\sum }_{X_1{X}_2}.

(13)

Next consider TC X₂ given {\mathbf{Y}}_1^{\prime } as decoder side-information. From Theorem 1, it follows that the E{∥{\mathbf{X}}_2-{\widehat{\mathbf{X}}}_2∥}^2 is minimized by choosing T₂ as the CKLT of X₂ given {\mathbf{Y}}_1^{\prime } and by applying RD optimal WZ quantization to each element of {\mathbf{U}}_2={\mathbf{T}}_2^T{\mathbf{X}}_2 given decoder side information {\mathbf{Y}}_1^{\prime } based on a bit-allocation specified by the eigenvalues of {\sum }_{X_2|Y_1^{\prime }}, {\mathit{\lambda }}_2=\left({\lambda }_{2,1},\dots ,{\lambda }_{2,M_2}\right). The optimal bit allocation for U_2,mis given by

r_{2,m}={\rho }_m\left({\mathit{\lambda }}_2,B_2,N_2\right),m=1,\dots ,M_2,

(14)

for some N₂≤ M₂. The resulting MSE is given by

D_2\left(B_1^{\prime },B_2\right)=E{∥{\mathbf{X}}_2-{\widehat{\mathbf{X}}}_2∥}^2=N_2{d}^{*}\left({\mathit{\lambda }}_2,B_2,N_2\right)+\sum _{m=N_2+1}^{M_2}{\lambda }_{2,m}.

(15)

As before, the quantized value of U₂ up to a scaling factor, can be represented by [[8], Theorem 3],

{\mathbf{Y}}_2={\mathbf{K}}_2^{\left(N_2\right)T}{\mathbf{X}}_2+{\mathbf{Z}}_2,

(16)

where {\mathbf{K}}_2^{\left(N2\right)} denotes the M₂ × N₂ matrix consisting of first N₂ columns of K₂ and the quantization noise {\mathbf{Z}}_2\in {ℝ}^{N_2} is a mean zero iid Gaussian vector independent of X₂. The covariance matrix of Z₂ is given by {\sum }_{Z_2}=\mathsf{\text{diag}}\left(Z_{2,1},\dots ,Z_{2,N_2}\right), where E{Z}_{2,m}^2, m = 1, . . ., N₂ are given by

E{Z}_{2,m}^2=\frac{{\lambda }_{2,m}{d}^{*}\left({\mathit{\lambda }}_2,B_2,N_2\right)}{{\lambda }_{2,m}-d^{*}\left({\mathit{\lambda }}_2,B_2,N_2\right)}.

(17)

The covariance matrix of Y₂ is

{\sum }_{Y_2}={\mathbf{K}}_2^{\left(N_2\right)T}{\sum }_{X_2}{\mathbf{K}}_2^{\left(N_2\right)}+{\sum }_{Z_2}.

(18)

Therefore X₁ and \mathbf{V}={\left({\mathbf{Y}}_1^{\prime }{\mathbf{Y}}_2\right)}^T are jointly Gaussian with the cross-covariance matrix

{\sum }_{X_1|V}={\sum }_{X_1}-{\sum }_{V{X}_1}^T{\sum }_V^{-1}{\sum }_{V{X}_1},

(19)

where

{\sum }_{V{X}_1}=\left(\begin{array}{c}\hfill {\mathbf{K}}_1^{\left(N_1^{\prime }\right)}{\sum }_{X_1}\hfill \\ \hfill {\sum }_{X_2}{\sum }_{X_2{X}_1}\hfill \end{array}\right)

and

{\sum }_V=\left(\begin{array}{cc}{\sum }_{Y_1^{\prime }}\hfill & {\mathbf{K}}_1^{\left(N_1^{\prime }\right)}{\sum }_{X_1{X}_2}{\sum }_{X_2}^T\hfill \\ {\sum }_{X_2}{\sum }_{X_2{X}_1}{\mathbf{K}}_1^{\left(N_1^{\prime }\right)T}\hfill & {\sum }_{Y_2}\hfill \end{array}\right).

Finally, consider quantizing X₁, given \mathbf{V}={\left(\begin{array}{cc}\hfill {\mathbf{Y}}_1^{\prime }\hfill & \hfill {\mathbf{Y}}_2\hfill \end{array}\right)}^T\in {ℝ}^{N_1^{\prime }+N_2} as the decoder side-information. As before, E{∥{\mathbf{X}}_1-{\widehat{\mathbf{X}}}_1∥}^2 is minimized by choosing {\mathbf{T}}_1^{″} as the CKLT of X₂ given V, and RD optimal WZ quantization of each element of {\mathbf{U}}_1^{″}={\mathbf{T}}_1^{″T}{\mathbf{X}}_1 given V, based on a bit-allocation specified by the eigenvalues of {\sum }_{X_1|V}, {\mathit{\lambda }}_1^{″}={\left({\lambda }_{1,m}^{″},\dots ,{\mathit{\lambda }}_{M_1}^{″}\right)}^T. The bit rate allocated to quantizing U_{1,m}^{″} given by

r_{1,m}^{″}={\rho }_m\left({\mathit{\lambda }}_1^{″},B_1^{″},N_1^{″}\right),m=1,\dots ,M_1,

(20)

for some N_1^{″}\le M1 and the resulting MSE is

D_1\left(B_1^{\prime },B_1^{″},B_2\right)=E{∥{\mathbf{X}}_1-{\widehat{\mathbf{X}}}_1∥}^2=N_1^{″}{d}^{*}\left({\mathit{\lambda }}_1^{″},B_1^{″},N_1^{″}\right)+\sum _{m=N_1^{″}+1}^{M_1}{\lambda }_{1,m}^{″}.

(21)

Given a rate tuple \left(B_1^{\prime },B_1^{″},B_2\right), the MMSE achievable with a SP-DKLT code for two jointly Gaussian vectors is given by

D\left(B_1^{\prime },B_1^{″},B_2\right)=D_1\left(B_1^{\prime },B_1^{″},B_2\right)+D_2\left(B_1^{\prime },B_2\right).

(22)

3.1.2 High-resolution scalar quantization and SW coding

Due to Theorem 2, the expressions for bit-allocations given by (9), (14) and (20) applies to SWC-HRSQ model as well. However, the resulting MSE and hence the quantization noise variances are different to those of the RD-WZQ model. More specifically, since the decoder side-information in a SP-DTC depends on quantization noise of the other terminals, the optimal transforms T₂ and {\mathbf{T}}_1^{″} and the associated bit allocations obtained with the SWC-HRSQ model are different to those obtained with the RD-WZQ model. In order to make the problem tractable, we assume that the quantization noise of SWC-HRSQ model also follows (11) and (16) (for a discussion on the validity of this assumption, see [17]). This assumption essentially allows us to compute the conditional covariance matrices {\sum }_{X_2|Y_1^{\prime }} and {\sum }_{X_1|V} as in the previous case, and then apply (9), (14) and (20) to find the optimal transforms and the bit-allocations. However, due to (27), the quantization noise variance in (10) in this case is given by

E{Z}_{1,m}^{\prime 2}=\frac{\left(\pi e/6\right){\lambda }_{1,m}^{\prime }{d}^{*}\left({\mathit{\lambda }}_1^{\prime },B_1^{\prime },N_1^{\prime }\right)}{{\lambda }_{1,m}^{\prime }-\left(\pi e/6\right)d^{*}\left({\mathit{\lambda }}_1^{\prime },B_1^{\prime },N_1^{\prime }\right)}.

(23)

A similar expression exists for the quantization noise variance in (17).

3.2 A tree-search solution to the unconstrained bit-allocation problem

We note that the optimal solution to the unconstrained problem defined in (8) corresponds to the MMSE solution of the constrained problem over the set of rate-tuples \mathcal{S}=\left\{\left(B_1^{\prime },B_1^{″},B_2\right):B_1^{\prime }\in \left(0,B\right),B_1^{″}\in \left(0,B\right),B_2\in \left(0,B\right),B_1^{\prime }+B_1^{″}+B_2\le B\right\}. This set is shown in Figure 2. One approach to locating the MMSE solution is to search over an appropriately discretized grid of points inside . As we will see, even though an exhaustive search on a fine grid can be prohibitively complex, a much simpler constrained tree-search algorithm exists which can be used to locate the required solution with a very high probability.

Figure 2

The solution space for the unconstrained bit-allocation problem. The tree-search algorithm uses a search-grid of regularly spaced points (i.e., a cubic lattice) with a separation of ΔB inside .

The proposed algorithm is a generalization of a class of bit-allocation algorithms in which a small fraction ΔB of the total bit-budget B is allocated to the "most deserving" quantizer among a set of quantizers in an incremental fashion, until the entire bit-budget is exhausted [[4], Section 8.4]. Unfortunately, this type of a greedy search cannot guarantee that the final solution is overall optimal and can yield poor results in our problem where the bit allocation among three sets of dependent quantizers must be achieved. On the other hand, if the increment ΔB is chosen small enough, a near-optimal solution can be found by resorting to a tree-search. Even though a full tree-search is intractable, a simple algorithm referred to as the (M, L)-algorithm[18] exists for detecting the minimum cost path in the tree with a high probability. We use this insight to formulate a tree-search algorithm for solving the unconstrained bit allocation problem, in which a set of constrained bit allocation problems are solved in each iteration.

In order to describe the proposed tree-search algorithm in detail, let ΔB be the incremental amount of bits to be allocated in each step of the search, where 0 < ΔB ≪ B. The algorithm is initialized by setting \left(B_1^{\prime },B_1^{″},B_2\right)=\left(0,0,0\right), i.e., the origin in Figure 2. Now if we are to allocate ΔB bits to only one of the three transform codes {\mathbf{T}}_1^{\prime }, {\mathbf{T}}_1^{″} or T₂, then there are three possible choices for the rate-tuple \left(B_1^{\prime },B_1^{″},B_2\right), namely (ΔB, 0, 0), (0, ΔB, 0), and (0, 0, ΔB). For each of these choices, we can explicitly solve the constrained bit allocation problem as described in the previous section and find the MMSE solution. Each of these candidate solutions can be viewed as a node in a tree as shown in Figure 3. The root node of the tree corresponds to a SP-DTC of rate 0, and a node in the first level of nodes obtained in the first iteration of the algorithm corresponds to a SP-DTC of rate of ΔB bits per source pair (X₁, X₂). In the second iteration of the algorithm, we allocate ΔB more bits to each of the three candidate SP-DTCs (but to one of the SP-DTCs at a time) in the first level of nodes. Note that, for each SP-DTC we can allocate ΔB bits in three different ways, i.e., ΔB bits can be added to either B_1^{\prime }, B_1^{″}, or B₂. This requires the solution of three constrained bit allocations problems for each of the 3 nodes in the first-level. As a result, the tree will be extended to a second level of 3² nodes, in which each node corresponds to a SP-DTC of 2ΔB bits per source-pair, as shown in Figure 3. We can repeat this procedure, allocating ΔB bits to each of terminal node of the tree in a given iteration, until all B bits are exhausted. After the final iteration, the tree would consist of L = ⌈B/ΔB⌉ levels with 3^Lnodes in the last level (terminal nodes). Each terminal node corresponds to a candidate SP-DTC of rate B, and rate-tuples of these SP-DTCs lie on the plane B_1^{\prime }+B_1^{″}+B_2=B in Figure 2. The MMSE terminal node of the tree is the optimal solution to the unconstrained bit allocation problem, provided that the latter solution is on the search-grid. If ΔB is chosen small enough, then we can ensure that the optimal solution is nearly on the search-grid. Suppose that, in each iteration, the algorithm saves the MSE of the solution to the constrained bit-allocation problem associated with each node. In theory, the optimal solution can be found by an exhaustive tree-search, using the MSE of a node [given by (22)] as the path-cost. In order to practically implement the tree-search, we use the (M, L)-algorithm, in which the parameter M can be chosen to reduce the complexity at the expense of decreased accuracy (i.e., the probability of detecting the lowest cost path in the tree). In the (M, L) algorithm [[18], p. 216] for a tree of depth L, one only retains the M best (lowest MSE) nodes in each iteration. When M = 1 we have a completely greedy search. On the other hand when M _n = 3ⁿin the n th iteration, we have a full-tree search which has a complexity that grows exponentially with the iteration number. When M (1 ≤ M ≤ 3^L) is a prescribed constant, the complexity is linear in M, independent of the number of iterations. In obtaining the simulation results presented in Section 4, M = 27 and L = 135 (ΔB = 0.2) were found to be sufficient to obtain near optimal results. For example, it was observed that even for M = 81 and L = 405, nearly the same result was obtained.

Figure 3

Bit allocation tree with the values of the rate-tuple \left(B_1^{\prime },B_1^{″},B_2\right) after two iterations of the tree-search algorithm.

4 Numerical results and discussion

Source model A: Let the components of X₁ be M₁ consecutive samples of a first-order Gauss-Markov process with a unit-variance and the correlation coefficient |ρ| < 1, i.e., X_1,m= ρX_1,(m-1)+ Z _m , m = 2, . . ., M₁, where Z _m , m = 1, . . ., M₁ are mean-zero iid Gaussian variables such that E{Z}_m^2=1-{\rho }^2. The auto-covariance matrix {\sum }_{X_1} is a Teoplitz matrix with the first row \left(1,\rho ,{\rho }^2,\dots ,{\rho }^{M_1-1}\right). Now define the components of X₂ to be noisy observations of the components of X₁, i.e., X_2,m= γX_1,m+ W _m , where |γ| < 1 and W _m is a mean-zero, iid Gaussian variable with E{W}_m^2=1-{\gamma }^2, m = 1, . . ., M₁ (M₂ = M₁). It follows that, {\sum }_{X_2} is a Teoplitz matrix with the first row \left(1,{\gamma }^2\rho ,{\gamma }^2{\rho }^2,\dots ,{\gamma }^2{\rho }^{M_1-1}\right). Furthermore, the cross-covariance matrix {\sum }_{X_1{X}_2}={\gamma }^2{\sum }_{X_1}. Note that X₁ and X₂ are not statistically similar and the components of X₁ are more correlated than those of X₂.

Source model B: Consider a spatial Gaussian random field in which the correlation function decays with distance d according to the squared exponential model [19]. We define the random vectors X₁ and X₂ to be observations picked-up by a pair of sensor arrays placed in this random filed. In this case, the auto-covariance matrix of X₁ is given by {\left[{\sum }_{X_1}\right]}_{ij}=exp\left\{-{\left(\alpha d_{ij}\right)}^2\right\}, where α is a constant and d _ij is the distance between X_1,iand X_1,j. The auto-covariance matrix of X₂ also has a similar form. For simplicity assume that the sensors in each array are placed on a M × M square grid of unit spacing (i.e., M₁ = M₂ = M²), the two arrays are on parallel planes separated by a distance r, and the two grids are aligned so that the distance between X_1iand X_2iis r for all i. With this setup, the distance between X_1iand X_2jis \sqrt{d_{ij}^2+r^2}, and the cross-covariance matrix is given by {\left[{\sum }_{X_1{X}_2}\right]}_{ij}=\theta exp\left\{-{\left(\alpha d_{ij}\right)}^2\right\}, where θ = exp {- (αr)²}. This sensor structure ensures that X₁ and X₂ are statistically similar. However, {\sum }_{X_1{X}_2} can be chosen independently (by choosing array separation r) of {\sum }_{X_1} and {\sum }_{X_2}.

4.1 RD performance

We compute the rate-pairs (R₁, R₂) achievable with a SP-DKLT code for a given a total MSE D, by fixing R₁ (or R₂) and then searching for minimum R₂ (or R₁) required to achieve the MSE D [given by (6)]. The rate-pairs achievable for D = 0.01 with SP-DKLT coding of vectors from source model A with ρ = 0.9, γ = 0.9, are plotted in Figure 4. These values of ρ and γ result in a source cross-covariance matrix with the largest element 0.9. In Figure 4, the curve "SP-DKLT (X₁ split)" corresponds to a system in which the input to terminal 1 (which applies source splitting) is X₁ as shown in Figure 1. The curve "SP-DKLT (X₂ split)" corresponds to a system in which the input to the terminal 1 is X₂. Note that the two curves are not symmetric in rates and they coincide if R₁ and R₂ are inter-changed in one of the curves. This is because X₁ and X₂ have different auto-covariance matrices, and hence inter-changing the rates is equivalent to inter-changing the terminals. Importantly, this result indicates that when the two sources are not statistically identical, which source is chosen for splitting does not affect the SP-DKLT performance. Table 1 lists the best bit allocations found by the tree-search algorithm for SP-DKLT codes shown in Figure 4. Note that, for the same (B₁, B₂), the rate-split between B_1^{\prime } and B_2^{″} when X₁ is applied to the terminal 1 is not identical to that when X₂ is applied to the terminal 1.

Figure 4

Comparison of rate-regions achievable with different TC approaches in quantizing 16-dimensional vectors ( M ₁ = M ₂ = 16) of source model A ( ρ = 0.9, γ = 0.9). "SP-DKLT (X₂ split)" refers to the case when X₂ is used as the input to terminal 1 (and hence transmitted at rate R₁) which applies source-splitting.

Table 1 Bit allocations found by the tree-search algorithm for 16-dimensional SP-DKLT coding of source model A (ρ = 0.9, γ = 0.9)

Figure 4 also shows the rate region achievable if each source is independently compressed using the KLT (i.e., only intra-vector correlation is utilized), labeled IKLT, and the OPTA lower bound for distributed TC predicted by the iterative DKLT algorithm [8]. The performance of both distributed and non-distributed TC of source model A degrades as ρ decreases, since both auto- and cross-covariance matrices of X₁ and X₂ are functions of ρ. Next consider the source model B for which the lowest achievable rate-pairs corresponding to D = 0.005 are plotted in Figure 5. The source parameter α = 0.32 results in auto-covariance matrices whose largest off-diagonal element is 0.9. Also recall that θ is the largest element in the source cross-covariance matrix. Note that changing θ only affects the cross-covariance matrix, and hence has no effect on the best achievable rates for independent coding of the two sources. On the other hand, as θ increases, the rates achievable with distributed coding do improve. Since in source model B, the two sources are statistically similar, the curves in Figure 5 are symmetric in rates and the optimal bit allocation does not depend on which source is chosen for splitting.

Figure 5

Comparison of rate-regions achievable with different TC approaches in quantizing 16-dimensional vectors ( M ₁ = M ₂ = 16) of source model B for α = 0.32 and different values of θ.

The RD performance in Figures 4 and 5 indicate that SP-DKLT codes can significantly outperform IKLT codes at all rates when there is sufficient correlation between the two distributed sources. The performance of SP-DKLT coding necessarily approaches OPTA (DKLT) bound when either R₁ or R₂ is sufficiently high. That is, the terminal with the higher bit rate can independently transform code its input with negligible distortion, and the other terminal can then apply WZ-TC at the minimum rate achievable with "almost unquantized" decoder side-information. However, for both source-models the rate-region achievable with source-splitting is strictly inside that of DKLT. In other words, there are some rate-pairs inside the DKLT rate-region for a given MSE D, which cannot to be achieved by a SP-DKLT code. A closely related issue is that, for a range of values of (R₁, R₂), the sumrate R₁ + R₂ of SP-DKLT codes remains constant and reaches its minimum. For example, it can be seen from Figure 4 and Table 1 that the sum-rate is about 4.125 bits when the rate of X₁ is in the range 1.375 - 2.5 bits/sample. From Table 1, it can be seen that when the sum-rate is greater than its minimum value, the optimal SP-DKLT code approaches a WZ transform code, i.e., no source-splitting occurs. This situation, which also exists in Figure 5, suggests that optimal SP-DKLT codes for the sum-rate at which a given D can be achieved, are equivalent to time-sharing [12] of two "corner points". Figure 6 illustrates this situation for optimal SP-DKLT codes at D = 0.005 for source model B (θ = 0.9 in Figure 5). It should however be noted that, unlike source-splitting, time-sharing between the two terminals requires synchronization of their encoders [11].

Figure 6

The optimal SP-DKLT codes at D = 0.005 for source model B with α = 0.32 and θ = 0.9. The minimum sum-rate is 2.5 bits/sample, which can also be achieved by time sharing of codes C₁ and C₂.

4.2 Design examples

In this section, we focus on the practical design of SP-DKLT codes for a given pair of rates (R₁, R₂) based on both scalar and block-quantization. RD-WZQ model used in Section 3.1.1 implies infinite block-length WZ-VQ of each coefficient. A practically realizable approach to block WZ quantization is SWC-TCQ [20]. Experimental results obtained with LDPC codes of block length up to 10⁶ bits and TCQs up to 8,192 states are presented in [20] for quadratic Gaussian WZ quantization, which indicate that performance very close to the theoretical limit can be achieved with SWC-TCQ. Motivated by these results, we aim to implement SP-DTCs which can approach theoretical performance predicted in Section 3.1.1 using TCQ and SW coding for encoding transform coefficients. However, the SWC-TCQ design procedure followed in [20] is to first design a TCQ whose MSE satisfies a constraint (by choosing a sufficiently high rate) and then to estimate the output conditional entropy (which is the target rate of the SW code) of the resulting TCQ. This is sufficient for verifying the achievable rate pairs for a given MSE which is the goal of [20]. Our problem is different in that the rate of the SW code is specified by the solution to the bit-allocation problem and our goal is to design a TCQ which minimizes the MSE, subject to a constraint on the output conditional entropy. This requires an alternative formulation of the design procedure, which we refer to as CEC-TCQ. In previous work on non-distributed quantization, entropy constrained TCQ (EC-TCQ) has been investigated in [21–24]. CEC-TCQ is a modification of EC-TCQ in [21, 22] to accommodate block SW-coding of the TCQ output relative to a decoder side-information sequence. Our formulation of CEC-TCQ follows the supersetentropy formulation of EC-TCQ in [22].

Suppose that a sequence of source samples {U _n ∈ℝ} has to be quantized, given that the sequence {Y _n ∈ℝ}, is available at the decoder as side-information, where n = 1, 2, . . . denotes the discrete-time. Similar to an ordinary TCQ [25], a CEC-TCQ uses a size 2^{R_{\mathsf{\text{TCQ}}}+1} scalar codebook to quantize the input sequence U₁, U₂, . . ., into a R_TCQ bits/sample output sequence Û₁, Û₂, . . . . However, CEC-TCQ output satisfies the additional property that the conditional entropy H (Û _n |Y _n ) = E{- log₂P (Û _n |Y _n )} ≤ R for some given R. It follows from [9] that if the CEC-TCQ output is SW-coded relative to the decoder side-information sequence {Y _n } then LR bits are sufficient to (almost) losslessly transmit a sequence of L source samples as L → ∞. The optimal CEC-TCQ minimizes the MSE E{(U _n - Û _n )²}, subject to the constraint E{- log₂P (Û _n |Y _n )} ≤ R, or equivalently, minimizes the Lagrangian

J=\underset{L\to \infty }{lim}\sum _{n=1}^L\left[E\left\{{\left(U_n-{\mathit{Û}}_n\right)}^2\right\}+\beta E\left\{-{log}_{2}P\left({\mathit{Û}}_n|Y_n\right)\right\}\right]

(24)

where β > 0 is the Lagrange multiplier. This implies that, given a specific sequence of input samples u₁, u₂, . . ., the CEC-TCQ encoder should use the Viterbi algorithm based on the path-cost function

\sum _n\left[{\left(u_n-{\mathit{û}}_n\right)}^2+\beta E\left\{-{log}_{2}P\left({\mathit{û}}_n|Y_n\right)\right\}\right].

(25)

In a rate R_TCQ TCQ, each codeword c _k , k=1,\dots ,2^{R_{\mathsf{\text{TCQ}}}+1}, in the codebook is labeled with R_TCQ-bit binary string [25]. Let b _i , i=1,\dots ,2^{R_{\mathsf{\text{TCQ}}}} be these binary labels. Then, to compute (25), the cost βE{- log₂P (b _i |Y)} must also be stored for each binary label, where Y is the random variable representing the decoder side-information. For a fixed value of β, we can use a slight modification of the algorithm in [21] for optimizing the CEC-TCQ codebook, by replacing the codeword entropies E{- log₂P(b _i )} by the conditional entropies E{- log₂P (b _i |Y)}, and by using a training sequence of (U _n , Y _n ) pairs. In order to approximate the expectations by sample averages computed from training data, the side-information variable Y is discretized to \mathit{Ŷ}\in \left\{{\eta }_1,\dots {\eta }_{\mathcal{Y}}\right\}, where is a large enough positive integer. Then E{- log₂P (B = b _i |Y)} can be approximated by

\mathit{Ĥ}\left(b_i\right)=-\sum _{k=1}^{\mathcal{Y}}{log}_2\left[P\left(B=b_i|\mathit{Ŷ}={\eta }_k\right)\right]P\left(\mathit{Ŷ}={\eta }_k|B=b_i\right)

(26)

where B\in \left\{b_1,\dots ,b_{2^{R_{\mathsf{\text{TCQ}}}}}\right\} is the binary-labeled output of the TCQ. Given a TCQ code-book, the probabilities P(B = b _i |Ŷ = η _k ) and P(Ŷ = η _k |B = b _i ) can be estimated using the training set [it is sufficient to estimate p _{i, k} = P(B = b _i , Ŷ = η _k ), k=1,\dots \mathcal{Y},i=1,\dots ,N]. To complete the design, it is necessary to search for the value of β for which E{- log₂P (Û _n |Y _n )} ≈ R by repeating the codebook optimization for an appropriately chosen sequence of β values.

For block WZ-code designs, the transforms and the bit allocations are found by RD-WZQ model (Section 3.1.1) and WZ quantizers are implemented using CEC-TCQ followed by binary SW coding. More specifically, the rate found by the bit allocation algorithm for each transform coefficient is used as the conditional entropy constraint in the design of a CEC-TCQ for that coefficient. As described in Section 2.3, the CEC-TCQ designs are based on scalar-side information obtained by a linear transform of the vector side-information at the decoder, see Theorem 3. All CEC-TCQ designs are based on the 8-state trellis used in JPEG2000 [[26], Figure 3.16]. For trellis encoding and decoding, a sequence length of 256 source samples has been used. For design and testing quantizers, sample sequences of length 5 × 10⁵ have been used. Since, the main focus this paper is the design of transforms and the quantizers, we assume ideal SW coding of the binary output of each CEC-TCQ, so that our results do not depend on any particular SW coding method. In a practical implementation (e.g., [20]), near optimal performance can be obtained by employing a sufficiently long SW code (note that sequence length for SW-coding can be chosen arbitrary larger than the sequence length used for TCQ encoding). This type of coding is well suited for applications such as distributed image compression, where the coding is inherently block-based.

We also consider SP-DKLT code designs based on scalar quantization. In this case, the transforms and bit allocations are found by using the SWC-HRSQ model (Section 3.1.2). While it is possible to use the step-size predicted by SWC-HRSQ model to design uniform quantizers, we found that such quantizers in reality do not satisfy the required entropy constraint at lower rates. We instead use conditional entropy constrained scalar quantizers (CEC-SQ), designed by modifying the algorithm in [27] to accommodate a conditional entropy constraint similar to CEC-TCQ approach above.

The reconstruction signal-to-noise ratio (RSNR) [with MSE as given by (6)] of SP-DKLT code designs for source model B is shown in the rows labeled Design in Table 2 where SP-DKLT/CECSQ and SP-DKLT/CECTCQ refer to scalar quantization and TCQ based designs respectively. The rows labeled Analytical show the performance predicted by the SWC-HRSQ and RD-WZQ models upon which the transforms and bit-allocations are based (note however that the performance predicted by SWC-HRSQ model is not necessarily an upper-bound for CEC-SQ designs which are not constrained to be uniform quantizers). We compare the performance of our SP-DKLT code designs with IKLT codes for which the bit-allocations are obtained by using either entropy coded high-rate quantization model (for scalar quantizer design) [[4], Section 9.9] or RD-optimal quantization model (for block quantizer design) [[16], Section 10.3.3] for Gaussian variables. The IKLT codes with scalar quantization have been implemented by using entropy constrained scalar quantizers (EC-SQ) while those with block quantizers have been implemented by using EC-TCQ [23], where we assume ideal entropy coding of the quantizer outputs. In Table 2, IKLT/ECSQ and IKLT/ECTCQ respectively refer to these designs.

Table 2 RSNR (in dB) of KLT-based transform code designs for quantizing 16-dimensional vectors (M₁ = M₂ = 16) of source model B (α = 0.32, θ = 0.9) at R₁ = R₂ = R bits/sample

From a practical view point, the use of DCT instead of KLT is interesting [26]. We therefore consider the design of SP-DTCs based on the DCT, referred to as source-split distributed DCT (SP-DDCT) codes. Since DCT is a fixed transform, we only need to optimize the bit allocations. To do this, we assume that DCT is approximately a decorrelating transform for Gaussian vectors [4]. Then, the bit-allocations given by (9), (14), and (20) are still valid provided that the eigenvalues in these expressions are replaced by the variances of the corresponding DCT coefficients. The rest of the design procedure is the same as that with SP-DKLT. The RSNR of DCT based designs are presented in Table 3 (again, the performance predicted by SWC-HRSQ model is not an upper bound for corresponding practical codes). The results show that, for this particular source model, even the scalar quantization based SP-DDCT codes outperform TCQ-based IDCT codes.

Table 3 SNR (in dB) of DCT-based transform code designs for quantizing 16-dimensional vectors (M₁ = M₂ = 16) of source model B (α = 0.32, θ = 0.9) at R₁ = R₂ = R bits/sample

5 Concluding remarks

Rate-distortion analysis and experimental results demonstrate that SP-DTC is a promising practical approach to implementing distributed VQ of high-dimensional correlated vectors. The comparisons shown in Table 2, as well as similar comparisons for source model A and the source model in [[8], Example 6], indicate that these codes can substantially outperform the independent transform codes, when there is sufficient inter-vector correlation. This approach has also been demonstrated to be effective for DCT-based systems. Therefore, the proposed approach can be potentially used in applications such as stereo image compression when inter-camera communication is impractical. Our RD analysis however indicates that the achievable rate-region of SP-DKLT codes for jointly Gaussian sources is strictly inside that predicted by the DKLT of [8]. An interesting avenue of future work is to find implementable distributed transforms codes which can achieve the rate-pairs below the "time-sharing" line in Figure 6. Another issue is the extension of the proposed approach to more than two vector sources. In principle, source-splitting can be easily applied to more than two sources. However, with more than two vector sources, the complexity of the bit-allocation will be significantly higher.

1 Appendix

1.1 Proof of Theorem 2

The optimality of CKLT is proved in [5]. To prove the optimality of the bit allocation, consider high-rate scalar quantization of the transform coefficient U _m and ideal SW coding of the quantizer output Û _m at the rate r _m = H (Û _m | Y) bits/sample, m = 1, . . ., M₁, where H (·|·) denotes the conditional entropy [16]. In this case, the asymptotically optimal scalar quantizer for each coefficient is known to be uniform [5]. For high-rate uniform quantization, H (Û _m | Y) ≈ h(U _m | Y) - log₂(Δ _m ), where Δ _m is the quantizer step-size and h(·|·) is the conditional differential entropy [[16], Section 8.3]. Since the conditional variance of the transform coefficient U _m , given the side-information Y is E\left\{U_m^2|\mathbf{Y}\right\}={\lambda }_m, and (U _m , Y) are jointly Gaussian, it follows that h(U _m | Y) = (1/2) log₂ (2πeλ _m ) [16], and hence {\mathrm{\Delta }}_m=\sqrt{2\pi e{\lambda }_m}{2}^{-r_m}. Therefore, the MSE of high-rate uniform quantization followed by ideal SW coding of U _m is

d_m^{\mathsf{\text{SWC}}-\mathsf{\text{HRSQ}}}=\frac{{\mathrm{\Delta }}_m^2}{12}=\left(\frac{\pi e}{6}\right){\lambda }_m{2}^{-2r_m}.

(27)

Since (27) and (1) are the same within a constant factor of πe/ 6 (which is identical for all transform coefficients), it is easy to verify that the optimal bit-allocation solution under SWC-HRSQ model is also given by (2). However, the MSE of m th coefficient in this case is

\begin{array}{c}{d}_m=\left(\frac{\pi e}{6}\right){\lambda }_m{2}^{-2{\rho }_m}\hfill \\ =\left\{\begin{array}{cc}\left(\frac{\pi e}{6}\right)d^{*}\left(\mathit{\lambda },B,N\right),\hfill & m=1,\dots ,N\hfill \\ {\lambda }_m,\hfill & m=N+1,\dots ,M\hfill \end{array}\right.\hfill \end{array}

(28)

and hence the overall MSE is given by (5). Furthermore,

{\mathrm{\Delta }}_m=\mathrm{\Delta }=\sqrt{\left(2\pi e\right)d^{*}\left(\mathit{\lambda },B,N\right)}

for m = 1, . . ., N.

1.2 Proof of Theorem 3

For jointly Gaussian and mean-zero X and Y, there exists a matrix A and a mean-zero Gaussian vector W₁ independent of Y such that X = AY + W₁, where \mathbf{A}={\sum }_{XY}{\sum }_Y^{-1}, {\sum }_{W_1}={\sum }_{X|Y}=\mathbf{T}\mathbf{\Lambda }{\mathbf{T}}^T, and Λ is the diagonal matrix of eigenvalues of ∑_X|Y. Furthermore U = T^TAY + W₂, where W₂ = T^TW₁ is an uncorrelated Gaussian vector since {\sum }_{W_2}={\mathbf{T}}^T{\sum }_{W_1}\mathbf{T}=\mathbf{\Lambda } (note that for CKLT, T^-1 = T). Therefore ∑_U|Y= Λ. The MMSE estimate of U given Y is Ũ = E{U|Y} = T^TE{X | Y} = T^TAY. Thus, U = Ũ + W₂ and ∑ _U|Ũ = Λ, and it follows that U _i is independent of Ũ _j if j ≠ i and var(U _i | Ũ _i ) = var(U _i | Y), where var(·|·) denotes the conditional variance. Now, since h(U _i | Y) = h(U _i | Ũ _i ), we conclude that Ũ _i is a sufficient statistic [16] for decoder side-information Y in WZ quantization of U _i .