Distributed transform coding via source-splitting

Many new applications such as multi-camera imaging systems rely on networks of distributed wireless sensors to acquire signals in the form of high-dimensional vectors [1]. In such situations, an encoder in each sensor quantizes a vector of observation variables (without exchanging any information with other sensors) and transmits its output to a central processor which jointly decodes all the sources. The strong statistical dependencies among the signals observed by different sensors can be exploited in the decoder to reduce the transmission bit-rate of each sensor. This problem, in general, is referred to as distributed (or multiterminal) vector quantization (VQ). The design of a distributed VQ for a large number of source variables is a difficult task. A practically simpler, yet very effective approach to quantizing a large number of correlated variables by using a bank of single variable quantizers is transform coding (TC) [2–4]. Clearly, TC can be used for distributed VQ when separately observed vectors have both inter-vector and intra-vector statistical dependencies, a situation typical in applications such as camera networks. Most of the previous work [5–7] studies Wyner-Ziv (WZ) transform coding (WZ-TC), which is a special case of more general multiterminal transform coding (MT-TC) [8]. In WZ-TC, a single source is quantized given that the decoder has access to side information about the source.

Information-theoretic studies of distributed transform coding (DTC) can be found in [5, 8]. In [8], the optimal linear transform for Gaussian WZ-TC under the mean square-error (MSE) criterion is shown to be the conditional Karhunen-Loéve transform (CKLT), which is a natural extension of the result in [2]. This result is based on the assumption that each transform coefficient is compressed by a rate-distortion (RD) optimal WZ quantizer and hence describes the optimal performance theoretically attainable (OPTA) in Gaussian WZ-TC. However, the optimal solution to the more general MT-TC problem remains unsolved, even for the Gaussian case. In [8], an iterative descent algorithm for determining the OPTA of Gaussian MT-TC problem is given. It is shown that, while this algorithm [referred to as the distributed KLT (DKLT)] always converges to a solution, the final solution is not necessarily the global optimum. In any case, the practical implementation of distributed quantizers implied by the DKLT remains an open problem. In [5] WZ-TC based on high-rate scalar quantization and ideal Slepian-Wolf (SW) coding [9] is studied. In particular, it is shown that, for jointly Gaussian vectors, CKLT followed by uniform scalar quantization is asymptotically optimal, a natural extension of the result in [10] for entropy-coded quantization at high-rate. More importantly, the bit-allocations and quantizer-step sizes found in [5] can be used for practical design of WZ-transform codes as long as high-rate approximations hold. However, we note that, even when scalar quantizers are used, achieving good performance with this approach still requires the use of a subsequent block-based SW coding method (e.g., Turbo codes or LDPC codes). Other previous studies on WZ-TC can be found in [6, 7]. However, they rely on WZ scalar quantization of transform coefficients. Such methods are therefore most suitable for applications requiring low coding delay as their performance is strictly inferior to block-based quantization.

In contrast to WZ-TC, we consider in this article the practical design of two-terminal transform codes for jointly Gaussian vectors in which arbitrary transmission rates can be assigned to each terminal. Our approach is based on the idea of source-splitting[11, 12] to convert the two-terminal TC problem into two WZ-TC problems. Since transform codes quantize linear projections, we perform source splitting in terms of optimal linear approximations, i.e., a linear approximation of one source is provided as decoder side-information for the other source. The proposed source-split DTC (SP-DTC) approach only requires the design of two-sets of WZ quantizers sequentially, and avoids having to iteratively optimize two sets of WZ quantizers to each other as in [8]. However, this approach requires the solution of a bit allocation problem involving dependent WZ quantizers. To solve this problem for Gaussian sources, we propose an efficient tree-search algorithm, which can be used to find the a good SP-DTC under different models for quantization of transform coefficients. When used with the RD-optimal WZ quantization model [8], this algorithm can potentially locate the optimal SP-DTC for Gaussian sources. In practice, with constraints imposed on tree-search complexity, the algorithm yields a near-optimal solution. We refer to the optimal solution to the Gaussian problem as the source-split DKLT (SP-DKLT). Using this algorithm, we numerically compute the rate-region achievable with a SP-DKLT code for two examples of jointly Gaussian vector sources. This study shows that, when there is sufficient inter-source correlation, optimal SP-DKLT codes can achieve substantially better performance than independent transform codes for the two sources. However, we find that the rates achievable with SP-DKLT codes are strictly inside the optimal achievable rate-region predicted by the DKLT algorithm of [8]. In order to approach the performance predicted by the optimal SP-DKLT in practice, block WZ quantization of transform coefficients is required. For implementation of block WZ quantizers, we consider the use of trellis-coded quantization (TCQ) followed by SW coding. This two stage approach is known to achieve the RD function of Gaussian WZ coding [13]. In order practically implement this approach, we introduce the idea of designing conditional entropy constrained TCQ (CEC-TCQ) based on analytically found bit-allocations. We present experimental results to demonstrate that practical implementations of SP-DTCs for Gaussian sources can closely approach the performance limits indicated by the optimal SP-DKLT. On the other hand when SW coded high-rate scalar quantization model [5] is assumed for encoding transform coefficients, the tree-search algorithm proposed in this article can also be used to find asymptotically good SP-DTC codes for scalar quantization based implementations. These codes can be readily implemented using CEC scalar quantizers (CEC-SQ) as demonstrated by experimental studies presented in this article. In our experimental study, we also investigate the design of good SP-DTCs based on widely used discrete cosine transform (DCT).

This article is organized as follows. Section 2 presents a review of WZ-TC of Gaussian vectors and motivates the particular approach introduced in this article. Section 3 presents the idea of SP-DTC and develops the tree-search algorithm for finding the optimal transforms and the bit-allocation for SP-DKLT codes. Section 4 computes the achievable rate region of SP-DKLT codes for two example Gaussian source models, and presents experimental results obtained by designing SP-DTCs based on both KLT and DCT. Finally, some concluding remarks are given in Section 5.

Notation: As usual, bold letters denote vectors and matrices, upper case denotes random variables, and lower case denotes realizations. Σ _X denotes the auto-covariance matrix of the vector X. Σ _XY and Σ _X|Y , respectively denote the joint covariance matrix of (X, Y) and the conditional covariance matrix of X given Y. The eigenvalues λ₁ , . . ., λ _M of a M × M covariance matrix are always indexed such that λ₁≥ λ₂ . . . ≥ λ _M , and the corresponding KLT matrix has the structure $\mathbf{T}=\left({\mathbf{u}}_1^T,\dots ,{\mathbf{u}}_M^T\right)$, where u _m is the eigenvector associated λ _m .

Consider encoding of a Gaussian vector $\mathbf{X}\in {ℝ}^{M_1}$ using B bits per vector, given that the decoder has access to a jointly Gaussian vector $\mathbf{Y}\in {ℝ}^{M_2}$. Assume that both vectors have zero mean, and let the auto-covariance matrix of X be Σ _X = E{XX ^T }. In WZ-TC, a linear transform is first applied to X and each component of the transform coefficient vector U = T ^T X is separately compressed by a WZ quantizer, considering Y as decoder side-information, where T is a M₁ × M₁ unitary matrix. Let Û be the quantized value of U. The decoder then estimates the source vector based on Û and Y. We wish to find the optimal transform and the allocation of B bits among M₁ transform coefficients, which minimize the quantization MSE $E\left\{{∥\mathbf{X}-\widehat{\mathbf{X}}∥}^2\right\}$, where $\widehat{\mathbf{X}}=E\left\{\mathbf{X}|\widehat{\mathbf{U}},\mathbf{Y}\right\}$ is the optimal estimate (at the decoder) of the source vector. The solution of this problem requires an analytical model for coefficient quantization. To this end, [8] considers RD optimal WZ quantization (RD-WZQ) model, the solution based on which is appropriate for practical block quantization techniques such as TCQ. On the other hand [5] considers SW-coded high-rate scalar quantization (SWCHRSQ) model.

where N ≤ M₁ is a positive integer.

In general, designing a multi-terminal VQ is more difficult than designing a WZ-VQ, due to the mutual dependence among the encoders. However, one could use WZ-VQs to realize a multi-terminal VQ by using source-splitting [12]. It is known that in the quadratic-Gaussian case, source-splitting can be used to realize any rate-pair in the achievable rate-region by only using ideal WZ-VQs which correspond to the corner-points of the achievable rate-region [[12], Section V-C], [15]. While the same optimality properties cannot be claimed for source-splitting by linear transforms, the aforementioned observation still provides us the motivation to take a similar approach in practically realizing the DTCs which can operate at arbitrary rates, by using only WZ quantizers.

A block diagram of the SP-DTC system is shown in Figure 1. Let the total number of bits available for encoding two jointly Gaussian vectors ${\mathbf{X}}_1\in {ℝ}^{M_1}$ and ${\mathbf{X}}_2\in {ℝ}^{M_2}$ be B bits. The terminal 1 performs source splitting by providing an approximation ${\mathbf{Y}}_1^{\prime }\in {ℝ}^{N_1^{\prime }}$ of X₁ at the rate $B_1^{\prime }\left(<B\right)$ bits/vector as decoder side-information for WZ coding of terminal 2, where $N_1^{\prime }\le M_1$. In a TC framework, the goal is to provide the best (in MMSE sense) linear approximation of X₁ as the decoder side-information. Therefore, ${\mathbf{Y}}_1^{\prime }$ is the $B_1^{\prime }$-bit approximation of a linear projection ${\mathbf{U}}_1^{\prime }={\mathbf{T}}_1^{\prime T}{\mathbf{X}}_1$, where ${\mathbf{T}}_1^{\prime }$ is a M₁ × M₁ unitary matrix. Given the side-information ${\mathbf{Y}}_1^{\prime }$ at the decoder, the terminal 2 quantizes a linear projection ${\mathbf{U}}_2={\mathbf{T}}_2^T{\mathbf{X}}_2$ of X₂ using $B_2\left(<B-B_1^{\prime }\right)$ bits/vector, where T₂ is a M₂ × M₂ unitary matrix. Let ${\mathbf{Y}}_2\in {ℝ}^{N_2}$, be the quantized value of U₂, where N₂≤ M₂. Then, given the quantized linear projections of both X₁ and X₂ available at the decoder, the terminal 1 quantizes the a linear projection ${\mathbf{U}}_1^{″}={\mathbf{T}}_1^{″T}{\mathbf{X}}_1$ of X₁ using $B_1^{″}=B-B_1^{\prime }-B_2$ bits/vector, where ${\mathbf{T}}_1^{″}$ is a M₁ × M₁ unitary matrix. Let ${\mathbf{Y}}_1^{″}\in {ℝ}^{N_1^{″}}$ be the quantized value of ${\mathbf{U}}_1^{″}$, where $N_1^{″}\le M_1$. In the receiver, each source vector is reconstructed by a WZ decoder. The MMSE optimal reconstructions for X₁ and X₂ are, respectively given by ${\widehat{\mathbf{X}}}_1=E\left\{{\mathbf{X}}_1|{\mathbf{Y}}_1^{″},\mathbf{V}\right\}$ and ${\widehat{\mathbf{X}}}_2=E\left\{{\mathbf{X}}_2|\mathbf{V}\right\}$, where $\mathbf{V}={\left({\mathbf{Y}}_1^{\prime }{\mathbf{Y}}_2\right)}^T$. The total transmission rate for source X₁ is thus $B_1=B_1^{\prime }+B_1^{″}$ bits/vector. The rates used by terminals 1 and 2 in bits/sample are given by R₁ = B₁/M₁ and R₂ = B₂/M₂, respectively.

The bit allocation problem can now be stated as follows:

where $\mathbf{r}={\left(r_1,\dots ,r_{2M_1+M_2}\right)}^T$. The explicit solution of this minimization problem is unfortunately intractable due to the inter-dependence of the three transform codes involved.

However, an explicit solution can be found for a variant of this problem obtained by fixing $B_1^{\prime }$, $B_1^{″}$ and B₂, so that the number of bits allocated to each transform code is fixed and it is only required to optimize the bit allocation among the quantizers within each transform code. For simplicity, we refer to this problem as the constrained bit-allocation problem. In the following, an explicit solution to this problem is derived. Based on the result, we then present a tree-search algorithm to solve the unconstrained problem (8). Under both RD-WZQ and SWC-HRSQ models, the optimal transforms for Gaussian sources are CKLTs. Therefore, we refer to the solution to problem (8) as the SP-DKLT.

3.2 A tree-search solution to the unconstrained bit-allocation problem

We note that the optimal solution to the unconstrained problem defined in (8) corresponds to the MMSE solution of the constrained problem over the set of rate-tuples $\mathcal{S}=\left\{\left(B_1^{\prime },B_1^{″},B_2\right):B_1^{\prime }\in \left(0,B\right),B_1^{″}\in \left(0,B\right),B_2\in \left(0,B\right),B_1^{\prime }+B_1^{″}+B_2\le B\right\}$. This set is shown in Figure 2. One approach to locating the MMSE solution is to search over an appropriately discretized grid of points inside . As we will see, even though an exhaustive search on a fine grid can be prohibitively complex, a much simpler constrained tree-search algorithm exists which can be used to locate the required solution with a very high probability.

The proposed algorithm is a generalization of a class of bit-allocation algorithms in which a small fraction ΔB of the total bit-budget B is allocated to the "most deserving" quantizer among a set of quantizers in an incremental fashion, until the entire bit-budget is exhausted [[4], Section 8.4]. Unfortunately, this type of a greedy search cannot guarantee that the final solution is overall optimal and can yield poor results in our problem where the bit allocation among three sets of dependent quantizers must be achieved. On the other hand, if the increment ΔB is chosen small enough, a near-optimal solution can be found by resorting to a tree-search. Even though a full tree-search is intractable, a simple algorithm referred to as the (M, L)-algorithm[18] exists for detecting the minimum cost path in the tree with a high probability. We use this insight to formulate a tree-search algorithm for solving the unconstrained bit allocation problem, in which a set of constrained bit allocation problems are solved in each iteration.

In order to describe the proposed tree-search algorithm in detail, let ΔB be the incremental amount of bits to be allocated in each step of the search, where 0 < ΔB ≪ B. The algorithm is initialized by setting $\left(B_1^{\prime },B_1^{″},B_2\right)=\left(0,0,0\right)$, i.e., the origin in Figure 2. Now if we are to allocate ΔB bits to only one of the three transform codes ${\mathbf{T}}_1^{\prime }$, ${\mathbf{T}}_1^{″}$ or T₂, then there are three possible choices for the rate-tuple $\left(B_1^{\prime },B_1^{″},B_2\right)$, namely (ΔB, 0, 0), (0, ΔB, 0), and (0, 0, ΔB). For each of these choices, we can explicitly solve the constrained bit allocation problem as described in the previous section and find the MMSE solution. Each of these candidate solutions can be viewed as a node in a tree as shown in Figure 3. The root node of the tree corresponds to a SP-DTC of rate 0, and a node in the first level of nodes obtained in the first iteration of the algorithm corresponds to a SP-DTC of rate of ΔB bits per source pair (X₁, X₂). In the second iteration of the algorithm, we allocate ΔB more bits to each of the three candidate SP-DTCs (but to one of the SP-DTCs at a time) in the first level of nodes. Note that, for each SP-DTC we can allocate ΔB bits in three different ways, i.e., ΔB bits can be added to either $B_1^{\prime }$, $B_1^{″}$, or B₂. This requires the solution of three constrained bit allocations problems for each of the 3 nodes in the first-level. As a result, the tree will be extended to a second level of 3² nodes, in which each node corresponds to a SP-DTC of 2ΔB bits per source-pair, as shown in Figure 3. We can repeat this procedure, allocating ΔB bits to each of terminal node of the tree in a given iteration, until all B bits are exhausted. After the final iteration, the tree would consist of L = ⌈B/ΔB⌉ levels with 3 ^L nodes in the last level (terminal nodes). Each terminal node corresponds to a candidate SP-DTC of rate B, and rate-tuples of these SP-DTCs lie on the plane $B_1^{\prime }+B_1^{″}+B_2=B$ in Figure 2. The MMSE terminal node of the tree is the optimal solution to the unconstrained bit allocation problem, provided that the latter solution is on the search-grid. If ΔB is chosen small enough, then we can ensure that the optimal solution is nearly on the search-grid. Suppose that, in each iteration, the algorithm saves the MSE of the solution to the constrained bit-allocation problem associated with each node. In theory, the optimal solution can be found by an exhaustive tree-search, using the MSE of a node [given by (22)] as the path-cost. In order to practically implement the tree-search, we use the (M, L)-algorithm, in which the parameter M can be chosen to reduce the complexity at the expense of decreased accuracy (i.e., the probability of detecting the lowest cost path in the tree). In the (M, L) algorithm [[18], p. 216] for a tree of depth L, one only retains the M best (lowest MSE) nodes in each iteration. When M = 1 we have a completely greedy search. On the other hand when M _n = 3 ⁿ in the n th iteration, we have a full-tree search which has a complexity that grows exponentially with the iteration number. When M (1 ≤ M ≤ 3 ^L ) is a prescribed constant, the complexity is linear in M, independent of the number of iterations. In obtaining the simulation results presented in Section 4, M = 27 and L = 135 (ΔB = 0.2) were found to be sufficient to obtain near optimal results. For example, it was observed that even for M = 81 and L = 405, nearly the same result was obtained.

Source model A: Let the components of X₁ be M₁ consecutive samples of a first-order Gauss-Markov process with a unit-variance and the correlation coefficient |ρ| < 1, i.e., X_1,m= ρX_1,(m-1)+ Z _m , m = 2, . . ., M₁, where Z _m , m = 1, . . ., M₁ are mean-zero iid Gaussian variables such that $E{Z}_m^2=1-{\rho }^2$. The auto-covariance matrix ${\sum }_{X_1}$ is a Teoplitz matrix with the first row $\left(1,\rho ,{\rho }^2,\dots ,{\rho }^{M_1-1}\right)$. Now define the components of X₂ to be noisy observations of the components of X₁, i.e., X_2,m= γX_1,m+ W _m , where |γ| < 1 and W _m is a mean-zero, iid Gaussian variable with $E{W}_m^2=1-{\gamma }^2$, m = 1, . . ., M₁ (M₂ = M₁). It follows that, ${\sum }_{X_2}$ is a Teoplitz matrix with the first row $\left(1,{\gamma }^2\rho ,{\gamma }^2{\rho }^2,\dots ,{\gamma }^2{\rho }^{M_1-1}\right)$. Furthermore, the cross-covariance matrix ${\sum }_{X_1{X}_2}={\gamma }^2{\sum }_{X_1}$. Note that X₁ and X₂ are not statistically similar and the components of X₁ are more correlated than those of X₂.

Source model B: Consider a spatial Gaussian random field in which the correlation function decays with distance d according to the squared exponential model [19]. We define the random vectors X₁ and X₂ to be observations picked-up by a pair of sensor arrays placed in this random filed. In this case, the auto-covariance matrix of X₁ is given by ${\left[{\sum }_{X_1}\right]}_{ij}=exp\left\{-{\left(\alpha d_{ij}\right)}^2\right\}$, where α is a constant and d _ij is the distance between X_1,iand X_1,j. The auto-covariance matrix of X₂ also has a similar form. For simplicity assume that the sensors in each array are placed on a M × M square grid of unit spacing (i.e., M₁ = M₂ = M²), the two arrays are on parallel planes separated by a distance r, and the two grids are aligned so that the distance between X_1iand X_2iis r for all i. With this setup, the distance between X_1iand X_2jis $\sqrt{d_{ij}^2+r^2}$, and the cross-covariance matrix is given by ${\left[{\sum }_{X_1{X}_2}\right]}_{ij}=\theta exp\left\{-{\left(\alpha d_{ij}\right)}^2\right\}$, where θ = exp {- (αr)²}. This sensor structure ensures that X₁ and X₂ are statistically similar. However, ${\sum }_{X_1{X}_2}$ can be chosen independently (by choosing array separation r) of ${\sum }_{X_1}$ and ${\sum }_{X_2}$.

4.1 RD performance

We compute the rate-pairs (R₁, R₂) achievable with a SP-DKLT code for a given a total MSE D, by fixing R₁ (or R₂) and then searching for minimum R₂ (or R₁) required to achieve the MSE D [given by (6)]. The rate-pairs achievable for D = 0.01 with SP-DKLT coding of vectors from source model A with ρ = 0.9, γ = 0.9, are plotted in Figure 4. These values of ρ and γ result in a source cross-covariance matrix with the largest element 0.9. In Figure 4, the curve "SP-DKLT (X₁ split)" corresponds to a system in which the input to terminal 1 (which applies source splitting) is X₁ as shown in Figure 1. The curve "SP-DKLT (X₂ split)" corresponds to a system in which the input to the terminal 1 is X₂. Note that the two curves are not symmetric in rates and they coincide if R₁ and R₂ are inter-changed in one of the curves. This is because X₁ and X₂ have different auto-covariance matrices, and hence inter-changing the rates is equivalent to inter-changing the terminals. Importantly, this result indicates that when the two sources are not statistically identical, which source is chosen for splitting does not affect the SP-DKLT performance. Table 1 lists the best bit allocations found by the tree-search algorithm for SP-DKLT codes shown in Figure 4. Note that, for the same (B₁, B₂), the rate-split between $B_1^{\prime }$ and $B_2^{″}$ when X₁ is applied to the terminal 1 is not identical to that when X₂ is applied to the terminal 1.

X1 Split					X2 Split
X 1		X 2			X 2		X 1
${\mathit{B}}_{\mathbf{1}}^{\mathbf{\prime }}$	${\mathit{B}}_{\mathbf{1}}^{″}$	B 1	B 2	B	${\mathit{B}}_{\mathbf{1}}^{\prime }$	${\mathit{B}}_{\mathbf{1}}^{″}$	B 1	B 2	B
0	14.8	14.8	96	110.8	0	22	22	96	118
0	15	15	80	95	0	22.4	22.4	80	102.4
0	15.8	15.8	64	79.8	0	22.6	22.6	64	86.6
0	19	19	48	67	0	24.4	24.4	48	72.4
0	22	22	44	66	0	25.6	25.6	44	69.6
3.2	22.8	26	40	66	0	26.8	26.8	40	66.8
10	20	30	36	66	0	30	30	36	66
16.6	16.4	33	33	66	2.8	30.2	33	33	66
18.6	15.4	34	32	66	4.2	29.8	34	32	66
23.2	12.8	36	30	66	7.6	28.4	36	30	66
34.2	5.8	40	27.2	67.2	17.4	22.6	40	26	66
44	0	44	25.8	69.8	32	12	44	22	66
48	0	48	24.4	72.4	43.8	4.2	48	19.4	68.4
64	0	64	22.6	86.6	64	0	64	16	90
80	0	80	22.2	102.2	80	0	80	15.2	95.2
96	0	96	22	118	96	0	96	14.8	110.8

Figure 4 also shows the rate region achievable if each source is independently compressed using the KLT (i.e., only intra-vector correlation is utilized), labeled IKLT, and the OPTA lower bound for distributed TC predicted by the iterative DKLT algorithm [8]. The performance of both distributed and non-distributed TC of source model A degrades as ρ decreases, since both auto- and cross-covariance matrices of X₁ and X₂ are functions of ρ. Next consider the source model B for which the lowest achievable rate-pairs corresponding to D = 0.005 are plotted in Figure 5. The source parameter α = 0.32 results in auto-covariance matrices whose largest off-diagonal element is 0.9. Also recall that θ is the largest element in the source cross-covariance matrix. Note that changing θ only affects the cross-covariance matrix, and hence has no effect on the best achievable rates for independent coding of the two sources. On the other hand, as θ increases, the rates achievable with distributed coding do improve. Since in source model B, the two sources are statistically similar, the curves in Figure 5 are symmetric in rates and the optimal bit allocation does not depend on which source is chosen for splitting.

The RD performance in Figures 4 and 5 indicate that SP-DKLT codes can significantly outperform IKLT codes at all rates when there is sufficient correlation between the two distributed sources. The performance of SP-DKLT coding necessarily approaches OPTA (DKLT) bound when either R₁ or R₂ is sufficiently high. That is, the terminal with the higher bit rate can independently transform code its input with negligible distortion, and the other terminal can then apply WZ-TC at the minimum rate achievable with "almost unquantized" decoder side-information. However, for both source-models the rate-region achievable with source-splitting is strictly inside that of DKLT. In other words, there are some rate-pairs inside the DKLT rate-region for a given MSE D, which cannot to be achieved by a SP-DKLT code. A closely related issue is that, for a range of values of (R₁, R₂), the sumrate R₁ + R₂ of SP-DKLT codes remains constant and reaches its minimum. For example, it can be seen from Figure 4 and Table 1 that the sum-rate is about 4.125 bits when the rate of X₁ is in the range 1.375 - 2.5 bits/sample. From Table 1, it can be seen that when the sum-rate is greater than its minimum value, the optimal SP-DKLT code approaches a WZ transform code, i.e., no source-splitting occurs. This situation, which also exists in Figure 5, suggests that optimal SP-DKLT codes for the sum-rate at which a given D can be achieved, are equivalent to time-sharing [12] of two "corner points". Figure 6 illustrates this situation for optimal SP-DKLT codes at D = 0.005 for source model B (θ = 0.9 in Figure 5). It should however be noted that, unlike source-splitting, time-sharing between the two terminals requires synchronization of their encoders [11].

4.2 Design examples

In this section, we focus on the practical design of SP-DKLT codes for a given pair of rates (R₁, R₂) based on both scalar and block-quantization. RD-WZQ model used in Section 3.1.1 implies infinite block-length WZ-VQ of each coefficient. A practically realizable approach to block WZ quantization is SWC-TCQ [20]. Experimental results obtained with LDPC codes of block length up to 10⁶ bits and TCQs up to 8,192 states are presented in [20] for quadratic Gaussian WZ quantization, which indicate that performance very close to the theoretical limit can be achieved with SWC-TCQ. Motivated by these results, we aim to implement SP-DTCs which can approach theoretical performance predicted in Section 3.1.1 using TCQ and SW coding for encoding transform coefficients. However, the SWC-TCQ design procedure followed in [20] is to first design a TCQ whose MSE satisfies a constraint (by choosing a sufficiently high rate) and then to estimate the output conditional entropy (which is the target rate of the SW code) of the resulting TCQ. This is sufficient for verifying the achievable rate pairs for a given MSE which is the goal of [20]. Our problem is different in that the rate of the SW code is specified by the solution to the bit-allocation problem and our goal is to design a TCQ which minimizes the MSE, subject to a constraint on the output conditional entropy. This requires an alternative formulation of the design procedure, which we refer to as CEC-TCQ. In previous work on non-distributed quantization, entropy constrained TCQ (EC-TCQ) has been investigated in [21–24]. CEC-TCQ is a modification of EC-TCQ in [21, 22] to accommodate block SW-coding of the TCQ output relative to a decoder side-information sequence. Our formulation of CEC-TCQ follows the supersetentropy formulation of EC-TCQ in [22].

Suppose that a sequence of source samples {U _n ∈ℝ} has to be quantized, given that the sequence {Y _n ∈ℝ}, is available at the decoder as side-information, where n = 1, 2, . . . denotes the discrete-time. Similar to an ordinary TCQ [25], a CEC-TCQ uses a size $2^{R_{\mathsf{\text{TCQ}}}+1}$ scalar codebook to quantize the input sequence U₁, U₂, . . ., into a R_TCQ bits/sample output sequence Û₁, Û₂, . . . . However, CEC-TCQ output satisfies the additional property that the conditional entropy H (Û _n |Y _n ) = E{- log₂P (Û _n |Y _n )} ≤ R for some given R. It follows from [9] that if the CEC-TCQ output is SW-coded relative to the decoder side-information sequence {Y _n } then LR bits are sufficient to (almost) losslessly transmit a sequence of L source samples as L → ∞. The optimal CEC-TCQ minimizes the MSE E{(U _n - Û _n )²}, subject to the constraint E{- log₂P (Û _n |Y _n )} ≤ R, or equivalently, minimizes the Lagrangian

$J=\underset{L\to \infty }{lim}\sum _{n=1}^L\left[E\left\{{\left(U_n-{\mathit{Û}}_n\right)}^2\right\}+\beta E\left\{-{log}_{2}P\left({\mathit{Û}}_n|Y_n\right)\right\}\right]$

(24)

where β > 0 is the Lagrange multiplier. This implies that, given a specific sequence of input samples u₁, u₂, . . ., the CEC-TCQ encoder should use the Viterbi algorithm based on the path-cost function

$\sum _n\left[{\left(u_n-{\mathit{û}}_n\right)}^2+\beta E\left\{-{log}_{2}P\left({\mathit{û}}_n|Y_n\right)\right\}\right].$

(25)

In a rate R_TCQ TCQ, each codeword c _k , $k=1,\dots ,2^{R_{\mathsf{\text{TCQ}}}+1}$, in the codebook is labeled with R_TCQ-bit binary string [25]. Let b _i , $i=1,\dots ,2^{R_{\mathsf{\text{TCQ}}}}$ be these binary labels. Then, to compute (25), the cost βE{- log₂P (b _i |Y)} must also be stored for each binary label, where Y is the random variable representing the decoder side-information. For a fixed value of β, we can use a slight modification of the algorithm in [21] for optimizing the CEC-TCQ codebook, by replacing the codeword entropies E{- log₂P(b _i )} by the conditional entropies E{- log₂P (b _i |Y)}, and by using a training sequence of (U _n , Y _n ) pairs. In order to approximate the expectations by sample averages computed from training data, the side-information variable Y is discretized to $\mathit{Ŷ}\in \left\{{\eta }_1,\dots {\eta }_{\mathcal{Y}}\right\}$, where is a large enough positive integer. Then E{- log₂P (B = b _i |Y)} can be approximated by

$\mathit{Ĥ}\left(b_i\right)=-\sum _{k=1}^{\mathcal{Y}}{log}_2\left[P\left(B=b_i|\mathit{Ŷ}={\eta }_k\right)\right]P\left(\mathit{Ŷ}={\eta }_k|B=b_i\right)$

(26)

where $B\in \left\{b_1,\dots ,b_{2^{R_{\mathsf{\text{TCQ}}}}}\right\}$ is the binary-labeled output of the TCQ. Given a TCQ code-book, the probabilities P(B = b _i |Ŷ = η _k ) and P(Ŷ = η _k |B = b _i ) can be estimated using the training set [it is sufficient to estimate p _{i, k} = P(B = b _i , Ŷ = η _k ), $k=1,\dots \mathcal{Y},i=1,\dots ,N$]. To complete the design, it is necessary to search for the value of β for which E{- log₂P (Û _n |Y _n )} ≈ R by repeating the codebook optimization for an appropriately chosen sequence of β values.

For block WZ-code designs, the transforms and the bit allocations are found by RD-WZQ model (Section 3.1.1) and WZ quantizers are implemented using CEC-TCQ followed by binary SW coding. More specifically, the rate found by the bit allocation algorithm for each transform coefficient is used as the conditional entropy constraint in the design of a CEC-TCQ for that coefficient. As described in Section 2.3, the CEC-TCQ designs are based on scalar-side information obtained by a linear transform of the vector side-information at the decoder, see Theorem 3. All CEC-TCQ designs are based on the 8-state trellis used in JPEG2000 [[26], Figure 3.16]. For trellis encoding and decoding, a sequence length of 256 source samples has been used. For design and testing quantizers, sample sequences of length 5 × 10⁵ have been used. Since, the main focus this paper is the design of transforms and the quantizers, we assume ideal SW coding of the binary output of each CEC-TCQ, so that our results do not depend on any particular SW coding method. In a practical implementation (e.g., [20]), near optimal performance can be obtained by employing a sufficiently long SW code (note that sequence length for SW-coding can be chosen arbitrary larger than the sequence length used for TCQ encoding). This type of coding is well suited for applications such as distributed image compression, where the coding is inherently block-based.

We also consider SP-DKLT code designs based on scalar quantization. In this case, the transforms and bit allocations are found by using the SWC-HRSQ model (Section 3.1.2). While it is possible to use the step-size predicted by SWC-HRSQ model to design uniform quantizers, we found that such quantizers in reality do not satisfy the required entropy constraint at lower rates. We instead use conditional entropy constrained scalar quantizers (CEC-SQ), designed by modifying the algorithm in [27] to accommodate a conditional entropy constraint similar to CEC-TCQ approach above.

The reconstruction signal-to-noise ratio (RSNR) [with MSE as given by (6)] of SP-DKLT code designs for source model B is shown in the rows labeled Design in Table 2 where SP-DKLT/CECSQ and SP-DKLT/CECTCQ refer to scalar quantization and TCQ based designs respectively. The rows labeled Analytical show the performance predicted by the SWC-HRSQ and RD-WZQ models upon which the transforms and bit-allocations are based (note however that the performance predicted by SWC-HRSQ model is not necessarily an upper-bound for CEC-SQ designs which are not constrained to be uniform quantizers). We compare the performance of our SP-DKLT code designs with IKLT codes for which the bit-allocations are obtained by using either entropy coded high-rate quantization model (for scalar quantizer design) [[4], Section 9.9] or RD-optimal quantization model (for block quantizer design) [[16], Section 10.3.3] for Gaussian variables. The IKLT codes with scalar quantization have been implemented by using entropy constrained scalar quantizers (EC-SQ) while those with block quantizers have been implemented by using EC-TCQ [23], where we assume ideal entropy coding of the quantizer outputs. In Table 2, IKLT/ECSQ and IKLT/ECTCQ respectively refer to these designs.

R	IKLT		SP-DKLT
(bits/sample)	EC-SQ	EC-TCQ	CEC-SQ	CEC-TCQ
0.5	10.3	11.7	12.4	14.0	(Analytical)
	10.3	11.0	11.6	12.7	(design)
1	16.6	18.0	19.1	20.6
	16.4	17.3	18.6	19.6
1.5	21.6	23.0	23.8	25.6
	21.7	22.5	23.3	24.4
2	25.7	27.1	28.2	29.8
	25.2	26.6	27.8	28.6

Analytical values refer to SNR of the quantization model assumed for determining the optimal transforms and the bit-allocation

From a practical view point, the use of DCT instead of KLT is interesting [26]. We therefore consider the design of SP-DTCs based on the DCT, referred to as source-split distributed DCT (SP-DDCT) codes. Since DCT is a fixed transform, we only need to optimize the bit allocations. To do this, we assume that DCT is approximately a decorrelating transform for Gaussian vectors [4]. Then, the bit-allocations given by (9), (14), and (20) are still valid provided that the eigenvalues in these expressions are replaced by the variances of the corresponding DCT coefficients. The rest of the design procedure is the same as that with SP-DKLT. The RSNR of DCT based designs are presented in Table 3 (again, the performance predicted by SWC-HRSQ model is not an upper bound for corresponding practical codes). The results show that, for this particular source model, even the scalar quantization based SP-DDCT codes outperform TCQ-based IDCT codes.

R	IDCT		SP-DDCT
(bits/sample)	EC-SQ	EC-TCQ	CEC-SQ	CEC-TCQ
0.5	7.9	9.1	11.7	12.6	(Analytical)
	8.0	8.4	11.0	12.0	(design)
1	11.9	13.2	15.9	17.8
	11.9	12.6	16.6	17.0
1.5	15.3	16.8	19.7	21.9
	15.3	16.1	20.5	21.3
2	18.9	20.3	23.6	25.6
	18.7	19.4	24.5	25.1

Analytical values refer to SNR of the quantization model assumed for determining the bit-allocation.

Rate-distortion analysis and experimental results demonstrate that SP-DTC is a promising practical approach to implementing distributed VQ of high-dimensional correlated vectors. The comparisons shown in Table 2, as well as similar comparisons for source model A and the source model in [[8], Example 6], indicate that these codes can substantially outperform the independent transform codes, when there is sufficient inter-vector correlation. This approach has also been demonstrated to be effective for DCT-based systems. Therefore, the proposed approach can be potentially used in applications such as stereo image compression when inter-camera communication is impractical. Our RD analysis however indicates that the achievable rate-region of SP-DKLT codes for jointly Gaussian sources is strictly inside that predicted by the DKLT of [8]. An interesting avenue of future work is to find implementable distributed transforms codes which can achieve the rate-pairs below the "time-sharing" line in Figure 6. Another issue is the extension of the proposed approach to more than two vector sources. In principle, source-splitting can be easily applied to more than two sources. However, with more than two vector sources, the complexity of the bit-allocation will be significantly higher.