The High-Resolution Rate-Distortion Function under the Structural Similarity Index

—In this paper, we show that the structural similarity (SSIM) index, which is used in image processing to assess the similarity between an image representation and an original reference image, can be formulated as a locally quadratic distortion measure. We furthermore show that recent results of Linder and Zamir on the rate-distortion function (RDF) under locally quadratic distortion measures are applicable to this SSIM distortion measure. We ﬁnally derive the high-resolution SSIM-RDF and provide a simple method to numerically compute an approximation of the SSIM-RDF of real images.


I. INTRODUCTION
A VAST majority of the work on source coding with a fidelity criterion (i.e.rate-distortion theory) concentrates on the meansquared error (MSE) fidelity criterion.The MSE fidelity criterion is used mainly due its mathematical tractability.However, in applications involving a human observer it has been noted that distortion measures which include some aspects of human perception generally perform better than the MSE [1].A great number of perceptual distortion measures are non-difference distortion measures and, unfortunately, even for simple sources, their corresponding ratedistortion functions (RDFs), that is, the minimum bit-rate required to attain a distortion equal or smaller than some given value, are not known.However, in certain cases it is possible to derive their RDFs.For example, for a Gaussian process with a weighted squared error criterion, where the weights are restricted to be linear time-invariant operators, the complete RDF was first found in [2] and later rederived by several others [3], [4].Other examples include the special case of locally quadratic distortion measures for fixed rate vector quantizers and under high-resolution assumptions [5], results which are extended to variable-rate vector quantizers in [6], [7], and applied to perceptual audio coding in [8], [9].
In [10], Wang et al. proposed the structural similarity (SSIM) index as a perceptual measure of the similarity between an image representation and an original reference image.The SSIM index takes into account the cross-corelation between the image and its representation as well as the images first and second order moments.It has been shown that this index provides a more accurate estimate of the perceived quality than the MSE [1].The SSIM index was used for image coding in [11] and was cast in the framework of ℓ1compression of images and image sequences in [12].The relation between the coding rate of a fixed-rate uniform quantizer and the distortion measured by the SSIM index was first addressed in [13].In particular, for several types of source distributions and under highresolution assumptions, upper and lower bounds on the SSIM index were provided as a function of the operational coding rate of the quantizer [13].
In this paper, we present the high-resolution RDF for sources with finite differential entropy and under a SSIM index distortion measure.The SSIM-RDF is particularly important for researchers and practitioners within the image coding area, since it provides a lower bound on the number of bits that any coder, e.g., JPEG etc., will use when encoding an image into a representation, which has an SSIM index no smaller than a pre-specified level.Thus, it allows one to compare the performance of a coding architecture to the optimum performance theoretically attainable.The SSIM-RDF is non convex and does not appear to admit a simple closed-form expression.However, when the coding rate is high, i.e., when each pixel of the image is represented by a high number of bits, say more than 0.5 bpp, then we are able to find a simple expression, which is asymptotically (as the bit-rate increases) exact.For finite and small bit-rates, our results provides an approximation of the true SSIM-RDF.
In order to find the SSIM-RDF, we first show that the SSIM index can be formulated as a locally quadratic distortion measure.We then show that recent results of Linder and Zamir [7] on the RDF under locally quadratic distortion measures are applicable, and finally obtain a closed form expression for the high-resolution SSIM-RDF.We end the paper by showing how to numerically approximate the high-resolution SSIM-RDF of real images.

II. PRELIMINARIES
In this section we present an important existing result on ratedistortion theory for locally quadratic distortion measures and also present the SSIM index.We will need these elements when proving our main results, i.e., Theorems 2 and 3, in Section III.

A. Rate-Distortion Theory for Locally Quadratic Distortion Measures
Let x ∈ R n be a realization of a source vector process and let y ∈ R n be the corresponding reproduction vector.A distortion measure d(x, y) is said to be locally quadratic if it admits a Taylor series (i.e. it possesses derivatives of all orders in a neighborhood around the points of interest) and furthermore, if the second-order terms of its Taylor series dominate the distortion asymptotically as y → x (corresponding to the high-resolution regime).In other words, if d(x, y) is locally quadratic, then it can be written as d(x, y) = (x − y) T B(x)(x − y) + O( x − y 3 ), where B(x) is an input-dependent positive-definite matrix and where for y close to x, the quadratic term (i.e., (x − y) T B(x)(x − y)) is dominating [7].We use upper case X when referring to the stochastic process generating a realization x and use h(X) to denote the differential entropy of X, provided it exists.The determinant of a matrix B is denoted det(B) and E denotes the expectation operator.
The RDF for locally quadratic distortion measures and smooth sources1 was found by Linder and Zamir [7] and is given by the following theorem: Theorem 1 ([7]).Suppose d(x, y) and X satisfy some mild technical conditions 2 .Then, where R(D) is the RDF of X (in bits per block) under distortion d(x, y), and h(X) denotes the differential entropy of X.

B. The Structural Similarity Index
Let x, y ∈ R n where n ≥ 2. We define the following empirical quantities: the sample mean µx x n−1 , and the sample cross variance σxy = σyx We define µy and σ 2 y similarily.The SSIM index studied in [10] is defined as: where Ci > 0, i = 1, 2. The SSIM index ranges between −1 and 1, where positive values close to 1 indicate a small perceptual distortion.We can define a distortion "measure" as one minus the SSIM index, that is which ranges between 0 and 2 and where a value close to 0 indicates a small distortion.The SSIM index is locally applied to N×N -blocks of the image.Then, all block indexes are averaged to yield the SSIM index of the entire image.We treat each block as an n-dimensional vector where n = N 2 .

III. RESULTS
In this section we present the main theoretical contributions of this paper.We will first show that d(x, y) is locally quadratic, and then use Theorem 1 to obtain the high-resolution RDF for the SSIM index.
Proof: See the appendix.Theorem 3. The high-resolution RDF R(D) for the source X under the distortion measure d(x, y) defined in (3) and where h(X) < ∞ and 0 < E X 2 < ∞, is given by where a(X) and b(X) are given by Proof: Recall from Theorem 2 that d(x, y) is locally quadratic.Moreover, the weighting matrix B(X) in (1), which is also known as a sensitivity matrix [5], is given by (15), see the Appendix.In the Appendix it is also shown that B(x) is positive-definite since a(x) > 0, a(x) + b(x)n > 0, ∀x, where a(x) and b(x) are given by ( 4) and ( 5), respectively.From ( 16), it follows that 2 See conditions a) -g) in Section II-A in [7].
At this point, we note that the main technical conditions required for Theorem 1 to be applicable is boundedness in the following sense [7] and E(trace{B −1 (X)})3/2 < ∞ and furthermore uniformly bounded third-order partial derivatives of d(X, Y ).The first two conditions are satisfied by the assumptions of the Theorem.The next two conditions follow since all elements of B(x) are bounded ∀x (see the proof of Theorem 2).Moreover, due to the positive stabilization constants C1 and C2, trace{B(x)} −1 is clearly bounded.Finally, it was established in the proof of Theorem 2 that the third-order derivatives of d(X, Y ) are uniformly bounded.Thus, the proof now follows simply by using ( 6) in (1).

A. Evaluating the SSIM Rate-Distortion Function
In this section we propose a simple method for estimating the SSIM-RDF in practice based on real images.Conveniently, we do not need to encode the images in order to find their corresponding high-resolution RDF.Thus, the results in this section (as well as the results in the previous sections) are independent of any specific coding architecture.
In practice, the source statistics are often not available and must therefore be found empirically from the image data.Towards that end, one may assume that the individual vectors {x(i)} M i=1 (where x(i) denotes the ith N×N subblock of the image and M denotes the total number of subblocks in the image) of the image constitute approximately independent realizations of a vector process.In this case, we can approximate the expectation by the empirical arithmetic mean, i.e., where a(x(i)) and b(x(i)) indicates that the functions a and b defined in (4) and ( 5) are used on the ith subblock x(i).Several estimates of  In order to obtain the high-resolution RDF of the image, according to Theorem 3, we also need the differential entropy h(X) of the image, which is usually not known a-priori in practice.Thus, we need to numerically estimate h(X), e.g., by using the average empirical differential entropy over all blocks of the image.In order to do this, we apply the two-dimensional KLT on each of the subblocks of the image in order to reduce the correlation within the subblocks. 3 Then we use a nearest-neighbor entropy-estimation approach to approximate the marginal differential entropies of the elements within a subblock [16].Finally, we approximate h(X) by the sum of the marginal differential entropies, which yields the values presented in Table II

IV. SIMULATIONS
In this section we use the JPEG codec 4 on the images and measure the corresponding SSIM values of the reconstructed images.Then we compare these operational results to the information theoretic estimated high-resolution SSIM RDF obtained as described in the previous section.We are interested in the high-resolution region, which corresponds to small d(x, y) values (i.e., values close to zero) or equivalently large SSIM values (i.e., values close to one). Figure 1 shows the high-resolution SSIM-RDF for d(x, y) values below 0.27, corresponding to SSIM values above 0.73.Notice that the rate becomes negative at large distortions (i.e., small rates), which happens because the high-resolution assumption is clearly not satisfied and the approximations are therefore not accurate.Thus, it does not make sense to evaluate the asymptotic SSIM-RDF of Theorem 3 at large distortions.

V. DISCUSSION
The information-theoretic high-resolution RDF characterized by Theorem 3 constitutes a lower bound on the operationally achievable minimum rate for a givem SSIM distortion value.As discussed in [17], achieving the high-resolution RDF could require the use of optimal companding, which may not be feasible in some cases.Thus, the questions of whether the RDF obtained in Theorem 3 is achievable, and how to achieve it, remain open.Nevertheless, we can obtain a loose estimate of how close a practical coding scheme could get to the high-resolution SSIM-RDF by evaluating the operational performance of, for example, the baseline JPEG. Figure 2, shows the operational RDF for the JPEG coder used on the Lena image and using block sizes of 8 × 8.For comparison, we have also shown the SSIM-RDF.It may be noticed that the operational curve is up to 2 bpp above the corresponding SSIM-RDF.5The gap between the SSIM-RDF and the operational RDF based on JPEG encoding as can be observed in Figure 2, can be explained by the following observations: First, the JPEG coder aims at minimizing a frequency weighted MSE rather than maximizing the SSIM index.Second, JPEG is a practical algorithm with reduced complexity and is therefore not rate-distortion optimal even for the weighted MSE.Third, the differential entropy as well as the expectation of the log of the determinant of the sensitivity matrix are empirically foundbased on a finite amount of image data.Thus, they are only estimates of the true values.Finally, the SSIM-RDF becomes exact in the asymptotic limit where the coding rate diverges towards infinity (i.e., for small distortions).At finite coding rates, it is an approximation.Nevertheless, within these limitations, the numerical evaluation of the SSIM-RDF presented here suggests that significant compression gains could be obtained by an SSIM-optimal image coder, at least at high rate regimes.To obtain further insight into this question, the corresponding RDF under MSE distortion (MSE-RDF) for the Lena image is shown in Figure 3.We can see that the excess rate of JPEG with respect to the MSE-RDF at high rates is not greater than 1.4 bpp.This suggests that a JPEG-like algorithm aimed at minimizing SSIM distortion could reduce at least a fraction of the bit-rate gap seen in Figure 2.
It is interesting to note that in the MSE case, we have B(x) = I, which implies that log 2 (| det(B(x))|) = 0. Thus, the difference between the SSIM-RDF and the MSE-RDF, under high-resolution assumptions, is constant (e.g., independent of the bit-rate).In fact, if the MSE is measured per dimension, then the rate difference is given by the values in Table I, i.e., 1 2n E[log 2 (det(B(X)))] + log 2 (N ).It follows that the SSIM-RDF is simply a shifted version of the MSE-RDF at high resolutions.Moreover, the gap between the curves illustrates the fact that, in general, a representation of an image which is MSE optimal is not necessarily also SSIM optimal.

VI. CONCLUSIONS
We have shown that, under high-resolution assumptions, the RDF for a range of natural images under the commonly used SSIM index, has a simple form.In fact, the RDF only depends upon the differential entropy of the source image as well as the expected value of a function of the sensitivity matrix of the image.Thus, it is independent of any specific coding architecture.Moreover, we also provided a simple method to estimate the SSIM-RDF in practice for a given image.Finally, we compared the operational performance of the baseline JPEG image coder to the SSIM-RDF and showed by approximate numerical evaluations that potentially significant perceptual rate-distortion improvements could be obtained by using SSIM-optimal encoding techniques.

APPENDIX PROOF OF THEOREM 2
We need to show that the second-order terms of the Taylor series of d(x, y) are dominating in the high-resolution limit where y → x.In order to do this, we show that the Taylor series coefficients of the zero-and first-order terms vanish whereas the coefficients of the second-and third-order terms are non-zero.Then we upper bound the remainder due to approximating d(x, y) by its second-order Taylor series.This upper bound is established via the third-order partial derivatives of d(x, y).We finally show that the second-order terms decay more slowly towards zero than the remainder as y tends to x. and let h = f g.It follows that d(x, y) = 1 − h and we note that the secondorder partial derivatives with respect to yi and yj for any i, j, are given by

Let us define
, it is easy to show that ∂f ∂y i y=x = ∂g ∂y i y=x = 0, ∀i.Thus, the coefficients of the zero-and first-order terms of the Taylor series of d(x, y) are zero.Moreover, it follows from (8) that ∂ 2 h ∂y i ∂y j y=x = ∂ 2 f ∂y i ∂y j y=x + ∂ 2 g ∂y i ∂y j y=x , ∀i, j.With this, and after some algebra, it can be shown that We now let h (m) denote the mth partial derivative of h with respect to some m variables and note that from Leibniz generalized product rule [18] it follows that h (3) = gf (3) +3g (1) f (2) +3g (2) f (1) +g (3) f .When evaluated at y = x this reduces to h (3) |y=x = f (3) |y=x + g (3) |y=x, since f (1) |y=x and g (1) |y=x are both zero.For the thirdorder derivatives of f , we have: ∀i, j, k, Moreover, if i = j = k and i = k we obtain whereas if any two indices are equal, e.g., i = j = k, we obtain Finally, if i = j = k, we obtain Let B be an n-dimensional ball of radius ǫ centered at x, let ξ = y − x, and let T2(ξ) be the second-order Taylor series of d(x, x + ξ) centered at x (i.e., at ξ = 0).It follows that where B(x) is given by half the second-order partial derivatives of d(x, y), that is (see (9)) which has full rank and is well defined for 1 < n < ∞.This can be rewritten as B(x) = a(x)I + b(x)J, (16) where I is the identity matrix, J is the all-ones matrix, where λmin = min{λi} n−1 i=0 = min{a(x) + nb(x), a(x)} > 0, which implies that B(x) is positive-definite.
On the other hand, it is known from Taylor's theorem that for any y ∈ B, the remainder R2(ξ), where i.e., φ is upper bounded by the supremum over the set of third-order coefficients of the Taylor series of h.Since for real images, the pixel values are finite, and since Ci > 0, i = 1, 2, it follows from ( 10) -( 13) that the third-order derivatives are uniformly bounded and φ is therefore finite.Moreover, for all ξ such that ξ 2 ≤ ǫ, it follows using ( 14), (19), and (21) that  21) runs over all possible combinations of third-order partial derivatives of a vector of length n, i.e., i,j,k 1 = n 3 .Furthermore, (24) follows by use of (19) and the fact that |ξi| 3 < ξ 3 .Finally, (25) follows from the fact that φ is bounded by (22).Since the limit of (25) exists and is zero, we deduce that the second-order terms of the Taylor series of d(x, y) are asymptotically dominating as y tends to x.This completes the proof.

Fig. 2 .
Fig. 2. Operational RDF using the JPEG coder on the Lena image under the similarity measure d(x, y) = 1 − SSIM (x, y) for block size 8×8.For comparison we have also shown the high-resolution SSIM-RDF (thin line).

Fig. 3 .
Fig. 3. Operational RDF using the JPEG coder on the Lena image under the MSE distortion measure.For comparison we have also shown the highresolution MSE-RDF (thin line).The horizontal axes on the top and the bottom show the PSNR and MSE, respectively.