Open Access

An iterative enhanced super-resolution system with edge-dominated interpolation and adaptive enhancements

EURASIP Journal on Advances in Signal Processing20152015:9

https://doi.org/10.1186/s13634-014-0190-x

Received: 1 October 2014

Accepted: 29 December 2014

Published: 1 February 2015

Abstract

For super-resolution (4K × 2K) displays, super-resolution technologies, which can upsample videos to higher resolution and achieve better visual quality, become more and more important currently. In this paper, an iterative enhanced super-resolution (IESR) system which is based on two-pass edge-dominated interpolation, adaptive enhancement, and adaptive dithering techniques is proposed. The two-pass edge-dominated interpolation with a simple and regular kernel can sharpen visual quality while the adaptive enhancement can provide high-frequency perfection and the adaptive dithering conveys naturalization enhancement such that the proposed IESR system achieves better peak signal-to-noise ratio (PSNR) and exhibits better visual quality. Experimental results indicate that the proposed IESR system, which improves PSNR up to 28.748 dB and promotes structural similarity index measurement (SSIM) up to 0.917611 in averages, is better than the other existing methods. Simulations also exhibit that the proposed IESR system acquires lower computational complexity than the methods which achieve similar visual quality.

Keywords

Low-resolutionSuper-resolutionTwo-pass dominated-edge interpolationAdaptive enhancementAdaptive dithering

1 Introduction

Currently, the super-resolution displays with 4K × 2K pixels are vigorously available in the commercial market; however, the existing TV programs are mostly with either standard definition (SD) with 640 × 480 or high definition (HD) with 2K × 1K resolution. In other words, there are almost no super-resolution programs to match up with 4K × 2K TV displays. Thus, the super-resolution technologies, which can upsample SD or HD videos to higher resolution, become more and more important for current applications. Super-resolution (SR) is a technique to recover a higher resolution image from a given low-resolution (LR) image. Simply, the SR algorithm could be treated as an interpolation method to enhance the resolution of images or videos. The interpolated image usually could still lose some detailed information. For 4K × 2K TV displays, it is noted that the SR algorithms should consider real-time implementation issues.

In the literatures, the SR algorithms can be classified into interpolation-based, reconstruction-based, and learning-based approaches to solve the problem of recovering detailed information extracted from the low-resolution image. The interpolation-based approach involves in the prediction of the unknown pixels by filtering processes. Based on the concept of ideal low-pass filtering, the interpolation methods [1] always need to consider the balance of computational complexity and reconstruction quality. The linear, bi-cubic, and cubic spline interpolations [2] are the possible means for reducing the complexity. To prevent filtering pixels across edges, numerous edge-directed interpolation methods are proposed [3-11].

The reconstructed-based approach generates high-resolution images by exploiting the information from a set of successive low-resolution images in the same scene but with sub-pixel displacements. In the wavelet domain [12], the LR image is considered as the lower sub-band of the wavelet-transformed high-resolution (HR) image. However, they are difficult in estimating the unknown coefficients of the other three higher wavelet sub-bands due to their independencies. Instead of the frequency domain methods, the most contemporary methods turn their attentions to solve the problem in the spatial domain [13]. The back projection algorithm iteratively projects the error between the simulated and input LR images to estimate HR error by iteratively minimizing the reconstruction error [14]. However, many jaggy artifacts along the edges may affect the quality of images. To reduce these artifacts, Dong et al [15] employed the nonlocal image redundancy to improve the quality of SR images. In the same time, it brings heavy computation complexity for updating the reconstruction error in each step. The projection onto convex sets (POCS) algorithms [16,17] applied to the input LR images could increase the solution of the element on the convex set. The maximum a posteriori (MAP) methods [18,19] adopt the associated probability of target high-resolution images to form a prior probability to refer the solution based on Bayesian inference.

The learning-based approaches [20,21] attempt to capture the correlation between LR and HR patches to exploit the redundant high-frequency information which is remained in HR training samples. Although these algorithms need a large number of databases to store millions of LR and HR patch pairs, the quality of reconstructed images can be improved even that the magnificent factor is large.

The dictionary learning-based denoising approach [22] used taxonomy based on image representations for a better understanding of state-of-the-art image denoising techniques. The multiresolution structure and sparsity of wavelets are used for nonlocal dictionary learning in each decomposition level of the wavelets [23]. The classification-based least squares trained filters on picture quality improvement algorithms are suggested [24].

In this paper, we propose an iterative enhanced super-resolution (IESR) system, which is based on two-pass edge-dominated interpolation by adding adaptive enhancement and dithering mechanisms. The proposed (IESR) system is based on iterative back projection concept [14]; however, the proposed two-pass edge-dominated interpolation consists of two adaptive fixed-structure filters. Besides, we further include the adaptive enhancement and adaptive dithering units to improve the quality of the HR image in iterative cycles. The overview of the proposed super-resolution system is addressed in Section 2. The edge-dominated interpolation methods will be described in Section 3 while the adaptive enhancement and adaptive dithering algorithms will be present in Section 4. In Section 5, the experimental results for verifying the proposed algorithms in comparison to the well-known super-resolution algorithms are demonstrated. Finally, the conclusions about this paper are addressed in Section 4.

2 Overview of the proposed super-resolution system

The block diagram of the proposed super-resolution system is shown in Figure 1. The flow diagram of the proposed super-resolution system is exhibit in Figure 2. The detailed descriptions of the proposed system are shown as the follows.
Figure 1

Block diagram of the proposed IESR system.

Figure 2

Flowchart of the proposed IESR system.

For later performance comparisons, as shown in Figure 1, the LR image, I L as the test image is obtained by down sampling a HR image, I H . For down sampling, we adopt Lanczos2 low-pass filtering to eliminate high-frequency components to prevent from the aliasing effect. The reconstruction kernel of Lanczos one-dimensional (1-D) low-pass filter is given as:
$$ L(x)\kern0.5em =\kern0.5em \left\{\begin{array}{c}\hfill nsin\left(\pi x\right) \sin \left(\pi x/n\right)/{\pi}^2{x}^2,\hfill \\ {}\hfill 0,\hfill \end{array}\begin{array}{c}\hfill \kern0.36em \mathrm{if}\kern0.24em 0\kern0.5em <\kern0.5em \left|x\right|\kern0.5em <a\hfill \\ {}\hfill \mathrm{otherwise},\hfill \end{array}\right. $$
(1)
where n is typically two or three. As shown in Figure 3, Lanczos2 filter is an approximation of the ideal low-pass filter, which is specified by sinc function.
Figure 3

The reconstruction kernel of Lanczos2 sinc low-pass filter.

For half-pixel interpolation, the 1-D Lanczos2 filtering process can be expressed by:
$$ {l}_0\kern0.5em =\kern0.75em -\kern0.5em 0.032\ {s}_{-2}\kern0.5em +\kern0.5em 0.284{s}_{-1}\kern0.5em +\kern0.5em 0.496{s}_0\kern0.5em +\kern0.5em 0.284{s}_1\kern0.5em -\kern0.5em 0.032{s}_2, $$
(2)
where s-2, s-1, s 0, s 1, and s 2 denote the five consecutive inputs while l 0 represents the co-located low-pass filtering result at s 0. In (2), the coefficients are calculated by Lanczos kernel to predict half pixels through sinc functions. It is noted that the symmetrical extension of image pixels is conducted for the image borders while applying low-pass filtering. With the about filtering process of the SR image, we then perform the two-to-one down sampling for horizontal and vertical to get the LR image, which will be treated as the input for testing SR algorithms while the original HR image will be used for performance evaluation of the proposed and other super-resolution algorithms. It is noted that the LR images might not have their related HR image and the LR image obtained from the HR image could be different from the data generation exhibited in Figure 2 normally.
In the proposed super-resolution system, the input low-resolution image, I L is first upsampled by the proposed two-pass edge-dominate interpolation (TEI) to become the initially restored HR image, \( {\tilde{I}}_{(0)}^H \) at k = 0. For the kth (k > 0) iteration, the adaptive enhancement and dither noise, \( {D}_{(k)}^H \) is added to the previous (k − 1)th iteration result to obtain the kth restored HR image as:
$$ {\tilde{I}}_{(k)}^H={\tilde{I}}_{\left(k-1\right)}^H+{D}_{(k)}^H $$
(3)

The detailed descriptions about the adaptive enhancement and adaptive dithering to obtain \( {D}_{(k)}^H \) will be addressed in Section 4.

Similar to data generation, the kth LR image, \( {I}_{(k)}^L \) is computed by down sampling of the kth restored HR image, \( {\tilde{I}}_{(k)}^H \) after Lanczos2 sinc filtering. The restored LR error image with respect to the low-resolution image, I L is computed by:
$$ {e}_{(k)}^L\kern0.5em ={I}_{(k)}^L\kern0.5em -{I}^L. $$
(4)

If the restored error, \( {e}_T\kern0.5em =\kern0.5em \left\Vert {e}_{(k)}^L\right\Vert \) is less than a predetermined threshold ε or the number of iterations, k is equal to the maximum limited number M, the whole iterative super-resolution process will be terminated. Thus, the final restored HR image, \( {\tilde{I}}_{(k)}^H \) will be the output HR image.

If the iterative process is not terminated, the LR error image \( {e}_{(k)}^L \) is then upsampled by the proposed two-pass edge-dominated interpolation to become the HR error image, \( {e}_{(k)}^H \). From \( {e}_{(k)}^H \), we can estimate the enhancement HR image, \( {E}_{(k)}^H \), and its dithering image, \( {D}_{(k)}^H \) for the next iteration. The proposed TEI will be discussed in the next section.

3 Two-pass edge-dominated interpolation

The TEI is shown in Figure 4. To gain more accurate interpolation value, the YUV color space is adopted in the proposed method. The Y component, also called luma component, represents the details of the texture. Therefore, the edge-dominated weights of the Y component are used to perform the interpolations of U and V components. In other words, the dominated weights used for Y component will be directly adopted for U and V components to save the computation and raise the interpolation performance with respect to the TEI performed in the RGB space.
Figure 4

The proposed two-pass edge-dominated interpolation. Which is the simplified notation of the proposed TEI-based up sampling depicted in Figure 1.

Figure 5a shows the two-pass edge-dominated interpolation, which first performs diagonal interpolation and then vertical-horizontal interpolation. Figure 5b exhibits relationships among the known (black) pixels, the first-pass interpolated (yellow) pixels, and the second-pass interpolated (white) pixels.
Figure 5

Two-pass edge-dominated interpolation. (a) Flow diagram and (b) the known and interpolated pixels handled in the first and second passes.

The first pass is to perform diagonal pixel (yellow) interpolation by using the edge-dominated concept. For each target (yellow) pixel x 0, there are four adjacent known (black) pixels, x 1, x 2, x 3, and x 4 from the LR image used to estimate it as:
$$ {x}_0\kern0.5em =\kern0.5em {\displaystyle {\sum}_{m=1}^4\kern0.5em {a}_m{x}_m/\kern0.5em {\displaystyle {\sum}_{m=1}^4\kern0.5em {a}_m}}, $$
(5)
where a k for k = 1, 2, 3, and 4 are called as the edge-dominated weights for the target pixel. The edge-dominated weights can be computed by:
$$ {a}_m\kern0.5em =\kern0.5em {e}^{-{s}_m/c},\kern0.5em \mathrm{f}\mathrm{o}\mathrm{r}\kern0.5em m\kern0.5em =\kern0.5em 1,\kern0.5em 2,\kern0.5em 3,\kern0.5em \mathrm{and}\ 4, $$
(6)
where c is a control parameter and is set to 32 in this paper. To retrieve the edge information, we should compute the edge sensitivities s m first. As shown in Figure 6, the edge sensitivities s m are suggested as:
Figure 6

Diagonal interpolation and its associated weights and edge sensitivities: target pixel (red color) and known (black color) pixels.

$$ {s}_1\kern0.5em =\kern0.5em \left|{x}_1\kern0.5em -\kern0.5em {x}_3\right|\kern0.5em +\kern0.5em \gamma \left|{x}_5\kern0.5em -\kern0.5em {x}_4\right|\kern0.5em +\kern0.5em \gamma \left|{x}_6\kern0.5em -\kern0.5em {x}_2\right|, $$
(7)
$$ {s}_2\kern0.5em =\kern0.5em \left|{x}_2\kern0.5em -\kern0.5em {x}_4\right|\kern0.5em +\kern0.5em \gamma \left|{x}_7\kern0.5em -\kern0.5em {x}_1\right|\kern0.5em +\kern0.5em \gamma \left|{x}_8\kern0.5em -\kern0.5em {x}_3\right|, $$
(8)
$$ {s}_3\kern0.5em =\kern0.5em \left|{x}_1\kern0.5em -\kern0.5em {x}_3\right|\kern0.5em +\kern0.5em \gamma \left|{x}_2\kern0.5em -\kern0.5em {x}_9\right|\kern0.5em +\kern0.5em \gamma \left|{x}_4\kern0.5em -\kern0.5em {x}_{10}\right|, $$
(9)
$$ {s}_4\kern0.5em =\kern0.5em \left|{x}_2\kern0.5em -\kern0.5em {x}_4\right|\kern0.5em +\kern0.5em \gamma \left|{x}_1\kern0.5em -\kern0.5em {x}_{12}\right|\kern0.5em +\kern0.5em \gamma \left|{x}_3\kern0.5em -\kern0.5em {x}_{11}\right|, $$
(10)
where γ is a parameter to control the weights of reference paired pixels. If any subtracting paired pixels across an edge, the difference of two pixels stated in (7), (8), (9), or (10) becomes large. Physically, the larger the edge sensitivity s m is, the less similarity to x m will be.
After the first-pass interpolation, the rest (white) pixels are calculated in the second step. Of course, we will not only use the pixels (black) of the original LR image but also the pixels (yellow) obtained from the first step. As shown in Figure 7, to find the target pixel (the center white pixel), y 0 in either horizontal or vertical interpolation, we also perform the edge-dominated interpolation as:
Figure 7

Vertical/horizontal interpolation and its associated weights and the edge sensitivities. Vertical/horizontal interpolation and its associated weights and the edge sensitivities of the target pixel y 0 (white) and known (black and yellow) pixels.

$$ {y}_0\kern0.5em =\kern0.5em \left\{\begin{array}{c}\hfill \left({b}_1{y}_1\kern0.5em +\kern0.5em {b}_3{y}_3\right)/\left({b}_1\kern0.5em +\kern0.5em {b}_3\right),\mathrm{if}\kern0.4em {s}_h\kern0.5em >\kern0.5em {s}_v\hfill \\ {}\hfill \left({b}_2{y}_2\kern0.5em +\kern0.5em {b}_4{y}_4\right)/\left({b}_2\kern0.5em +\kern0.5em {b}_4\right),\kern1.60em \mathrm{if}\ {s}_h\kern0.5em <\kern0.5em {s}_v\hfill \\ {}\hfill {\displaystyle {\sum}_{n=1}^4{b}_n{y}_n/\kern0.5em {\displaystyle {\sum}_{n=1}^4{b}_n}},\kern2em \mathrm{otherwise}\hfill \end{array}\right., $$
(11)
where
$$ {b}_n\kern0.5em =\kern0.5em {e}^{-{D}_n/c},\kern0.75em n\kern0.5em =\kern0.5em 1,\kern0.5em 2,\kern0.5em 3,\kern0.5em 4, $$
(12)
and
$$ {D}_1\kern0.5em =\kern0.5em \left|{y}_1\kern0.5em -\kern0.5em {y}_3\right|\kern0.5em +\kern0.5em \left|{y}_5\kern0.5em -\kern0.5em {y}_4\right|\kern0.5em +\kern0.5em \left|{y}_6\kern0.5em -\kern0.5em {y}_2\right|, $$
(13)
$$ {D}_2\kern0.5em =\kern0.5em \left|{y}_4\kern0.5em -\kern0.5em {y}_2\right|\kern0.5em +\kern0.5em \left|{y}_1\kern0.5em -\kern0.5em {y}_7\right|\kern0.5em +\kern0.5em \left|{y}_3\kern0.5em -\kern0.5em {y}_8\right|, $$
(14)
$$ {D}_3\kern0.5em =\kern0.5em \left|{y}_1\kern0.5em -\kern0.5em {y}_3\right|\kern0.5em +\kern0.5em \left|{y}_4\kern0.5em -\kern0.5em {y}_{10}\right|\kern0.5em +\kern0.5em \left|{y}_2\kern0.5em -\kern0.5em {y}_9\right|, $$
(15)
$$ {D}_4\kern0.5em =\kern0.5em \left|{y}_4\kern0.5em -\kern0.5em {y}_2\right|\kern0.5em +\kern0.5em \left|{y}_{12}\kern0.5em -\kern0.5em {y}_1\right|\kern0.5em +\kern0.5em \left|{y}_{11}\kern0.5em -\kern0.5em {y}_3\right|, $$
(16)
The horizontal or vertical interpolation is determined by computation of s h and s v as shown in Figure 7. The horizontal and vertical sensitivities of y 0 can be respectively formulized as:
$$ {s}_h\kern0.5em =\kern0.5em \left|{y}_4\kern0.5em -\kern0.5em {y}_2\right|\kern0.5em +\kern0.5em \left|{y}_{12}\kern0.5em -\kern0.5em {y}_1\right|\kern0.5em +\kern0.5em \left|{y}_1\kern0.5em -\kern0.5em {y}_7\right|\kern0.5em +\kern0.5em \left|{y}_{11}\kern0.5em -\kern0.5em {y}_3\right|\kern0.5em +\kern0.5em \left|{y}_3\kern0.5em -\kern0.5em {y}_8\right|, $$
(17)
and
$$ {s}_v\kern0.5em =\kern0.5em \left|{y}_1\kern0.5em -\kern0.5em {y}_3\right|\kern0.5em +\kern0.5em \left|{y}_5\kern0.5em -\kern0.5em {y}_4\right|\kern0.5em +\kern0.5em \left|{y}_4\kern0.5em -\kern0.5em {y}_{10}\right|\kern0.5em +\kern0.5em \left|{y}_6\kern0.5em -\kern0.5em {y}_2\right|\kern0.5em +\kern0.5em \left|{y}_2\kern0.5em -\kern0.5em {y}_9\right|. $$
(18)

It is noted that the computation of edge sensitivities and horizontal/vertical sensitivities, which involves the sum of selected absolute differences, can be dramatically reduced if we could properly reuse the absolute differences.

4 Image enhancement and dithering algorithms

The most edge-directed interpolation algorithms [3-11] including the proposed TEI method can successfully enlarge the low-resolution images to super-resolution ones with less artifacts along the texture edges. However, the loss of high-frequency components cannot be properly restored by general interpolation algorithms. To recover high-frequency parts of the original image, in this section, we further propose to utilize the adaptive enhancement (AE) and adaptive dithering (AD) algorithms for the Y component to further improve the quality of HR images iteratively.

From the LR reconstruction error, \( {e}_{(k)}^L \) which is stated in (4), we can use the same two-pass edge-dominated interpolation to upsample \( {e}_{(k)}^L \) to the HR reconstruction error, \( {e}_{(k)}^H \) as shown in Figure 2. To eliminate the HR reconstruction error, of course, we should try to remove \( {e}_{(k)}^H \) from the restored HR image. It is noted that \( {e}_{(k)}^H \), which is recovered from the low-resolution error, only contains the low-frequency part of the HR reconstruction error.

In order to restore the high-frequency part, we suggest an adaptive high-pass image enhancement filter as:
$$ H(x)\kern0.5em =\kern0.5em 1\kern0.5em -\kern0.5em L(x)\kern0.5em =\kern0.5em \left\{\begin{array}{c}\hfill 1-n \sin \left(\pi x\right) \sin \left(\pi x/n\right)/{\pi}^2{x}^2,\hfill \\ {}\hfill 1,\hfill \end{array}\begin{array}{c}\hfill \kern0.36em \mathrm{if}\kern0.24em 0<\left|x\right|<a\hfill \\ {}\hfill \mathrm{otherwise},\hfill \end{array}\right. $$
(19)
which is the inverse of the reconstruction kernel of Lanczos sinc function. To obtain high-pass filtering, the associated Lanczos2 high-pass 1-D filtering result is given by:
$$ {h}_0\kern0.5em =\kern0.5em 0.032\ {s}_{-2}\kern0.5em -\kern0.5em 0.284{s}_{-1}\kern0.5em +\kern0.5em 0.504{s}_0\kern0.5em -\kern0.5em 0.284{s}_1\kern0.5em +\kern0.5em 0.032{s}_2, $$
(20)
where h 0 represents the co-located high-pass filtering result at s 0 . With symmetrical extension of image borders of \( {\tilde{I}}_{(k)}^H \), the high-pass part of the kth restored HR image, \( {h}_{(k)}^H \) can be retrieved.
Thus, the proposed adaptive enhancement algorithm is expressed by:
$$ {E}_{(k)}^H\kern0.5em =\kern0.5em \alpha (k){h}_{(k)}^H\kern0.5em -\kern0.5em \left(1\kern0.5em -\kern0.5em \alpha (k)\right){e}_{(k)}^H, $$
(21)
where α(k) is a decay function with k such that the high-frequency enhancement will be gradually reduced and the error compensation can be increased after iterations. The above HR enhancement algorithm can only recover the high-frequency components, whose magnitudes are partially distorted.
To further enhance the quality of the super-resolution image, we further add adaptive random Gaussian noise into \( {E}_{(k)}^H \) to become:
$$ {D}_{(k)}^H={E}_{(k)}^H+\beta (k){\sigma}_q{n}_{(k)}^H, $$
(22)
where \( \beta \)(k) is a decay dithering function in the kth iteration such that the dithering will be gradually reduced after iterations. In (22), σ q is the root mean square of \( {E}_{(k)}^H \) in a q × q window centered at the dithering pixel, and the size of window is set to three in this paper. By borrowing the MAP concept [18,19], the prior probability of the high-resolution images is based on Gaussian noise model, \( {n}_{(k)}^H=N\left(0,\;1\right) \) .If the termination is at the Mth iterations, the high-resolution image \( {\tilde{I}}_{(M)}^H \) will be the output of the proposed IESR system.

5 Simulation results

Figure 8 shows all the test images, whose resolutions are from 2,048 × 2,560 to 6,000 × 4,190. More details of the resolution for each image can be found in Table 1. In the experiments, the original HR images are downscaled by factor 2 to generate the LR images with Lanczos filtering. For comparisons, two objective measures, peak signal-to-noise ratio (PSNR) and structural similarity index measurement (SSIM) are used to evaluate the performances of the super-resolution algorithms. The PSNR is the ratio of the maximum power and the noise power, which is defined as:
Figure 8

Test images (read from left to right in rows). (a) N1.bmp, (b) N2.bmp, (c) N4.bmp, (d) N5.bmp, (e) N6.bmp, (f) N7.bmp, (g) N8.bmp, (h) QFHD_P01.bmp, (i) QFHD_P03.bmp, (j) QFHD_P04.bmp, and (k) world_statelite.bmp.

Table 1

Quality evaluation of the proposed methods in PSNR (dB)

Image

Resolution

TEI only

TEI + AE with α ( k )

TEI + AE + AD with β ( k )

Linear

Exponent

Linear

Exponent

N1

2,048 × 2,560

28.692

29.222

29.221

29.248

29.250

N2

2,048 × 2,560

23.287

23.745

23.756

23.784

23.786

N5

2,048 × 2,560

27.689

28.226

28.226

28.259

28.260

N4

2,560 × 2,048

25.866

26.421

26.448

26.509

26.509

N6

2,560 × 2,048

35.759

36.356

36.374

36.381

36.380

N7

2,560 × 2,048

24.804

25.277

25.279

25.308

25.309

N8

2,560 × 2,048

21.040

21.393

21.412

21.438

21.441

QFHD_P01

3,840 × 2,160

35.778

36.599

36.606

36.639

36.643

QFHD_P03

3,840 × 2,160

25.792

26.434

26.446

26.478

26.482

QFHD_P04

3,840 × 2,160

35.359

36.010

36.060

36.066

36.074

world_satellite

6,000 × 4,190

25.685

25.948

26.017

26.080

26.090

Average

28.159

28.694

28.711

28.739

28.748

$$ \mathrm{PSNR}\kern0.5em =10 \log {}_{10}\frac{255^2}{\mathrm{MSE}}, $$
(23)
where
$$ \mathrm{M}\mathrm{S}\mathrm{E}\kern0.5em =\kern0.5em \frac{{\displaystyle {\sum}_{M,N}{\left[{\tilde{I}}_{(M)}^H\left(x,y\right)\kern0.5em -\kern0.5em {I}^H\left(x,y\right)\right]}^2}}{P\times N}, $$
(24)
which denotes the minimum square error between computed super-resolution image, \( {\tilde{I}}_{(M)}^H \) and the original super-resolution image, I H (x, y), where M and N are the sizes of row and column, respectively, and (x, y) means the position of the pixel. The SSIM is another measurement system to compare the similarity of two images, which is defined as:
$$ \mathrm{SSIM}\left(x,y\right)\kern0.5em =\kern0.5em {\scriptscriptstyle \frac{\left(2{\mu}_c{\mu}_o\kern0.5em +\kern0.5em {c}_1\right)\left(2{\sigma}_{oc}\kern0.5em +\kern0.5em {c}_2\right)}{\left({\mu}_c^2\kern0.5em +\kern0.5em {\mu}_o^2\kern0.5em +\kern0.5em {c}_1\right)\left({\sigma}_c^2\kern0.5em +\kern0.5em {\sigma}_o^2\kern0.5em +\kern0.5em {c}_2\right)}}, $$
(25)
where μ c and μ o denote the average values and \( {\sigma}_c^2 \) and \( {\sigma}_o^2 \) are the variances of \( {\tilde{I}}_{(M)}^H \) and I H (x, y), respectively, while σ oc exhibits their covariance. The variables c 1 and c 2, which are used to avoid instability with weak denominator, are given by:
$$ {c}_1={\left({k}_1L\right)}^2\kern0.24em {c}_2={\left({k}_2L\right)}^2, $$
(26)
where L is dynamic range of the pixel values, like 255 for 8 bits. In (26), k 1 and k 2 are set to 0.01 and 0.03, respectively.

To exhibit the effectiveness of the proposal TEI, the AE, and AD algorithms are tested by various combinations, which are the TEI only, the ‘TEI + AE,’ and the ‘TEI + AE + AD’. For AE and AD algorithms, we suggest two decay functions. For the kth iteration, the linear decaying function is given as α 1(k) = 0.5 − 0.1235(k − 1) while the power-of-two exponential decay function is defined as α 2(k) = 2− k .

For evaluation of ‘TEI only,’ ‘TEI + AE,’ and ‘TEI + AE + AD’ combinations, Tables 1 and 2 show PSNR (dB) and SSIM performances for these three combinations, respectively. All the proposed two-pass edge-dominated interpolation, adaptive enhancement, and adaptive dithering could help to achieve better performances in the proposed IESR system. The experimental results show that the power-of-two exponential decay function achieves better performance and less computation since it only involve bit-shift operations.
Table 2

Quality evaluation of the proposed methods with SSIM

Image

Resolution

TEI only

TEI + AE with α ( k )

TEI + AE + AD with β ( k )

Linear

Exponent

Linear

Exponent

N1

2,048 × 2,560

0.924244

0.933028

0.933031

0.933280

0.933574

N2

2,048 × 2,560

0.869639

0.884614

0.885038

0.885420

0.885351

N5

2,048 × 2,560

0.899536

0.908142

0.908139

0.908400

0.908514

N4

2,560 × 2,048

0.909207

0.917733

0.918449

0.918500

0.918058

N6

2,560 × 2,048

0.955273

0.958016

0.958051

0.958057

0.958062

N7

2,560 × 2,048

0.817085

0.833128

0.832990

0.833602

0.834204

N8

2,560 × 2,048

0.824425

0.842889

0.843691

0.844313

0.844618

QFHD_P01

3,840 × 2,160

0.980694

0.983344

0.983406

0.983439

0.983475

QFHD_P03

3,840 × 2,160

0.909168

0.923118

0.922826

0.923674

0.923897

QFHD_P04

3,840 × 2,160

0.975670

0.978389

0.978555

0.978505

0.978391

world_satellite

6,000 × 4,190

0.913758

0.923650

0.925000

0.925313

0.925573

Average

0.907154

0.916914

0.917198

0.917500

0.917611

Tables 3 and 4 show the PSNR and SSIM performances achieved by the proposed and the well-known SR algorithms, respectively. We observed that the proposed (TEI + AE + AD) method achieves the PSNR up to 36.643 dB and SSIM up to 0.983475, while the averaged PSNR is 28.748 dB and the averaged SSIM reaches to 0.917611. The proposed IESR system outperforms the other well-known super-resolution algorithms. Although the iterative back projection (IBP) [14] and the nonlocal back projection (NBP) algorithms [15] achieve similar results, the proposed method takes less execution time as shown in Table 5. For subjective comparisons, Figures 9 and 10 show the cropped super-resolution (SR) images for N1 and QFHD_P01 test images. The cropped images also show exhibit that the proposed methods outperform the exiting algorithms in visual quality.
Table 3

Performance comparison of the proposed and the well-known SR algorithms in term of PSNR (dB)

Image

Resolution

NNI

Bilinear

Bi-cubic

Learn

IBP

NBP

Proposed method

N1

2,048 × 2,560

25.270

25.840

26.146

25.894

28.917

29.180

29.250

N2

2,048 × 2,560

20.194

20.764

21.049

20.930

23.504

23.513

23.786

N5

2,048 × 2,560

24.784

25.293

25.643

25.759

27.805

27.980

28.260

N4

2,560 × 2,048

22.649

23.250

23.661

23.831

26.008

26.233

26.509

N6

2,560 × 2,048

31.849

32.634

33.021

33.128

35.923

35.930

36.380

N7

2,560 × 2,048

22.113

22.560

22.850

22.843

24.966

24.914

25.309

N8

2,560 × 2,048

19.048

19.321

19.646

19.723

21.321

21.239

21.441

QFHD_P01

3,840 × 2,160

30.700

31.605

32.128

32.108

36.036

35.656

36.643

QFHD_P03

3,840 × 2,160

22.607

23.071

23.624

23.881

26.040

25.962

26.482

QFHD_P04

3,840 × 2,160

29.424

30.531

31.118

31.187

35.811

35.238

36.074

world_satellite

6,000 × 4,190

22.488

23.215

23.508

23.495

25.980

25.830

26.090

Average

24.648

25.280

25.672

25.707

28.392

28.286

28.748

Table 4

Performance comparison of the proposed and the well-known SR algorithms in term in term of SSIM

Images

Resolution

NNI

Bilinear

Bi-cubic

Learn

IBP

NBP

Proposed method

N1

2,048 × 2,560

0.87807

0.87695

0.89103

0.89878

0.92988

0.92936

0.93357

N2

2,048 × 2,560

0.79524

0.79704

0.81697

0.83103

0.87577

0.87184

0.88535

N5

2,048 × 2,560

0.86975

0.86842

0.87793

0.88466

0.90397

0.90015

0.90851

N4

2,560 × 2,048

0.87034

0.87128

0.88325

0.89372

0.91185

0.90992

0.918056

N6

2,560 × 2,048

0.94077

0.94292

0.94631

0.94561

0.95827

0.95437

0.95806

N7

2,560 × 2,048

0.76907

0.76136

0.77935

0.79448

0.82491

0.81377

0.83420

N8

2,560 × 2,048

0.76006

0.74829

0.77434

0.79841

0.83508

0.82489

0.84462

QFHD_P01

3,840 × 2,160

0.95470

0.96237

0.96766

0.96913

0.98298

0.97873

0.98348

QFHD_P03

3,840 × 2,160

0.84092

0.84207

0.86495

0.88436

0.91576

0.90496

0.92390

QFHD_P04

3,840 × 2,160

0.94705

0.95413

0.96020

0.96187

0.97881

0.97246

0.97839

world_satellite

6,000 × 4,190

0.84918

0.84998

0.86774

0.88021

0.91977

0.90765

0.92557

Average

0.86138

0.86135

0.87543

0.88566

0.91246

0.90619

0.91761

Table 5

Computational time (seconds) required by the proposed algorithm and the nonlocal back project methods

Image

Resolution

IBP

NBP

Proposed method

N1

2,048 × 2,560

365.221388

56,902.158

38.811761

N2

2,048 × 2,560

366.667311

49,838.346

37.482078

N5

2,048 × 2,560

361.005792

57,614.404

38.188074

N4

2,560 × 2,048

367.104573

59,545.188

39.593662

N6

2,560 × 2,048

360.240996

64,205.242

39.867318

N7

2,560 × 2,048

354.380109

58,801.846

38.624665

N8

2,560 × 2,048

355.603370

56,018.748

39.447973

QFHD_P01

3,840 × 2,160

562.299157

169,538.659

63.022658

QFHD_P03

3,840 × 2,160

559.187032

83,501.423

64.054184

QFHD_P04

3,840 × 2,160

569.389138

59,545.188

67.990431

world_satellite

6,000 × 4,190

1,933.869665

36,711.171

206.892703

Figure 9

Cropped SR (N1) images. (a) Ground truth and the reconstructed images by (b) NNI, (c) bilinear, (d) bi-cubic, (e) spare representation, (f) IBP, (g) NLIBP, and (h) the proposed methods.

Figure 10

Cropped (QFHD_P01) SR images. (a) Ground truth and the reconstructed images by (b) NNI, (c) bilinear, (d) bi-cubic, (e) spare representation, (f) IBP, (g) NLIBP, and (h) the proposed methods.

6 Conclusions

In this paper, a super-resolution algorithm based on edge-dominated interpolation adaptive enhancement and adaptive dithering is proposed. The edge-dominated interpolation can overcome the artifacts of interpolation such that we could have smoother results along the edges. The adaptive image enhancement algorithm can improve the distorted high-frequency parts while the adaptive dithering method can recover the loss of high-frequency components. In this paper, we only use Y component for edge detection, adaptive enhancement, and adaptive dithering such that we can reduce computation time and achieve better quality. The experimental results show that the proposed algorithm achieves PSNR up to 28.748 dB and SSIM up to about 0.918 in average while the computational time is also reasonably low for practical applications. Due to local data usage and regular structures in computation, the proposed super-resolution system is suitable for VLSI implementation.

Declarations

Acknowledgements

This work was supported in part by the Ministry of Economic Affairs and Ministry of Science and Technology of Taiwan, under Contract 103-EC-17-A-02-S1-201 and Grant MOST 103-2221-E-006-109-MY3.

Authors’ Affiliations

(1)
Institute of Computer and Communication Engineering, Department of Electrical Engineering, National Cheng Kung University
(2)
Department of Electrical Engineering, National Cheng Kung University

References

  1. E Maeland, On the comparison of interpolation methods. IEEE Trans Med Imaging 7(7), 213–217 (1988)View ArticleGoogle Scholar
  2. HS Hou, H Andrews, Cubic splines for image interpolation and digital filtering. IEEE Trans Acoustic, Speech Signal Process 26(6), 508–517 (1978)View ArticleMATHGoogle Scholar
  3. Y. Yun, J. Bae, and J. Kim, Adaptive multidirectional edge directed interpolation for selected edge regions, Proc. of Region 10 Conference (TENCON 2011), pp. 385-388, Bali, Nov. 2011.Google Scholar
  4. D Zhou, X Shen, W Dong, Image zooming using directional cubic convolution interpolation. IET Image Process 6(6), 627–634 (2012)View ArticleMathSciNetGoogle Scholar
  5. D Zhang, X Wu, An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8), 2226–2238 (2006)View ArticleGoogle Scholar
  6. X Li, NT Orchard, New edge-directed interpolation. IEEE Trans Image Process 10(10), 1521–1527 (2001)View ArticleGoogle Scholar
  7. J Allebach, PW Wong, Edge-directed interpolation. Proc Int Conf Image Process 3, 707–710 (1996)View ArticleGoogle Scholar
  8. C.-S. Wong and W.-C. Siu, Adaptive directional window selection for edge-directed Interpolation, Proc. of 19th International Conference on Computer Communications and Networks, vol., no., pp.1-6, Aug. 2010.Google Scholar
  9. C-S Wong, W-C Siu, Further improved edge-directed interpolation and fast EDI for SDTV to HDTV conversion, in Proc. of European Signal Processing Conference, 2010, pp. 23–27Google Scholar
  10. W-S Tam, CW Kok, S Wan-Chi, Modified edge-directed interpolation for images. J Electron Imaging 19, 013011 (2010)View ArticleGoogle Scholar
  11. N Asuni, A Giachetti, Accuracy improvements and artifacts removal in edge based image interpolation, in Proc. of the 3rd Int. Conf. on Computer Vision Theory and Applications, 2008Google Scholar
  12. SC Tai, TM Kuo, CH Lao, TW Liao, A fast algorithm for single image super resolution in both wavelet and spatial domain, in Proc. of International Symposium on Computer, Consumer and Control, 2012, pp. 702–705View ArticleGoogle Scholar
  13. H Su, L Tang, Y Wu, D Tretter, J Zhou, Spatially adaptive block-based super-resolution. IEEE Trans Image Process 21(3), 1031–1045 (2012)View ArticleMathSciNetGoogle Scholar
  14. B Zhao, Z Gan, Y Zhang, F Liu, H Wang, Novel back-projection framework for single image super-resolution, in Proc. of International Conf. on Signal Processing, 2012, pp. 894–898Google Scholar
  15. W Dong, L Zhang, G Shi, X Wu, Nonlocal back projection for adaptive image enlargement, in International Conf. on Image Processing, 2009, pp. 349–352Google Scholar
  16. C Fan, J Zhu, J Gong, C Kuang, POCS Super-resolution sequence image reconstruction based on improvement approach of Keren registration method. Proc Inter Conf Intell Syst Des Appl 2, 333–337 (2006)Google Scholar
  17. AJ Patti, Y Altunbasak, Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Trans Image Process 10(1), 179–186 (2002)View ArticleGoogle Scholar
  18. S Belekos, N Galatsanos, A Katsaggelos, Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans Image Process 19(6), 1451–1464 (2010)View ArticleMathSciNetGoogle Scholar
  19. LC Pickup, DP Capel, SJ Roberts, A Zisserman, Bayesian methods for image super-resolution. Comput J 52(1), 101–113 (2007)View ArticleGoogle Scholar
  20. WT Freeman, EC Pasztor, OT Carmichael, Learning low level vision. Int J Comput Vis 40(1), 25–47 (2000)View ArticleMATHGoogle Scholar
  21. J Yang, J Wright, T Huang, Y Ma, Image super-resolution via sparse representation. IEEE Trans Image Process 19(11), 2861–2873 (2010)View ArticleMathSciNetGoogle Scholar
  22. L Shao, R Yan, X Li, Y Liu, From heuristic optimization to dictionary learning: a review and comprehensive comparison of image denoising algorithms”. IEEE Trans Cybern 44(7), 1001–1013 (2014)View ArticleGoogle Scholar
  23. R Yan, L Shao, Y Liu, Nonlocal hierarchical dictionary learning using wavelets for image denoising. IEEE Trans Image Process 22(12), 4689–4698 (2013)View ArticleMathSciNetGoogle Scholar
  24. L Shao, H Zhang, G de Haan, An overview and performance evaluation of classification-based least squares trained filters. IEEE Trans Image Process 17(10), 1772–1782 (2008)View ArticleMathSciNetGoogle Scholar

Copyright

© Lin et al.; licensee Springer. 2015

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.