An iterative enhanced super-resolution system with edge-dominated interpolation and adaptive enhancements
- Chi-Kun Lin^{1},
- Yi-Hsien Wu^{2},
- Jar-Ferr Yang^{1}Email author and
- Bin-Da Liu^{2}
https://doi.org/10.1186/s13634-014-0190-x
© Lin et al.; licensee Springer. 2015
Received: 1 October 2014
Accepted: 29 December 2014
Published: 1 February 2015
Abstract
For super-resolution (4K × 2K) displays, super-resolution technologies, which can upsample videos to higher resolution and achieve better visual quality, become more and more important currently. In this paper, an iterative enhanced super-resolution (IESR) system which is based on two-pass edge-dominated interpolation, adaptive enhancement, and adaptive dithering techniques is proposed. The two-pass edge-dominated interpolation with a simple and regular kernel can sharpen visual quality while the adaptive enhancement can provide high-frequency perfection and the adaptive dithering conveys naturalization enhancement such that the proposed IESR system achieves better peak signal-to-noise ratio (PSNR) and exhibits better visual quality. Experimental results indicate that the proposed IESR system, which improves PSNR up to 28.748 dB and promotes structural similarity index measurement (SSIM) up to 0.917611 in averages, is better than the other existing methods. Simulations also exhibit that the proposed IESR system acquires lower computational complexity than the methods which achieve similar visual quality.
Keywords
1 Introduction
Currently, the super-resolution displays with 4K × 2K pixels are vigorously available in the commercial market; however, the existing TV programs are mostly with either standard definition (SD) with 640 × 480 or high definition (HD) with 2K × 1K resolution. In other words, there are almost no super-resolution programs to match up with 4K × 2K TV displays. Thus, the super-resolution technologies, which can upsample SD or HD videos to higher resolution, become more and more important for current applications. Super-resolution (SR) is a technique to recover a higher resolution image from a given low-resolution (LR) image. Simply, the SR algorithm could be treated as an interpolation method to enhance the resolution of images or videos. The interpolated image usually could still lose some detailed information. For 4K × 2K TV displays, it is noted that the SR algorithms should consider real-time implementation issues.
In the literatures, the SR algorithms can be classified into interpolation-based, reconstruction-based, and learning-based approaches to solve the problem of recovering detailed information extracted from the low-resolution image. The interpolation-based approach involves in the prediction of the unknown pixels by filtering processes. Based on the concept of ideal low-pass filtering, the interpolation methods [1] always need to consider the balance of computational complexity and reconstruction quality. The linear, bi-cubic, and cubic spline interpolations [2] are the possible means for reducing the complexity. To prevent filtering pixels across edges, numerous edge-directed interpolation methods are proposed [3-11].
The reconstructed-based approach generates high-resolution images by exploiting the information from a set of successive low-resolution images in the same scene but with sub-pixel displacements. In the wavelet domain [12], the LR image is considered as the lower sub-band of the wavelet-transformed high-resolution (HR) image. However, they are difficult in estimating the unknown coefficients of the other three higher wavelet sub-bands due to their independencies. Instead of the frequency domain methods, the most contemporary methods turn their attentions to solve the problem in the spatial domain [13]. The back projection algorithm iteratively projects the error between the simulated and input LR images to estimate HR error by iteratively minimizing the reconstruction error [14]. However, many jaggy artifacts along the edges may affect the quality of images. To reduce these artifacts, Dong et al [15] employed the nonlocal image redundancy to improve the quality of SR images. In the same time, it brings heavy computation complexity for updating the reconstruction error in each step. The projection onto convex sets (POCS) algorithms [16,17] applied to the input LR images could increase the solution of the element on the convex set. The maximum a posteriori (MAP) methods [18,19] adopt the associated probability of target high-resolution images to form a prior probability to refer the solution based on Bayesian inference.
The learning-based approaches [20,21] attempt to capture the correlation between LR and HR patches to exploit the redundant high-frequency information which is remained in HR training samples. Although these algorithms need a large number of databases to store millions of LR and HR patch pairs, the quality of reconstructed images can be improved even that the magnificent factor is large.
The dictionary learning-based denoising approach [22] used taxonomy based on image representations for a better understanding of state-of-the-art image denoising techniques. The multiresolution structure and sparsity of wavelets are used for nonlocal dictionary learning in each decomposition level of the wavelets [23]. The classification-based least squares trained filters on picture quality improvement algorithms are suggested [24].
In this paper, we propose an iterative enhanced super-resolution (IESR) system, which is based on two-pass edge-dominated interpolation by adding adaptive enhancement and dithering mechanisms. The proposed (IESR) system is based on iterative back projection concept [14]; however, the proposed two-pass edge-dominated interpolation consists of two adaptive fixed-structure filters. Besides, we further include the adaptive enhancement and adaptive dithering units to improve the quality of the HR image in iterative cycles. The overview of the proposed super-resolution system is addressed in Section 2. The edge-dominated interpolation methods will be described in Section 3 while the adaptive enhancement and adaptive dithering algorithms will be present in Section 4. In Section 5, the experimental results for verifying the proposed algorithms in comparison to the well-known super-resolution algorithms are demonstrated. Finally, the conclusions about this paper are addressed in Section 4.
2 Overview of the proposed super-resolution system
The detailed descriptions about the adaptive enhancement and adaptive dithering to obtain \( {D}_{(k)}^H \) will be addressed in Section 4.
If the restored error, \( {e}_T\kern0.5em =\kern0.5em \left\Vert {e}_{(k)}^L\right\Vert \) is less than a predetermined threshold ε or the number of iterations, k is equal to the maximum limited number M, the whole iterative super-resolution process will be terminated. Thus, the final restored HR image, \( {\tilde{I}}_{(k)}^H \) will be the output HR image.
If the iterative process is not terminated, the LR error image \( {e}_{(k)}^L \) is then upsampled by the proposed two-pass edge-dominated interpolation to become the HR error image, \( {e}_{(k)}^H \). From \( {e}_{(k)}^H \), we can estimate the enhancement HR image, \( {E}_{(k)}^H \), and its dithering image, \( {D}_{(k)}^H \) for the next iteration. The proposed TEI will be discussed in the next section.
3 Two-pass edge-dominated interpolation
It is noted that the computation of edge sensitivities and horizontal/vertical sensitivities, which involves the sum of selected absolute differences, can be dramatically reduced if we could properly reuse the absolute differences.
4 Image enhancement and dithering algorithms
The most edge-directed interpolation algorithms [3-11] including the proposed TEI method can successfully enlarge the low-resolution images to super-resolution ones with less artifacts along the texture edges. However, the loss of high-frequency components cannot be properly restored by general interpolation algorithms. To recover high-frequency parts of the original image, in this section, we further propose to utilize the adaptive enhancement (AE) and adaptive dithering (AD) algorithms for the Y component to further improve the quality of HR images iteratively.
From the LR reconstruction error, \( {e}_{(k)}^L \) which is stated in (4), we can use the same two-pass edge-dominated interpolation to upsample \( {e}_{(k)}^L \) to the HR reconstruction error, \( {e}_{(k)}^H \) as shown in Figure 2. To eliminate the HR reconstruction error, of course, we should try to remove \( {e}_{(k)}^H \) from the restored HR image. It is noted that \( {e}_{(k)}^H \), which is recovered from the low-resolution error, only contains the low-frequency part of the HR reconstruction error.
5 Simulation results
Quality evaluation of the proposed methods in PSNR (dB)
Image | Resolution | TEI only | TEI + AE with α ( k ) | TEI + AE + AD with β ( k ) | ||
---|---|---|---|---|---|---|
Linear | Exponent | Linear | Exponent | |||
N1 | 2,048 × 2,560 | 28.692 | 29.222 | 29.221 | 29.248 | 29.250 |
N2 | 2,048 × 2,560 | 23.287 | 23.745 | 23.756 | 23.784 | 23.786 |
N5 | 2,048 × 2,560 | 27.689 | 28.226 | 28.226 | 28.259 | 28.260 |
N4 | 2,560 × 2,048 | 25.866 | 26.421 | 26.448 | 26.509 | 26.509 |
N6 | 2,560 × 2,048 | 35.759 | 36.356 | 36.374 | 36.381 | 36.380 |
N7 | 2,560 × 2,048 | 24.804 | 25.277 | 25.279 | 25.308 | 25.309 |
N8 | 2,560 × 2,048 | 21.040 | 21.393 | 21.412 | 21.438 | 21.441 |
QFHD_P01 | 3,840 × 2,160 | 35.778 | 36.599 | 36.606 | 36.639 | 36.643 |
QFHD_P03 | 3,840 × 2,160 | 25.792 | 26.434 | 26.446 | 26.478 | 26.482 |
QFHD_P04 | 3,840 × 2,160 | 35.359 | 36.010 | 36.060 | 36.066 | 36.074 |
world_satellite | 6,000 × 4,190 | 25.685 | 25.948 | 26.017 | 26.080 | 26.090 |
Average | 28.159 | 28.694 | 28.711 | 28.739 | 28.748 |
To exhibit the effectiveness of the proposal TEI, the AE, and AD algorithms are tested by various combinations, which are the TEI only, the ‘TEI + AE,’ and the ‘TEI + AE + AD’. For AE and AD algorithms, we suggest two decay functions. For the kth iteration, the linear decaying function is given as α _{1}(k) = 0.5 − 0.1235(k − 1) while the power-of-two exponential decay function is defined as α _{2}(k) = 2^{− k }.
Quality evaluation of the proposed methods with SSIM
Image | Resolution | TEI only | TEI + AE with α ( k ) | TEI + AE + AD with β ( k ) | ||
---|---|---|---|---|---|---|
Linear | Exponent | Linear | Exponent | |||
N1 | 2,048 × 2,560 | 0.924244 | 0.933028 | 0.933031 | 0.933280 | 0.933574 |
N2 | 2,048 × 2,560 | 0.869639 | 0.884614 | 0.885038 | 0.885420 | 0.885351 |
N5 | 2,048 × 2,560 | 0.899536 | 0.908142 | 0.908139 | 0.908400 | 0.908514 |
N4 | 2,560 × 2,048 | 0.909207 | 0.917733 | 0.918449 | 0.918500 | 0.918058 |
N6 | 2,560 × 2,048 | 0.955273 | 0.958016 | 0.958051 | 0.958057 | 0.958062 |
N7 | 2,560 × 2,048 | 0.817085 | 0.833128 | 0.832990 | 0.833602 | 0.834204 |
N8 | 2,560 × 2,048 | 0.824425 | 0.842889 | 0.843691 | 0.844313 | 0.844618 |
QFHD_P01 | 3,840 × 2,160 | 0.980694 | 0.983344 | 0.983406 | 0.983439 | 0.983475 |
QFHD_P03 | 3,840 × 2,160 | 0.909168 | 0.923118 | 0.922826 | 0.923674 | 0.923897 |
QFHD_P04 | 3,840 × 2,160 | 0.975670 | 0.978389 | 0.978555 | 0.978505 | 0.978391 |
world_satellite | 6,000 × 4,190 | 0.913758 | 0.923650 | 0.925000 | 0.925313 | 0.925573 |
Average | 0.907154 | 0.916914 | 0.917198 | 0.917500 | 0.917611 |
Performance comparison of the proposed and the well-known SR algorithms in term of PSNR (dB)
Image | Resolution | NNI | Bilinear | Bi-cubic | Learn | IBP | NBP | Proposed method |
---|---|---|---|---|---|---|---|---|
N1 | 2,048 × 2,560 | 25.270 | 25.840 | 26.146 | 25.894 | 28.917 | 29.180 | 29.250 |
N2 | 2,048 × 2,560 | 20.194 | 20.764 | 21.049 | 20.930 | 23.504 | 23.513 | 23.786 |
N5 | 2,048 × 2,560 | 24.784 | 25.293 | 25.643 | 25.759 | 27.805 | 27.980 | 28.260 |
N4 | 2,560 × 2,048 | 22.649 | 23.250 | 23.661 | 23.831 | 26.008 | 26.233 | 26.509 |
N6 | 2,560 × 2,048 | 31.849 | 32.634 | 33.021 | 33.128 | 35.923 | 35.930 | 36.380 |
N7 | 2,560 × 2,048 | 22.113 | 22.560 | 22.850 | 22.843 | 24.966 | 24.914 | 25.309 |
N8 | 2,560 × 2,048 | 19.048 | 19.321 | 19.646 | 19.723 | 21.321 | 21.239 | 21.441 |
QFHD_P01 | 3,840 × 2,160 | 30.700 | 31.605 | 32.128 | 32.108 | 36.036 | 35.656 | 36.643 |
QFHD_P03 | 3,840 × 2,160 | 22.607 | 23.071 | 23.624 | 23.881 | 26.040 | 25.962 | 26.482 |
QFHD_P04 | 3,840 × 2,160 | 29.424 | 30.531 | 31.118 | 31.187 | 35.811 | 35.238 | 36.074 |
world_satellite | 6,000 × 4,190 | 22.488 | 23.215 | 23.508 | 23.495 | 25.980 | 25.830 | 26.090 |
Average | 24.648 | 25.280 | 25.672 | 25.707 | 28.392 | 28.286 | 28.748 |
Performance comparison of the proposed and the well-known SR algorithms in term in term of SSIM
Images | Resolution | NNI | Bilinear | Bi-cubic | Learn | IBP | NBP | Proposed method |
---|---|---|---|---|---|---|---|---|
N1 | 2,048 × 2,560 | 0.87807 | 0.87695 | 0.89103 | 0.89878 | 0.92988 | 0.92936 | 0.93357 |
N2 | 2,048 × 2,560 | 0.79524 | 0.79704 | 0.81697 | 0.83103 | 0.87577 | 0.87184 | 0.88535 |
N5 | 2,048 × 2,560 | 0.86975 | 0.86842 | 0.87793 | 0.88466 | 0.90397 | 0.90015 | 0.90851 |
N4 | 2,560 × 2,048 | 0.87034 | 0.87128 | 0.88325 | 0.89372 | 0.91185 | 0.90992 | 0.918056 |
N6 | 2,560 × 2,048 | 0.94077 | 0.94292 | 0.94631 | 0.94561 | 0.95827 | 0.95437 | 0.95806 |
N7 | 2,560 × 2,048 | 0.76907 | 0.76136 | 0.77935 | 0.79448 | 0.82491 | 0.81377 | 0.83420 |
N8 | 2,560 × 2,048 | 0.76006 | 0.74829 | 0.77434 | 0.79841 | 0.83508 | 0.82489 | 0.84462 |
QFHD_P01 | 3,840 × 2,160 | 0.95470 | 0.96237 | 0.96766 | 0.96913 | 0.98298 | 0.97873 | 0.98348 |
QFHD_P03 | 3,840 × 2,160 | 0.84092 | 0.84207 | 0.86495 | 0.88436 | 0.91576 | 0.90496 | 0.92390 |
QFHD_P04 | 3,840 × 2,160 | 0.94705 | 0.95413 | 0.96020 | 0.96187 | 0.97881 | 0.97246 | 0.97839 |
world_satellite | 6,000 × 4,190 | 0.84918 | 0.84998 | 0.86774 | 0.88021 | 0.91977 | 0.90765 | 0.92557 |
Average | 0.86138 | 0.86135 | 0.87543 | 0.88566 | 0.91246 | 0.90619 | 0.91761 |
Computational time (seconds) required by the proposed algorithm and the nonlocal back project methods
Image | Resolution | IBP | NBP | Proposed method |
---|---|---|---|---|
N1 | 2,048 × 2,560 | 365.221388 | 56,902.158 | 38.811761 |
N2 | 2,048 × 2,560 | 366.667311 | 49,838.346 | 37.482078 |
N5 | 2,048 × 2,560 | 361.005792 | 57,614.404 | 38.188074 |
N4 | 2,560 × 2,048 | 367.104573 | 59,545.188 | 39.593662 |
N6 | 2,560 × 2,048 | 360.240996 | 64,205.242 | 39.867318 |
N7 | 2,560 × 2,048 | 354.380109 | 58,801.846 | 38.624665 |
N8 | 2,560 × 2,048 | 355.603370 | 56,018.748 | 39.447973 |
QFHD_P01 | 3,840 × 2,160 | 562.299157 | 169,538.659 | 63.022658 |
QFHD_P03 | 3,840 × 2,160 | 559.187032 | 83,501.423 | 64.054184 |
QFHD_P04 | 3,840 × 2,160 | 569.389138 | 59,545.188 | 67.990431 |
world_satellite | 6,000 × 4,190 | 1,933.869665 | 36,711.171 | 206.892703 |
6 Conclusions
In this paper, a super-resolution algorithm based on edge-dominated interpolation adaptive enhancement and adaptive dithering is proposed. The edge-dominated interpolation can overcome the artifacts of interpolation such that we could have smoother results along the edges. The adaptive image enhancement algorithm can improve the distorted high-frequency parts while the adaptive dithering method can recover the loss of high-frequency components. In this paper, we only use Y component for edge detection, adaptive enhancement, and adaptive dithering such that we can reduce computation time and achieve better quality. The experimental results show that the proposed algorithm achieves PSNR up to 28.748 dB and SSIM up to about 0.918 in average while the computational time is also reasonably low for practical applications. Due to local data usage and regular structures in computation, the proposed super-resolution system is suitable for VLSI implementation.
Declarations
Acknowledgements
This work was supported in part by the Ministry of Economic Affairs and Ministry of Science and Technology of Taiwan, under Contract 103-EC-17-A-02-S1-201 and Grant MOST 103-2221-E-006-109-MY3.
Authors’ Affiliations
References
- E Maeland, On the comparison of interpolation methods. IEEE Trans Med Imaging 7(7), 213–217 (1988)View ArticleGoogle Scholar
- HS Hou, H Andrews, Cubic splines for image interpolation and digital filtering. IEEE Trans Acoustic, Speech Signal Process 26(6), 508–517 (1978)View ArticleMATHGoogle Scholar
- Y. Yun, J. Bae, and J. Kim, Adaptive multidirectional edge directed interpolation for selected edge regions, Proc. of Region 10 Conference (TENCON 2011), pp. 385-388, Bali, Nov. 2011.Google Scholar
- D Zhou, X Shen, W Dong, Image zooming using directional cubic convolution interpolation. IET Image Process 6(6), 627–634 (2012)View ArticleMathSciNetGoogle Scholar
- D Zhang, X Wu, An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8), 2226–2238 (2006)View ArticleGoogle Scholar
- X Li, NT Orchard, New edge-directed interpolation. IEEE Trans Image Process 10(10), 1521–1527 (2001)View ArticleGoogle Scholar
- J Allebach, PW Wong, Edge-directed interpolation. Proc Int Conf Image Process 3, 707–710 (1996)View ArticleGoogle Scholar
- C.-S. Wong and W.-C. Siu, Adaptive directional window selection for edge-directed Interpolation, Proc. of 19th International Conference on Computer Communications and Networks, vol., no., pp.1-6, Aug. 2010.Google Scholar
- C-S Wong, W-C Siu, Further improved edge-directed interpolation and fast EDI for SDTV to HDTV conversion, in Proc. of European Signal Processing Conference, 2010, pp. 23–27Google Scholar
- W-S Tam, CW Kok, S Wan-Chi, Modified edge-directed interpolation for images. J Electron Imaging 19, 013011 (2010)View ArticleGoogle Scholar
- N Asuni, A Giachetti, Accuracy improvements and artifacts removal in edge based image interpolation, in Proc. of the 3rd Int. Conf. on Computer Vision Theory and Applications, 2008Google Scholar
- SC Tai, TM Kuo, CH Lao, TW Liao, A fast algorithm for single image super resolution in both wavelet and spatial domain, in Proc. of International Symposium on Computer, Consumer and Control, 2012, pp. 702–705View ArticleGoogle Scholar
- H Su, L Tang, Y Wu, D Tretter, J Zhou, Spatially adaptive block-based super-resolution. IEEE Trans Image Process 21(3), 1031–1045 (2012)View ArticleMathSciNetGoogle Scholar
- B Zhao, Z Gan, Y Zhang, F Liu, H Wang, Novel back-projection framework for single image super-resolution, in Proc. of International Conf. on Signal Processing, 2012, pp. 894–898Google Scholar
- W Dong, L Zhang, G Shi, X Wu, Nonlocal back projection for adaptive image enlargement, in International Conf. on Image Processing, 2009, pp. 349–352Google Scholar
- C Fan, J Zhu, J Gong, C Kuang, POCS Super-resolution sequence image reconstruction based on improvement approach of Keren registration method. Proc Inter Conf Intell Syst Des Appl 2, 333–337 (2006)Google Scholar
- AJ Patti, Y Altunbasak, Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Trans Image Process 10(1), 179–186 (2002)View ArticleGoogle Scholar
- S Belekos, N Galatsanos, A Katsaggelos, Maximum a posteriori video super-resolution using a new multichannel image prior. IEEE Trans Image Process 19(6), 1451–1464 (2010)View ArticleMathSciNetGoogle Scholar
- LC Pickup, DP Capel, SJ Roberts, A Zisserman, Bayesian methods for image super-resolution. Comput J 52(1), 101–113 (2007)View ArticleGoogle Scholar
- WT Freeman, EC Pasztor, OT Carmichael, Learning low level vision. Int J Comput Vis 40(1), 25–47 (2000)View ArticleMATHGoogle Scholar
- J Yang, J Wright, T Huang, Y Ma, Image super-resolution via sparse representation. IEEE Trans Image Process 19(11), 2861–2873 (2010)View ArticleMathSciNetGoogle Scholar
- L Shao, R Yan, X Li, Y Liu, From heuristic optimization to dictionary learning: a review and comprehensive comparison of image denoising algorithms”. IEEE Trans Cybern 44(7), 1001–1013 (2014)View ArticleGoogle Scholar
- R Yan, L Shao, Y Liu, Nonlocal hierarchical dictionary learning using wavelets for image denoising. IEEE Trans Image Process 22(12), 4689–4698 (2013)View ArticleMathSciNetGoogle Scholar
- L Shao, H Zhang, G de Haan, An overview and performance evaluation of classification-based least squares trained filters. IEEE Trans Image Process 17(10), 1772–1782 (2008)View ArticleMathSciNetGoogle Scholar
Copyright
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.