- Research Article
- Open Access
A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video
EURASIP Journal on Advances in Signal Processing volume 2007, Article number: 074585 (2007)
Super-resolution (SR) reconstruction technique is capable of producing a high-resolution image from a sequence of low-resolution images. In this paper, we study an efficient SR algorithm for digital video. To effectively deal with the intractable problems in SR video reconstruction, such as inevitable motion estimation errors, noise, blurring, missing regions, and compression artifacts, the total variation (TV) regularization is employed in the reconstruction model. We use the fixed-point iteration method and preconditioning techniques to efficiently solve the associated nonlinear Euler-Lagrange equations of the corresponding variational problem in SR. The proposed algorithm has been tested in several cases of motion and degradation. It is also compared with the Laplacian regularization-based SR algorithm and other TV-based SR algorithms. Experimental results are presented to illustrate the effectiveness of the proposed algorithm.
Tsai RY, Huang TS: Multi-frame image restoration and registration. Advances in Computer Vision and Image Processing 1984,1(2):317-339.
Kim SP, Bose NK, Valenzuela HM: Recursive reconstruction of high resolution image from noisy undersampled multiframes. IEEE Transactions on Acoustics, Speech, and Signal Processing 1990,38(6):1013-1027. 10.1109/29.56062
Kim SP, Su W-Y: Recursive high-resolution reconstruction of blurred multiframe images. IEEE Transactions on Image Processing 1993,2(4):534-539. 10.1109/83.242363
Rhee S, Kang MG: Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Engineering 1999,38(8):1348-1356. 10.1117/1.602177
Chan RH, Chan TF, Shen L, Shen Z: Wavelet algorithms for high-resolution image reconstruction. SIAM Journal of Scientific Computing 2003,24(4):1408-1432. 10.1137/S1064827500383123
Ng MK, Sze CK, Yung SP: Wavelet algorithms for deblurring models. International Journal of Imaging Systems and Technology 2004,14(3):113-121. 10.1002/ima.20014
Nguyen N, Milanfar P: A wavelet-based interpolation-restoration method for superresolution (wavelet superresolution). Circuits, Systems, and Signal Processing 2000,19(4):321-338. 10.1007/BF01200891
Ur H, Gross D: Improved resolution from subpixel shifted pictures. CVGIP: Graphical Models and Image Processing 1992,54(2):181-186. 10.1016/1049-9652(92)90065-6
Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991,53(3):231-239. 10.1016/1049-9652(91)90045-L
Stark H, Oskoui P: High-resolution image recovery from image-plane arrays, using convex projections. Journal of the Optical Society of America A: Optics and Image Science, and Vision 1989,6(11):1715-1726. 10.1364/JOSAA.6.001715
Tekalp AM, Ozkan MK, Sezan MI: High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '92), March 1992, San Francisco, Calif, USA 3: 169–172.
Patti AJ, Sezan MI, Tekalp AM: High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur. Proceedings of IEEE International Conference Image Processing (ICIP '94), November 1994, Austin, Tex, USA 1: 343–347.
Patti AJ, Sezan MI, Tekalp AM: Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Transactions on Image Processing 1997,6(8):1064-1076. 10.1109/83.605404
Tom BC, Katsaggelos AK: Reconstruction of a high-resolution image from multiple-degraded misregistered low-resolution images. Visual Communications and Image Processing, September 1994, Chicago, Ill, USA, Proceedings of SPIE 2308: 971–981.
Schultz RR, Stevenson RL: Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing 1996,5(6):996-1011. 10.1109/83.503915
Hardie RC, Tuinstra TR, Bognar J, Barnard KJ, Armstrong EE: High resolution image reconstruction from digital video with global and non-global scene motion. Proceedings of IEEE International Conference on Image Processing (ICIP '97), October 1997, Santa Barbara, Calif, USA 1: 153–156.
Elad M, Feuer A: Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing 1997,6(12):1646-1658. 10.1109/83.650118
Elad M, Feuer A: Superresolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999,8(3):387-395. 10.1109/83.748893
Chung J, Haber E, Nagy J: Numerical methods for coupled super-resolution. Inverse Problems 2006,22(4):1261-1272. 10.1088/0266-5611/22/4/009
Hardie RC, Barnard KJ, Armstrong EE: Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing 1997,6(12):1621-1633. 10.1109/83.650116
Woods NA, Galatsanos NP, Katsaggelos AK: Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images. IEEE Transactions on Image Processing 2006,15(1):201-213.
Shen H, Zhang L, Huang B, Li P: A MAP approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image Processing 2007,16(2):479-490.
Sasahara R, Hasegawa H, Yamada I, Sakaniwa K: A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method. Proceedings of IEEE International Conference on Image Processing (ICIP '05), September 2005, Genova, Italy 1: 857–860.
Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006,15(1):141-159.
Akgun T, Altunbasak Y, Mersereau RM: Super-resolution reconstruction of hyperspectral images. IEEE Transactions on Image Processing 2005,14(11):1860-1875.
Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004,13(7):898-910. 10.1109/TIP.2004.827230
Segall CA, Molina R, Katsaggelos AK: High-resolution images from low-resolution compressed video. IEEE Signal Processing Magazine 2003,20(3):37-48. 10.1109/MSP.2003.1203208
Capel D, Zisserman A: Super-resolution enhancement of text image sequences. Proceedings of the 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 600–605.
Han YB, Wu LN: Super resolution reconstruction of video sequence based on total variation. Proceedings of International Symposium on Intelligent Multimedia, Video and Speech Processing (ISIMP '04), October 2004, Hong Kong 575–578.
Vazquez C, Aly H, Dubois E, Mitiche A: Motion compensated super-resolution of video by level sets evolution. Proceedings of IEEE International Conference on Image Processing (ICIP '04), October 2004, Singapore 3: 1767–1770.
Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multiframe super resolution. IEEE Transactions on Image Processing 2004,13(10):1327-1344. 10.1109/TIP.2004.834669
Chan TF, Shen J: Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics 2002,62(3):1019-1043. 10.1137/S0036139900368844
Borman S, Stevenson RL: Spatial resolution enhancement of low-resolution image sequences: a comprehensive review with directions for future research. Laboratory for Image and Signal Analysis (LISA), University of Notre Dame, Notre Dame, Ind, USA; 1998.
Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003,20(3):21-36. 10.1109/MSP.2003.1203207
Nguyen N, Milanfar P, Golub G: Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement. IEEE Transactions on Image Processing 2001,10(9):1299-1308. 10.1109/83.941854
Capel D, Zisserman A: Computer vision applied to super resolution. IEEE Signal Processing Magazine 2003,20(3):75-86. 10.1109/MSP.2003.1203211
Schultz RR, Meng L, Stevenson RL: Subpixel motion estimation for super-resolution image sequence enhancement. Journal of Visual Communication and Image Representation 1998,9(1):38-50. 10.1006/jvci.1997.0370
Tekalp AM: Digital Video Processing. Prentice-Hall, Englewood Clliffs, NJ, USA; 1995.
Farsiu S, Robinson D, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004,14(2):47-57. 10.1002/ima.20007
Rudin L, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992,60(1–4):259-268.
Vogel CR, Oman ME: Fast, robust total variation-based reconstruction of noisy, blurred images. IEEE Transactions on Image Processing 1998,7(6):813-824. 10.1109/83.679423
Chambolle A: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 2004,20(1-2):89-97.
Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996,5(6):987-995. 10.1109/83.503914
Bioucas-Dias JM, Figueiredo MAT, Oliveira JP: Total variation-based image deconvolution: a majorization-minimization approach. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '06), May 2006, Toulouse, France 2: 861–864.
Vogel CR, Oman ME: Iterative methods for total variation denoising. SIAM Journal of Scientific Computing 1996,17(1):227-238. 10.1137/0917016
Vogel CR: Computational Methods for Inverse Problems, Frontiers in Applied Mathematics. SIAM, Philadelphia, Pa, USA; 2002.
Lin F-R, Ng MK, Ching W-K: Factorized banded inverse preconditioners for matrices with Toeplitz structure. SIAM Journal of Scientific Computing 2005,26(6):1852-1870. 10.1137/030601272
Chan RH, Chan TF, Wong C-K: Cosine transform based preconditioners for total variation deblurring. IEEE Transactions on Image Processing 1999,8(10):1472-1478. 10.1109/83.791976
Ng MK, Chan RH, Chan TF, Yip AM: Cosine transform preconditioners for high resolution image reconstruction. Linear Algebra and Its Applications 2000,316(1–3):89-104.
Kolotilina LY, Yeremin AY: Factorized sparse approximate inverse preconditionings I: theory. SIAM Journal on Matrix Analysis and Applications 1993,14(1):45-58. 10.1137/0614004
Nguyen N, Milanfar P, Golub G: A computationally efficient superresolution image reconstruction algorithm. IEEE Transactions on Image Processing 2001,10(4):573-583. 10.1109/83.913592