Skip to main content

A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video

Abstract

Super-resolution (SR) reconstruction technique is capable of producing a high-resolution image from a sequence of low-resolution images. In this paper, we study an efficient SR algorithm for digital video. To effectively deal with the intractable problems in SR video reconstruction, such as inevitable motion estimation errors, noise, blurring, missing regions, and compression artifacts, the total variation (TV) regularization is employed in the reconstruction model. We use the fixed-point iteration method and preconditioning techniques to efficiently solve the associated nonlinear Euler-Lagrange equations of the corresponding variational problem in SR. The proposed algorithm has been tested in several cases of motion and degradation. It is also compared with the Laplacian regularization-based SR algorithm and other TV-based SR algorithms. Experimental results are presented to illustrate the effectiveness of the proposed algorithm.

References

  1. 1.

    Tsai RY, Huang TS: Multi-frame image restoration and registration. Advances in Computer Vision and Image Processing 1984,1(2):317-339.

    Google Scholar 

  2. 2.

    Kim SP, Bose NK, Valenzuela HM: Recursive reconstruction of high resolution image from noisy undersampled multiframes. IEEE Transactions on Acoustics, Speech, and Signal Processing 1990,38(6):1013-1027. 10.1109/29.56062

    Article  Google Scholar 

  3. 3.

    Kim SP, Su W-Y: Recursive high-resolution reconstruction of blurred multiframe images. IEEE Transactions on Image Processing 1993,2(4):534-539. 10.1109/83.242363

    Article  Google Scholar 

  4. 4.

    Rhee S, Kang MG: Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Engineering 1999,38(8):1348-1356. 10.1117/1.602177

    Article  Google Scholar 

  5. 5.

    Chan RH, Chan TF, Shen L, Shen Z: Wavelet algorithms for high-resolution image reconstruction. SIAM Journal of Scientific Computing 2003,24(4):1408-1432. 10.1137/S1064827500383123

    MathSciNet  MATH  Article  Google Scholar 

  6. 6.

    Ng MK, Sze CK, Yung SP: Wavelet algorithms for deblurring models. International Journal of Imaging Systems and Technology 2004,14(3):113-121. 10.1002/ima.20014

    Article  Google Scholar 

  7. 7.

    Nguyen N, Milanfar P: A wavelet-based interpolation-restoration method for superresolution (wavelet superresolution). Circuits, Systems, and Signal Processing 2000,19(4):321-338. 10.1007/BF01200891

    MATH  Article  Google Scholar 

  8. 8.

    Ur H, Gross D: Improved resolution from subpixel shifted pictures. CVGIP: Graphical Models and Image Processing 1992,54(2):181-186. 10.1016/1049-9652(92)90065-6

    Google Scholar 

  9. 9.

    Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991,53(3):231-239. 10.1016/1049-9652(91)90045-L

    Google Scholar 

  10. 10.

    Stark H, Oskoui P: High-resolution image recovery from image-plane arrays, using convex projections. Journal of the Optical Society of America A: Optics and Image Science, and Vision 1989,6(11):1715-1726. 10.1364/JOSAA.6.001715

    Article  Google Scholar 

  11. 11.

    Tekalp AM, Ozkan MK, Sezan MI: High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '92), March 1992, San Francisco, Calif, USA 3: 169–172.

    Google Scholar 

  12. 12.

    Patti AJ, Sezan MI, Tekalp AM: High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur. Proceedings of IEEE International Conference Image Processing (ICIP '94), November 1994, Austin, Tex, USA 1: 343–347.

    Article  Google Scholar 

  13. 13.

    Patti AJ, Sezan MI, Tekalp AM: Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Transactions on Image Processing 1997,6(8):1064-1076. 10.1109/83.605404

    Article  Google Scholar 

  14. 14.

    Tom BC, Katsaggelos AK: Reconstruction of a high-resolution image from multiple-degraded misregistered low-resolution images. Visual Communications and Image Processing, September 1994, Chicago, Ill, USA, Proceedings of SPIE 2308: 971–981.

    Article  Google Scholar 

  15. 15.

    Schultz RR, Stevenson RL: Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing 1996,5(6):996-1011. 10.1109/83.503915

    Article  Google Scholar 

  16. 16.

    Hardie RC, Tuinstra TR, Bognar J, Barnard KJ, Armstrong EE: High resolution image reconstruction from digital video with global and non-global scene motion. Proceedings of IEEE International Conference on Image Processing (ICIP '97), October 1997, Santa Barbara, Calif, USA 1: 153–156.

    Article  Google Scholar 

  17. 17.

    Elad M, Feuer A: Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing 1997,6(12):1646-1658. 10.1109/83.650118

    Article  Google Scholar 

  18. 18.

    Elad M, Feuer A: Superresolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999,8(3):387-395. 10.1109/83.748893

    Article  Google Scholar 

  19. 19.

    Chung J, Haber E, Nagy J: Numerical methods for coupled super-resolution. Inverse Problems 2006,22(4):1261-1272. 10.1088/0266-5611/22/4/009

    MathSciNet  MATH  Article  Google Scholar 

  20. 20.

    Hardie RC, Barnard KJ, Armstrong EE: Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing 1997,6(12):1621-1633. 10.1109/83.650116

    Article  Google Scholar 

  21. 21.

    Woods NA, Galatsanos NP, Katsaggelos AK: Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images. IEEE Transactions on Image Processing 2006,15(1):201-213.

    MathSciNet  Article  Google Scholar 

  22. 22.

    Shen H, Zhang L, Huang B, Li P: A MAP approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image Processing 2007,16(2):479-490.

    MathSciNet  Article  Google Scholar 

  23. 23.

    Sasahara R, Hasegawa H, Yamada I, Sakaniwa K: A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method. Proceedings of IEEE International Conference on Image Processing (ICIP '05), September 2005, Genova, Italy 1: 857–860.

    Google Scholar 

  24. 24.

    Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006,15(1):141-159.

    Article  Google Scholar 

  25. 25.

    Akgun T, Altunbasak Y, Mersereau RM: Super-resolution reconstruction of hyperspectral images. IEEE Transactions on Image Processing 2005,14(11):1860-1875.

    Article  Google Scholar 

  26. 26.

    Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004,13(7):898-910. 10.1109/TIP.2004.827230

    Article  Google Scholar 

  27. 27.

    Segall CA, Molina R, Katsaggelos AK: High-resolution images from low-resolution compressed video. IEEE Signal Processing Magazine 2003,20(3):37-48. 10.1109/MSP.2003.1203208

    Article  Google Scholar 

  28. 28.

    Capel D, Zisserman A: Super-resolution enhancement of text image sequences. Proceedings of the 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 600–605.

    Article  Google Scholar 

  29. 29.

    Han YB, Wu LN: Super resolution reconstruction of video sequence based on total variation. Proceedings of International Symposium on Intelligent Multimedia, Video and Speech Processing (ISIMP '04), October 2004, Hong Kong 575–578.

    Google Scholar 

  30. 30.

    Vazquez C, Aly H, Dubois E, Mitiche A: Motion compensated super-resolution of video by level sets evolution. Proceedings of IEEE International Conference on Image Processing (ICIP '04), October 2004, Singapore 3: 1767–1770.

    Google Scholar 

  31. 31.

    Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multiframe super resolution. IEEE Transactions on Image Processing 2004,13(10):1327-1344. 10.1109/TIP.2004.834669

    Article  Google Scholar 

  32. 32.

    Chan TF, Shen J: Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics 2002,62(3):1019-1043. 10.1137/S0036139900368844

    MathSciNet  MATH  Article  Google Scholar 

  33. 33.

    Borman S, Stevenson RL: Spatial resolution enhancement of low-resolution image sequences: a comprehensive review with directions for future research. Laboratory for Image and Signal Analysis (LISA), University of Notre Dame, Notre Dame, Ind, USA; 1998.

    Google Scholar 

  34. 34.

    Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003,20(3):21-36. 10.1109/MSP.2003.1203207

    Article  Google Scholar 

  35. 35.

    Nguyen N, Milanfar P, Golub G: Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement. IEEE Transactions on Image Processing 2001,10(9):1299-1308. 10.1109/83.941854

    MathSciNet  MATH  Article  Google Scholar 

  36. 36.

    Capel D, Zisserman A: Computer vision applied to super resolution. IEEE Signal Processing Magazine 2003,20(3):75-86. 10.1109/MSP.2003.1203211

    Article  Google Scholar 

  37. 37.

    Schultz RR, Meng L, Stevenson RL: Subpixel motion estimation for super-resolution image sequence enhancement. Journal of Visual Communication and Image Representation 1998,9(1):38-50. 10.1006/jvci.1997.0370

    Article  Google Scholar 

  38. 38.

    Tekalp AM: Digital Video Processing. Prentice-Hall, Englewood Clliffs, NJ, USA; 1995.

    Google Scholar 

  39. 39.

    Farsiu S, Robinson D, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004,14(2):47-57. 10.1002/ima.20007

    Article  Google Scholar 

  40. 40.

    Rudin L, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992,60(1–4):259-268.

    MathSciNet  MATH  Article  Google Scholar 

  41. 41.

    Vogel CR, Oman ME: Fast, robust total variation-based reconstruction of noisy, blurred images. IEEE Transactions on Image Processing 1998,7(6):813-824. 10.1109/83.679423

    MathSciNet  MATH  Article  Google Scholar 

  42. 42.

    Chambolle A: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 2004,20(1-2):89-97.

    MathSciNet  MATH  Google Scholar 

  43. 43.

    Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996,5(6):987-995. 10.1109/83.503914

    Article  Google Scholar 

  44. 44.

    Bioucas-Dias JM, Figueiredo MAT, Oliveira JP: Total variation-based image deconvolution: a majorization-minimization approach. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '06), May 2006, Toulouse, France 2: 861–864.

    Google Scholar 

  45. 45.

    Vogel CR, Oman ME: Iterative methods for total variation denoising. SIAM Journal of Scientific Computing 1996,17(1):227-238. 10.1137/0917016

    MathSciNet  MATH  Article  Google Scholar 

  46. 46.

    Vogel CR: Computational Methods for Inverse Problems, Frontiers in Applied Mathematics. SIAM, Philadelphia, Pa, USA; 2002.

    Google Scholar 

  47. 47.

    Lin F-R, Ng MK, Ching W-K: Factorized banded inverse preconditioners for matrices with Toeplitz structure. SIAM Journal of Scientific Computing 2005,26(6):1852-1870. 10.1137/030601272

    MathSciNet  MATH  Article  Google Scholar 

  48. 48.

    Chan RH, Chan TF, Wong C-K: Cosine transform based preconditioners for total variation deblurring. IEEE Transactions on Image Processing 1999,8(10):1472-1478. 10.1109/83.791976

    Article  Google Scholar 

  49. 49.

    Ng MK, Chan RH, Chan TF, Yip AM: Cosine transform preconditioners for high resolution image reconstruction. Linear Algebra and Its Applications 2000,316(1–3):89-104.

    MathSciNet  MATH  Article  Google Scholar 

  50. 50.

    Kolotilina LY, Yeremin AY: Factorized sparse approximate inverse preconditionings I: theory. SIAM Journal on Matrix Analysis and Applications 1993,14(1):45-58. 10.1137/0614004

    MathSciNet  MATH  Article  Google Scholar 

  51. 51.

    Nguyen N, Milanfar P, Golub G: A computationally efficient superresolution image reconstruction algorithm. IEEE Transactions on Image Processing 2001,10(4):573-583. 10.1109/83.913592

    MATH  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Michael K. Ng.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://doi.org/creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Ng, M.K., Shen, H., Lam, E.Y. et al. A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video. EURASIP J. Adv. Signal Process. 2007, 074585 (2007). https://doi.org/10.1155/2007/74585

Download citation

Keywords

  • Reconstruction Algorithm
  • Motion Estimation
  • Reconstruction Technique
  • Digital Video
  • Reconstruction Model