Open Access

Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences

  • Sina Farsiu1,
  • Michael Elad2 and
  • Peyman Milanfar1
EURASIP Journal on Advances in Signal Processing20062006:061859

https://doi.org/10.1155/ASP/2006/61859

Received: 17 December 2004

Accepted: 15 March 2005

Published: 8 February 2006

Abstract

We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color super-resolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength.

[1234567891011121314151617181920212223242526272829303132333435363738394041424344]

Authors’ Affiliations

(1)
Electrical Engineering Department, University of California Santa Cruz
(2)
Computer Science Department, Technion – Israel Institute of Technology

References

  1. Huang TS, Tsai RY: Multi-frame image restoration and registration. In Advances in Computer Vision and Image Processing. Volume 1. JAI Press, Greenwich, Conn, USA; 1984:317-339. chapter 7Google Scholar
  2. Nguyen N, Milanfar P, Golub GH: A computationally efficient super-resolution image reconstruction algorithm. Transactions on Image Processing 2001, 10(4):573-583. 10.1109/83.913592MATHMathSciNetView ArticleGoogle Scholar
  3. Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991, 53(3):231-239. 10.1016/1049-9652(91)90045-LGoogle Scholar
  4. Elad M, Feuer A: Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images. Transactions on Image Processing 1997, 6(12):1646-1658. 10.1109/83.650118View ArticleGoogle Scholar
  5. Zomet A, Peleg S: Efficient super-resolution and applications to mosaics. Proceedings of IEEE 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 579-583.View ArticleGoogle Scholar
  6. Ng MK, Bose NK: Mathematical analysis of super-resolution methodology. IEEE Signal Processing Magazine 2003, 20(3):62-74. 10.1109/MSP.2003.1203210View ArticleGoogle Scholar
  7. Altunbasak Y, Patti AJ, Mersereau RM: Super-resolution still and video reconstruction from MPEG-coded video. IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(4):217-226. 10.1109/76.999200View ArticleGoogle Scholar
  8. Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multi-frame super-resolution. IEEE Transactions on Image Processing 2004, 13(10):1327-1344. 10.1109/TIP.2004.834669View ArticleGoogle Scholar
  9. Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004, 13(7):898-911. 10.1109/TIP.2004.827230View ArticleGoogle Scholar
  10. Borman S, Stevenson RL: Super-resolution from image sequences—a review. Proceedings of Midwest Symposium on Circuits and Systems (MWSCAS '98), August 1998, Notre Dame, Ind, USA 374-378.Google Scholar
  11. Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003, 20(3):21-36. 10.1109/MSP.2003.1203207View ArticleGoogle Scholar
  12. Farsiu S, Robinson MD, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004, 14(2):47-57. 10.1002/ima.20007View ArticleGoogle Scholar
  13. Elad M, Feuer A: Super-resolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999, 8(3):387-395. 10.1109/83.748893View ArticleGoogle Scholar
  14. Elad M, Feuer A: Super-resolution reconstruction of image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999, 21(9):817-834. 10.1109/34.790425View ArticleGoogle Scholar
  15. Elad M, Hel-Or Y: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing 2001, 10(8):1187-1193. 10.1109/83.935034MATHView ArticleGoogle Scholar
  16. Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution from undersampled color images. Computational Imaging II, January 2004, San Jose, Calif, USA, Proceedings of SPIE 5299: 222-233.View ArticleGoogle Scholar
  17. Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006, 15(1):141-159.View ArticleGoogle Scholar
  18. Kay SM: Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Prentice-Hall, Englewood Cliffs, NJ, USA; 1993.MATHGoogle Scholar
  19. Jazwinski AH: Stochastic Processes and Filtering Theory. Academic Press, New York, NY, USA; 1970.MATHGoogle Scholar
  20. Elad M: Super-resolution reconstruction of continuous image sequence, Ph.D. dissertation. Technion-Israel Institute of Technology, Haifa, Israel; 1997.Google Scholar
  21. Rauch HE, Striebel CT, Tung F: Maximum likelihood estimates of dynamic linear systems. American Institute of Aeronautics and Astronautics 1965, 3(8):1445-1450.MathSciNetView ArticleGoogle Scholar
  22. Harvey AC: Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press, Cambridge, UK; 1990.MATHView ArticleGoogle Scholar
  23. Bovik A: Handbook of Image and Video Processing. Academic Press, New York, NY, USA; 2000.MATHGoogle Scholar
  24. Rudin LI, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992, 60(1–4):259-268.MATHView ArticleMathSciNetGoogle Scholar
  25. Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996, 5(6):987-995. 10.1109/83.503914View ArticleGoogle Scholar
  26. Chan TF, Osher S, Shen J: The digital TV filter and nonlinear denoising. IEEE Transactions on Image Processing 2001, 10(2):231-241. 10.1109/83.902288MATHView ArticleGoogle Scholar
  27. Tomasi C, Manduchi R: Bilateral filtering for gray and color images. Proceedings of IEEE 6th International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 839-846.Google Scholar
  28. Elad M: On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing 2002, 11(10):1141-1151. 10.1109/TIP.2002.801126MathSciNetView ArticleGoogle Scholar
  29. Laroche C, Prescott M: Apparatus and method for adaptive for adaptively interpolating a full color image utilizing chrominance gradients. United States Patent 5,373,322, 1994Google Scholar
  30. Kimmel R: Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 1999, 8(9):1221-1228. 10.1109/83.784434View ArticleGoogle Scholar
  31. Keren D, Osadchy M: Restoring subsampled color images. Machine Vision and Applications 1999, 11(4):197-202. 10.1007/s001380050102View ArticleGoogle Scholar
  32. Hel-Or Y, Keren D: Demosaicing of color images using steerable wavelets. In Tech. Rep. HPL-2002-206R1 20020830. HP Laboratories Israel, Haifa, Israel; 2002. Online, available: http://citeseer.nj.nec.com/Google Scholar
  33. Muresan DD, Parks TW: Optimal recovery demosaicing. Proceedings of IASTED International Conference on Signal and Image Processing (SIP '02), August 2002, Kauai, Hawaii, USAGoogle Scholar
  34. Gunturk BK, Altunbasak Y, Mersereau RM: Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 2002, 11(9):997-1013. 10.1109/TIP.2002.801121View ArticleGoogle Scholar
  35. Alleysson D, Süsstrunk S, Hérault J: Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain. Proceedings of IS&T/SID 10th Color Imaging Conference, November 2002, Scottsdale, Ariz, USA 331-336.Google Scholar
  36. Ramanath R, Snyder WE, Bilbro GL, Sander WA: Demosaicking methods for the Bayer color arrays. Journal of Electronic Imaging 2002, 11(3):306-315. 10.1117/1.1484495View ArticleGoogle Scholar
  37. Pei S-C, Tam I-K: Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 2003, 13(6):503-513. 10.1109/TCSVT.2003.813422View ArticleGoogle Scholar
  38. Zomet A, Peleg S: Multi-sensor super-resolution. Proceedings of IEEE 6th Workshop on Applications of Computer Vision (WACV '02), December 2002, Orlando, Fla, USA 27-31.Google Scholar
  39. Gotoh T, Okutomi M: Direct super-resolution and registration using raw CFA images. Proceedings of IEEE Computer Society Conference on Computer Vision and Patern Recognition (CVPR '04), June–July 2004, Washington, DC, USA 2: 600-607.Google Scholar
  40. Shah NR, Zakhor A: Resolution enhancement of color video sequences. IEEE Transactions on Image Processing 1999, 8(6):879-885. 10.1109/83.766865View ArticleGoogle Scholar
  41. Tom BC, Katsaggelos AK: Resolution enhancement of monochrome and color video using motion compensation. IEEE Transactions on Image Processing 2001, 10(2):278-287. 10.1109/83.902292MATHView ArticleGoogle Scholar
  42. Pratt WK: Digital Image Processing. 3rd edition. John Wiley & Sons, New York, NY, USA; 2001.View ArticleMATHGoogle Scholar
  43. Golland P, Bruckstein AM: Motion from color. Computer Vision and Image Understanding 1997, 68(3):346-362. 10.1006/cviu.1997.0553View ArticleGoogle Scholar
  44. Bergen JR, Anandan P, Hanna KJ, Hingorani R: Hierarchical model-based motion estimation. Proceedings of European Conference on Computer Vision (ECCV '92), May 1992, Santa Margherita Ligure, Italy 237-252.Google Scholar

Copyright

© Farsiu et al. 2006