Skip to main content

Advertisement

Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences

Article metrics

Abstract

We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color super-resolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength.

References

  1. 1.

    Huang TS, Tsai RY: Multi-frame image restoration and registration. In Advances in Computer Vision and Image Processing. Volume 1. JAI Press, Greenwich, Conn, USA; 1984:317–339. chapter 7

  2. 2.

    Nguyen N, Milanfar P, Golub GH: A computationally efficient super-resolution image reconstruction algorithm. Transactions on Image Processing 2001, 10(4):573–583. 10.1109/83.913592

  3. 3.

    Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991, 53(3):231–239. 10.1016/1049-9652(91)90045-L

  4. 4.

    Elad M, Feuer A: Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images. Transactions on Image Processing 1997, 6(12):1646–1658. 10.1109/83.650118

  5. 5.

    Zomet A, Peleg S: Efficient super-resolution and applications to mosaics. Proceedings of IEEE 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 579–583.

  6. 6.

    Ng MK, Bose NK: Mathematical analysis of super-resolution methodology. IEEE Signal Processing Magazine 2003, 20(3):62–74. 10.1109/MSP.2003.1203210

  7. 7.

    Altunbasak Y, Patti AJ, Mersereau RM: Super-resolution still and video reconstruction from MPEG-coded video. IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(4):217–226. 10.1109/76.999200

  8. 8.

    Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multi-frame super-resolution. IEEE Transactions on Image Processing 2004, 13(10):1327–1344. 10.1109/TIP.2004.834669

  9. 9.

    Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004, 13(7):898–911. 10.1109/TIP.2004.827230

  10. 10.

    Borman S, Stevenson RL: Super-resolution from image sequences—a review. Proceedings of Midwest Symposium on Circuits and Systems (MWSCAS '98), August 1998, Notre Dame, Ind, USA 374–378.

  11. 11.

    Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003, 20(3):21–36. 10.1109/MSP.2003.1203207

  12. 12.

    Farsiu S, Robinson MD, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004, 14(2):47–57. 10.1002/ima.20007

  13. 13.

    Elad M, Feuer A: Super-resolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999, 8(3):387–395. 10.1109/83.748893

  14. 14.

    Elad M, Feuer A: Super-resolution reconstruction of image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999, 21(9):817–834. 10.1109/34.790425

  15. 15.

    Elad M, Hel-Or Y: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing 2001, 10(8):1187–1193. 10.1109/83.935034

  16. 16.

    Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution from undersampled color images. Computational Imaging II, January 2004, San Jose, Calif, USA, Proceedings of SPIE 5299: 222–233.

  17. 17.

    Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006, 15(1):141–159.

  18. 18.

    Kay SM: Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Prentice-Hall, Englewood Cliffs, NJ, USA; 1993.

  19. 19.

    Jazwinski AH: Stochastic Processes and Filtering Theory. Academic Press, New York, NY, USA; 1970.

  20. 20.

    Elad M: Super-resolution reconstruction of continuous image sequence, Ph.D. dissertation. Technion-Israel Institute of Technology, Haifa, Israel; 1997.

  21. 21.

    Rauch HE, Striebel CT, Tung F: Maximum likelihood estimates of dynamic linear systems. American Institute of Aeronautics and Astronautics 1965, 3(8):1445–1450.

  22. 22.

    Harvey AC: Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press, Cambridge, UK; 1990.

  23. 23.

    Bovik A: Handbook of Image and Video Processing. Academic Press, New York, NY, USA; 2000.

  24. 24.

    Rudin LI, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992, 60(1–4):259–268.

  25. 25.

    Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996, 5(6):987–995. 10.1109/83.503914

  26. 26.

    Chan TF, Osher S, Shen J: The digital TV filter and nonlinear denoising. IEEE Transactions on Image Processing 2001, 10(2):231–241. 10.1109/83.902288

  27. 27.

    Tomasi C, Manduchi R: Bilateral filtering for gray and color images. Proceedings of IEEE 6th International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 839–846.

  28. 28.

    Elad M: On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing 2002, 11(10):1141–1151. 10.1109/TIP.2002.801126

  29. 29.

    Laroche C, Prescott M: Apparatus and method for adaptive for adaptively interpolating a full color image utilizing chrominance gradients. United States Patent 5,373,322, 1994

  30. 30.

    Kimmel R: Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 1999, 8(9):1221–1228. 10.1109/83.784434

  31. 31.

    Keren D, Osadchy M: Restoring subsampled color images. Machine Vision and Applications 1999, 11(4):197–202. 10.1007/s001380050102

  32. 32.

    Hel-Or Y, Keren D: Demosaicing of color images using steerable wavelets. In Tech. Rep. HPL-2002-206R1 20020830. HP Laboratories Israel, Haifa, Israel; 2002. Online, available: https://doi.org/citeseer.nj.nec.com/

  33. 33.

    Muresan DD, Parks TW: Optimal recovery demosaicing. Proceedings of IASTED International Conference on Signal and Image Processing (SIP '02), August 2002, Kauai, Hawaii, USA

  34. 34.

    Gunturk BK, Altunbasak Y, Mersereau RM: Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 2002, 11(9):997–1013. 10.1109/TIP.2002.801121

  35. 35.

    Alleysson D, Süsstrunk S, Hérault J: Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain. Proceedings of IS&T/SID 10th Color Imaging Conference, November 2002, Scottsdale, Ariz, USA 331–336.

  36. 36.

    Ramanath R, Snyder WE, Bilbro GL, Sander WA: Demosaicking methods for the Bayer color arrays. Journal of Electronic Imaging 2002, 11(3):306–315. 10.1117/1.1484495

  37. 37.

    Pei S-C, Tam I-K: Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 2003, 13(6):503–513. 10.1109/TCSVT.2003.813422

  38. 38.

    Zomet A, Peleg S: Multi-sensor super-resolution. Proceedings of IEEE 6th Workshop on Applications of Computer Vision (WACV '02), December 2002, Orlando, Fla, USA 27–31.

  39. 39.

    Gotoh T, Okutomi M: Direct super-resolution and registration using raw CFA images. Proceedings of IEEE Computer Society Conference on Computer Vision and Patern Recognition (CVPR '04), June–July 2004, Washington, DC, USA 2: 600–607.

  40. 40.

    Shah NR, Zakhor A: Resolution enhancement of color video sequences. IEEE Transactions on Image Processing 1999, 8(6):879–885. 10.1109/83.766865

  41. 41.

    Tom BC, Katsaggelos AK: Resolution enhancement of monochrome and color video using motion compensation. IEEE Transactions on Image Processing 2001, 10(2):278–287. 10.1109/83.902292

  42. 42.

    Pratt WK: Digital Image Processing. 3rd edition. John Wiley & Sons, New York, NY, USA; 2001.

  43. 43.

    Golland P, Bruckstein AM: Motion from color. Computer Vision and Image Understanding 1997, 68(3):346–362. 10.1006/cviu.1997.0553

  44. 44.

    Bergen JR, Anandan P, Hanna KJ, Hingorani R: Hierarchical model-based motion estimation. Proceedings of European Conference on Computer Vision (ECCV '92), May 1992, Santa Margherita Ligure, Italy 237–252.

Download references

Author information

Correspondence to Sina Farsiu.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Farsiu, S., Elad, M. & Milanfar, P. Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences. EURASIP J. Adv. Signal Process. 2006, 061859 (2006) doi:10.1155/ASP/2006/61859

Download citation

Keywords

  • Color
  • Information Technology
  • Real Data
  • Deblurring
  • Quantum Information