Skip to main content

Advertisement

Overcoming Registration Uncertainty in Image Super-Resolution: Maximize or Marginalize?

Article metrics

  • 1113 Accesses

  • 17 Citations

Abstract

In multiple-image super-resolution, a high-resolution image is estimated from a number of lower-resolution images. This usually involves computing the parameters of a generative imaging model (such as geometric and photometric registration, and blur) and obtaining a MAP estimate by minimizing a cost function including an appropriate prior. Two alternative approaches are examined. First, both registrations and the super-resolution image are found simultaneously using a joint MAP optimization. Second, we perform Bayesian integration over the unknown image registration parameters, deriving a cost function whose only variables of interest are the pixel values of the super-resolution image. We also introduce a scheme to learn the parameters of the image prior as part of the super-resolution algorithm. We show examples on a number of real sequences including multiple stills, digital video, and DVDs of movies.

References

  1. 1.

    Irani M, Peleg S: Super resolution from image sequences. Proceedings of the 10th International Conference on Pattern Recognition (ICPR '90), June 1990, Atlantic City, NJ, USA 2: 115–120.

  2. 2.

    Patti AJ, Sezan MI, Tekalp AM: Robust methods for high-quality stills from interlaced video in the presence of dominant motion. IEEE Transactions on Circuits and Systems for Video Technology 1997,7(2):328-342. 10.1109/76.564111

  3. 3.

    Salient Stills https://doi.org/www.salientstills.com/

  4. 4.

    Cheeseman P, Kanefsky B, Kraft R, Stutz J, Hanson R: Super-resolved surface reconstruction from multiple images. In Maximum Entropy and Bayesian Methods. Edited by: Heidbreder GR. Kluwer Academic Publishers, Dordrecht, The Netherlands; 1996:293-308.

  5. 5.

    Hardie RC, Barnard KJ, Armstrong EE: Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing 1997,6(12):1621-1633. 10.1109/83.650116

  6. 6.

    Joshi MV, Chaudhuri S, Panuganti R: A learning-based method for image super-resolution from zoomed observations. IEEE Transactions on Systems, Man, and Cybernetics, Part B 2005,35(3):527-537. 10.1109/TSMCB.2005.846647

  7. 7.

    Capel DP, Zisserman A: Automated mosaicing with super-resolution zoom. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '98), June 1998, Santa Barbara, Calif, USA 885–891.

  8. 8.

    Altunbasak Y, Patti AJ, Mersereau RM: Super-resolution still and video reconstruction from MPEG-coded video. IEEE Transactions on Circuits and Systems for Video Technology 2002,12(4):217-226. 10.1109/76.999200

  9. 9.

    Baker S, Kanade T: Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(9):1167-1183. 10.1109/TPAMI.2002.1033210

  10. 10.

    Bascle B, Blake A, Zisserman A: Motion deblurring and super-resolution from an image sequence. In Proceedings of the 4th European Conference on Computer Vision (ECCV '96), April 1996, Cambridge, UK. Volume 2. Springer; 573–582.

  11. 11.

    Elad M, Feuer A: Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing 1997,6(12):1646-1658. 10.1109/83.650118

  12. 12.

    Nguyen N, Milanfar P, Golub G: Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement. IEEE Transactions on Image Processing 2001,10(9):1299-1308. 10.1109/83.941854

  13. 13.

    Schultz RR, Stevenson RL: A Bayesian approach to image expansion for improved definition. IEEE Transactions on Image Processing 1994,3(3):233-242. 10.1109/83.287017

  14. 14.

    Robinson D, Milanfar P: Fundamental performance limits in image registration. IEEE Transactions on Image Processing 2004,13(9):1185-1199. 10.1109/TIP.2004.832923

  15. 15.

    Tipping ME, Bishop CM: Bayesian image super-resolution. Proceedings of Advances in Neural Information Processing Systems 15 (NIPS '02), December 2002, Vancouver, British Columbia, Canada 1279–1286.

  16. 16.

    Borman S: Topics in multiframe superresolution restoration, Ph.D. thesis. University of Notre Dame, Notre Dame, Ind, USA; 2004.

  17. 17.

    Borman S, Stevenson RL: Simultaneous multi-frame MAP super-resolution video enhancement using spatio-temporal priors. Proceedings of International Conference on Image Processing (ICIP '99), October 1999, Kobe, Japan 3: 469–473.

  18. 18.

    Capel DP: Image Mosaicing and Super-Resolution, Distinguished Dissertations. Springer, New York, NY, USA; 2004.

  19. 19.

    Farsiu S, Elad M, Milanfar P: A practical approach to super-resolution. Visual Communications and Image Processing, January 2006, San Jose, Calif, USA, Proceedings of SPIE 6077:

  20. 20.

    Pickup LC, Roberts SJ, Zisserman A: A sampled texture prior for image super-resolution. Proceedings of Advances in Neural Information Processing Systems 16 (NIPS '03), December 2004, Vancouver, British Columbia, Canada 1587–1594.

  21. 21.

    Pickup LC, Capel DP, Roberts SJ, Zisserman A: Bayesian image super-resolution, continued. Advances in Neural Information Processing Systems 19, December 2006, Cambridge, Mass, USA 1089–1096.

  22. 22.

    Pickup LC, Roberts SJ, Zisserman A: Optimizing and learning for super-resolution. Proceedings of the 17th British Machine Vision Conference (BMVC '06), September 2006, Edinburgh, UK

  23. 23.

    Hartley RI, Zisserman A: Multiple View Geometry in Computer Vision. 2nd edition. Cambridge University Press, Cambridge, UK; 2004.

  24. 24.

    Nabney I: NETLAB: Algorithms for Pattern Recognition. Springer, New York, NY, USA; 2002.

  25. 25.

    Triggs B, McLauchlan PF, Hartley RI, Fitzgibbon AW: Bundle adjustment—a modern synthesis. In Proceedings of International Workshop on Vision Algorithms on Vision Algorithms: Theory and Practice, September 1999, Corfu, Greece, Lecture Notes in Computer Science. Volume 1883. Edited by: Triggs B, Zisserman A, Szeliski R. Springer; 298–372.

  26. 26.

    Hardie RC, Barnard KJ, Bognar JG, Armstrong EE, Watson EA: High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system. Optical Engineering 1998,37(1):247-260. 10.1117/1.601623

  27. 27.

    Reeves SJ, Mersereau RM: Blur identification by the method of generalized cross-validation. IEEE Transactions on Image Processing 1992,1(3):301-311. 10.1109/83.148604

  28. 28.

    Lagendijk RL, Tekalp AM, Biemond J: Maximum likelihood image and blur identification: a unifying approach. Optical Engineering 1990,29(5):422-435. 10.1117/12.55611

  29. 29.

    Kundur D, Hatzinakos D: Blind image deconvolution. IEEE Signal Processing Magazine 1996,13(3):43-64. 10.1109/79.489268

  30. 30.

    Freeman WT, Jones TR, Pasztor EC: Example-based super-resolution. IEEE Computer Graphics and Applications 2002,22(2):56-65. 10.1109/38.988747

Download references

Author information

Correspondence to Lyndsey C Pickup.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://doi.org/creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Keywords

  • Cost Function
  • Image Registration
  • Digital Video
  • Imaging Model
  • Generative Imaging