Robust video super resolution algorithm using measurement validation method and scene change detection
© Kim et al; licensee Springer. 2011
Received: 28 February 2011
Accepted: 15 November 2011
Published: 15 November 2011
Explicit motion estimation is considered a major factor in the performance of classical motion-based super resolution (SR) algorithms. To reconstruct video frames sequentially, we applied a dynamic SR algorithm based on the Kalman recursive estimator. Our approach includes a novel measurement validation process to attain robust image reconstruction results under inexplicit motion estimation. In our method, the suitability for high-resolution pixel estimation is determined by the accuracy of motion estimation. We measured the accuracy of the image registration result using the Mahalanobis distance between the input low-resolution frame and the motion compensated high-resolution estimation. We also incorporate an effective scene change detection method dedicated to the proposed SR approach for minimizing erroneous results when abrupt scene changes occur in the video frames. According to the ratio of well-aligned pixels (i.e., motion is compensated accurately) to the total number of pixels, we are able to detect sudden changes of scene and context in the input video. Representative experiments on synthetic and real video data show robust performance of the proposed algorithm in terms of its reconstruction quality even with errors in the estimated motion.
In imaging devices and applications, we often have to deal with degraded low resolution (LR) images due to because of the theoretical and practical limits of imaging devices. In visual surveillance and satellite imaging systems, certain regions of interest in the input video must be magnified for more detailed analyses. However, it is difficult to obtain satisfactory images using conventional image zooming techniques and the interpolation methods. Expensive imaging devices capable of capturing images of higher resolution or higher quality may not be desirable for higher cost.
Nowadays, the super resolution (SR) algorithm has been considered one of the most promising methods to overcome the limits of imaging devices since it does not induce any additional expensive hardware. The SR algorithm is an image processing technique that can recover an HR image from multiple LR images.
Researchers have investigated a variety of SR approaches over the past last two decades in an attempt to achieve better image reconstruction results [1, 2]. SR algorithms can be divided into two broad categories. The first is motion-based SR which considers movement between the LR image frames as a cue [3–9]. By making certain assumptions in the image acquisition model, this approach becomes straightforward and easy to implement. In this scheme, however, precise motion estimation and compensation are very important to reconstruct the HR image. Since the estimation of complex motions of multiple objects in LR video is difficult and time-consuming, new approaches have recently been developed to avoid the high dependency of motion-based SR on accurate motion estimation [10–14]. These approaches constitute the second category of SR algorithms and are referred to as motion-free SR . Instead of directly estimating the motion, motion-free SR obtains spatial enhancement by incorporating cues such as blur.
Among the various motion-free SR approaches, the example-based SR algorithm  is one of the most promising methods. This method involves the concept of prior information to reconstruct HR image. They use learned data sets of image patches capturing the relationship between LR and HR images and find appropriate patches for estimating an HR image. However, because a large amount of training data is required to obtain a robust reconstruction results, example- or learning-based SR incurs an enormous computational load.
Daniel et al.  tried to handle this problem by combining the motion- based SR and example-based SR. Based on an assumption that patches in a single natural image tend to recur many times in an image, their approach uses LR/HR pairs of patches within and across the scales of a single image. However, the quality of the reconstructed image still depends on the accuracy of motion estimation when compensating motions of the patches. In addition, the desired LR/HR pairs of patches might be insufficient when the observed image is small or severely degraded. This makes it hard to apply their approach to practical applications such as video surveillance systems.
For the point of view of estimation criteria, SR algorithms may be divided into static and dynamic SR . Static SR fuses multiple LR images to reconstruct a single HR image at a specific time point, while dynamic SR exploits the temporal evolution which reconstructs the HR image sequence. Dynamic SR requires relatively lower memory and numbers of computations than static SR, and is therefore regarded as being a more appropriate approach for real-time applications.
In this article, we propose a robust dynamic motion-based SR algorithm for LR video input. Our approach iteratively fuses the pixel data from an LR image sequence to estimate the pixel data of the HR image sequence based on the Kalman recursive estimation . To deal with the performance degradation because of the inexplicit motion estimation, we suggest a validation process to filter out the irregularly registered pixels caused by inaccurate motion estimation. By implementing the proposed validation method, our SR approach was able to show robust HR image reconstruction results, even when the motion estimates were not accurate at the sub-pixel level. Moreover, abrupt changes in the scene input video can be detected in this validation process, so the fusion of pixels from two different scenes can be prevented. Since the quality of the reconstructed images is stable even with inaccurate motion estimation with low memory usage (requires only two frame memory) because of the sequential estimation, and each updated HR frame can be viewed during the estimation process, our approach is suitable for practical applications, especially in visual surveillance systems.
The remainder of this article is structured as follows. In Section 2, we describe the image acquisition modeling and basic concept of the dynamic SR process using the Kalman filter framework. In Section 3, we describe the proposed validation method for observed image data, and in Section 4 the scene change detection process developed for the robust sequential estimation of HR video has been described. In Section 5, we demonstrate both synthetic and real real-data experiments. Section 6 concludes this effort and discusses future study.
2. Dynamic SR
In this section, we review the dynamic SR approach proposed in , which is based on the Kalman recursive estimation. The main contribution of our approach will be described in Sections 3 and 4.
2.1. Image acquisition modeling
We used the underscore notation to indicate a vector derived from an image scanned in lexicographic order . Thus, the HR frame at time t, X(t) with a size of [r2MN × 1] is the warped version of the previous HR frame where r is the resolution-enhancement factor, since M(t) with a size of [r2MN × r2MN], indicates the existing motions between the two neighboring frames. The [r2MN × 1] vector, U(t), can be explained as the system noise that represents the accuracy of the motion estimation. In Equation 2, Y(t) with a size of [MN × 1] is the observed LR image at time t, and the [r2MN × r2MN] matrix, B, describes the blur operations resulting from the sensor's point spread function. The [MN × r2MN] matrix, D, reflects the downsample operation in the image acquisition and saving. The [MN × 1] vector W(t) is the measurement noise.
Only translational (planar) motion is considered in the input video.
The blur and downsampling operation are invariant in time. This is why there are no time indices in B and D.
Both the system and measurement noise are assumed to be additive white Gaussian noise.
2.2. Kalman recursive for data fusion
where denotes the estimated state vector, i.e., the blurred HR image. Equation 5 indicates that the final estimate of the blurred HR image is the sum of the prediction (i.e., motion compensated version of the previous estimate, and innovation or measurement residual (i.e., the difference between the new observation, Y(t), and prediction) multiplied by K(t), which is the Kalman gain defined as the ratio of the prediction covariance P(t) to the innovation covariance S(t). Analogously, the updated covariance of can be derived as in Equation 6.
Since the inversion of the covariance matrix in Equation 7 is very cumbersome and requires substantial computation and memory, further assumptions are needed to achieve a faster implementation. As proven in , if the covariance matrix of V(t) denoted as C v (t) and the initial covariance are diagonal, P(t) and become diagonal for all t. This enables a pixel-by-pixel implementation, so all of the procedures from Equations 1 to 9 can be computed as a single scalar value (i.e., single pixel). A more detailed description can be found in .
To estimate and compensate the motions existing among the input frames modeled by M(t), we adopt the image registration method in frequency-domain  since it is simple and accurate for translational motions. It estimates the horizontal and vertical shifts in spatial domain by computing the phase shift in the frequency domain. Moreover, the frequency-domain approach benefits when the aliasing effect exists in input LR frames.
3. Measurement validation
Explicit motion estimation is a major factor that affects the performance of the motion-based SR algorithm as mentioned in [13, 14]. Various research efforts have been dedicated to enable precise (sub-pixel accuracy) motion estimation; however, the methods developed are insufficient to guarantee perfect motion compensation and, even though perfect motion estimation is potentially possible, it usually requires a large amount of computation.
Some novel approaches not involving accurate motion estimation were recently suggested in [10–14], but they are not suitable for practical real-time surveillance system applications because of their computation requirements. In this article, we added a validation method in the sequential estimation process to enhance erroneous reconstructed HR images caused by inexplicit motion estimations.
Since we assume that all covariance matrices including S(t) are diagonal, computing the distance of one measured frame at time t, d(t) which is referred to as the 'Mahalanobis distance' or 'Statistical distance', is the same as computing the sum of the distances of each pixel in that frame, d k (t), in Equation 11. y k (t) is the k th pixel in a measured frame Y(t) and S k (t) is the k th diagonal element of S(t). D k is the k th row of the downsampling operator D size of [1 × r 2 MN].
Y(t) in Equation 12 is the measurement at time t and Yt-1is the sequence of measurements from the initial time to time t - 1. Thus, Equation 12 represents that the conditional probability of Y(t) given the measurements up to time t - 1, namely Yt-1is normally distributed with the mean equal to the predicted measurement and the covariance equal to the innovation covariance S(t). The theoretical description for this can be found in the sections on the Kalman filter in [16, 17].
As represented in Equations 5 and 6, K(t) determines the amount of updates required for estimating and . In the proposed measurement validation method, only valid pixel values should be used in the update equations. When K(t) is equal to zero, no updates will be made in Equations 5 and 6, thus the estimations for and are only dependent on the prediction terms. In our implementation, after the new measurement is obtained, i.e., MN pixels are observed at time t, each pixel is investigated to determine whether or not it falls inside the validation region in Equation 13. After we determine the misaligned pixels among MN pixels, we can prevent them from being used in the update equations by setting those elements of K(t), whose indices correspond to the indices of misaligned pixels, to zero.
Chi-square distribution table
4. Scene change detection
Since the dynamic SR algorithm recursively fuses the pixel data from the sequentially observed images, it is highly likely for an erroneous HR estimation result to occur when the scene or contents of two adjacent frames are totally different. This problem arises frequently when the input LR video contains many different scenes or the motions in it are too large to be estimated. There is no possible motion between different frames from different scenes and, hence, these frames can never be aligned correctly. Even though the measurement validation method can detect and filter out misaligned pixels, fusing pixels from two different scenes is not a desired situation.
Instead of applying one of the conventional scene change detection methods [21, 22], we suggest a simple but effective way to detect a sudden change of scene in the input LR video by exploiting the statistical distance already discussed in the previous section.
In this article, we set the threshold value, Th to 0.3, which means that about 30% of the pixels from the current input LR frame are different from those of the previous frame. This threshold value is determined experimentally with more than ten real video data containing scene changes. If a sudden scene change is detected with this method, we reset the estimation process (i.e., reinitialize the Kalman filter). The procedure is summarized in Figure 4.
5. Experimental results
We evaluated the performance of the proposed dynamic SR algorithm with synthetic and real video data. The threshold for measurement validation was set to 15.1 for all experiments, which represents that a confidence probability of 99.99% according to the chi-square distribution table. For the deblurring method in the last step of the proposed SR algorithm, we used the classical but effective Wiener filter approach with a constant noise-to-signal ratio (NSR) to reduce the computation complexity. The parameter NSR for the Wiener filter was tuned to obtain the best performance in all experiments.
5.1. Synthetic video data test
The method in  was implemented directly from the MATLAB GUI (http://users.soe.ucsc.edu/~milanfar/software/superresolution.html). According to , they used the image registration algorithm in  which is different from the algorithm we exploited. As mentioned in the earlier sections and previous related studies, the major factor contributing to the reconstruction image result of the multi-frame SR algorithm is the accuracy of the image registration. Thus, if a different image registration algorithm is used in the reference method, we cannot say that the improved HR image result is completely because of the proposed measurement validation. For a fair comparison, we also implemented the method in  using the frequency-domain image registration algorithm  which is used in the proposed method. Therefore, we compared the proposed method with two reference methods, one from the author's website and the other from our own implementation by modifying the image registration part. In addition, we applied the Wiener filter to the method in , instead of the bilateral-total variation (BTV) regularization to see the effect of the measurement validation only. The quality of the reconstructed HR image is evaluated quantitatively with the PSNRc (Peak SNR) metric.
5.2. Real video data test
Assuming that a sufficient number of LR frames are available and the proper image registration algorithm is used for compensating the motions existing among the LR frames, multi-frame SR generally outperforms the single image interpolation method. In the extreme case where we do not register the LR frames at all, the estimated HR image result will be worse than the Bicubic interpolation result. However, if we apply the measurement validation while still not registering all LR frames, the HR image result will be almost the same as the initial estimated HR image since most of the unregistered LR pixels will be regarded as invalid. Thus, if we set the initial estimated HR image as the Bicubic interpolated one of the initial LR frames, the HR image result obtained with the proposed method cannot be worse than the Bicubic interpolation result even though most of the LR data are excluded.
If all of the frames are aligned perfectly or well enough to fall in the preset validation region, all of the measured pixel values will contribute to the HR image estimation process. The benefit of the measurement validation process is that it prevents the misaligned pixel values from contributing to the HR image estimation. By setting the confidence level for the image registration result (i.e., the threshold for the validation region), we can exclude undesired updates of the pixel values. Thus, it becomes more beneficial when there is a higher possibility of misalignment because of the poor performance of the image registration algorithm or because of the existence of LR frames with fast motion. This is the reason why the results obtained with the proposed method in Figure 10d,h show more robust performance when large motion estimation errors occur frequently.
5.3. Scene change detection performance test
In this article, we proposed a robust dynamic SR algorithm to alleviate the performance degradation because of inaccurate motion estimation and sudden scene changes. We adopted the dynamic SR algorithm based on the Kalman filter approach, because of its effectiveness when applied to real-time applications. When the size of the output super-resolved image is about 200 × 200, the proposed dynamic SR algorithm estimates the HR images sequentially at a speed of over 20 fps while necessitating a memory size corresponding to only two frames.
In the case of misalignment caused by motion estimation error, the proposed measurement validation method determines whether each of the pixels is suitable for data fusion or not with the statistical distance of intensity. It is preferable to set the pixels with a large distance as invalid and filter them out after the estimation process enters the steady state. Otherwise, the estimated HR pixels tend to remain the same as the previous LR pixel since every input pixel with a large intensity difference would be filtered out and, hence, the update process in Kalman filtering would be prevented. The starting point of the measurement validation and the appropriate threshold remain as an ongoing research topic.
In addition, we developed a scene change detection method to handle various input videos containing one or more scene changes. By virtue of the proposed scene change detection method, we can handle input video containing more than one scene. Adaptive threshold setting for the scene change detection method is preferable for robust detection performance, and so this remains as a future study. Throughout this study, we fixed, defined a relatively large validation region, V(γ), whose threshold is equal to 15.1, because we assumed that the image registration algorithm performs well enough to align most of the LR frames correctly. If we can predict the accuracy of image registration, we can control the validation region by varying the threshold, γ.
As shown in the several representative experiments, a considerable degree of enhancement and the restoration of the deteriorated visual information can be achieved by the proposed SR algorithm. Especially, for input images of small size, such as human face and license plate images, the proposed SR algorithm is appropriate for real-time visual surveillance applications considering the processing speed and the visual quality of the reconstructed image.
aThe threshold value has no digital unit since d(t) is a normalized random variable (i.e., statistical distance). bThe test video can be downloaded from http://users.soe.ucsc.edu/~milanfar/software/sr-datasets.html. cThe PSNR of two images X and Y of size M by N is defined as .
This research was supported by the Seoul R&BD Program (WR080951).
- Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Proc. Mag 2003,20(3):21-36. 10.1109/MSP.2003.1203207View ArticleGoogle Scholar
- Elad M, Feuer A: Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Process 1997,6(12):1646-1658. 10.1109/83.650118View ArticleGoogle Scholar
- Elad M, Hel-Or Y: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Trans. Image Process 2001,10(8):1187-1193. 10.1109/83.935034View ArticleGoogle Scholar
- Farsiu S, Robinson D, Elad M, Milanfar P: Fast and robust multiframe super resolution. IEEE Trans. Image Process 2004,13(10):1327-1344. 10.1109/TIP.2004.834669View ArticleGoogle Scholar
- Elad M: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Trans. Image Process 2001,10(8):1187-1193. 10.1109/83.935034MathSciNetView ArticleGoogle Scholar
- Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Trans. Image Process 2006,15(1):141-159.View ArticleGoogle Scholar
- Elad M, Feuer A: Super-resolution reconstruction of image sequences. IEEE Trans. Pattern Anal. Mach. Intell 1999,21(9):817-834. 10.1109/34.790425View ArticleGoogle Scholar
- Farsiu S, Elad M, Milanfar P: Video-to-video dynamic super-resolution for grayscale and color sequences. EURASIP J. Appl. Signal Process 2006, 1-15. Article ID 61859Google Scholar
- Narayanan B, Hardie RC, Barner KE, Shao M: A computationally efficient super-resolution algorithm for video processing using partition filters. IEEE Trans. Circuits Syst. Video Technol 2007,17(5):621-634.View ArticleGoogle Scholar
- Protter M, Elad M, Takeda H, Milanfar P: Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans. Image Process 2009,18(1):36-51.MathSciNetView ArticleGoogle Scholar
- Freeman W, Jones T, Pasztor E: Example-based super-resolution. Comput. Graph. Appl 2002,22(2):56-65. 10.1109/38.988747View ArticleGoogle Scholar
- Glasner D, Bagon S, Irani M: Super-resolution from a single image. International Conference on Computer Vision (ICCV) 2009.Google Scholar
- Protter M, Elad M: Super resolution with probabilistic motion estimation. IEEE Trans. Image Process 2009,18(8):1899-1904.MathSciNetView ArticleGoogle Scholar
- Takeda H, Milanfar P, Protter M, Elad M: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process 2009,18(9):1958-1975.MathSciNetView ArticleGoogle Scholar
- Chaudhuri S, Manjunath J: Motion-free Super-Resolution. Springer; 2005.Google Scholar
- Louis L: Statistical Signal Processing. Scharf, Addison-Wesley Pub. Co; 1991.Google Scholar
- Bar-Shalom Y, Fortmann TE: Tracking and Data Association. Academic Press, Inc; 1988.Google Scholar
- Vandewalle P, Susstrunk S, Vetterli M: A frequency domain approach to registration of aliased images with application to super-resolution. EURASIP J. Appl. Signal Process 2006, 1-14. Article ID 71459Google Scholar
- Ku BH, Lee YH, Hong WY, Ko H: Suppressing ghost targets via gating and tracking history in Y-shaped passive linear array sonars. IEEE Trans. AES 2011,47(3):1605-1616.Google Scholar
- Ko H, Lee IK, Lee JH, Han D: Effective multi-vehicle tracking in nighttime condition using imaging sensors. IEICE Trans-Inform. Syst 2003,E86-D(9):1887-1895.Google Scholar
- El-Qawasmeh E: Scene change detection schemes for video indexing in uncompressed domain. Informatica 2003,14(1):19-36.Google Scholar
- Ngo C, Pong T, Chin R, Zhang H: Motion-based video representation for scene change detection. Int. J. Comput. Vis 2002,50(2):127-142. 10.1023/A:1020341931699View ArticleGoogle Scholar
- Bergen JR, Anandan P, Hanna KJ, Hingorani R: Hierarchical model-based motion estimation. Proceedings of European Conference on Computer Vision (ECCV '92), 237-252., Santa Margherita Ligure, Italy 1992.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.