Optical Flow and Principal Component Analysis-Based Motion Detection in Outdoor Videos
© Kui Liu et al. 2010
Received: 6 December 2009
Accepted: 16 January 2010
Published: 8 March 2010
We propose a joint optical flow and principal component analysis (PCA) method for motion detection. PCA is used to analyze optical flows so that major optical flows corresponding to moving objects in a local window can be better extracted. This joint approach can efficiently detect moving objects and more successfully suppress small turbulence. It is particularly useful for motion detection from outdoor videos with low quality. It can also effectively delineate moving objects in both static and dynamic background. Experimental results demonstrate that this approach outperforms other existing methods by extracting the moving objects more completely with lower false alarms.
The detection of moving objects is critical in many defense and security applications, where motion detection is usually performed in a preprocessing step, a key to the success in the following target tracking and recognition. Many videos used in defense and security applications are outdoor videos whose quality may be degraded by various noisy sources, such as atmospheric turbulence, and sensor platform scintillation. Meanwhile, moving objects may be very small occupying a few pixels only, which makes motion detection very challenging. Under this circumstance, existing approaches may generate significant amount of false alarms.
Motion detection has been extensively investigated [1–3]. Many research works are conducted for indoor videos with large objects. As one of the major techniques, optical flow-based approaches have been widely used for motion detection. There are two classic methods of optical flow computation in computer vision: Horn-Schunck (HS) method and Lucas-Kanade (LK) method [4–7]. Both of them are based on the two-frame differential algorithms. LK method may not perform well in dense flow field; on the other hand, HS method can detect minor motion of objects and provide a 100% flow field . Thus, we focus on HS method for optical flow computation in our research. Considering outdoor videos with low quality, special care needs to be taken in order to better extract features related to moving objects from optical flows while suppressing false alarms.
Principal component analysis (PCA) is a typical approach in multivariate analysis . It is also named the discrete Karhunen-Loève transform (KLT) or the Hotelling transform . PCA includes the eigen-decomposition of a data covariance matrix or singular value decomposition of a data matrix, usually after mean centering. It projects the original data onto an orthogonal subspace, where each direction is mutually decorrelated and major data information is present in the first several principal components (PCs). For optical flows in a local window, moving objects have consistent flows while pixels with only turbulence have random flows. Thus, if PCA is applied to the two-dimensional (2D) data of optical flows, the difference between desired motion pixels and random motion pixels may be magnified because their contributions to the two eigenvalues are very different; the contribution from random motion pixels can be very small, even to the second eigenvalue. Experimental results show that this approach actually is an effective way of analyzing outdoor videos; it can reduce false alarms for videos with either static or dynamic background, and it is also useful to delineate the size of moving objects.
This paper is organized as follows. Section 2 explains the proposed method based on optical flow and PCA. Section 3 presents experiments using ground-based and airborne videos. Section 4 draws the conclusion.
2. Proposed Method
HS method is a special approach of using global constraint of smoothness to express a brightness variation in certain areas of the frames in a video sequence. It is also a specially defined framework to lay out the smoothness of the flow field. Let represent the brightness of a pixel at coordinates and the frame. According to , the image constraint at with Taylor series can be expressed as
where and are the and components of the velocity or optical flow of respectively, and and are the derivatives of the image at in the corresponding directions. A constrained minimization problem can be formulated to calculate optical flow vector for the th frame:
where and are the estimated local average optical flow velocities, and is a weighting factor. A larger value of results in a smoother flow; in our experiments using 8-bit videos, it is empirically set to be 30000. Based on the norm of an optical flow vector, one can determine if the motion exists or not, while the direction of this vector provides the motion orientation.
where is the optical flow matrix after mean removal. After eigen-decomposition, two eigenvalues are assigned to the central pixel of the mask. Motion detection is accomplished by analyzing or thresholding the eignenvalue (s). Since is the major flow component and is the minor flow component, it may be more effective to considering than the values in the original space.
Intuitively, only needs to be considered because it corresponds to the major flow component and corresponds to the minor flow component or even turbulence. An appropriate threshold can be determined by using the Ostu's method on the histogram . However, in practice, should be considered as well since pixels inside object boundaries usually have quite large but not Thus, thresholding may need to be taken on the histogram; a pixel is claimed to have motion if either or are above the corresponding thresholds.
In Step we may use the optical flow data from multiple frames. For instance, optical flow data from Frames 1 and 2 can be combined with optical flow data from Frames 2 and 3; this may help to emphasize the desired optical flows of moving objects and to emphasize the randomness of turbulence.
In the experiments, we use two adjacent frames, a mask, and only for thresholding. It is to show that such simplest implementation is sufficient to provide better performance than other widely used techniques.
In the experiments, videos with both static and dynamic backgrounds were analyzed. They were taken by a commercial Sony Camcorder. We compared our proposed method with the original optical flow method, the motion detection methods based on Kalman filtering , background modeling using Gaussian mixture model , difference-based spatial temporal entropy image (DSTEI) , and forward-backward motion history images (MHI) . They were chosen for comparison because they are either typical methods or designed specifically for more complicated videos (e.g., those with dynamic background).
3.1. Experiment 1: Ground-Based Video with Relatively Large Object
3.2. Experiment 2: Airborne Videos with Small Objects
The second experiment used an airborne video with low quality. It was taken by the camcorder mounted on the helicopter in the video shown in Experiment 1. In addition to atmospheric turbulence, scintillation from the airborne platform (i.e., the small helicopter) further degraded the video quality. As shown in Figure 9, there were three moving vehicles on the highway, highlighted in yellow circles. They consisted of only a few pixels. The two frames were pre-registered using the method in .
In this paper, we propose a joint optical flow and PCA approach for motion detection. Instead of considering the original optical flow, the two eigenvalues of the covariance matrix of local optical flows are analyzed. Since the first eigenvalue represents the major motion component and the second eigenvalue represents the minor motion component or turbulence, they are more useful to detect true motions while more successfully suppressing false alarms. The proposed method is also effective in extracting the actual size of moving objects.
The computational complexity involved in PCA includes the calculation of covariance matrix of local optical flow and its eigen-decomposition. For a mask of size the number of multiplications in calculating the covariance matrix of size is and complexity of eigen-decomposition is generally For an image frame with m pixels, the total computational complexity is It can be reduced to if using iterative PCA (IPCA) as discussed in , where is a small integer. As the future work, we will investigate the performance when using IPCA to expedite motion detection.
This research was supported by National Geospatial-Intelligence Agency of the United States.
- Mitiche A, Bouthemy P: Computation and analysis of image motion: a synopsis of current problems and methods. International Journal of Computer Vision 1996, 19(1):29-55. 10.1007/BF00131147View ArticleGoogle Scholar
- Hu W, Tan T, Wang L, Maybank S: A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man and Cybernetics Part C 2004, 34(3):334-352. 10.1109/TSMCC.2004.829274View ArticleGoogle Scholar
- Yilmaz A, Javed O, Shah M: Object tracking: a survey. ACM Computing Surveys 2006, 38(4):1-45.View ArticleGoogle Scholar
- Horn BKP, Schunck BG: Determining optical flow. Artificial Intelligence 1981, 17(1–3):185-203.View ArticleGoogle Scholar
- Horn BKP, Schunck BG: Determining optical flow. Artificial Intelligence 1993, 59(1-2):81-87. 10.1016/0004-3702(93)90173-9View ArticleGoogle Scholar
- Lucas BD, Kanade T: An iterative image registration technique with an application to stereo vision. Proceedings of the 7th International Joint Conference on Artificial Intelligence, 1981 2: 674-679.Google Scholar
- Bruhn A, Weickert J, Schnorr C: Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. International Journal of Computer Vision 2005, 61(3):211-231.View ArticleGoogle Scholar
- Pearson K: On lines and planes of closest fit to systems of points in space. Philosophical Magazine 1901, 2(6):559-572.View ArticleMATHGoogle Scholar
- Hotelling H: Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology 1933, 24(6):417-441.View ArticleMATHGoogle Scholar
- Otsu N: A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 1979, 9(1):62-66.MathSciNetView ArticleGoogle Scholar
- Karmann K-P, Brandt AV: Moving object recognition using an adaptive background memory. In Time-Varing Image Processing and Moving Object Recognition. Volume 2. Edited by: Cappellini V. Elsevier, Amsterdam, The Netherlands; 1990:297-307.Google Scholar
- Kaewtrakulpong P, Bowden R: An improved adaptive background mixture model for real-time tracking with shadow detection. Proceedings of 2nd European Workshop on Advanced Video-Based Surveillance Systems(AVBS '01), September 2001Google Scholar
- Jing G, Siong CE, Rajan D: Foreground motion detection by difference-based spatial temporal entropy image. Proceedings of IEEE TENCON Conference, 2004 A379-A382.Google Scholar
- Yin Z, Collins R: Moving object localization in thermal imagery by forward-backward MHI. Proceedings of IEEE Computer Vision and Pattern Recognition Workshops (CVPRW '06), 2006Google Scholar
- Bergen JR, Anandan P, Hanna KJ, Hingorani R: Hierarchical model-based motion estimation. In Proceedings of the 2nd European Conference on Computer Vision, 1992. Volume 588. Springer; 237-252.Google Scholar
- Yang H, Ma B, Du Q: Very small moving object detection from airborne videos using corners in differential images. Proceedings of IEEE International Conference on Image Processing, 2010Google Scholar
- Du Q, Fowler JE: Low-complexity principal component analysis for hyperspectral image compression. International Journal of High Performance Computing Applications 2008, 22(4):438-448. 10.1177/1094342007088380View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.