# Covariance Tracking via Geometric Particle Filtering

- Yunpeng Liu
^{1, 2, 3, 4}Email author, - Guangwei Li
^{5}and - Zelin Shi
^{1, 3}

**2010**:583918

https://doi.org/10.1155/2010/583918

© Yunpeng Liu et al. 2010

**Received: **30 November 2009

**Accepted: **24 June 2010

**Published: **13 July 2010

## Abstract

Region covariance descriptor recently proposed has been approved robust and elegant to describe a region of interest, which has been applied to visual tracking. We develop a geometric method for visual tracking, in which region covariance is used to model objects appearance; then tracking is led by implementing the particle filter with the constraint that the system state lies in a low dimensional manifold: affine Lie group. The sequential Bayesian updating consists of drawing state samples while moving on the manifold geodesics; the region covariance is updated using a novel approach in a Riemannian space. Our main contribution is developing a general particle filtering-based racking algorithm that explicitly take the geometry of affine Lie groups into consideration in deriving the state equation on Lie groups. Theoretic analysis and experimental evaluations demonstrate the promise and effectiveness of the proposed tracking method.

## Keywords

## 1. Introduction

Visual tracking in an image sequence, which is now an active area of research in computer vision, is widely applied to vision guidance, surveillance, robotic navigation, human-computer interaction, and so forth. Dynamic deformation of object is a distinct problem in image-based tracking.

Conventional correlation-based trackers [1, 2] use either a region's gray information or edges and other features as the target signatures, but it is difficult to solve the problem of object region deformation in the tracking. Over the last 10 years, numerous approaches [3–10] have been proposed to address this problem. The main idea of them is molding geometric parameter models for the image motions of points within a target region. The parameter models including affine model, projective model, or other nonlinear models. The classic Lucas-Kanade tracker [3, 4] and Meanshift tracker [5] get the model parameters through gradient descent which minimizes the difference between the template and the current region of the image. These methods are computationally efficient. However, the methods may converge to a local maximum, they are sensitive to background clutter, occlusion, and quick moving objects. These problems can be mitigated by stochastic methods which maintain multiple hypotheses in the state space and in this way, achieve more robustness to the local maximum. Among various stochastic methods, particle filters [5–10] are very successful. Particle filters provide a robust tracking framework as they are neither limited to linear systems nor require the noise to be Gaussian. Particle filters simultaneously track multiple hypotheses and recursively approximate the posterior probability density function in the state space with a set of random sampled particles.

Many papers, such as [5–10] utilize particle filter method to track deformable target. They use affine transform as parameter model, and the six affine parameters were treated as a vector. However, the affine parameters belong to spaces which are not vector spaces, but instead a curved Lie group. In general, the system state of the particle filter lies in a constrained subspace whose dimension is much lower than the whole space dimension. Only a few recent papers have tried to use the geometry of the manifold to design Bayesian filtering algorithms [11, 12]. However, there is little discussion in the literature using the intrinsic geometry of manifold to develop particle filter-based tracking algorithms.

Object representation is one of major components for a typical visual tracker. Extensive researches have been done on this topic. Recently Tuzel et al. [13, 14] proposed an elegant and simple solution to integrate multiple features. In this method, covariance matrix was employed to represent the target. Using a covariance matrix to represent the target (region covariance descriptor) has many advantages: ( ) it embodies both spatial and statistical properties of the objects; ( ) it provides an elegant solution to fuse multiple features and modalities; ( ) it has a very low-dimensionality; ( ) it is capable of comparing regions without being restricted to a constant window size; and ( ) the estimation of the covariance matrix can be easily implemented.

In this paper, we integrate covariance descriptor into Mont Carlo technique for visual tracking, study the geometry structure of affine Lie groups, and propose a tracking algorithm through particle filtering on manifolds, which implement the particle filter with the constraint that the system state lies in a low dimensional manifold, The sequential Bayesian updating consists drawing state samples while moving on the manifold geodesics; this provides a smooth prior for the state space change. The regions covariance matrices are updated using a novel approach in a Riemannian space. Theoretic analysis and experimental results shows the promise and effectiveness of the approach proposed.

The paper is organized as follows. In Section 2, The mathematical background is described. Section 3 shows the object regions descriptor and the new update solution for those descriptors. Section 4 describes the tracking algorithm using geometric particle filtering. Results on real image sequences for evaluating algorithm performance are discussed in Section 5.Section 6 concludes this paper.

## 2. Manifold and Lie Group

The tools used here come primarily from differential geometry. For more information on these subjects, the reader is referred to [15, 16].

A manifold is a topological space that is locally similar to an Euclidean space. Intuitively, we can think of a manifold as a continuous surface lying in a higher dimensional Euclidean space. Analytic manifolds satisfy some further conditions of smoothness [16]. From now onwards, we restrict ourselves to analytic manifolds and by manifold we mean analytic manifold.

are analytic [15]. The local neighborhood of any group element can be adequately described by its tangent-space. The tangent-space at the identity element forms its Lie algebra.

The set of nonsingular square matrices forms a Lie group where the group product is modeled by matrix multiplication, usually denoted by for the general linear group of the order . Lie groups are differentiable manifolds on which we can do calculus.

In our task, we use affine transformation as parameter model. The set of all affine transformation forms a matrix Lie group.

## 3. Region Covariance Descriptor

where is the number of pixels in the region. is the mean of the feature points.

where and are the pixel location in ; is the gray value; and are first derivatives of ; In this way, the region is mapped into a covariance matrix.

In a tracking process, the objects appearance changes over time. This dynamic behavior requires a robust temporal update of the region covariance descriptors and the definition of dissimilarity metric for the region covariance. The important question here is how to measure the dissimilarity between two region covariance matrices and how to update the regions covariance matrix in the next time slot. Note that the covariance matrices do not lie on Euclidean space. For example, the space is not closed under multiplication with negative scalars. So, it is necessary to get the dissimilarity between two covariance matrices in a different space. To overcome this problem a Riemannian Manifold is used.

### 3.1. Dissimilarity Metric

The dissimilarity between two regions covariance matrices can be given by the distance between two points of the manifold , considering that those points are the two regions.

### 3.2. Covariance Update

where is the average distance between two points on a Riemannian Manifold (the updated covariance matrix). This update means that the present covariance is more important than the previous covariances. Since we are tracking objects that can change over time, the last information about them is more reliable.

## 4. Tracking Model

Equation (11) is called the prediction equation and (12) is called the update equation. The tracking process is governed by the observation model , where we estimate the likelihood of observing , and the dynamical model between two states .

### 4.1. Dynamical Model

where is a discrete-time trajectory on a six-dimensional affine Lie group, is a velocity on the corresponding Lie algebra, are Gaussian white zero-mean stochastic processes.

The tracking algorithm will not require the explicit functional form of the prior density; it will be dependent on the samples generated from the prior density. In a Markovian time-series analysis, often there is a standard characterization of a time-varying posterior density, in a convenient recursive form. This characterization relates an underlying Markov process to its observations at each observation time via a pair of state transition equations. The following algorithm specifies a procedure to sample from the conditional prior :

Algorithm 1.

### 4.2. Observation Model

where be covariance features of the template image, and denote covariance features at the transformation .

### 4.3. Sequential Monte Carlo Approach

The Monte Carlo idea is to approximate the posterior density of by a large number of samples drawn from it. Having obtained the samples, any estimate of (MMSE, MAP, etc.) can be approximated using sample averages.

A recursive formulation, which takes samples from and generates the samples from in an efficient fashion, is desirable. We accomplish this task using ideas from sequential methods and importance sampling. Assume that, at the observation time , we have a set of samples from the posterior, . Following are the steps to generate the set .

Prediction

The first step is to sample from given the samples from . According to (11), is the integral of the product of a marginal and a conditional density. This implies that, for each element , by generating a sample from the conditional, we can generate a sample from . In our case, this is accomplished using Algorithm 1. Now we have samples from ; these samples are called predictions, but we have used a geodesics prediction different to classic particle filter on vector space.

Resampling

Then, resample values from the set according to probability . These values are desired samples from the posterior . Denote the resampled set by , .

Averaging on the Lie Group

Now that we have samples from the posterior , we can average them appropriately to approximate the posterior mean of .

It may be recalled that for a vector space, the sample mean or average of a set is given by . However, such a notion cannot be applied directly to elements of a group manifold. There are at least two ways of define a mean value on a manifold: extrinsic means and intrinsic means. The extrinsic mean depends on the geometry of the ambient space and the embedding. The intrinsic mean is defined using only the intrinsic geometry of the manifold. In general, the intrinsic average is preferable over the extrinsic average but is often hard to compute due to the nonlinearity of the Riemannian distance function and the need to parameterize the group manifold. However, as we will see here, for matrix Lie groups the intrinsic average can be computed efficiently. In several applications, the Lie algebra is used for computing intrinsic means of points having Lie group structure [17–19]. We adopt the similar idea to obtain the intrinsic mean of the affine lie group.

### 4.4. Detail of Tracking Algorithm

Algorithm 2.

Generate samples from the prior distribution . Set initial weights .

Draw from the conditional prior according to Algorithm1.

Compute the probability according to (18).

Generate samples from the set with the associated probabilities . Denote these samples by .

Calculate the sample average according to (19) which is the target state. Set and go to step .

## 5. Experimental Results

In order to evaluate the performance of the proposed tracking algorithm based on geometric particle filtering and the new update method. We start by comparing proposed algorithm (referred as ) with the tracking algorithm based on Particle filtering on vector space ( ) [5–10] with the same real image sequences. After that, we evaluated the proposed update method with the one previously proposed in the literature. We also tested the proposed algorithm under varying illumination conditions. These algorithms are implemented in C++ running on an Intel Core-2 2.5 GHz processor with 2 GB memory.

### 5.1. Compared with VPF

Two typical image sequences where the objects undergo large changes in pose and scale were tested using
and *VPF*. Thus, the performance of the two algorithms has been compare with the same experimental setup.

*VPF*are shown in Figure 3. For a better visualization, we just show the tracking results of four representative frames 52, 87, 135 and 148. The frame number is shown on the top left corner of each image. The value below each image is the likelihood of the matching, the smaller the matching error, the larger the likelihood. Figure 4(a) shows the likelihood curves.

In summary, we observe that the tracker outperforms in the scenarios of scale, rotation, and shear changes of target.

### 5.2. Update Method

To evaluate the effectiveness of the proposed update solution, we compare the result of it with the ones obtained by the Porikli update proposed in [14]. We compare the likelihood curves between above two image sequences; the results were obtained by just changing the update method.

Execution time of two update methods size.

Method | Execution time (ms) |
---|---|

Porikli update | 129.6 |

New update | 0.6 |

### 5.3. Illumination Changes

### 5.4. Experimental Analyses

The algorithm described in the paper consists of three components.

- (1)
We develop a general particle filtering based tracking algorithm that explicitly take the geometry of affine Lie groups into consideration in deriving the state equation on Lie groups. This one is our main contribution and the dominating factor in improving the tracking performance.

- (2)
We use region covariance descriptor to model objects appearance, the edge-like information more robust to the illumination changes than the image grayscale can be simultaneously considered with the image grayscale information and pixel spatial information, and the consequence is the quite robust tracking results as seen in Figure 7.

- (3)
We updated region covariance using a novel approach in a Riemannian space. The new update method has improved the real-time performance.

So the order of importance to the performance among these components is 1, 2, 3.

## 6. Conclusion

In this paper, we have proposed a visual tracking method, which integrate covariance descriptor into Mont Carlo tracking technique for visual tracking. The distinct advantage of this new approach is carrying Sequential Monte Carlo method over the affine Lie group, which consider the geometry prior of the parameter space. Theoretic analysis and experimental results shows the promise and effectiveness of the approach proposed.

This paper highlights the role of Monte Carlo methods in statistical inferences over affine lie group for visual tracking problem. There are several directions for extending the new idea. One is to consider more general differentiable manifolds beyond the affine lie group. In addition, we can deepen and broaden this research to other image processing problems.

## Declarations

### Acknowledgments

This work is partly supported by the National Natural Science Foundation of China (Grant no. 60603097) and the National Defense Innovation Foundation of Chinese Academy Sciences (CXJJ-65).

## Authors’ Affiliations

## References

- Montera DA, Rogers SK, Ruck DW, Oxley ME: Object tracking through adaptive correlation.
*Optical Engineering*1994, 33: 294-302. 10.1117/12.152013View ArticleGoogle Scholar - Parry HS, Marshall AD, Markham KC: Tracking targets in FLIR images by region template correlation.
*Acquisition, Tracking, and Pointing XI, April 1997, Orlando, Fla, USA, Proceedings of SPIE*3086: 221-232.View ArticleGoogle Scholar - Hager GD, Belhumeur PN: Efficient region tracking with parametric models of geometry and illumination.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*1998, 20(10):1025-1039. 10.1109/34.722606View ArticleGoogle Scholar - Baker S, Matthews I: Lucas-Kanade 20 years on: a unifying framework.
*International Journal of Computer Vision*2004, 56(3):221-255.View ArticleGoogle Scholar - Zhang H, Huang W, Huang Z, Li L: Affine object tracking with kernel-based spatial-color representation.
*Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05), June 2005*293-300.Google Scholar - Isard M, Blake A: Condension-conditional density propagation for visual tracking.
*International Journal of Computer Vision*1998, 29(1):5-28. 10.1023/A:1008078328650View ArticleGoogle Scholar - Zhou SK, Chellappa R, Moghaddam B: Visual tracking and recognition using appearance-adaptive models in particle filters.
*IEEE Transactions on Image Processing*2004, 13(11):1491-1506. 10.1109/TIP.2004.836152View ArticleGoogle Scholar - Rathi Y, Vaswani N, Tannenbaum A, Yezzi A: Tracking deforming objects using particle filtering for geometric active contours.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*2007, 29(8):1470-1475.View ArticleGoogle Scholar - Odobez J-M, Gatica-Perez D, Ba SO: Embedding motion in model-based stochastic tracking.
*IEEE Transactions on Image Processing*2006, 15(11):3514-3530.View ArticleGoogle Scholar - Ross DA, Lim J, Lin R-S, Yang M-H: Incremental learning for robust visual tracking.
*International Journal of Computer Vision*2008, 77(1–3):125-141.View ArticleGoogle Scholar - Srivastava A, Klassen E: Monte Carlo extrinsic estimators of manifold-valued parameters.
*IEEE Transactions on Signal Processing*2002, 50(2):299-308. 10.1109/78.978385View ArticleGoogle Scholar - Snoussi H, Mohammad-Djafari A: Particle filering on Riemannian manifold.
*Proceedings of the 27th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, 2006, AIP Conference Proceedings*872: 219-226.View ArticleGoogle Scholar - Tuzel O, Porikli F, Meer P: Region covariance: a fast descriptor for detection and classification.
*Proceedings of the 9th European Conference on Computer Vision (ECCV '06), 2006, Lecture Notes in Computer Science*3952: 589-600.Google Scholar - Porikli F, Tuzel O, Meer P: Covariance tracking using model update based on Lie algebra.
*Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), June 2006, New York, NY, USA*1: 728-735.Google Scholar - Hall BC:
*Lie Algebras, and Representations: An Elementary Introduction*. Springer, New York, NY, USA; 2003.MATHGoogle Scholar - Berger M:
*A Panoramic View of Riemannian Geometry*. Springer, Berlin, Germany; 2003.View ArticleMATHGoogle Scholar - Begelfor E, Werman M: How to put probabilities on homographies.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*2005, 27(10):1666-1670.View ArticleGoogle Scholar - Govindu VM: Lie-algebraic averaging for globally consistent motion estimation.
*Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '04), July 2004, Washington, DC, USA*1: 684-691.Google Scholar - Tuzel O, Subbarao R, Meer P: Simultaneous multiple 3D motion estimation via mode finding on lie groups.
*Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV '05), October 2005, Beijing, China*1: 18-25.View ArticleGoogle Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.