A Grassmann graph embedding framework for gait analysis

Abstract

Gait recognition is important in a wide range of monitoring and surveillance applications. Gait information has often been used as evidence when other biometrics is indiscernible in the surveillance footage. Building on recent advances of the subspace-based approaches, we consider the problem of gait recognition on the Grassmann manifold. We show that by embedding the manifold into reproducing kernel Hilbert space and applying the mechanics of graph embedding on such manifold, significant performance improvement can be obtained. In this work, the gait recognition problem is studied in a unified way applicable for both supervised and unsupervised configurations. Sparse representation is further incorporated in the learning mechanism to adaptively harness the local structure of the data. Experiments demonstrate that the proposed method can tolerate variations in appearance for gait identification effectively.

1 Introduction

The use of CCTV video cameras for surveillance is common in public and commercial establishments like banks, shopping malls, parks, and railway stations. Most of the current video surveillance systems require human operators to constantly supervise the cameras. In other words, the effectiveness of the system is largely dependent on the vigilance of the person monitoring the system. To resolve this shortcoming, research is under way to develop automated systems for real-time cameras monitoring. Among the efforts, gait recognition is a popular study for automatic human identification. Gait recognition is a biometric technology that identifies people based on the manner they walk. This technology is suitable for person identification at a distance when other biometrics like face, iris, or fingerprint might be obscured or at too low a resolution. In many situations, gait is the only evidence available from a crime scene [1].

With the advent of visual surveillance, it is not difficult to obtain multiple-viewpoint shot of a subject or video outputs over a period of time. These multiple sets of images can be combined to yield better performance as compared to single-shot images. Subspace-based approaches have been shown effective in modeling data consisting of multiple sets of images [2]. For example, Jacobs et al. [3] showed that illumination on human faces can be modeled as a nine-dimensional subspace under mild assumptions. Subsequent to this finding, sets of images of the same person under varying lighting conditions are often modeled as low dimensional subspaces [4–6]. While a subspace is a linear space, the collection of linear subspaces is a completely different space known as the Riemannian manifold [7]. More formally, the d-dimensional subspace in ℝⁿ is called the Grassmann manifold, named after the famous mathematician Hermann Günther Grassmann [8]. The Grassmann manifold has long been known for its fascinating mathematical properties. However, its applications in computer vision and machine learning have appeared rather recently.

Turaga et al. [9] demonstrated the use of computer vision applications such as video-based face recognition, activity recognition, and image set-based object recognition on the Grassmann manifold. The Grassmann manifold structure of the face shape is also utilized in [10] for age estimation and face verification. In [11], geometrical structure of the Grassmann manifold was exploited for visual tracking scheme.

Hamm and Lee [12] showed that using a suitable Grassmann kernel, the Grassmann space can be embedded to a higher-dimensional reproducing kernel Hilbert space (RKHS) where many Euclidean algorithms can be generalized. Subsequent to this finding, several studies extended the use of dimension reduction methods on the Grassmann manifold [13, 14]. Considerable improvement in recognition accuracy has been reported for this application.

In this paper, we propose an approach called Grassmann graph embedding (GGE) for gait analysis. Motivated by the success of the graph embedding (GE) framework [15], we show how GE can be integrated in the Grassmann manifold for the gait recognition problem through the use of well-defined kernel functions on the manifold. We provide a general formulation that supports both supervised and unsupervised dimension reduction mechanisms. We further attach semantic meaning to the gait data by incorporating sparse representation in our learning mechanism.

The rest of the paper is organized as follows. In Section 2, we review the different approaches to gait recognition. In Section 3, we provide the background of the methods used in this paper. The overall framework for the proposed Grassmann GE learning is described in Section 4. In Section 5, we present experimental results on different settings. Lastly, some concluding remarks are given in Section 6.

2 Related work

We provide a background study for the methods addressing view angle, clothing, and also speed factors in gait recognition. Besides, some subspace-based techniques related to our work are also reviewed in this Section.

2.1 Gait recognition under various viewing angles

Appearance change due to varying view angles is one of the greatest challenges in gait analysis. Studies show that single-view gait recognition performance drops when the view angle changes [16, 17]. Current approaches to gait recognition under various viewing angles can be classified into one of the three major categories: (1) extraction of view-invariant gait feature, (2) generation of three-dimensional (3D) gait information, and (3) learning projection or mapping functions to transform gait features from various views into a common feature space.

The first approach attempts to find gait features that are invariant to view changes. Jean et al. [18] introduced body part trajectories as the view-invariant feature. The 2D trajectories of the feet and head were normalized to make them appear as if they were always seen from the front-to-parallel viewpoint. A method was proposed by Kale et al. [19] to synthesize the lateral view from arbitrary view through perspective projection in a sagittal plane. Recently, Goffredo et al. [20] derived gait features based on estimated joint positions. A reconstruction method was employed to normalize the gait features from different viewpoints into the side plane. The methods in the first category can only work with limited range of view angles, and the accuracy of the methods can be affected by self-occlusion.

The methods in the second category integrate 3D information from multiple cameras to construct a gait model. An image-based rendering method was employed by Bodor et al. [21] to reconstruct the 3D view of the subject from a blend of different views. Zhao et al. [22] used video sequences acquired by multiple cameras to setup a human 3D model. Matching for the 3D models was performed using a linear time normalization technique. Yamauchi et al. [23] captured the body data using a high-resolution projector-camera system. They were able to obtain fairly accurate reconstructed synthetic human poses. The methods in the second category are able to provide reliable performance. However, these 3D analysis methods require complicated setup of a calibrated multi-camera system. Besides, these methods demand complex computation which makes them unsuitable for practical application.

The methods in the third category have some learnt mapping/projection function to normalize the gait features obtained from various viewing points to a shared feature space. Makihara et al. [24] extracted a frequency-domain gait feature using Fourier analysis. After that, a view transformation model (VTM) was used to learn a mapping function for the gait features obtained from different views. Some other variations based on VTMs had also been introduced [25–27]. Studies that utilize VTM [24–27] assume that the feature matrix in the training set can be completely decomposed into view and subject independent submatrices without overlapping elements. However, the view angle may sometimes be difficult to obtain a priori.

In [28], the correlation of gait sequences from different views was modeled using canonical-correlation analysis (CCA). The CCA strengths were directly used to match two gait sequences. Lee and Elgammal [29] presented a multi-linear generative model using higher-order singular value decomposition. View factors, body configuration factors, and gait-style factors could be obtained using such model. The methods in the third category generate more stable gait features and are less sensitive towards noise as compared to the methods in the first category. Furthermore, the methods in the third category deploy a simpler camera setup as compared to those in the second category.

2.2 Gait recognition with clothing and carrying conditions

Clothing is another challenging factor for gait recognition. The appearance of a person changes when the person wears different types of clothes. Besides, a recent study [30] shows that gait spoofing is possible by imitating the clothing of a person with similar build. These observations imply that the clothing factor yields high intraclass variation and low interclass variation which makes personal identification difficult.

Hossain et al. [31] attempted to address the clothing factor in gait recognition by proposing an approach to adaptively assign weights to different body parts based on how much that area is affected by clothing variation. For example, the head will usually be affected if a person wears a hat, while the leg will be affected if the person wears a long skirt. The algorithm assigns less weight to the head when the person wears a hat and similarly assigns less weight to the leg when the person wears a long skirt. This method thus reduces the influence of clothing by the adaptive weight tuning mechanism. However, the method makes strong assumption on the types of clothing the person wears (e.g., the clothing types must be known beforehand), and this makes it not very practical in real-life application.

Another study [32] approached the clothing factor using a random subspace method. Multiple subspaces were randomly formed using the coefficients generated by 2DPCA. A promising result was obtained as the method combined the evidences from multiple subspaces which provided different information about the clothing aspects when classification was performed.

There is also a group of researchers who introduced the use of gait energy image (GEI) with sway alignment [33] to overcome the clothing and carrying effects. Instead of taking the whole body to generate GEIs, only the area below the knee was used. The authors claimed that their method produced better accuracy as they believed that the lower part of the body was usually unaffected by the clothing and carrying conditions. Nevertheless, this method easily fails when the person's leg is obscured (e.g., the person wears a long skirt or carries a briefcase).

2.3 Gait recognition across various walking speeds

The approaches towards speed factor in gait recognition bear some resemblance to those methods addressing viewpoint variation. There are two general approaches that deal with gait with varying speeds: (1) learning mapping functions to transform the gait features from various speeds into a common walking speed and (2) extraction of speed-invariant gait feature. In the first approach, Tanawongsuwa and Bobic [34] proposed a stride normalization technique to transform the gait feature across various speeds into a common walking speed. On the other hand, Tsuji et al. [35] viewed cross-speed gait recognition as a similar problem as cross-view gait recognition and applied the VTM [24] technique to transform the gait from different speeds to a common speed for recognition.

In the second approach, Kusakunniran et al. [36] showed that the use of Procrustes shape analysis could tolerate the gait changes due to speed differences. They extended the technique to a higher-order shape configuration that could better represent the gait signature across speeds. They further introduced a differential composition model to assign different weights to different shape boundary to cope with large changes in walking speeds. Liu and Sarkar [37] proposed a population hidden Markov model to normalize the gait features based on a generic walking model. The proposed model, when combined with linear discriminant analysis, could distinguish the shapes of different subjects and suppress the differences of the same subject under various conditions, including speed changes. Tan et al. [38] represented the gait features using eight projective representations. The representation using projection from different directions yielded acceptable accuracy for gait recognition across speeds. Recently, Guan and Li [39] deployed the random subspace method [32] to address the cross-speed problem. This method also seemed to respond well towards speed changes.

2.4 Subspace-based approaches

In the computer vision community, the subspace method [40] has been used to represent an image set by a linear subspace that is spanned by all the images in the set. A number of algorithms have been proposed to measure the distances/similarities among the subspaces. Among the many distance/similarity measures, the concept of principal angle [41] between two subspaces has been widely adopted due to its efficient, accurate, and robust characteristics. Yamaguchi et al. [42] presented a method called mutual subspace method (MSM) that directly used the angles between two subspaces as the similarity score of two face image sets. Li et al. [43] further introduced the idea of weighted subspace distance to more effectively account for the characteristics of the underlying data distribution. This method was adopted by Liu et al. [44] in gait recognition to compare two subspaces comprising gait images captured from different view angles. A nonlinear extension of the principal angle method has also been presented in [45, 46].

Fukui and Yamaguchi proposed a constrained CSM (CMSM) [47] to learn a subspace in which the entire class exhibited small variance. This method greatly outperformed the original MSM. Later on, the nonlinear extension of the method using kernel trick was presented in [48]. The concept of multiple CMSM was proposed in [49] to create multiple constrained subspaces using ensemble learning, and MSM was used for classification. Inspired by linear discriminant analysis, Kim et al. [4] developed a technique that minimizes the canonical correlations of between-class sets and maximizes the canonical correlations of within-class sets. This method was shown to perform well in several object recognition problems.

2.5 Motivation and contribution

The subspace-based approach is shown to be promising in modeling video sequences. Subspaces can accommodate the effect of a wide range of variations and capture the dynamic properties in the video sequences. In many video surveillance applications, multiple snapshots of the same subject at different time instances can be obtained for recognition. Similarly, multiple images of the same subjects under varying viewpoints are also available in video camera networks. Therefore, it is natural to utilize these multiple sets of images instead of the conventional single snapshot image in our recognition task.

Clearly, the subspace-structure data resides on a nonlinear manifold. The non-Euclidean domain which suits the subspace-structure data is the Grassmann manifold. The Grassmann manifold G(m, D) is the set of m-dimensional linear subspaces of the ℝ^D. Hence, a set of linear subspaces can be perceived as points on the Grassmann manifold. Most of the computer vision algorithms are developed for data lying in ℝ^D. Applying these algorithms directly on the nonlinear manifold will yield poor accuracy as the underlying geometry of the manifold is ignored. Therefore, this paper aims to generalize the algorithm developed for ℝ^D to the Grassmann manifold through the use of well-defined Grassmann kernels.

Our primary contributions in this paper are (1) a formulation for modeling gait subspaces on the Grassmann manifold, (2) a framework to integrate supervised and unsupervised GE techniques in the Grassmann manifold, (3) a method to incorporate sparse representation in the learning algorithms, and (4) extensive experiment to corroborate the proposed approach.

A preliminary version of this paper was presented in [50], which explored the use of gait recognition on the Grassmann manifold. This paper provides the road block for modeling gait image sets on the Grassmann manifold. A local-based discriminant analysis method called Grassmann locality preserving discriminant analysis was deployed, and an encouraging result was reported. In this paper, we provide a more detailed analysis and present a framework to integrate supervised and unsupervised GE methods. On top of that, we also propose three graph learning mechanisms, namely global, local, and adaptive learning, which operate around the GE framework which was not studied in the previous paper.

3 Preliminaries

Brief reviews of the Grassmann manifold and sparse representation are provided in this section. The theory behind the Grassmann kernel would be helpful to understand how points on the manifold could be measured. Some background knowledge of sparse representation would be beneficial in understanding how adaptive learning is accomplished in this work.

3.1 Grassmann manifold

The geometric property of the Grassmann manifold has received significant attention, and a good introduction for this topic can be found in [7]. For image set matching problem, an image set comprising of m images, with each image having D pixels, can be represented as a point on G(m, D). Two points on the Grassmann manifold, which correspond to two image sets, are equivalent if one can be mapped to the other by an m × m orthogonal matrix [7].

The distance between two subspaces can be measured by canonical distance, which is the length of geodesic path connecting two points on the Grassmann manifold. However, it is more computationally efficient to compute the distances between the subspaces using the principal angles [51]. Given two subspaces,P₁ and P₂, or referred to as points on the Grassmann manifold, principal angles are related to the geodesic distance by

$$D_{\mathrm{Geo}}^2\left(P_1,P_2\right)={\displaystyle \sum _i}{\theta }_i^2$$

(1)

where θ = [θ₁, …, θ_m]^′ denotes the distance between span (P _i ) and span (P _j ). Principal angles can be conveniently computed using singular value decomposition as

$$P_1^{\prime }{P}_2=\mathit{US}{V}^{\prime }$$

(2)

where U = [u₁…u_m], u _k ∈ span (P₁), V = [v₁…v _m ], v _k ∈ span (P₂), and S is the diagonal matrix S = diag (cosθ₁…cosθ_m).

Various distances have been defined based on the principal angles, and some well-known distances are the Binet-Cauchy, projection, and Procrustes distances. Among the various distances, the projection distance, Binet-Cauchy distance, and canonical-correlation distance (the largest principal angle) are induced from positive definite kernels. This means that we can define the corresponding kernels on the Grassmann manifold based on these matrices.

In this paper, the projection kernel and canonical-correlation kernel are adopted as they are reported to provide good result [12, 14]. Given two points on a Grassmann manifold, X _i and X _j ∈ ℝ^{D xm}, the similarity between the points is defined as

$$k\_{\text{proj}}_{i,j}={∥X_i^{\prime },X_j∥}_F^2$$

(3)

$$k\_{\mathrm{cc}}_{i,j}=\underset{a_p\in \text{span}\left(X_i\right)}{\text{max}}\underset{b_q\in \text{span}\left(X_j\right)}{\text{max}}{a}_p^T{b}_q$$

(4)

subject to $a_p^T{a}_p=b_p^T{b}_p=1$ and $a_p^T{a}_q=b_p^T{b}_q=0$, p ≠ q; k_ proj denotes the projection kernel while k_ cc signifies the canonical-correlation kernel.

3.2 Sparse representation

In the past few years, sparse representation (SR) has proven to be a powerful tool for computer vision, computational biology, statistics, pattern recognition, and other applications [32, 52, 53]. Given a signal, or the column vector of an image in our case, x _i ∈ ℝ^k and an overcomplete dictionary [54] with k bases, X = [x₁, x₂, …, x _n ] ∈ ℝ^{n × k} (k > n), the goal of SR is to represent x _i using as few entries of X as possible. The objective function can be defined as follows:

$$\text{min}∥S_i∥0s.t.x_i=X{S}_i$$

(5)

where S _i denotes the sparse coefficient matrix and ∥∙∥₀ denotes the l₀ norm of a vector.

However, it is NP-hard to find the sparsest solution for Equation 2 using l₀-minimization. As such, l₁-minimization is often used to solve the problem [54]. In practical applications, there might be noises in signal x _i . Therefore, the following optimization model is used to estimate S _i :

$$\text{min}∥S_i∥1s.t.∥x_i-X{S}_i∥2<ϵ$$

(6)

where ∥∙∥₁ is l₁-norm and ϵ is the error-tolerant term.

4 Proposed approach

The detail of the proposed approach is given in this section. The proposed method mainly consists of three stages: GEI construction, Grassmann projection, and GGE. Two types of GGE configurations are introduced: supervised and unsupervised. Three different graph learning mechanisms are further presented for each of the GGE learning modes. The general framework for the proposed approach is depicted in Figure 1.

Figure 1

General framework for the GGE method.

4.1 Gait energy image

The simple yet effective GEI [55] approach is deployed in this paper. Given a gait sequence ${\left\{I_t,\left(i\text{,},,j\right)\right\}}_{t=1}^F$, where I _t (i, j) is a pixel at position (i, j) in the image I _t , and F is the total number of frames in the gait sequence, GEI is defined as

$$\mathrm{GEI}\left(i,j\right)={\displaystyle \sum _{t=1}^F}{I}_t\left(i,j\right)/F.$$

(7)

One advantage of representing the gait feature using GEI is that we do not need to consider the underlying dynamics of the walking motion. This representation enables us to study the gait sequence from a holistic view by implicitly characterizing the structural statistics of the spatiotemporal patterns of the walking person. The original silhouette images and the resulting GEI images of three subjects are illustrated in Figure 2. We observe that the subjects can be favorably distinguished from the GEI images.

Figure 2

Samples of original silhouette images and the resulting GEI images.

4.2 Grassmann projection

The set of GEI images taken from the video sequence are modeled as a collection of linear subspaces. In this way, the undesired variability due to view angle, pose, and appearance changes can be absorbed within subspaces, and the variability of subject identity can be emphasized as variability among the subspaces. Most subspace-based learning techniques [4–6] employ an inconsistent mechanism, e.g., feature extraction is performed in the Euclidean space while non-Euclidean subspace distances are used. Optimization and convergence will be difficult to achieve using this inconsistent approach [12]. Under the Grassmann framework, the feature extraction and distance measurement can be integrated in a graceful manner, resulting in a simpler and more familiar algorithm.

Given sets of GEIs calculated using Equation 7, we compute SVD over the image sets to obtain the corresponding subspaces{X₁, X₂, …, X _n } where X _i ∈ ℝ^{D xm} and D refers to the length of the gait feature while m signifies the number of images comprising the subspaces. After that, the Grassmann kernel is applied on these subspaces. To this end, we have tested two types of kernel functions, namely the projection and canonical kernels [12, 14] given in Equations 3 and 4.

4.3 Grassmann graph embedding

Grassmann kernels allow us to embed the manifold in a higher-dimensional RKHS to which many Euclidean algorithms can be generalized. Conventional dimension reduction techniques like linear discriminant analysis (LDA), principal component analysis (PCA), and locality preserving projection (LPP) can thus be applied on the Grassmann manifold to further improve recognition accuracy [12–14]. The GE framework [15] has proven to be effective in unifying the various dimension reduction algorithms. Given points from the underlying Grassmann manifold ℳ, the local geometrical structure of ℳ can be modeled by constructing a similarity graph W. Let G = {V, W} denotes an undirected weighted graph with vertices V and similarity matrix W. The values for W can be directly obtained from the output of the Grassmann kernel. On the other hand, the diagonal matrix D and the Laplacian matrix L of the graph G are defined as L = D – W where $D_{\mathit{ii}}={\displaystyle \sum _{j\ne i}}{W}_{\mathit{ij}}.$

The task of GE is to determine a low-dimensional representation of the vertex set V that preserves similarities between vertex pairs in the original high-dimensional space. The solution can be directly obtained using eigenvalue decomposition [15]. In the following text, we formulate the GE dimension reduction problem over the Grassmann manifolds for unsupervised and supervised configurations.

The unsupervised GGE approach is suitable for open surveillance systems like applications to monitor pedestrians at the streets and customers at the shopping malls. It is very difficult, if not impossible, to obtain the subject's identity in such settings; thus, unsupervised GGE will be useful in discerning an individual with unknown identity. On the contrary, supervised GGE is appropriate for closed-set identification like monitoring employees in a workplace. As the identity of the legitimate subject is known, supervised GGE would be able to classify the gait data reliably using identity information.

4.3.1 Unsupervised GGE

We formulate the unsupervised GGE method by first forming the similarity graph W. We want to find a mapping function F:Y _i  → Z _i to map the points on the Grassmann manifold, ℳ, to a new manifold, ℳ’, to preserve the local geometry of the manifold. In other words, we want to find a transformation which maps the connected points on W as close as possible. The following objective function realizes this criterion:

$$\text{min}{\displaystyle \sum _{\mathit{ij}}}{\left(Z_i-Z_j\right)}^2{W}_{\mathit{ij}}.$$

(8)

The objective function W _ij incurs a heavy penalty if the connected neighbors are mapped far apart in ℳ’. Therefore, minimizing W _ij ensures that Z _i and Z _j are close if Y _i and Y _j are close.

Suppose$⏟\mathbb{U}$ is a projection matrix, Z^T =$⏟\mathbb{U}$^TY, that fulfills the objective function (8) and Y is the kernel matrix produced by the Grassmann kernel. By simple algebra manipulation, the objective function can be reduced to

$$\begin{array}{ll}1/2{\displaystyle \sum _{\mathit{ij}}}{\left(Z_i-Z_j\right)}^2{W}_{\mathit{ij}}& =1/2{\displaystyle \sum _{\mathit{ij}}}{\left(U_i^T{Y}_i-U_j^T{Y}_j\right)}^2{W}_{\mathit{ij}}\\ ={\mathbb{U}}^T\left(D-W_{\mathit{ij}}\right){\mathbb{Y}}^T\mathbb{U}\\ ={\mathbb{U}}^T\mathbb{Y}L{\mathbb{Y}}^T\mathbb{U}\end{array}$$

(9)

where D is a diagonal matrix given by $D_{\mathit{ij}}={\displaystyle \sum _j}{W}_{\mathit{ij}}$. The optimization problem can be reduced to finding

$$\text{arg}\underset{\begin{array}{c}\hfill \mathbb{U}\hfill \\ \hfill {\mathbb{U}}^T\mathbb{Y}L{\mathbb{Y}}^T\mathbb{U}=1\hfill \end{array}}{\text{min}}\left\{{\mathbb{U}}^T\mathbb{Y}L{\mathbb{Y}}^T\mathbb{U}\right\}.$$

(10)

The projection matrix $⏟\mathbb{U}$ that minimizes Equation 8 is given by the maximum eigenvalue solution to the generalized eigenvalue problem:

$$\mathbb{Y}L{\mathbb{Y}}^T\mathbb{U}=\mathbf{\lambda }\mathbb{Y}D{\mathbb{Y}}^T\mathbb{U}.$$

(11)

4.3.2 Supervised GGE

The unsupervised GGE method can be extended to the supervised version by constructing two similarity graphs, W _w,ij and W _b,ij , which denote the within-class and between-class similarity matrices, respectively. The extension is desirable as we can take advantage of the class label information to improve the classification accuracy. The mapping function for supervised GGE is slightly different from its unsupervised counterpart. The new mapping function $F^{\prime }\text{:}{Y}_i\to Z_t^{\prime }$ is formed such that the connected points of the within-class similarity matrix, W _w,ij , stay as close as possible while connected points of the between-class similarity matrix, W _b,ij , stay as distant as possible. The class label information is used in this method to discover the discriminant structure of the samples. The objective functions for supervised GGE are defined as follows:

$$\text{min}{\displaystyle \sum _{\mathit{ij}}}{\left(Z_i^{\prime }-Z_j^{\prime }\right)}^2{W}_{w,\mathit{ij}}$$

(12)

$$\text{max}{\displaystyle \sum _{\mathit{ij}}}{\left(Z_i^{\prime }-Z_j^{\prime }\right)}^2{W}_{b,\mathit{ij}}.$$

(13)

The objective function W _w,ij incurs a heavy penalty if neighboring points $Z_i^{\prime }$ and $Z_j^{\prime }$ are mapped far apart while they are actually in the same class. Likewise, the objective function W _b,ij incurs a heavy penalty if neighboring points $Z_i^{\prime }$ and $Z_j^{\prime }$ are mapped close together while they belong to different classes.

Suppose$⏟\mathbb{U}$ is a projection matrix, Z^′T =$⏟\mathbb{U}$^TY, to realize the objective functions (12) and (13). By simple algebra manipulation, the objective function (12) can be reduced to

$$\begin{array}{ll}1/2{\displaystyle \sum _{\mathit{ij}}}{\left(Z_i^{\prime }-Z_j^{\prime }\right)}^2{W}_{w,\mathit{ij}}& =1/2{\displaystyle \sum _{\mathit{ij}}}{\left(V_i^T{Y}_i-V_j^T{Y}_j\right)}^2{W}_{w,\mathit{ij}}\\ ={\displaystyle \sum _i}{V}_i^T{Y}_i{D}_{w,\mathit{ii}}{Y}_i^T{V}_i-{\displaystyle \sum _{i,j}}{V}_j^T{Y}_j{D}_{w,\mathit{ij}}{Y}_j^T{V}_j\\ ={\mathbb{U}}^T\mathbb{Y}{D}_w{\mathbb{Y}}^T{\mathbb{U}}^T-{\mathbb{U}}^T\mathbb{Y}{W}_w{\mathbb{Y}}^T{\mathbb{U}}^T\end{array}$$

(14)

where D _w is a diagonal matrix given by $D_{w,\mathit{ij}}={\displaystyle \sum _j}{W}_{w,\mathit{ij}}$. Similarly, the objective function (13) can be condensed to the following form:

$$\begin{array}{ll}1/2{\displaystyle \sum _{\mathit{ij}}}{\left(Z_i^{\prime }-Z_j^{\prime }\right)}^2{W}_{b,\mathit{ij}}& =1/2{\displaystyle \sum _{\mathit{ij}}}{\left(V_i^T{Y}_i-V_j^T{Y}_j\right)}^2{W}_{b,\mathit{ij}}\\ ={\mathbb{U}}^T\left(D_b-W_b\right){\mathbb{Y}}^T\mathbb{U}={\mathbb{U}}^T\mathbb{Y}{L}_b{\mathbb{Y}}^T\mathbb{U}\end{array}$$

(15)

where D _b is a diagonal matrix obtained through $D_{b,\mathit{ii}}={\displaystyle \sum _j}{W}_{b,\mathit{ij}}$. The optimization problem can be condensed into the following form:

$$\text{arg}\underset{\begin{array}{c}\hfill \mathbb{U}\hfill \\ \hfill {\mathbb{U}}^T\mathbb{Y}{D}_w{\mathbb{Y}}^T\mathbb{U}=1\hfill \end{array}}{\text{max}}\left\{{\mathbb{U}}^T\mathbb{Y}{L}_b{W}_w{\mathbb{Y}}^T\mathbb{U}\right\}.$$

(16)

The projection matrix that minimizes Equation 16 can be obtained by solving the generalized eigenvalue problem:

$$\mathbb{Y}{L}_b{W}_w\mathbb{U}=\mathbf{\lambda }\mathbb{Y}{D}_w{\mathbb{Y}}^T\mathbb{U}.$$

(17)

The procedure to implement GGE for supervised and unsupervised configurations is summarized in Algorithm 1.

4.3.3 Constructing the similarity graphs

Graph relations play a crucial role in the GE framework to determine how the methods behave based on the connectivity and weight assignment of the neighboring points in the data. We present three approaches for graph construction: global, local, and adaptive. The first approach constructs fully connected graphs where all nodes are connected using predefined weights. The representative methods for this approach are PCA and LDA for unsupervised and supervised configurations, respectively.

The second approach takes into consideration the neighborhood information where only the k neighboring nodes are connected in the graph. If k = N, the local approach is the same as the global approach. Some popular methods for this approach are LPP and locality preserving discriminant analysis [56] for unsupervised and supervised modes, respectively.

The third approach adaptively assigns weights to the nodes based on how the rest of the samples contribute to the sparse representation of the nodes. This is an unconventional approach for graph construction, and the detail of constructing the adaptive graph is given in the subsequent section.

Weight assignment for the similarity graphs for the global approach is straightforward where all nodes in the graph are connected with equal weights. For the unsupervised mode, the simplest graph structure is to set W _ij  = 1. Another way to form the similarity graph is using the heat kernel equation $W_{\mathit{ij}}=\left\{-∥{\widehat{x}}_i-{{\widehat{x}}_j∥}^2/t\right\}$[15] where t is an adjustable constant. In contrast to the unsupervised mode, two graphs are constructed in the supervised mode. Weights are assigned to the within-class similarity graph, W _w,ij , if two nodes share the same class label; 0 otherwise. Similarly, weights are assigned to the between-class similarity graph, W _b,ij , if two nodes are not from the same class; 0 otherwise.

For the local approach, the simplest graph structure is the simple-minded graph where the similarity matrix W _ij is set to 1 if ${\widehat{x}}_i$ is among the k th nearest neighbors of ${\widehat{x}}_j$; 0 otherwise. The weight can also be replaced by the heat kernel equation. On the other hand, the supervised method takes into consideration the class information and sets the within-class similarity graph W _w,ij  = 1 if ${\widehat{x}}_i$ is among the k th nearest neighbors of ${\widehat{x}}_j$ in the same class;0 otherwise. In a similar manner, the between-class similarity graph assigns W _b,ij  = 1 if ${\widehat{x}}_i$ is among the k th nearest neighbors of ${\widehat{x}}_j$ in different classes; 0 otherwise.

We also propose a self-adaptive graph structure. Suppose S(i, j) is the sparse output estimated by Equation 6 using the column vector of $\widehat{\mathbf{X}}$ (output of the Grassmann kernel), the similarity graph for unsupervised self-adaptive graph is defined as W _ij  = S(i, j). On the other hand, the within-class similarity graph for the supervised method is defined as W_w,ij = S _w (i, j). S _w is the output from Equation 6 fulfilling the conditions ${\widehat{x}}_i\in N_w\left({\widehat{x}}_j\right)$ or ${\widehat{x}}_i\in N_w\left({\widehat{x}}_j\right)$ where $N_w\left({\widehat{x}}_j\right)$ is the set of k neighbors sharing the same label with ${\widehat{x}}_i$. The between-class similarity graph is characterized by W_b,ij = S _b (i, j). S _b is the output from Equation 6 and ${\widehat{x}}_i\in N_b\left({\widehat{x}}_j\right)$ or ${\widehat{x}}_j\in N_b\left({\widehat{x}}_i\right)$ where $N_b\left({\widehat{x}}_i\right)$ is the set of k neighbors having different labels. This is the basic approach to construct an adaptive graph where a single dictionary is learnt for all classes. Since the dictionary is learnt only once, some computational burden can be saved.

A number of variations can be derived from this basic idea. For example, class-specific dictionary can be learnt where each class is modeled independently of the others. W _w can be modeled from the SR output using the column vector of $\widehat{{\mathbf{X}}_w}$, where $\widehat{{\mathbf{X}}_w}$ is the Grassmann output sharing the same labels with the test sample. W _b can also be constructed using the SR output using the column vector of $\widehat{{\mathbf{X}}_b}$, where $\widehat{{\mathbf{X}}_b}$ is the Grassmann output having different labels with the test sample. This approach enables the learnt dictionary to have an efficient representation for each class. However, dictionary learning has to be performed multiple times for different classes.

If one wishes to uncover only the semantic information in the between-class similarity graph (due to the fact that perhaps not much interesting information can be revealed in the sparse within-class similarity graph as large values are expected for nodes coming from the same class), the between-class similarity graph could be generated using the sparse approach while the within-class similarity graph be constructed using the simple-minded or heat kernel functions. The combination of fully connected and sparse graphs benefits from the flexibility of sparse graph and low computational cost of the fully connected graph. Table 1 summarizes the different graph construction methods for GGE.

Table 1 Summary of the different graph construction methods

5 Experiments

Two databases were used to evaluate the proposed method namely, the Chinese Academy of Sciences, Institute of Automation (CASIA) gait database: dataset B [57] and the Osaka University, Institute of Scientific and Industrial Research (OU-ISIR) gait database: datasets A and B [58]. The CASIA gait database is good for assessing the view variation effect on gait as it contains a large number of subjects taken from different viewing angles. The CASIA gait database consists of 124 subjects captured from 11 different angles. The viewing angles range from 0° to 180°, separated by an interval of 18°. There are ten walking sequences for each subject, with six samples containing subjects walking under normal condition, two samples with subjects walking with coats, and two samples with subjects carrying bags. Therefore, there are altogether 13,640 (10 × 11 × 124) gait sequences in the database. All the images were cropped and normalized to 120 × 120 pixels.

The OU-ISIR gait database is suitable for assessing the influence of speed changes and clothing variations on gait. The OU-ISIR gait database: dataset A contains 35 subjects captured from side view with speed variation from 2 to 7 km/h, at an interval of 1 km/h. There are two walking sequences for each speed level. Thus, there are 420 (2 × 6 × 35) gait sequences in this dataset. On the other hand, dataset B is made up of 68 subjects acquired from side view with clothing variations. There are many clothing combinations in this dataset which include pants, half shirt, rain coat, skirt, and cap. All the images for the OU-ISIR database were cropped and resized to 128 × 88 pixels.

5.1 Experiment result

5.1.1 Evaluation on view variations

The CASIA gait database was used to testify the performance of the proposed method under view changes. All the six gait sequences under the normal walking condition were used. For clear indication, each of the viewing angles {0°, 18°, …, 180°} were labeled as θ = {1, 2, …, 11}. We formulated three cases to evaluate the proposed method against viewpoint changes. We simulated realistic scenarios where the multiple views could have been acquired from fairly different viewpoints:

• 1. Same view setting, θ _test  = θ _train . In this setting, all the viewpoints used in the training and testing sets were the same, e.g., θ _train = {1, …, 11} and θ _test = {1, …, 11}.

• 2. Mixed view setting, θ _test  = θ; θ _train  = θ. In this setting, we made it challenging in which not all the poses in the testing sets were available for training, e.g., θ _train = {2, 3, 4, 6, 8} and θ _test = {2, 4, 6, 7, 9}.

• 3. Different view setting, θ _test  = θ - θ _train . This is a difficult case where the testing set contains images which were totally different from those in the training set, e.g., θ _train = {2, 4, 6, 9} and θ _test = {1, 3, 5, 8}. We further included more challenging scenarios to test how the proposed method was able to generalize unseen viewpoints, e.g., θ _train = {1, 2, 3, 4} and θ _test = {7, 8, 9, 10}. This is an interesting experiment to see how well the proposed method performs in extrapolating view angles beyond the known view angles. The previous setting where the estimation of view angles is within the range of known view angles (e.g., θ _train = {2, 4, 6, 9} and θ _test = {1, 3, 5, 8}) can be seen as an interpolation case.

The following setup was deployed to run the experiment. We randomly selected four gait sequences from each subject to form the training set, and the remaining sequences were for the testing set. The selected view angles for the different settings were modeled as the subspace for each sample in the training and testing sets. We then computed the similarity score for every pair of training–testing matches. The random division of the gait sequences into training and testing sets was repeated several times, and the average result was recorded. The k-nearest neighbor method was used to measure the similarity score between the training and testing sets.

For SR dictionary learning, we deployed the l₁-regularized least square problem solver distributed by Boyd's research group [59]. The algorithms are sensitive towards several parameters listed in Table 2. The values or range of values that generally yield good performance based on empirical test are also given in Table 2. The results reported in this paper were obtained based on the best possible combination of the parameters. The rank-1 recognition rate was used as the performance indicator. The correct match was counted when the sample in the testing set was the best match (top one) from the training set.

Table 2 List of parameters used in GGE

The experimental results for evaluating the changes in view angles are shown in Table 3. The canonical-correlation kernel is denoted as CC, kernel while projection kernel is termed as ‘Proj’. The prefixes ‘SM’, ‘HK’, and ‘SR’ are the abbreviations for simple-minded (the binary graph), heat kernel function, and sparse representation. We included comparison with the multi-view subspace representation (MSR) method. On top of that, we also added the classical score-level fusion method to benchmark the algorithm. The scores from the different view angles were fused together using the minimum dissimilarity selection rule [60].

Table 3 Evaluating the effect of view angle changes for the same, mixed, and different view settings

When all of the viewing angles are used to train the system, 100% accuracy could be achieved for all the methods except for score-level fusion and MSR. It is not surprising to get such good result because GGE captures the variations in viewpoint changes when recognition is performed. In the mixed view settings, promising results close to 100% accuracy is obtained. This is encouraging as the proposed methods are shown to possess cross-view capability. The performance is still favorable in the different view settings when the viewpoints in the testing set are close to that in the training set. However, the accuracy drops when the viewpoints in the testing sets are far apart from the training set (e.g., in the case of TR 1, 2, 3, 4; TT 7, 8, 9, 10). We accept such poor result because we understand that it is a challenging problem to extrapolate unseen views which are very different from the existing views. Based on the results shown in Table 3, we notice that the proposed method using CC kernel consistently yields good results. The effectiveness of SR has been verified in the experiments.

5.1.2 Variants of SR analysis

In this experiment, we evaluate the different variants derived from the SR similarity graph construction approach described in Section 4.3.3. Table 4 displays the performance of the different graph combination methods. Adaptive supervised GGE with CC kernel was applied in this experiment. SR variation 1 refers to the basic adaptive graph construction approach which learns a shared dictionary for all classes. This method constructs the within-class and between-class similarity graphs using W_w,ij = S _w (i, j) and W_b,ij = S _b (i, j). The l₁-minimization algorithm given in Equation 6 is run once per test sample, and the output is split into W _w,ij and W _b,ij based on the class labels.

Table 4 Rank-1 recognition rate (%) of the different combinations of SR similarity graphs

On the contrary, SR variation 2 learns class-specific dictionary. It builds the within-class similarity graph using $W_{w,\mathit{ij}}={\widehat{S}}_w\left(i,j\right)$ where ${\widehat{S}}_w\left(i,j\right)$ is the result of running l₁-minimization on the Grassmann output sharing the same class labels with the test sample. The between-class similarity graph is constructed using $W_{b,\mathit{ij}}={\widehat{S}}_b\left(i,j\right)$ where ${\widehat{S}}_b\left(i,j\right)$ is the result of running l₁-minimization on the Grassmann output having different class labels from the test sample. In this respect, the l₁-minimization algorithm was run twice per test sample: the first time using the training data from the same class to construct W _w,ij and the second time using the training data from the other classes to construct W _b,ij .

As for the SM + SR method, the simple-minded function was used to generate the within-class similarity graph, W _w,ij , while SR was used to build the between-class similarity graph, W _b,ij . Likewise, the heat kernel function and SR were used to generate W _w,ij and W _b,ij , respectively, for the HK + SR method.

We observe that the performances of the different SR variations do not deviate significantly. We use SR variation 1 in the subsequent sections as it gives slightly better results in most situations. Besides, it is less time-consuming as compared to SR variation 2 and does not require additional parameters like the number of neighbors k and the constant term t as compared to SM + SR and HK + SR.

5.1.3 Evaluation on clothing and carrying conditions

We conducted experiments to examine the performance of the proposed methods under clothing variations. The main purpose of this experiment is to simulate the condition where suspects captured by the surveillance cameras are trying to masquerade themselves by wearing covers like rain coat or hat. This experiment is also useful to identify the ability of the proposed method to discriminate individuals who wear loose outfits like baggy pants and skirt (for ladies) which can obstruct the gait pattern from being observed properly.

The CASIA and OU-ISIR gait databases were used for this evaluation. For the CASIA database, we took four normal gait sequences as the training set and two bags-carrying and two coats-wearing sequences as the testing sets. All the 11 viewing angles were applied in the test. Using two types of data for training and testing (one from the normal walking sequence and the other from the carrying/clothing conditions) is a more realistic setting where we need to generalize the unknown carrying/clothing type from the existing dataset. In real-life scenario, there is no way for us to predict the types of clothes the person wears or the things the person carry when he/she walks. As for the OU-ISIR database, six different clothing combinations were tested. Most of the clothing combinations were from types A (e.g., regular pants and parka) to M (e.g., baggy pants and down jacket) [58]. The clothing types were chosen such that we could get the largest possible variations for the test. Only 16 subjects were tested in this experiment. This is because we could only identify 16 corresponding pairs between dataset A, the normal walking sequence, and dataset B, walking with clothing variations. Six sequences from dataset A were used as the training set, while the six sequences in dataset B were used as the testing set.

The results of the tests are shown in Table 5. We find that the variations in clothing alter an individual's appearance and make the problem of gait identification challenging. For example, the images depicted in Figure 3 are taken from the same subject. The images look different when different types of clothing are worn. The experiment suggests that further investigation has to be carried out to study gait recognition with substantial clothing variations. Nevertheless, the methods could handle the carrying condition satisfactorily.

Table 5 Evaluating the effect of clothing and carrying conditions

Figure 3

Images of the same person with different clothing types.

5.1.4 Evaluation on walking speeds

We have also conducted experiments to assess the effect of walking speed on gait. We are interested in this study as the perpetrator usually walks faster in order to leave the crime scenes immediately. The OU-ISIR gait database was used for this evaluation. Using similar treatment as the view angle evaluation, we labeled the speed {2, 3, …, 7 km/h} as S = {1, 2, …, 6}. Table 6 shows the result of evaluating speed variations. Some methods could achieve 100% accuracy when all the speeds are used. Unlike clothing variations, speed changes do not drastically affect the accuracy of gait identification. Therefore, the methods could tolerate speed variations quite robustly.

Table 6 Evaluating the effect of speed variations using rank-1 recognition rate (%)

5.2 Summary and discussion

The important findings of this work are summarized below:

•  The Grassmann manifold provides a platform to reduce the subspace-to-subspace matching problem to a point-to-point matching model. This is immensely useful for gait recognition as the gait video sequences naturally fall in the subspace learning paradigm (unlike face recognition which can be carried out using single image).

•  The unsupervised and supervised GGE configurations provide different treatments to gait dataset of different natures (labeled and unlabeled). Nevertheless, the two approaches can be unified gracefully under a general formulation.

•  GGE outperforms the benchmark methods for all cases. The proposed adaptive learning approach, in particular, yields considerable improvement in classification accuracy.

•  As a comparison among the different graph construction approaches, the adaptive graph construction method obviously outperforms its counterparts. However, no conclusive remark can be drawn between the global and local methods. The global approach performs better than the local approach under view angle changes, but the opposite happens for the clothing and carrying conditions, while almost similar results were obtained for speed variation. As such, we conjecture that the topological structure of the graph has disparate impact on different scenarios. No single graph structure (referring to the global and local graphs) works best for all cases.

•  Unsupervised GGE surprisingly outperforms its supervised counterpart in a number of scenarios, e.g., gait image sets with varying view angles and different clothing appearances. The reason why the unsupervised method performs better than the supervised scheme may be because the similarity graph W encodes general information about the relationship among the nodes, whereas the within- and between-similarity graphs, W _w and W _b , may overlook some subtle discriminative connection in the graphs. There may also be some outliers in the labeled training set, for example, an image of a person wearing thick sweater with hood that confuses the true appearance of the person, which explains why the supervised method is slightly inferior to the unsupervised method. This counterintuitive result suggests that it might be better to resort to unsupervised method when the cost of labeling the data is high where the class information would not lead to a dramatic improvement in recognition rate.

•  The canonical-correlation kernel generally performs better than the projection kernel in the changing view scenario. We attribute this to the nature of the canonical-correlation kernel which is based on the notion of principal angle. The canonical-correlation kernel will always find the closest view angle in the training set for comparison. This concept is illustrated in Figure 4. Nevertheless, the projection kernel performs better than the canonical-correlation kernel in the clothing and carrying conditions. This may be due to the fact that projection kernel treats the image subspaces from a more holistic aspect. However, if the sample size is small, e.g., for the OU dataset, projection kernel does not have any advantage over the canonical-correlation kernel. The result suggests that the kernels describe different aspects of the subspaces.

Figure 4

Subspace-based method will always find the closest view angle between training and testing subspaces for matching.

6 Conclusions

This paper demonstrates how it is possible to formulate the gait recognition problem on the Grassmann manifold. This formulation enables us to work in higher-order data structure to harness the nonlinear structure of the data and yet benefit from conventional vector-based computation. We present a method comprising unsupervised and supervised learning modes on the Grassmann manifold. We further introduce the concept of adaptive graph in the learning mechanism to adaptively tailor the graph content based on the nature of the dataset. Experimental results suggest that the proposed method has a potential for practical application as it demonstrates view- and speed-invariant capabilities.