A Human Gait Classification Method Based on Radar Doppler Spectrograms
© Fok Hing Chi Tivive et al. 2010
Received: 1 February 2010
Accepted: 24 June 2010
Published: 12 July 2010
An image classification technique, which has recently been introduced for visual pattern recognition, is successfully applied for human gait classification based on radar Doppler signatures depicted in the time-frequency domain. The proposed method has three processing stages. The first two stages are designed to extract Doppler features that can effectively characterize human motion based on the nature of arm swings, and the third stage performs classification. Three types of arm motion are considered: free-arm swings, one-arm confined swings, and no-arm swings. The last two arm motions can be indicative of a human carrying objects or a person in stressed situations. The paper discusses the different steps of the proposed method for extracting distinctive Doppler features and demonstrates their contributions to the final and desirable classification rates.
In the past few years, human gait analysis has received significant interest due to its numerous applications, such as border surveillance, video understanding, biometric identification, and rehabilitation engineering. Besides the advances in vision-based gait recognition technology, there is a large amount of research concerned with the development of automatic radar gait recognition systems. Radars have certain advantages over optical-based systems in that it can operate in all types of weather, is insensitive to lighting conditions and the size of the object, and can penetrate clothes. The general concept of radar-based systems is to transmit an electromagnetic wave at a certain range of frequencies and analyze the radar return signal to estimate the velocity of a moving object by measuring the frequency shift of the wave radiated or scattered by the object, known as the Doppler effect. For an articulated object such as a walking person, the motion of various components of the body including arms and legs induces frequency modulation on the returned signal and generates sidebands about the Doppler frequency, referred to as micro-Doppler signatures. These micro-Doppler signatures have been studied in a number of publications [1–4] using joint time-frequency representations.
Signals characterized with multiple components having different frequency laws leave distinct features when examined in the time-frequency domain . Therefore, to extract useful information, a type of joint time-frequency analysis is usually performed on the radar data to convert a one-dimensional nonstationary temporal signal into a two-dimensional joint-variable distribution [6–9]. When presenting the signal power distribution over time and frequency, the time-frequency signal representation can be cast as a typical image in which the two spatial axes are replaced by the time and frequency variables. This similarity invites the application of image-based classification techniques to non-stationary signal analysis.
In this paper, we apply an image processing method for classification of people based on the Doppler signatures they produce when walking. In this respect, we consider received radar data of human walking motion and represent the corresponding signal in the time-frequency domain using spectrograms. Herein, three types of human walking motion are considered: ( ) free-arm motion (FAM) characterized by swinging of both arms, ( ) partial-arm motion (PAM), which corresponds to a motion of only one arm, and ( ) no-arm motion (NAM), which corresponds to no motion of both arms. The NAM is referred to as a stroller or sauntere . The last two classes are commonly associated with a person walking with his/her hand(s) in the trouser pockets or a person carrying light small or heavy large objects, respectively. All three categories are considered important for police and law enforcement, especially when humans are behind opaque material, that is, inside buildings and in enclosed structures, or they are monitored while moving in city canyons and street corners.
Existing human gait classification methods for radar systems can be categorized as parametric and nonparametric approaches. In parametric approaches, explicit parameters are extracted from the respective time-frequency distributions and used as features for classification . Some important features could be the periods characterizing the repetitive arm and leg motions, the Doppler frequency of the torso, which is indicative of walking or running motion, the radar cross-section (RCS), the relative times of positive and negative Doppler describing the forward and backward swings, among others. In nonparametric approaches, portions or segments of the time-frequency distributions, or their subspace representations, are employed as features, followed by a classifier [11, 12].
The proposed method for the above gait classification problem is nonparametric in nature. It is based upon a hierarchical image classification architecture, which has recently been developed for visual pattern classification . Instead of processing optical images, the time-frequency representation of Doppler is used as input to the image classification architecture, which comprises a set of nonlinear directional and adaptive two-dimensional filters, followed by a classifier. We show that each stage of the proposed architecture captures salient features from the Doppler spectrograms which are useful for classification of human motions.
The remainder of the paper is organized as follows. Section 2 describes the application of Short-Time Fourier Transform (STFT) technique to capture the micro-Doppler signatures of the three types of arm motion, FAM, PAM, and NAM. Section 3 presents the proposed classification method which consists of a cascade of directional filters and adaptive filters. Section 4 presents experimental results demonstrating that the proposed image classification technique can be successfully applied to time-frequency signal representations. Finally, concluding remarks are given in Section 5.
2. Human Motion Signatures inTime Frequency
The proposed classification technique is applied to real data collected in the Radar Imaging Lab, Center for Advanced Communications, Villanova University, USA. The radar is a continuous wave (CW) operating at 2.4 GHz and with direct line of sight to the target. The data for five persons (labelled as A, B, C, D, and E) were collected and sampled at 1 kHz with a transmit power level of 5 dBm. The motion of each subject was recorded for 20 seconds, with the person moving forwards (towards the radar) and backwards. When a person is walking, various components of the body, such as the torso, legs, and arms have different velocities, and the signal reflected from these components will have a Doppler shift. To capture the Doppler frequency at various instances of time, a joint time-frequency analysis method is used.
3. Hierarchical Image ClassificationArchitecture (HICA)
3.1. Stage 1—Oriented Feature Extraction
3.2. Stage 2—Learning Intrinsic Motion Features
3.3. Stage 3—Classifier
3.4. Training Method
where and are the th element of the desired output vector and the actual response , respectively, and is the number of arm motions, that is, . The Levenberg-Marquardt (LM) algorithm  is used to learn the optimum adaptive filter parameters in Stage 2 and the parameters of the classifier in Stage 3. The LM algorithm is a fast and effective training method; it combines the stability of the gradient descent with the speed of Newton algorithm. Given that all parameters of the adaptive filters and the linear classifier are arranged as a column vector, . The main steps of the LM algorithm are given as follows.
Perform forward computation to find the outputs of each stage in response to the training patterns.
where is the Jacobian of the error function , is the identity matrix, and is a regularization term to avoid the singularity problem. During training, the regularization parameter is increased or decreased by a factor of ten, depending on the decrease or increase of the MSE, respectively. The Jacobian matrix can be computed from a modified version of the error-backpropagation algorithm, which is explained in .
Repeat Steps 2 to 3 until the maximum number of training epochs is reached or the error is below a predefined limit.
4. Experimental Methods and Results
Before the spectrogram is computed, the radar trace is downsampled by a factor of two to reduce the amount of data to be processed. Furthermore, the spectrogram is normalized by dividing by its maximum value. Overlapping spectrogram windows of size are used for training and testing the HICA presented in Section 3. The spectrogram windows are centred at the location of the torso, that is, at the maximum magnitude spectrum for each given time interval. There is a tradeoff between the input window size and the HICA classification performance; a too small window does not allow the HICA to learn the salient features of each motion, and a too large window increases the complexity of the HICA, which affects its generalization ability. Therefore, the input window is chosen as the minimum window size that achieves good classification performance. Previous studies on visual pattern recognition problems showed that the HICA achieves good classification performance when using convolution masks of size for each adaptive filter in Stage 2 [28, 29]. Thus, the size of the convolution masks and is set to in all experiments, and the exponential and hyperbolic tangent activation functions are chosen for and , respectively. For Stage 1 the directional filters are designed with kernel size of and .
The optimum configuration of the HICA depends on a number of factors, including the number of directional filters used in Stage 1, the time/frequency resolution of the spectrogram window, and the classifier type for Stage 3. Several experiments were conducted to determine the effects of these factors on the classification performance. The classification rate is used as a measure of performance, which is computed as a ratio of the number of correctly classified windows over the total number of test windows. The optimum parameters are chosen when the maximum classification rate is achieved on a validation set. The effects of the various parameters are investigated using the incidence angle motion data only. The experimental results are presented in the following three subsections.
4.1. Performance of Various HICA Configurations
4.2. Effect of Time/Frequency Resolution
4.3. Performance of the Feature Extraction Stages
Classification accuracy of a linear classifier using as input the features extracted at different stages.
Features extracted from spectrogram
Features extracted from Stage 1
Features extracted from Stage 2
4.4. Comparison with Other Classifiers
Classification performances of different classifiers using the spectrogram as input.
MLP with one hidden layer
4.5. Classification of Short-Time Segments
Several existing methods use the entire frame to classify the motion of a subject. For example, Mobasseri and Amin  used principal component analysis (PCA) on the same data set to extract features from the spectrogram and applied a quadratic classifier based on the mahalanobis distance for classifying the spectrogram of human motion. When extracting feature vector parallel to the frequency axis, they achieved 82.5% for classifying no-arm motion (NAM), 69.1% for classifying PAM and, 70.7% for classifying FAM. However, when the feature vectors are computed parallel to the time axis (Doppler snapshots), the classification performance is increased to 100% for PAM, 98.3% for FAM, and 100% for NAM. This improvement is due to large changes in the Doppler frequency across time.
Average classification rate
A three-stage classification method employing both fixed directional and adaptive filters, in addition to a linear classifier, is introduced for classifying various types of human walking. The filters are applied in the time-frequency domain which depicts the Doppler signal power distribution over time and frequency. Three types of arm motion are considered: free-arm swings, one-arm confined swings, and two-arm confined swings. The proposed method determines the optimum time-frequency window for training and testing, and is able to detect and extract distinct Doppler features from the spectrogram. The data used for testing and training correspond to five subjects moving towards and away from the radar with and aspect angle, and with nonobstructed line of sight. The paper shows the importance of each stage of the classification method in improving the classification rates. The attractiveness of the proposed method lies in its robustness to data misalignments, forward/backward walking motions, including the acceleration-deceleration phases exhibited when turning, and to the specific quadratic distribution used for time-frequency signal representations.
This work is supported in part by a grant from the Australian Research Council (ARC).
- Geisheimer JL, Greneker EF, Marshall WS: High-resolution Doppler model of the human gait. Radar Sensor Technology and Data Visualization, April 2002, Orlando, Fla, USA, Proceedings of SPIE 4744: 8-18.View ArticleGoogle Scholar
- Van Dorp P, Groen FCA: Human walking estimation with radar. IEE Proceedings: Radar, Sonar and Navigation 2003, 150(5):356-366. 10.1049/ip-rsn:20030568Google Scholar
- Chen VC: Analysis of radar micro-Doppler signature with time-frequency transform. Proceedings of IEEE Signal Processing Workshop on Statistical Signal and Array Processing (SSAP '00), 2000, Pocono, Pa, USA 463-466.Google Scholar
- Smith GE, Woodbridge K, Baker CJ: Multistatic micro-Doppler signature of personnel. Proceedings of IEEE Radar Conference (RADAR '08), May 2008Google Scholar
- Cohen L: Time-Frequency Analysis. Prentice Hall, Upper Saddle River, NJ, USA; 1995.Google Scholar
- Amin M, Sarabandi K: Special issue on remote sensing of building interior. IEEE Transactions on Geoscience and Remote Sensing 2009, 47(5):1267-1268.View ArticleGoogle Scholar
- Amin M: Special issue on advances in indoor radar imaging. Journal of the Franklin Institute 2008, 345(6):556-722. 10.1016/j.jfranklin.2008.01.005View ArticleGoogle Scholar
- Borek SE: An overview of through the wall surveillance for homeland security. Proceedings of the 34th Applied Imagery and Pattern Recognition Workshop: Multi-Modal Imaging, October 2005 42-47.View ArticleGoogle Scholar
- Hunt A: Image formation through walls using a distributed radar sensor array. Proceedings of the 32nd Applied Imagery Pattern Recognition Workshop, 2003 232-237.Google Scholar
- Kim Y, Ling H: Human activity classification based on micro-Doppler signatures using a support vector machine. IEEE Transactions on Geoscience and Remote Sensing 2009, 47(5):1328-1337.View ArticleGoogle Scholar
- Mobasseri BG, Amin MG: A time-frequency classifier for human gait recognition. Optics and Photonics in Global Homeland Security V and Biometric Technology for Human Identification VI, April 2009, Orlando, Fla, USA, Proceedings of SPIE 7306:View ArticleGoogle Scholar
- Lyonnet B, Ioana C, Amin M: Human gait classification using micro-Doppler time-frequency signal representations. Proceedings of IEEE International Radar Conference (RADAR '10), May 2010, Washington, DC, USAGoogle Scholar
- Tivive FHC, Bouzerdoum A, Phung SL, Iftekharuddin KM: Adaptive hierarchical architecture for visual recognition. Applied optics 2010, 49(10):B1-B8. 10.1364/AO.49.0000B1View ArticleGoogle Scholar
- Mitchell SJ, Silver RA: Shunting inhibition modulates neuronal gain during synaptic excitation. Neuron 2003, 38(3):433-445. 10.1016/S0896-6273(03)00200-9View ArticleGoogle Scholar
- Prescott SA, De Koninck Y: Gain control of firing rate by shunting inhibition: roles of synaptic noise and dendritic saturation. Proceedings of the National Academy of Sciences of the United States of America 2003, 100(4):2076-2081. 10.1073/pnas.0337591100View ArticleGoogle Scholar
- Arulampalam G, Bouzerdoum A: A generalized feedforward neural network architecture for classification and regression. Neural Networks 2003, 16(5-6):561-568. 10.1016/S0893-6080(03)00116-3View ArticleGoogle Scholar
- Arulampalam G, Bouzerdoumn A: Training shunting inhibitory artificial neural networks as classifiers. Neural Network World 2000, 10(3):333-350.Google Scholar
- Bouzerdoum A: Classification and function approximation using feed-forward shunting inhibitory artificial neural networks. Proceedings of the International Joint Conference on Neural Networks (IJCNN '00), July 2000 613-618.Google Scholar
- Cheung HN, Bouzerdoum A, Newland W: Properties of shunting inhibitory cellular neural networks for colour image enhancements. Proceedings of the 6th International Conference on Neural Information Processing, 1999 3: 1219-1223.Google Scholar
- Hammadou T, Bouzerdoum A: Novel image enhancement technique using shunting inhibitory cellular neural networks. IEEE Transactions on Consumer Electronics 2001, 47(4):934-940. 10.1109/30.982811View ArticleGoogle Scholar
- Bamberger RH, Smith MJT: A filter bank for the directional decomposition of images: theory and design. IEEE Transactions on Signal Processing 1992, 40(4):882-893. 10.1109/78.127960View ArticleGoogle Scholar
- Park S-I, Smith MJT, Mersereau RM: Improved structures of maximally decimated directional filter banks for spatial image analysis. IEEE Transactions on Image Processing 2004, 13(11):1424-1431. 10.1109/TIP.2004.836186View ArticleGoogle Scholar
- Nguyen TT, Oraintara S: A class of multiresolution directional filter banks. IEEE Transactions on Signal Processing 2007, 55(3):949-961.MathSciNetView ArticleGoogle Scholar
- Folsom TC, Pinter RB: Primitive features by steering, quadrature, and scale. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998, 20(11):1161-1173. 10.1109/34.730552View ArticleGoogle Scholar
- Bovik AC, Clark M, Geisler WS: Multichannel texture analysis using localized spatial filters. IEEE Transactions on Pattern Analysis and Machine Intelligence 1990, 12(1):55-73. 10.1109/34.41384View ArticleGoogle Scholar
- Hagan MH, Menhaj MB: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 1994, 5(6):989-993. 10.1109/72.329697View ArticleGoogle Scholar
- Tivive FHC, Bouzerdoum A: Efficient training algorithms for a class of shunting inhibitory convolutional neural networks. IEEE Transactions on Neural Networks 2005, 16(3):541-556. 10.1109/TNN.2005.845144View ArticleGoogle Scholar
- Tivive FHC, Bouzerdoum A: A gender recognition system using shunting inhibitory convolutional neural networks. Proceedings of the International Joint Conference on Neural Networks (IJCNN '06), July 2006 5336-5341.Google Scholar
- Tivive FHC, Bouzerdoum A: A hierarchical learning network for face detection with in-plane rotation. Neurocomputing 2008, 71(16–18):3253-3263.View ArticleGoogle Scholar
- Chang C-C, Lin C-J: LIBSVM: a library for support vector machines. 2001, http://www.csie.ntu.edu.tw/~cjlin/libsvm/Google Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.