This article presents a feature-specific infrared sensing paradigm and its application in building a lightweight human biometric detection system for wireless sensor network (WSN). The proposed paradigm exploits the novel use of pyroelectric infrared (PIR) sensor arrays in pursuit of unifying motion detection and biometric sensing with a ceiling view based fashion. Three PIR sensors with compound-eye structured field of view (FOV) are used to organize the biometric feature acquisition. The application of the proposed sensing paradigm to lightweight biometric sensing is addressed in the context of the path-constrained walker recognition with the vector quantization (VQ) method. Experimental studies are conducted to validate the proposed method.
Wireless sensor networks (WSNs) have come into view as an important tool both in surveillance applications and the internet of things (IoT) [1, 2]. They have provided flexible sensing capabilities for the human movement understanding in diverse environments. Human motion acquisition, detection and biometric recognition are fundamental problems in these systems. The higher level understanding for the environment can be deduced from these fundamental clues. However, the challenges in WSN are how to acquire and process the sensor data efficiently , which desiderates low-power and inexpensive sensor nodes, low communication cost and real time data processing.
The developments of recent surveillance applications need the WSN to provide the functionalities of detecting the human motions and reporting their attributes (e.g., location, identity) . This is useful for information-associated human tracking  and behavior analysis . Fingerprint, iris and face based identification techniques have been demonstrated as the most powerful methods for representing human identity. However, these feature acquisitions belong to invasive fashion and the use is not flexible in the WSN. Walking feature based biometric sensing as a non-invasive technique plays an influential role in representing human identity . In addition, the walking feature has the inherent advantages of being hard to hide and disguise. Although the walking feature is not a strong trust and robust identity, compared to the sophisticated identification representation, the lightweight identity plays an important role in the small group circumstances . This article focuses on the lightweight biometric sensing and walker recognition for the WSN and IoT.
Recent advances in imaging technologies and micro-electro-mechanical systems have provided new opportunities to sense with small, low-power, and low-cost image/video devices . Visual information acquisition and human gait based biometric sensing may be attracted because of the plenty research of vision based gait recognition. However, visual sensor based WSN suffers from the imbalances among information acquisition, transmission and utilization. On the one hand, high-resolution video-rate data streams are acquired and then compressed with various algorithms to send to the center station. While the center station needs to decode these compressive data streams to recover the raw image or video. On the other hand, the scene is recorded as an image representation by a high-resolution visual sensor, while the interest or actionable areas are often sparse. In general, the image representations are high-dimensional, which makes the following analysis computationally more expensive. Researchers prefer to seek a more compact and stable feature presentation by projecting the image representation into a lower dimensional space. In addition, image-based gait methods need to record images, which is a violation of human privacy and may not be accepted.
Figure 1 shows a detailed vision based information processing framework for conventional gait sensing and recognition. Usually, the first step of visual processing method is to detect the human targets . The background modeling and motion optical flow extraction are the most used methods for human detection [8–10]. However, how to detect the silhouette of a person from a reference background is still a difficult problem, which may need complex image processing algorithms. Also, optical flow based methods are susceptible to the noise, which is unstable in the practical environments . And the estimation of optical flow is complex and intractable in real-time. The second step is the motion modeling and to extract the features according to various motion representations from the target region . However, the dimension of the extracted motion feature is still high. It is inflexible and difficult for realtime applications especially in WSN.
In this article, we present a new ceiling view based biometric sensing that using the pyroelectric infrared (PIR) sensor. Our main contribution is two-fold. First, to overcome the constraints of WSN, we propose a ceiling view based sensing module. PIR sensors are employed to detect the infrared radiation changes produced by motion in a passive sensing way. Each sensor is equipped with a compound-eye shaped high-resolution Fresnel lens array. Three sensors form a fine-level division for the sensing field. More biometric details can be captured. Second, the sensor arrays can form a feature-specific sensing pattern for human detection, identity recognition and behavior analysis, as shown in Figure 2. Human identity can be derived from the sensors’ output. The vector quantization (VQ) based methods are used for modeling each pedestrian without considering the temporal order dependency. We test and evaluate the proposed method on a laboratory database containing 11 registered members. Each member walks along a constrained path. Experimental results demonstrate the efficiency of the feature-specific sensing and the ability for lightweight recognition.
This article is organized as follows: In Section ‘Related study’, we review the related study. In Section ‘Sensing model’, we present the sensing model. In Section ‘Walker recognition method’, we show the VQ based walker recognition method. In Section ‘Experiments’, we describe the experiments and evaluate our approach on a small laboratory database. In Section ‘Conclusion’, we give the conclusion.
Some attentions have been invested on the infrared radial (IR) based human motion acquisition for behavior understanding. The PIR sensors are suitable for surveillance applications because of their advanced sensitivity to human motion and resistance to illumination changes [11–16]. Shankar et al. explore the response characteristics of pyroelectric sensors caused by human motion and their ability for human tracking . Feller et al. show the common use of pyroelectric sensors is a low-cost and commercially available solution for motion sensing . Structured lenslets are used to extract frequency components of targets for classification, the location and orientation of the sensors are combined to track targets when they move through the space. Hao et al. present a multiple-lateral-view based wireless pyroelectric sensor system to perform human tracking . Human location can be surmised from the angle of arrival (AoA) in distributed sensor modules. In the study of , Hao et al. subsequently show the PIR sensor has the potentiality to be a reliable biometric solution for the verification/identification of a small group of human subjects. The complex multiple person tracking can be realized by identity linking. Above studies show the PIR sensors can be used as the lightweight sensor network nodes for the distributed human-target-surveillance.
Human identification is of practical importance for the information-associated tracking. In a tracking application, the body’s identity information can help distinguish the walking path. Gu et al. confirm lightweight detection and classification are more suitable for WSN . Several studies exploit the multi-granularity ability of PIR sensor for human detection and walker recognition simultaneously. In the study of , Fang et al. develop a pyroelectric infrared system for real-time walker recognition. A pyroelectric infrared sensor with a Fresnel lens is used for gait acquisition, and a linear regression technique is employed for recognition. In the study of , Fang et al. introduce a path-dependent human identification using a pyroelectric infrared sensor and coded Fresnel lens array. In the study of , Fang et al. use digital sequential data generated by walkers as the gait features, and the hidden Markov model to model the characteristics of individuals.
Unlike the lateral view based sensing approaches, the ceiling view method is more flexible and manageable. Lateral view based approaches inevitably suffer from occlusion, especially at home or office. In addition, the characteristics of the human body are difficult to collect, because the amplitude of sensor response is varying with human motion at different distances. While the ceiling view based method is able to overcome these shortcomings. In [13, 21], Hosokawa et al. propose a ceiling view based person tracking with infrared sensors. Walkers’ biometrics are used for feature-associated tracking and Bayesian network is employed for inferring the position of the walker. The multiple people tracking can be achieved based on a high probability of correct identification. Burchett et al. give a lightweight biometric detection system with PIR sensors . Their method not only enables to find the location and identification of the human body, but also to recognize some typical actions. However, most of these systems use a visual modulation with low-resolution sampling. The physiological diversities, such as the details of limb-level motion, may not be sensed.
Compared with the ceiling view based approaches [13, 21, 22], the main differences of this article are in two-fold. For the sensing task, we employ multiple ceiling view based PIR sensors with compound-eye structured modulation for splitting the object space into multiple subregions. High spatial resolution enables us to capture the limb-level diversities caused by the different body size and walking postures. For the recognition task, human identity is represented in a set of one’s codewords and without modeling any temporal order. Each member gets his or her own VQ codebook based on several training repetitions of the walk. Codebooks are obtained by the efficient k-means clustering method . The test biometrics sequence is evaluated against all the codebooks and the identity with the lowest distortion measure is chosen as the output of the recognition. The proposed recognition method can simplify the transmission and processing drastically, which is suitable for the applications in WSN.
In this section, we will present the pyroelectric sensor model and the high-resolution visibility modulation with Fresnel lens arrays.
Pyroelectric Sensor Modeling
Figure 3 shows the sensing model of a PIR sensor. The pyroelectric sensors D 205b commercially available are employed for sensing the thermal radiation changes in the object space . The sensor has the characteristic of being a bandpass filter with the wavelengths of 8 μ m to 14 μ m, which just corresponds to the thermal radiation emitted from the human body. The intrusion of visible illumination can be removed. Object space is defined as the collection of the thermal radiation field in a walker’s body. Fresnel lenses are used to bridge the field of view (FOV) of the PIR sensor to the motion sensing space. When a person moves across the object space continuously, one can motivate the corresponding sensors.
The sensing model can be represented by a form of reference structure :
where ∗ is the convolution operator; H(t) = [ dPs / dT ] [ dT/dt ] denotes the impulse response function of the PIR sensor; T is the temperature; t is the time tag andPs denotes the polarization per unit volume . The quantity P = dPs / dT is known as the pyroelectric coefficient and related with the pyroelectric materials, so the H(t) = P [dT / dt] is the rate of temperature change. In particular, a stationary human body does not trigger the sensor and the PIR sensor only responds to human movement without considering the textures of clothing. S and m denote the radiation state vector and the measurement vector, respectively. V(rmrs) is the visibility function, which is “1” whenrs is visible to the sensor atrm, otherwise is “0”.
Sensor modules with visibility modulation
An important issue in the walker recognition task is to extract the salient features that will most effectively represent identity characteristics. In a seminal work, Johansson has demonstrated that human could distinguish the gender from the motion of a few light points attached to the body . The movement information is the essential feature of human gait. Following this idea, Little and Boyd introduced the dense optical flow of a walker for individual recognition . Their research also highlights the importance of motion information for gait representation. Besides, limb-level motion information are able to aid in designing efficient and task-specific biometric sensing. Lee and Grimson extracted each silhouette from video sequence, and then divided them into 7 parts . Head, shoulder region, front of the torso were matched in accordance with these parts and ellipses were fitted to each region. Finally, the joint motion parameters of these ellipses were extracted as gait features for recognition task. The above research results provide noteworthy clues for biometric feature acquisition. Human motion is connected with locomotion of the motion by limbs explicitly. Moreover, walking movements can be decomposed into the limb-swinging along the horizontal and vertical directions.
In the study of , Fang et al. explores the performances of PIR based walker recognition with the different horizontal resolutions. They conclude the higher horizontal resolution enables to get more details and better results. However, our design focuses on how to capture the motion feature caused by horizontal and vertical movements with the looking-down fashion.
The compound eyes of insect provide the noteworthy intuitions for our visual modulation, due to its outstanding ability to find and track the moving objects in the horizontal and radial direction accurately . The Fresnel lenses are employed to modulate the FOV of the PIR sensor, which are made of a light-weight, low-cost plastic material and has good focusing characteristics. Figure 4a shows the top view of the compound eye structured Fresnel lens, where the white and dark regions correspond to the visible and invisible regions, respectively. The sampling space is split into 19 non-overlapping subregions, and denoted as:
Figure 4b shows the lateral view of the Fresnel lens. The characteristics of this compound-eye visual modulation enable higher resolution and sensitivity in the horizontal direction. More precisely, we can represent a single sensing channel in the form of:
We should note that a single view based sensor is only sensitive to horizontal motion.
To capture the motion details in the radial direction, two more same modulated sensors are provided with small disparity. Figure 5 shows the proposed sensing setup. The object space is divided into many subregions by three compound-eye modulated sensors, and the radial movement information is fused in the parallel multi-channel output. This sensing method leads to capture characteristics of omnidirectional movements during walking. Comparing with previously mentioned studies, the designed field structure not only has high resolution in the horizontal direction, but also in the vertical direction.
With the sensor arrays deployed as in Figure 5, the object space is partitioned into a set of subregions. When one walks through the prescribed path, the corresponding sensor will be activated by the limb movements. Figure 6a shows the response collected from a single sensor channel. Human motion detection can be achieved easily based on the energy of sensors’ signal.
The three feature sequences produced from three sensor channels are combined in a simplest way, as illustrated in Figure 6b. The fused feature vector is built by concatenating the feature sequences into a higher dimensional sequence, which can be denoted by:
where them1(t),m2(t),m3(t) are the outputs of the three sensors in Figure 5. Compared with the image based feature exaction, the proposed method reduces the dimension of feature drastically, which just contains a three-dimensional sequence. This not only saves the sensor resources, but also the power consumption, storage space and bandwidth needed for transmission.
Walker recognition method
We model each walker’s identity using the VQ method, since it has successful applications archived in efficient data compression  and voice-based speaker recognition . The utility of VQ lies in the efficiency of using compact codebooks for modeling sequence pattern. In VQ based method, the codebooks have no explicit temporal order information and can come from any part of the training feature sequence. However, some indirect clues are preserved because the entries of codebook are trained to minimize average distortion through one’s training sequences. Each codeword in the codebook is corresponded with the more frequent feature slices or key-poses in the individual training data. Often a small size of codebook is enough to represent one’s identity during relatively steady walk. Compared with the well-studied time-varying signal modeling approach (such as the Hidden Markov Models, HMMs ), the VQ is a lightweight modeling approach and has not the ability to model complex time series. Therefore, the VQ is a weak classifier for walker recognition. Under the assumption of using the proposed sensing method as the data acquisition method, if a lightweight and weak classifier would achieve good results, then we can demonstrate the validity of the proposed sensing method indirectly.
The overall recognition procedure is illustrated in Figure 7. First, we collect several repetitions as the training sequence for each person. The training sequence can be denoted as:
where w is the identity label and num is the number of repetitions. Second, the k-means clustering method is used here for searching the codewords in the training sequenceMw. The k-means clustering is one of the most commonly used method for finding K clusters or codebook in the N observations and each observation belongs to the corresponding cluster with the nearest mean . Suppose that a set of observations M = m1,…,mn,…,mN is given and the number of clusters is K, then we can define an objective function as the sum of square distances between a data point and its nearest cluster centers. Suppose the data points in cluster k are represented by a setSk with a cluster center or codewordck, the k-means clustering is able to be formed as minimizing the object function, denoted by:
Then the problem of clustering or searching codewords can be formulated as a problem of optimization. But this object function is non-convex, existing methods try to use various iterative schemes for finding a locally minimal solution. Reference  gives an efficient k-means clustering algorithm. Then, each person gets his or her own identity representation with a bag of codewords denoted byCw = [cw,1cw,2,…,cw,K]. Here K is the size of the codebook. Third, with total W reference identities, the matching of an unknown walker is performed as follow. A test sequence M= m(1),…,m(t),…,m(T)] will be evaluated against all codebooks and the distortion d(MCw) is recorded. Here the distortion is defined by:
where is the codeword with the nearest distance to the vector m(t) in the codebookCw. The corresponding identity is given by:
As illustrated in Figure 7, it is obvious that, with a Person 1’s testing sequence, the output of quantizer 2 has a larger distortion than that in quantizer 1.
The walker verification has a similar framework. Only the codebook of the claimed identity is used and the resultant average distortion is compared with a fixed threshold for rejecting or accepting the identity for an unknown walker.
Our proposed method is integrated by the smart system-on-chip (SoC) CC2430. Figure 8 shows the prototype of a sensor module. Figure 5 presents the experimental setup. A sensor module, which contains three high-resolution modulated PIR sensors, is mounted on the ceiling at a height of 320 cm to look down to detect the motion. A typical wireless sensor network in surveillance applications often provides local sensing coverage for a relatively limited area. Acquiring the biometric information of the walkers will contribute to multi-human tracking with wireless distributed sensors. Basically the number of monitored members is limited. Therefore, the sensory data is collected with 11 members in our laboratory walked along a constrained straight path. The horizontal width of the FOV of the sensor module is about 450 cm. Each member repeats his or her walks along the path 40 times. The analog feature inputs are sampled at 50 Hz.
For testing the ability of the proposed sensing method, the recognition accuracy is used and defined by:
All recognition accuracies are computed with the leave-half-out cross-validation. At the training phase, we randomly selected 20 data sets from each person for the parameters training. Each vector quantizer can be considered as one’s reference template. At the testing phase, the remaining data sets are used to estimate the performance of recognition. The distortion obtained through the vector quantizer defines the matching similarity between the testing sequence and registered template.
The recognition accuracy on the testing set with respect to different sizes of codebooks is shown in Figure 9. We repeat the experiment 50 times with all walkers and average over the results. Our method presents a recognition rate of above 80% when the number of codebook is greater than 30. When the size of the codebook is greater than 50, the rising trend of the average accuracy becomes smooth gradually. The possible reason is that the average accuracy depends on the features of the collected signal intrinsically. While the distinguishable ability of the features is determined by the proposed sensing method. When the complexity of parameters reaches a certain size, the performance would not be better. Similarly to the last experiment, we compute the average accuracy of per member with the number of codebook, which is shown in Figure 10. It can be seen from Figure 10, even if there is a larger size codebook, some people are still difficult to be identified. It may require more detailed clues and more effective classification method.
Figure 11 shows the confusion matrix based on our proposed sensing method with the coobook size 50. The confusion matrix shows the identity label (vertical) versus the classification results (horizontal). Each pixel (i,j) in the matrix denotes the percentage of person i being recognized as person j. The percentage of the correctly recognized identity can be obtained by calculating the trace of matrix. The remaining darker cells present the percentage of misclassification. It can be seen that among the 11 walkers, the lowest identification rate is 65%, the highest is 100%, and the average is 85.45%. Some people are difficult to be identified. The reasons are two aspects. First, the proposed sensing method is part of the lightweight biometric sensing. The ceiling view based sensing pattern restricts the better acquisition of the walker’s physical details. Second, the VQ is a lightweight modeling approach and has not the ability to model complex time series. Under the assumption of using the proposed sensing method as the data acquisition method, if a lightweight and weak classifiers recognition method would achieve good results, then we can demonstrate the validity of the proposed sensing method indirectly. Also, we could find the misclassification will lead this sensing system to a lightweight recognition.
Figure 12 illustrates the influence of group and codebook’s size on the average correct recognition rate. We can find the identification rate degrades when the number of registered members grows from 2 to 11. Three kinds of sizes of codebooks give similar performance trends.
Another important issue for the surveillance applications is that the proposed method should be able to recognize whether a walker is a legal member or not. We define the legal walker as the positive events. That is, if a legal walker is detected correctly, the detection result is said to be a true-positive. If an illegal member is detected as a legal one, the detection result is said to be a false-positive. We randomly select a member from the total set as the illegal one, and the following results are based on the average values of 100 repeats. With the well trained codebooks all for the legal walker, a testing multidimensional sequence M can be identified as a legal member or not based on the following criterion:
where Threshold is the threshold value for distinguishing the legal members from the illegal ones. Figure 13 shows the receiver operating characteristic (ROC) curves corresponding to the above method with respect to different parameters of codebooks and threshold values. It is a tool for selecting the optimal codebooks and identifying the optimal threshold values. A test with the point (0,1) is accurate with the true positive rate 100 and the false positive rate 0. It is desirable that the point (0,1) can be close to the ROC curve as possible. It is evident that a codebook with size 40, shown as the curve with green color in Figure 13, is sufficiently satisfactory for separating the legal walkers from the illegal ones. The thresholds can be specified to make a tradeoff between true-positive and false-positive. Once the parameters and threshold values are well determined, the unknown walker can be detected as a legal member or not. Here, threshold can be fixed based on the optimal decision region. Table 1 shows the accuracy for detecting legal and illegal walkers.
It should be noted that the recognition is very fast, since the testing feature is represented by a low-dimensional sequence (three dimensions). The relevant algorithms in our experimental studies run on an Intel Pentium4 2.8 GHz computer by Matlab codes. Even in the case of 11 members with codebook size 50, the longest recognition and legal detection time will not exceed 0.1 s.
Recent developments in the WSN for surveillance reveal urgent needs with the feature-specific sensing paradigm. It is determined by the special constraints in WSN. To meet the above needs, we presented a compact, low-cost and efficient ceiling view based sensing method for human biometric detection. The proposed sensing method gives rise to two main advantages. First, the sensing module consists of three PIR sensors, and each sensor is equipped with a compound-eye shaped Fresnel lens array. The characteristics of omnidirectional movement under walking can be captured. Second, under the feature-specific infrared sensing paradigm, the features can be directly encoded into low-dimensional measurement data and thus the operations for recognition tasks can be performed directly on measurement space, which can facilitate the development of realtime tracking systems.
For the recognition task, VQ as an uncomplicated modeling method, were used to build a template for each member. Experiments in the context of the path-constrained walker identification were conducted to validate the proposed sensing method. Human motion detection and identification can be carried out in real-time. In the experiments, we do not provide experimental results to demonstrate the benefit of the proposed framework over the conventional image based feature acquisition framework, the reasons are two-fold. First, the motivation of this article aims to exploit the feature-specific infrared sensing paradigm for lightweight walker recognition. The scope of the study is part of wireless sensor networks. While the conventional image based feature acquisition framework is an extensive sampling method that the information obtained contains a large amount of redundancy information. Figure 1 is just used for illustrating the asymmetry between the data acquisition and utilization. The large amount of video data will increases the sampling and transmission loads directly, thus it is not suitable to apply it to the WSN experimental platform. Second, from the signal processing perspective, we concern about the identification of the low-dimensional sensory signal instead of the body image. These two types of signals provide different levels of features and the proposed method is a complementary sensing mode for the conventional image based feature acquisition framework under certain circumstances.
We are exploiting the novel use of PIR sensors in the analysis of typical and abnormal behavior. The distributed PIR nodes with different modulations are able to form a multi-granularity and feature-specific perception of the scene, which will play an important role in the WSN and IoT.
Gu L, Jia D, Vicaire P, Yan T, Luo L, Tirumala A, Cao Q, He T, Stankovic JA, Abdelzaher T, Krogh BH: Lightweight detection and classification for wireless sensor networks in realistic environments. In Proceedings of the 3rd international conference on Embedded networked sensor systems. (ACM, New York, USA,); 2005. pp. 205–217
Lee L, Grimson WEL: Gait analysis for recognition and classification. In Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition. Washington, DC, USA, IEEE Computer Society; 2002. pp. 148–155
Chang HT, Chen CH, Hung CW, Shen DF: Frequency domain analysis of human motions in surveillance video. In Proceedings of the 2010 International Conference on Broadband, Wireless Computing, Communication and Applications. Fukuoka, Japan, IEEE; 2010. pp. 522–526
Feller SD, Burchett JB, Hao Q, Fang JS, Shankar M, Guenther BD, Brady DJ: Human characterization and tracking using pyroelectric sensors. In Optical Solutions for Homeland and National Security. Washington, DC, USA, Opt. Soc. Am; 2005. p. 14
Honda S, Fukui KI, Moriyama K, Kurihara S, Numao M: Extracting human behaviors with infrared sensor network. In Proceedings of the Fourth International Conference on Networked Sensing Systems. Braunschweig, Germany, IEEE; 2007. pp. 122–125
Fang JS, Hao Q, Brady DJ, Guenther BD, Hsu KY: A pyroelectric infrared biometric system for real-time walker recognition by use of a maximum likelihood principal components estimation (MLPCE) method. Opt. Express 2007, 15(6):3271-3284. 10.1364/OE.15.003271
This study was supported by the national Natural Science Foundation (NSF) of China, Grant Nos. 60970118 and 61002012. The authors would like to thank the anonymous reviewers for their constructive comments and suggestions. They also wish to thank all staffs of Information Processing & Human-Robot Systems lab in Sun Yat-sen University for their aids in conducting the measurement experiments.
Authors and Affiliations
Department of Audio-Visual and Image Technology, China Criminal Police University, Shenyang, 110854, China
School of Information Science and Technology, Sun Yat-sen University, Guangzhou, 510006, China
College of Physics & Electronic Information Engineering, Wenzou University, Wenzhou, 325035, China
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.