Feature-specific biometric sensing using ceiling view based pyroelectric infrared sensors

Abstract

This article presents a feature-specific infrared sensing paradigm and its application in building a lightweight human biometric detection system for wireless sensor network (WSN). The proposed paradigm exploits the novel use of pyroelectric infrared (PIR) sensor arrays in pursuit of unifying motion detection and biometric sensing with a ceiling view based fashion. Three PIR sensors with compound-eye structured field of view (FOV) are used to organize the biometric feature acquisition. The application of the proposed sensing paradigm to lightweight biometric sensing is addressed in the context of the path-constrained walker recognition with the vector quantization (VQ) method. Experimental studies are conducted to validate the proposed method.

Introduction

Wireless sensor networks (WSNs) have come into view as an important tool both in surveillance applications and the internet of things (IoT) [1, 2]. They have provided flexible sensing capabilities for the human movement understanding in diverse environments. Human motion acquisition, detection and biometric recognition are fundamental problems in these systems. The higher level understanding for the environment can be deduced from these fundamental clues. However, the challenges in WSN are how to acquire and process the sensor data efficiently [3], which desiderates low-power and inexpensive sensor nodes, low communication cost and real time data processing.

The developments of recent surveillance applications need the WSN to provide the functionalities of detecting the human motions and reporting their attributes (e.g., location, identity) [4]. This is useful for information-associated human tracking [5] and behavior analysis [6]. Fingerprint, iris and face based identification techniques have been demonstrated as the most powerful methods for representing human identity. However, these feature acquisitions belong to invasive fashion and the use is not flexible in the WSN. Walking feature based biometric sensing as a non-invasive technique plays an influential role in representing human identity [7]. In addition, the walking feature has the inherent advantages of being hard to hide and disguise. Although the walking feature is not a strong trust and robust identity, compared to the sophisticated identification representation, the lightweight identity plays an important role in the small group circumstances [5]. This article focuses on the lightweight biometric sensing and walker recognition for the WSN and IoT.

Recent advances in imaging technologies and micro-electro-mechanical systems have provided new opportunities to sense with small, low-power, and low-cost image/video devices [2]. Visual information acquisition and human gait based biometric sensing may be attracted because of the plenty research of vision based gait recognition. However, visual sensor based WSN suffers from the imbalances among information acquisition, transmission and utilization. On the one hand, high-resolution video-rate data streams are acquired and then compressed with various algorithms to send to the center station. While the center station needs to decode these compressive data streams to recover the raw image or video. On the other hand, the scene is recorded as an image representation by a high-resolution visual sensor, while the interest or actionable areas are often sparse. In general, the image representations are high-dimensional, which makes the following analysis computationally more expensive. Researchers prefer to seek a more compact and stable feature presentation by projecting the image representation into a lower dimensional space. In addition, image-based gait methods need to record images, which is a violation of human privacy and may not be accepted.

Figure 1 shows a detailed vision based information processing framework for conventional gait sensing and recognition. Usually, the first step of visual processing method is to detect the human targets [7]. The background modeling and motion optical flow extraction are the most used methods for human detection [8–10]. However, how to detect the silhouette of a person from a reference background is still a difficult problem, which may need complex image processing algorithms. Also, optical flow based methods are susceptible to the noise, which is unstable in the practical environments [8]. And the estimation of optical flow is complex and intractable in real-time. The second step is the motion modeling and to extract the features according to various motion representations from the target region [10]. However, the dimension of the extracted motion feature is still high. It is inflexible and difficult for realtime applications especially in WSN.

Figure 1

Conventional image based feature acquisition framework.

In this article, we present a new ceiling view based biometric sensing that using the pyroelectric infrared (PIR) sensor. Our main contribution is two-fold. First, to overcome the constraints of WSN, we propose a ceiling view based sensing module. PIR sensors are employed to detect the infrared radiation changes produced by motion in a passive sensing way. Each sensor is equipped with a compound-eye shaped high-resolution Fresnel lens array. Three sensors form a fine-level division for the sensing field. More biometric details can be captured. Second, the sensor arrays can form a feature-specific sensing pattern for human detection, identity recognition and behavior analysis, as shown in Figure 2. Human identity can be derived from the sensors’ output. The vector quantization (VQ) based methods are used for modeling each pedestrian without considering the temporal order dependency. We test and evaluate the proposed method on a laboratory database containing 11 registered members. Each member walks along a constrained path. Experimental results demonstrate the efficiency of the feature-specific sensing and the ability for lightweight recognition.

Figure 2

Feature-specific infrared sensing framework.

This article is organized as follows: In Section ‘Related study’, we review the related study. In Section ‘Sensing model’, we present the sensing model. In Section ‘Walker recognition method’, we show the VQ based walker recognition method. In Section ‘Experiments’, we describe the experiments and evaluate our approach on a small laboratory database. In Section ‘Conclusion’, we give the conclusion.

Related study

Some attentions have been invested on the infrared radial (IR) based human motion acquisition for behavior understanding. The PIR sensors are suitable for surveillance applications because of their advanced sensitivity to human motion and resistance to illumination changes [11–16]. Shankar et al. explore the response characteristics of pyroelectric sensors caused by human motion and their ability for human tracking [11]. Feller et al. show the common use of pyroelectric sensors is a low-cost and commercially available solution for motion sensing [12]. Structured lenslets are used to extract frequency components of targets for classification, the location and orientation of the sensors are combined to track targets when they move through the space. Hao et al. present a multiple-lateral-view based wireless pyroelectric sensor system to perform human tracking [17]. Human location can be surmised from the angle of arrival (AoA) in distributed sensor modules. In the study of [16], Hao et al. subsequently show the PIR sensor has the potentiality to be a reliable biometric solution for the verification/identification of a small group of human subjects. The complex multiple person tracking can be realized by identity linking. Above studies show the PIR sensors can be used as the lightweight sensor network nodes for the distributed human-target-surveillance.

Human identification is of practical importance for the information-associated tracking. In a tracking application, the body’s identity information can help distinguish the walking path. Gu et al. confirm lightweight detection and classification are more suitable for WSN [5]. Several studies exploit the multi-granularity ability of PIR sensor for human detection and walker recognition simultaneously. In the study of [18], Fang et al. develop a pyroelectric infrared system for real-time walker recognition. A pyroelectric infrared sensor with a Fresnel lens is used for gait acquisition, and a linear regression technique is employed for recognition. In the study of [19], Fang et al. introduce a path-dependent human identification using a pyroelectric infrared sensor and coded Fresnel lens array. In the study of [20], Fang et al. use digital sequential data generated by walkers as the gait features, and the hidden Markov model to model the characteristics of individuals.

Unlike the lateral view based sensing approaches, the ceiling view method is more flexible and manageable. Lateral view based approaches inevitably suffer from occlusion, especially at home or office. In addition, the characteristics of the human body are difficult to collect, because the amplitude of sensor response is varying with human motion at different distances. While the ceiling view based method is able to overcome these shortcomings. In [13, 21], Hosokawa et al. propose a ceiling view based person tracking with infrared sensors. Walkers’ biometrics are used for feature-associated tracking and Bayesian network is employed for inferring the position of the walker. The multiple people tracking can be achieved based on a high probability of correct identification. Burchett et al. give a lightweight biometric detection system with PIR sensors [22]. Their method not only enables to find the location and identification of the human body, but also to recognize some typical actions. However, most of these systems use a visual modulation with low-resolution sampling. The physiological diversities, such as the details of limb-level motion, may not be sensed.

Compared with the ceiling view based approaches [13, 21, 22], the main differences of this article are in two-fold. For the sensing task, we employ multiple ceiling view based PIR sensors with compound-eye structured modulation for splitting the object space into multiple subregions. High spatial resolution enables us to capture the limb-level diversities caused by the different body size and walking postures. For the recognition task, human identity is represented in a set of one’s codewords and without modeling any temporal order. Each member gets his or her own VQ codebook based on several training repetitions of the walk. Codebooks are obtained by the efficient k-means clustering method [23]. The test biometrics sequence is evaluated against all the codebooks and the identity with the lowest distortion measure is chosen as the output of the recognition. The proposed recognition method can simplify the transmission and processing drastically, which is suitable for the applications in WSN.

Sensing model

In this section, we will present the pyroelectric sensor model and the high-resolution visibility modulation with Fresnel lens arrays.

Pyroelectric Sensor Modeling

Figure 3 shows the sensing model of a PIR sensor. The pyroelectric sensors D 205b commercially available are employed for sensing the thermal radiation changes in the object space [24]. The sensor has the characteristic of being a bandpass filter with the wavelengths of 8 μ m to 14 μ m, which just corresponds to the thermal radiation emitted from the human body. The intrusion of visible illumination can be removed. Object space is defined as the collection of the thermal radiation field in a walker’s body. Fresnel lenses are used to bridge the field of view (FOV) of the PIR sensor to the motion sensing space. When a person moves across the object space continuously, one can motivate the corresponding sensors.

Figure 3

Pyroelectric infrared sensing model.

The sensing model can be represented by a form of reference structure [25]:

\mathbf{m}({\mathbf{r}}_m,t)=\mathrm{H}\left(t\right)\ast \sum \mathbf{V}({\mathbf{r}}_m,{\mathbf{r}}_s)\mathbf{S}({\mathbf{r}}_s,t),

(1)

where ∗ is the convolution operator; H(t) = [ dP _s  / dT ] [ dT/dt ] denotes the impulse response function of the PIR sensor; T is the temperature; t is the time tag andP _s denotes the polarization per unit volume [26]. The quantity P = dP _s  / dT is known as the pyroelectric coefficient and related with the pyroelectric materials, so the H(t) = P [dT / dt] is the rate of temperature change. In particular, a stationary human body does not trigger the sensor and the PIR sensor only responds to human movement without considering the textures of clothing. S and m denote the radiation state vector and the measurement vector, respectively. V(r _m r _s ) is the visibility function, which is “1” whenr _s is visible to the sensor atr _m , otherwise is “0”.

Sensor modules with visibility modulation

An important issue in the walker recognition task is to extract the salient features that will most effectively represent identity characteristics. In a seminal work, Johansson has demonstrated that human could distinguish the gender from the motion of a few light points attached to the body [27]. The movement information is the essential feature of human gait. Following this idea, Little and Boyd introduced the dense optical flow of a walker for individual recognition [8]. Their research also highlights the importance of motion information for gait representation. Besides, limb-level motion information are able to aid in designing efficient and task-specific biometric sensing. Lee and Grimson extracted each silhouette from video sequence, and then divided them into 7 parts [9]. Head, shoulder region, front of the torso were matched in accordance with these parts and ellipses were fitted to each region. Finally, the joint motion parameters of these ellipses were extracted as gait features for recognition task. The above research results provide noteworthy clues for biometric feature acquisition. Human motion is connected with locomotion of the motion by limbs explicitly. Moreover, walking movements can be decomposed into the limb-swinging along the horizontal and vertical directions.

In the study of [18], Fang et al. explores the performances of PIR based walker recognition with the different horizontal resolutions. They conclude the higher horizontal resolution enables to get more details and better results. However, our design focuses on how to capture the motion feature caused by horizontal and vertical movements with the looking-down fashion.

The compound eyes of insect provide the noteworthy intuitions for our visual modulation, due to its outstanding ability to find and track the moving objects in the horizontal and radial direction accurately [28]. The Fresnel lenses are employed to modulate the FOV of the PIR sensor, which are made of a light-weight, low-cost plastic material and has good focusing characteristics. Figure 4a shows the top view of the compound eye structured Fresnel lens, where the white and dark regions correspond to the visible and invisible regions, respectively. The sampling space is split into 19 non-overlapping subregions, and denoted as:

O=\underset{L}{\cup }{o}_l\text{and}{o}_i\cap o_j=\varnothing \text{for}i\ne \mathrm{j.}

(2)

Figure 4

Characteristic of the visibility in one Fresnel lens (a) Top view,(b) lateral view.

Figure 4b shows the lateral view of the Fresnel lens. The characteristics of this compound-eye visual modulation enable higher resolution and sensitivity in the horizontal direction. More precisely, we can represent a single sensing channel in the form of:

m({\mathbf{r}}_m,t)=\mathrm{H}\left(t\right)\ast \sum _{l=1}^{19}\mathbf{S}({\mathbf{r}}_{o_l},t).

(3)

We should note that a single view based sensor is only sensitive to horizontal motion.

To capture the motion details in the radial direction, two more same modulated sensors are provided with small disparity. Figure 5 shows the proposed sensing setup. The object space is divided into many subregions by three compound-eye modulated sensors, and the radial movement information is fused in the parallel multi-channel output. This sensing method leads to capture characteristics of omnidirectional movements during walking. Comparing with previously mentioned studies, the designed field structure not only has high resolution in the horizontal direction, but also in the vertical direction.

Figure 5

Experimental setup.

Feature generation

With the sensor arrays deployed as in Figure 5, the object space is partitioned into a set of subregions. When one walks through the prescribed path, the corresponding sensor will be activated by the limb movements. Figure 6a shows the response collected from a single sensor channel. Human motion detection can be achieved easily based on the energy of sensors’ signal.

Figure 6

Typical signals collected from the sensors when a person walks through their FOVs. (a) Signal collected from a single sensor, (b) Fused feature sequence by concatenating three feature vectors into a higher dimensional vector

The three feature sequences produced from three sensor channels are combined in a simplest way, as illustrated in Figure 6b. The fused feature vector is built by concatenating the feature sequences into a higher dimensional sequence, which can be denoted by:

\mathbf{m}\left(t\right)={\left[m_1\right(t),m_2(t),m_3(t\left)\right]}^T,

(4)

where them₁(t),m₂(t),m₃(t) are the outputs of the three sensors in Figure 5. Compared with the image based feature exaction, the proposed method reduces the dimension of feature drastically, which just contains a three-dimensional sequence. This not only saves the sensor resources, but also the power consumption, storage space and bandwidth needed for transmission.

Walker recognition method

We model each walker’s identity using the VQ method, since it has successful applications archived in efficient data compression [29] and voice-based speaker recognition [30]. The utility of VQ lies in the efficiency of using compact codebooks for modeling sequence pattern. In VQ based method, the codebooks have no explicit temporal order information and can come from any part of the training feature sequence. However, some indirect clues are preserved because the entries of codebook are trained to minimize average distortion through one’s training sequences. Each codeword in the codebook is corresponded with the more frequent feature slices or key-poses in the individual training data. Often a small size of codebook is enough to represent one’s identity during relatively steady walk. Compared with the well-studied time-varying signal modeling approach (such as the Hidden Markov Models, HMMs [31]), the VQ is a lightweight modeling approach and has not the ability to model complex time series. Therefore, the VQ is a weak classifier for walker recognition. Under the assumption of using the proposed sensing method as the data acquisition method, if a lightweight and weak classifier would achieve good results, then we can demonstrate the validity of the proposed sensing method indirectly.

The overall recognition procedure is illustrated in Figure 7. First, we collect several repetitions as the training sequence for each person. The training sequence can be denoted as:

{\mathbf{M}}_w=[{\mathbf{M}}_1,{\mathbf{M}}_2,\dots ,{\mathbf{M}}_{\text{num}}],

(5)

where w is the identity label and num is the number of repetitions. Second, the k-means clustering method is used here for searching the codewords in the training sequenceM _w . The k-means clustering is one of the most commonly used method for finding K clusters or codebook in the N observations and each observation belongs to the corresponding cluster with the nearest mean [23]. Suppose that a set of observations M = m₁,…,m _n ,…,m _N is given and the number of clusters is K, then we can define an objective function as the sum of square distances between a data point and its nearest cluster centers. Suppose the data points in cluster k are represented by a setS _k with a cluster center or codewordc _k , the k-means clustering is able to be formed as minimizing the object function, denoted by:

{\text{arg min}}_{\mathbf{S}}\sum _{k=1}^K\sum _{{\mathbf{m}}_n\in {\mathbf{S}}_k}\parallel {\mathbf{m}}_n-{\mathbf{S}}_k{\parallel }^2.

(6)

Then the problem of clustering or searching codewords can be formulated as a problem of optimization. But this object function is non-convex, existing methods try to use various iterative schemes for finding a locally minimal solution. Reference [23] gives an efficient k-means clustering algorithm. Then, each person gets his or her own identity representation with a bag of codewords denoted byC _w  = [c _w,1c_w,2,…,c_w,K]. Here K is the size of the codebook. Third, with total W reference identities, the matching of an unknown walker is performed as follow. A test sequence M= m(1),…,m(t),…,m(T)] will be evaluated against all codebooks and the distortion d(M C _w ) is recorded. Here the distortion is defined by:

d(\mathbf{M},{\mathbf{C}}_w)=\sum _{t=1}^T\parallel \mathbf{m}\left(t\right)-{\widehat{\mathbf{c}}}_{w,k}\left(t\right){\parallel }^2,

(7)

Figure 7

Overview of the VQ based recognition method.

where {\widehat{\mathbf{c}}}_{w,k}\left(t\right) is the codeword with the nearest distance to the vector m(t) in the codebookC _w . The corresponding identity is given by:

w^{\ast }=\underset{w\in W}{\text{arg min}}d(\mathbf{M},{\mathbf{C}}_w).

(8)

As illustrated in Figure 7, it is obvious that, with a Person 1’s testing sequence, the output of quantizer 2 has a larger distortion than that in quantizer 1.

The walker verification has a similar framework. Only the codebook of the claimed identity is used and the resultant average distortion is compared with a fixed threshold for rejecting or accepting the identity for an unknown walker.

Experiments

Our proposed method is integrated by the smart system-on-chip (SoC) CC2430[32]. Figure 8 shows the prototype of a sensor module. Figure 5 presents the experimental setup. A sensor module, which contains three high-resolution modulated PIR sensors, is mounted on the ceiling at a height of 320 cm to look down to detect the motion. A typical wireless sensor network in surveillance applications often provides local sensing coverage for a relatively limited area. Acquiring the biometric information of the walkers will contribute to multi-human tracking with wireless distributed sensors. Basically the number of monitored members is limited. Therefore, the sensory data is collected with 11 members in our laboratory walked along a constrained straight path. The horizontal width of the FOV of the sensor module is about 450 cm. Each member repeats his or her walks along the path 40 times. The analog feature inputs are sampled at 50 Hz.

Figure 8

The ceiling view based sensor module with three sensors.

For testing the ability of the proposed sensing method, the recognition accuracy is used and defined by:

\mathrm{TR}=\frac{\text{the number of the sets being truly recognized}}{\text{the number of the testing sets}}.

(9)

All recognition accuracies are computed with the leave-half-out cross-validation. At the training phase, we randomly selected 20 data sets from each person for the parameters training. Each vector quantizer can be considered as one’s reference template. At the testing phase, the remaining data sets are used to estimate the performance of recognition. The distortion obtained through the vector quantizer defines the matching similarity between the testing sequence and registered template.

The recognition accuracy on the testing set with respect to different sizes of codebooks is shown in Figure 9. We repeat the experiment 50 times with all walkers and average over the results. Our method presents a recognition rate of above 80% when the number of codebook is greater than 30. When the size of the codebook is greater than 50, the rising trend of the average accuracy becomes smooth gradually. The possible reason is that the average accuracy depends on the features of the collected signal intrinsically. While the distinguishable ability of the features is determined by the proposed sensing method. When the complexity of parameters reaches a certain size, the performance would not be better. Similarly to the last experiment, we compute the average accuracy of per member with the number of codebook, which is shown in Figure 10. It can be seen from Figure 10, even if there is a larger size codebook, some people are still difficult to be identified. It may require more detailed clues and more effective classification method.

Figure 9

Average accuracy with respect to different sizes of codebook.

Figure 10

Average accuracy of per member with respect to different sizes of codebook.

Figure 11 shows the confusion matrix based on our proposed sensing method with the coobook size 50. The confusion matrix shows the identity label (vertical) versus the classification results (horizontal). Each pixel (i,j) in the matrix denotes the percentage of person i being recognized as person j. The percentage of the correctly recognized identity can be obtained by calculating the trace of matrix. The remaining darker cells present the percentage of misclassification. It can be seen that among the 11 walkers, the lowest identification rate is 65%, the highest is 100%, and the average is 85.45%. Some people are difficult to be identified. The reasons are two aspects. First, the proposed sensing method is part of the lightweight biometric sensing. The ceiling view based sensing pattern restricts the better acquisition of the walker’s physical details. Second, the VQ is a lightweight modeling approach and has not the ability to model complex time series. Under the assumption of using the proposed sensing method as the data acquisition method, if a lightweight and weak classifiers recognition method would achieve good results, then we can demonstrate the validity of the proposed sensing method indirectly. Also, we could find the misclassification will lead this sensing system to a lightweight recognition.

Figure 11

Confusion matrix with the codebook of size 50.

Figure 12 illustrates the influence of group and codebook’s size on the average correct recognition rate. We can find the identification rate degrades when the number of registered members grows from 2 to 11. Three kinds of sizes of codebooks give similar performance trends.

Figure 12

Average identification accuracy as a function of the number of members with three sizes of codebook.

Another important issue for the surveillance applications is that the proposed method should be able to recognize whether a walker is a legal member or not. We define the legal walker as the positive events. That is, if a legal walker is detected correctly, the detection result is said to be a true-positive. If an illegal member is detected as a legal one, the detection result is said to be a false-positive. We randomly select a member from the total set as the illegal one, and the following results are based on the average values of 100 repeats. With the well trained codebooks all for the legal walker, a testing multidimensional sequence M can be identified as a legal member or not based on the following criterion:

(10)

where Threshold is the threshold value for distinguishing the legal members from the illegal ones. Figure 13 shows the receiver operating characteristic (ROC) curves corresponding to the above method with respect to different parameters of codebooks and threshold values. It is a tool for selecting the optimal codebooks and identifying the optimal threshold values. A test with the point (0,1) is accurate with the true positive rate 100 and the false positive rate 0. It is desirable that the point (0,1) can be close to the ROC curve as possible. It is evident that a codebook with size 40, shown as the curve with green color in Figure 13, is sufficiently satisfactory for separating the legal walkers from the illegal ones. The thresholds can be specified to make a tradeoff between true-positive and false-positive. Once the parameters and threshold values are well determined, the unknown walker can be detected as a legal member or not. Here, threshold can be fixed based on the optimal decision region. Table 1 shows the accuracy for detecting legal and illegal walkers.

Figure 13

ROC curves for detecting the legal walkers with different codebooks.

Table 1 The optimal results for detecting legal and illegal walkers with the codebook of size 50

It should be noted that the recognition is very fast, since the testing feature is represented by a low-dimensional sequence (three dimensions). The relevant algorithms in our experimental studies run on an Intel Pentium4 2.8 GHz computer by Matlab codes. Even in the case of 11 members with codebook size 50, the longest recognition and legal detection time will not exceed 0.1 s.

Conclusion

Recent developments in the WSN for surveillance reveal urgent needs with the feature-specific sensing paradigm. It is determined by the special constraints in WSN. To meet the above needs, we presented a compact, low-cost and efficient ceiling view based sensing method for human biometric detection. The proposed sensing method gives rise to two main advantages. First, the sensing module consists of three PIR sensors, and each sensor is equipped with a compound-eye shaped Fresnel lens array. The characteristics of omnidirectional movement under walking can be captured. Second, under the feature-specific infrared sensing paradigm, the features can be directly encoded into low-dimensional measurement data and thus the operations for recognition tasks can be performed directly on measurement space, which can facilitate the development of realtime tracking systems.

For the recognition task, VQ as an uncomplicated modeling method, were used to build a template for each member. Experiments in the context of the path-constrained walker identification were conducted to validate the proposed sensing method. Human motion detection and identification can be carried out in real-time. In the experiments, we do not provide experimental results to demonstrate the benefit of the proposed framework over the conventional image based feature acquisition framework, the reasons are two-fold. First, the motivation of this article aims to exploit the feature-specific infrared sensing paradigm for lightweight walker recognition. The scope of the study is part of wireless sensor networks. While the conventional image based feature acquisition framework is an extensive sampling method that the information obtained contains a large amount of redundancy information. Figure 1 is just used for illustrating the asymmetry between the data acquisition and utilization. The large amount of video data will increases the sampling and transmission loads directly, thus it is not suitable to apply it to the WSN experimental platform. Second, from the signal processing perspective, we concern about the identification of the low-dimensional sensory signal instead of the body image. These two types of signals provide different levels of features and the proposed method is a complementary sensing mode for the conventional image based feature acquisition framework under certain circumstances.

We are exploiting the novel use of PIR sensors in the analysis of typical and abnormal behavior. The distributed PIR nodes with different modulations are able to form a multi-granularity and feature-specific perception of the scene, which will play an important role in the WSN and IoT.