Object Association and Identification in Heterogeneous Sensors Environment
© Shung Han Cho et al. 2010
Received: 12 June 2010
Accepted: 8 November 2010
Published: 29 December 2010
An approach for dynamic object association and identification is proposed for heterogeneous sensor network consisting of visual and identification sensors. Visual sensors track objects by a 2D localization, and identification sensors (i.e., RFID system, fingerprint, or iris recognition system) are incorporated into the system for object identification. This paper illustrates the feasibility and effectiveness of information association between the position of objects estimated by visual sensors and their simultaneous registration of multiple objects. The proposed approach utilizes the object dynamics of entering and leaving the coverage of identification sensors, where the location information of identification sensors and objects is available. We investigate necessary association conditions using set operations where the sets are defined by the dynamics of the objects. The coverage of identification sensor is approximately modeled by the maximum sensing coverage for a simple association strategy. The effect of the discrepancy between the actual and the approximated coverage is addressed in terms of the association performance. We also present a coverage adjustment scheme using the object dynamics for the association stability. Finally, the proposed method is evaluated with a realistic scenario. The simulation results demonstrate the stability of the proposed method against nonideal phenomena such as false detection, false tracking, and inaccurate coverage model.
Recently, heterogeneous sensor network has received much attention in the field of multiple objects tracking to exploit advantages of using different modalities [1, 2]. Visual sensor is one of the most popular sensors due to its reliability and ease of analysis [3–5]. However, the visual sensor-based tracking system is limited only to recording the trajectory of objects because visual sensors have several limitations for object identification [6–9]. One of the main difficulties for the visual sensor-based object tracking is that distinguishable characteristics of the objects are nontrivial to be constructed for all the detected targets due to the objects' similarity in color, size, and shape. Moreover, accurate feature extraction is not always guaranteed. Therefore, identifying an object with features is a challenging problem. Also, several identification sensors, such as RFID (Radio Frequency Identification) system, fingerprint, or iris recognition system, have been utilized for object identification. However, the functionality of these sensors is limited only to the object identification and they are difficult to be used for the object tracking [10–12]. They can only alarm human operators for events triggered by identification sensors but cannot make intelligent decisions for them. For example, they cannot monitor the movement pattern of authorized people in special areas. Therefore, an identification sensor can only complement the visual sensor-based tracking system for the intelligent surveillance system.
There have been some related works regarding the issue of surveillance using heterogeneous types of sensors. The specific issues considered are various such as heterogeneous data association and efficient network architecture. Schulz et al.  proposed the method to track and identify multiple objects by using ID-sensors such as infrared badges and anonymous sensors such as laser range-finders. Although the system successfully associates the anonymous sensor data with ID-sensor data, the transition of the two phases is simply done by the heuristic of the average number of different assignments in Markov chains. Moreover, it does not provide a recovery method against losing the correct ID and the number of hypotheses grows extremely fast whenever several people are close to each other. Shin et al.  proposed the network architecture for a large-scale surveillance system that supports heterogeneous sensors such as video and RFID sensors. Although the event-driven control effectively minimizes the system load, the paper does not deal with the association problem of heterogeneous data but only the mitigation of the data overload. Cho et al. [15, 16] proposed the heterogeneous sensor node with an acoustic and RFID sensor where the coverage of an acoustic sensor is identical to the coverage of an RFID sensor. The association of the estimated position and the identification of an object is achieved by using a simple association rule that one and only one identification is registered within the coverage of the sensor node while its corresponding position is estimated within the coverage of the sensor node. The performance of these approaches, however, can be significantly degraded by the coverage uncertainty of the acoustic and RFID sensors. The coverage uncertainty is caused by the characteristics of acoustic and RFID signal. The system cannot accurately calibrate the time-varying coverage of those sensors. Moreover, multiple objects near the boundary of the sensor coverage may obscure the object identification by identification sensors and the object localization by acoustic sensors. Therefore, an effective association algorithm is needed which can manage the inconsistent registrations of identifications.
In this paper, we present an approach for dynamic object identification in heterogeneous sensor networks where two functionally different sensors are incorporated. Visual sensors associate objects and track them using the geometric relationship of multiple cameras [17, 18]. The visual sensor-based tracking system is assisted by identification sensors in identifying the estimated positions of objects. The coverage of identification sensors is assumed by its maximum sensing coverage and the association system applies the simple association strategy for the estimated position from the visual sensor and the identification from the identification sensor. The important issue in heterogeneous sensor networks is to provide the association system with a common reference information fusing heterogeneous data. The visual sensors-based tracking system utilizes the known coverage of the identification sensors to associate the heterogeneous data. The locations of identification sensors are known and they are jointly used with the locations of objects to check the object dynamics of entering and leaving the sensor coverage. The sets of estimated positions and identifications are defined for the coverage of each identification sensor. The association of them is established by checking the temporal change of the sets. In order to solve the association problem with the coverage uncertainty issue, a group and incomplete group associations are introduced. The group and incomplete group associations enable the association system to maintain identification candidates for the corresponding estimated positions until a single association is established. Also, a group association can stabilize the association performance against the inconsistent registration of identifications by an identification sensor. Additional association cases are investigated to increase the association performance by checking the object dynamics. We also identify more association problems with the discrepancy between the actual coverage by the identification sensor and the approximated coverage by the visual sensor and present a coverage adjustment scheme using the object dynamics. Finally, the proposed association method is evaluated with a realistic scenario and is analyzed to show the stability of the proposed method according to degree of the discrepancy between approximated and actual identification sensor coverage, variance of actual identification sensor coverage, and tracking performance.
The remainder of this paper has 4 sections. In Section 2, we present the overview of an application model and problem descriptions. Section 3 explains an association method for multiple objects by a group association and incomplete group association with the consideration of the coverage uncertainty problems. In Section 4, the proposed method is evaluated with a realistic application scenario and is analyzed with nonideal problems such as the discrepancy between approximated and actual identification sensor coverage and variance of actual identification sensor coverage. Finally, the paper is summarized in Section 5.
2. Application Model and Problem Description
2.1. Application Model
The proposed approach can be applied to not only public areas (e.g., schools, hospitals, and shopping malls) but also highly secured areas (e.g., airports, military facilities, and government organizations). As an example of possible scenarios, serious offenders with attached ID tags can be tracked with the proposed method in order to ensure the safety at public places in cities. Also, the surveillance system with the proposed approach can keep tracking passengers in an airplane check-in or military personnel in a special area. It assumes that each object has its own identification such as an RFID tag, fingerprint, and iris. Identification sensors are usually installed at the gates of restricted areas, and a visual sensor tracks objects. For the airport application, the check-in counter can play the role of the ID-sensor. Whenever an object goes across the gates, the registered ID by an identification sensor is associated with the position estimated by a visual sensor. The system continuously watches the surveillance region by checking authorized IDs in the restricted areas.
where represents the number of elements in the set [15, 16]. In other words, for an identification sensor at a time instance, if there is one unassociated (from identification sensor) and one unassociated object position (from visual sensor), the association can simply be made. However, in practical applications, the condition in (3) may not be satisfied.
2.2. Problem Description
The association problems can be nontrivial, especially when RFID-type identification sensors are used. For those types of sensors, as they are based on the reception of the radio frequency signal, which can be easily distorted by the environment, the coverage of the sensor can become time-varying without being known to the visual sensor. Then, the actual coverage of an identification sensor can be different from the approximated coverage by a visual sensor and the condition in (3) may not be satisfied—there are more than one unassociated objects' positions but fewer number of unassociated ID's, or vice versa. Even if the coverage of the identification sensor is not time-varying, there can still be the coverage uncertainty problem, when objects are densely populated near the boundary of the coverage. In order to adapt to the time-varying coverage of the identification sensor, the maximum sensing coverage of the identification sensor can be assumed by the visual sensor.
Violation of the condition in (3) can happen due to the coverage discrepancy between the sampling intervals of two sensors. For example, an registered during one sampling interval of the visual sensor can be associated with multiple estimated positions within the coverage of an identification sensor. An ideal situation for an association is that one and only one is registered during one sampling interval of the visual sensor and one position is newly added and estimated at each sampling time within the coverage of an identification sensor. However, the registration of identifications within the approximated coverage of the visual sensor is not always guaranteed due to the coverage uncertainty. Identifications may not be registered sequentially as multiple objects enter the approximated coverage of the visual sensor. Also, the registration times of identifications may not coincide with the estimation time of the corresponding positions. Then, it is difficult to associate identifications with estimated positions by using only the simple association condition in (3).
The association problems become more difficult when objects with and without identifications coexist. Especially, when there is the coverage uncertainty issue, the association system cannot clearly determine whether an object has an or not. The deterministic association approach by one-to-one assignment may falsely associate identifications with unassociated estimated positions. Moreover, the association system may switch ID's while tracking multiple objects when objects collide with each other. Therefore, the association system requires an effective association algorithm that can recover association failures by managing the coverage uncertainty.
3. Association and Identification with Coverage Uncertainty
3.1. Multiple Objects Association
3.1.1. Association without Coverage Uncertainty
Even when the coverages of the identification sensor and the visual sensor are identical, the association failure, the violation of the condition in (3), can happen mainly due to the two reasons—the simultaneous entrance and the collision. When multiple objects simultaneously enter the coverage of the identification sensor, the condition in (3) is not satisfied, since multiple objects are registered during a single sampling time of the visual sensor and . As investigated in , increasing the sampling time of the visual sensor can alleviate the problem, but it cannot be the fundamental solution to the simultaneous entrance problem. A collision between the objects can lead to a failure in tracking objects since they are too close to be differentiated for position and ID assignments. Although the visual sensor can track multiple objects after the collision, the associations between the objects and the ID's are no longer valid. If the dynamic transition model of objects is known, an identification assignment can be estimated through the tracking. However, the accurate model is not always known to the association system. The existing method shown in [13, 15] waits for a new association until the association-failure objects enter the coverage of a new identification sensor. Although this method can provide an association recovery, all the established associations are lost by the collision.
In other words, for an identification sensor at a time instance, if there are more than one unassociated ID's (from the identification sensor) and the same number of unassociated object positions (from the visual sensor), then a group association can be made.
which is the condition of association, modified from the condition in (3). Although the condition in (6) establishes a single association for a newly added object, such a single association cannot be established for an object in a group association by the condition in (6).
respectively. A group association is divided into single association(s) or other group associations by these conditions.
3.1.2. Effects of Coverage Uncertainty
where each object is registered as an element of the incomplete group association with possible identification candidates.
3.2. Group Association by Temporal Set Maintenance
The column of "With Temporal Set Maintenance" in the table of Figure 4 shows how the sets of estimated positions and identifications vary using the temporal set maintenance. are associated with as a group at . Since is associated with at the next sampling time, and are removed in and .
3.3. Association Stability in Mismatched Model
Association performance is also influenced by the discrepancy between the approximated coverage and the actual coverage. When the approximated coverage is greater than the actual coverage, positions of objects with nonregistered identifications can be estimated within the approximated coverage. Then, a group or incomplete group association increases by the condition in (5) or (11). This can frequently occur when objects move around the boundary of coverage of an identification sensor. Moreover, the effect of the smaller approximated coverage than the actual coverage is similar to the effect of the larger approximated coverage than actual coverage. Since the number of registered identifications is different from the number of estimated positions within the approximated coverage, this may increase group or incomplete group associations. However, the estimated positions of objects are eventually identified when single associations are established. While the inaccurate coverage model may delay the establishment of single associations, the number of single associations eventually increases by the object dynamics.
The irregular sensor coverage causes a false association with a noncorresponding identification when objects move around the boundary of the modeled coverage. For example, is not estimated but is estimated inside the coverage of the visual sensor. Also, on the other hand, only is registered inside the coverage. Then, can be falsely associated with by the condition in (3). Since a single association is established, the association system cannot confirm the false association immediately. However, the association system can cope with false associations using two approaches. One is a passive approach that uses the property of a group association. If objects in relation with a false association collide inside or outside the coverage, a false association naturally becomes a group or incomplete group association. The other approach is to confirm the association by checking whether duplicated identifications exist in the association system. If the false association is confirmed, the falsely associated position changes to an unassociated position. Therefore, false associations are eventually resolved by a group association or checking the identification with duplicate registrations at the coverage of different identification sensors.
3.4. Coverage Adjustment Scheme
At the initial state, the approximated radius of an identification sensor is set as a physical variable in the system. Since the radius is used to determine whether objects enter or leave the coverage of an identification sensor, it needs to be accurately estimated for the improved association performance. However, the association performance is also affected by the simultaneous entrance and the collision. These phenomena frequently occur where objects are densely populated. The association performance is not improved proportionally to the degree of the accurate estimation of the radius but the time to stabilize the association performance is inversely proportional to the degree of the accurate estimation of the radius. In order to adjust the initial radius of an identification sensor, we utilize the object dynamics of entering and leaving the coverage of an identification sensor.
The basic idea of the coverage adjustment scheme is to compare the number of estimated positions with the number of registered identifications within the coverage of an identification sensor. If the approximated radius of an identification sensor is accurate enough, the number of the estimated positions is mostly equivalent to the number of the registered identifications. Otherwise, it means that the approximated coverage differs from the actual coverage. The radius of an identification sensor is adjusted by checking the difference between them. In some cases, the system needs to check the farthest or closet estimated position from the center of an identification sensor. For example, when the number of the estimated positions is equivalent to the number of the registered identifications, the coverage of an identification should be adjust to the farthest estimated position. Then, the problem in the coverage adjustment scheme is to determine how degree the radius is adjusted by at each sampling time. Since the coverage of an identification sensor can vary temporally, the large change of the radius may cause a reverse effect and the association performance may degenerate. Thus, we use the average speed of tracked objects measured by the association system as the degree of the radius adjustment to be unsusceptible to the object dynamics.
The temporal change of sets of positions and identifications is utilized to adjust the initial coverage, while the coverage of an identification sensor is assumed to slowly vary. Since an association can be established at every sampling time , the approximated coverage of the visual sensor is also adjusted by the change of a radius at time . The average speed of tracked objects, measured by the association system, can be used to determine , since the registration is related to the object dynamics. Define as the adjusted radius between radii and for an identification sensor at time . The set of estimated positions within is denoted by and the set of registered identifications within is denoted by .
Moreover, the goal of the coverage adjustment is to prevent a significant discrepancy between the initial approximated coverage and the actual coverage as conserving current association information of objects. Hence, the adjusted radius should not violate the positions of objects having association information.
3.5. Association Algorithm
Algorithm 1 summarizes the conditions for multiple objects association with the coverage uncertainty. If is in a group association, possible associable identifications are limited to . Objects in incomplete group associations also have identification candidates. Therefore, the possibility is increased that an estimated position will be uniquely paired with its identification. After the association system finishes checking the association conditions for each object in the coverage, it determines whether remaining objects are a group association or incomplete group association. Then, the association system removes associated identifications and estimated positions in all the sets of group associations and incomplete group associations. Single associations can also be established in this process if the number of elements in group associations is two.
Algorithm 1: The proposed association algorithm.
if the entering condition in the group association is satisfied then
Remove them in group associations
if the condition in (12) then
Remove them in incomplete group associations
if the leaving condition in the group association is satisfied then
Remove them in group associations
if The remaining objects satisfy the condition in (3) then
Register them as a single association
else if The remaining objects satisfy the condition in (5) then
Register them as a group association;
Update candidate identifications of objects in group associations
until Association system stops;
4.1. Simulation Setup
The association performance for the identification is compared against the simple association rule. In the simple association rule, a position and an identification of a single object are associated when each signal exists in the sensor coverage [15, 16]. It is assumed that an object is localized and tracked by multiple cameras without failure. We use a simple object tracking algorithm since object models are not known to the association system.
4.2. Effect of Modeled Region Accuracy
4.3. Effect of Region Overlapping
4.4. Association Performance
4.5. Robustness against False Detection and False Tracking
The proposed method has robustness against two nonideal phenomena possibly caused by visual sensors. One case involves falsely detected objects according to the classification capability of detection algorithms. When objects are falsely detected inside the region, this leads to a group or incomplete group association. However, this is eventually resolved when the true position of an object is associated with its identification. The other issue is false tracking, which usually occurs when objects collide with each other. Identifications can be switched depending on the tracking capability. In this case, the proposed method utilizes a group association. Then, their identifications are also eventually found by the object dynamics. However, the system cannot clearly determine whether an object has identification information or not because of the coverage uncertainty.
The data association and management scheme is proposed to complement two different types of signals in heterogeneous sensor environment. Visual sensors estimate and track positions of objects, and identification sensors register identifications of objects. The uncertain sensing coverage of an identification sensor is approximately modeled for a simple association strategy. The location information of identification sensors and objects is utilized to resolve the association problems with the object dynamics. We also present a coverage adjustment method using the object dynamics around the coverage of the identification sensor. The simulation-based analysis shows that the association performance is improved as the time elapses even with realistic problems such as error of estimated positions, a discrepancy between approximated and actual identification sensor overage, variance of actual identification sensor coverage, and imperfect tracking. To improve the association performance, the identification sensors should be installed at the places where objects dynamically move around for a fast association establishment or recovery, as the associations are established by the object dynamics of crossing the coverage of identification sensors.
This paper was supported in part by the Mid-career Researcher Program of Korea Science and Engineering Foundation (KOSEF) Grant funded by the Korea government (MEST) (no.2010-0000487) and the National Research Foundation of Korea (NRF) Grant funded by the Korea government (MEST) (no. 2010-0027499). Part of this paper was presented at 6th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009) Genoa, Italy, September 2009.
- Strobel N, Spors S, Rabenstein R: Joint audio-video object localization and tracking: a presentation general methodology. IEEE Signal Processing Magazine 2001, 18(1):22-31. 10.1109/79.911196View ArticleGoogle Scholar
- Zhou H, Taj M, Cavallaro A: Target detection and tracking with heterogeneous sensors. IEEE Journal on Selected Topics in Signal Processing 2008, 2(4):503-513.View ArticleGoogle Scholar
- Hu W, Tan T, Wang L, Maybank S: A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man and Cybernetics C 2004, 34(3):334-352. 10.1109/TSMCC.2004.829274View ArticleGoogle Scholar
- Zhao W, Chellappa R, Phillips PJ, Rosenfeld A: Face recognition: a literature survey. ACM Computing Surveys 2003, 35(4):399-458. 10.1145/954339.954342View ArticleGoogle Scholar
- Yang MH, Kriegman DJ, Ahuja N: Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002, 24(1):34-58. 10.1109/34.982883View ArticleGoogle Scholar
- Brunelli R, Poggio T: Face recognition: features versus templates. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993, 15(10):1042-1052. 10.1109/34.254061View ArticleGoogle Scholar
- Grudin MA: On internal representations in face recognition systems. Pattern Recognition 2000, 33(7):1161-1177. 10.1016/S0031-3203(99)00104-1View ArticleGoogle Scholar
- Garcia C, Tziritas G: Face detection using quantized skin color regions merging and wavelet packet analysis. IEEE Transactions on Multimedia 1999, 1(3):264-277. 10.1109/6046.784465View ArticleGoogle Scholar
- Hsu RL, Abdel-Mottaleb M, Jain AK: Face detection in color images. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002, 24(5):696-706. 10.1109/34.1000242View ArticleGoogle Scholar
- Römer K, Schoch T, Mattern F, Dübendorfer T: Smart identification frameworks for ubiquitous computing applications. Wireless Networks 2004, 10(6):689-700.View ArticleGoogle Scholar
- Roberts CM: Radio frequency identification (RFID). Computers and Security 2006, 25(1):18-26. 10.1016/j.cose.2005.12.003View ArticleGoogle Scholar
- Roussos G, Kostakos V: RFID in pervasive computing: state-of-the-art and outlook. Pervasive and Mobile Computing 2009, 5(1):110-131. 10.1016/j.pmcj.2008.11.004View ArticleGoogle Scholar
- Schulz D, Fox D, Hightower J: People tracking with anonymous and ID-sensors using Rao-Blackwellised particle filters. Proceedings of the International Joint Conference on Artificial Intelligence, August 2003Google Scholar
- Shin J, Kumar R, Mohapatra D, Ramachandran U, Ammar M: ASAP: a camera sensor network for situation awareness. Proceedings of the 11th International Conference on Principles of Distributed Systems (OPODIS '07), December 2007 31-47.Google Scholar
- Cho SH, Lee J, Deng X, Hong S, Cho W-D: Passive sensor based dynamic object association method in wireless sensor networks. Proceedings of the 50th Midwest Symposium on Circuits and Systems (MWSCAS '07), August 2007 1221-1224.Google Scholar
- Cho SH, Lee J, Hong S: Passive sensor based dynamic object association with particle filtering. Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance (AVSS '07), September 2007 206-211.Google Scholar
- Kyong Y, Cho SH, Hong S: Local initiation method for multiple object association in surveillance environment with multiple cameras. Proceedings of the IEEE 5th International Conference on Advanced Video and Signal Based Surveillance (AVSS '08), September 2008 348-355.Google Scholar
- Park K-S, Lee J, Stanaćević M, Hong S, Cho W-D: Iterative object localization algorithm using visual images with a reference coordinate. EURASIP Journal on Image and Video Processing 2008., 2008:Google Scholar
- Yoon SM, Kim H: Real-time multiple people detection using skin color, motion and appearance information. Proceedings of the 13th IEEE International Workshop on Robot and Human Interactive Communication, September 2004 331-334.Google Scholar
- Eng HL, Wang J, Kam AH, Yau WY: A bayesian framework for robust human detection and occlusion handling using human shape model. Proceedings of the 17th International Conference on Pattern Recognition (ICPR '04), August 2004 257-260.Google Scholar
- Elzein H, Lakshmanan S, Watta P: A motion and shape-based pedestrian detection algorithm. Proceedings of the IEEE on Intelligent Vehicles Symposium, June 2003 500-504.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.