Comparison of Spectral-Only and Spectral/Spatial Face Recognition for Personal Identity Veriﬁcation

Face recognition based on spatial features has been widely used for personal identity veriﬁcation for security-related applications. Recently, near-infrared spectral reﬂectance properties of local facial regions have been shown to be su ﬃ cient discriminants for accurate face recognition. In this paper, we compare the performance of the spectral method with face recognition using the eigenface method on single-band images extracted from the same hyperspectral image set. We also consider methods that use multiple original and PCA-transformed bands. Lastly, an innovative spectral eigenface method which uses both spatial and spectral features is proposed to improve the quality of the spectral features and to reduce the expense of the computation. The algorithms are compared using a consistent framework.


Introduction
Automatic personal identity authentication is an important problem in security and surveillance applications, where physical or logical access to locations, documents, and services must be restricted to authorized persons.Passwords or personal identification numbers (PINs) are often assigned to individuals for authentication.However, the password or PIN is vulnerable to unauthorized exploitation and can be forgotten.Biometrics, on the other hand, use personal intrinsic characteristics which are harder to compromise and more convenient to use.Consequently, the use of biometrics has been gaining acceptance for various applications.Many different sensing modalities have been developed to verify personal identities.Fingerprints are a widely used biometric.Iris recognition is an emerging technique for personal identification which is an active area of research.There are also studies to use voice and gait as primary or auxiliary means to verify personal identities.
Face recognition has been studied for many years for human identification and personal identity authentication and is increasingly used for its convenience and noncontact measurements.Most modern face recognition systems are based on the geometric characteristics of human faces in an image [1][2][3][4].Accurate verification and identification performance has been demonstrated for these algorithms based on mug shot type photographic databases of thousands of human subjects under controlled environments [5,6].Various 3D face models [7,8] and illumination models [9,10] have been studied for pose and illuminationinvariant face recognition.In addition to methods based on gray-scale and color face images over the visible spectrum, thermal infrared face images [11,12] and hyperspectral face images [13] have also been used for face recognition experiments.An evaluation of different face recognition algorithms using a common dataset has been of general interest.This approach provides a solid basis to draw conclusions on the performance of different methods.The Face Recognition Technology (FERET) program [5] and the Face Recognition Vendor Test (FRVT) [6] are two programs which provided independent government evaluations for various face recognition algorithms and commercially available face recognition systems.
Most biometric methods, including face recognition methods, are subject to possible false acceptance or rejection.Although biometric information is difficult to duplicate, these methods are not immune to forgery, or so-called spoofing.This is a concern for automatic personal identity authentication since intruders can use artificial materials or objects to gain unauthorized access.There are reports showing that fingerprint sensor devices have been deceived by Gummi fingers in Japan [14] and fake latex fingerprints in Germany [15].Face and iris recognition systems can also be compromised since they use external observables [16].To counter this vulnerability, many biometric systems employ a liveness detection function to foil attempts at biometric forgery [17,18].To improve system accuracy, there is strong interest in research to combine multiple biometric characteristics for multimodal personal identity authentication [19,20].Since hyperspectral sensors capture spectral and spatial information they provide the potential for improved personal identity verification.
Methods that have been developed consider the use of representations for visible wavelength color images for face recognition [21,22] as well as the combination of color and 3D information [23].In this work, we examine the use of combined spectral/spatial information for face recognition over the near-infrared (NIR) spectral range.We show that the use of spatial information can be used to improve on the performance of spectral-only methods [13].We also use a large NIR hyperspectral dataset to show that the choice of spectral band over the NIR does not have a significant effect on the performance of single-band eigenface methods.On the other hand, we show that band selection does have a significant effect on the performance of multiband methods.In this paper we develop a new representation called the spectral-face which preserves both high-spectral and highspatial resolution.We show that the spectral eigenface representation outperforms single-band eigenface methods and has performance that is comparable to multiband eigenface methods but at a lower computational cost.

Face Recognition in Single-Band Images
A hyperspectral image provides spectral information, normally in radiance or reflectance, at each pixel.Thus, there is a vector of values for each pixel corresponding to different wavelengths within the sensor spectral range.The reflectance spectrum of a material remains constant in different images while different materials exhibit distinctive reflectance properties due to different absorbing and scattering characteristics as a function of wavelength.In the spatial domain, there are several gray-scale images that represent the hyperspectral imager responses of all pixels for a single spectral band.In a previous study [24], seven hyperspectral face images were collected for each of 200 human subjects.These images have a spatial resolution of 468 × 494 and 31 bands with band centers separated by 0.01 μm over the near-infrared (0.7 μm-1.0 μm). Figure 1 shows calibrated hyperspectral face images of two subjects at seven selected bands which are separated by 0.06 μm over 0.7 μm-1.0 μm.We see that the ratios of pixel values on skin or hair between different bands are dissimilar for the two subjects.That is, they have unique hyperspectral signatures for each tissue type.Based on these spectral signatures, a Mahalanobis distance-based method was applied for face recognition tests and accurate face  recognition rates were achieved.However, the performance was not compared with classic face recognition methods using the same dataset.
The CSU Face Identification Evaluation System [25] provides a standard set of well-known algorithms and established experimental protocols for evaluating face recognition algorithms.We selected the Principal Components Analysis (PCA) Eigenfaces [26] algorithm and used cumulative match scores as in the FERET study [5] for performance comparisons.To prepare for the face recognition tests, a gray-scale image was extracted for each of the 31 bands from a hyperspectral image.The coordinates of both eyes were manually positioned before processing by the CSU evaluation programs.In the CSU evaluation system all images were transformed and normalized so that they have a fixed spatial resolution of 130 × 150 pixels and the eye coordinates are the same.Masks were used to void nonfacial features.Histogram equalization was also performed on all images before the face recognition tests were conducted.For each of the 200 human subjects, there are three frontview images with the first two (fg and fa) having neutral expression and the other (fb) having a smile.All 600 images were used to generate the eigenfaces.Figure 2 shows one single-band image before and after the normalization, and the first 10 eigenfaces for the dataset.The number of eigenfaces used for face recognition was determined by selecting the set of most significant eigenfaces which account for 90% of the total energy.
Given the wth band of hyperspectral images U and V , the Mahalanobis Cosine distance [27] is used to measure the similarity of the two images.Let u w,i be the projection of the wth band of U onto the ith eigenface and let σ w,i be the standard deviation of the projections from all of the wth band images onto the ith eigenface.projection of V w .The Mahalanobis Cosine distance between U and V for the wth band is defined by which is the negative of the cosine between the two vectors.
For the 200 subjects, the fg images were grouped in the gallery set and the fa and fb images were used as probes [5].The experiments follow the closed universe model where the subject in every image in the probe set is included in the gallery.For each probe image, the Mahalanobis Cosine distance between the probe and all gallery images is computed.If the correct match is included in the group of gallery images with the N smallest distances, we say that the probe is correctly matched in the top N.The cumulative match score for a given N is defined as the fraction of correct matches in the top N from all probes.The cumulative match score for N = 1 is called the recognition rate.Figure 3 plots the cumulative match scores for N = 1, 5, and 10 respectively.Band 1 refers to the image acquired at 700 nm and band 31 refers to the image acquired at 1000 nm.We see that all bands provide high recognition rates, with more than 96% of the probes correctly identified for N = 1 and over 99% for N = 10.It is important to consider the statistical significance of the results.For this purpose, we model the fraction of the probes that are correctly identified by a binomial distribution with a mean given by the measured identification rate p.The variance σ 2 of p is given by 400p(1 − p) where 400 is the number of probes [28].For an identification rate of 0.97 we have σ = 3.4 which corresponds to a standard deviation in the identification rate of 0.009 and for an identification rate of 0.99 we have σ = 1.99 which corresponds to a standard deviation in the identification rate of 0.005.Thus, for each of the three curves plotted in Figure 3 the variation in performance across bands is not statistically  significant.Figure 4 compares the cumulative match scores using the spectral signature method [13] and the single-band eigenface method using the most effective band.We see that the spectral signature method performs well but somewhat worse than the best single-band method for matches with N less than 8.For N = 1, a recognition rate of 0.92 corresponds to a standard deviation in the recognition rate of 0.014 which indicates that the difference between the two methods in Figure 4 is statistically significant.The advantage of the spectral methods is pose invariance which was discussed in a previous work [13] but which is not considered in this paper.

Face Recognition in Multiband Images
We have shown that both spatial and spectral features in hyperspectral face images provide useful discriminants for recognition.Thus, we can consider the extent of performance improvements when both features are utilized.We define a distance between images U and V using where the index w takes values over a group of W-selected bands that are not necessarily contiguous.Note that the additive 1 is to ensure a nonnegative value before the square.
Redundancy in a hyperspectral image can be reduced by a Principal Component Transformation (PCT) [29].For a hyperspectral image U = (U 1 , U 2 , . . ., U W ), the PCT generates U = (U 1 , U 2 , . . ., U W ), where U i = j ε i j U j .The principal components U 1 , U 2 , . . ., U W are orthogonal to each other and sorted in order of decreasing modeled variance.Figure 5 shows a single-band image at 700 nm and the first five principal components that are extracted from the corresponding hyperspectral image.We see that the first principal component image resembles the single-band image while the second and third component images highlight features of the lips and eyes.We also see that there are few visible features remaining in the fourth and fifth principal components.Figure 6 plots the recognition rates for different multiband eigenface methods.First we selected the bands in order of increasing center wavelength and performed eigenface recognition tests for the first one band, two bands and up to 31 bands, respectively.We also sorted all 31 bands in descending order of recognition rate and performed the same procedure for the face recognition tests.From Figure 6 we see that both methods reach a maximum recognition rate of 98% when using multiple bands.However, when the number of bands is less than 16, the multiband method performs better if the bands are sorted in advance from the highest recognition rate to the lowest.We also used the leading principal components for multiband recognition.We see in Figure 6 that over 99% of the probes were correctly recognized when using the first three principal bands.Increasing the number of principal bands beyond 3 causes performance degradation.The original-order algorithm in Figure 6 achieves a recognition rate of approximately 0.965 for less than ten bands which corresponds to a standard deviation in recognition rate of 0.009.Thus, the performance difference between this method and the PCT-based method is significant between 3 and 9 bands.Note that the PCT was performed on each hyperspectral image individually with  different sets of ε i j .The PCT can also be implemented using the same coefficients for faster computation.Figure 7 also compares the recognition performance of the three multiband methods discussed in the previous paragraph where each algorithm uses only the first three bands.It is interesting that sorting the bands according to performance improves the recognition rate for N = 1 but worsens the performance somewhat for larger values of N. In either case, the multiband method based on the PCT has the best performance for N < 7 and is equivalent to the originalorder method for larger values of N.

Face Recognition Using Spectral Eigenfaces
We showed in Section 3 that multiband eigenface methods can improve face recognition rates.In these algorithms, the multiple bands are processed independently.A more general approach is to consider the full spectral/spatial structure of the data.One way to do this is to apply the eigenface method to large composite images that are generated by concatenating the 31 single-band images.This approach, however, will significantly increase the computational cost of the process.An alternative is to subsample each band of the hyperspectral image before concatenation into the large composite image.For example, Figure 8 shows a 31-band image after subsampling so that the total number of pixels is equivalent to the number of pixels in a 130×150 pixel singleband image.We see that significant spatial detail is lost due to the subsampling.
A new representation, called spectral-face, is proposed to preserve both spectral and spatial properties.The spectralface has the same spatial resolution as a single-band image so the spatial features are largely preserved.In the spectral domain, the pixel values in the spectral-face are extracted sequentially from band 1 to band 31 then from band 1 again.For example, the value of pixel i in spectral-face equals the value of pixel i in band w where w is the remainder of i divided by 31.Figure 9 shows an original single-band image together with the normalized spectral-face image in the left column.Spectral-face has improved spatial detail as compared with Figure 8.The pattern on the face in Figure 9 demonstrates the variation in the spectral domain.With the spectral-face images, the same eigenface technique is applied for face recognition.The first 10 spectral eigenfaces are shown on the right side of Figure 9.It is interesting to observe that the eighth spectral eigenface highlights the teeth feature in smiling faces.
The spectral eigenface method was applied to the same dataset as the single-band and multiband methods.The cumulative match scores for N = 1 to 20 are shown in Figure 10.The best of the single-band methods, which corresponds to band 19 (880 nm), is included for performance comparison with the spectral eigenface method.We see that the spectral eigenface method has better performance for all ranks.The best of the multiband methods, which combines the first three principal bands, is also considered.The multiband method performs better than the spectral eigenface method for small values of the rank, but performs worse for larger values of the rank.For this case, an identification rate of 0.99 corresponds to a standard deviation in identification rate of 0.005.Thus, the two multiple-band methods have a statistically significant advantage over the single-band eigenface method for ranks between 3 and 10.Note that the multiple principal band method requires more computation than the spectral eigenface method.

Conclusion
Multimodal personal identity authentication systems have gained popularity. Hyperspectral imaging systems capture both spectral and spatial information.The previous work [24] has shown that spectral signatures are powerful discriminants for face recognition in hyperspectral images.In this work, various methods that utilize spectral and/or spatial features were evaluated using a hyperspectral face image dataset.The single-band eigenface method uses spatial features exclusively and performed better than the pure spectral  method.However, the computational requirements increase significantly for eigenface generation and projection.The recognition rate was further improved by using multiband eigenface methods which require more computation.The best performance was achieved with the highest computational complexity by using principal component bands.The spectral eigenface method transforms a multiband hyperspectral image to a spectral-face image which samples from all of the bands while preserving spatial resolution.We showed that this method performs as well as the PCT-based multiband method but with a much lower computational requirement.
campus.The authors would like to thank J. Stuart Nelson and Montana Compton for their valuable assistance in the process of IRB approval and human subject recruitment.

Figure 1 :
Figure 1: Selected single-band images of two subjects.

Figure 2 :
Figure 2: Example of eigenfaces in one single-band.
The Mahalanobis projection of U w is M w = (m w,1 , m w,2 , . . ., m w,I ) where m w,i = u w,i /σ w,i .Let N w be the similarly computed Mahalanobis EURASIP Journal on Advances in Signal Processing 3

Figure 3 :
Figure 3: Cumulative match scores of single-band images at different wavelengths.

Figure 4 :
Figure 4: Cumulative match scores of spectral signature method and the best single-band eigenface method.

Figure 5 :Figure 6 :
Figure 5: Five principal band images of one subject after PCT.

Figure 8 :
Figure 8: A sample image composed from 31 bands with lowspatial resolution.

Figure 9 :
Figure 9: One sample spectral-face and the first 10 spectral eigenfaces.

Figure 10 :
Figure 10: Comparison of spectral eigenface method with singleband and multiband methods.