- Research Article
- Open Access
Subspace-Based Holistic Registration for Low-Resolution Facial Images
© B. J. Boom et al. 2010
- Received: 9 December 2009
- Accepted: 14 July 2010
- Published: 29 July 2010
Subspace-based holistic registration is introduced as an alternative to landmark-based face registration, which has a poor performance on low-resolution images, as obtained in camera surveillance applications. The proposed registration method finds the alignment by maximizing the similarity score between a probe and a gallery image. We use a novel probabilistic framework for both user-independent as well as user-specific face registration. The similarity is calculated using the probability that the face image is correctly aligned in a face subspace, but additionally we take the probability into account that the face is misaligned based on the residual error in the dimensions perpendicular to the face subspace. We perform extensive experiments on the FRGCv2 database to evaluate the impact that the face registration methods have on face recognition. Subspace-based holistic registration on low-resolution images can improve face recognition in comparison with landmark-based registration on high-resolution images. The performance of the tested face recognition methods after subspace-based holistic registration on a low-resolution version of the FRGC database is similar to that after manual registration.
- Face Recognition
- Facial Image
- Registration Method
- Edge Image
- Gallery Image
Face recognition in the context of camera surveillance is still a challenging problem. For reliable face recognition, it is crucial that an acquired facial image is registered to a reference coordinate system. Most conventional registration methods are based on landmarks. To locate these landmarks accurately, high-resolution images are needed. For those methods, it is problematic to register low resolution facial images as obtained in video surveillance. In the Face Recognition Vendor Test , low-resolution face images are defined to contain an interocular distance of 75 pixels, we used even lower resolutions with interocular distances of 50 pixels and lower. High-resolution face images have an interocular distance of more than 100 pixels. Face registration on low-resolution images is in these cases often omitted and the region found by the face detection is directly used for face recognition [2, 3]. In our opinion, accurate face registration can contribute to better recognition performance on low-resolution images. Therefore, we developed a Subspace-based Holistic Registration (SHR) method, which uses the entire face region to correct for translation, rotation, and scale transformation of the face, which enables us to accurately register low-resolution facial images. The face registration is performed after a frontal face detector, which detects a face at a certain scale and rotation variations, limiting the search for the final registration parameters.
As already pointed out above, registration methods can be divided into two categories: landmark-based registration, using landmarks to register the face image, and holistic registration, using the entire image for registration. Of the latter only a few methods have been reported.
In the first category, the object detection method of Viola and Jones , originally proposed for face detection, is a popular approach to locating landmarks [5–7]. The advantages of this method are that it is fast and robust in comparison with other landmark methods. Many papers report good results especially in uncontrolled scenarios. However, occasionally landmarks are not found by this method. In , a probabilistic approach using Principal Component Analysis (PCA) is used to locate the landmarks. Subspace methods for facial feature detection are also used in [9–11]. Some landmarking techniques are not only based on texture, but also use geometric relations between landmarks, for instance [12–15]. These methods usually require more landmarks and high-resolution facial images. A well-known example of such a method is Elastic Bunch Graphs . Elastic Bunch Graphs are used to determine the relation between different landmarks. The relation between the landmarks and the scores of Gabor Jets are combined to register and recognize the face. Active Shape Models  and Active Appearance Models  can also be used to perform a fine registration of a face, by using both texture and the relation between the landmarks. Both methods, however, need a good initialization to find an accurate registration, which can be provided by, for instance, the Viola and Jones landmark finding method.
In the second category of registration, there are correlation-based registration methods that are invariant to translation. The MACE filter originally described in  and used in face recognition in [19, 20], is invariant for translations. In , a face registration method using super resolution is described that performs correlation to compare the original image with a reconstructed image obtained using super resolution, correcting for translation and scale variations. The method described in [22, 23] is a correlation-based method that finds a rigid transformation to align the facial images, which is done using robust correlation to a user template.
Another way of evaluating the registration quality is by using the similarity score determined by a face recognition algorithm. In , the manually labelled eye coordinates are used as a starting point from which the eye coordinates are varied to obtain different registrations. The registration that resulted in the best similarity score is selected. This experiment was performed using several different face recognition algorithms. In , we performed a similar experiment and in addition showed that small changes in the registration parameters can have a huge effect on the similarity scores of face recognition algorithms. In [26, 27], we proposed a matching score-based face registration approach, which searches for the optimal alignment by maximizing the similarity score of several holistic face recognition algorithms, for example, PCA Mahalanobis distance. In , the PCA Mahalanobis distance is used to find the registration parameters for low-resolution images using a different search strategy as in , where the focus of the paper is face hallucination. In , this face registration method is extended especially for the purpose of face hallucination. We performed no experiment using face hallucination, because our focus is on face registration and its effect on the recognition. In this paper, we extended the work in [26, 27], by developing Subspace-based Holistic Registration (SHR) method. The novelty of this method is that we use a probabilistic framework designed to evaluate the registration of faces, instead of maximizing the score of a face recognition method, which might not be suited for comparing unregistered face images.
2.1. Subspace-Based Holistic Registration
where are the eigenvalues in and which is the average eigenvalue in . This distance measure consist of two parts, the first is called "distance-in-feature-space" (DIFS) and the second is called "distance-from-feature-space" (DFFS). In our experiments, we compare the results of using only DIFS for face registration, which is used in [27, 28], and using both DIFS and DFFS (see Section 4.1). We show that using both distances result in a better performance than using DIFS.
Two important issues in the evaluation function are the model and the features. The model can be either user independent as explained in the previous section or user specific. This we will discuss in the first paragraph below. As features, we propose edge images, instead of grey level images, which reduce the number of local minima in the evaluation. This will be explained in the second paragraph.
2.2.1. Evaluation to a User Specific Face Model
Instead of registration to a mean face model, which may differ substantially from individual faces, registration to a user-specific model, if available may improve registration results. For user-specific face registration, we need a user template to register a probe image. For face identification, user-specific registration has the drawback that we have to register the probe to every user template in the database.
2.2.2. Using Edge Images to Avoid Local Minima
The default features used in this paper are the "edge images", and a comparison between the features is performed in Section 4.1.
This allows us to obtain an aligned image by backward mapping and interpolation. Most landmark-based methods also perform this transformation based on the found landmarks in order to obtain a registered face image .
2.4. Search Methods
In (1), we have to maximize the similarity score to find the best alignment parameters . Ideally, an iterative search method should be able to find the optimal solution using a small number of evaluations, making it possible to register the probe image almost real time. The search method also has to be robust against local minima. Confirmed by our observations, we assume reasonably smooth search landscapes. We applied two different search methods the first is the downhill simplex method  that we also used in [26, 27], and the second is a gradient-based method.
2.4.1. Downhill Simplex Search Method
where is the maximum expected offset for a single registration parameter in positive or negative direction, where we use the offset which gives the best similarity score. The downhill simplex methods is however able to find optimal registration parameters that lay outside the maximum expected offsets. This search method maximizes the similarity function by replacing those registration parameters in the simplex that gives the worst similarity score by a better set using some simple heuristics.
2.4.2. Gradient-Based Search Method
In the appendix, it is shown how this equation is solved and how updated parameters are obtained analytically. This procedure is repeated until convergence has been reached.
Comparison with earlier versions of SHR . These experiments are included to illustrate the positive effect of the new evaluation criteria given in (6) and of the features discussed in Section 2.2.2;
Comparison with landmark-based registration based on automatically detected landmarks as well as on manual landmarks;
Comparison between user-independent and user-specific registration;
Comparison between two search methods (Section 2.4) in both performance and computation time;
Comparison of SHR performed on lower resolutions.
3.1. Experimental Setup
3.1.1. Face Database
The verification rates at FAR 0.1% of several Face Recognition Methods which allow us to compare the registration methods, these verification rates are achieved using manually registered images.
3.1.2. Face Detection
Face registration depends on the input of a Face Detection method. We used the OpenCV implementation [35, 36] of the Viola and Jones algorithm  to find the faces. We used the pretrained model called "haarcascade_frontalface_default.xml". In order to avoid misdetections, we included some simple heuristics based on the manually labelled landmarks to determine if the face regions were correctly found. All landmarks have to be inside the face region and the width and height of this region is less than four times the distance between the eyes. Facial images in which the face is not correctly found are removed from all experiments.
3.1.3. Low Resolution
SHR is developed for low-resolution images. Because there are no large low-resolution face databases, we used the FRGCv2 database and created low-resolution facial images by low-pass filtering and subsequent downsampling. Using low-resolution facial images makes the comparison of the performance of our face recognition methods with the state of the art difficult, because these are primarily focussed on high-resolution facial images. Also, landmark-based registration methods work poorly on these resolutions. For this reason, we performed the landmark finding on high-resolutions images, thus given them an advantage over SHR.
3.1.4. Face Recognition
3.1.5. Landmark Methods for Comparison
We compared SHR to two landmark registration methods. The first method is the Viola and Jones detector  trained to find facial landmarks. The second method is called MLLL (Most Likely Landmark Locator) , which finds the landmarks by maximizing the likelihood ratio using PCA and LDA. This algorithm is run in combination with BILBO, which is a subspace-based method to correct for outliers. We have trained both methods on the FRGCv2 database and evaluated them using high-resolution images. Both the Viola-Jones and MLLL + BILBO find four landmarks (eyes, nose, and mouth). Based on the found landmarks, we calculate the Procrustes transformation to align the images.
3.2. Experimental Settings
4.1. Comparison with Earlier Work
4.2. Subspace-Based Holistic Registration versus Landmark-Based Face Registration
Verification rate at and in parenthesis the relative contribution that automatic registration has in comparison with manual registration on FRGC [4, Experiment ], comparing all registration methods using all face classifiers. The best automatic registration is achieved using user-independent SHR using low-resolutions, this often performs even better than manual registration.
Viola-Jones (high resolution)
MLLL + BILBO (high resolution)
57.3% ( 8.9%)
44.5% ( 5.8%)
34.0% ( 3.4%)
73.2% ( 13.8%)
85.2% ( 1.7%)
57.4% ( 10.8%)
68.2% ( 0.0%)
39.7% ( 9.3%)
86.7% ( 10.1%)
94.0% ( 2.8%)
95.9% ( 0.9%)
76.9% ( 13.9%)
86.9% ( 4.7%)
65.5% ( 14.8%)
77.2% ( 3.1%)
86.5% ( 8.4%)
93.5% ( 1.4%)
94.1% ( 0.8%)
78.3% ( 10.5%)
87.1% ( 1.7%)
87.9% ( 1.0%)
69.8% ( 11.1%)
78.9% ( 2.0%)
80.1% ( 0.8%)
4.3. User Independent versus User Specific
4.4. Comparing Search Algorithms
4.5. Lower Resolutions
In video surveillance, the resolution of the facial images is often below the interocular distance of 50 pixels used in previous section. To simulate this, we downsampled the images even more. In this section, we ran experiments using several lower resolutions to test the performance of SHR. After finding the alignment parameters for these resolutions, we use the alignment to register the facial images using an interocular distance of 50 pixels. This allows us to show the effects of low-resolution on the registration, while ignoring the effects of low-resolution on the face recognition.
We presented a novel subspace-base holistic registration (SHR) method, which is developed to perform registration on low-resolution face images. In contrast to most landmark-based registration methods, which can only perform accurate registration on high resolutions. SHR is able to use a user-independent face model or a user-specific face model to register face images. For the user-specific registration, we defined two scenarios to register the gallery images. We show that by using edges as features for the registration, we obtain better results than using the grey levels of the image. The search for the best registration parameters is iterative, and we proposed two search methods, namely, the downhill simplex method and a gradient-based method.
To evaluate the face registration, we measured the effects it has on the results of face recognition. We used the FRGCv2 database to perform our face registration experiments. We compared SHR with two landmark-based registration methods, working on high resolution facial images. Nevertheless, the recognition results of SHR were better than those of the landmark-based methods. User-independent SHR gives a similar performance in face recognition results than registration with manually labelled landmarks. User-specific SHR performs better than the user-independent SHR and manual registration. One of the advantages over the landmark-based methods is that SHR is able to register low-resolution face images with an interocular distance as low as 25 pixels. The results at this resolution make SHR suitable for use in video surveillance.
- Phillips JP, Scruggs TW, Otoole AJ, et al.: FRVT 2006 and ice 2006 large-scale results. National Institute of Standards and Technology; March 2007.Google Scholar
- Acosta E, Torres L, Albiol A, Delp E: An automatic face detection and recognition system for video indexing applications. Proceedings of the IEEE International Conference on Acoustic, Speech, and Signal Processing (ICASSP '02), May 2002 3644-3647.Google Scholar
- Balcan M, Blum A, Choi PP, et al.: Person identification in webcam images: an application of semi-supervised learning. Proceedings of the International Conference on Machine Learning Workshop on Learning from Partially Classified Training Data, 2005 1-9.Google Scholar
- Viola PA, Jones MJ: Rapid object detection using a boosted cascade of simple features. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '01), December 2001 511-518.Google Scholar
- Cristinacce D, Cootes T, Scott I: A multi-stage approach to facial feature detection. Proceedings of the 15th British Machine Vision Conference, 2004, London, UK 277-286.Google Scholar
- Chen L, Zhang L, Zhu L, Li M, Zhang H: A novel facial feature localization method using probabilistic-like output. Proceedings of the Asian Conference on Computer Vision, 2004 1-10.Google Scholar
- Castrilln-Santana M, Dniz-Surez O, Antn-Canals L, Lorenzo-Navarro J: Face and facial feature detection. Proceedings of the 3rd International Conference on Computer Vision Theory and Applications (VISAPP '08), 2008 2: 167-172.Google Scholar
- Moghaddam B, Pentland A: Probabilistic visual learning for object representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997, 19(7):696-710. 10.1109/34.598227View ArticleGoogle Scholar
- Bazen A, Veldhuis R, Croonen G: Likelihood ratio-based detection of facial features. Proceedings of the 14th Annual Workshop on Circuits, Systems and Signal Processing (ProRisc '03), November 2003, Veldhoven, The Netherlands 2: 323-329.Google Scholar
- Beumer GM, Tao Q, Bazen AM, Veldhuis RNJ: A landmark paper in face recognition. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR '06), April 2006 73-78.View ArticleGoogle Scholar
- Everingham M, Zisserman A: Regression and classification approaches to eye localization in face images. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR '06), April 2006 441-446.View ArticleGoogle Scholar
- Wiskott L, Fellous J-M, Krüger N, von der Malsburg C: Face recognition by elastic bunch graph matching. In Intelligent Biometric Techniques in Fingerprint and Face Recognition. Edited by: Jain LC, Halici U, Hayashi I, Lee SB. CRC Press, Boca Raton, Fla, USA; 1999:355-396.Google Scholar
- Shi J, Samal A, Marx D: How effective are landmarks and their geometry for face recognition? Computer Vision and Image Understanding 2006, 102(2):117-133. 10.1016/j.cviu.2005.10.002View ArticleGoogle Scholar
- Arca S, Campadelli P, Lanzarotti R: A face recognition system based on automatically determined facial fiducial points. Pattern Recognition 2006, 39(3):432-443. 10.1016/j.patcog.2005.06.015View ArticleMATHGoogle Scholar
- Salah AA, Çinar H, Akarun L, Sankur B: Robust facial landmarking for registration. Annals of Telecommunications 2007, 62(1-2):1608-1633.Google Scholar
- Cootes TF, Taylor CJ, Cooper DH, Graham J: Active shape models—their training and application. Computer Vision and Image Understanding 1995, 61(1):38-59. 10.1006/cviu.1995.1004View ArticleGoogle Scholar
- Cooles TF, Edwards GJ, Taylor CJ: Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001, 23(6):681-685. 10.1109/34.927467View ArticleGoogle Scholar
- Mahalanobis A, Kumar BVKV, Casasent D: Minimum average correlation energy filters. Applied Optics 1987, 26(6):3633-3640.View ArticleGoogle Scholar
- Savvides M, Vijaya Kumar B: Efficient design of advanced correlation filters for robust distortion-tolerant face recognition. Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, July 2003 45-52.View ArticleGoogle Scholar
- Savvides M, Abiantun R, Heo J, Park S, Xie C, Vijayakumar BVK: Partial & holistic face recognition on frgc-ii data using support vector machine. Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops (CVPRW '06), June 2006 48-48.Google Scholar
- Jia K, Gong S, Leung A: Coupling face registration and super-resolution. Proceedings of the British Machine Vision Conference, September 2006 2: 449-458.Google Scholar
- Jonsson K, Matas J, Kittler J, Haberl S: Saliency-based robust correlation for real-time face registration and verification. Proceedings of the British Machine Vision Conference (BMVC '98), 1998 44-53.Google Scholar
- Matas J, Jonsson K, Kittler J: Fast face localization and verification. Image and Vision Computing 1999, 17(8):575-581. 10.1016/S0262-8856(98)00176-0View ArticleGoogle Scholar
- Wang P, Tran LC, Ji Q: Improving face recognition by online image alignment. Proceedings of the 18th International Conference on Pattern Recognition (ICPR '06), August 2006 1: 311-314.View ArticleGoogle Scholar
- Spreeuwers L, Boom B, Veldhuis R: Better than best: matching score based face registration. Proceedings of the 28th Symposium on Information Theory in the Benelux, 2007 125-132.Google Scholar
- Boom B, Beumer G, Spreeuwers L, Veldhuis R: Matching score based face registration. In Proceedings of the 17th Annual Workshop on Circuits, Systems and Signal Processing (ProRISC '06), 2006, Veldhoven, The Netherlands. STW;Google Scholar
- Boom B, Spreeuwers L, Veldhuis R: Automatic face alignment by maximizing similarity score. Proceedings of the 7th International Workshop on Pattern Recognition in Information Systems (PRIS '07), June 2007 221-230.Google Scholar
- Liu C, Shum H-Y, Freeman WT: Face hallucination: theory and practice. International Journal of Computer Vision 2007, 75(1):115-134. 10.1007/s11263-006-0029-5View ArticleGoogle Scholar
- Jia K, Gong S: Generalized face super-resolution. IEEE Transactions on Image Processing 2008, 17(6):873-886.MathSciNetView ArticleGoogle Scholar
- Cootes TF, Taylor CJ: On representing edge structure for model matching. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '01), December 2001 1: 1114-1119.Google Scholar
- Nelder J, Mead R: A simplex method for function minimization. The Computer Journal 1965, 7(10):308-315.View ArticleMATHGoogle Scholar
- Hager GD, Belhumeur PN: Efficient region tracking with parametric models of geometry and illumination. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998, 20(10):1025-1039. 10.1109/34.722606View ArticleGoogle Scholar
- Baker S, Matthews I: Lucas-Kanade 20 years on: a unifying framework. International Journal of Computer Vision 2004, 56(3):221-255.View ArticleGoogle Scholar
- Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Chang J, Hoffman K, Marques J, Min J, Worek W: Overview of the face recognition grand challenge. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05), June 2005 1: 947-954.Google Scholar
- Lienhart R, Kuranov A, Pisarevsky V: Empirical analysis of detection cascades of boosted classifiers for rapid object detection. In Pattern Recognition, Lecture Notes in Computer Science. Volume 2781. Springer, Berlin, Germany; 2003:297-304. 10.1007/978-3-540-45243-0_39Google Scholar
- Intel : Open computer vision library. http://sourceforge.net/projects/opencvlibrary/
- Wang P, Green M, Ji Q, Wayman J: Automatic eye detection and its validation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '05), June 2005 164-164.Google Scholar
- Perlibakas V: Distance measures for PCA-based face recognition. Pattern Recognition Letters 2004, 25(6):711-724. 10.1016/j.patrec.2004.01.011View ArticleGoogle Scholar
- Zhang G, Huang X, Li SZ, Wang Y, Wu X: Boosting local binary pattern (lbp)-based face recognition. Proceedings of the Chinese Conference on Biometric Recognition (SINOBIOMETRICS '04), 2004, Guangzhou, China 179-186.Google Scholar
- Veldhuis R, Bazen A, Booij W, Hendrikse A: Hand-geometry recognition based on contour parameters. Biometric Technology for Human Identification II, March 2005, Orlando, Fla, USA, Proceedings of SPIE 344-353.View ArticleGoogle Scholar
- Jonathon Phillips P, Flynn PJ, Scruggs T, Bowyer KW, Worek W: Preliminary face recognition grand challenge results. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR '06), April 2006 15-24.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.