Skip to main content

Face recognition via edge-based Gabor feature representation for plastic surgery-altered images

Abstract

Plastic surgery procedures on the face introduce skin texture variations between images of the same person (intra-subject), thereby making the task of face recognition more difficult than in normal scenario. Usually, in contemporary face recognition systems, the original gray-level face image is used as input to the Gabor descriptor, which translates to encoding some texture properties of the face image. The texture-encoding process significantly degrades the performance of such systems in the case of plastic surgery due to the presence of surgically induced intra-subject variations. Based on the proposition that the shape of significant facial components such as eyes, nose, eyebrow, and mouth remains unchanged after plastic surgery, this paper employs an edge-based Gabor feature representation approach for the recognition of surgically altered face images. We use the edge information, which is dependent on the shapes of the significant facial components, to address the plastic surgery-induced texture variation problems. To ensure that the significant facial components represent useful edge information with little or no false edges, a simple illumination normalization technique is proposed for preprocessing. Gabor wavelet is applied to the edge image to accentuate on the uniqueness of the significant facial components for discriminating among different subjects. The performance of the proposed method is evaluated on the Georgia Tech (GT) and the Labeled Faces in the Wild (LFW) databases with illumination and expression problems, and the plastic surgery database with texture changes. Results show that the proposed edge-based Gabor feature representation approach is robust against plastic surgery-induced face variations amidst expression and illumination problems and outperforms the existing plastic surgery face recognition methods reported in the literature.

1 Introduction

The much attention given to face recognition within the research and commercial community can be associated with its real-world application potentials in areas such as surveillance, homeland security, and border control. Among the most challenging tasks for face recognition in these application scenarios is the development of robust face recognition systems [1]. This implies that apart from recognizing faces under normal scenario, such systems should also be able to successfully handle issues arising from unconstrained conditions. The face recognition under unconstrained conditions results in faces which are termed here and throughout this paper as the unconstrained faces.

Typically, unconstrained faces include faces that are subject to factors such as changes in illumination direction, pose, expression, and recently introduced variations due to plastic surgery [2]. The problem of pose, expressions, and illumination in face recognition has been addressed in a good number of literatures, some of which are [39]. However, there has been scanty literature on the recognition of surgically altered faces. Like changes in illumination direction, plastic surgery procedures induce intra-subject (face image versions of the same person) dissimilarity, which are impediments to robust face recognition. Such problem can be exacerbated when other conditions such as pose and expression are included. The main focus of this paper is to address the recognition problems that arise from conditions where the face is surgically altered.

To solve the problem of face recognition under unconstrained conditions, let us take a quick look at a typical face recognition system as shown in Figure 1. This system mainly consists of the face preprocessing, face representation, and face classification stages. Among these three stages, the face representation stage has been identified as a fundamental component in face recognition that is necessary for minimizing intra-subject variations as well as increasing inter-subject discrimination margin [1012]. Over the past years, many face representation approaches such as eigenface [13], fisherface [14], Gabor [11], and local binary pattern (LBP) [15] have been introduced. The eigenface and fisherface are categorized as global approaches, while Gabor [11] and LBP [15] are categorized as local approaches. In a study by Heisele et al. [16], a comparison between the global approaches and the local approaches shows that the local matching methods outperform the global matching methods in accurate face identification. Hence, in the ensuing discussions, emphasis will be on the local approach.

Figure 1
figure 1

Architecture of a typical face recognition system. Dashed rectangles highlight the contribution points of this paper.

The local approach, LBP, describes a central point pixel by the changes in its neighboring pixels. According to Vu and Caplier [17], the LBP is basically a fine-scale descriptor that mostly captures small texture details. Some of the existing LBP-based descriptors are the monogenic-local binary pattern (M-LBP) [18], local binary pattern histogram Fourier features (LBP-HF) [9], and local Gabor binary pattern histogram sequence (LGBPHS) [8], which utilize image texture properties that LBP encodes. For faces altered by plastic surgery procedures that introduce texture variations on face images, the LBP-based descriptors fall short since they mostly encode texture properties of the face. Hence, the face recognition system that utilizes texture variant descriptors may not be hardy against faces altered by plastic surgery procedures.

On the other hand, Gabor descriptor captures salient visual properties such as spatial localization, orientation, selectivity, and spatial frequency characteristics [9]. Gabor typically encodes facial shape and appearance [17], which makes it robust against factors such as facial expression [46], mutilated faces [19], occlusion [7], and pose [3]. Gabor is also good for small sample size problem [11]. However, studies have shown that Gabor features are sensitive to gross changes in illumination direction [12] and does retain some elements of small-scale textures [20]. Hence, compensating for the influence of illumination changes and texture changes for unconstrained faces is a necessary step towards a robust Gabor feature descriptor for surgically altered images.

Before addressing the illumination problem for the descriptor, one has to take into cognizance the type of face images that are input to the descriptor. Basically, in face recognition tasks, the inputs are the original gray-level (intensity) image. The intensity-based methods encode image texture properties [21]. The robustness of the texture-encoding approach in the recognition of surgically altered faces can be assessed by considering a typical scenario. A typical gross case of plastic surgery is the rhytidectomy, which is a procedure that changes the global appearance of a face. This surgery procedure fundamentally enhances facial skin texture from an aging state to a younger state, hence bringing about change in skin texture. In most cases, rhytidectomy is combined with some surgery procedures such as nose reshaping, eye lift, and jaw enhancement, which change the face appearance (increases intra-subject variation), but might not necessarily change the shape of the facial components. In other words, rhytidectomy is an embodiment of local and global appearance-changing surgery procedures, which may explain the challenges that the existing intensity-based recognition methods in the case of rhytidectomy [2] and subsequent works [2225] faced. In contrast to the original gray-level image, the image edge information is an alternative because only valuable representation of the most significant details of the face is retained, which we presume is a good candidate for the recognition of surgically altered faces. However, as pointed out by Gao and Qi [26], edge information are insensitive to illumination changes but only to a certain extent. The question then is to what extent? The extent to which the edge information is insensitive to illumination is dependent on light distribution across the illuminated object. Hence, if we can eliminate or reduce the effect of the inconsistent lighting of the face, then the edge information will richly retain the shape of significant facial components, which may minimize intra-subject variations induced by plastic surgery. And to the best of our knowledge, this alternative has not been explored in the recognition of surgically altered face images. Our work considers exploring the shape of the facial components to address the intra-subject dissimilarity problem due to plastic surgery.

In compensating for illumination variations, illumination normalization techniques are often employed. These techniques compensate for illumination variations while retaining image feature shape characteristics [27]. Some of the existing illumination normalization techniques include the histogram equalization (HE) [28], gamma correction (GC) [29], the logarithm transform (LT) [30], and the quotient image techniques [31]. The HE is normally used to make an image have a uniform histogram to produce an optimal global contrast in the image. However, HE may make an image that has uneven illumination turn to be more uneven. The LT works best at shadow regions of a given image [30]. For the quotient image-based techniques, it is known that they are dependent on the albedo (texture) [31]. Since the quotient image-based techniques comprise the ratio of albedo (texture) between a test face and a given face, edge information obtained from such techniques have the likelihood of containing many false edges [32]. The GC corrects the overall brightness of a face image to a pre-defined ‘canonical form’ , which fades away the effect of varying lighting. But, the GC is still affected by some level of directional lighting as pointed out by [33].

We may note at this point that all the above mentioned illumination normalization techniques are used on gray-scale images. However, in recent times, most face images that are acquired and are available for face recognition task are color images. Studies have shown that illumination effect due to changes in light direction can be addressed in the color domain when the source color is known and constant over a scene [34, 35]. According to Zickler et al. [34], red, green, and blue (rgb) color space transformations remove the effect of illumination direction without explicit specular/diffuse separation. This claim is also supported by the work of Finlayson et al. [35], where it is highlighted that the image dependencies due to lighting geometry can be removed by normalizing the magnitude of the rgb pixel triplets. Therefore, inspired by these studies, the proposition of illumination normalization steps that take advantage of color domain normalization to improve the performance of edge-based face recognition systems is desired.

1.1 Related works

Face recognition performance in plastic surgery scenarios for cases such as rhytidectomy (face and mid-face lift), rhinoplasty (nose reshaping), blepharoplasty (eye surgery), otoplasty (ear surgery), browlift, dermabrasion, and skin peeling has been investigated [2, 2224, 3538]. Bhatt et al. [24, 25] adopted non-disjoint face granulation approach where the granules are obtained from HE-normalized images. The features were then extracted using the extended uniform circular local binary patterns (EUCLBP) and scale invariant feature transform (SIFT). The performance of their method is significantly impacted by rhytidectomy procedure. In [23], a Gabor patch classifier that uses the rank-order list fused from equal and non-overlapping patches of surgically altered face images for discrimination was proposed. Aggarwal et al. [36] adopted a part-wise sparse representation approach for matching plastic surgery-altered faces. They employed the intensity characteristics of the principal component analysis (PCA)-based representation of the six facial components cropped from each face image. The facial components were then fused to determine the sparse representation error. A match is found if the probe sample produces smallest representation error to a test sample. In [37, 38], the multimodal biometrics, such as holistic face information and the periocular region, were adopted. Then, the features were extracted using shape local binary texture and Gabor. In [22], the method of face analysis for commercial entities (FACE) was adopted. The FACE utilizes correlation index obtained from defined subregions between two images. By correcting for illumination problem using self-quotient image, an improved performance was obtained using the FACE method. With particular interest in rhytidectomy, it is worth pointing out that though the recognition results in [2, 22] suggest that the algorithms have tried to address the challenges in face recognition in the event of rhytidectomy, there is still a significant scope for further improvement.

1.2 Our contributions and paper organization

In the following, we summarize the main contributions of this paper.

  • We propose illumination normalization steps that reduce image dependency on illumination direction and control edge extraction sensitivity to illumination for the purpose of face recognition. The proposed illumination normalization steps are obtained from fusing RGB normalization (rgbN) method and the non-linear pixel power transform method termed GC for color images. We term this proposed steps the rgb-gamma encoding (rgbGE) technique.

  • Under the assumption that the shape of the facial components might remain unchanged after surgery, we propose edge-based Gabor face representation for face recognition of surgically altered face images. The shape of significant facial components is retained by extracting gradient magnitude information from the rgbGE-normalized face image so as to minimize the intra-subject variations due to surgery.

  • By means of experimental results, we show that the edge-based Gabor face representation approach performs significantly well in a simple nearest neighbor search face recognition framework. And the robustness of the proposed approach is investigated first with typically investigated problems such as pose, expression, and illumination problems and then plastic surgery.

The rest of this paper is organized as follows. In Section 2, the illumination normalization is presented. We present in Section 3 the proposed edge-based Gabor face representation approach. In Section 4, the face recognition experiment using the proposed face representation approach is presented. Finally, conclusions are drawn in Section 5.

2 Illumination normalization

As highlighted earlier, the changes in illumination condition results in false edges with respect to edge information extraction, and this has to be properly addressed. In this section, a brief review of the face reflectance model is firstly provided in such a manner that establishes a coherent basis for presenting the proposed illumination normalization technique. Subsequently, the proposed technique and the related step-by-step procedure for actualizing the technique are presented.

2.1 Face reflectance model

Light reflections from most surfaces are of two basic types, namely, the diffuse and specular reflections. The diffuse reflection defines the case where the incident light is reflected equally in all directions [39] and is well described by the Lambertian model [40]. The specular reflection for a smooth surface defines the case where the incident light is reflected in a mirror-like direction from the surface [41]. These reflections are often modelled using the Phong reflectance model [42].

To model a typical image captured using RGB camera sensor, we use the dichromatic reflection model described by Shafer [43], which includes the Lambertian term and the Phong's specular term. This model is given by

I k c = w d c w S c λ E λ C k λ + w s c w E λ C k λ , k = r , g , b
(1)
= w d c D k c + w s c G k ,
(2)

where D k c = w S c λ E λ C k λ , G k = w E λ C k λ , I k  = {Ir, Ig, Ib} is the color vector of image intensity, λ is the wavelength of the light, Sc(λ) is the spectral reflectance on a surface point c (where c is of spatial coordinates {x, y}). E(λ) is the spectral power distribution of the incident light, and C k (λ) is the spectral sensitivity of the sensor. The terms wd and ws are the diffuse and specular terms of the incoming light, respectively. The first part of the right-hand side of (2) is the diffuse component, while the second part is the specular component.

For the color vectors {r, g, b}, (2) can be rewritten as

I k c = w d c D r c + w s c G r w d c D g c + w s c G g w d c D b c + w s c G b = I r c I g c I b c
(3)

Equation 3 describes the intensity components, which comprise diffuse and specular reflections for an RGB image captured in uncontrolled lighting environment. Mathematically, the objective of the proposed normalization technique is to reduce/eliminate the dependency of I k (c) on the factors ws, Gr, Gg, and Gb. To achieve this objective, we employ the merits of RGB and GC normalizations. The RGB normalization will address the directional lighting effect, while the GC normalizes a face image to a pre-defined ‘canonical form’ , which fades away the effect of illumination.

2.2 RGB normalization

The normalized RGB (Nrgb) is used in [35, 44]. The Nrgb is expressed by [44]

β k c = I k c I r c + I g c + I b c ,
(4)

where β k  = {βr, βg, βb} represents each color channel.

The computation of (4) results in the removal of intensity variations from the image, so that the RGB components of the image specify color only, and no luminance. This ensures that the normalized image becomes insensitive to changes in illumination direction.

2.3 Gamma correction

Gamma correction is a non-linear operation generally used to control image overall brightness. It is simply defined by the power law expression with respect to the input image Iinput and output image Ioutput, as

I output = I input γ ,
(5)

where γ is the exponent of the power function. Usually, the gamma value can be between the range [0,1] and is referred to as the encoding gamma. Gamma encoding, which is a form of non-linear transformation of pixels, enhances the local dynamic range of the images in dark or shadowed regions while compressing it in bright regions and at highlights [45]. For a given image I(c), the gamma encoding transform is expressed as

G c = I 1 / γ c
(6)

In essence, the transformation technique corrects the problem of non-uniform intensity, where too much bits belong to high intensities and too few bits to low intensities.

2.4 Fusion of Nrgb and GC

The proposed illumination normalization technique rgbGE fuses the merits of RGB normalization and GC described above in order to compensate for illumination problem. The steps in the fusion of Nrgb and GC techniques are presented as follows:

  • Step 1: Separate the image I k (c) into the respective RGB color channels, Ir(c), Ig(c), and Ib(c).

  • Step 2: Obtain the square magnitude Im(c) of the separated RGB color channel images in step 1; thus,

    I m c = I r 2 c + I g 2 c + I b 2 c
    (7)
  • Step 3: Normalize the images Ir(c), Ig(c), and Ib(c) in step 1 by the Im(c) (step 2) as follows:

    I r c = I r c I m c I g c = I g c I m c I b c = I b c I m c
    (8)
  • The essence of this step is to reduce the image intensity variation so that the RGB components of the image specify color only, and no luminance.

  • Step 4: Compute the gamma encoding transformation of the Im(c) in step 2.

    G m c = I m 1 / γ c , γ 0 , 1
    (9)
  • Step 5: Multiply the result of step 3 by G m ' c as shown below:

    f r c = I r c G m c f g c = I g c G m c f b c = I b c G m c
    (10)
  • When the Ir(c), Ig(c), and Ib(c) in step 3 are recombined to form a color image, the resultant image is of low contrast. Step 5 is used to restore the contrast level to a level adequate for further processing.

  • Step 6: By concatenating the expressions in (10) along the third dimension, we obtain the rgbGE-normalized image; thus,

    Ψ k c = f r c 3 f g c 3 f b c ,
    (11)

where Ψ k (c) is the rgbGE-normalized image, and 3 symbolizes concatenation along the third dimension.

By this transformation, the illumination problem in the original image is compensated, and edge information obtained from Ψ k (c) will have little or no false edges. It is important to note that the working principle of the rgbGE is based on the dichromatic reflectance model, which does not consider the ambient component. Hence, the performance of the technique will be significantly affected by outdoor captured images that mostly have ambient lighting components. More also, it should be noted that the variation in the degree of illumination problem across all images in a given database also affects the performance of rgbGE. Basically, the rgbGE performs well only when the degree of illumination variation across all images in a database is insignificant.The main idea behind the proposed illumination normalization technique is to minimize the intra-subject variation due to illumination as well as skin texture differences. In Figure 2, example images (images of a person) from three datasets are used to illustrate the performance of the proposed illumination normalization technique under varying lighting conditions (but without any ambient component). A number of images for each subject with varying degree of lighting problem are used in the illustration. It can be seen from Figure 2 that the normalized images of the subjects appear more similar to each other.

Figure 2
figure 2

Example images from three datasets. (a1, b1, and c1) The original images of three different subjects. (a2, b2, and c2) The corresponding normalized images obtained with the proposed preprocessing method.

3 Edge-based Gabor face representation

As can be observed in image 1 of Figure 2b, the pre-surgery face images (the images on the first two columns) and the post-surgery images (the image on the third column) show a great amount of skin texture changes. Such differences between the images of the same subject are likely to impact on the face recognition accuracy. A plausible solution is to exploit the face information that are not likely to be affected by plastic surgery. Hence, we exploit the shape of the facial components, i.e., the shape of the eyes, nose (nostrils), eyebrow, and mouth that do not change after plastic surgery procedures. We put forward that this frame of reference serves as a platform for constructing robust and efficient feature descriptors for recognizing surgically altered face images. Under these assumptions, we utilize edge information, which are dependent on the shapes of the significant facial components of the face to address the intra-subject variations due to plastic surgery procedures. The basic idea of the proposed edge-based Gabor face representation approach is aimed at mitigating the intra-subject variations induced by plastic surgery procedures. This is achieved via computing the edge gradient magnitude of the illumination-normalized image. Applying Gabor wavelet on the resultant edge gradient magnitude image accentuates on the uniqueness of significant facial components, which enlarges the discrimination margin between different person face images. These processes are discussed below.

3.1 Edge gradient magnitude computation

Let the grayscale version of the illumination-normalized image Ψ k (c) be denoted as Ψ(c). The edge information g(c) of the image Ψ(c) is obtained via the computation of the gradient magnitude of the image; thus [46],

g c = x Ψ c 2 + y Ψ c 2 ,
(12)

where x Ψ = Ψ δ δx S and y Ψ = Ψ δ δy S denote partial derivatives, with S as smoothening Sobel filter function.

The false edges in the gradient magnitude image g(c) are substantially reduced when the rgbGE normalization technique is employed. This can be observed in Figure 3a.In Figure 3, the gradient of the rgbGE normalized face images (three images of a subject) is compared with the original image without correction and with various illumination normalization methods such as LT, HE, and GC. It can be seen from the figure that the gradient of the rgbGE face images shows less facial appearance differences in comparison to the other methods. In subsequent subsections, the Gabor encoding process is given in detail.

Figure 3
figure 3

Illustration of edge gradient magnitude for intra-subjects with different illumination normalization techniques. (a) rgbGE, (b) LT, (c) without normalization, (d) HE, and (e) GC.

3.2 Gabor wavelets

Gabor wavelets (kernels and filters) have proven useful in pattern representation due to their computational properties and biological relevance [3, 7, 11, 19]. It is a powerful tool that provides spatial domain and frequency domain information on an object.

The Gabor kernels can be expressed by [47]

ψ μ , ν c = l μ , ν 2 σ 2 e l μ , ν 2 c 2 / 2 σ 2 e ilμ , ν c e σ 2 / 2 ,
(13)

where μ and ν define the orientation and scale of the Gabor kernels, respectively, c = (x, y), . denotes the norm operator. The term lμ,ν is defined as [11]

l μ , ν = l ν e iφμ ,
(14)

where l ν = l max / s f ν and φ μ  = πμ/8. lmax is the maximum frequency, sf is the spacing factor between kernels in the frequency domain [47], and σ is a control parameter for the Gaussian function.

The family of self-similar Gabor kernels in (13) is generated from a mother wavelet by selecting different center frequencies (scales) and orientations. In most cases, the Gabor wavelets at five scales ν {0, …, 4} and eight orientations μ {0, …, 7} are used [11, 19]. This paper uses Gabor kernels at five scales and eight orientations with the following parameters: σ = 2π, lmax = π/2, s f = 2 [11, 19] as shown in Figure 4. The edge image g(c) is convolved with a family of Gabor kernels at five scales and eight orientations; thus,

Figure 4
figure 4

Gabor wavelets at five scales and eight orientations. (a) The real part of the Gabor wavelet. (b) The magnitude part of the of the Gabor wavelet.

Ο μ , ν c = g c ψ μ , ν c ,
(15)

where denotes the convolution operator, and Ομ,ν(c) is the corresponding convolution result at different scales ν and orientations μ.

Applying the convolution theorem, each Ομ,ν(c) from (15) can be derived via the fast Fourier transform (FFT) [11]:

Ο μ , ν c = 1 g c ψ μ , ν c ,
(16)

where and −1 denote the Fourier transform and its inverse, respectively.

The Gabor wavelet representation of the edge image g(c) is shown in Figure 5, where only the magnitude responses of Ομ,ν(c) is used to construct the Gabor feature. Having computed Ομ,ν(c), the augmented feature vector, namely the edge-based Gabor magnitude (EGM) feature matrix Ζ(p), is obtained by concatenating each Ομ,ν(c) already downsampled by a factor p, where p = 64 becomes Ο μ , ν c and normalized to zero mean and unit variance. By so doing, the augmented EGM feature matrix Ζ(p) encompasses every possible orientation selectivity, spatial locality, and frequency of the representation result; thus,

Figure 5
figure 5

Edge-based Gabor magnitude representation of a sample face image.

Ζ p = Ο 0 , 0 , jk T Ο 0 , 1 , jk T Ο 7 , 4 , jk T T = Z rq
(17)

where T is the transpose operator, Ο μ , ν , jk are the respective J × K downsampled image matrices with orientation μ and scale ν, and Z rq are the elements of the R × Q EGM feature matrix. The procedure for obtaining the EGM feature is clearly illustrated in Figure 6.

Figure 6
figure 6

Overview of the stages taken into consideration for describing a face image. The stages enclosed by the broken line highlight the proposed EGM descriptor.

3.3 Dimensionality reduction and discriminant analysis

The EGM features are of high dimensional space, such that Ζ(p) RN, where N is the dimensionality of the vector space. To address the dimensionality problem and still retain the discriminating information for identifying a face, we apply the two-stage (PCA + LDA) approach [14, 48]. Each same person face is defined as belonging to a class. Let ω1, ω2, , ω L and N1, N1, , N L denote the classes and the number of images within each class, respectively. Let M1, M1, , M L and M be the mean values of the classes and the grand mean value. The within-class scatter matrix S ω and the between-class scatter matrix Sb are defined as [14, 48]

S ω = i = 1 L P Ω i ϵ Y p M i Y p M i T Ω i
(18)
S b = i = 1 L P Ω i M i M M i M T ,
(19)

where Y(p) is the most expressive feature of the original data Ζ(p) obtained with a PCA step so that LDA is implemented in the PCA subspace [14]. P(Ω i ) is the probability of the i th class, and L denotes the number of classes.

The LDA derives a projection matrix A that maximizes the Fisher's discriminant criterion:

J A = arg max A A S b A T A S ω A T
(20)

The Fisher's discriminant criterion is maximized when A consists of the eigenvectors of the matrix S ω 1 S b [48].

S ω 1 S b A = ,
(21)

where A and Δ are the eigenvector and eigenvalue matrices of S ω 1 S b , respectively. The two-stage (PCA + LDA) dimensionality reduction approach is employed to maximize the between-class variations and minimize the within-class variations of the projected face subspace.For validation purpose, the face recognition performance of Gabor, i.e., with the original gray-level face images, and the EGM, i.e., with the gradient magnitude face images, are shown in Figure 7.

Figure 7
figure 7

Face recognition performances using Gabor with original gray-level face images and gradient magnitude faces.

It can be observed that the use of gradient magnitude image improved the performance of the Gabor descriptor significantly compared to using the original gray-level face images. At this point, it is important to note that the illumination normalization technique that can be used with the EGM include any of the existing normalization techniques discussed in this work. For simplicity, and hence forth, we use the acronym EGM-rgbGE to represent the EGM that employs the rgbGE illumination normalization technique, EGM-HE to represent the EGM that employs the HE illumination normalization technique, and EGM-GC to represent the EGM that employs the GC illumination normalization technique.

4 Face recognition experiment

In this section, the results of the proposed EGM-based face recognition method on a plastic surgery database [2], the Georgia Tech (GT) face database [49], and the Labeled Faces in the Wild (LFW) database [50] are presented. The details of the datasets and the experimental setups for the face recognition experiment are provided. We show through simulations the result of the proposed method compared with the existing face recognition methods.

4.1 Datasets and experimental setup

4.1.1 Georgia Tech dataset

The Georgia Tech face database [49] contains of 750 color images of 50 subjects, some of which where captured during different sessions. These images comprise variations in illumination direction, scale, pose, and expression; see sample images in Figure 8. The images were manually cropped and resized to size 128 × 128. The database is partitioned into training and testing sets. The number of training and test images is selected to resemble a real-time scenario where only one image per person is tested on a large database in which there exist numerous images of the same person.

Figure 8
figure 8

Original sample faces from the GT dataset.

4.1.2 Plastic surgery dataset

The plastic surgery dataset [2] contains frontal face images of plastic surgery-altered faces, which vary by scale and small expression, small pose, and majorly by plastic surgery procedures. There are only two images (pre-surgery and post-surgery) per subject available. The dataset consists of different plastic surgery procedures such as rhinoplasty (nose surgery), blepharoplasty (eyelid surgery), brow lift, skin peeling, and rhytidectomy (face and mid-face lift surgery). According to Bhatt et al. [24, 25], global surgeries like rhytidectomy severely impacted on their recognition algorithm. To investigate this impact, we used the rhytidectomy plastic surgery dataset, which we arranged to combine full-face lift and mid-face lift surgery images. The rhytidectomy dataset consists of two images per subject. There are 321 subjects; hence, the total number of images is 642. The samples of the rhytidectomy images from the dataset are shown in Figure 9.

Figure 9
figure 9

Rhytidectomy plastic surgery sample faces, pre-surgery (top row), post-surgery (bottom row).

Motivated by the fact that a face can be identified from its mirror image [51], we include a mirror version of the pre-surgery images in the database in order to increase the sample size of each subject. Hence, there are a total of 963 face images cropped and resized to size 128 × 128. For this database, we consider two scenarios s 1 and s 2 in the experimental setup. In s 1, out of the 321 subjects, we used 128 subjects for training and performance is evaluated on the remaining subjects, where the pre-surgery images are used as gallery and post-surgery images are used as the probe. In s 2, pre-surgery images of all the subjects are used for training and gallery while the post-surgery images are used as the probe.

4.1.3 Heterogeneous dataset

This database consists of both plastic surgery images and other images that are randomly selected from other databases. The main idea behind the use of this database structure in performance analysis of the EGM is to consider a typical real-world scenario of training-testing where the system is unaware of any plastic surgery cases. In this dataset, the images of 321 subjects with three images per subject are added to the plastic surgery database used above. This brings the total number of subjects to 642, with every subject having three images. For this database, we consider two scenarios s 1 and s 2 just like in the plastic surgery database above. In s 2, the pre-surgery images of all the subjects are used for training and gallery, while the post-surgery images are used as the probe.

4.1.4 Labeled Faces in the Wild (LFW) dataset

The LFW database [50] is a well-known database with gross pose, expression, and illumination problems. Out of all the subjects in the database, we choose subjects with up to three images per subject. Hence, our dataset consists of 869 subjects with images resized to size 128 × 128. For this database, there is only one scenario where 869 images are used as test set, while the training and gallery consist of the remaining images. The LFW sample face images are shown in Figure 10.

Figure 10
figure 10

Original sample faces from the LFW dataset.

4.2 Experimental results

Here, we evaluate the proposed method on all the datasets presented above. In all the experimental setups presented above, the gamma value used in the illumination correction is chosen as γ = 0.75, but the value of γ can be user defined. We used a simple nearest neighbor classifier with the cosine distance measure in order to measure the similarity between two images. Experimental results are obtained for datasets with illumination correction and without illumination correction. The evaluation is of two categories. In the first category, the proposed method is evaluated on the GT face database and LFW database. The performance of the EGM in the case where the proposed rgbGE illumination normalization technique is used is compared to where GC, LT, and HE illumination normalization techniques are used for the GT. This performance comparison is also extended to the case where GC and HE illumination normalization techniques are used for the LFW. The results of the experiment on the GT face database are shown in Table 1 and Figure 11. A comparison of the recognition accuracy of the proposed EGM-rgbGE with existing methods on GT database is shown in Table 2. The results of the experiment on the LFW database are shown in Table 3 and Figure 12.

Table 1 Recognition performance comparisons of the proposed EGM-based face recognition method on Georgia Tech face database
Figure 11
figure 11

Recognition rate for EGM-based face recognition method with different illumination normalization methods on GT database.

Table 2 Recognition rate comparisons with some existing methods on GT face database
Table 3 Recognition performance comparisons of the proposed EGM-based face recognition method on LFW Database
Figure 12
figure 12

Recognition rate for EGM-based face recognition method with different illumination normalization methods on LFW database.

For the second category, we conduct an experiment on a rhytidectomy plastic surgery database. In Table 4 and Figure 13, the rank-1 recognition performance of the proposed EGM-rgbGE-based face recognition method on rhytidectomy plastic surgery database is compared to the existing face recognition methods. The recognition performance of the EGM-rgbGE on the heterogeneous database is shown in Figure 14.

Table 4 Recognition rate comparison with existing methods on plastic surgery database: a case of rhytidectomy
Figure 13
figure 13

EGM recognition performance for plastic surgery database. The experimental scenarios s 1 and s 2 are observed on the basis of with and without rgbGE illumination normalization technique.

Figure 14
figure 14

EGM recognition performance for heterogeneous database. The experimental scenarios s 1 and s 2 are observed on the basis of with and without rgbGE illumination normalization technique.

From the results in Figure 11 and Table 1, it can be observed that the EGM-LT did not improve on the rank-1 recognition rate (RR) of EGM without correction. This shows that it does not address the presence of false edges induced by illumination as is evident from Figure 3. However, it improved in terms of equal error rate (EER). On the other hand, the RR performances of EGM-HE and EGM-GC degraded in comparison with EGM without correction. In this case, such degradation may be associated with the introduction of more false edges to the edge information. The EGM-rgbGE-based face recognition method is observed to be more insensitive to performance degradation factors such as illumination, pose, and expression with 98% RR and 0.34% EER.

In Table 2, the performance of the proposed EGM-based face recognition method compared with some existing face recognition methods for the GT database is shown. The performance of the face recognition methods considered are sorted by their rank-1 recognition rates. The performance of EGM-rgbGE in terms of RR with no face alignment is significantly higher than those of all other methods with no face alignment. For the methods with face alignment, the EGM-rgbGE performed better except for the method by Wouter and Peter [19]; of course with face alignment, the EGM-rgbGE will perform better. In the case of LFW, the performance of the proposed method performs equally well as the existing results in the literature. The reader is referred to [57] for results on LFW database. We note at this point that the goal of the current paper is to demonstrate the efficiency of the proposed descriptor based face recognition method, not to compete in the LFW challenge.

In Figure 13, the proposed EGM-rgbGE method performs better in the s 2 scenario than in s 1 scenario. This performance difference may not be unconnected with the method of training and evaluation. In both s 1 and s 2 scenarios, the recognition rates are better when rgbGE normalization is used than when it is not incorporated into the EGM method. Table 4 shows the performance comparison of the EGM-rgbGE method with the existing methods on rhytidectomy plastic surgery database. The EGM-rgbGE rank-1 recognition rate of over 89% is a difference of 17.48% in comparison to granular approach [24, 25], 15.24% in comparison to FACE [22], and 2.56% in comparison to GROF [23]. However, it should be noted that face alignment was employed to obtain the result in the GPROF method, while the EGM-rgbGE result is without face alignment. The RR and EER results of the different experimental setups and scenarios for the plastic surgery database are shown in Table 5.

Table 5 Recognition performance of the proposed EGM-based face recognition method for the plastic surgery database

The result of the experiment on the heterogeneous database (a combination of the plastic surgery-altered images and non-surgery images) for training-testing, whereby the face recognition system is set to resemble a real-world scenario, unaware of any plastic surgery image, is shown in Figure 14. The experimental results in the two scenarios show how the proposed method, which considers the shape of facial components, performs across surgery and non-surgery images. Again, the proposed EGM-rgbGE method performs better in scenarios s 1 and s 2 in comparison with other existing methods. In both s 1 and s 2 scenarios, the recognition rates are better when the rgbGE normalization is used than when it is not incorporated into the EGM method. As can be observed, the result of the experiments in the case of heterogeneous database outperforms that of the experiments in the case of only-plastic-surgery database. This performance variation is expected since the average performance will be expectedly higher due to the inclusion of non-plastic surgery images in the database with lesser recognition challenges than purely plastic surgery database. The RR and EER results of the different experimental setups and scenarios for the heterogeneous database are shown in Table 6.

Table 6 RR and EER performance comparisons of the proposed EGM-based face recognition method for the heterogeneous database

5 Conclusion

We have presented the proposed edge-based Gabor feature representation approach for appearance representation of faces altered by plastic surgery procedures. The proposed face representation approach exploited the shape of significant facial components to address the intra-subject dissimilarity problem due to plastic surgery procedures. The face recognition experiment using the proposed face representation approach considered other unconstrained conditions, which included usual pose, illumination, and expression problems. Comparative experimental results on the GT and LFW databases were provided to show the robustness of the representation approach under usually experimented-on problems and then on the rhytidectomy plastic surgery database. The results indicate that the proposed face representation approach in a simple nearest neighbor search face recognition framework (with no face alignment) performed better than the previously reported methods in the literature on rhytidectomy plastic surgery database. In relation to the GT database, the proposed method performed better than the existing methods, except in comparison with one particular method, in which prior face alignment on the database was carried out, and 0.88% performance difference was reported. However, with face alignment on the database, it is believed that the proposed method will greatly outperform all methods reported so far in the literature. On the LFW database, our method provides up to 76.33% recognition rate. In future work, we will integrate texture insensitive and illumination invariant processes into a single processing step and investigate its performance with other plastic surgery procedures and cases of gross illumination problems.

References

  1. Zhao W, Chellappa R, Phillips PJ, Rosenfeld A: Face recognition: a literature survey. ACM Comput Surv 2003, 35(4):399-458. 10.1145/954339.954342

    Article  Google Scholar 

  2. Singh R, Vatsa M, Bhatt H, Bharadwaj S, Noore A, Nooreyezdan S: Plastic surgery: a new dimension to face recognition. IEEE Trans Inf Technol Biomed 2010, 5: 441-448.

    Google Scholar 

  3. Zheng Z, Yang F, Tan W, Jia J, Yang J: Gabor feature-based face recognition using supervised locality preserving projection. Signal Processing 2007, 87: 2473-2483. 10.1016/j.sigpro.2007.03.006

    Article  MATH  Google Scholar 

  4. Olshausen BA, Field DJ: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 1996, 381: 607-609. 10.1038/381607a0

    Article  Google Scholar 

  5. Rao R, Ballard D: An active vision architecture based on iconic representations. Artif Intell 1995, 78: 461-505. 10.1016/0004-3702(95)00026-7

    Article  Google Scholar 

  6. Schiele B, Crowley JL: Recognition without correspondence using multidimensional receptive field histograms. Int J Comput Vis 1996, 36: 31-52.

    Article  Google Scholar 

  7. Kepenekci B, Tek FB, Akar GB: Occluded face recognition based on Gabor wavelets. Proc ICIP 2002, 1: 293-296.

    Google Scholar 

  8. Zhang W, Shan S, Gao W, Chen X, Zhang H: Local Gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and recognition. Proc ICCV 2005, 1: 786-791.

    Google Scholar 

  9. Ahonen T, Matas J, He C, Pietikäinen M: Lecture notes in computer science: rotation invariant image description with local binary pattern histogram Fourier features. In Image Analysis. Edited by: Salberg A-B, Hardeberg JY, Jenssen R. Springer, Berlin; 2009. vol. 5575, pp. 61–70

    Google Scholar 

  10. Vu NS, Caplier A: Face recognition with patterns of oriented edge magnitudes. Proc ECCV 2010, 6311: 313-326.

    Google Scholar 

  11. Liu C, Wechsler H: Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition. IEEE Trans Image Process 2002, 11(4):467-476. 10.1109/TIP.2002.999679

    Article  Google Scholar 

  12. Adini Y, Moses Y, Ullman S: Face recognition: the problem of compensating for changes in illumination direction. IEEE Trans Pattern Anal Machine Intell 1997, 19: 721-732. 10.1109/34.598229

    Article  Google Scholar 

  13. Turk M, Pentland A: Eigenfaces for recognition. J Cogn Nuerosci 1991, 3: 71-86. 10.1162/jocn.1991.3.1.71

    Article  Google Scholar 

  14. Belhumeur P, Hespanha J, Kriegman D: Eigenfaces vs fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 1997, 19: 711-720. 10.1109/34.598228

    Article  Google Scholar 

  15. Ahonen T, Hadid A, Pietikainen M: Face recognition with local binary patterns. Proc ECCV 2004, 3021: 469-481.

    Google Scholar 

  16. Heisele B, Ho P, Wu J, Poggio T: Face recognition: component-based versus global approaches. Comput Vis Image Understanding 2003, 91: 6-12. 10.1016/S1077-3142(03)00073-0

    Article  Google Scholar 

  17. Vu NS, Caplier A: Enhanced patterns of oriented edge magnitudes for face recognition and image matching. IEEE Trans on Image Processing 2011, 2: 1352-1365.

    MathSciNet  Google Scholar 

  18. Zhang L, Zhang L, Guo Z, Zhang D: Monogenic-LBP: a new approach for rotation invariant texture classification. In Paper presented at the 17th IEEE international conference on image processing (ICIP). Hong Kong; 2677-2680. 26–29 Sept 2010

  19. Wouter JR, Peter HN: Robust face recognition algorithm for the identification of disaster victims. SPIE, Burlingame, CA, USA; 03 Feb 2013, vol. 8655. doi:10.1117/12.2001634

  20. Manjunath BS, Ma WY: Texture feature for browing and retrieval of image data. IEEE Trans Pattern Anal Machine Intell 1996, 12: 55-73.

    Google Scholar 

  21. Liu C, Wechsler H: An integrated shape and intensity coding scheme for face recognition. Proc IJCNN 1999, 5: 3300-3304.

    Google Scholar 

  22. Marsico M, Nappi M, Riccio D, Wechsler H: Lecture notes in computer science: robust face recognition after plastic surgery using local region analysis. In Image Analysis and Recognition. Edited by: Kamel M, Campilho A. Springer, Berlin; 2011:191-200. vol. 6754

    Chapter  Google Scholar 

  23. Liu X, Shiguang S, Xilin C: Lecture notes in computer science: face recognition after plastic surgery: a comprehensive study. In Computer Vision. Edited by: Lee KM, Matsushita Y, Rehg JM, Hu Z. Springer, Berlin; 2013:565-576. vol. 7725

    Google Scholar 

  24. Bhatt H, Singh R, Vatsa M: Evolutionary granular approach for recognizing faces altered due to plastic surgery. In Paper presented at the IEEE international conference on automatic face and gesture recognition and workshops (AFGR). Santa Barbara, CA, USA; 720-725. 21–25 March 2011

    Google Scholar 

  25. Bhatt H, Baradwaj S, Singh R, Vatsa M: Recognizing surgically altered face images using multi-objective evolutionary algorithm. IEEE Trans Inf Forensics Security 2013, 8: 89-100.

    Article  Google Scholar 

  26. Gao Y, Qi Y: Robust visual similarity retrieval in single model face databases. Pattern Recognition 2005, 39: 1009-1020.

    Article  Google Scholar 

  27. Chen W, Er M, Wu S: Illumination compensation and normalization for robust face recognition using discrete cosine transform in logarithm domain. IEEE Trans System, Man, Cybern B 2006, 36: 458-466.

    Article  Google Scholar 

  28. Pizer SM, Amburn EP: Adaptive histogram equalization and its variations. Comput Vis Graph, Image Process 1987, 39: 355-368. 10.1016/S0734-189X(87)80186-X

    Article  Google Scholar 

  29. Shan S, Gao W, Cao B, Zhao D: Illumination normalization for robust face recognition against varying lighting conditions. In Paper presented at the IEEE international workshop on analysis and modeling of faces and gestures (AMFG). Nice, France; 17 Oct 2003:157-164.

    Google Scholar 

  30. Marios S, Kumar BK: Lecture notes in computer science: illumination normalization using logarithm transforms for face authentication. In Audio- and Video-Based Biometric Person Authentication. Edited by: Kittler J, Nixon MS. Springer, Berlin; 2003:549-556. vol. 2688

    Google Scholar 

  31. Xie X, Zheng WS, Lai J, Yuen C, Suen Y: Normalization of face illumination based on large- and small-scale features. IEEE Trans Image Processing 2011, 20: 1807-1821.

    MathSciNet  Article  Google Scholar 

  32. Arandjelovic O: Gradient edge map features for frontal face recognition under extreme illumination conditions. Paper presented at BMVC, Surrey, England; 1-11. 3–7 Sept 2012

    Google Scholar 

  33. Al-Osaimi R, Bennamoun M, Mian A: Illumination normalization for color face images. Proc ISCV 2006, 4291: 90-101.

    Google Scholar 

  34. Zickler T, Mallick P, Kriegman J, Belhumeur N: Color subspaces as photometric invariants. Int J Comput Vision 2008, 79: 13-30. 10.1007/s11263-007-0087-3

    Article  Google Scholar 

  35. Finlayson G, Schiele B, Crowley J: Comprehensive colour normalization. Proc ECCV 1998, 1: 475-490.

    Google Scholar 

  36. Aggarwal G, Biswas S, Flynn JP, Bowyer KW: A sparse representation approach to face matching across plastic surgery. In Paper presented at the IEEE conference on applications of computer vision. Breckenridge, CO, USA; 113-119. 9–11 Jan 2012

    Google Scholar 

  37. Lakshmiprabha NS, Bhattacharya J, Majumder S: Face recognition using multimodal biometric features. In Paper presented at the IEEE conference on image information processing. Himachal, Pradesh; 3–5 Nov 2011

    Google Scholar 

  38. Lakshmiprabha NS, Bhattacharya J, Majumder S: A novel face recognition approach using multimodal feature vector. Int J Adv Eng Tech 2012, 3(1):477-486.

    Google Scholar 

  39. Cook L, Torrance K: A reflectance model for computer graphics. ACM T Graphic 1982, 1: 7-24. 10.1145/357290.357293

    Article  Google Scholar 

  40. Mallick S, Zickler T: Lecture notes in computer science: specularity removal in images and videos: a PDE approach. In Computer Vision. Edited by: Leonardis A, Bischof H, Pinz A. Springer, Berlin; 2006:550-563. vol. 3951

    Google Scholar 

  41. Nayar K, Ikeuchi K, Kanade T: Surface reflection: physical and geometrical perspectives. IEEE Trans Pattern Anal Mach Intell 1991, 13: 611-634. 10.1109/34.85654

    Article  Google Scholar 

  42. Phong BT: Illumination for computer generated pictures. Comm ACM 1975, 18: 311-317. 10.1145/360825.360839

    Article  Google Scholar 

  43. Shafer S: Using color to separate reflection components. Color Res Appl 1985, 10: 210-218. 10.1002/col.5080100409

    Article  Google Scholar 

  44. Tan R, Ikeuchi R: Separating reflection components of textured surfaces using a single image. IEEE Trans Pattern Anal Machine Intell 2005, 27: 178-1935.

    Article  Google Scholar 

  45. Tan X, Triggs B: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 2010, 19(6):1635-1650.

    MathSciNet  Article  Google Scholar 

  46. Zhu J, Zheng W, Lai L: Lecture notes in computer science: complete gradient face: a novel illumination invariant descriptor. In Biometric Recognition. Edited by: Lai J, Wang Y, Chen X, Yuen PC, Zheng W-S, Su Z. Springer, Berlin; 2012:17-24. vol. 7701

    Chapter  Google Scholar 

  47. Lades M, Vorbruggen J, Buhmann J, Lange L, Malsburg C, Wurtz RP, Konen W: Distortion invariant object recognition in the dynamic link architecture. IEEE Trans Comput 1993, 42: 300-311. 10.1109/12.210173

    Article  Google Scholar 

  48. Swets D, Weng J: Using discriminant eigenfeatures for image retrieval. IEEE Trans Pattern Anal Machine Intell 1996, 18: 831-836. 10.1109/34.531802

    Article  Google Scholar 

  49. Georgia Tech Face Database 2007.http://www.anefian.com/research/face_reco.htm . Accessed 2 Feb 2013

  50. Huang G, Manu Ramesh T, Learned-Miller E: Labeled Faces in the Wild: a database for studying face recognition in unconstrained environments, Technical report. University of Massachusetts, Amherst; 2007.

    Google Scholar 

  51. Etemad K, Chellappa R: Discriminant analysis for recognition of human face images. JOSA A 1997, 14(8):1724-1733. 10.1364/JOSAA.14.001724

    Article  Google Scholar 

  52. Majumdar A, Bhattacharya A: Face recognition by multi-resolution curvelet transform on bit quantized facial images. In Paper presented at the international conference on computational intelligence and multimedia applications (ICCIMA). Sivakasi, Tamil Nadu; 209-213. 13–15 Dec 2007

    Google Scholar 

  53. Maturana D, Mery D, Soto A: Face recognition with local binary patterns, spatial pyramid histograms and naive Bayes nearest neighbor classification. In Paper presented at the international conference of the Chilean Computer Science Technology (SCCC). Santiago, Chile; 125-132. 10–12 Nov 2009

    Google Scholar 

  54. Naseem I, Togneri R, Bennamoun M: Linear regression for face recognition. IEEE Trans Pattern Anal Machine Intell 2010, 32: 2106-2112.

    Article  Google Scholar 

  55. Geng C, Jiang X: Fully automatic face recognition framework based on local and global features. Mach Vision Appl 2013, 24: 537-549. 10.1007/s00138-012-0423-7

    Article  Google Scholar 

  56. Li B, Liu W, An S, Krishna A, Xu T: Face recognition using various scales of discriminant color space transform. Neurocomputing 2012, 94: 68-76.

    Article  Google Scholar 

  57. LFW: Results. 1999.http://vis-www.cs.umass.edu/lfw/results.html . Accessed 17 Jan 2014

    Google Scholar 

Download references

Acknowledgements

The authors thank the Universiti Teknologi Malaysia for the International Doctoral Fellowship (IDF) scholarship.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chollette C Chude-Olisah.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Authors’ original file for figure 3

Authors’ original file for figure 4

Authors’ original file for figure 5

Authors’ original file for figure 6

Authors’ original file for figure 7

Authors’ original file for figure 8

Authors’ original file for figure 9

Authors’ original file for figure 10

Authors’ original file for figure 11

Authors’ original file for figure 12

Authors’ original file for figure 13

Authors’ original file for figure 14

Authors’ original file for figure 15

Authors’ original file for figure 16

Authors’ original file for figure 17

Authors’ original file for figure 18

Authors’ original file for figure 19

Authors’ original file for figure 20

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chude-Olisah, C.C., Sulong, G., Chude-Okonkwo, U.A.K. et al. Face recognition via edge-based Gabor feature representation for plastic surgery-altered images. EURASIP J. Adv. Signal Process. 2014, 102 (2014). https://doi.org/10.1186/1687-6180-2014-102

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1687-6180-2014-102

Keywords

  • Face recognition
  • Plastic surgery
  • Illumination normalization
  • Edge information
  • Gabor wavelets