Face recognition via edge-based Gabor feature representation for plastic surgery-altered images

Abstract

Plastic surgery procedures on the face introduce skin texture variations between images of the same person (intra-subject), thereby making the task of face recognition more difficult than in normal scenario. Usually, in contemporary face recognition systems, the original gray-level face image is used as input to the Gabor descriptor, which translates to encoding some texture properties of the face image. The texture-encoding process significantly degrades the performance of such systems in the case of plastic surgery due to the presence of surgically induced intra-subject variations. Based on the proposition that the shape of significant facial components such as eyes, nose, eyebrow, and mouth remains unchanged after plastic surgery, this paper employs an edge-based Gabor feature representation approach for the recognition of surgically altered face images. We use the edge information, which is dependent on the shapes of the significant facial components, to address the plastic surgery-induced texture variation problems. To ensure that the significant facial components represent useful edge information with little or no false edges, a simple illumination normalization technique is proposed for preprocessing. Gabor wavelet is applied to the edge image to accentuate on the uniqueness of the significant facial components for discriminating among different subjects. The performance of the proposed method is evaluated on the Georgia Tech (GT) and the Labeled Faces in the Wild (LFW) databases with illumination and expression problems, and the plastic surgery database with texture changes. Results show that the proposed edge-based Gabor feature representation approach is robust against plastic surgery-induced face variations amidst expression and illumination problems and outperforms the existing plastic surgery face recognition methods reported in the literature.

1 Introduction

The much attention given to face recognition within the research and commercial community can be associated with its real-world application potentials in areas such as surveillance, homeland security, and border control. Among the most challenging tasks for face recognition in these application scenarios is the development of robust face recognition systems [1]. This implies that apart from recognizing faces under normal scenario, such systems should also be able to successfully handle issues arising from unconstrained conditions. The face recognition under unconstrained conditions results in faces which are termed here and throughout this paper as the unconstrained faces.

Typically, unconstrained faces include faces that are subject to factors such as changes in illumination direction, pose, expression, and recently introduced variations due to plastic surgery [2]. The problem of pose, expressions, and illumination in face recognition has been addressed in a good number of literatures, some of which are [3–9]. However, there has been scanty literature on the recognition of surgically altered faces. Like changes in illumination direction, plastic surgery procedures induce intra-subject (face image versions of the same person) dissimilarity, which are impediments to robust face recognition. Such problem can be exacerbated when other conditions such as pose and expression are included. The main focus of this paper is to address the recognition problems that arise from conditions where the face is surgically altered.

To solve the problem of face recognition under unconstrained conditions, let us take a quick look at a typical face recognition system as shown in Figure 1. This system mainly consists of the face preprocessing, face representation, and face classification stages. Among these three stages, the face representation stage has been identified as a fundamental component in face recognition that is necessary for minimizing intra-subject variations as well as increasing inter-subject discrimination margin [10–12]. Over the past years, many face representation approaches such as eigenface [13], fisherface [14], Gabor [11], and local binary pattern (LBP) [15] have been introduced. The eigenface and fisherface are categorized as global approaches, while Gabor [11] and LBP [15] are categorized as local approaches. In a study by Heisele et al. [16], a comparison between the global approaches and the local approaches shows that the local matching methods outperform the global matching methods in accurate face identification. Hence, in the ensuing discussions, emphasis will be on the local approach.

Figure 1

Architecture of a typical face recognition system. Dashed rectangles highlight the contribution points of this paper.

The local approach, LBP, describes a central point pixel by the changes in its neighboring pixels. According to Vu and Caplier [17], the LBP is basically a fine-scale descriptor that mostly captures small texture details. Some of the existing LBP-based descriptors are the monogenic-local binary pattern (M-LBP) [18], local binary pattern histogram Fourier features (LBP-HF) [9], and local Gabor binary pattern histogram sequence (LGBPHS) [8], which utilize image texture properties that LBP encodes. For faces altered by plastic surgery procedures that introduce texture variations on face images, the LBP-based descriptors fall short since they mostly encode texture properties of the face. Hence, the face recognition system that utilizes texture variant descriptors may not be hardy against faces altered by plastic surgery procedures.

On the other hand, Gabor descriptor captures salient visual properties such as spatial localization, orientation, selectivity, and spatial frequency characteristics [9]. Gabor typically encodes facial shape and appearance [17], which makes it robust against factors such as facial expression [4–6], mutilated faces [19], occlusion [7], and pose [3]. Gabor is also good for small sample size problem [11]. However, studies have shown that Gabor features are sensitive to gross changes in illumination direction [12] and does retain some elements of small-scale textures [20]. Hence, compensating for the influence of illumination changes and texture changes for unconstrained faces is a necessary step towards a robust Gabor feature descriptor for surgically altered images.

Before addressing the illumination problem for the descriptor, one has to take into cognizance the type of face images that are input to the descriptor. Basically, in face recognition tasks, the inputs are the original gray-level (intensity) image. The intensity-based methods encode image texture properties [21]. The robustness of the texture-encoding approach in the recognition of surgically altered faces can be assessed by considering a typical scenario. A typical gross case of plastic surgery is the rhytidectomy, which is a procedure that changes the global appearance of a face. This surgery procedure fundamentally enhances facial skin texture from an aging state to a younger state, hence bringing about change in skin texture. In most cases, rhytidectomy is combined with some surgery procedures such as nose reshaping, eye lift, and jaw enhancement, which change the face appearance (increases intra-subject variation), but might not necessarily change the shape of the facial components. In other words, rhytidectomy is an embodiment of local and global appearance-changing surgery procedures, which may explain the challenges that the existing intensity-based recognition methods in the case of rhytidectomy [2] and subsequent works [22–25] faced. In contrast to the original gray-level image, the image edge information is an alternative because only valuable representation of the most significant details of the face is retained, which we presume is a good candidate for the recognition of surgically altered faces. However, as pointed out by Gao and Qi [26], edge information are insensitive to illumination changes but only to a certain extent. The question then is to what extent? The extent to which the edge information is insensitive to illumination is dependent on light distribution across the illuminated object. Hence, if we can eliminate or reduce the effect of the inconsistent lighting of the face, then the edge information will richly retain the shape of significant facial components, which may minimize intra-subject variations induced by plastic surgery. And to the best of our knowledge, this alternative has not been explored in the recognition of surgically altered face images. Our work considers exploring the shape of the facial components to address the intra-subject dissimilarity problem due to plastic surgery.

In compensating for illumination variations, illumination normalization techniques are often employed. These techniques compensate for illumination variations while retaining image feature shape characteristics [27]. Some of the existing illumination normalization techniques include the histogram equalization (HE) [28], gamma correction (GC) [29], the logarithm transform (LT) [30], and the quotient image techniques [31]. The HE is normally used to make an image have a uniform histogram to produce an optimal global contrast in the image. However, HE may make an image that has uneven illumination turn to be more uneven. The LT works best at shadow regions of a given image [30]. For the quotient image-based techniques, it is known that they are dependent on the albedo (texture) [31]. Since the quotient image-based techniques comprise the ratio of albedo (texture) between a test face and a given face, edge information obtained from such techniques have the likelihood of containing many false edges [32]. The GC corrects the overall brightness of a face image to a pre-defined ‘canonical form’ , which fades away the effect of varying lighting. But, the GC is still affected by some level of directional lighting as pointed out by [33].

We may note at this point that all the above mentioned illumination normalization techniques are used on gray-scale images. However, in recent times, most face images that are acquired and are available for face recognition task are color images. Studies have shown that illumination effect due to changes in light direction can be addressed in the color domain when the source color is known and constant over a scene [34, 35]. According to Zickler et al. [34], red, green, and blue (rgb) color space transformations remove the effect of illumination direction without explicit specular/diffuse separation. This claim is also supported by the work of Finlayson et al. [35], where it is highlighted that the image dependencies due to lighting geometry can be removed by normalizing the magnitude of the rgb pixel triplets. Therefore, inspired by these studies, the proposition of illumination normalization steps that take advantage of color domain normalization to improve the performance of edge-based face recognition systems is desired.

1.1 Related works

Face recognition performance in plastic surgery scenarios for cases such as rhytidectomy (face and mid-face lift), rhinoplasty (nose reshaping), blepharoplasty (eye surgery), otoplasty (ear surgery), browlift, dermabrasion, and skin peeling has been investigated [2, 22–24, 35–38]. Bhatt et al. [24, 25] adopted non-disjoint face granulation approach where the granules are obtained from HE-normalized images. The features were then extracted using the extended uniform circular local binary patterns (EUCLBP) and scale invariant feature transform (SIFT). The performance of their method is significantly impacted by rhytidectomy procedure. In [23], a Gabor patch classifier that uses the rank-order list fused from equal and non-overlapping patches of surgically altered face images for discrimination was proposed. Aggarwal et al. [36] adopted a part-wise sparse representation approach for matching plastic surgery-altered faces. They employed the intensity characteristics of the principal component analysis (PCA)-based representation of the six facial components cropped from each face image. The facial components were then fused to determine the sparse representation error. A match is found if the probe sample produces smallest representation error to a test sample. In [37, 38], the multimodal biometrics, such as holistic face information and the periocular region, were adopted. Then, the features were extracted using shape local binary texture and Gabor. In [22], the method of face analysis for commercial entities (FACE) was adopted. The FACE utilizes correlation index obtained from defined subregions between two images. By correcting for illumination problem using self-quotient image, an improved performance was obtained using the FACE method. With particular interest in rhytidectomy, it is worth pointing out that though the recognition results in [2, 22] suggest that the algorithms have tried to address the challenges in face recognition in the event of rhytidectomy, there is still a significant scope for further improvement.

1.2 Our contributions and paper organization

In the following, we summarize the main contributions of this paper.

• We propose illumination normalization steps that reduce image dependency on illumination direction and control edge extraction sensitivity to illumination for the purpose of face recognition. The proposed illumination normalization steps are obtained from fusing RGB normalization (rgbN) method and the non-linear pixel power transform method termed GC for color images. We term this proposed steps the rgb-gamma encoding (rgbGE) technique.

• Under the assumption that the shape of the facial components might remain unchanged after surgery, we propose edge-based Gabor face representation for face recognition of surgically altered face images. The shape of significant facial components is retained by extracting gradient magnitude information from the rgbGE-normalized face image so as to minimize the intra-subject variations due to surgery.

• By means of experimental results, we show that the edge-based Gabor face representation approach performs significantly well in a simple nearest neighbor search face recognition framework. And the robustness of the proposed approach is investigated first with typically investigated problems such as pose, expression, and illumination problems and then plastic surgery.

The rest of this paper is organized as follows. In Section 2, the illumination normalization is presented. We present in Section 3 the proposed edge-based Gabor face representation approach. In Section 4, the face recognition experiment using the proposed face representation approach is presented. Finally, conclusions are drawn in Section 5.

2 Illumination normalization

As highlighted earlier, the changes in illumination condition results in false edges with respect to edge information extraction, and this has to be properly addressed. In this section, a brief review of the face reflectance model is firstly provided in such a manner that establishes a coherent basis for presenting the proposed illumination normalization technique. Subsequently, the proposed technique and the related step-by-step procedure for actualizing the technique are presented.

2.1 Face reflectance model

Light reflections from most surfaces are of two basic types, namely, the diffuse and specular reflections. The diffuse reflection defines the case where the incident light is reflected equally in all directions [39] and is well described by the Lambertian model [40]. The specular reflection for a smooth surface defines the case where the incident light is reflected in a mirror-like direction from the surface [41]. These reflections are often modelled using the Phong reflectance model [42].

To model a typical image captured using RGB camera sensor, we use the dichromatic reflection model described by Shafer [43], which includes the Lambertian term and the Phong's specular term. This model is given by

$${\mathit{I}}_{\mathit{k}}\left(\mathit{c}\right)={\mathit{w}}_{\mathrm{d}}\left(\mathit{c}\right){\displaystyle \underset{\mathit{w}}{\int }{\mathit{S}}^{\mathit{c}}\left(\mathit{\lambda }\right)\mathit{E}\left(\mathit{\lambda }\right){\mathit{C}}_{\mathit{k}}\left(\mathit{\lambda }\right)\mathit{d\lambda }}+{\mathit{w}}_{\mathrm{s}}\left(\mathit{c}\right){\displaystyle \underset{\mathit{w}}{\int }\mathit{E}\left(\mathit{\lambda }\right){\mathit{C}}_{\mathit{k}}\left(\mathit{\lambda }\right)\mathit{d\lambda }},\mathit{k}=\mathit{r},\mathit{g},\mathit{b}$$

(1)

$$={\mathit{w}}_{\mathrm{d}}\left(\mathit{c}\right){\mathit{D}}_{\mathit{k}}\left(\mathit{c}\right)+{\mathit{w}}_{\mathrm{s}}\left(\mathit{c}\right){\mathit{G}}_{\mathit{k}},$$

(2)

where ${\mathit{D}}_{\mathit{k}}\left(\mathit{c}\right)={\displaystyle \underset{\mathit{w}}{\int }}{\mathit{S}}^{\mathit{c}}\left(\mathit{\lambda }\right)\mathit{E}\left(\mathit{\lambda }\right){\mathit{C}}_{\mathit{k}}\left(\mathit{\lambda }\right)\mathit{d\lambda }$, ${\mathit{G}}_{\mathit{k}}={\displaystyle \underset{\mathit{w}}{\int }}\mathit{E}\left(\mathit{\lambda }\right){\mathit{C}}_{\mathit{k}}\left(\mathit{\lambda }\right)\mathit{d\lambda }$, I _k  = {I_r, I_g, I_b} is the color vector of image intensity, λ is the wavelength of the light, S^c(λ) is the spectral reflectance on a surface point c (where c is of spatial coordinates {x, y}). E(λ) is the spectral power distribution of the incident light, and C _k (λ) is the spectral sensitivity of the sensor. The terms w_d and w_s are the diffuse and specular terms of the incoming light, respectively. The first part of the right-hand side of (2) is the diffuse component, while the second part is the specular component.

For the color vectors {r, g, b}, (2) can be rewritten as

$${\mathit{I}}_{\mathit{k}}\left(\mathit{c}\right)=\left[\begin{array}{l}{\mathit{w}}_{\mathrm{d}}\left(\mathit{c}\right){\mathit{D}}_{\mathrm{r}}\left(\mathit{c}\right)+{\mathit{w}}_{\mathrm{s}}\left(\mathit{c}\right){\mathit{G}}_{\mathrm{r}}\\ {\mathit{w}}_{\mathrm{d}}\left(\mathit{c}\right){\mathit{D}}_{\mathrm{g}}\left(\mathit{c}\right)+{\mathit{w}}_{\mathrm{s}}\left(\mathit{c}\right){\mathit{G}}_{\mathrm{g}}\\ {\mathit{w}}_{\mathrm{d}}\left(\mathit{c}\right){\mathit{D}}_{\mathrm{b}}\left(\mathit{c}\right)+{\mathit{w}}_{\mathrm{s}}\left(\mathit{c}\right){\mathit{G}}_{\mathrm{b}}\end{array}\right]=\left[\begin{array}{l}{\mathit{I}}_{\mathrm{r}}\left(\mathit{c}\right)\\ {\mathit{I}}_{\mathrm{g}}\left(\mathit{c}\right)\\ {\mathit{I}}_{\mathrm{b}}\left(\mathit{c}\right)\end{array}\right]$$

(3)

Equation 3 describes the intensity components, which comprise diffuse and specular reflections for an RGB image captured in uncontrolled lighting environment. Mathematically, the objective of the proposed normalization technique is to reduce/eliminate the dependency of I _k (c) on the factors w_s, G_r, G_g, and G_b. To achieve this objective, we employ the merits of RGB and GC normalizations. The RGB normalization will address the directional lighting effect, while the GC normalizes a face image to a pre-defined ‘canonical form’ , which fades away the effect of illumination.

2.2 RGB normalization

The normalized RGB (Nrgb) is used in [35, 44]. The Nrgb is expressed by [44]

$${\mathit{\beta }}_{\mathit{k}}\left(\mathit{c}\right)=\frac{{\mathit{I}}_{\mathit{k}}\left(\mathit{c}\right)}{{\mathit{I}}_{\mathrm{r}}\left(\mathit{c}\right)+{\mathit{I}}_{\mathrm{g}}\left(\mathit{c}\right)+{\mathit{I}}_{\mathrm{b}}\left(\mathit{c}\right)},$$

(4)

where β _k  = {β_r, β_g, β_b} represents each color channel.

The computation of (4) results in the removal of intensity variations from the image, so that the RGB components of the image specify color only, and no luminance. This ensures that the normalized image becomes insensitive to changes in illumination direction.

2.3 Gamma correction

Gamma correction is a non-linear operation generally used to control image overall brightness. It is simply defined by the power law expression with respect to the input image I_input and output image I_output, as

$${\mathit{I}}_{\mathrm{output}}={\mathit{I}}_{\mathrm{input}}^{\mathit{\gamma }},$$

(5)

where γ is the exponent of the power function. Usually, the gamma value can be between the range [0,1] and is referred to as the encoding gamma. Gamma encoding, which is a form of non-linear transformation of pixels, enhances the local dynamic range of the images in dark or shadowed regions while compressing it in bright regions and at highlights [45]. For a given image I(c), the gamma encoding transform is expressed as

$${\mathit{G}}^{\prime }\left(\mathit{c}\right)={\mathit{I}}^{1/\mathit{\gamma }}\left(\mathit{c}\right)$$

(6)

In essence, the transformation technique corrects the problem of non-uniform intensity, where too much bits belong to high intensities and too few bits to low intensities.

2.4 Fusion of Nrgb and GC

The proposed illumination normalization technique rgbGE fuses the merits of RGB normalization and GC described above in order to compensate for illumination problem. The steps in the fusion of Nrgb and GC techniques are presented as follows:

• Step 1: Separate the image I _k (c) into the respective RGB color channels, I_r(c), I_g(c), and I_b(c).

• Step 2: Obtain the square magnitude I_m(c) of the separated RGB color channel images in step 1; thus,

  $${\mathit{I}}_{\mathrm{m}}\left(\mathit{c}\right)=\sqrt{{\mathit{I}}_{\mathrm{r}}^2\left(\mathit{c}\right)+{\mathit{I}}_{\mathrm{g}}^2\left(\mathit{c}\right)+{\mathit{I}}_{\mathrm{b}}^2\left(\mathit{c}\right)}$$

  (7)

• Step 3: Normalize the images I_r(c), I_g(c), and I_b(c) in step 1 by the I_m(c) (step 2) as follows:

  $$\begin{array}{l}{\mathit{I}}_{\mathrm{r}}^{\prime }\left(\mathit{c}\right)=\frac{{\mathit{I}}_{\mathrm{r}}\left(\mathit{c}\right)}{{\mathit{I}}_{\mathrm{m}}\left(\mathit{c}\right)}\\ {\mathit{I}}_{\mathrm{g}}^{\prime }\left(\mathit{c}\right)=\frac{{\mathit{I}}_{\mathrm{g}}\left(\mathit{c}\right)}{{\mathit{I}}_{\mathrm{m}}\left(\mathit{c}\right)}\\ {\mathit{I}}_{\mathrm{b}}^{\prime }\left(\mathit{c}\right)=\frac{{\mathit{I}}_{\mathrm{b}}\left(\mathit{c}\right)}{{\mathit{I}}_{\mathrm{m}}\left(\mathit{c}\right)}\end{array}$$

  (8)

• The essence of this step is to reduce the image intensity variation so that the RGB components of the image specify color only, and no luminance.

• Step 4: Compute the gamma encoding transformation of the I_m(c) in step 2.

  $${\mathit{G}}_{\mathrm{m}}^{\prime }\left(\mathit{c}\right)={\mathit{I}}_{\mathrm{m}}^{1/\mathit{\gamma }}\left(\mathit{c}\right),\mathit{\gamma }\in \left[0,1\right]$$

  (9)

• Step 5: Multiply the result of step 3 by ${\mathit{G}}_{\mathrm{m}}^{\text{'}}\left(\mathit{c}\right)$ as shown below:

  $$\begin{array}{l}{\mathit{f}}_{\mathrm{r}}^{\prime }\left(\mathrm{c}\right)={\mathit{I}}_{\mathrm{r}}^{\prime }\left(\mathrm{c}\right){\mathit{G}}_{\mathrm{m}}^{\prime }\left(\mathrm{c}\right)\\ {\mathit{f}}_{\mathrm{g}}^{\prime }\left(\mathrm{c}\right)={\mathit{I}}_{\mathrm{g}}^{\prime }\left(\mathrm{c}\right){\mathit{G}}_{\mathrm{m}}^{\prime }\left(\mathrm{c}\right)\\ {\mathit{f}}_{\mathrm{b}}^{\prime }\left(\mathrm{c}\right)={\mathit{I}}_{\mathrm{b}}^{\prime }\left(\mathrm{c}\right){\mathit{G}}_{\mathrm{m}}^{\prime }\left(\mathrm{c}\right)\end{array}$$

  (10)

• When the I_r(c), I_g(c), and I_b(c) in step 3 are recombined to form a color image, the resultant image is of low contrast. Step 5 is used to restore the contrast level to a level adequate for further processing.

• Step 6: By concatenating the expressions in (10) along the third dimension, we obtain the rgbGE-normalized image; thus,

  $${\mathit{\Psi }}_{\mathit{k}}\left(\mathit{c}\right)={\mathit{f}}_{\mathrm{r}}^{\prime }\left(\mathit{c}\right){\oplus }^3{\mathit{f}}_{\mathrm{g}}^{\prime }\left(\mathit{c}\right){\oplus }^3{\mathit{f}}_{\mathrm{b}}^{\prime }\left(\mathit{c}\right),$$

  (11)

where Ψ _k (c) is the rgbGE-normalized image, and ⊕ ³ symbolizes concatenation along the third dimension.

By this transformation, the illumination problem in the original image is compensated, and edge information obtained from Ψ _k (c) will have little or no false edges. It is important to note that the working principle of the rgbGE is based on the dichromatic reflectance model, which does not consider the ambient component. Hence, the performance of the technique will be significantly affected by outdoor captured images that mostly have ambient lighting components. More also, it should be noted that the variation in the degree of illumination problem across all images in a given database also affects the performance of rgbGE. Basically, the rgbGE performs well only when the degree of illumination variation across all images in a database is insignificant.The main idea behind the proposed illumination normalization technique is to minimize the intra-subject variation due to illumination as well as skin texture differences. In Figure 2, example images (images of a person) from three datasets are used to illustrate the performance of the proposed illumination normalization technique under varying lighting conditions (but without any ambient component). A number of images for each subject with varying degree of lighting problem are used in the illustration. It can be seen from Figure 2 that the normalized images of the subjects appear more similar to each other.

Figure 2

Example images from three datasets. (a1, b1, and c1) The original images of three different subjects. (a2, b2, and c2) The corresponding normalized images obtained with the proposed preprocessing method.

3 Edge-based Gabor face representation

As can be observed in image 1 of Figure 2b, the pre-surgery face images (the images on the first two columns) and the post-surgery images (the image on the third column) show a great amount of skin texture changes. Such differences between the images of the same subject are likely to impact on the face recognition accuracy. A plausible solution is to exploit the face information that are not likely to be affected by plastic surgery. Hence, we exploit the shape of the facial components, i.e., the shape of the eyes, nose (nostrils), eyebrow, and mouth that do not change after plastic surgery procedures. We put forward that this frame of reference serves as a platform for constructing robust and efficient feature descriptors for recognizing surgically altered face images. Under these assumptions, we utilize edge information, which are dependent on the shapes of the significant facial components of the face to address the intra-subject variations due to plastic surgery procedures. The basic idea of the proposed edge-based Gabor face representation approach is aimed at mitigating the intra-subject variations induced by plastic surgery procedures. This is achieved via computing the edge gradient magnitude of the illumination-normalized image. Applying Gabor wavelet on the resultant edge gradient magnitude image accentuates on the uniqueness of significant facial components, which enlarges the discrimination margin between different person face images. These processes are discussed below.

3.1 Edge gradient magnitude computation

Let the grayscale version of the illumination-normalized image Ψ _k (c) be denoted as Ψ(c). The edge information g(c) of the image Ψ(c) is obtained via the computation of the gradient magnitude of the image; thus [46],

$$\mathit{g}\left(\mathit{c}\right)=\sqrt{{\left(\partial \mathit{x}\mathit{\Psi }\left(\mathit{c}\right)\right)}^2+{\left(\partial \mathit{y}\mathit{\Psi }\left(\mathit{c}\right)\right)}^2},$$

(12)

where $\partial \mathit{x}\mathit{\Psi }=\left(\mathit{\Psi }\otimes \frac{\mathit{\delta }}{\mathit{\delta x}}\right)\otimes \mathit{S}$ and $\partial \mathit{y}\mathit{\Psi }=\left(\mathit{\Psi }\otimes \frac{\mathit{\delta }}{\mathit{\delta y}}\right)\otimes \mathit{S}$ denote partial derivatives, with S as smoothening Sobel filter function.

The false edges in the gradient magnitude image g(c) are substantially reduced when the rgbGE normalization technique is employed. This can be observed in Figure 3a.In Figure 3, the gradient of the rgbGE normalized face images (three images of a subject) is compared with the original image without correction and with various illumination normalization methods such as LT, HE, and GC. It can be seen from the figure that the gradient of the rgbGE face images shows less facial appearance differences in comparison to the other methods. In subsequent subsections, the Gabor encoding process is given in detail.

Figure 3

Illustration of edge gradient magnitude for intra-subjects with different illumination normalization techniques. (a) rgbGE, (b) LT, (c) without normalization, (d) HE, and (e) GC.

3.2 Gabor wavelets

Gabor wavelets (kernels and filters) have proven useful in pattern representation due to their computational properties and biological relevance [3, 7, 11, 19]. It is a powerful tool that provides spatial domain and frequency domain information on an object.

The Gabor kernels can be expressed by [47]

$${\mathit{\psi }}_{\mathit{\mu },\mathit{\nu }}\left(\mathit{c}\right)=\frac{{∥{\mathit{l}}_{\mathit{\mu },\mathit{\nu }}∥}^2}{{\mathit{\sigma }}^2}{\mathit{e}}^{\left(-{∥{\mathit{l}}_{\mathit{\mu },\mathit{\nu }}∥}^2{∥\mathit{c}∥}^2/2{\mathit{\sigma }}^2\right)}\left[{\mathit{e}}^{\mathit{il\mu },{\mathit{\nu }}^{\mathit{c}}}-{\mathit{e}}^{-{\mathit{\sigma }}^2/2}\right],$$

(13)

where μ and ν define the orientation and scale of the Gabor kernels, respectively, c = (x, y), ∥ . ∥ denotes the norm operator. The term l_μ,ν is defined as [11]

$${\mathit{l}}_{\mathit{\mu },\mathit{\nu }}={\mathit{l}}_{\mathit{\nu }}{\mathit{e}}^{\mathit{i\phi \mu }},$$

(14)

where ${\mathit{l}}_{\mathit{\nu }}={\mathit{l}}_{\text{max}}/{\mathit{s}}_{\mathrm{f}}^{\mathit{\nu }}$ and φ _μ  = πμ/8. l_max is the maximum frequency, s_f is the spacing factor between kernels in the frequency domain [47], and σ is a control parameter for the Gaussian function.

The family of self-similar Gabor kernels in (13) is generated from a mother wavelet by selecting different center frequencies (scales) and orientations. In most cases, the Gabor wavelets at five scales ν ∈ {0, …, 4} and eight orientations μ ∈ {0, …, 7} are used [11, 19]. This paper uses Gabor kernels at five scales and eight orientations with the following parameters: σ = 2π, l_max = π/2, ${\mathit{s}}_{\mathrm{f}}=\sqrt{2}$[11, 19] as shown in Figure 4. The edge image g(c) is convolved with a family of Gabor kernels at five scales and eight orientations; thus,

Figure 4

Gabor wavelets at five scales and eight orientations. (a) The real part of the Gabor wavelet. (b) The magnitude part of the of the Gabor wavelet.

$${\mathit{{\rm O}}}_{\mathit{\mu },\mathit{\nu }}\left(\mathit{c}\right)=\mathit{g}\left(\mathit{c}\right)\ast {\mathit{\psi }}_{\mathit{\mu },\mathit{\nu }}\left(\mathit{c}\right),$$

(15)

where ∗ denotes the convolution operator, and Ο_μ,ν(c) is the corresponding convolution result at different scales ν and orientations μ.

Applying the convolution theorem, each Ο_μ,ν(c) from (15) can be derived via the fast Fourier transform (FFT) [11]:

$${\mathit{{\rm O}}}_{\mathit{\mu },\mathit{\nu }}\left(\mathit{c}\right)={\Im }^{-1}\left\{\Im \left\{\mathit{g}\left(\mathit{c}\right)\right\}\Im \left\{{\mathit{\psi }}_{\mathit{\mu },\mathit{\nu }}\left(\mathit{c}\right)\right\}\right\},$$

(16)

where ℑ and ℑ⁻¹ denote the Fourier transform and its inverse, respectively.

The Gabor wavelet representation of the edge image g(c) is shown in Figure 5, where only the magnitude responses of Ο_μ,ν(c) is used to construct the Gabor feature. Having computed Ο_μ,ν(c), the augmented feature vector, namely the edge-based Gabor magnitude (EGM) feature matrix Ζ^(p), is obtained by concatenating each Ο_μ,ν(c) already downsampled by a factor p, where p = 64 becomes ${\stackrel{⌣}{\mathit{{\rm O}}}}_{\mathit{\mu },\mathit{\nu }}\left(\mathit{c}\right)$ and normalized to zero mean and unit variance. By so doing, the augmented EGM feature matrix Ζ^(p) encompasses every possible orientation selectivity, spatial locality, and frequency of the representation result; thus,

Figure 5

Edge-based Gabor magnitude representation of a sample face image.

$$\begin{array}{l}{\mathit{{\rm Z}}}^{\left(\mathit{p}\right)}={\left({\stackrel{⌣}{\mathit{{\rm O}}}}_{0,0,\mathit{jk}}^{\mathit{T}}{\stackrel{⌣}{\mathit{{\rm O}}}}_{0,1,\mathit{jk}}^{\mathit{T}}\cdots {\stackrel{⌣}{\mathit{{\rm O}}}}_{7,4,\mathit{jk}}^{\mathit{T}}\right)}^{\mathit{T}}\\ ={\mathit{Z}}_{\mathit{rq}}\end{array}$$

(17)

where T is the transpose operator, ${\stackrel{⌣}{\mathit{{\rm O}}}}_{\mathit{\mu },\mathit{\nu },\mathit{jk}}$ are the respective J × K downsampled image matrices with orientation μ and scale ν, and Z _rq are the elements of the R × Q EGM feature matrix. The procedure for obtaining the EGM feature is clearly illustrated in Figure 6.

Figure 6

Overview of the stages taken into consideration for describing a face image. The stages enclosed by the broken line highlight the proposed EGM descriptor.

3.3 Dimensionality reduction and discriminant analysis

The EGM features are of high dimensional space, such that Ζ^(p) ∈ R^N, where N is the dimensionality of the vector space. To address the dimensionality problem and still retain the discriminating information for identifying a face, we apply the two-stage (PCA + LDA) approach [14, 48]. Each same person face is defined as belonging to a class. Let ω₁, ω₂, ⋯, ω _L and N₁, N₁, ⋯, N _L denote the classes and the number of images within each class, respectively. Let M₁, M₁, ⋯, M _L and M be the mean values of the classes and the grand mean value. The within-class scatter matrix S _ω and the between-class scatter matrix S_b are defined as [14, 48]

$${\mathit{S}}_{\mathit{\omega }}={\displaystyle \sum _{\mathit{i}=1}^{\mathit{L}}\mathit{P}\left({\mathit{\Omega }}_{\mathit{i}}\right)}\mathit{ϵ}\left\{\left({\mathit{Y}}^{\left(\mathit{p}\right)}-{\mathit{M}}_{\mathit{i}}\right){\left({\mathit{Y}}^{\left(\mathit{p}\right)}-{\mathit{M}}_{\mathit{i}}\right)}^{\mathit{T}}\left|{\mathit{\Omega }}_{\mathit{i}}\right.\right\}$$

(18)

$${\mathit{S}}_{\mathrm{b}}={\displaystyle \sum _{\mathit{i}=1}^{\mathit{L}}\mathit{P}}\left({\mathit{\Omega }}_{\mathit{i}}\right)\left({\mathit{M}}_{\mathit{i}}-\mathit{M}\right){\left({\mathit{M}}_{\mathit{i}}-\mathit{M}\right)}^{\mathit{T}},$$

(19)

where Y^(p) is the most expressive feature of the original data Ζ^(p) obtained with a PCA step so that LDA is implemented in the PCA subspace [14]. P(Ω _i ) is the probability of the i th class, and L denotes the number of classes.

The LDA derives a projection matrix A that maximizes the Fisher's discriminant criterion:

$$\mathit{J}\left(\mathit{A}\right)=arg\underset{\left(\mathit{A}\right)}{max}\frac{\left|\mathit{A}{\mathit{S}}_{\mathrm{b}}{\mathit{A}}^{\mathit{T}}\right|}{\left|\mathit{A}{\mathit{S}}_{\mathit{\omega }}{\mathit{A}}^{\mathit{T}}\right|}$$

(20)

The Fisher's discriminant criterion is maximized when A consists of the eigenvectors of the matrix ${\mathit{S}}_{\mathit{\omega }}^{-1}{\mathit{S}}_{\mathrm{b}}$[48].

$${\mathit{S}}_{\mathit{\omega }}^{-1}{\mathit{S}}_{\mathrm{b}}\mathit{A}=\mathit{A\Delta },$$

(21)

where A and Δ are the eigenvector and eigenvalue matrices of ${\mathit{S}}_{\mathit{\omega }}^{-1}{\mathit{S}}_{\mathrm{b}}$, respectively. The two-stage (PCA + LDA) dimensionality reduction approach is employed to maximize the between-class variations and minimize the within-class variations of the projected face subspace.For validation purpose, the face recognition performance of Gabor, i.e., with the original gray-level face images, and the EGM, i.e., with the gradient magnitude face images, are shown in Figure 7.

Figure 7

Face recognition performances using Gabor with original gray-level face images and gradient magnitude faces.

It can be observed that the use of gradient magnitude image improved the performance of the Gabor descriptor significantly compared to using the original gray-level face images. At this point, it is important to note that the illumination normalization technique that can be used with the EGM include any of the existing normalization techniques discussed in this work. For simplicity, and hence forth, we use the acronym EGM-rgbGE to represent the EGM that employs the rgbGE illumination normalization technique, EGM-HE to represent the EGM that employs the HE illumination normalization technique, and EGM-GC to represent the EGM that employs the GC illumination normalization technique.

4 Face recognition experiment

In this section, the results of the proposed EGM-based face recognition method on a plastic surgery database [2], the Georgia Tech (GT) face database [49], and the Labeled Faces in the Wild (LFW) database [50] are presented. The details of the datasets and the experimental setups for the face recognition experiment are provided. We show through simulations the result of the proposed method compared with the existing face recognition methods.

4.1 Datasets and experimental setup

4.1.1 Georgia Tech dataset

The Georgia Tech face database [49] contains of 750 color images of 50 subjects, some of which where captured during different sessions. These images comprise variations in illumination direction, scale, pose, and expression; see sample images in Figure 8. The images were manually cropped and resized to size 128 × 128. The database is partitioned into training and testing sets. The number of training and test images is selected to resemble a real-time scenario where only one image per person is tested on a large database in which there exist numerous images of the same person.

Figure 8

Original sample faces from the GT dataset.

4.1.2 Plastic surgery dataset

The plastic surgery dataset [2] contains frontal face images of plastic surgery-altered faces, which vary by scale and small expression, small pose, and majorly by plastic surgery procedures. There are only two images (pre-surgery and post-surgery) per subject available. The dataset consists of different plastic surgery procedures such as rhinoplasty (nose surgery), blepharoplasty (eyelid surgery), brow lift, skin peeling, and rhytidectomy (face and mid-face lift surgery). According to Bhatt et al. [24, 25], global surgeries like rhytidectomy severely impacted on their recognition algorithm. To investigate this impact, we used the rhytidectomy plastic surgery dataset, which we arranged to combine full-face lift and mid-face lift surgery images. The rhytidectomy dataset consists of two images per subject. There are 321 subjects; hence, the total number of images is 642. The samples of the rhytidectomy images from the dataset are shown in Figure 9.

Figure 9

Rhytidectomy plastic surgery sample faces, pre-surgery (top row), post-surgery (bottom row).

Motivated by the fact that a face can be identified from its mirror image [51], we include a mirror version of the pre-surgery images in the database in order to increase the sample size of each subject. Hence, there are a total of 963 face images cropped and resized to size 128 × 128. For this database, we consider two scenarios s 1 and s 2 in the experimental setup. In s 1, out of the 321 subjects, we used 128 subjects for training and performance is evaluated on the remaining subjects, where the pre-surgery images are used as gallery and post-surgery images are used as the probe. In s 2, pre-surgery images of all the subjects are used for training and gallery while the post-surgery images are used as the probe.

4.1.3 Heterogeneous dataset

This database consists of both plastic surgery images and other images that are randomly selected from other databases. The main idea behind the use of this database structure in performance analysis of the EGM is to consider a typical real-world scenario of training-testing where the system is unaware of any plastic surgery cases. In this dataset, the images of 321 subjects with three images per subject are added to the plastic surgery database used above. This brings the total number of subjects to 642, with every subject having three images. For this database, we consider two scenarios s 1 and s 2 just like in the plastic surgery database above. In s 2, the pre-surgery images of all the subjects are used for training and gallery, while the post-surgery images are used as the probe.

4.1.4 Labeled Faces in the Wild (LFW) dataset

The LFW database [50] is a well-known database with gross pose, expression, and illumination problems. Out of all the subjects in the database, we choose subjects with up to three images per subject. Hence, our dataset consists of 869 subjects with images resized to size 128 × 128. For this database, there is only one scenario where 869 images are used as test set, while the training and gallery consist of the remaining images. The LFW sample face images are shown in Figure 10.

Figure 10

Original sample faces from the LFW dataset.

4.2 Experimental results

Here, we evaluate the proposed method on all the datasets presented above. In all the experimental setups presented above, the gamma value used in the illumination correction is chosen as γ = 0.75, but the value of γ can be user defined. We used a simple nearest neighbor classifier with the cosine distance measure in order to measure the similarity between two images. Experimental results are obtained for datasets with illumination correction and without illumination correction. The evaluation is of two categories. In the first category, the proposed method is evaluated on the GT face database and LFW database. The performance of the EGM in the case where the proposed rgbGE illumination normalization technique is used is compared to where GC, LT, and HE illumination normalization techniques are used for the GT. This performance comparison is also extended to the case where GC and HE illumination normalization techniques are used for the LFW. The results of the experiment on the GT face database are shown in Table 1 and Figure 11. A comparison of the recognition accuracy of the proposed EGM-rgbGE with existing methods on GT database is shown in Table 2. The results of the experiment on the LFW database are shown in Table 3 and Figure 12.

Table 1 Recognition performance comparisons of the proposed EGM-based face recognition method on Georgia Tech face database

Figure 11

Recognition rate for EGM-based face recognition method with different illumination normalization methods on GT database.

Table 2 Recognition rate comparisons with some existing methods on GT face database

Table 3 Recognition performance comparisons of the proposed EGM-based face recognition method on LFW Database

Figure 12

Recognition rate for EGM-based face recognition method with different illumination normalization methods on LFW database.

For the second category, we conduct an experiment on a rhytidectomy plastic surgery database. In Table 4 and Figure 13, the rank-1 recognition performance of the proposed EGM-rgbGE-based face recognition method on rhytidectomy plastic surgery database is compared to the existing face recognition methods. The recognition performance of the EGM-rgbGE on the heterogeneous database is shown in Figure 14.

Table 4 Recognition rate comparison with existing methods on plastic surgery database: a case of rhytidectomy

Figure 13

EGM recognition performance for plastic surgery database. The experimental scenarios s 1 and s 2 are observed on the basis of with and without rgbGE illumination normalization technique.

Figure 14

EGM recognition performance for heterogeneous database. The experimental scenarios s 1 and s 2 are observed on the basis of with and without rgbGE illumination normalization technique.

From the results in Figure 11 and Table 1, it can be observed that the EGM-LT did not improve on the rank-1 recognition rate (RR) of EGM without correction. This shows that it does not address the presence of false edges induced by illumination as is evident from Figure 3. However, it improved in terms of equal error rate (EER). On the other hand, the RR performances of EGM-HE and EGM-GC degraded in comparison with EGM without correction. In this case, such degradation may be associated with the introduction of more false edges to the edge information. The EGM-rgbGE-based face recognition method is observed to be more insensitive to performance degradation factors such as illumination, pose, and expression with 98% RR and 0.34% EER.

In Table 2, the performance of the proposed EGM-based face recognition method compared with some existing face recognition methods for the GT database is shown. The performance of the face recognition methods considered are sorted by their rank-1 recognition rates. The performance of EGM-rgbGE in terms of RR with no face alignment is significantly higher than those of all other methods with no face alignment. For the methods with face alignment, the EGM-rgbGE performed better except for the method by Wouter and Peter [19]; of course with face alignment, the EGM-rgbGE will perform better. In the case of LFW, the performance of the proposed method performs equally well as the existing results in the literature. The reader is referred to [57] for results on LFW database. We note at this point that the goal of the current paper is to demonstrate the efficiency of the proposed descriptor based face recognition method, not to compete in the LFW challenge.

In Figure 13, the proposed EGM-rgbGE method performs better in the s 2 scenario than in s 1 scenario. This performance difference may not be unconnected with the method of training and evaluation. In both s 1 and s 2 scenarios, the recognition rates are better when rgbGE normalization is used than when it is not incorporated into the EGM method. Table 4 shows the performance comparison of the EGM-rgbGE method with the existing methods on rhytidectomy plastic surgery database. The EGM-rgbGE rank-1 recognition rate of over 89% is a difference of 17.48% in comparison to granular approach [24, 25], 15.24% in comparison to FACE [22], and 2.56% in comparison to GROF [23]. However, it should be noted that face alignment was employed to obtain the result in the GPROF method, while the EGM-rgbGE result is without face alignment. The RR and EER results of the different experimental setups and scenarios for the plastic surgery database are shown in Table 5.

Table 5 Recognition performance of the proposed EGM-based face recognition method for the plastic surgery database

The result of the experiment on the heterogeneous database (a combination of the plastic surgery-altered images and non-surgery images) for training-testing, whereby the face recognition system is set to resemble a real-world scenario, unaware of any plastic surgery image, is shown in Figure 14. The experimental results in the two scenarios show how the proposed method, which considers the shape of facial components, performs across surgery and non-surgery images. Again, the proposed EGM-rgbGE method performs better in scenarios s 1 and s 2 in comparison with other existing methods. In both s 1 and s 2 scenarios, the recognition rates are better when the rgbGE normalization is used than when it is not incorporated into the EGM method. As can be observed, the result of the experiments in the case of heterogeneous database outperforms that of the experiments in the case of only-plastic-surgery database. This performance variation is expected since the average performance will be expectedly higher due to the inclusion of non-plastic surgery images in the database with lesser recognition challenges than purely plastic surgery database. The RR and EER results of the different experimental setups and scenarios for the heterogeneous database are shown in Table 6.

Table 6 RR and EER performance comparisons of the proposed EGM-based face recognition method for the heterogeneous database

5 Conclusion

We have presented the proposed edge-based Gabor feature representation approach for appearance representation of faces altered by plastic surgery procedures. The proposed face representation approach exploited the shape of significant facial components to address the intra-subject dissimilarity problem due to plastic surgery procedures. The face recognition experiment using the proposed face representation approach considered other unconstrained conditions, which included usual pose, illumination, and expression problems. Comparative experimental results on the GT and LFW databases were provided to show the robustness of the representation approach under usually experimented-on problems and then on the rhytidectomy plastic surgery database. The results indicate that the proposed face representation approach in a simple nearest neighbor search face recognition framework (with no face alignment) performed better than the previously reported methods in the literature on rhytidectomy plastic surgery database. In relation to the GT database, the proposed method performed better than the existing methods, except in comparison with one particular method, in which prior face alignment on the database was carried out, and 0.88% performance difference was reported. However, with face alignment on the database, it is believed that the proposed method will greatly outperform all methods reported so far in the literature. On the LFW database, our method provides up to 76.33% recognition rate. In future work, we will integrate texture insensitive and illumination invariant processes into a single processing step and investigate its performance with other plastic surgery procedures and cases of gross illumination problems.