Projective complex matrix factorization for facial expression recognition

In this paper, a dimensionality reduction method applied on facial expression recognition is investigated. An unsupervised learning framework, projective complex matrix factorization (proCMF), is introduced to project high-dimensional input facial images into a lower dimension subspace. The proCMF model is related to both the conventional projective nonnegative matrix factorization (proNMF) and the cosine dissimilarity metric in the simple manner by transforming real data into the complex domain. A projective matrix is then found through solving an unconstraint complex optimization problem. The gradient descent method was utilized to optimize a complex cost function. Extensive experiments carried on the extended Cohn-Kanade and the JAFFE databases show that the proposed proCMF model provides even better performance than state-of-the-art methods for facial expression recognition.

Facial expression recognition (FER) plays an important role in many applications related to computer vision and pattern recognition, such as human computer interface, surveillance, and multimedia [1,2,3,4,5]. Intensive techniques have been proposed for FER problem [6,7,8,9,10,11,12,13,14]. Much attention is put on the facial action coding system (FACS) approach which attempts to decompose facial expressions into varied action units and facial expression could be recognized based on the mixture of action units [6, 7]. A new perspective of using deep neural networks [8,9,10,11] to extract powerful temporal features hidden in facial images is also an interesting approach for FER, in particular for dynamic facial expression recognition. The dimensionality reduction technique that reduces the size of the feature space has been widely utilized because of its effectiveness on feature representation [12,13,14]. It is known that the changes of local appearance (e.g., eyes, nose, mouth) are usually related to facial expression variations; meanwhile, the global features that are related to the whole facial image may fail on expression analysis. Local facial components contain more discriminative information and outperform global features for face recognition [15]. A nonholistic representation and a low rank approximation of the data make better performance for an FER system [16].

Nonnegative matrix factorization (NMF) [12, 13] is among the most popular dimensionality reduction methods by its natural part-based representation ability. NMF decomposes the high-dimensional nonnegative data matrix into two low-rank nonnegative matrices with additional non-negativity constraints. Most NMF techniques estimate the linear subspace of the given data by the square of the Euclidean distance or the generalized Kullback-Leibler divergence. They work well when image noise is independent and identically distributed. However, for the data corrupted by outliers, the estimated subspace can be arbitrarily biased [17]. To overcome this drawback, researchers improved NMF in different distance metrics. Specifically, D. Kong et al. proposed a robust NMF by using L₂ and L₁ norms, where the noise was assumed to follow the Laplacian distribution [18]. Similarly, Earth Mover’s distance (EMD) and the Manhattan distance were also suggested in the work of Sandler et al. [19] and N. Guan et al. [20], respectively.

Besides, integrating constraints or using different metrics to modify the structure is also another attractive strategy to extend NMF. Starting from the ideas of SVD and NMF, for instance, Yuan et al. proposed a novel method called projective nonnegative matrix factorization (proNMF) [21]. Instead of operating two parameter matrices as NMF, there is only one computed matrix in proNMF. The coefficient matrix is replaced by the inner product of the base and the input matrix. As a result, there exists much fewer free-learned parameters. Furthermore, inspite of having no any extra regularization term, proNMF is still able to learn more spatially localized and part-based representations of visual patterns.

Recently, many FER algorithms have adopted NMF. Buciu and Pitas employed NMF for FER by enforcing local constraints to create a local NMF (LMF) model [22]. Nikitidis et al. incorporated discriminant constraints to build the supervised NMF learning method [23, 24]. Based on clustering-based discriminant analysis (CDA), the algorithm in [25] efficiently decomposes the provided data to discriminant parts and successfully extended the well-known NMF to subclass discriminant NMF (SDNMF). R. Zhi et al. [26] provided another extended NMF which imposed both sparse constraints on the basis matrix and graph-preserving criterions to improve the classification performance. Interpreting expressive images into two subspaces including the identity subspace and the expression subspace was a new model investigated in dual subspace nonnegative matrix factorization (DSNMF) [27] and dual subspace nonnegative graph embedding [28]. It can be seen that, in order to extract representative expression facial features, most existing algorithms tried to integrate label information via different constraints to derive supervised NMF. Therefore, the overall performance highly depends on the expression label which is insufficient to characterize data by the variations of pose, illumination, etc.

How to develop an unsupervised learning algorithm that performs efficiently FER was aimed in this paper. Encouraged by the localized and part-based representations of proNMF as well as the challenges on choosing a suitable metric that quantifies the approximation error in the NMF model, we proposed a new dimensionality reduction approach, named projective complex matrix factorization (proCMF). The proCMF model attempts to learn the localized basis that highlights the salient features of facial expressions; meanwhile, the reconstruction error affected by outliers was also minimized as much as possible. Moreover, the underlying intrinsic cosine dissimilarity referenced from the equivalence between two different metrics is also exploited. We also adopt a cosine-based distance measure to define an explicit mapping function from the space of pixel intensity into a high-dimensional sphere where complex projection is performed. The Euler formula was used for constructing isomorphism mapping from the real space to the complex space. In this context, the cosine dissimilarity for approximating real data was replaced by the Frobenius norm to evaluate the reconstruction error of complex approximation. After transforming the real data into the complex field, the obtained complex matrix was approximately factorized into a product of two low-rank matrices through an unconstraint complex optimization problem. The proposed method addresses the general problem of finding projections that enhance class separability without attached label information in the reduced dimensional space. There is no any extra regularization term in the objective functions that reduce the complication on solving the optimization problem. Furthermore, the representation of newly coming samples in the FER framework by using only the trained projection matrix is one of the important factors on decreasing the computational complexity.

In summary, the contributions of this paper are shown as follows.

First, we construct a new dimensionality reduction model for learning low rank projection in the complex domain. The adoption of cosine dissimilarity and the Frobenius norm significantly enhance the performance of the FER system.

Second, proCMF can be performed without limiting the sign of data. Our proposed method can be applied to both negative and positive data, which yields extension on real-world applications. Therefore, several operations that can extract complex features, such as the short-time Fourier transform, are going to be utilized directly instead of their absolute values (magnitude/power spectrogram) on sound processing.

To satisfy nonnegative requirement, NMF usually uses various strategies on minimizing a function, which leads to computational complexity. On the contrary, the significant superiority compared to NMF approaches of the proposed proCMF is to construct an unconstraint optimization problem that simplified the framework of extracting the basis and intrinsic features.

The rest of this paper is organized as follows. In Section 2, we recall some basic facts about proNMF. The motivation, formulation, and algorithms of the proposed model as well as its optimal solution are presented in Section 3. Some optimization techniques and computation methods are also described in this section. The convincing experimental results of the proposed approach on FER are depicted in Section 4. Finally, conclusion and future work are drawn in Section 5.

The problem on ensuring the learned basis vectors to be part-based standard NMF was treated by projective nonnegative matrix factorization (proNMF) [29, 30]. ProNMF approximately factorizes a projection matrix P into a positive low-rank matrix W and its transpose W^T such that X = PX. The proNMF optimization problem has the following form:

It can be seen that the projection matrix is constructed only from the basic matrix, so the optimization problem in (1) mainly takes account of one variable W. Using the popular multiplicative update algorithm [13, 29] to solve (1), one can update W iteratively from

Observations on cosine dissimilarity

Assume that we are given the representations of two images I₁ and I₂ that are written by the N-dimensional vector x _i  (i = 1, 2) in the lexicographic order. First, x _i  ∈ ℝ^N is normalized to get x _i (c) ∈ [0, 1] where c is the element vector index or the vector spatial location. Then, x _i is mapped into the ℝ^2N sphere by

where

and ‖Z(x _i )‖ = 1.

We have

Recall that the cosine distance measure between two vectors x and y is given by

If the distance between Z(x₁) and Z(x₂) have the form

then it is equal to a cosine-based distance measure and

It can be seen that, if I₁ ≈ I₂, e.g., ∀c, x₁(c) − x₂(c) ≈ 0, then d(Z(x₁), Z(x₂)) → 0. This implies that if the two images are unrelated, then their local elements are unmatched.

Moreover, the mapping function (3) from ℝ^N to ℝ^2N is equivalent to a mapping function f : ℝ^N → ℂ^N defined by:

where the Euler’s formula [31] is

Therefore, the cosine dissimilarity of a data pair in the input real space equals to the Frobenious distance of the corresponding data pair in the complex domain. It is known that the robustness of cosine dissimilarity in the real domain has been found to suppress outliers [32]. With the idea of utilizing different robust similarity metrics to extend NMF, we introduce a new dimensionality reduction method (proCMF) which relates to conventional proNMF and uses the cosine dissimilarity metric as a measurement of the reconstruction error. However, the complexity of optimizing the real function with the cosine distance is wisely addressed by converting to a complex optimization problem with the Frobenius norm which is described in the next sections.

Problem formulation

In this section, we formulate the problem of multi-variants data factorization within the framework of complex data decomposition. Given the sample dataset X = [x₁, x₂, …, x _M ], x _i  ∈ ℝ^N, we convert the real data matrix X ∈ ℝ^N × M to a complex matrix Z ∈ ℂ^N × M by the mapping (8) and perform matrix factorization in this complex feature space.

The basic idea of proCMF is the coefficient of each data point z _i  (i = 1, 2, …, M) that lies within the subspace spanned by the column vectors of one projection matrix. The coefficient matrix H ∈ ℂ^K × M is obtained by linear transformation from samples. More specifically, given a matrix Z ∈ ℂ^N × M, we need to find out two matrices W ∈ ℂ^N × K and H ∈ ℂ^K × M to minimize \( {\left\Vert \mathbf{Z}-\mathbf{WH}\right\Vert}_F^2\ \mathrm{s}.\mathrm{t}\ \mathbf{H}=\mathbf{VZ} \) where V ∈ ℂ^K × N is the projection matrix. The proCMF objective function is as follows:

Complex-valued gradient decent method

It can be seen that (10) is a nonconvex minimization problem with respect to both variables W and V. Therefore, they are impractical to obtain the optimal solution. This NP-hard problem can be tackled by applying the block coordinate descent (BCD) with two matrix blocks [33] to obtain a local solution by the following scheme:

Given an initial W⁽⁰⁾, we find the optimal solution V^(t+1) such that:

Because of no nonnegative constraint, the basis can be updated simply by the Moore–Penrose pseudoinverse [34]

To find optimal solutions of (11), we use the complex-valued gradient descent algorithm (CGD). Here, (11) is considered as a real-valued function that is needed to be minimized subject to the complex variable V. Generally, the problem of one scalar function of a complex variable (11) is defined as to solve the following unconstrained optimization problem:

where

and (.)^H is the matrix Hermitian operation.

Let V = Re(V) + iIm(V) where i is the imaginary unit and i² = − 1. Then, f(V) can be viewed as a real bivariate function of its real and imaginary components.

In most of complex-variable optimization problems, the objective functions are the real-valued functions of complex arguments. They are not analytic on the complex plane and do not satisfy Cauchy-Riemann conditions. Brandwood’s analytic theory [35] can be applied to overcome this common problem. Recall that if f(V) satisfies Brandwood’s analytic condition, i.e., f(V) is analytic with respect to the complex-valued variable V and its complex conjugate \( \overline{\mathbf{V}} \) where V and \( \overline{\mathbf{V}} \) are treated as independent variables, then the first-order Taylor expansion of \( f\left(\mathbf{V},\overline{\mathbf{V}}\right) \) is as follows:

and the complex gradient of f(V) is defined as

Therefore, the function f(V) is treated as \( f\left(\mathbf{V},\overline{\mathbf{V}}\right) \), where

The gradient of \( f\left(\mathbf{V},\overline{\mathbf{V}}\right) \) with respect to V is given by:

The gradient decent method for the unconstrained optimization problem in (13) builds a sequence {V^(t)}_{t ∈ ℕ} according to the following iterative form:

where β^(t) is the step size variable, which is a small positive constant minimizing \( f\left({\mathbf{V}}^{(t)}-{\beta}^{(t)}{\nabla}_{\overline{\mathbf{V}}}f\left({\mathbf{V}}^{(t)},{\overline{\mathbf{V}}}^{(t)}\right)\right) \) over ℝ. In this paper, backtracking line search, which is also known as the Armijo rule [36], is used to estimate the step size. In this rule, \( {\beta}^{(t)}={\mu}^{k_t} \), where 0 < μ <1 and k _t is the first non-negative integer k such that:

There always exists a step length β^(t), which is among 1, μ¹, μ², … . A stationary point of (13) also exists among the limit points of {V^(t)}_{t ∈ ℕ} [37]. The iteration will be stopped when the solution is close to a stationary point. Practically, a common condition to check if a point V^(t) is close to a stationary point is:

where ε is a pre-defined threshold.

The following Algorithm 2 summarizes the optimization process of the proposed proCMF model.

We evaluated the proposed proCMF on FER. The classification capability of the derived encoding coefficient vectors was compared with various NMF-based methods.

Here, the minimization of the objective function \( \frac{1}{2}{\left\Vert {\mathbf{Z}}_{tr}-{\mathbf{W}}_{tr}{\mathbf{V}}_{tr}{\mathbf{Z}}_{tr}\right\Vert}^2 \) of proCMF was operated on the training phase that extracts new feature H _tr =V _tr Z _tr from the image training set Z _tr . After creating a projection matrix V _tr , for each new testing sample z _te in a query face image Z _te , we can easily obtain its corresponding projection h _te such that h _te =V _tr z _te . To make a decision about the facial expression class, the projected expression representation h _te will be fed to the nearest neighbor classifier. Then, the Euclidean distance between h _te and each training datum of H _tr is computed. The tested image is assigned to the class with the closest training datum.

Data description

The proposed algorithm is tested on two well-known datasets, the extended Cohn-Kanade (CK+) [38] and the Japanese Female Facial Expression (JAFFE) [39] datasets of six “basic” facial expressions (happiness, sadness, surprise, anger, disgust, and fear).

The CK+ dataset consists of 593 video sequences from 123 subjects. Each video sequence shows distinct facial expressions. For each expression of a subject, the last five frames in the videos are selected and these frames were treated as static different facial expressions. Some samples in the CK+ dataset are shown in Fig. 1.

Cropped face images of six facial expressions from the CK+ dataset [38]

Full size image

Altogether, the JAFFE dataset has 213 grayscale images of ten subjects posed 2–4 examples for each expressions. Figure 2 shows some figures in the JAFFE dataset.

Cropped face images of six facial expressions from the JAFFE dataset [39]

Full size image

Each cropped facial image in the datasets was isotropically scaled to the fixed size of 32 × 32 pixels.

Baselines and experiment settings

The proposed algorithm is compared to the following popular PCA and NMF algorithms: (1) basic NMF [4]; (2) projective NMF (proNMF) [9]; (3) convex NMF (conNMF) [40]; (4) weighted NMF (weiNMF), which assigns binary weights to the data matrix [41]; (5) NeNMF, which applies efficient Nesterov’s optimal gradient method in the optimization process [42]; (6) principal component analysis (PCA) [43]; (7) graph preserving sparse nonnegative matrix factorization (GSNMF) [26]; (8)(9) unsupervised and supervised robust nonnegative graph embedding (uRNGE) and (sRNGE), which are robust nonnegative graph embedding methods that replace the L₂-norm with the L₂₁-norm [44, 45]; and (10)(11) unsupervised and supervised robust semi-nonnegative graph embedding (usRNGE) and (ssRNGE), which are robust semi-nonnegative graph embedding methods that impose no constraint on the base matrix [44].

In the implementation of the complex valued conjugate gradient decent method with the Amijo rule (CGD), we set the decreasing rate of the step size μ to satisfy (19) with the sufficient decreasing condition at 0.01 and the stopping criteria of 10,000 times of iterations or that the error ε is smaller than 10⁻⁴.

Visualization of learned basis components and reconstructed images

In order to visualize the basis and reconstructed images, we computed the complex matrix z back to the real domain by applying the angle operator ∠, which returns the angle of a complex number. The inverse mapping f ⁻¹of (8) gives the pre-image x in the following form:

Some basis images learned by NMF, proNMF, and proCMF from the JAFFE dataset are shown as in Fig. 3. One can see that the NMF bases are generally nearly holistic and less sparse than others bases, while proNMF extracted more localized and non-overlapped bases which correspond to several facial parts. Unlike small single regions as proNMF bases, the basic components of the proposed proCMF are combined from several localized regions to highlight specific local facial features which are salient areas on the face, such as the mouth, the nose, the eyes, and the eyebrows. This can be attributed to the fact that the proposed proCMF is more robust to extract discriminant facial features and has potential superior for FER. Figure 4 gives the reconstructed images of the proposed proCMF, NMF, and proNMF. The faces reconstructed by proCMF are much clearer than those reconstructed by NMF and proNMF. Due to outliers and noise, NMF and proNMF are hard to identify well test samples.

Learned basis images by a NMF, b proNMF, and c the proposed proCMF from the training data of the JAFFE dataset

Full size image

Reconstructed images by a NMF, b proNMF, and c the proposed proCMF from the JAFFE dataset

Full size image

Facial expression recognition on CK+ dataset

Generally, the training set and the testing set were created by dividing the dataset into two subsets. On the CK+ dataset, we designed two experiments using different numbers of selected images for training and testing. In case 1, one image among five frames of each expression per subject is used for training and the remaining four are used for testing. In case 2, for each expression per subject, two collected images formed the training set and the remaining three formed the testing set. The average recognition rates of two cases versus different subspace dimensionalities are described in Tables 1 and 2, respectively. The best results are indicated in italics.

Most of the algorithms tend to get higher recognition rates when the subspace dimension increases. NMF performs better than proNMF and conNMF but worse than weiNMF and NeNMF. As mentioned in Section 4.3, the facial bases extracted by proNMF are spatially localized and lack expression-related information. These small basis regions of proNMF are clearly not appropriate for facial expression discrimination and result in unsatisfactory performance. The eigenface approach PCA, a well-known framework for dimensionality reduction, just rated at 86.80% in average achievement. The supervised scheme of the NGE model herein is better than unsupervised ones in all cases, and no nonnegative constraint on the base matrix of semi-NMF has significant effect on FER. GSNMF and the proposed proCMF are superior to baseline methods. Their average performances are over 97% with the highest rate of 97.51% is achieved by the proposed proCMF algorithm.

Facial expression recognition on JAFFE dataset

Regarding to the JAFFE dataset, one image for each expression per person is taken randomly to construct the training data and the rest of the images are used to the test phase (case 1). Similarly, the case 2 was conducted by collecting two images for the training set. However, the JAFFE dataset is more challenging and the recognition rate is much lower than that achieved for the CK+ dataset. The recognition results are detailed in Tables 3 and 4. The proposed proCMF attained the best performance and reaches the average recognition rates of 82.10 and 70.42%, respectively, which are 1.15 ~ 36.13% higher than those of other methods. In case 1 of experiments, when the number of learned basis K exceeds the size of the training data, the overfitting problem occurs on PCA and seminonnegative models.

Facial expression recognition on occluded CK+ images

As mentioned in previous sections, the proposed proCMF exhibits the best performance on facial expression recognition in both CK+ and JAFFE datasets. In this section, we aim to firm the robustness of the proposed method when the dataset is corrupted by partial occlusion. The occlusion experiments were conducted on the facial images in the CK+ dataset. We constructed three experimental assessments through varying the occluded position on the tested face images. The mouth and eyes occlusions are simulated by placing a mask at mouth or eyes position, respectively. The sheltering patches of size 70 × 70 were put randomly on the original image of size 640 × 490 to create the random occlusion case. Some samples from occluded CK+ images are shown in Fig. 5. In a similar way to the unocclusion scene, the five last frames of each video sequence were treated as static images. Two of them were processed to become occlusion images and collected to form the testing set, the remaining three images were used for training set. The detailed results of random, mouth, and eyes occlusions are reported in Tables 5, 6, and 7, respectively. It can be seen that the highest recognition rate is achieved repeatedly by our proposed method, proCMF. Moreover, proCMF performs much more robust and stable than other methods. In this occlusion situation, the proposed proCMF maintains its discriminating ability, while the performances of other methods decrease significantly. In order to witness this statement, we calculated the gap between the non-occlusion and the occlusion cases on the CK+ dataset. The smallest decline on the average recognition rate is achieved by proCMF and uRNGE (15.98%), followed by ssRNGE with 17.86%. Inversely, proNMF, NeNMF, and conNMF dropped by the biggest proportions of 34.43, 34.29, and 40.40%, respectively. The recognition rates of the rest of the methods range from 20.84 to 27.54%. In overall, it can be stated that the proposed method works well not only in recognizing unoccluded faces but also in the occlusion case.

Cropped face images of six facial expressions with occlusions from the CK+ dataset

Full size image

The confusion matrix is figured out in terms of unocclusion, eyes and mouth occlusion for the proposed proCMF. The merged results are given in Table 8. It is observed that, in case of no occlusion, happiness and neutral can be classified well with highest accuracy (100%), while the other five expressions are recognized with lower accuracy (96–99%). For occluded mouth images, sadness is recognized with the lowest accuracy (71.86%) since sadness is highly confused to disgust and surprise. In contrast, it seems that eyes occlusion most affect to recognizing disgust expression. There is a significant decreasing to 63.16% on discriminating disgust by the confusing with sadness and anger.

Regarding the evaluation of the influence of different parts on facial image to the expression recognition rates, we considered on both occlusion and unocclusion local regions. From the aligned images, we cropped fixed regions on faces to create the testing set. Figure 6 shows the 12 local facial regions with the corresponding facial expression recognition accuracies. It is shown that the eye regions have more important information than the nose and mouth regions. In fact, in case of local region experiments, the eyes-nose regions obtain 60.77% in accuracy rate; meanwhile, the mouth-nose regions just reach 33.35%. The eye regions perform 30.71% accuracy which is better than the mouth regions 9.10%. In case of occlusion experiment settings, 62.82% is the average accuracy of all algorithms on recognizing the occluded mouth images and this rate is degraded to 59.9% for the occluded eyes images.

Face regions with the recognition accuracies (%), respectively

Full size image

We explored a new dimensionality reduction model which employs an implicitly cosine dissimilarity metric by transforming the real data to the complex domain and sets up a complex projection matrix in the setting of an unbounded optimization problem. Under the simple framework without any added label information, the proposed method, proCMF, successfully extracted local facial features which are salient areas in facial expression formation. The experiments validate convincingly that the proposed proCMF algorithm can much outperform the popular NMF algorithms for facial expression recognition. The proposed model shows its potential on dealing with diverse scenarios that involve a whole face and an occluded face. In the future, we will develop the current model by incorporating more constraints and improving optimization methods to enhance the FER performance. In order to firm the effectiveness of the proposed method, we will extend evaluation on spontaneous facial expressions and apply directly complex spectrogram features on sound processing, especially the problems about sound source separation.

This research was financially supported in part by the National Central University, Taiwan, through the NCU International Student Scholarship. The authors would like to thank the anonymous reviewers for their valuable comments that significantly improved the quality of this paper.

Affiliations

1. Department of Computer Science and Information Engineering, National Central University, Taoyuan, Taiwan

   • Viet-Hang Duong
   • , Yuan-Shan Lee
   • , Bach-Tung Pham
   • , Manh-Quan Bui
   •  & Jia-Ching Wang

2. Graduate Institute of Communication Engineering, National Taiwan University, Taipei, Taiwan

   • Jian-Jiun Ding
   •  & Pham The Bao

3. Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh, Vietnam

   • Jia-Ching Wang

Contributions

VHD made the main contributions to the conception and tracking algorithms’ design, as well as drafting the article. BTP contributed to the design of experiments. YSL, JJD, and PTB provided the technical advices. MQB checked the manuscript and contributed to the rearrangement of the materials. JJW offered critical suggestions on the algorithms’ design, provided significant revising for important intellectual content, and gave final approval of the current version to be submitted. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jia-Ching Wang.

Consent for publication

Informed consent was obtained from all authors included in the study.

Competing interests

The authors declare that they have no competing interests.

Ethical approval

All data and procedures performed in paper were in accordance with the ethical standards of research community. This paper does not contain any studies with human participants or animals performed by any of the authors.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Reprints and Permissions