Audio visual speech source separation via improved context dependent association model

2.1 Blind source separation problem

where $\mathbf{s}\left(t\right)={\left[s_1\right(t),\dots ,s_N(t\left)\right]}^T\in {ℝ}^N$ is vector of source samples, $\mathbf{x}\left(t\right)={\left[x_1\right(t),\dots ,x_M(t\left)\right]}^T\in {ℝ}^M$ is vector of mixed signals and $\mathbf{A}\left(t\right)\in {ℝ}^{M\times N}$ is mixing matrix at the time instance t. It should be noticed that both s(t) and A are unknown. Hence, the problem is designated to BSS that is estimation of unmixed signals y(t)=B(t)x(t) from mixed signals x using an unknown de-mixing matrix B such that they are as similar as possible to unknown sources s.

where τ is the batch index iterating over all batches of the signals. In this case, the mixing and de-mixing models (A _τ , B _τ ) are time invariant during each batch. Another solution is to consider adaptive source separation techniques which is beyond the scope of this paper.

2.2 Audio-visual speech processing

Before explaining AV source separation methods, it is necessary to review some issues in AV speech processing which also inherently arises in AV source separation:

To cope with first two issues, in most speech processing problems, speech is processed frame-wise with frames of 20−30 ms length where speech signal can be considered stationary. In AV speech processing, it is convenient to choose speech frame length such that audio and video frame rates are equal.

For handling the third issue, the routine solution is to extract compact and informative acoustic and visual features from speech and video frames such that each frame is represented with a few number of parameters. Frame-wise processing of speech is practical in most speech processing tasks, but the amount of speech signal in a single frame may be insufficient for source separation algorithms to perform accurately. Hence, a couple of consecutive frames must be used in each batch τ.

2.3 Audio-visual source separation

that is signal’s amplitude gain and their order cannot be determined using these algorithms. The permutation of estimated sources may also change within consecutive frames, because sources are non-stationary in time and space. Having true or at least stable ordering of sources is crucial in most automatic speech processing applications. Furthermore, ICA-based methods do not consider or perform weakly in case of noisy and degenerate mixtures (i.e., mixtures with M<N).

Incorporation of visual modality of speech as a source of extra information, can help to solve these problems. The permutation problem can be simply resolved and enhancement in the separation performance is gained in regular and degenerate mixtures.

Most AVSS algorithms work based on maximization of AV coherency between unmixed signals y and their corresponding video streams. It is shown in [12] that given coarse spectral envelope of sources, one can solve a system of equations for calculation of de-mixing matrix in regular mixtures. Moreover there exists a stochastic coherent relation between the speech spectral envelope and the lip visual features [9, 12]. These two facts have guided researchers toward capturing AV relation using different models and adopt it for AVSS tasks.

In [12] and [14], authors have proposed a joint statistical distribution $p_{\text{av}}(\mathcal{S},\mathcal{V})$ as an AV model which measures the coherency between the acoustic spectral ( ) and lip visual ( ) features of the speech in each frame. The distribution $p_{\text{av}}(\mathcal{S},\mathcal{V})$ is modeled by the GMM and is trained using a corpus of corresponding AV streams via the Expectation Maximization (EM) algorithm.

The summation in (6) is based on the assumption that AV frames in consecutive frames are independent from each other.

In the rest of this text unless mentioned otherwise, we always consider a single row de-mixing vector denoted by B corresponding to a single visual stream. For the sake of brevity we omit the superscript (.)¹ of variables. It is clear that in case of existence of multiple video streams corresponding to more than one speech sources, all the described methods can be repeated for each video stream.

3.1 MLP audio visual model

Having acoustic and visual streams of feature frames and extracted from pairs of corresponding AV signals s and V (see Section 4.1), a context-dependent AV associator can be trained using a suitable non-linear function approximator: $\widehat{\mathcal{V}}\left(k\right)=h\left({\mathcal{S}}_e\right(k\left)\right)$, where $\mathcal{Se}\left(k\right)=E\left(\mathcal{S}\right(k-K/2-1:k+K/2\left)\right)$ is an embedded vector obtained from a context of K audio frames around frame k. An option for embedding E, is to stacks the center frame of the context and the first-order temporal difference of other frames. In this paper, we adopt an MLP with K input audio frames, one hidden layer of N _H neurons and a single visual frame as output, to approximate the AV mapping h(.). The MLP-based AV incoherency model ${\mathcal{ℳ}}_{MLP}$ is then defined as

${\mathcal{ℳ}}_{MLP}\left(\mathcal{Ye}\right(k),\mathcal{V}(k\left)\right)={\parallel \mathcal{V}\left(k\right)-h\left(\mathcal{Ye}\right(k\left)\right)\parallel }^2$

(7)

3.2 Audio visual source separation algorithm

Beside the difference in negative log probability and mean square error, another difference between AV objective functions (6) and (8) is the form of independence assumption in measuring the incoherency. The former considers absolute independence (i.e., i.i.d.) between the frames while the later assumes conditional independence.

In the last summation, the first term is gradient of MLP AV model with respect to its input acoustic context ${\mathcal{Y}}_e\left(k\right)$ and the second term is gradient of acoustic features with respect to the de-mixing model B. Gradient-based algorithm iteratively minimizes the problem (9). Starting from an initial point B _τ (0), at each iteration i, the gradient (11) is calculated, and using (10) or a quasi-Newton method, the improved de-mixing vector B _τ (i+1) is estimated. This continues until the change in the norm of B _τ or J_{av MLP}(B _τ ) becomes smaller than a pre-defined threshold.

Since the AV contrast function is not convex, the optimization algorithm is prune to local minima. Thus, selection of a good initialization point B _τ (0) is important. A simple option may be to start from random initial points multiple times. Most ICA algorithms (including FastICA [26] and JADE [27]) start from uncorrelated or white signals. Thus, another suggestion for initial point B _τ (0) is to apply PCA on mixed signals x _τ of the current batch τ and, among eigenvectors, select a vector W that produces a signal y=W x which is most coherent with the visual stream ${\mathcal{V}}_{\tau }$ and use it as the initial point B _τ (0).

3.3 AVSS using AV coherency and independence criterion

Although the existing and the proposed AV coherency-based methods provide improvements in speech source separation, but these methods totally neglect the useful constraint of independence of the sources. The statistical independence criteria used by ICA methods has been successful in many BSS methods. In this section, we consider the benefit of using AV coherency and statistical independence together to gain more enhancement in speech source separation.

3.4 Toward a time domain AVSS for convolutive mixtures

It contains M(2L+1) observation samples and using it the convolutive de-mixing process for separation of signal s¹ can be expressed as y(n)=B x^′(n) where B is a row vector containing coefficients of M de-mixing FIR filters each one having 2L+1 taps. This is just an instantaneous mixture with M(2L+1) virtual (embedded) observations and can be solved using the kurtosis-based method of [31] or using our proposed criteria (14).

As a final note, it should be mentioned that the reference method of [31], can estimate de-mixing filters up to a scale and time delay. Thus, a cross-correlation step is necessary to fix the possible delay of filters. For further details please refer to [31]. When dealing with convolutive mixtures, it is necessary to calculate the objective scores on longer segments of signals since there are larger number of parameters to estimate.

4.1 Audio-visual data

To evaluate the proposed algorithm, we have recorded a proper AV corpora which is comparable in (size and complexity) to the corpora used in former research. Unlike [11, 12], we have not used lip blue make-ups in data recordings since we do not need lip segmentation for extraction of geometric features such as width and height. Instead, the pixel gray values of speaker’s mouth region are used to extract the visual parameters. We have recorded two different types of corpora. The first corpus consists of discrete Persian and English alphabet and digits with a vocabulary size of 78 words (32 + 10 Persian and 26 + 10 English alpha-digits). The second corpus is continuous and consists of 140 verses of Persian poets. Both corpora are uttered by a male speaker. Each corpus is recorded two times. The first recording is used for training AV models and the second recording is used in evaluation phase.

In each recording, camera is focused on the speaker’s mouth and a video stream together with a mono audio stream is recorded. The raw video is captured in VGA size, true-color format (RGB 24 bits/pixel 8 bits/color) and at the frame rate of F _r ≃30 f p s and audio is recorded using 16 bits/sample and at sampling frequency of F _s =16,000 Hz. The final mouth region video used in the experiments of this paper, is stored in true-color 160 × 120 resolution frames. Sample lip region images from audio-visual corpus are shown in Figure 1a.

4.2 Audio and video parameter extraction

As discussed before, speech signal and lip image frames are high-dimensional data with sparse information related to our task. Thus, parametrizing audio and visual frames to compact vectors is necessary. Here, we clarify the methods for audio and visual feature extraction.

4.2.1 Audio parametrization

In most speech processing tasks, log spectral envelope (cepstral) features are utilized as effective features. We use PCA projected (whitened) log power spectral density for the speech frames parametrization. Let $\mathcal{Y}\left(k\right)=f\left(y\right(k\left)\right)=f\left(B\mathbf{x}\right(k\left)\right)$ be acoustic feature mapping function which extracts k _a spectral envelope features from every frame k of the estimated signal y. In practice, audio features are extracted from the spectrum Y of the de-mixed signal y. Since for the separation algorithm we need to efficiently calculate and its derivative with respect to the de-mixing vector B, we define an alternate audio feature extractor function F(.) which efficiently extracts features from the frequency domain representation of y:

$\mathcal{Y}\left(k\right)=F\left(Y\right(k\left)\right)=F\left(B\mathbf{X}\right(k\left)\right)$

(17)

where $Y=F\left\{y\right\}$ and $\mathbf{X}=F\left\{\mathbf{x}\right\}$ are the fast Fourier transform (FFT) of y and x, respectively. In the right-hand side (RHS) of (17), we have used the linear property of FFT that is, for every matrix B, $F\left\{\mathbf{B}\mathbf{x}\right\}=\mathbf{B}F\left\{\mathbf{x}\right\}$. Thus, we can pre-calculate X using FFT and then for any value of the de-mixing vector B the frequency domain de-mixed signal Y (and its derivative) can be efficiently obtained without FFT recalculation. As in [14], we have considered n=32 spectral coefficients in the range [0,5,000]H z as Y(k) for each frame. Power spectrum vector of each frame Y(k) is then defined as P S _Y (k)=[Y(k)⊗Y^∗(k)] ^T , where ⊗, (.)^∗ and (.) ^T are element-wise product, complex conjugation and transpose operators. Although the lip and the speech spectral envelope shapes are correlated, but there is not any meaningful relation between the lip shape and speech loudness (energy). Thus, it is important to normalize the energy of power spectrum in each frame resulting in power spectral density (PSD). The PSD coefficients are then converted to decibels (dB) using logarithm

${\log \mathit{\text{PSD}}}_Y\left(k\right)=\text{log}\left(\frac{{\mathit{\text{PS}}}_Y\left(k\right)}{\parallel Y\left(k\right){\parallel }^2}\right)$

(18)

Finally, whitening is applied to reduce the dimension of acoustic feature vectors to k _a elements. The acoustic whitening matrix ${\mathbf{W}}_a\in {ℝ}^{n\times k_a}$ is computed from the train data using eigen-value decomposition and is used to project train and test feature vectors to k _a -element compact spectral acoustic vectors. The overall acoustic feature extraction function F(Y) is defined as follows:

$\mathcal{Y}\left(k\right)=F(B;\mathbf{X}(k\left)\right)={\mathbf{W}}_a^T\text{log}\left(\frac{{\left[\left(B\mathbf{X}\right(k\left)\right)\otimes {\left(B\mathbf{X}\right(k\left)\right)}^{\ast }\right]}^T}{\parallel B\mathbf{X}\left(k\right){\parallel }^2}\right)$

(19)

The Jacobian of F with respect to B is derived in the Appendix in Equations 22, 23 and 24. The derived formulas are efficient and do not need FFT recalculation for different values of B. It is also worth to mention that the mapping F is invariant regarding a scalar multiplication (i.e. F(α B;X(k))=F(B;X(k)),∀α≠0). A property that entails gain invariance property (12) in AV contrast functions (6) and (8).

4.2.2 Video parametrization

In previous works, such as [11, 12, 14], authors have used geometric lip parameters that need lip contour detection to estimate the width and height of interior lip contour. We extract holistic visual features from all pixels of the mouth region. This requires less computation and does not require contour fitting. Let function g(.) be visual feature mapping function which extracts k _v visual features from any video frame. We assume that mouth region can be extracted from video using detection and tracking algorithms. There exists efficient parametric head tracking algorithms such as [36] which can be adopted for this task. The corpus used in this paper simply provides lip region in each frame. To extract k _v visual features, the mouth region of each frame is shrunk to 32 × 24 pixels and then reshaped to 768 × 1 image vectors. Finally, a PCA transform is applied to extract visual features. The PCA matrix ${\mathbf{W}}_v\in {ℝ}^{768\times k_v}$ is computed from the train data and is used to project train and test mouth region images to k _v -element visual parameter vectors. Figure 1b represents top major eigen-vectors (eigen-lips) in order.

The overall visual feature extraction function is defined as normalized projected gray values of mouth region: $\mathcal{V}\left(k\right)=g\left(\mathbf{V}\right(k\left)\right)={\mathbf{\text{Q}}}_v^T{\mathbf{W}}_v^T\mathbf{V}\left(k\right)$, where Q _v is the diagonal scaling matrix calculated from square root of corresponding eigen-values.

To assess and understand the virtue of visual features, a simple yet insightful simulation is illustrated in Figure 1c. In PCA, the eigenvector with largest eigenvalue captures most of the variance of dataset. Most variance of lip images during speaking is along opening and closing of lips. Thus, it is expected that the principal eigenvector ${\mathbf{W}}_v^1$ will model this direction of variation. To check this, we calculated mean vector µ _v of all video frames in poet-verses train corpus and illustrated its variations along the principal eigenvector ${\mathbf{W}}_v^1$ with negative and positive integer multiplies of square root of corresponding eigenvalue σ₁. Results presented in Figure 1c show that this has resulted in synthesized opening and closing of lip images.

Figure 2a,c demonstrates two segments of discrete and continuous speech from alpha-digits and poet-verses corpora and Figure 2b,d shows the corresponding first visual feature (before normalization). In discrete or slow speech, first visual feature shows a quasi-periodic shape corresponding to the periodic lip opening and closing. In continuous or fast speech, lip opening and closing is partial and this makes it more complex.

4.3 Building audio-visual models

In addition to estimation of transforms W _a and W _v , that are part of the AV feature mapping functions f(.) and g(.), the train set of each corpus is used to learn the AV models. The training set consists of synchronous sequences of AV pairs $\left(\mathcal{S}\right(k),\mathcal{V}(k\left)\right)$ extracted from the raw AV data. For training models with K>1, first, the embedded acoustic stream ${\mathcal{S}}_e$ is formed by K-fold embedding of frames of acoustic stream . Instead of stacking the K frames of context, it is better to stack the center frame together with temporal difference of other frames. This reduces the redundancy in the embedded vector. Then, embedded pairs $\left({\mathcal{S}}_e\right(k),\mathcal{V}(k\left)\right)$ are used to train models. Both GMM and MLP models are trained with different context sizes for fair comparison. But as experimental results of Section 5.1 shows, GMM degrades with K>1.

For training GMM models, AV components of each pair are concatenated and considered as samples of joint PDF p_av(.,.). These samples are used for estimation of GMM parameters using maximum likelihood via expectation maximization (EM) algorithm [37]. GMM distributions with various configuration of parameters (k _a , k _v , N _M , K) are trained. To assure good training, for each setting, GMM distribution is trained 20 times using EM with random initialization and the best model is selected based on a validation subset of training data. Regularization by adding a small positive number in range [10⁻¹⁰,10⁻²] to diagonal elements of covariance matrices was adopted to hold positive definiteness where necessary (specially for models with larger random vector dimensions).

MLP AV models are also trained on AV pairs $\left({\mathcal{S}}_e\right(k),\mathcal{V}(k\left)\right)$ with ${\mathcal{S}}_e\left(k\right)$ as input and $\mathcal{V}\left(k\right)$ as output. MSE criterion between true and estimated outputs $\mathcal{V}\left(k\right)$ and $\widehat{\mathcal{V}}\left(k\right)$ is used as the performance measure in training. This is the same criterion as what is used in contrast function (8). Networks were trained using the Levenberg-Marquardt algorithm [38] and via early stopping based on validation subset to avoid over-fit. As for GMM, MLP models with various configuration of parameters (k _a , k _v , N _H , K) are trained. To avoid local minima in training, each model is trained 20 times with random initialization and the best model is selected based on the validation subset.

5.1 Audio-visual models assessment and selection

In this experiment, we pre-evaluate fitness of AV models and explore the effect of different parameters on their performance. Both MLP- and GMM-based AV models need training and have hyper-parameters to be selected. We should choose proper dimensions k _a and k _v of acoustic and visual parameters, the embedding context size K, the number of hidden neurons N _H of MLP and the number of Gaussian components N _M of GMM models. Although validation scores of trained models can be used to select best GMM and MLP models, but selection of models based on their capability of discrimination between coherent and incoherent speech is more reasonable since models are aimed to be used for source separation. Furthermore, such an experiment provides insights in virtual potentials of coherency-based AVSS methods.

5.1.1 Audio-visual pure relevant source detection

In this experiment, we compare incoherency scores between a visual stream V¹ and two pure audio signals: a coherent signal s¹ and an irrelevant signal s². For each frame in the test set, the signal which produces minimum incoherency score is recognized to be coherent with V¹. Experiments are performed for both AV models ${\mathcal{ℳ}}_{GMM}(\mathcal{S},\mathcal{V})$ (5) and ${\mathcal{ℳ}}_{MLP}(\mathcal{S},\mathcal{V})$ (7). For each model, different values of hyper-parameters k _a ∈ {2,4,6,8,10,12}, k _v ∈{2,4,6,8}, N _M ,N _H ∈{4,8,12,16,20,24,28}, K∈{1,2,3,4,5,6} are examined. Finally, the percent of all frames which signal s¹ is truly selected is reported as classification accuracy for different values of batch size T.

In [14], authors have evaluated the classification rate just against a single irrelevant signal which is uttered by a different male speaker. Our initial experiments revealed that classification accuracy for different irrelevant signals is variable depending on the speaker, the speech content of signal and alignment of silent parts of coherent and incoherent signals. Thus, to provide classification rates with high confidence, we conducted multiple simulations by performing coherency classification on the relevant signal s¹y against six distinct speech signals for s² and reported the average recognition rate as the performance of models.

Furthermore, it is possible that AV models, in addition to AV coherency of speech, capture some parts of AV identity of speaker. To check for this, we chose coherent and irrelevant speech signals both from the same speaker. As much as AV models have captured speaker identity, this provides a classification problem which is more confusing and complex for them relative to choosing irrelevant signals from different speaker(s).

Comprehensive classification rates are presented in Figure 3 for both GMM and MLP models and for different values of k _a , k _v and T as free parameters. In this figure, other parameters (N _H of MLP, N _M of GMM and K of both models) are marginalized by selecting the maximum accuracy among them.

Table 1 presents optimal values for hidden (marginalized) parameters (N _H , N _M and K) of Figure 3 for different k _a and k _v configurations in both GMM and MLP models.

	GMM(K/N M )					MLP(K/N H )
k a	k v :2	4	6	8		2	4	6	8
2	1/4	1/16	1/28	1/8		4/20	4/16	4/20	6/28
4	1/24	1/24	1/24	1/20		4/20	3/16	2/24	3/16
6	1/28	1/20	1/24	1/24		4/8	2/24	3/16	2/8
8	1/8	1/8	1/28	1/28		2/20	2/16	2/28	2/16
10	1/24	1/28	1/28	1/24		2/16	2/28	2/16	2/16
12	1/8	1/16	1/20	1/20		4/4	2/16	2/20	4/8

Optimal values are selected using cross validation for different values of k _a and k _v .

The common trends in classification rates of Figure 3 reveals following points:

1. 1.

   Accuracy of both MLP and GMM models is enhanced by increasing the number of batch frames T, acoustic features k _a and visual features k _v .

    
2. 2.

   Among these factors, batch size T has the highest impact and this is followed by k _a ; finally, k _v has the lowest impact.

    

Comparing the trends in Figure 3 for MLP and GMM models, also reveals interesting points:

1. 1.

   MLP model performs significantly better relative to GMM in various AV dimensions.

    
2. 2.

   In lower visual dimensions (k _v =2), MLP outperforms with a 10−15% gap relative to GMM model. In this case, even performance of MLP with worst condition (batch size T=4) is 5−6% higher than GMM with best condition (batch size T=16).

    
3. 3.

   In higher visual dimensions (k _v =6,8), the difference between GMM and MLP is somewhat reduced.

    
4. 4.

   The improvements by increasing number of features k _a and k _v is bounded. For k _v >8, in MLP, k _v >6 in GMM and k _a >8 in both models, no more significant enhancement is achieved. The model complexity increases in O(K.k _a +k _v ) for MLP and O((K.k _a +k _v )²) for GMM and in some point, this results in over-complex models for the problem (considering the amount of available training data).

    
5. 5.

   Contrarily, improvements by increasing batch size (T) continues upward and may reach perfect accuracy for enough large T values. This is because the value of T does not change the model size while increasing it introduces more information for decision making. However, it is important to mention that for real AVSS tasks, we cannot increase T arbitrarily. This makes the stationary assumption considered in the mixture model (2) invalid. Hence, there is a trade-off on the value of T between the AV model accuracy and the mixing model fitness.

    

Finally, results of optimal K, N _H and N _M values presented in Table 1 reveals that

1. 1.

   In various k _a and k _v s, GMM always has performed better with K=1 frames in embedded context which means GMM can not capture temporal dynamics by frame embedding due to quadratic order of parameters.

    
2. 2.

   MLP always has performed better with K=2 frames (for greater k _a ) or K=4,6 frames (for smaller k _a ) in embedded context showing that it can capture some temporal dynamics.

    
3. 3.

   Both GMM and MLP models exploit maximum average number of latent units in k _v =6 which seems to be efficient optimal visual dimension size according to results of Figure 3.

    

5.1.2 Audio-visual mixed relevant source detection

Recall that classification results in Section 5.1.1 are based on comparing the incoherency scores between pure relevant and irrelevant speech signals. This entails that AV models are well suited for selection of a clean relevant source among multiple available irrelevant signals. For example, it will perform well for relevant source selection in AV-assisted ICA-based source separation method (i.e. JADE-AV) discussed in Section 3.3.1.

In AV separation algorithms, models must provide scores for signal of relevant source which is more or less contaminated by other sources specially during first iterations of the optimization algorithm. Hence, a good AV model must be such that it provides decreasing incoherency scores for increasing amounts of SIR. Therefore, we conducted another experiment to assess how well AV models comply with this property. Let ξ _i =s¹+α _i s² be a mixed signal composed of source s¹ coherent with visual stream V¹ and an irrelevant speech or acoustic signal s². Mixed signals at different SIRs can be generated using different values for mixing coefficient α _i . We generated a set of mixed signals ξ _i with SIRs in range [ −5,30] dB and performed classification using incoherency score comparisons between signal pairs (ξ _i ,ξ_i+1) at different SIR levels. Here, the classification accuracy is defined as percent of all frames which the signal with higher SIR is selected. Figure 4 shows average classification accuracy measured at different SIR levels for all GMM and MLP models tested in previous experiment.

Generally, trends of Figure 4 shows similar properties as was discussed for Figure 3. The major point is that classification accuracies of best models on mixed signals ξ _i , ξ_i+1 is something about 10% less relative to classification of pure relevant and irrelevant signals. Such a degradation is predictable since signals ξ _i and ξ_i+1 are very similar. But the interesting note is that superior models in the pure classification have approximately kept their superiority in the mixed case. This means that optimal model configurations which are better for classification task, may keep their position in separation task. As before MLP models are superior to GMM models but the large gap between them is somewhat reduced.

5.2 Source separation experiments