Dereverberation and denoising based on generalized spectral subtraction by multi-channel LMS algorithm using a small-scale microphone array

A blind dereverberation method based on power spectral subtraction (SS) using a multi-channel least mean squares algorithm was previously proposed to suppress the reverberant speech without additive noise. The results of isolated word speech recognition experiments showed that this method achieved significant improvements over conventional cepstral mean normalization (CMN) in a reverberant environment. In this paper, we propose a blind dereverberation method based on generalized spectral subtraction (GSS), which has been shown to be effective for noise reduction, instead of power SS. Furthermore, we extend the missing feature theory (MFT), which was initially proposed to enhance the robustness of additive noise, to dereverberation. A one-stage dereverberation and denoising method based on GSS is presented to simultaneously suppress both the additive noise and nonstationary multiplicative noise (reverberation). The proposed dereverberation method based on GSS with MFT is evaluated on a large vocabulary continuous speech recognition task. When the additive noise was absent, the dereverberation method based on GSS with MFT using only 2 microphones achieves a relative word error reduction rate of 11.4 and 32.6% compared to the dereverberation method based on power SS and the conventional CMN, respectively. For the reverberant and noisy speech, the dereverberation and denoising method based on GSS achieves a relative word error reduction rate of 12.8% compared to the conventional CMN with GSS-based additive noise reduction method. We also analyze the effective factors of the compensation parameter estimation for the dereverberation method based on SS, such as the number of channels (the number of microphones), the length of reverberation to be suppressed, and the length of the utterance used for parameter estimation. The experimental results showed that the SS-based method is robust in a variety of reverberant environments for both isolated and continuous speech recognition and under various parameter estimation conditions.


Introduction
In a distant-talking environment, channel distortion drastically degrades speech recognition performance because of a mismatch between the training and testing environments. The current approach focusing on automatic speech recognition (ASR) robustness to reverberation and noise can be classified as speech signal processing, robust feature extraction, and model adaptation [1][2][3].
In this paper, we focus on speech signal processing in the distant-talking environment. Because both the speech signal and the reverberation are nonstationary signals, dereverberation to obtain clean speech from the convolution of nonstationary speech signals and impulse responses is very hard work. Several studies have focused on mitigating the above problem. A blind deconvolution-based approach for the restoration of speech degraded by the acoustic environment was proposed in [4]. The proposed scheme processed the outputs of two microphones using cepstra operations and the theory of signal reconstruction from the phase only. Avendano et al. [5,6] explored a speech dereverberation technique for which the principle was the recovery of the envelope modulations of the original (anechoic) speech. They applied a technique that they originally developed to treat background noise [7] to the dereverberation problem. A novel approach for multimicrophone speech dereverberation was proposed in [8]. The method was based on the construction of the null subspace of the data matrix in the presence of colored noise, employing generalized singular-value decomposition or generalized eigenvalue decomposition of the respective correlation matrices. A reverberation compensation method for speaker recognition using SS, in which late reverberation is treated as additive noise, was proposed in [9,10]. However, the drawback of this approach is that the optimum parameters for SS are empirically estimated from a development dataset and the late reverberation cannot be subtracted correctly as it is not modeled precisely.
In [1,[11][12][13], an adaptive multi-channel least mean squares (MCLMS) algorithm was proposed to blindly identify the channel impulse response in a time domain. However, the estimation error of the impulse response was very large. Therefore, the isolated word recognition rate of the compensated speech using the estimated impulse response was significantly worse than that of unprocessed received distorted speech [14]. The reason might be that the tap number of the impulse response was very large and the duration of the utterance (that is, a word with duration of about 0.6 s) was very short. Therefore, the variable step-size unconstrained MCLMS (VSS-UMCLMS) algorithm in the time domain might not be convergent. The other problem with the algorithm in the time domain is the estimation cost. Previously, Wang et al. [14] proposed a robust distant-talking speech recognition method based on power SS employing the MCLMS algorithm (see Figure 1a). They treated the late reverberation as additive noise, and a noise reduction technique based on power SS was proposed to estimate the power spectrum of the clean speech using an estimated power spectrum of the impulse response. To estimate the power spectra of the impulse responses, we extended the VSS-UMCLMS algorithm for identifying the impulse responses in a time domain [1] to a frequency domain. The early reverberation was normalized by CMN.
Power SS is the most commonly used SS method. A previous study has shown that GSS with a lower exponent parameter is more effective than power SS for noise reduction [15]. In this paper, instead of using power SS, GSS is employed to suppress late reverberation. We also investigate the use of missing feature theory (MFT) [16] to enhance the robustness to noise, in combination with GSS, since the reverberation cannot be suppressed completely owing to the estimation error of the impulse response. Soft-mask estimation-based MFT calculates the reliability of each spectral component from the signal-tonoise ratio (SNR). This idea is applied to reverberant speech. However, the reliability estimation is complicated in a distant-talking environment. In [17], reliability is estimated from the time lag between the power spectrum of the clean speech and that of the distorted speech. In this paper, reliability is estimated by the signal-to-reverberation ratio (SRR) since the power spectra of clean speech and the reverberation signal can be estimated by power SS or GSS using MCLMS. A diagram of the modified proposed method combining GSS with MFT is shown in Figure 1b.  The precision of impulse response estimation is drastically degraded when the additive noise is absent. The traditional method used two-stage processing progress, in which the reverberation suppression is performed after additive noise reduction. We present a one-stage dereverberation and denoising based on GSS. A diagram of the processing method is shown in Figure 2.

ĂƌůǇ ƌĞ|ĞƌďĞƌĂƚŝŽŶ
In this paper, we also investigate the robustness of the SS-based reverberation under various reverberant conditions for large vocabulary continuous speech recognition (LVCSR). We analyze the effect factors (numbers of reverberation windows and channels, length of utterance, and the distance between sound source and microphone) of compensation parameter estimation for dereverberation based on SS.
The remainder of this paper is organized as follows: Section 2 describes the outline of blind dereverberation based on SS. A MFT for dereverberation is described in Section 3. A one-stage dereverberation and denoising method is proposed in Section 4, while Section 5 describes the experimental results of distant speech recognition in a reverberant environment. Finally, Section 6 summarizes the paper.

If speech s[t] is corrupted by convolutional noise h[t]
and additive noise n[t], the observed speech x[t] becomes where * denotes the convolution operation. In this paper, additive noise is ignored for simplification, so If the length of the impulse response is much smaller than the size T of the analysis window used for short time Fourier transform (STFT), the STFT of the distorted speech equals that of the clean speech multiplied by the STFT of the impulse response h[t]. However, if the length of the impulse response is much greater than the analysis window size, the STFT of the distorted speech is usually approximated by where f is the frame index, H(ω) is the STFT of the impulse response, S(f, ω) is the STFT of clean speech s, D is number of reverberation windows, and H(d, ω) denotes the part of H(ω) corresponding to the frame delay d. That is, with a long impulse response, the channel distortion is no longer of a multiplicative nature in a linear spectral domain but is rather convolutional [3].
In [14], Wang et al. proposed a dereverberation method based on power SS to estimate the STFT of the clean speechŜ (f , ω) based on Equation (2). The spectrum of the impulse response for the SS is blindly estimated using the method described in Section 2.3. Assuming that phases of different frames is noncorrelated for simplification, the power spectrum of Equation (2) can be approximated as The power spectrum of clean speech |Ŝ(f , ω)| 2 can be estimated as Equation (4), where H(d, ω), d = 0,1, ...,D-1 is the STFT of impulse response, which can be calculated from the known impulse response or can be blindly estimated.
Furthermore, the early reverberation is compensated by subtracting the cepstral mean of the utterance. As is well known, cepstrum of the input speech x(t) is calculated as: where X(ω) is the spectrum of the input speech x(t). The early reverberation is normalized by the cepstral mean C in a cepstral domain (linear cepstrum is used) and then it is converted into a spectral domain as: whereX(f , ω) is the mean vector of X(f, ω). After this normalization processing, Equation (6) becomes as where The SS is used to prevent the estimated clean power spectrum being negative value; Equation (8) is adopted as:

Dereverberation based on GSS
Previous studies have shown that GSS with an arbitrary exponent parameter is more effective than power SS for noise reduction. In this paper, we extend GSS to suppress late reverberation. Instead of the power SS-based dereverberation given in Equation (9), GSS-based dereverberation is modified as where n is the exponent parameter. For power SS, the exponent parameter n is equal to 1. In this paper, the exponent parameter n is set to 0.1 as this value yielded the best results in [15].
The methods given in Eqs. (9) and (10) are referred to as SS-based (original) and GSS-based (proposed) dereverberation methods, respectively.

Compensation parameter estimation for SS by multichannel LMS algorithm
In [1], an adaptive multi-channel LMS algorithm for blind single-input multiple-output (SIMO) system identification was proposed.
In the absence of additive noise, we can take advantage of the fact that (11) and have the following relation at time t: where h i (t) is the i-th impulse response at time t and where x i (t) is the speech signal received from the i-th channel at time t and L is the number of taps of the impulse response. Multiplying Equation (12) by x i (t) and taking expectation yields, (13) where Over all channels, we then have a total of N equations. In matrix form, this set of equations is written as: where where h n (t, l) is the lth tap of the nth impulse response at time t. If the SIMO system is blindly identifiable, the matrix R x+ is rank deficient by 1 (in the absence of noise) and the channel impulse responses can be uniquely determined.
When the estimation of channel impulse responses is deviated from the true value, an error vector at time t + 1 is produced by: whereRx i x j (t + 1) = x i (t + 1)x T j (t + 1), i, j = 1, 2, ..., N andĥ (t) is the estimated model filter at time t. Here, we put a tilde inRx i x j to distinguish this instantaneous value from its mathematical expectation Rx i x j .
This error can be used to define a cost function at time t + 1 By minimizing the cost function J of Equation (21), the impulse response can be blindly derived. Wang et al. [14] extended this VSS-UMCLMS algorithm [1], which identifies the multi-channel impulse responses, for processing in a frequency domain with SS applied in combination.

Missing feature theory for dereverberation
MFT [16] enhances the robustness of speech recognition to noise by rejecting unreliable acoustic features using a missing feature mask (MFM). The MFM is the reliability corresponding to each spectral component, with 0 and 1 being unreliable and reliable, respectively. The MFM is typically a hard and a soft mask. The hard mask applies binary reliability values of 0 or 1 to each spectral component and is generated using the signal-to-noise ratio (SNR). The reliability is 0 when the SNR is greater than a manually-defined threshold, otherwise it is 1. The soft mask is considered a better approach than the hard mask and applies a continuous value between 0 and 1 using a sigmoid function.
In a distant-talking environment, it is difficult to estimate the reliability of each spectral component since it is difficult to estimate the spectral components of clean speech and reverberant speech. Therefore, in [17], the reliability was estimated from a priori information by measuring the difference between the spectral components of clean speech and reverberant speech at given times. In this paper, a soft mask is calculated using the signal-to-reverberation ratio (SRR). From Equation (10), the SRR is calculated as The reliability r(f, ω) for the soft mask is generated as where a and b are the gradient and center of the sigmoid function, respectively, and are empirically determined. Finally, the estimated spectrum of clean speech from Equation (10) is multiplied by the reliability r(f, ω), and the inverse DFT of |Ŝ(f , ω)| 2n r(f , ω) forms the dereverberant speech.

One-stage dereverberation and denoising based on GSS
The precision of impulse response estimation is drastically degraded when the additive noise is present. The traditional method used two-stage processing progress, in which the reverberation suppression is performed after additive noise reduction. We present a one-stage dereverberation and denoising based on GSS. A diagram of the processing method is shown in Figure 2. At first, the spectra of additive noise and impulse responses are estimated, and then the reverberation and additive noise are suppressed simultaneously. When additive noise is present, the power spectrum of Equation (2) becomes whereN(ω) is the mean of noise spectrum N(ω). To suppress the noise and reverberation simultaneously, Equation (10) is modified as where X N (f, ω) is spectrum by subtracting the spectrum of observed speech with the spectrum of noisē N(ω) andX N (f , ω) is mean vector of X N (f, ω).

Experimental setup
Multi-channel distorted speech signals simulated by convolving multi-channel impulse responses with clean speech were used to evaluate our proposed algorithm. Fifteen kinds of multi-channel impulse responses measured in various acoustical reverberant environments were selected from the real world computing partnership (RWCP) sound scene database [18,19] and the CEN-SREC-4 database [20]. Table 1 lists the details of 15 recording conditions. The illustration of microphone array is shown in Figure 3. For RWCP database, a 2-8 channel circular or linear microphone array was taken from a circular + linear microphone array (30 channels). The circle type microphone array had a diameter of 30 cm. The microphones of the linear microphone array were located at 2.83 cm intervals. Impulse responses were measured at several positions 2 m from the microphone array. For the CENSREC-4 database, 2 or 4 channel microphones were taken from a linear microphone array (7 channels) with the two microphones located at 2.125 cm intervals. Impulse responses were measured at several positions 0.5 m from the microphone array. The Japanese Newspaper Article Sentences (JNAS) corpus [21] was used as clean speech. Hundred utterances from the JNAS database convolved with the multi-channel impulse responses shown in Table 1 were used as test data. The average time for all utterances was about 5.8 s. Table 2 gives the conditions for speech recognition. The acoustic models were trained with the ASJ speech databases of phonetically balanced sentences (ASJ-PB) and the JNAS. In total, around 20K sentences (clean speech) uttered by 132 speakers were used for each gender. Table 3 gives the conditions for SS-based dereverberation. The parameters shown in Table 3 were determined empirically. An illustration of the analysis window is shown in Figure 4. For the proposed dereverberation method based on SS, the previous clean power spectra estimated with a skip window were used to estimate the current clean power spectrum since the frame shift was half the frame length in this study a . The spectrum of the impulse response H(d, ω) was estimated for each utterance to be recognized. An opensource LVCSR decoder software "Julius" [22] that is based on word trigram and triphone context-dependent HMMs is used. The word accuracy for LVCSR with clean speech was 92.59% (Table 4).

Effect factor analysis of compensation parameter estimation
In this section, we describe the use of four microphones b to estimate the spectrum of the impulse responses without a particular explanation. Delay-and-sum beamforming (BF) was performed on the 4-channel  dereverberant speech signals. For the proposed method, each speech channel was compensated by the corresponding estimated impulse response. Preliminary experimental results for isolated word recognition showed that the SS-based dereverberation method significantly improved the speech recognition performance significantly compared with traditional CMN with beamforming [14].
In this paper, we also evaluated the SS-based dereverberation method on LVCSR with the experimental results shown in Figure 5. Naturally, the speech recognition rate deteriorated as the reverberation time increased. Using the SS-based dereverberation method, the reduction in the speech recognition rate was smaller than in conventional CMN, especially for impulse responses with a long reverberation time. For RWCP database, the SS-based dereverberation method achieved a relative word recognition error reduction rate of 19.2% relative to CMN with delay-and-sum beamforming. We also conducted an LVCSR experiment with SS-based dereverberation under different reverberant conditions (CENSREC-4), with the reverberation time between 0.25 and 0.75 s and the distance between microphone and sound source 0.5 m. A similar trend to the above results was observed. Therefore, the SS-based dereverberation method is robust to various reverberant conditions for both isolated word recognition and LVCSR. The reason is that the SS-based dereverberation method can compensate for late reverberation through SS using an estimated power spectrum of the impulse response.
In this section, we also analyzed the effect factor (number of reverberation windows D in Equation (9), channel number, and length of utterance) for compensation parameter estimation for the dereverberation method based on SS using RWCP database.
The effect of the number of reverberation windows on speech recognition is shown in Figure 6. The detail results based on different number of reverberation windows D and reverberant environments (that is, different reverberation times) were shown in Table 5. The results shown on Figure 6 and Table 5 were not performed delay-and-sum beamforming. The results show that the optimal number of reverberation windows D depends on the reverberation time. The best average result of all reverberant speech was obtained when D equals 6. The speech recognition performance with the number of reverberation windows between 4 and 10 did not vary greatly and was significantly better than the baseline.
We analyzed the influence of the number of channels on parameter estimation and delay-and-sum beamforming. Besides four channels, two and eight channels were also used to estimate the compensation parameter and perform beamforming. Channel numbers corresponding to Figure 3a shown in Table 4 were used. The results are shown in Figure 7. The speech recognition performance of the SS-based dereverberation method without beamforming was hardly affected by the number of channels. That is, the compensation parameter estimation is robust to the number of channels. Combined with beamforming, the more channels that are used and the better is the speech recognition performance.
Thus far, the whole utterance has been used to estimate the compensation parameter. The effect of the length of utterance used for parameter estimation was    Table 4 Channel number corresponding to Figure 3a using for dereverberation and denoising (RWCP database) investigated, with the results shown in Figure 8. The longer the length of utterance used, the better is the speech recognition performance. Deterioration in speech recognition was not experienced with the length of the utterance used for parameter estimation greater than 1 s. The speech recognition performance of the SS-based dereverberation method is better than the baseline even if only 0.1 s of utterance is used to estimate the compensation parameter.

Experimental results of dereverberation and denoising
In this section, reverberation and noise suppression using only 2 speech channels is described. c In both SS-based and GSS-based dereverberation methods, speech signals from two microphones were used to estimate blindly the compensation parameters for the power SS and GSS (that is, the spectra of the channel impulse responses), and then reverberation was suppressed by SS and the spectrum of dereverberant speech was inverted into a time domain. Finally, delay-and-sum beamforming was performed on the two-channel dereverberant speech. The schematic of dereverberation is shown in Figure 1. Table 6 shows the speech recognition results for the original and proposed methods. "Distorted speech #" in Table 6 corresponds to "array no" in Table 1. The word accuracy by CMN without beamforming was 40.46%. The speech recognition performance was drastically degraded under reverberant conditions because the conventional CMN did not suppress the late reverberation. Delay-and-sum beamforming with CMN (41.91%) could not markedly improve the speech recognition performance because of the small number of microphones and the small distance between the microphone pair. In contrast, the power SS-based dereverberation using Equation (9) markedly improved the speech recognition performance. The GSS-based dereverberation using Equation (10)    The results with bold font indicate the best result corresponding to each array all reverberant conditions. The GSS-based method without MFT achieved an average relative word error reduction rate of 31.4% compared to the conventional CMN and 9.8% compared to the power SS-based method without MFT. When MFT was combined with both our methods, a further improvement was achieved. Finally, the GSS-based method with MFT achieved an average relative word error reduction rate of 32.6% compared to conventional CMN and 11.4% compared to the original proposed method [14]. Table 7 gives a breakdown of the word error rates obtained by the power SS-and GSS-based methods. The power SS-based method improved the substitution and deletion error rates but degraded the insertion error rate compared with CMN. The GSS-based method improved all error rates compared with the power SSbased method and achieved almost the same word insertion error as CMN.
To evaluate the proposed one-stage dereverberation and denoising based on GSS, computer room noise  Table 6 based on one-stage dereverberation and denoising based on GSS were shown in Table 8. The one-stage dereverberation and denoising method improved the speech recognition performance under all reverberant and noisy speech at each SNR level and reverberation time. The one-stage dereverberation and denoising method based on GSS achieved a relative word error reduction rate of 12.8% compared to the conventional CMN with GSS-based additive noise reduction method. The improvement under the additive noise condition was smaller than that for the noise-free condition. The reason might be the difference between the estimated spectrum of impulse response H(d, ω) for each condition; we compared the estimated H(d, ω) for both by first denoting the estimated spectrum of the impulse response for each as H 1 (d,ω) and H 2 (d,ω) and defining their average values asH   The normalized average differenceH n between H 1 (d,ω) and H 2 (d,ω) is then defined as The average values of these estimated spectra of impulse responses and their difference are shown in Table 9. In Table 9, only the multi-channel speech of array 2 was used to calculate the average values. The result showed that H 1 (d,ω) and H 2 (d,ω) were quit different.

Conclusions
Previously, Wang et al. [14] proposed a blind dereverberation method based on power SS employing the multi-channel LMS algorithm for distant-talking speech recognition. Previous studies showed that GSS with an arbitrary exponent parameter is more effective than power SS for noise reduction. In this paper, GSS is applied instead of power SS to suppress late reverberation. However, reverberation cannot be completely suppressed owing to the estimation error of the impulse response. MFT is used to enhance the robustness of noise. Soft-mask estimation-based MFT calculates the reliability of each spectral component from SNR. In this paper, reliability was estimated through the signal-to-reverberation ratio. Furthermore, delayand-sum beamforming was also applied to the multichannel speech compensated by the reverberation compensation method. Our SS and GSS-based dereverberation methods were evaluated using distorted speech signals simulated by convolving multi-channel impulse responses with clean speech. When the additive noise was absent, the GSS-based method without MFT achieved an average relative word error reduction rate of 31.4% compared to conventional CMN and 9.8% compared to the power SS-based method without MFT. When MFT was combined with both our methods, further improvement was obtained. The GSSbased method with MFT achieved average relative word error reduction rates of 32.6 and 11.4% compared to conventional CMN and the original proposed method, respectively. The one-stage dereverberation and denoising method based on GSS achieved a relative word error reduction rate of 12.8% compared to the conventional CMN with GSS-based additive noise reduction method.
In this paper, we also investigated the effect factors (numbers of reverberation windows and channels, and length of utterance) for compensation parameter estimation. We reached the following conclusions: (1) the speech recognition performance with the number of reverberation windows between 4 and 10 did not vary greatly and was significantly better than the baseline, (2) the compensation parameter estimation was robust to the number of channels, and (3) degradation of speech recognition did not occur with the length of utterance used for parameter estimation longer than 1 s. Endnotes a For example, to estimate the clean power spectrum of the 2ith window W 2i , the estimated clean power spectra of the 2(i-1)th window W 2(i-1) , the 2(i-2)th window W 2(i-2) , ... were used. b For RWCP database, 4 speech channels shown in Table 4 were used. For CENSREC-4 database, speech channels 1, 3, 5, and 7 shown in Figure 3b were used. c For RWCP database, 2 speech channels shown in Table 4 were used. For CENSREC-4 database, speech channels 1 and 3 shown in Figure 3b were used.