Speech recognition in reverberant and noisy environments employing multiple feature extractors and i-vector speaker adaptation

2.1 Multi-taper filterbank features

where f ∈ {0, 1,⋯, K − 1} denotes the discrete frequency index, N is the frame length, s(j) is the time-domain speech signal, and w(j) denotes the time-domain window function, also known as the taper. The taper, such as the Hamming window, is usually symmetric and decreases towards the frame boundaries.

Windowing reduces the bias (the bias of a spectrum estimator \( \widehat{\theta} \) is defined as the expected difference between the estimated value and the true value of the spectrum θ being estimated and is defined as \( \mathrm{bias}\left(\widehat{\theta}\right)=E\left[\left(\widehat{\theta}-\theta \right)\right] \)), but it does not reduce the variance of the spectral estimate [14, 15]; therefore, the variance of the cepstral/filterbank features computed from this estimated spectrum remains large.

The multi-taper spectrum estimate is therefore obtained as the weighted average of M individual spectra. A multi-taper spectrum estimator is somewhat similar to averaging the spectra from a variety of conventional tapers such as Hamming and Hann tapers, but in this case, there will be strong redundancy as the different tapers are highly correlated (they have a common time-domain shape).

Figure 1 depicts the multi-taper spectrum estimation process from a frame of speech signal with M = 6 orthogonal tapers. Unlike conventional tapers, the M orthonormal tapers used in a multi-taper spectrum estimator provide M statistically independent (hence uncorrelated) estimates of the underlying spectrum. The weighted average of the M individual spectral estimates Ŝ _MT (f) then has smaller variance than that of the single-taper spectrum estimates Ŝ _d (f) by a factor that approaches 1/M, i.e., \( \operatorname{var}\left({\widehat{S}}_{MT}(f)\right)\approx \frac{1}{M}\operatorname{var}\left({\widehat{S}}_d(f)\right) \) [14]. According to [16], variance in the feature vectors has a direct bearing to the variance of the Gaussian modeling speech classes. In general, reduction in feature vector variance increases class separability and thereby increases recognition accuracy [16]. Multi-taper mel-filterbank (MMFB) features are then computed from a multiple windowed (e.g., Thomson) spectrum estimate instead of the Hamming windowed periodogram estimate as used in the conventional mel-filterbank (MFB)/cepstral features. The motivation behind using the multi-taper method in the REVERB challenge 2015 tasks is its improved speaker recognition performance on the microphone speech portions of the NIST-SRE corpora [14, 15]. In this work, two variants of MMFB are used:

MMFBl: MMFB features with logarithmic nonlinearity

MMFBp: MMFB features with power function nonlinearity

where β is the time half bandwidth product. The optimum number of tapers for a continuous speech recognition task was found to be M = 6 [4, 14, 15] and β = 3.0.

Both of the MMFB features were normalized using the short-time mean and scale normalization (STMSN) method [19] with a sliding window of 1.5 s duration. Our baseline system uses conventional MFB features extracted using the Kaldi toolkit [20]. Various steps for the extraction of MMFB features are shown in Fig. 2. MFB features can be obtained as a special case of MMFB features from Fig. 2 by selecting the number of taper M = 1, taper weights λ(p) = 1, and w _p (j) as the symmetric Hamming taper.

2.2 Robust compressive gammachirp filterbank and mel-filterbank features

Robust compressive gammachirp filterbank (RCGFB) and robust mel-filterbank (RMFB) features were computed following a similar framework to the robust compressive gammachirp filterbank cepstral coefficient (RCGCC) features proposed in [21]. The main motivation for using RCGFB and RMFB feature extractors in the REVERB challenge 2014 tasks is the better recognition accuracy obtained by the RCGCC features on the AURORA-5 and AURORA-4 corpora which represents reverberant acoustic conditions and additive noise as well as different microphone channel conditions, respectively [21, 22].

Figure 3 presents the block diagram for the RCGFB and RMFB feature extractors that incorporate a sigmoid shape suppression rule based on subband a posteriori signal-to-noise ratios (SNRs) in order to enhance the auditory spectrum.

and N _as(k, m) is the noise power spectrum mapped onto the auditory frequency axis. In order to remove the outliers from the weighting factor H(k, m), as given by Eq. (3), due to noise variability, we use a two-dimensional median filter. For smoothing the decision regions, a two-dimensional moving average filter is also applied [21, 22].

Noise power spectrum estimation from the noisy speech signal plays a very important rule in noise reduction/speech enhancement algorithms. Here, we use a minimum mean square error (MMSE)-soft speech presence probability (SPP) (MMSE-SPP)-based noise estimation approach, proposed in [23], for estimation of noise power spectra. In this method, the initial estimate of the noise power spectrum is computed by averaging the first ten frames of the speech spectrum. The advantage of this method is that it does not require a bias correction term as required by a MMSE-based noise spectrum estimation method; it also results in less overestimation of noise power and is computationally less expensive [24]. The reason behind choosing the MMSE-SPP-based noise spectrum estimation method is that it is computationally simple and requires only one parameter to be tuned.

RCGFB utilizes a power function nonlinearity with a coefficient of 0.07 to approximate the loudness nonlinearity of human perception, whereas RMFB uses a logarithmic nonlinearity. For feature normalization, a short-term mean and scale normalization (STMSN) technique is used with a sliding window of 1.5 s. Under mismatched conditions, STMSN helps to remove the difference of log spectrum between the training and test environments by adjusting the short-term mean and scale [19].

2.3 Iterative deconvolution-based features

A general nonnegative matrix factorization (NMF) framework decomposes spectra of reverberated speech in to those of the clean and room impulse response filter. An iterative least squares deconvolution technique can be employed for spectral factorization [25]. The iterative deconvolution (ITD)-based dereverberated mel-filterbank feature extraction method is presented in Fig. 4. It was introduced in [25]. A gammatone filterbank integrated auditory spectrum is computed for each windowed frame. ITD is then applied to each subband in the gammatone frequency domain. ITD, an iterative least squares approach that minimizes the errors [25]

$$ {e}_k={{\displaystyle \sum_i\left(S\left(i,k\right)-{\displaystyle \sum_mX\left(m,k\right){H}_r\left(i-m,k\right)}\right)}}^2, $$

(5)

initialized by NMF, is used to estimate the clean signal X(m, k) and room impulse response H _r (k, m) from the reverberated signal S(m, k) where k is the subband index and m is the frame index. After reconstructing the dereverberated signal, 23-dimensional mel-filterbank features are then computed using the Kaldi toolkit [4].

2.4 Maximum likelihood inverse filtering-based dereverberated features

Maximum likelihood inverse filtering-based dereverberation of a reverberated signal in the cepstral domain, proposed in [2], is shown in Fig. 5. The purpose of cepstral post-filtering is to partially decorrelate the features. If P (P(z) = 1/(1 + pz ⁻¹)) is an inverse impulse response (IIR) dereverberation filter of M taps long, then the dereverberated cepstral features c ^d [m] can be given as

$$ {c}^d\left[m\right]=c\left[m\right]-{\displaystyle \sum_{k=1}^{M-1}p\left[k\right]}{c}^d\left[m-k\right], $$

(6)

where m is the frame index, and c[m] is the cepstral features of the m-th frame of a reverberated speech signal.

Parameters that best describe P can be obtained by maximizing the log likelihood with respect to the Gaussian mixture models (GMM) for all the frames of the speech [2]. A common approximation in GMMs is to replace overall GMM likelihood score by the top N scoring Gaussian density among the set of Gaussian mixtures [2]. Here, similar to [2], we use top-1 approximation for filter updates. We apply cepstral mean normalization (CMN) to remove any constant additive shift in the cepstral features and to partially decorrelate the features cepstral post filtering (CPF) [26] is used.

4.1 Training and test data

In REVERB challenge 2014, the task is to recognize read speech in eight different acoustic conditions. Six of which are simulated by convolving the WSJCAM0 corpus [32] with three measured room impulse responses (RIRs) at near- and far-field microphones and by adding stationary noise from the same room at a signal-to-noise ratio (SNR) of 20 dB (SimData). The other two conditions (RealData) correspond to real recordings of speakers in a reverberant meeting room at two microphone distances with ambient, stationary noise taken from the MC-WSJ-AV corpus [33]. The T60 times range from 0.25 to 0.7 s. The T60 times are assumed to be unknown at the test time. In all conditions, eight channels of a circular microphone array are available, of which the first is used as a reference channel. The SimData set has 1484/2176 utterances from 20/20 speakers in the development and evaluation data. The RealData set has 179/372 utterances from 5/10 speakers [34].

For training the DNN models, we use the multi-condition training data generated from the clean training data (WSJCAM0 corpus) using the scripts provided by the organizer [32, 35]. The multi-condition training set contains artificially distorted data similar to the SimData set. It is of the same size as the clean WSJCAM0 training set, containing 7861 utterances from 92 speakers. The RIRs and noise types are chosen randomly with equal probability. The total number of recordings in the training data is 70,155, and 594 utterances are held out in order to provide a cross-validation set for DNN training. The same training and validation sets are used for training the DNN for different feature parameters derived from the raw signal. This training data corresponds to training with 20K vocabulary recognition tasks [32, 35].

4.2 Training with MFCC features

The maximum likelihood inverse filtering-based dereverberated MFCC features (i.e., MLIFD features) are used to train a DNN-HMM hybrid system. In order to train the DNN-HMM hybrid system, we first train a GMM-HMM system with 3435 tied triphone states and 40K Gaussians. The training process uses the Kaldi toolkit [20], and the training process is similar to that outlined in [36]. In short, the GMM-HMM is trained on features obtained by first normalizing MFCCs to zero mean per speaker, then splicing together 7 frames of 13-dimensional MFCCs and reducing them to 40 dimensions through linear discriminant analysis (LDA) [37], followed by a semi-tied covariance (STC) transform [38]. The resulting features are then transformed through a feature space maximum likelihood linear regression (FMLLR) transform [39]. The LDA + STC + FMLLR [20, 37–40] transformed MFCC features are then used to train the GMM-HMM system having 3435 tied triphone states and 40K Gaussians. The DNN is also trained on the LDA + STC + FMLLR [20, 37–40] transformed MFCC features. These features are globally normalized to have zero mean and unit variance. The input to the neural net is 11 frames (or 440 feature values). The DNN is initialized with stacked RBMs. The resulting DNN (with 5 hidden layers, 1024 neurons in each hidden layer, and 3435 outputs in the output layer) goes through one iteration of sequence training with MPE (minimum phone error) criteria. The training data is re-aligned with the new DNN, and we go through two more iterations of sequence training with the MPE criteria. The resulting DNN-HMM hybrid system is then used for recognition.

4.3 Training with filterbank features

For training DNN-HMM models from the baseline (i.e., conventional mel-filterbank (MFB)) features, from the MMFB_l (multi-taper mel-filterbank with logarithmic nonlinearity) and MMFB_p (multi-taper mel-filterbank with power-law nonlinearity), from the RCGFB (robust compressive gammachirp filterbank) and RMFB (robust mel-filterbank) features, and from the ITD-based dereverberated MFB (ITD-MFB) features, we generate 23-dimensional filterbank features per frame for each of the abovementioned front-ends. The process for generating the DNN from these filterbank features is similar to that outlined in [13] for the DNN-HMM system with speaker adaptation. From the filterbank features, we compute the TRAP features [30] as follows: we first normalize the 23-dimensional filterbank features to zero mean per speaker. Then, 31 frames of these 23-dimensional filterbank features (15 frames on each side of current frame) are spliced together to form a 713-dimensional feature vector. This 713-dimensional feature vector is transformed using a Hamming window (to emphasize the center), passed through a discrete cosine transform, and the dimensionality is reduced to 368. This 368-dimensional feature vector is globally normalized to have zero mean and unit variance. This normalized 368-dimensional feature vector together with a 100 dimensional i-vector [28] (computed from the full utterance) is then input to the seven-layer DNN as shown in Fig. 7. The i-vector was length normalized by dividing the i-vector by the square root of the sum of the squares of its elements. Note that the un-normalized i-vectors are approximately Gaussianized by length normalization [41]. The DNN has 5 hidden layers with 1024 neurons each, and the output softmax layer has around 3500 outputs. The i-vector characterizing a speaker is used as an additional input to the feature layer in order to adapt the DNN to the speaker. The resulting DNN was then used to re-align the training data. This was then followed with four iterations of sequence training with the MMI criteria.

We generated DNNs with two different variants. In the first variant, the i-vectors were computed from only the current utterance. In the second variant, all the utterances in one room per test condition corresponding to the same speaker were used to compute the 100-dimensional length-normalized i-vector for this speaker. Note that we did not use data from other test condition/conditions to perform feature normalization and adaptation per room (Fig. 7).

4.4 Algorithm used for decoding

The decoding algorithm is slightly different depending on whether we are doing utterance-based batch processing or full batch processing. For the maximum likelihood inverse filtering-based dereverberation (MLIFD) features that use FMLLR transform, we only use full batch processing, since we need to compute the FMLLR transform for the speaker from all the utterances of the speaker in the room. Computing the FMLLR transform from a single short utterance gives poor results (we need over 20 s of audio to estimate reasonable FMLLR transforms).

For full batch processing, we first diarize all the utterances in a room per test condition using a modified version of the multi-stage segmentation and clustering system [42]. The modification is that each utterance corresponds to one speaker. There is no sub-segmentation of the utterance. Filterbank features in each speaker cluster are then normalized to zero. We also compute one 100-dimensional length-normalized i-vector per speaker [28].

For utterance-based batch processing, each utterance was labeled as a different speaker, and an i-vector was computed per utterance. In this case, the filterbank features are normalized per utterance and the 100-dimensional length-normalized i-vector is computed per utterance.

4.5 ROVER for combining the results

For the single-channel task, we use ROVER to combine the results of all seven systems to get the lowest possible word error rates (WERs). For the two-channel and eight-channel tasks, we, in place of applying any multi-microphone signal processing algorithms for doing the dereverberation, exploit ROVER to combine the results from all two and all eight channels, respectively, to get the best possible results.