A New Bigram-PLSA Language Model for Speech Recognition

Language models are important in various applications especially in speech recognition. Statistical language models are obtained using different approaches depending on the resources and tasks requirements. Extracting -gram statistics is a prevalent approach for statistical language modeling. -gram takes the order of words into account and calculates the probability of the word occurring after other known words.

Many attempts have been made to incorporate semantic knowledge in language modeling. Latent topic modeling approaches such as Latent Semantic Analysis (LSA) [1, 2], Probabilistic Latent Semantic Analysis (PLSA) [3], and Latent Dirichlet Allocation (LDA) [4] are the most recent techniques. Latent semantic information is extracted by these models through decomposing word-document cooccurrence matrix. These topic models have been successful in reducing the perplexity and improving the accuracy rate of speech recognition systems [2, 5, 6]. The main deficiency of the topic models is that they do not take the order of words into consideration due to the assumption of "bag of words" intrinsically.

The useful semantic modeling of the topic models and the potential of considering words history in the -gram language model motivate researchers to combine the capabilities of both approaches. Bellegarda [2] proposed the combination of the -gram and the LSA models and Federico [7] utilized the PLSA framework to adapt the -gram language model. Both [2, 7] used rescaling approach for the combination. Griffiths et al. [8] presented an extension of the topic model that is sensitive to word order and automatically learns the syntactic factors as well as the semantic ones. In [9, 10] the collocation of words was incorporated in the LDA model. Girolami and Kaban [11] relaxed the "bag of words" assumption in the LDA model by applying the Markov chain assumption on symbol sequences. Wallach [12] proposed a combination of bigram and LDA models (the bigram topic model) and achieved a significant performance improvement on perplexity by exploring latent semantics following different context words. This research was a basis for Nie et al.'s work [13] that proposed the combination of bigram and PLSA models. The performance improvements achieved in [12, 13] motivated us to propose a general framework for combining bigram and PLSA models. As discussed in Section 3.6, our model is different from Nie et al.'s work and can be considered as a generalization to that model. One cannot derive the re-estimation formulae via the standard EM procedure based on Nie et al.'s model. In this paper, we propose an EM procedure for re-estimating the parameters of our model.

The remainder of the paper is organized as follows. In Section 2, the PLSA model is briefly reviewed. In Section 3, the combination of bigram and PLSA models is introduced and its parameter estimation procedure is described. In Section 4, experimental results are presented and finally in Section 5 the conclusions are made.

Suppose that we have a set of words that composes a set of documents . In the PLSA model, the occurrence probability of word w _i given document d _j is defined as below [3].

(1)

where is a latent class variable (or a topic) belonging to a set of class variables (topics) . Equation (1) is a weighted mixture of word distributions called aspect model [14]. The aspect model is a latent variable model for co-occurrence data that associates an unobserved class variable to each observation (i.e., words and documents). The aspect model introduces a conditional independence assumption, that is, d _j and w _i are independent conditioned on the state of the associated latent variable [15]. In (1), , , are the word distributions and , , are the weights of distributions.

In another view, the PLSA model is a decomposition of word-document co-occurrence matrix . The matrix is decomposed into and matrices in order to minimize the cross entropy (KL divergence) between the matrix and empirical distribution.

The PLSA parameters and are re-estimated via the EM procedure. The EM procedure includes two alternate steps: (i) an expectation (E) step where posterior probabilities are computed for the latent variables based on the current estimates of the parameters, (ii) a maximization (M) step where PLSA parameters are updated based on the posterior probabilities computed in the E-step [15].

Before describing the proposed model, the previous research on combining bigram and PLSA model by Nie et al. [13] is reviewed. This method is a special case (with certain independence assumptions) of our proposed method.

3.1. Nie et al.'s Bigram-PLSA Model

Nie et al. presented a combination of bigram and PLSA models [13]. Instead of in (1), their bigram-PLSA model employs resulting in

(2)

The EM procedure for training the combined model contains the following two steps.

E-step:

(3)

M-step:

(4)

(5)

where is the number of times that the word pair occurs in the document d _k , and is the number of words in the document d _k .

3.2. Proposed Bigram-PLSA Model

We intend to combine the bigram and the PLSA models to take advantage of the strengths of both models for increasing the predictability of words in documents. In order to combine bigram and PLSA models, we incorporate the context word in the PLSA parameters. In other words, we associate the generation of words and documents to the context word in addition to the latent topics.

The generative process of bigram-PLSA model can be defined by the following scheme:

1. (1)

   Generate a context word as the word history with probability .

    
2. (2)

   Select a document d _k with probability .

    
3. (3)

   Pick a latent variable z _l with probability .

    
4. (4)

   Generate a word w _j with probability .

    

Translating the generative process into a joint probability model results in

(6)

According to (6), the occurrence probability of the word w _j given the document d _k and the word history is defined as

(7)

Equation (7) is an extended version of the aspect model that considers the word history in the word-document modeling and can be considered as a combination of bigram and PLSA models. In (7), the distributions and are the model parameters that should be estimated from training data. This model is similar to the original PLSA model except that the context words (word history) is incorporated in the model parameters.

Like the original aspect model, the extended aspect model assumes conditional independence between word w _j and document d _k , that is, w _j and d _k are independent conditioned on the latent parameter z _l and the context word :

(8)

The justification behind the assumed conditional independence in the proposed model is the same reasoning that the PLSA model is using to make an analytical model, that is, simplification of the model formulation and reasonable reduction of the computational cost.

As in the original PLSA model, the equivalent parameterization of the joint probability in (6) can be written as

(9)

3.3. Parameter Estimation Using the EM Algorithm

Like original PLSA model, we re-estimate the parameters of bigram-PLSA model using the EM procedure. In the EM procedure, for E-step, we simply apply Bayes' rule to obtain the posterior probability of the latent variable z _l given the observed data d _k , , and w _j .

E-step:

(10)

We can rewrite (10) as

(11)

In the M-step, the parameters are updated by maximizing the log-likelihood of the complete data (words and documents) with respect to the probabilistic model. The likelihood of the complete data with respect to the probabilistic model is computed as

(12)

where is the occurrence probability of the word pair in the document and is the frequency of word pair in the document .

Let be the set of model parameters. For estimating , we use MLE to maximize the log-likelihood of the complete data:

(13)

Considering (7), we expand the above equation to

(14)

In (14), the left factor before the plus sign is omitted because it is independent of . In order to maximize the log-likelihood, (14) should be differentiated. Differentiating (14) with respect to the parameters does not lead to well-formed formulae, so we try to find a lower bound for (14) using Jensen's inequality

(15)

The obtained lower bound should be maximized, that is, maximizing the right hand side of (15) instead of its left hand side. For maximizing the lower bound and re-estimating the parameters, we have a constrained optimization problem because all parameters indicate probability distributions. Therefore, the parameters should satisfy the constraints

(16)

In order to consider the above constraints, the right hand side of (15) has to be augmented by the appropriate Lagrange multipliers

(17)

where and are the Lagrange multipliers related to constraints specified in (16).

Differentiating the above equation partially with respect to the different parameters leads to(18)

(18)

Solving (18) and applying the constraints (16), the M-step re-estimation formulae, (19), are obtained:

(19)

The E-step and M-step are repeated until convergence criterion is met.

3.5. Extension to -gram

We can extend the bigram-PLSA model to -gram-PLSA model by considering the context words instead of only one context word w _i as the word history. The generative process of the -gram-PLSA model is similar to the bigram-PLSA model except that in step 1, instead of generating one context word, context words should be generated. Therefore, the combined model can be expressed by

(20)

where is a sequence of words. We can follow the same EM procedure for parameter estimation in the -gram-PLSA model where w _i is replaced by in all formulae. In the re-estimation formulae, we have that is the number of occurrences of the word sequence in the document d _k .

Combining PLSA model and -gram model for leads to high complexity in time and memory of the training process. As discussed in Section 3.4, the time complexity of the EM algorithm is for . Consequently, the time complexity for higher order -grams is that grows exponentially as n increases. In addition, the memory requirement for -gram-PLSA combination is very high. For example, for saving the normalization parameters, we need a ( )-dimensional matrix which contains elements. Therefore, the memory requirement also grows exponentially as n increases.

3.6. Comparison with Nie et al.'s Bigram-PLSA Model

As discussed in Section 3.1, Nie et al. have presented a combination of bigram and PLSA models in 2007 [13]. This work does not have a strong mathematical foundation and one cannot derive the re-estimation formulae via the standard EM procedure based on that. Nie et al.'s work is based on an assumption of independence between the latent topics z _l and the context words w _i . According to this assumption, we can rewrite (7) as

(21)

According to (21), the difference between our model and Nie et al.'s model is in the definition of the topic probability. In Nie et al.'s model the topic probability is conditioned on the documents, but in our model, the topic probability is further conditioned on the bigram history. In Nie et al.'s model, the assumption of independence between the latent topics and the context words leads to assigning the latent topics to each context word evenly, that is, the same numbers of latent variables are assigned to decompose the word-document matrices of all context words despite their different complexities. Thus, they propose a refining procedure that unevenly assigns the latent topics to the context words according to an estimation of their latent semantic complexities.

In our proposed bigram-PLSA model, we relax the assumption of independence between the latent topics and the context words and achieve a general form of the aspect model that considers the word history in the word-document modeling. Our model automatically assigns the latent topics to the context words unevenly because for each context h _i , there is a distribution that assigns the appropriate number of latent topics to that context. Consequently, remains zero for those z _l inappropriate to the context word w _i .

The number of free parameters in our proposed model is , whereM, N, and K are the number of words, the number of documents, and the number latent topics, respectively. On the other hand, the number of free parameters in Nie et al.'s model is that is less than the number of free parameters in our model. Consequently, the training time of Nie et al.'s model is less than the training time of our model.

The bigram-PLSA model was evaluated using two different criteria: perplexity and word error rate of a CSR system. We selected 500 documents containing about 248600 words from BLLIP WSJ corpus and used them to train our proposed bigram-PLSA model. We replaced all stop words of the training documents with a unique symbol (#STOP) and considered all infrequent words (the words occurring only once) as unknown words and replaced them with UNK symbol. After these replacements, the vocabulary contained about 3800 words. We could not include more documents in the training process because the computational cost and memory requirement grow rapidly as the size of the training set increases (as discussed in Section 3.4). For training the bigram-PLSA model, first we set the number of the latent topics between 10 and 50 and initialized the model randomly, then we executed the EM algorithm until it converged. We evaluated the bigram-PLSA model on 50 documents, with 22300 words in total, not overlapped with the training data. This evaluation process was run ten times for different random initial models and the results were averaged.

The perplexity of evaluation data was calculated as follows:

(22)

where was obtained from the value of   in the bigram-PLSA model. Since document d was not present in the training data, we had to follow the folding-in procedure mentioned in [5] to calculate . Within this procedure, the parameters were assumed constant and the EM algorithm was employed to calculate only parameters for and for those w _i present in the document d. After convergence of the EM procedure, was found. Obtained matrix contained many zero probabilities, thus we smoothed it using Witten-Bell smoothing method [16]. Note that the folding-in procedure gives the PLSA and the bigram-PLSA models an unfair advantage by allowing them to adapt the model parameters to the test data. Nevertheless, we applied it to avoid overfitting.

To have a valid comparison, the PLSA and Nie et al.'s bigram-PLSA models were trained by the same data employed to train our proposed bigram-PLSA model. The folding-in procedure and Witten-Bell smoothing were also applied on the PLSA and Nie et al.'s bigram-PLSA models. Figure 1 shows the perplexities of the proposed and Nie et al.'s bigram-PLSA models for different numbers of latent topics averaged over ten times of running the experiment. In this figure, the error bars show the standard errors of the average perplexities. As seen in Figure 1, the perplexity of our proposed bigram-PLSA model is lower than the perplexity of Nie et al.'s bigram-PLSA model. The best perplexity was obtained when the number of latent topics was set to 40 in both models. Therefore, in the rest of experiments the numbers of latent topics were set accordingly.

In addition, we performed the paired t-test on the perplexity results of both methods with the significance level of 0.01. As stated, each experiment was carried out ten times. The null hypothesis is whether the average perplexities of two methods are the same. Table 1 shows the -value obtained from the paired t-test for our experiments performed with different numbers of latent topics. The right column of Table 1 shows the -value where the alternative hypothesis is whether the average perplexity of our method is less than the average perplexity of Nie et al.'s method. All -values obtained are smaller than the specified significance level. Therefore, the perplexity improvements are statistically significant.

	-value
10
20
30
40
50

Table 2 shows the comparison between the average perplexities of the bigram-PLSA model and other language models. The standard errors of the average perplexities, the number of model parameters and the approximate time of each EM iteration are reported in this table. Note that the number of model parameters for the bigram and trigram language models are equal to the number of word pairs and word triplets observed in the training data, respectively. The numbers shown in Table 2 are the maximum possible number of the word pairs and triplets. In this table, the perplexities of the bigram and trigram language models, the PLSA model, and linear interpolations of the PLSA model and the bigram model are also shown. The bigram and trigram language models were trained by the training data discussed above and the Katz backoff smoothing method [17] was applied on them. Stop words and infrequent words of training data were replaced by #STOP and UNK symbols. The number of latent topics was set to 40 in the bigram-PLSA models and 50 in the PLSA model because for the PLSA model the best perplexity was obtained when the number of latent topics was set to 50. In case of linear interpolation, in (22) was calculated as follows:

(23)

We set in our experiments. This value for was obtained by optimizing it on the held-out data.

As Table 3 shows, the PLSA and the bigram-PLSA models improve the word error rate. In addition, the word error rate obtained from the bigram-PLSA model is meaningfully lower than that of the PLSA model. Our proposed bigram-PLSA model shows slight improvement compared to Nie et al.'s bigram-PLSA model. The third column better demonstrates the effect of the bigram-PLSA model in reducing the word error rate. The average decoding time is given in the last column of Table 3. It is observed that WER is improved for the cost of increasing the decoding time, but the increase in the decoding time compared to the Nie et al.'s model is insignificant.

In addition, we performed paired t-test on WER results of the Nie et al.'s and the proposed methods. The significance level was set to be 0.01. Table 4 shows the -values obtained from the paired t-test. As this table shows, the WER improvements are statistically significant.

LM in decoding	-value
Trigram
No LM

In this paper, a general framework for combining bigram and PLSA models was proposed. The combined model was obtained from incorporating the word history in the PLSA parameters. Furthermore, the EM procedure for estimating the parameters of the combined model was described. Finally, the proposed model was compared to the previous work done on combining the bigram and the PLSA models by Nie et al. Our proposed model is different from Nie et al.'s model in the definition of the topic probability. In Nie et al.'s model the topic probability is conditioned on the documents, but in our model, the topic probability is further conditioned on the bigram history. The proposed model automatically assigns latent topics to each context word unevenly in contrast to the even assignment of them by Nie et al.'s initial bigram-PLSA model. We arranged experiments to evaluate our combined model based on the perplexity and the word error rate criteria. Experiments showed that our proposed bigram-PLSA model outperformed the PLSA model according to the both criteria. The proposed model also showed slight superiority over Nie et al.'s bigram-PLSA model in improving perplexity and WER. As our future research work, we intend to suggest a similar framework to combine -gram and LDA models. We also plan to use automatic smoothing in our parameter estimation process without requiring it to be done as an extra step as it is the state-of-the-art in Bayesian machine learning methods.