 Research Article
 Open Access
A New BigramPLSA Language Model for Speech Recognition
 Mohammad Bahrani^{1}Email author and
 Hossein Sameti^{1}
https://doi.org/10.1155/2010/308437
© M. Bahrani and H. Sameti. 2010
 Received: 3 March 2010
 Accepted: 8 July 2010
 Published: 27 July 2010
Abstract
A novel method for combining bigram model and Probabilistic Latent Semantic Analysis (PLSA) is introduced for language modeling. The motivation behind this idea is the relaxation of the "bag of words" assumption fundamentally present in latent topic models including the PLSA model. An EMbased parameter estimation technique for the proposed model is presented in this paper. Previous attempts to incorporate word order in the PLSA model are surveyed and compared with our new proposed model both in theory and by experimental evaluation. Perplexity measure is employed to compare the effectiveness of recently introduced models with the new proposed model. Furthermore, experiments are designed and carried out on continuous speech recognition (CSR) tasks using word error rate (WER) as the evaluation criterion. The superiority of the new bigramPLSA model over Nie et al.'s bigramPLSA and simple PLSA models is demonstrated in the results of our experiments. Experiments on BLLIP WSJ corpus show about 12% reduction in perplexity and 2.8% WER improvement compared to Nie et al.'s bigramPLSA model.
Keywords
 Language Model
 Word Pair
 Topic Model
 Latent Semantic Analysis
 Latent Topic
1. Introduction
Language models are important in various applications especially in speech recognition. Statistical language models are obtained using different approaches depending on the resources and tasks requirements. Extracting gram statistics is a prevalent approach for statistical language modeling. gram takes the order of words into account and calculates the probability of the word occurring after other known words.
Many attempts have been made to incorporate semantic knowledge in language modeling. Latent topic modeling approaches such as Latent Semantic Analysis (LSA) [1, 2], Probabilistic Latent Semantic Analysis (PLSA) [3], and Latent Dirichlet Allocation (LDA) [4] are the most recent techniques. Latent semantic information is extracted by these models through decomposing worddocument cooccurrence matrix. These topic models have been successful in reducing the perplexity and improving the accuracy rate of speech recognition systems [2, 5, 6]. The main deficiency of the topic models is that they do not take the order of words into consideration due to the assumption of "bag of words" intrinsically.
The useful semantic modeling of the topic models and the potential of considering words history in the gram language model motivate researchers to combine the capabilities of both approaches. Bellegarda [2] proposed the combination of the gram and the LSA models and Federico [7] utilized the PLSA framework to adapt the gram language model. Both [2, 7] used rescaling approach for the combination. Griffiths et al. [8] presented an extension of the topic model that is sensitive to word order and automatically learns the syntactic factors as well as the semantic ones. In [9, 10] the collocation of words was incorporated in the LDA model. Girolami and Kaban [11] relaxed the "bag of words" assumption in the LDA model by applying the Markov chain assumption on symbol sequences. Wallach [12] proposed a combination of bigram and LDA models (the bigram topic model) and achieved a significant performance improvement on perplexity by exploring latent semantics following different context words. This research was a basis for Nie et al.'s work [13] that proposed the combination of bigram and PLSA models. The performance improvements achieved in [12, 13] motivated us to propose a general framework for combining bigram and PLSA models. As discussed in Section 3.6, our model is different from Nie et al.'s work and can be considered as a generalization to that model. One cannot derive the reestimation formulae via the standard EM procedure based on Nie et al.'s model. In this paper, we propose an EM procedure for reestimating the parameters of our model.
The remainder of the paper is organized as follows. In Section 2, the PLSA model is briefly reviewed. In Section 3, the combination of bigram and PLSA models is introduced and its parameter estimation procedure is described. In Section 4, experimental results are presented and finally in Section 5 the conclusions are made.
2. Review of the PLSA Model
where is a latent class variable (or a topic) belonging to a set of class variables (topics) . Equation (1) is a weighted mixture of word distributions called aspect model [14]. The aspect model is a latent variable model for cooccurrence data that associates an unobserved class variable to each observation (i.e., words and documents). The aspect model introduces a conditional independence assumption, that is, d _{ j } and w _{ i } are independent conditioned on the state of the associated latent variable [15]. In (1), , , are the word distributions and , , are the weights of distributions.
In another view, the PLSA model is a decomposition of worddocument cooccurrence matrix . The matrix is decomposed into and matrices in order to minimize the cross entropy (KL divergence) between the matrix and empirical distribution.
The PLSA parameters and are reestimated via the EM procedure. The EM procedure includes two alternate steps: (i) an expectation (E) step where posterior probabilities are computed for the latent variables based on the current estimates of the parameters, (ii) a maximization (M) step where PLSA parameters are updated based on the posterior probabilities computed in the Estep [15].
3. Combining Bigram and PLSA Models
Before describing the proposed model, the previous research on combining bigram and PLSA model by Nie et al. [13] is reviewed. This method is a special case (with certain independence assumptions) of our proposed method.
3.1. Nie et al.'s BigramPLSA Model
The EM procedure for training the combined model contains the following two steps.
where is the number of times that the word pair occurs in the document d _{ k } , and is the number of words in the document d _{ k }.
3.2. Proposed BigramPLSA Model
We intend to combine the bigram and the PLSA models to take advantage of the strengths of both models for increasing the predictability of words in documents. In order to combine bigram and PLSA models, we incorporate the context word in the PLSA parameters. In other words, we associate the generation of words and documents to the context word in addition to the latent topics.
 (1)
Generate a context word as the word history with probability .
 (2)
Select a document d _{ k } with probability .
 (3)
Pick a latent variable z _{ l } with probability .
 (4)
Generate a word w _{ j } with probability .
Equation (7) is an extended version of the aspect model that considers the word history in the worddocument modeling and can be considered as a combination of bigram and PLSA models. In (7), the distributions and are the model parameters that should be estimated from training data. This model is similar to the original PLSA model except that the context words (word history) is incorporated in the model parameters.
The justification behind the assumed conditional independence in the proposed model is the same reasoning that the PLSA model is using to make an analytical model, that is, simplification of the model formulation and reasonable reduction of the computational cost.
3.3. Parameter Estimation Using the EM Algorithm
Like original PLSA model, we reestimate the parameters of bigramPLSA model using the EM procedure. In the EM procedure, for Estep, we simply apply Bayes' rule to obtain the posterior probability of the latent variable z _{ l } given the observed data d _{ k }, , and w _{ j }.
where is the occurrence probability of the word pair in the document and is the frequency of word pair in the document .
where and are the Lagrange multipliers related to constraints specified in (16).
The Estep and Mstep are repeated until convergence criterion is met.
3.4. Implementation and Complexity Analysis
For implementing the EM algorithm, in the Estep, we need to calculate for all i, j, k, and l. It requires four nested loops. Thus the time complexity of the Estep is , where M, N, and K are the number of words, the number of documents, and the number of latent topics respectively. The memory requirements in the Estep include a fourdimensional matrix for saving and a threedimensional matrix for saving the normalization parameter (denominator of (11)). For reducing the memory requirements, note that it is not necessary to calculate and save at the Estep; rather, it can be calculated in the Mstep by multiplying the previous and and dividing the result by the normalization parameter. Therefore, we save only the normalization parameter at the Estep. According to (7), the normalization parameter is equal to , thus the related matrix contains elements, which is a large number for typical values of M and N.
In the Mstep, we need to calculate the model parameters and specified in (19). These calculations require four nested loops, but note that we can decrease the number of loops to three nested loops by considering only the word pairs that are present in the training documents instead of all word pairs. Thus the time complexity in the Mstep is O(KNB) where B is the average number of the word pairs in the training documents.
The memory requirements in the Mstep include two threedimensional matrices for saving and and two twodimensional matrices for saving the denominators of (19). Saving these large matrices results in high memory requirements in the training process. is another matrix that can be implemented by a sparse matrix containing the indices of the word pairs presented in each training document and the counts of the word pairs.
3.5. Extension to gram
where is a sequence of words. We can follow the same EM procedure for parameter estimation in the gramPLSA model where w _{ i } is replaced by in all formulae. In the reestimation formulae, we have that is the number of occurrences of the word sequence in the document d _{ k }.
Combining PLSA model and gram model for leads to high complexity in time and memory of the training process. As discussed in Section 3.4, the time complexity of the EM algorithm is for . Consequently, the time complexity for higher order grams is that grows exponentially as n increases. In addition, the memory requirement for gramPLSA combination is very high. For example, for saving the normalization parameters, we need a ( )dimensional matrix which contains elements. Therefore, the memory requirement also grows exponentially as n increases.
3.6. Comparison with Nie et al.'s BigramPLSA Model
According to (21), the difference between our model and Nie et al.'s model is in the definition of the topic probability. In Nie et al.'s model the topic probability is conditioned on the documents, but in our model, the topic probability is further conditioned on the bigram history. In Nie et al.'s model, the assumption of independence between the latent topics and the context words leads to assigning the latent topics to each context word evenly, that is, the same numbers of latent variables are assigned to decompose the worddocument matrices of all context words despite their different complexities. Thus, they propose a refining procedure that unevenly assigns the latent topics to the context words according to an estimation of their latent semantic complexities.
In our proposed bigramPLSA model, we relax the assumption of independence between the latent topics and the context words and achieve a general form of the aspect model that considers the word history in the worddocument modeling. Our model automatically assigns the latent topics to the context words unevenly because for each context h _{ i }, there is a distribution that assigns the appropriate number of latent topics to that context. Consequently, remains zero for those z _{ l } inappropriate to the context word w _{ i }.
The number of free parameters in our proposed model is , whereM, N, and K are the number of words, the number of documents, and the number latent topics, respectively. On the other hand, the number of free parameters in Nie et al.'s model is that is less than the number of free parameters in our model. Consequently, the training time of Nie et al.'s model is less than the training time of our model.
4. Experimental Results
The bigramPLSA model was evaluated using two different criteria: perplexity and word error rate of a CSR system. We selected 500 documents containing about 248600 words from BLLIP WSJ corpus and used them to train our proposed bigramPLSA model. We replaced all stop words of the training documents with a unique symbol (#STOP) and considered all infrequent words (the words occurring only once) as unknown words and replaced them with UNK symbol. After these replacements, the vocabulary contained about 3800 words. We could not include more documents in the training process because the computational cost and memory requirement grow rapidly as the size of the training set increases (as discussed in Section 3.4). For training the bigramPLSA model, first we set the number of the latent topics between 10 and 50 and initialized the model randomly, then we executed the EM algorithm until it converged. We evaluated the bigramPLSA model on 50 documents, with 22300 words in total, not overlapped with the training data. This evaluation process was run ten times for different random initial models and the results were averaged.
where was obtained from the value of in the bigramPLSA model. Since document d was not present in the training data, we had to follow the foldingin procedure mentioned in [5] to calculate . Within this procedure, the parameters were assumed constant and the EM algorithm was employed to calculate only parameters for and for those w _{ i } present in the document d. After convergence of the EM procedure, was found. Obtained matrix contained many zero probabilities, thus we smoothed it using WittenBell smoothing method [16]. Note that the foldingin procedure gives the PLSA and the bigramPLSA models an unfair advantage by allowing them to adapt the model parameters to the test data. Nevertheless, we applied it to avoid overfitting.
The values obtained from the pairedttest on perplexity results of Nie et al.'s and proposed method for different numbers of latent topics ( ).
 value 

10 

20 

30 

40 

50 

Perplexities, number of parameters, and the computation cost of the bigramPLSA model and other language models.
Model  Calculated parameter  Number of model parameters  Time of each EM iteration  Perplexity 

bigram 
 Maximum 3800^{2}  —  198 
trigram 
 Maximum 3800^{3}  —  134 
PLSA 
 215000  0.6 second 

Bigram & PLSA (linear interpolation) 
 14655000  0.6 second 

BigramPLSA (Nie et al.'s) 
 577620000  19 minutes 

BigramPLSA (proposed) 
 653600000  24 minutes 

We set in our experiments. This value for was obtained by optimizing it on the heldout data.
Average word error rates of the CSR system using PLSAbased language models with and without trigram language model in decoding.
Language Model (for Nbest rescoring)  WER (%) (trigram in decoding)  WER (%) (No LM in decoding)  Average decoding time (Sec.) 

—  12.66  74.24  0.8 
PLSA 

 4.5 
BigramPLSA (Nie et al.'s) 

 131 
BigramPLSA (proposed) 

 140 
As Table 3 shows, the PLSA and the bigramPLSA models improve the word error rate. In addition, the word error rate obtained from the bigramPLSA model is meaningfully lower than that of the PLSA model. Our proposed bigramPLSA model shows slight improvement compared to Nie et al.'s bigramPLSA model. The third column better demonstrates the effect of the bigramPLSA model in reducing the word error rate. The average decoding time is given in the last column of Table 3. It is observed that WER is improved for the cost of increasing the decoding time, but the increase in the decoding time compared to the Nie et al.'s model is insignificant.
The values obtained from the pairedttest on WER results of Nie et al.'s and proposed method.
LM in decoding  value 

Trigram 

No LM 

5. Conclusions and Future Work
In this paper, a general framework for combining bigram and PLSA models was proposed. The combined model was obtained from incorporating the word history in the PLSA parameters. Furthermore, the EM procedure for estimating the parameters of the combined model was described. Finally, the proposed model was compared to the previous work done on combining the bigram and the PLSA models by Nie et al. Our proposed model is different from Nie et al.'s model in the definition of the topic probability. In Nie et al.'s model the topic probability is conditioned on the documents, but in our model, the topic probability is further conditioned on the bigram history. The proposed model automatically assigns latent topics to each context word unevenly in contrast to the even assignment of them by Nie et al.'s initial bigramPLSA model. We arranged experiments to evaluate our combined model based on the perplexity and the word error rate criteria. Experiments showed that our proposed bigramPLSA model outperformed the PLSA model according to the both criteria. The proposed model also showed slight superiority over Nie et al.'s bigramPLSA model in improving perplexity and WER. As our future research work, we intend to suggest a similar framework to combine gram and LDA models. We also plan to use automatic smoothing in our parameter estimation process without requiring it to be done as an extra step as it is the stateoftheart in Bayesian machine learning methods.
Declarations
Acknowledgment
This paper was in part supported by a grant from Iran Telecommunication Research Center (ITRC).
Authors’ Affiliations
References
 Deerwester S, Dumais S, Furnas G, Landauer T, Harshman R: Indexing by latent semantic analysis. Journal of the American Society of Information Science 1990, 41: 391407. 10.1002/(SICI)10974571(199009)41:6<391::AIDASI1>3.0.CO;29View ArticleGoogle Scholar
 Bellegarda JR: Exploiting latent semantic information in statistical language modeling. Proceedings of the IEEE 2000, 88(8):12791296. 10.1109/5.880084View ArticleGoogle Scholar
 Hofmann T: Probabilistic latent semantic indexing. Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1999, Berkeley, Calif, USA 5057.Google Scholar
 Blei DM, Ng AY, Jordan MI: Latent Dirichlet allocation. Journal of Machine Learning Research 2003, 3(45):9931022.MATHGoogle Scholar
 Gildea D, Hofmann T: Topicbased language models using EM. Proceedings of the 6th European Conference on Speech Communication and Technology (EUROSPEECH '99), 1999, Budapest, Hungary 235238.Google Scholar
 Mrva D, Woodland PC: Unsupervised language model adaptation for mandarin broadcast conversation transcription. Proceedings of International Conference on Spoken Language Processing, 2006, Pittsburgh, Pa, USA 15491552.Google Scholar
 Federico M: Language model adaptation through topic decomposition and MDI estimation. Proceedings of International Conference on Acoustics, Speech and Signal Processing, 2002, Orlando, Fla, USA 773776.Google Scholar
 Griffiths T, Steyvers M, Blei D, Tenenbaum J: Integrating topics and syntax. Advances in Neural Information Processing Systems 17, December 2004, Vancouver, Canada 8794.Google Scholar
 Griffiths TL, Steyvers M, Tenenbaum JB: Topics in semantic representation. Psychological Review 2007, 114(2):211244.View ArticleGoogle Scholar
 Wang X, McCallum A: A note on topical ngrams. University of Massachusetts, Amherst, Mass, USA; December 2005.Google Scholar
 Girolami M, Kaban A: Simplicial mixtures of Markov chains: distributed modeling of dynamic user profiles. In Advances in Neural Information Processing Systems 16, December 2003, Vancouver, Canada. MIT Press; 916.Google Scholar
 Wallach HM: Topic modeling: beyond bagofwords. Proceedings of the 23rd International Conference on Machine Learning (ICML '06), June 2006, Pittsburgh, Pa, USA 977984.View ArticleGoogle Scholar
 Nie J, Li R, Luo D, Wu X: Refine bigram PLSA model by assigning latent topics unevenly. Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, 2007, Kyoto, Japan 141146.Google Scholar
 Hofmann T, Puzicha J, Jordan MI: Learning from dyadic data. Advances in Neural Information Processing Systems 11, NovemberDecember 1998, Denver, Colo, USA 466472.Google Scholar
 Hofmann T: Unsupervised learning by probabilistic latent semantic analysis. Machine Learning 2001, 42(12):177196.View ArticleMATHGoogle Scholar
 Witten IH, Bell TC: The zerofrequency problem: estimating the probabilities of novel events in adaptive text compression. IEEE Transactions on Information Theory 1991, 37(4):10851094. 10.1109/18.87000View ArticleGoogle Scholar
 Katz SM: Estimation of probabilities from sparse data for the language model component of speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing 1987, 35(3):400401. 10.1109/TASSP.1987.1165125View ArticleGoogle Scholar
 Walker W, Lamere P, Kwok P, et al.: Sphinx4: a flexible open source framework for speech recognition. SUN Microsystems; November 2004.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.