- Research Article
- Open Access
On the Soft Fusion of Probability Mass Functions for Multimodal Speech Processing
EURASIP Journal on Advances in Signal Processing volume 2011, Article number: 294010 (2011)
Multimodal speech processing has been a subject of investigation to increase robustness of unimodal speech processing systems. Hard fusion of acoustic and visual speech is generally used for improving the accuracy of such systems. In this paper, we discuss the significance of two soft belief functions developed for multimodal speech processing. These soft belief functions are formulated on the basis of a confusion matrix of probability mass functions obtained jointly from both acoustic and visual speech features. The first soft belief function (BHT-SB) is formulated for binary hypothesis testing like problems in speech processing. This approach is extended to multiple hypothesis testing (MHT) like problems to formulate the second belief function (MHT-SB). The two soft belief functions, namely, BHT-SB and MHT-SB are applied to the speaker diarization and audio-visual speech recognition tasks, respectively. Experiments on speaker diarization are conducted on meeting speech data collected in a lab environment and also on the AMI meeting database. Audiovisual speech recognition experiments are conducted on the GRID audiovisual corpus. Experimental results are obtained for both multimodal speech processing tasks using the BHT-SB and the MHT-SB functions. The results indicate reasonable improvements when compared to unimodal (acoustic speech or visual speech alone) speech processing.
To access the full article, please see PDF.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Kumar, D., Vimal, P. & Hegde, R.M. On the Soft Fusion of Probability Mass Functions for Multimodal Speech Processing. EURASIP J. Adv. Signal Process. 2011, 294010 (2011). https://doi.org/10.1155/2011/294010
- Probability Mass Function
- Belief Function
- Visual Speech
- Multiple Hypothesis Testing
- Acoustic Speech