Open Access

Noise Adaptive Stream Weighting in Audio-Visual Speech Recognition

  • Martin Heckmann1Email author,
  • Frédéric Berthommier2 and
  • Kristian Kroschel1
EURASIP Journal on Advances in Signal Processing20022002:720764

Published: 28 November 2002


It has been shown that integration of acoustic and visual information especially in noisy conditions yields improved speech recognition results. This raises the question of how to weight the two modalities in different noise conditions. Throughout this paper we develop a weighting process adaptive to various background noise situations. In the presented recognition system, audio and video data are combined following a Separate Integration (SI) architecture. A hybrid Artificial Neural Network/Hidden Markov Model (ANN/HMM) system is used for the experiments. The neural networks were in all cases trained on clean data. Firstly, we evaluate the performance of different weighting schemes in a manually controlled recognition task with different types of noise. Next, we compare different criteria to estimate the reliability of the audio stream. Based on this, a mapping between the measurements and the free parameter of the fusion process is derived and its applicability is demonstrated. Finally, the possibilities and limitations of adaptive weighting are compared and discussed.


audio-visual speech recognition adaptive weighting robust recognition multi-stream recognition ANN/HMM

Authors’ Affiliations

Institut für Nachrichtentechnik, Universität Karlsruhe
Institut de la Communication Parlée (ICP), Institut National Polytechnique de Grenoble


© Heckmann et al. 2002