Open Access

Automatic Speechreading with Applications to Human-Computer Interfaces

  • Xiaozheng Zhang1Email author,
  • Charles C. Broun2,
  • Russell M. Mersereau1 and
  • Mark A. Clements1
EURASIP Journal on Advances in Signal Processing20022002:240192

Received: 30 October 2001

Published: 28 November 2002


There has been growing interest in introducing speech as a new modality into the human-computer interface (HCI). Motivated by the multimodal nature of speech, the visual component is considered to yield information that is not always present in the acoustic signal and enables improved system performance over acoustic-only methods, especially in noisy environments. In this paper, we investigate the usefulness of visual speech information in HCI related applications. We first introduce a new algorithm for automatically locating the mouth region by using color and motion information and segmenting the lip region by making use of both color and edge information based on Markov random fields. We then derive a relevant set of visual speech parameters and incorporate them into a recognition engine. We present various visual feature performance comparisons to explore their impact on the recognition accuracy, including the lip inner contour and the visibility of the tongue and teeth. By using a common visual feature set, we demonstrate two applications that exploit speechreading in a joint audio-visual speech signal processing task: speech recognition and speaker verification. The experimental results based on two databases demonstrate that the visual information is highly effective for improving recognition performance over a variety of acoustic noise levels.


automatic speechreading visual feature extraction Markov random fields hidden Markov models polynomial classifier speech recognition speaker verification

Authors’ Affiliations

Center for Signal and Image Processing, Georgia Institute of Technology
Motorola Human Interface Lab


© Zhang et al. 2002