Skip to content


  • Research Article
  • Open Access

Multimodal Translation System Using Texture-Mapped Lip-Sync Images for Video Mail and Automatic Dubbing Applications

EURASIP Journal on Advances in Signal Processing20042004:509796

Received: 25 November 2002

Published: 18 September 2004


We introduce a multimodal English-to-Japanese and Japanese-to-English translation system that also translates the speaker's speech motion by synchronizing it to the translated speech. This system also introduces both a face synthesis technique that can generate any viseme lip shape and a face tracking technique that can estimate the original position and rotation of a speaker's face in an image sequence. To retain the speaker's facial expression, we substitute only the speech organ's image with the synthesized one, which is made by a 3D wire-frame model that is adaptable to any speaker. Our approach provides translated image synthesis with an extremely small database. The tracking motion of the face from a video image is performed by template matching. In this system, the translation and rotation of the face are detected by using a 3D personal face model whose texture is captured from a video frame. We also propose a method to customize the personal face model by using our GUI tool. By combining these techniques and the translated voice synthesis technique, an automatic multimodal translation can be achieved that is suitable for video mail or automatic dubbing systems into other languages.

Keywords and phrases

audio-visual speech translationlip-sync talking headface tracking with 3D templatevideo mail and automatic dubbingtexture-mapped facial animationpersonal face model

Authors’ Affiliations

School of Science and Engineering, Waseda University, Tokyo, Japan
ATR Spoken Language Translation Research Laboratories, Kyoto, Japan


© Morishima and Nakamura 2004

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.