Skip to content


  • Research Article
  • Open Access

Semantic Context Detection Using Audio Event Fusion

EURASIP Journal on Advances in Signal Processing20062006:027390

  • Received: 31 August 2004
  • Accepted: 5 April 2005
  • Published:


Semantic-level content analysis is a crucial issue in achieving efficient content retrieval and management. We propose a hierarchical approach that models audio events over a time series in order to accomplish semantic context detection. Two levels of modeling, audio event and semantic context modeling, are devised to bridge the gap between physical audio features and semantic concepts. In this work, hidden Markov models (HMMs) are used to model four representative audio events, that is, gunshot, explosion, engine, and car braking, in action movies. At the semantic context level, generative (ergodic hidden Markov model) and discriminative (support vector machine (SVM)) approaches are investigated to fuse the characteristics and correlations among audio events, which provide cues for detecting gunplay and car-chasing scenes. The experimental results demonstrate the effectiveness of the proposed approaches and provide a preliminary framework for information mining by using audio characteristics.


  • Support Vector Machine
  • Content Analysis
  • Markov Model
  • Hide Markov Model
  • Quantum Information

Authors’ Affiliations

Department of Computer Science and Information Engineering, National Taiwan University, Taipei, 106, Taiwan
Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, 106, Taiwan


  1. Lienhart RW: Comparison of automatic shot boundary detection algorithms. Storage and Retrieval for Image and Video Databases VII, January 1999, San Jose, Calif, USA, Proceedings of SPIE 3656: 290–301.View ArticleGoogle Scholar
  2. Hanjalic A: Shot-boundary detection: unraveled and resolved? IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(2):90–105. 10.1109/76.988656View ArticleGoogle Scholar
  3. Chang S-F, Vetro A: Video adaptation: concepts, technologies, and open issues. Proceedings of the IEEE 2005, 93(1):148–158.View ArticleGoogle Scholar
  4. Lu L, Zhang H-J, Jiang H: Content analysis for audio classification and segmentation. IEEE Transactions Speech Audio Processing 2002, 10(7):504–516. 10.1109/TSA.2002.804546View ArticleGoogle Scholar
  5. Zhang T, Jay Kuo C-C: Hierarchical system for content-based audio classification and retrieval. Multimedia Storage and Archiving Systems III, November 1998, Boston, Mass, USA, Proceedings of SPIE 3527: 398–409.View ArticleGoogle Scholar
  6. Tzanetakis G, Cook P: Musical genre classification of audio signals. IEEE Transactions Speech Audio Processing 2002, 10(5):293–302. 10.1109/TSA.2002.800560View ArticleGoogle Scholar
  7. Lu L, Zhang H-J: Automated extraction of music snippets. Proc. 11th ACM International Conference on Multimedia, November 2003, Berkeley, Calif, USA 140–147.Google Scholar
  8. Fischer S, Lienhart R, Effelsberg W: Automatic recognition of film genres. Proc. 3rd ACM International Conference on Multimedia, November 1995, San Francisco, Calif, USA 295–304.Google Scholar
  9. Liu Z, Huang J, Wang Y: Classification of TV programs based on audio information using hidden Markov model. Proc. IEEE 2nd Workshop on Multimedia Signal Processing (MMSP '98), December 1998, Redonda Beach, Calif, USA 27–31.Google Scholar
  10. Wang Y, Liu Z, Huang J-C: Multimedia content analysis-using both audio and visual clues. IEEE Signal Processing Magazine 2000, 17(6):12–36. 10.1109/79.888862View ArticleGoogle Scholar
  11. Zettl H: Sight Sound Motion: Applied Media Aesthetics. Wadsworth, Belmont, Calif, USA; 1999.Google Scholar
  12. Dorai C, Venkatesh S: Media Computing: Computational Media Aesthetics. Kluwer Academic, Boston, Mass, USA; 2002.View ArticleGoogle Scholar
  13. Moncrieff S, Venkatesh S, Dorai C: Horror film genre typing and scene labeling via audio analysis. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 2: 193–196.Google Scholar
  14. Cai R, Lu L, Zhang H-J, Cai L-H: Highlight sound effects detection in audio stream. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 3: 37–40.Google Scholar
  15. Naphade MR, Kristjansson T, Frey B, Huang TS: Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems. Proc. International Conference on Image Processing (ICIP '98), October 1998, Chicago, Ill, USA 3: 536–540.View ArticleGoogle Scholar
  16. Naphade MR, Huang TS: Extracting semantics from audio-visual content: the final frontier in multimedia retrieval. IEEE Transactions on Neural Networks 2002, 13(4):793–810. 10.1109/TNN.2002.1021881View ArticleGoogle Scholar
  17. Smith JR, Naphade M, Natsev A: Multimedia semantic indexing using model vectors. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 2: 445–448.Google Scholar
  18. Hyvärinen A, Karhunen J, Oja E: Independent Component Analysis. John Wiley & Sons, New York, NY, USA; 2001.View ArticleGoogle Scholar
  19. Rabiner LR: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 1989, 77(2):257–286. 10.1109/5.18626View ArticleGoogle Scholar
  20. Duda RO, Hart PE, Stork DG: Pattern Classification. John Wiley & Sons, New York, NY, USA; 2001.MATHGoogle Scholar
  21. Li SZ: Content-based audio classification and retrieval using the nearest feature line method. IEEE Transactions Speech Audio Processing 2000, 8(5):619–625. 10.1109/89.861383View ArticleGoogle Scholar
  22. Bow S-T: Pattern Recognition and Image Preprocessing. Marcel Dekker, New York, NY, USA; 2002.View ArticleGoogle Scholar
  23. Sound Ideas: Sound Effects Library
  24. Zilca RD: Text-independent speaker verification using covariance modeling. IEEE Signal Processing Letters 2001, 8(4):97–99. 10.1109/97.911465View ArticleGoogle Scholar
  25. Vapnik VN: Statistical Learning Theory. John Wiley & Sons, New York, NY, USA; 1998.MATHGoogle Scholar
  26. Platt JC, Cristianini N, Shawe-Taylor J: Large margin DAGs for multiclass classification. In Advances in Neural Information Processing Systems. Volume 12. MIT Press, Cambridge, Mass, USA; 2000:547–553.Google Scholar
  27. Hsu C-W, Lin C-J: A comparison of methods for multiclass support vector machines. IEEE Transactions Neural Networks 2002, 13(2):415–425. 10.1109/72.991427View ArticleGoogle Scholar
  28. Wang J, Xu C, Chng E, Tian Q: Sports highlight detection from keyword sequences using HMM. Proc. IEEE International Conference on Multimedia and Expo (ICME '04), June 2004, Taipei, Taiwan 1: 599–602.Google Scholar
  29. Naphade MR, Garg A, Huang TS: Audio-visual event detection using duration dependent input output Markov models. Proc. IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '01), December 2001, Kauai, Hawaii, USA 39–43.View ArticleGoogle Scholar
  30. TREC Video Retrieval Evaluation


© Chu et al. 2006