- Research Article
- Open Access
Semantic Context Detection Using Audio Event Fusion
EURASIP Journal on Advances in Signal Processingvolume 2006, Article number: 027390 (2006)
Semantic-level content analysis is a crucial issue in achieving efficient content retrieval and management. We propose a hierarchical approach that models audio events over a time series in order to accomplish semantic context detection. Two levels of modeling, audio event and semantic context modeling, are devised to bridge the gap between physical audio features and semantic concepts. In this work, hidden Markov models (HMMs) are used to model four representative audio events, that is, gunshot, explosion, engine, and car braking, in action movies. At the semantic context level, generative (ergodic hidden Markov model) and discriminative (support vector machine (SVM)) approaches are investigated to fuse the characteristics and correlations among audio events, which provide cues for detecting gunplay and car-chasing scenes. The experimental results demonstrate the effectiveness of the proposed approaches and provide a preliminary framework for information mining by using audio characteristics.
Lienhart RW: Comparison of automatic shot boundary detection algorithms. Storage and Retrieval for Image and Video Databases VII, January 1999, San Jose, Calif, USA, Proceedings of SPIE 3656: 290–301.
Hanjalic A: Shot-boundary detection: unraveled and resolved? IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(2):90–105. 10.1109/76.988656
Chang S-F, Vetro A: Video adaptation: concepts, technologies, and open issues. Proceedings of the IEEE 2005, 93(1):148–158.
Lu L, Zhang H-J, Jiang H: Content analysis for audio classification and segmentation. IEEE Transactions Speech Audio Processing 2002, 10(7):504–516. 10.1109/TSA.2002.804546
Zhang T, Jay Kuo C-C: Hierarchical system for content-based audio classification and retrieval. Multimedia Storage and Archiving Systems III, November 1998, Boston, Mass, USA, Proceedings of SPIE 3527: 398–409.
Tzanetakis G, Cook P: Musical genre classification of audio signals. IEEE Transactions Speech Audio Processing 2002, 10(5):293–302. 10.1109/TSA.2002.800560
Lu L, Zhang H-J: Automated extraction of music snippets. Proc. 11th ACM International Conference on Multimedia, November 2003, Berkeley, Calif, USA 140–147.
Fischer S, Lienhart R, Effelsberg W: Automatic recognition of film genres. Proc. 3rd ACM International Conference on Multimedia, November 1995, San Francisco, Calif, USA 295–304.
Liu Z, Huang J, Wang Y: Classification of TV programs based on audio information using hidden Markov model. Proc. IEEE 2nd Workshop on Multimedia Signal Processing (MMSP '98), December 1998, Redonda Beach, Calif, USA 27–31.
Wang Y, Liu Z, Huang J-C: Multimedia content analysis-using both audio and visual clues. IEEE Signal Processing Magazine 2000, 17(6):12–36. 10.1109/79.888862
Zettl H: Sight Sound Motion: Applied Media Aesthetics. Wadsworth, Belmont, Calif, USA; 1999.
Dorai C, Venkatesh S: Media Computing: Computational Media Aesthetics. Kluwer Academic, Boston, Mass, USA; 2002.
Moncrieff S, Venkatesh S, Dorai C: Horror film genre typing and scene labeling via audio analysis. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 2: 193–196.
Cai R, Lu L, Zhang H-J, Cai L-H: Highlight sound effects detection in audio stream. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 3: 37–40.
Naphade MR, Kristjansson T, Frey B, Huang TS: Probabilistic multimedia objects (multijects): a novel approach to video indexing and retrieval in multimedia systems. Proc. International Conference on Image Processing (ICIP '98), October 1998, Chicago, Ill, USA 3: 536–540.
Naphade MR, Huang TS: Extracting semantics from audio-visual content: the final frontier in multimedia retrieval. IEEE Transactions on Neural Networks 2002, 13(4):793–810. 10.1109/TNN.2002.1021881
Smith JR, Naphade M, Natsev A: Multimedia semantic indexing using model vectors. Proc. IEEE International Conference on Multimedia and Expo (ICME '03), July 2003, Baltimore, Md, USA 2: 445–448.
Hyvärinen A, Karhunen J, Oja E: Independent Component Analysis. John Wiley & Sons, New York, NY, USA; 2001.
Rabiner LR: A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 1989, 77(2):257–286. 10.1109/5.18626
Duda RO, Hart PE, Stork DG: Pattern Classification. John Wiley & Sons, New York, NY, USA; 2001.
Li SZ: Content-based audio classification and retrieval using the nearest feature line method. IEEE Transactions Speech Audio Processing 2000, 8(5):619–625. 10.1109/89.861383
Bow S-T: Pattern Recognition and Image Preprocessing. Marcel Dekker, New York, NY, USA; 2002.
Sound Ideas: Sound Effects Library https://doi.org/www.sound-ideas.com/
Zilca RD: Text-independent speaker verification using covariance modeling. IEEE Signal Processing Letters 2001, 8(4):97–99. 10.1109/97.911465
Vapnik VN: Statistical Learning Theory. John Wiley & Sons, New York, NY, USA; 1998.
Platt JC, Cristianini N, Shawe-Taylor J: Large margin DAGs for multiclass classification. In Advances in Neural Information Processing Systems. Volume 12. MIT Press, Cambridge, Mass, USA; 2000:547–553.
Hsu C-W, Lin C-J: A comparison of methods for multiclass support vector machines. IEEE Transactions Neural Networks 2002, 13(2):415–425. 10.1109/72.991427
Wang J, Xu C, Chng E, Tian Q: Sports highlight detection from keyword sequences using HMM. Proc. IEEE International Conference on Multimedia and Expo (ICME '04), June 2004, Taipei, Taiwan 1: 599–602.
Naphade MR, Garg A, Huang TS: Audio-visual event detection using duration dependent input output Markov models. Proc. IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '01), December 2001, Kauai, Hawaii, USA 39–43.
TREC Video Retrieval Evaluation https://doi.org/www-nlpir.nist.gov/projects/trecvid/