Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Data Driven Representations
Conclusion
References
All Topics
Computer Science
Computational Linguistics

A Survey on Multimodal Video Representation for Semantic Retrieval

Profile image of Janko CalicJanko Calic

2005

https://doi.org/10.1109/EURCON.2005.1629877
visibility

…

description

4 pages

link

1 file

Abstract

This paper surveys the approaches to video representation, focusing on semantic analysis for content-based indexing and retrieval. A problem of adaptive representation of digital multimedia is critically assessed and some novel ideas are presented. Furthermore, the concept of video multimodality is reevaluated and redefined in order to introduce modalities such as editing technique or affect to the audience.

Related papers

Video semantics exploration for indexing and retrieval
eslam mofreh

Al-Azhar Bulletin of Science, 2020

Video semantic concepts exploration is a fundamental problem in a video indexing and retrieval. It has a long history of investigation since early days till recent achievements. The challenges lie in bridging the gap between low level features and semantic level ones. To stand on the thoroughly situation, video semantic concepts exploration for indexing and retrieval purposes evolution from conventional methods to the state-ofthe-art ones will be reviewed. The main contribution is to unify concepts involved and evolution in this interesting topic.

View PDFchevron_right
Multimodal Video Indexing: A Review of the State-of-the-art
Marcel Worring

Multimedia Tools and Applications, 2000

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. In this paper we survey several methods aiming

View PDFchevron_right
Semantic Video Indexing
Marcel Worring

Multimedia Retrieval, 2007

View PDFchevron_right
Multimodal Fuzzy Fusion System for Semantic Video Indexing
Adel M. Alimi

In this paper, we propose a semantic indexing system for reducing the semantic gap between the machine and human interpretations on a video document by generating a finer indexing quality. To do so, data fusion of analyzed interpretation (concepts) from different video modalities (sound and images) is viewed as the required choice.

View PDFchevron_right
Tracking the Progression of Multimedia Semantics: from Text Based Retrieval to Semantic Based Retrieval
Talha Akram

2012

2 Abstract: With the emergence of the low cost multimedia enable devices and storage the progression of the multimedia increases at an incredible velocity. These advancements in the technology have culminated with the immense amount of data. Effective utilization of this colossal data requires a need for the effective and intelligent system and technique to store, annotate, manage, search and retrieve. Semantically analyzing and interpreting the multimedia data is an open challenge. Multimedia analysis starts from the text based system to the content based and finally to the semantic based retrieval systems.

View PDFchevron_right
Multimedia search and retrieval using multimodal annotation propagation and indexing techniques
Michalis Lazaridis

Signal Processing: Image Communication, 2013

In this paper, a complete solution for search and retrieval of rich multimedia content over modern databases is presented. The framework proposed in this paper combines the advantages of multimodal search with those of annotation propagation into a unified system. Moreover, an effective technique, which is appropriate for large-scale indexing, is adopted, extended and integrated to the proposed framework so as to achieve optimized search and retrieval of rich media content even from large-scale databases.

View PDFchevron_right
Semantic Representation of Multimedia Content - Knowledge Representation and Semantic Indexing
Phivos Mylonas

In this paper we present a framework for unified, personalized access to heterogeneous multimedia content in distributed repositories. Focusing on semantic analysis of multimedia documents, metadata, user queries and user profiles, it contributes to the bridging of the gap between the semantic nature of user queries and raw multimedia documents. The proposed approach utilizes as input visual content analysis results, as well as analyzes and exploits associated textual annotation, in order to extract the underlying semantics, construct a semantic index and classify documents to topics, based on a unified knowledge and semantics representation model. It may then accept user queries, and, carrying out semantic interpretation and expansion, retrieve documents from the index and rank them according to user preferences, similarly to text retrieval. All processes are based on a novel semantic processing methodology, employing fuzzy algebra and principles of taxonomic knowledge representation. Part I of this work presented in this paper deals with data and knowledge models, manipulation of multimedia content annotations and semantic indexing, while Part II will continue on the use of the extracted semantic information for personalized retrieval.

View PDFchevron_right
Multimedia Ontology Based Computational Framework for Video Annotation and Retrieval
Marco Bertini

Lecture Notes in Computer Science, 2007

Ontologies are defined as the representation of the semantics of terms and their relationships. Traditionally, they consist of concepts, concept properties, and relationships between concepts, all expressed in linguistic terms. In order to support effectively video annotation and content-based retrieval the traditional linguistic ontologies should be extended to include structural video information and perceptual elements such as visual data descriptors.

View PDFchevron_right
towards multimodal content representation Discussion paper
Harry Bunt

An intelligent multimodal interface requires a number of functionalities concerning media input processing and output rendering, deeper analysis and synthesis drawing at least upon underlying models of media and modalities (language, gesture, facial expression of user or animated agent), fusion and coordination of multimodal input and output at a semantic level, interpretation of multimodal input within the current state of the interaction and the context, and reasoning about and planning of multimodal messages. This implies an architecture with many ...

View PDFchevron_right
An integrated semantic-based approach in concept based video retrieval
Norwati Mustapha, Lilly Affendey

Multimedia Tools and Applications, 2013

Multimedia content has been growing quickly and video retrieval is regarded as one of the most famous issues in multimedia research. In order to retrieve a desirable video, users express their needs in terms of queries. Queries can be on object, motion, texture, color, audio, etc. Low-level representations of video are different from the higher level concepts which a user associates with video. Therefore, query based on semantics is more realistic and tangible for end user. Comprehending the semantics of query has opened a new insight in video retrieval and bridging the semantic gap. However, the problem is that the video needs to be manually annotated in order to support queries expressed in terms of semantic concepts. Annotating semantic concepts which appear in video shots is a challenging and time-consuming task. Moreover, it is not possible to provide annotation for every concept in the real world. In this study, an integrated semantic-based approach for similarity computation is proposed with respect to enhance the retrieval effectiveness in concept-based video retrieval. The proposed method is based on the integration of knowledge-based and corpus-based semantic word similarity measures in order to retrieve video shots for concepts whose annotations are not available for the system. The TRECVID 2005 dataset is used for evaluation purpose, and the results of applying proposed method are then compared against the individual knowledgebased and corpus-based semantic word similarity measures which were utilized in previous studies in the same domain. The superiority of integrated similarity method is shown and evaluated in terms of Mean Average Precision (MAP).

View PDFchevron_right

References (43)

  1. B.D. Adams. Where does computational media aesthetics fit? IEEE Multimedia Magazine, spec. ed. Computational Media Aesthetics, April-June 2003.
  2. Steve Anderson. Select and combine: The rise of database narratives. Res Magazine, 7(1):52-53, Jan/Feb 2004.
  3. Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M. Blei, and Michael I. Jordan. Matching words and pictures. J. Mach. Learn. Res., 3:1107-1135, 2003.
  4. M. Bertini, A. Del Bimbo, and W. Nunziati. Highlights modeling and detection in sports videos. Pattern Analysis and Applications, 2005.
  5. S. Bloehdorn, K. Petridis, C. Saathoff, N. Simou, V. Tzouvaras, Y. Avrithis, S. Handschuh, I. Kompatsiaris, S. Staab, and M. G. Strintzis. Semantic annotation of images and videos for multimedia analysis. In Proceedings of the 2nd European Semantic Web Confer- ence, ESWC 2005, Heraklion, Greece, May 2005.
  6. Tilo Burghardt, Janko Calic, and Barry Thomas. Tracking animals in wildlife videos using face detection. In European Workshop on the Integration of Knowledge, Semantics and Digital Media Technology, October 2004.
  7. Tilo Burghardt, Barry Thomas, Peter J Barham, and Janko Calic. Au- tomated visual recognition of individual african penguins. In Fifth International Penguin Conference, Ushuaia, Tierra del Fuego, Ar- gentina, September 2004.
  8. Janko Calic, Neill Campbell, Majid Mirmehdi, Barry Thomas, Ron Laborde, Sarah Porter, and Nishan Canagarajah. ICBR -multimedia management system for intelligent content based retrieval. In Inter- national Conference on Image and Video Retrieval CIVR 2004, pages 601-609. Springer LNCS 3115, July 2004.
  9. Janko Calic and Barry Thomas. Spatial analysis in key-frame extrac- tion using video segmentation. In Workshop on Image Analysis for Multimedia Interactive Services, April 2004.
  10. Gert Cauwenberghs and Tomaso Poggio. Incremental and decremen- tal support vector machine learning. In Proc. of Neural Information Processing Systems (NIPS) 2000, Denver, USA.
  11. Glorianna Davenport, Thomas Aguirre Smith, and Natalio Pincever. Cinematic primitives for multimedia. IEEE Comput. Graph. Appl., 11(4):67-74, 1991.
  12. Marc Davis. Media streams: representing video for retrieval and repurposing. In MULTIMEDIA '94: Proceedings of the second ACM international conference on Multimedia, pages 478-479, New York, NY, USA, 1994. ACM Press.
  13. Marc Davis. Media streams: representing video for retrieval and repurposing. PhD thesis, Cambridge, MA, USA, 1995.
  14. Nevenka Dimitrova. Context and memory in multimedia content analysis. IEEE MultiMedia, 11(3):7-11, 2004.
  15. A. Dorado, J. Calic, and E. Izquierdo. A rule-based video annotation system. Circuits and Systems for Video Technology, IEEE Transac- tions on, 14(5):622-633, May 2004.
  16. Chitra Dorai and Svetha Venkatesh. Media computing: computa- tional media aesthetics. The Kluwer international series in video computing. Kluwer Academic Publishers, Boston; London, 2002.
  17. Venkatesh S. Dorai, C. Bridging the semantic gap with computational media aesthetics. Multimedia, IEEE, 10:15-17, 2003.
  18. Sergei Eisenstein and Jay Leyda. Film Form Essays in film theory. Dennis Dobson Ltd, [S.l.], 1949. edited and translated by Jay Leyda.
  19. Louis D. Giannetti. Understanding movies. Prentice Hall; London: Prentice-Hall International (UK), Upper Saddle River, N.J., 9th ed. edition, 2002.
  20. A. Hanjalic and L. Q. Xu. Affective video content representation and modeling. IEEE Transactions on Multimedia, 7(1):143-154, 2005.
  21. Sion Hannuna, Neill Campbell, and David Gibson. Segmenting quadruped gait patterns from wildlife video. In The IEE Interna- tional Conference on Visual Information Engineering: Convergence in Graphics and Vision., pages 235-243. Institution of Electrical En- gineers, April 2005.
  22. I. Kompatsiaris, Y. Avrithis, P. Hobson, T. May, and J. Tromp. Achieving Integration of Knowledge and Content Technologies: The AceMedia Project. In Proc. European Workshop on the Integration of Knowledge, Semantics and Digital Media Technology, Royal Sta- tistical Society, London, UK, Nov. 2004.
  23. Lev Kuleshov and Ronald Levaco. Kuleshov on film: writings by Lev Kuleshov. University of California Press, Berkeley; London, 1974.
  24. R. Leonardi and P. Migliorati. Semantic indexing of multimedia doc- uments. Multimedia, IEEE, 9(2):44, 2002.
  25. R. Leonardi, P. Migliorati, and M. Prandini. Semantic indexing of soccer audio-visual sequences: a multimodal approach based on con- trolled Markov chains. Circuits and Systems for Video Technology, IEEE Transactions on, 14(5):634, 2004.
  26. Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K. Sethi. Multimedia content processing through cross-modal association. In MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia, pages 604-611, New York, NY, USA, 2003. ACM Press.
  27. Lev Manovich. The language of new media. Leonardo. MIT Press, Cambridge, Mass.; London, 2001.
  28. Christian Metz. [Essais sur la signification au cinéma.] Film lan- guage. A semiotics of the cinema. Translated by Michael Taylor. New York: Oxford University Press, 1974.
  29. F. Nack and A. Parkes. Toward the automated editing of theme- oriented video sequences. Applied Artificial Intelligence, 11(4):331- 366, 1997.
  30. Milind R. Naphade. On supervision and statistical learning for se- mantic multimedia analysis. Journal of Visual Communication and Image Representation, 15(3):348-369, 2004.
  31. Laurence Nigay and Joëlle Coutaz. A design space for multimodal systems: concurrent processing and data fusion. In CHI '93: Pro- ceedings of the SIGCHI conference on Human factors in computing systems, pages 172-178, New York, NY, USA, 1993. ACM Press.
  32. M. Ramesh Naphade, I. V. Kozintsev, and T. S. Huang. Factor graph framework for semantic video indexing. Circuits and Systems for Video Technology, IEEE Transactions on, 12(1):40, 2002.
  33. J.J. Rocchio, Jr. The SMART Retrieval System: Experiments in Au- tomatic Document Processing, chapter Relevance Feedback in Infor- mation Retrieval, pages 313-323. Prentice-Hall, 1971.
  34. S. Santini, A. Gupta, and R. Jain. Emergent semantics through inter- action in image databases. Knowledge and Data Engineering, IEEE Transactions on, 13(3):337, 2001.
  35. C. Saraceno and R. Leonardi. Indexing audiovisual databases through joint audio and video processing. IJIST, 9(5):320-331, 1999.
  36. Ferdinand de Saussure. Course in general linguistics. Duckworth, London, 1983.
  37. Arnold W. M. Smeulders, Marcel Worring, Simone Santini, Amar- nath Gupta, and Ramesh Jain. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell., 22(12):1349-1380, 2000.
  38. C.G.M. Snoek and M. Worring. Multimodal video indexing: A review of the state-of-the-art. Multimedia Tools and Applications, 25(1):5-35, 2005.
  39. S. Staab. Emergent semantics. IEEE Intelligent Systems, 17(1):78- 81, 2002.
  40. M. E. J. Wood, N. W. Campbell, and B. T. Thomas. Iterative refine- ment by relevance feedback in content-based digi tal image retrieval. In ACM Multimedia 98, pages 13-20. ACM, September 1998.
  41. Wang Yao, Liu Zhu, and Huang Jin-Cheng. Multimedia content analysis-using both audio and visual clues. Signal Processing Mag- azine, IEEE, 17(6):12, 2000.
  42. A. Yoshitaka and T. Ichikawa. A survey on content-based retrieval for multimedia databases. IEEE Transactions on Knowledge and Data Engineering, 11(1):81-93, Jan/Feb 1999.
  43. Xiang Sean Zhou and Thomas S. Huang. Relevance feedback in im- age retrieval: A comprehensive review. Multimedia Syst., 8(6):536- 544, 2003.

Related papers

AN OVERVIEW OF MULTIMODAL VIDEO REPRESENTATION FOR SEMANTIC ANALYSIS
Janko Calic

This paper gives an overview of approaches to video representation targeting semantic analysis for content-based indexing and retrieval. It highlights the major achieve-

View PDFchevron_right
Growing Trend from Uni to Multimodal Video Indexing
Jonathan Loo

International Journal of Digital Content: Technology and its Applications, 2009

The ultimate challenge of the semantic multimedia database research is to provide a system that can retrieve the most relevant and semantically accurate multimedia data (image, audio, text) according to the user requirement. In order to cope with this challenge there is a need for a system that can efficiently and effectively index the multimedia data. This paper provides an overview of multimodal video indexing. It cover different aspects and issues in video indexing and the approaches that are used. The ultimate main challenge is to mind the semantic gap. The structure of this paper is as follow. The following section outlines the introduction. The section-II brings multimodal analysis of videos. Section-III is about semantic indexes. Section-IV covers content based video indexing. Section-V is about different indexing techniques that are used. The paper ends with the major research issues and problem and last section conclusion about this paper.

View PDFchevron_right
A review on multimodal video indexing
Marcel Worring

Proceedings. IEEE International Conference on Multimedia and Expo, 2002

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. Efficient, single modality based, video indexing methods have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. In this paper we present a framework for multimodal video indexing, which views a video document from the perspective of its author. The framework serves as a blueprint for a generic and flexible multimodal video indexing system, and generalizes different state-of-the-art video indexing methods. It furthermore forms the basis for categorizing these different methods.

View PDFchevron_right
Multimodal Video Indexing: A Review of
Marcel Worring

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. Efficient, single modality based, video indexing methods have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. In this paper we present a framework for multimodal video indexing, which views a video document from the perspective of its author. The framework serves as a blueprint for a generic and flexible multimodal video indexing system, and generalizes different state-of-the-art video indexing methods. It furthermore forms the basis for categorizing these different methods.

View PDFchevron_right
Extracting and Integrating Multimodality Features via Multidimensional Approach for Video Retrieval
Fatimah Khalid

IJCSNS, 2009

This work discusses the application of an Artificial Intelligence technique called data extraction and a process-based ontology in constructing experimental qualitative models for video retrieval and detection. We present a framework architecture that uses multimodality features as the knowledge representation scheme to model the behaviors of a number of human actions in the video scenes. The main focus of this paper placed on the design of two main components (model classifier and inference engine) for a tool abbreviated as VSAD (Video Action Scene Detector) for retrieving and detecting human actions from video scenes. The discussion starts by presenting the workflow of the retrieving and detection process and the automated model classifier construction logic. We then move on to demonstrate how the constructed classifiers can be used with multimodality features for detecting human actions. Finally, behavioral explanation manifestation is discussed. The simulator is implemented in bilingual; Math Lab and C++ are at the backend supplying data and theories while Java handles all front-end GUI and action pattern updating.

View PDFchevron_right

Related topics

  • Semantic gap
  • Semantic retrieval
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved