Academia.eduAcademia.edu

Outline

Title
Abstract
Content-Based Video Retrieval Conclusions
References
All Topics
Computer Science
Computer Vision

Content-based video retrieval: Three example systems from TRECVid

Profile image of Marcel WorringMarcel Worring

2008, International Journal of Imaging Systems and Technology

https://doi.org/10.1002/IMA.20150
visibility

…

description

7 pages

link

1 file

Abstract

The growth in available online video material over the Internet is generally combined with user-assigned tags or content description, which is the mechanism by which we then access such video. However, user-assigned tags have limitations for retrieval and often we want access where the content of the video itself is directly matched against a user's query rather than against some manually assigned surrogate tag. Content-based video retrieval techniques are not yet scalable enough to allow interactive searching on Internet-scale, but the techniques are proving robust and effective for smaller collections. In this article, we show three exemplar systems which demonstrate the state of the art in interactive, content-based retrieval of video shots, and these three are just three of the more than 20 systems developed for the 2007 iteration of the annual TRECVid benchmarking activity. The contribution of our article is to show that retrieving from video using content-based methods is now viable, that it works, and that there are many systems which now do this, such as the three outlined herein. These systems, and others can provide effective search on hundreds of hours of video content and are samples of the kind of contentbased search functionality we can expect to see on larger video archives when issues of scale are addressed.

Related papers

Content Based Video Retrieval
IRJET Journal

Searching for digital information on web, especially images, music, and video, is quickly gaining importance for the business and entertainment industry. Content-based video retrieval (CBVR) is a prominent research interest. The ability of a computer to automatically recognize objects in videos is so low that the existing technique for extracting semantic features from all kinds of videos is incapable of retrieving videos based on semantic feature. Here a new approach is proposed for video clip retrieval based on Earth Mover's Distance (EMD). Instead of imposing one-to-one matching constraint it allows many-to-many matching methodology and is capable of tolerating errors due to video partitioning and various video editing effects. We formulate clip-based retrieval as a graph matching problem in two stages. In the first stage, to allow the matching between a query and a long video, an online clip segmentation algorithm is employed to rapidly locate candidate clips for similarity measure. In the second stage, a weighted graph is constructed to model the similarity between two clips. EMD is proposed to compute the minimum cost of the weighted graph as the similarity between two clips.

View PDFchevron_right
A Synergistic Approach to Efficient Interactive Video Retrieval
John Adcock

Lecture Notes in Computer Science, 2005

A video database can contain a large number of videos ranging from several minutes to several hours in length. Typically, it is not sufficient to search just for relevant videos, because the task still remains to find the relevant clip, typically less than one minute of length, within the video. This makes it important to direct the users attention to the most promising material and to indicate what material they already investigated. Based on this premise, we created a video search system with a powerful and flexible user interface that incorporates dynamic visualizations of the underlying multimedia objects. The system employes an automatic story segmentation, combines text and visual search, and displays search results in ranked sets of story keyframe collages. By adapting the keyframe collages based on query relevance and indicating which portions of the video have already been explored, we enable users to quickly find relevant sections. We tested our system as part of the NIST TRECVID interactive search evaluation, and found that our user interface enabled users to find more relevant results within the allotted time than other systems employing more sophisticated analysis techniques but less helpful user interfaces.

View PDFchevron_right
CONTENT BASED VIDEO QUERYING
arief Setyanto

JURNA DASI, 2010

View PDFchevron_right
VISIONE: A Large-Scale Video Retrieval System with Advanced Search Functionalities
Giuseppe Amato

2023

VISIONE is a large-scale video retrieval system that integrates multiple search functionalities, including free text search, spatial color and object search, visual and semantic similarity search, and temporal search. The system leverages cutting-edge AI technology for visual analysis and advanced indexing techniques to ensure scalability. As demonstrated by its runner-up position in the 2023 Video Browser Showdown competition, VISIONE effectively integrates these capabilities to provide a comprehensive video retrieval solution. A system demo is available online, showcasing its capabilities on over 2300 hours of diverse video content (V3C1+V3C2 dataset) and 12 hours of highly redundant content (Marine dataset). The demo can be accessed at https://visione.isti.cnr.it/. CCS CONCEPTS • Information systems → Information retrieval; Users and interactive retrieval; Retrieval models and ranking; Search engine architectures and scalability; Multimedia and multimodal retrieval; Video search.

View PDFchevron_right
Automated Content Based Video Retrieval
Stefan Rueger

2009

Abstract. We describe our experiments for the search task. Eight runs were submitted, all of them corresponding to the fully automated mode, without human interaction in the loop. The system was based on determining the distance from a query image to a pre-indexed ...

View PDFchevron_right
A Survey of Content-Based Video Retrieval
Vasumathi Narayanan

Journal of Computer Science, 2008

This study surveys current trends/methods in video retrieval. The major themes covered by the study include shot segmentation, key frame extraction, feature extraction, clustering, indexing and video retrieval-by similarity, probabilistic, transformational, refinement and relevance feedback. This work has done in an aim to assist the upcoming researchers in the field of video retrieval, to know about the techniques and methods available for video retrieval.

View PDFchevron_right
iARM - an interactive video retrieval system
Paisarn Muneesawang

2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763), 2004

This paper presents the iARM sysfem for contenfbased video retrieval in the interactive framework This system explores a new model-based video indexing technique fo improve effectiveness of relevance feedback to make interactive video retrieval a user-friendly environmenf. The system emphasizes the accuracy in modeling spafio-temporal information in a video clip, so that relevance feedhack analysis needs only a few cycles and a few fraining samples, which greatly reduce the search time for video transmissions over the network We investigate the resilience of the system, and apply the interactive content-based retrieval method to an automafically indexed database of 20 hours of video.

View PDFchevron_right
An Image Retrieval System for Video
Lucia Vadicamo, Fabrizio Falchi

Similarity Search and Applications 12th International Conference, SISAP 2019, 2019

Since the 1970's the Content-Based Image Indexing and Retrieval (CBIR) has been an active area. Nowadays, the rapid increase of video data has paved the way to the advancement of the technologies in many different communities for the creation of Content-Based Video Indexing and Retrieval (CBVIR). However, greater attention needs to be devoted to the development of effective tools for video search and browse. In this paper, we present Visione, a system for large-scale video retrieval. The system integrates several content-based analysis and retrieval modules , including a keywords search, a spatial object-based search, and a visual similarity search. From the tests carried out by users when they needed to find as many correct examples as possible, the similarity search proved to be the most promising option. Our implementation is based on state-of-the-art deep learning approaches for content analysis and leverages highly efficient indexing techniques to ensure scalability. Specifically, we encode all the visual and textual descriptors extracted from the videos into (surrogate) textual representations that are then efficiently indexed and searched using an off-the-shelf text search engine using similarity functions.

View PDFchevron_right
A Review Paper on Content based Video Retrieval
IJSRD Journal

Nowadays visual information is gaining much more importance than textual information as pictures convey more than words. As image has richer content than text, Video database is increasing exponentially over the Internet. Content-based Video retrieval (CBVR) from large video storage has become a challenging and important issue for researchers over worldwide. There is lack of tools so it is difficult to distinguish between low and high level feature. There has been lot of CBVR systems developed but still there is a lack of efficient retrieval system, as many retrieval techniques are not suitable for extensive search and so it deteriorates the overall performance of system. In this paper we have given an outline of the sequential steps of existing CBVR systems.

View PDFchevron_right
VERGE: A video interactive retrieval engine
Stefanos Vrochidis

2010

This paper presents the video retrieval engine VERGE, which combines indexing, analysis and retrieval techniques in various modalities (i.e. textual, visual and concept search). The functionalities of the search engine are demonstrated through the supported user interaction modes.

View PDFchevron_right

References (13)

  1. on Image and Video Retrieval (CIVR'07), Amsterdam, The Netherlands, 2007.
  2. P. Joly, J. Benois-Pineau, E. Kijak, and G. Que ´not, The argos campaign: Evaluation of video analysis tools. In Proceedings of the International Work- shop on Content-Based Multimedia Indexing, 2007. CBMI'07, 2007, pp. 130-137.
  3. N. Lazarevic-McManus, J. Renno, and G.A. Jones, Performance evaluation in visual surveillance using the F-measure. In Proceedings of the 4th ACM international Workshop on Video Surveillance and Sensor Networks (Santa Barbara, California, USA, October 27, 2006). VSSN'06, 2006, pp. 45-52.
  4. P. Over, G. Awad, W. Kraaij, and A.F. Smeaton, TRECVID 2007-An Introduction, In Proceedings of TRECVID 2007, Gaithersburg, MD, No- vember 2007.
  5. G.M. Que ´not. Active learning for multimedia. In Proceedings of the 15th international Conference on Multimedia (Augsburg, Germany, September 25-29, 2007). MULTIMEDIA'07. ACM Press, 2007.
  6. A.F. Smeaton, P. Over, and W. Kraaij, Evaluation campaigns and TRECVid, In Proceedings of the 8th ACM International Workshop on Multimedia In- formation Retrieval (Santa Barbara, California, USA, October 26-27, 2006). MIR'06. ACM Press, 2006, pp. 321-330.
  7. C.G.M. Snoek, I. Everts, J.C. van Gemert, J.M. Geusebroek, B. Huurnink, D.C. Koelma, M. van Liempt, O. de Rooij, K.E.A. van de Sande, A.W.M. Smeulders, J.R.R. Uijlings, and M. Worring, The MediaMill TRECVID 2007 Semantic Video Search Engine. In Proceedings of TRECVID 2007, Gaithersburg, MD, November 2007a.
  8. C.G.M. Snoek, J.C. van Gemert, Th. Gevers, B. Huurnink, D.C. Koelma, M. Van Liempt, O. De Rooij, K.E.A. van de Sande, F.J. Seinstra, A.W.M. Smeulders, A.H.C. Thean, C.J. Veenman, and M. Worring. The MediaMill TRECVID 2006 Semantic Video Search Engine. In Proceedings of TREC- VID 2006, Gaithersburg, MD, November 2006.
  9. C.G.M. Snoek, M. Worring, D.C. Koelma, and A.W.M. Smeulders, A learned lexicon-driven paradigm for interactive video retrieval, IEEE Trans Multimedia, 9(2007b), 280-292.
  10. D. Tao, X. Tang, and X. Li, Which components are important for interactive image searching? IEEE Trans Circuits Systems Video Technol 18(2008), pp. 3-11.
  11. P. Wilkins, P. Ferguson, and A.F. Smeaton, Using score distributions for quer- ytime fusion in multimedia retrieval, In Proceedings of the 8th ACM Interna- tional Workshop on Multimedia Information Retrieval (Santa Barbara, Cali- fornia, USA, October 26-27, 2006). MIR'06. ACM Press, pp. 51-60.
  12. P. Wilkins, T. Adamek, P. Ferguson, M. Hughes, G.J.F. Jones, G. Keenan, K. McGuinness, N.E. O'Connor, D. Sadlier, and A.F. Smeaton, K-Space at TRECVid 2007, In Proceedings of TRECVID 2007, Gaithersburg, MD, November 2007.
  13. X. Zhou and T.S. Huang, Relevance feedback in content-based image retrieval: some recent advances, Inform Sci Appl 148(2002), 129-137.

Related papers

[Invited Paper] Strategies for Searching Video Content with Text Queries or Video Examples
Chuang Gan

ITE Transactions on Media Technology and Applications, 2016

Shoou-I Yu* (student member) †1 , Yi Yang (member) †2 , Zhongwen Xu (student member) †2 , Shicheng Xu (student member) †1 , Deyu Meng (member) †3 , Zexi Mao (member) †1 , Zhigang Ma (member) †1 , Ming Lin (member) †1 , Xuanchong Li (student member) †1 , Huan Li (member) †1 , Zhenzhong Lan (student member) †1 , Lu Jiang (student member) †1 , Alexander G. Hauptmann (member) †1 , Chuang Gan (student member) †4 , Xingzhong Du (student member) †5 , Xiaojun Chang (student member) †2

View PDFchevron_right
TRECVID: the utility of a content-based video retrieval evaluation
Alexander Hauptmann

SPIE Proceedings, 2006

TRECVID, an annual retrieval evaluation benchmark organized by NIST, encourages research in information retrieval from digital video. TRECVID benchmarking covers both interactive and manual searching by end users, as well as the benchmarking of some supporting technologies including shot boundary detection, extraction of semantic features, and the automatic segmentation of TV news broadcasts. Evaluations done in the context of the TRECVID benchmarks show that generally, speech transcripts and annotations provide the single most important clue for successful retrieval. However, automatically finding the individual images is still a tremendous and unsolved challenge. The evaluations repeatedly found that none of the multimedia analysis and retrieval techniques provide a significant benefit over retrieval using only textual information such as from automatic speech recognition transcripts or closed captions. In interactive systems, we do find significant differences among the top systems, indicating that interfaces can make a huge difference for effective video/image search. For interactive tasks efficient interfaces require few key clicks, but display large numbers of images for visual inspection by the user. The text search finds the right context region in the video in general, but to select specific relevant images we need good interfaces to easily browse the storyboard pictures. In general, TRECVID has motivated the video retrieval community to be honest about what we don't know how to do well (sometimes through painful failures), and has focused us to work on the actual task of video retrieval, as opposed to flashy demos based on technological capabilities.

View PDFchevron_right
The TREC2001 Video Track: Information Retrieval on Digital Video Information
Alexander Hauptmann

Lecture Notes in Computer Science, 2002

The development of techniques to support content-based access to archives of digital video information has recently started to receive much attention from the research community. During 2001, the annual TREC activity, which has been benchmarking the performance of information retrieval techniques on a range of media for 10 years, included a "track" or activity which allowed investigation into approaches to support searching through a video library. This paper is not intended to provide a comprehensive picture of the different approaches taken by the TREC2001 video track participants but instead we give an overview of the TREC video search task and a thumbnail sketch of the approaches taken by different groups. The reason for writing this paper is to highlight the message from the TREC video track that there are now a variety of approaches available for searching and browsing through digital video archives, that these approaches do work, are scalable to larger archives and can yield useful retrieval performance for users. This has important implications in making digital libraries of video information attainable.

View PDFchevron_right
An interactive video content-based retrieval system
Frédéric Precioso

2008

The actual generation of video search engines offers low-level abstractions of the data while users seek for high-level semantics. The main challenge in video retrieval remains bridging the semantic gap. Thus, the effectiveness of video retrieval is based on the result of the interaction between query selection and a goal-oriented human user. The system exploits the human capability for rapidly scanning imagery augmenting it with an active learning loop, which tries to always present the most relevant material based on the current information. We describe in this paper, a machine learning system for interactive video retrieval. The core of this system is a kernel-based SVM classifier. The video retrieval uses the core as an active learning classifier. We perform an experiment against the 2005 NIST TRECVID benchmark in the high-level task.

View PDFchevron_right
The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval
Fabrizio Falchi

Journal of Imaging

This paper describes in detail VISIONE, a video search system that allows users to search for videos using textual keywords, the occurrence of objects and their spatial relationships, the occurrence of colors and their spatial relationships, and image similarity. These modalities can be combined together to express complex queries and meet users’ needs. The peculiarity of our approach is that we encode all information extracted from the keyframes, such as visual deep features, tags, color and object locations, using a convenient textual encoding that is indexed in a single text retrieval engine. This offers great flexibility when results corresponding to various parts of the query (visual, text and locations) need to be merged. In addition, we report an extensive analysis of the retrieval performance of the system, using the query logs generated during the Video Browser Showdown (VBS) 2019 competition. This allowed us to fine-tune the system by choosing the optimal parameters and st...

View PDFchevron_right

Related topics

  • Content-Based Video Retrieval
  • Environmental Pollution
  • Video Retrieval
  • Content based Retrieval

    • Cited by

    A crowdsourcing framework for the production and use of film and television data
    Carlos Eugenio Ovalle

    New Review of Hypermedia and Multimedia, 2011

    This paper outlines a framework that would enable the detailed indexing of film and television media through crowdsourcing. By making it easier to generate detailed data about these media on a large scale, fans and scholars can more efficiently produce a wide range of artifacts that reflect their interests in this content. Our development of a test collection included detailed indexing of 12 feature films and 8 television programs. We describe the conditions that make crowdsourcing an ideal approach for accomplishing this work on a larger scale; present a three-level development framework; and discuss how automated indexing, crowdsourcing quality, and copyright concerns might influence continued development of the project. Our framework highlights the potential of both multimedia indexing and crowdsourcing and can serve as a model for others embarking on projects that involve indexing, annotating, or labeling large multimedia collections.

    View PDFchevron_right
    A Content Based Video Retrieval Analysis System with Extensive Features by Using Kullback-Leibler
    Dr. T.N. Shanmugam

    International Journal of Computational Intelligence Systems, 2014

    Content-based video retrieval systems have shown great potential in supporting decision making in clinical activities, teaching, and biological research. In content-based video retrieval, feature combination plays a key role. As a result content-based retrieval of all different type video data turns out to be a challenging and vigorous problem. This paper presents an effective content based video retrieval system, which recognizes and retrieves videos with three different types of visual effects. The raw video information is divided into shots and also the object feature, movement feature and also the occlusion options are extracted from these shots and also the feature library is used for the storage method of those options. Advanced on, the Kullback-Leibler distance is computed among the options of the feature library and also the options of the question clip that's extracted within the similar manner. The results show that it is possible to improve a system for content-based video retrieval by using Kullback-Leibler distance model, which takes careful consideration of the structure and distribution of visual features. Hence the final results with the aid of the Kullback-Leibler distance the similar videos are extracted from the collection of videos based on the given query video clip in an effective manner.

    View PDFchevron_right
    Audio Interval Retrieval Using Convolutional Neural Networks
    Ievgeniia Kuzminykh

    Lecture Notes in Computer Science, 2020

    Modern streaming services are increasingly labeling videos based on their visual or audio content. This typically augments the use of technologies such as AI and ML by allowing to use natural speech for searching by keywords and video descriptions. Prior research has successfully provided a number of solutions for speech to text, in the case of a human speech, but this article aims to investigate possible solutions to retrieve sound events based on a natural language query, and estimate how effective and accurate they are. In this study, we specifically focus on the YamNet, AlexNet, and ResNet-50 pre-trained models to automatically classify audio samples using their respective melspectrograms into a number of predefined classes. The predefined classes can represent sounds associated with actions within a video fragment. Two tests are conducted to evaluate the performance of the models on two separate problems: audio classification and intervals retrieval based on a natural language query. Results show that the benchmarked models are comparable in terms of performance, with YamNet slightly outperforming the other two models. YamNet was able to classify single fixed-size audio samples with 92.7% accuracy and 68.75% precision while its average accuracy on intervals retrieval was 71.62% and precision was 41.95%. The investigated method may be embedded into an automated event marking architecture for streaming services.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved