ECL-LIRIS at TrecVid 2011: Semantic Indexing

This year EURECOM participated in the TRECVID 2013 Semantic INdexing (SIN) Task [11] for the submission of four different runs for 60 concepts. Our submission builds on the runs submitted last year at the 2012 SIN task, the details of which can be found in [8]. In 2013, two runs are combinations of basic descriptors. One run adds uploaders bias to the pool of visual features while another run was prepared in collaboration with Aalto University. All runs are trained on annotations provided by the IRIM collaborative effort [4]. When compared with last year system, our runs use a larger set of visual features, and larger visual dictionaries to provide a finer representation of the visual/clustering space and increase the precision of the retrieval system. Like in last year's submission we add a global descriptor to visual features capturing salient details or gist of a keyframe. We further benefit from metadata information by including an uploader bias to increase the scores of videos uploaded by same users. A major difference this year is that we have used a new training algorithm based on a combination of PEGASOS [14] mixed with Homogeneous Kernel Maps [16], while our previous system was based on the libsvm library [5].

2011

2010

The IRIM group is a consortium of French teams working on Multimedia Indexing and Retrieval. This paper describes our participation to the TRECVID 2010 semantic indexing and instance search tasks. For the semantic indexing task, we evaluated a number of different descriptors and tried different fusion strategies, in particular hierarchical fusion. The best IRIM run has a Mean Inferred Average Precision of 0.0442, which is above the task median performance. We found that fusion of the classification scores from different classifier types improves the performance and that even with a quite low individual performance, audio descriptors can help. For the instance search task, we used only one of the example images in our queries. The rank is nearly in the middle of the list of participants. The experiment showed that HSV features outperform the concatenation of HSV and Edge histograms or the Wavelet features.

2015

This year EURECOM participated in the TRECVID 2015 Semantic INdexing (SIN) Task [24] for the submission of four different runs for 60 concepts, and Video Hyperlinking (LNK) Task [24] with the four submissions. Our submission to the SIN Task builds on the runs submitted in the previous years for the 2013 and 2014 SIN tasks, the details of which can be found in [20] and [19], while the LNK submissions are based on our previous experiments as in [28] and [9]. The major changes for 2015 are the use of new Deep Network models to produce extra descriptors for the video shots, and the introduction of various fusion schemes at all levels of the processing, to reduce the problem of overfitting. This year, we did not use our uploader model, partly because of lack of time, and partly because initial experiments showed only marginal improvement after the new features were added. For the LNK Task our approach targeted to connect the textual stream of the videos within the collection and its vocabulary context, as defined by word2vec algorithm, with the output of visual concepts detection tools for the corresponding hyperlinks candidates within one framework. We combined visual concepts detection confidence scores with the information about corresponding word vectors distances in order to rerank the baseline text based search. The reranked runs did not outperform the baseline, however they exposed potential of our method for further improvement. Beside this participation, EURECOM took part in the collaborative IRIM submission, the details of this contribution is included in the corresponding publication from the IRIM group. The remainder of this paper briefly describes the descriptors that we have been using, the training and the various fusion schemes, and the content of the submitted runs; and the framework of the confidence scores combinations used for reranking in the LNK task.

J Amer Coll Cardiol, 2011

In this paper we describe our TRECVID 2010 video retrieval experiments. The MediaMill team participated in three tasks: semantic indexing, known-item search, and instance search. The starting point for the MediaMill concept detection approach is our top-performing bag-of-words system of TRECVID 2009, which uses multiple color SIFT descriptors, sparse codebooks with spatial pyramids, kernelbased machine learning, and multi-frame video processing. We improve upon this baseline system by further speeding up its execution times for both training and classification using GPU-optimized algorithms, approximated histogram intersection kernels, and several multi-frame combination methods. Being more efficient allowed us to supplement the Internet video training collection with positively labeled examples from international news broadcasts and Dutch documentary video from the TRECVID 2005-2009 benchmarks. Our experimental setup covered a huge training set of 170 thousand keyframes and a test set of 600 thousand keyframes in total. Ultimately leading to 130 robust concept detectors for video retrieval. For retrieval, a robust but limited set of concept detectors justifies the need to rely on as many auxiliary information channels as possible. For automatic known item search we therefore explore how we can learn to rank various information channels simultaneously to maximize video search results for a given topic. To further improve the video retrieval results, our interactive known item search experiments investigate how to combine metadata search and visualization into a single interface. The 2010 edition of the TRECVID benchmark has again been a fruitful participation for the MediaMill team, resulting in the top ranking for concept detection in the semantic indexing task. Feature Extraction Feature Extraction Word Projection Word Projection Machine Learning Machine Learning

2015

The IRIM group is a consortium of French teams supported by the GDR ISIS and working on Multimedia Indexing and Retrieval. This paper describes its participation to the TRECVID 2015 semantic indexing (SIN). Our approach uses a six-stages processing pipelines for computing scores for the likelihood of a video shot to contain a target concept. These scores are then used for producing a ranked list of images or shots that are the most likely to contain the target concept. The pipeline is composed of the following steps: descriptor extraction, descriptor optimization, classification, fusion of descriptor variants, higher-level fusion, and re-ranking. We evaluated a number of different descriptors and tried different fusion strategies. The best IRIM run has a Mean Inferred Average Precision of 0.2947, which ranked it 4th out of 15 participants.

2010

In this paper we describe our TRECVID 2009 video retrieval experiments. The MediaMill team participated in three tasks: concept detection, automatic search, and interactive search. The starting point for the MediaMill concept detection approach is our top-performing bag-of-words system of last year, which uses multiple color descriptors, codebooks with soft-assignment, and kernel-based supervised learning. We improve upon this baseline system by exploring two novel research directions. Firstly, we study a multimodal extension by including 20 audio concepts and fusion using two novel multi-kernel supervised learning methods. Secondly, with the help of recently proposed algorithmic refinements of bag-of-word representations, a GPU implementation, and compute clusters, we scale-up the amount of visual information analyzed by an order of magnitude, to a total of 1,000,000 i-frames. Our experiments evaluate the merit of these new components, ultimately leading to 64 robust concept detectors for video retrieval. For retrieval, a robust but limited set of concept detectors justifies the need to rely on as many auxiliary information channels as possible. For automatic search we therefore explore how we can learn to rank various information channels simultaneously to maximize video search results for a given topic. To further improve the video retrieval results, our interactive search experiments investigate the roles of visualizing preview results for a certain browse-dimension and relevance feedback mechanisms that learn to solve complex search topics by analysis from user browsing behavior. The 2009 edition of the TRECVID benchmark has again been a fruitful participation for the MediaMill team, resulting in the top ranking for both concept detection and interactive search. Again a lot has been learned during this year's TRECVID campaign; we highlight the most important lessons at the end of this paper.

2012

This paper describes our method developed for TRECVID 2012 Semantic INdexing (SIN) Task. Our main research purpose is the development of a fast method, which can work on a single processor with no performance degra- dation. To this end, computationally expensive processes are re-formulated based on matrix operation. We re-formulate the Euclidian distance computation for the kernel value computa- tion in an SVM, and the probability density computation of multivariate normal distributions for the GMM supervector representation. This enables accurate concept detection using a large number of training examples, and spatially-temporally dense features. The following four runs were submitted to SIN (light) task: L A kobe muro l5 4: This is our baseline run using five

This paper provides an overview of the tasks submitted to TRECVID 2012 by ITI-CERTH. ITI-CERTH participated in the Known-item search (KIS), in the Semantic Indexing (SIN), as well as in the Event Detection in Internet Multimedia (MED) and the Multimedia Event Recounting (MER) tasks. In the SIN task, techniques are developed, which combine video representations that express motion semantics with existing well-performing descriptors such as SIFT and Bag-of-Words for shot representation. In the MED task, two methods are evaluated, one that is based on Gaussian mixture models (GMM) and audio features, and a "semantic model vector approach" that combines a pool of subclass kernel support vector machines (KSVMs) in an ECOC framework for event detection exploiting visual information only. Furthermore, we investigate fusion strategies of the two systems in an intermediate semantic level or in score level (late fusion). In the MER task, a "model vector approach" is used to describe the semantic content of the videos, similar to the MED task, and a novel feature selection method is utilized to select the most discriminant concepts regarding the target event. Finally, the KIS search task is performed by employing VERGE, which is an interactive retrieval application combining retrieval functionalities in various modalities. Representation Classifiers 12 local-image-feature-based Classifiers (3 descriptors (SIFT, Opponent-SIFT, RGB-SIFT) x Key-frame 2 sampling strategies (Dense, Harris-Laplace) x 2 BoW strategies (soft-, hard-assignment)) 1 global-image-feature-based classifier (color histograms) Tomographs 12 tomograph-based Classifiers (2 types of video tomographs (horizontal, vertical) x 3 descriptors (SIFT, Opponent-SIFT, RGB-SIFT) x 2 BoW strategies (soft-, hard-assignment))

2013

This paper describes the system we used for the main task of Semantic INdexing (SIN) at TRECVID 2015. Our system uses a fivestage processing pipeline including feature extraction, pooling, encoding, classification and reranking. We employed CNN-based representations, as well as Dense and Root SIFTs as features for our system. We also report results of our experiments with SentiBank features and data augmentation techniques that did not contribute to the performance of the final system. Our second run ‘Rostam’ achieved an infAP of 26.67% on the 30 concepts evaluated for SIN 2015.