Rushes summarization by IRIM consortium

This paper presents the framework of a general video summarisation system on the rushes collection, which formalises the summarisation process as an 0 − 1 Knapsack optimisation problem. Three stages are included, namely content analysis, content selection and summary composition. Content analysis is the pre-processing step, consisting of shot segmentation, feature extraction, raw video discrimination and shot clustering. Content selection weights the importance of video segments by an attention model. A greedy approximation approach is employed in the composition of summary videos with a cost function, which balances the video importance gain and the duration cost. The average content coverage achieved on the rushes test collection is about 29%, while the average score on readability is 3.13 with the redundancy credit at 4.08.

2008

This paper describes the video summarization system developed by the partners of the K-Space European Network of Excellence for the TRECVID 2008 BBC rushes summarization evaluation. We propose an original method based on individual content segmentation and selection tools in a collaborative system. Our system is organized in several steps. First, we segment the video, secondly we identify relevant and redundant segments, and finally, we select a subset of segments to concatenate and build the final summary with video acceleration incorporated. We analyze the performance of our system through the TRECVID evaluation.

Proceeding of the 2nd ACM workshop on Video summarization - TVS '08, 2008

Generating short summary videos for rushes is a challenging task due to the difficulty in eliminating redundancy and determining the important objects and events to be placed in the summary. Redundancy elimination is difficult since repetitive segments, which are takes of the same scene, usually have different lengths and motion patterns. This makes approaches using one keyframe for a shot representation fail when doing clustering. In addition, even repetitive segments can be precisely determined, but the summary generated by concatenating together the selected segments still takes longer than the upper limit. Selecting a sub-segment that conveys as much of the information concerning a given scene as possible might be a good way to improve this process. We introduce two approaches to solve these problems. In the first approach, one keyframe is used for representing a shot when doing clustering; and sub-segments are selected using the motion information for generating the summary. Meanwhile, in the second approach, all the frames of a given shot are used for clustering; and a simple skimming method is used to select the sub-segments. The experimental results on the TRECVID 2008 dataset and a comparison between the two approaches are also reported.

Proceedings of the international workshop on TRECVID video summarization - TVS '07, 2007

In this paper, we describe our approach for video summarization that was applied to the BBC rushes material as part of the TRECVID 2007 evaluations. The method consists of initial shot boundary detection followed by shot similarity assessment and pruning, with both stages implemented using multiple parallel Self-Organizing Maps and within our content-based multimedia information retrieval and analysis framework named PicSOM. The results indicate that our approach can be successfully applied to rushes summarization. Compared to other submissions, our method resulted in the overall shortest summaries with close to median performance in the fraction of ground-truth inclusions found.

2008

In this paper we describe a system for video rushes summarization. The basic problems of rushes videos are three. First, the presence of useless frames such as colorbars, monochrome frames and frames containing clapboards. Second, the repetition of similar segments produced from multiple takes of the same scene and finally, the efficient representation of the original video in the video summary. In the method we proposed herein, the input video is segmented into shots. Then, colorbars and monochrome frames are removed by checking their edge direction histogram, whereas frames containing clapboards are removed by checking their SIFT descriptors. Next, an enhanced spectral clustering algorithm that both estimates the number of clusters and employs the fast global k-means algorithm in the clustering stage after the eigenvector computation of the similarity matrix is used to extract the key-frames of each shot, to efficiently represent shot content. Similar shots are clustered in one group by comparing their key-frames using a sequence alignment algorithm. Each group is represented from the shot with the largest duration and the final video summary is generated by concatenating frames around the key-frames of each shot. Experiments on TRECVID 2008 Test Data indicate that our method exhibits good performance.

For the four year we have participated to the high-level feature extraction task and we pursued our effort on the fusion of classifier outputs. Unfortunatly a single run was submitted for evaluation this year, due to lack of computationnal ressources during the limited time available for training and tuning the entire system. This year's run is based on a SVM classification scheme. Localised color and texture features were extracted from shot key-frames. Then, SVM classifiers were build per concept on the training data set. The fusion of classifier outputs is finally provided by a multilayer neural network. In BBC rushes exploitation, we explore the description of rushes through a visual dictionary. A set of non-redundant images are segmented into blocks. These blocks are clustered in a small number of classes to create a visual dictionary. Then, we can describe each image by the number of blocks of each class. After, we evaluate the power of this visual dictionary for retrieving images from rushes: if we use one or more blocks from an image as a query, are we able to retrieve the original image, and in which position in the result list. And finally, we organize and present video using this visual dictionary.

2008

For the first time in 2007, TRECVID considered structured evaluation of automated video summarization, utilizing BBC rushes video. In 2007, we conducted user evaluations with the published TRECVID summary assessment procedure to rate a cluster method for producing summaries, a 25x (sampling every 25th frame), and pz (emphasizing pans and zooms). Data from 4 human assessors shows significant differences between the cluster, pz, and 25x approaches. The best coverage (text inclusion performance) is obtained by 25x, but at the expense of 25x taking the most time to evaluate and judged as being the most redundant. Method pz was easier to use than cluster and rated best on redundancy. A question following the TRECVID workshop was whether simple speed-ups would still work at 50x or 100x, leading to a study with 15 human assessors looking at pzA (pz but with better audio), 25x, 50x, and 100x summaries (these latter 3 with an unsynchronized more comprehensive audio track as well). 100x gives the fastest time on task but with poor usability and performance. PzA gives the best usability measures but poor time on task and performance. 25x does well on performance as before, with 50x doing just as well but with much less time on task and better ease of use and redundancy scores. Based on these results, 50x with its audio skimming is recommended as the best way to summarize video rushes materials.

2003

In our previous work, we described an adaptive fast playback framework for video summarization where we changed the playback rate using the motion activity feature so as to maintain a constant pace. This method provides an effective way of skimming through video, especially when the motion is not too complex and the background is mostly still, such as in surveillance video. In this paper, we present an extended summarization framework that, in addition to motion activity, uses semantic cues such as face or skin color appearance, speech and music detection, or other domain dependent semantically significant events to control the playback rate. The semantic features we use are computationally inexpensive and can be computed in compressed domain, yet are robust, reliable, and have a wide range of applicability across different content types. The presented framework also allows for adaptive summaries based on preference, for example, to include more dramatic vs. action elements, or vice versa. The user can switch at any time between the skimming and the normal playback modes. The continuity of the video is preserved, and complete omission of segments that may be important to the user is avoided by using adaptive fast playback instead of skipping over long segments. The rule-set and the input parameters can be further modified to fit a certain domain or application. Our framework can be used by itself, or as a subsequent presentation stage for a summary produced by any other summarization technique that relies on generating a sub-set of the content.

2021

Automatic video summarization is still an unsolved problem due to several chal1 lenges. The currently available datasets either have very short videos or have a few 2 long videos of only a particular type. We introduce a new benchmarking video 3 dataset called VISIOCITY (VIdeo SummarIzatiOn based on Continuity, Intent 4 and DiversiTY) which comprises of longer videos across six different categories 5 with dense concept annotations capable of supporting different flavors of video 6 summarization and other vision problems. Secondly, human reference summaries 7 necessary for supervised video summarization techniques are difficult to obtain, 8 especially for long videos. We explore strategies to automatically generate mul9 tiple reference summaries from indirect ground truth present in VISIOCITY. We 10 also present a study of different desired characteristics of a good summary and 11 argue that evaluating a summary against one or more human summaries and us12 ing a single measure has it...