An audio-visual saliency model for movie summarization

International Journal of Multimedia Data Engineering and Management, 2019

Video summarization condenses a video by extracting its informative and interesting segments. In this article, a novel video summarization approach is proposed based on spatiotemporal salient region detection. The proposed approach first segments a video into a set of shots which are ranked with spatiotemporal saliency scores. The score for a shot is computed by aggregating the frame level spatiotemporal saliency scores. This approach detects spatial and temporal salient regions separately using different saliency theories related to objects present in a visual scenario. The spatial saliency of a video frame is computed using color contrast and color distribution estimations and center prior integration. The temporal saliency of a video frame is estimated as an integration of local and global temporal saliencies computed using patch level optical flow abstractions. Finally, top ranked shots with the highest saliency scores are selected for generating the video summary. The objective...

2001

Compact representations of video data can enable efficient video browsing. Such representations provide the user with information about the content of the particular sequence being examined while preserving the essential message. We propose a method to automatically generate video summaries for long videos. Our video summarization approach involves mainly two tasks: first, segmenting the video into small, coherent segments and second, ranking the resulting segments. Our proposed algorithm scores segments based on word frequency analysis of speech transcripts. Then a summary is generated by selecting the segments with the highest score to duration ratios and these are concatenating them. We have designed and performed a user study to evaluate the quality of summaries generated. Comparisons are made using our proposed algorithm and a random segment selection scheme based on statistical analysis of the user study results. Finally we discuss various issues that arise in summary evaluation with user studies.

Human Vision and Electronic Imaging XIII, 2008

According to the literature, automatic video summarization techniques can be classified in two parts, following the output nature: "video skims", which are generated using portions of the original video and "key-frame sets", which correspond to the images, selected from the original video, having a significant semantic content. The difference between these two categories is reduced when we consider automatic procedures. Most of the published approaches are based on the image signal and use either pixel characterization or histogram techniques or image decomposition by blocks. However, few of them integrate properties of the Human Visual System (HVS). In this paper, we propose to extract keyframes for video summarization by studying the variations of salient information between two consecutive frames. For each frame, a saliency map is produced simulating the human visual attention by a bottom-up (signal-dependent) approach. This approach includes three parallel channels for processing three early visual features: intensity, color and temporal contrasts. For each channel, the variations of the salient information between two consecutive frames are computed. These outputs are then combined to produce the global saliency variation which determines the key-frames. Psychophysical experiments have been defined and conducted to analyze the relevance of the proposed key-frame extraction algorithm.

Computers & Electrical Engineering, 2014

Video summarization is an integral component of video archiving systems. It provides small versions of the videos that are suitable for enhancing browsing and navigation capabilities. A popular method to generate summaries is to extract a set of key frames from the video, which conveys the overall message of the video. This paper introduces a novel feature aggregation based visual saliency detection mechanism and its usage for extracting key frames. The saliency maps are computed based on the aggregated features and motion intensity. A non-linear weighted fusion mechanism combines the two saliency maps. On the resultant map, a Gaussian weighting scheme is used to assign more weight to the pixels close to the center of the frame. Based on the final attention value of each frame, the key frames are extracted adaptively. The experimental results, based on different evaluation standards, demonstrate that the proposed scheme extracts semantically significant key frames.

Nowadays, with the explosion of multimedia content availability, the selectiveness of its consumption increases in importance. Audiovisual content is no longer brought to us only by the television, being available through many other systems, like Personal Video Recorders and Video on Demand systems, on each one's Personal Computer or, obviously, in the Internet. The exponential growth of websites like YouTube shows that people assign great relevance to audiovisual content in these days. Moreover, common people can easily produce, store, distribute and view those contents as it is not required any specialized skills to do so. To browse for a specific content in any of these systems can be a long, painful task. To manually summarize videos for the user's own purposes, as showing a summary of his/her vacations to friends and family also takes much time and it is a complex task. As people's time is getting more precious and scarce every day, an application capable of saving the time spent in these tasks by automatically summarizing audiovisual generic content arises as useful. Motivated by these factors, this report describes the developed solution for automatic audiovisual summarization for generic content, its motivations, the architecture as well as the process for summarization designed and implemented in the course of this work. To evaluate the quality of the created summaries, a user evaluation study was conducted with encouraging results, showing that the developed application is able, with relative success, to summarize audiovisual generic content.

Proceedings of the tenth ACM international conference on Multimedia - MULTIMEDIA '02, 2002

Automatic generation of video summarization is one of the key techniques in video management and browsing. In this paper, we present a generic framework of video summarization based on the modeling of viewer's attention. Without fully semantic understanding of video content, this framework takes advantage of computational attention models and eliminates the needs of complex heuristic rules in video summarization. A set of methods of audiovisual attention model features are proposed and presented. The experimental evaluations indicate that the computational attention based approach is an effective alternative to video semantic analysis for video summarization.

Proceedings of the 19th International Conference on World Wide Web, WWW '10, 2010

Video summarization is a mechanism for generating short summaries of the video to help people quickly make sense of the content of the video before downloading or seeking more detailed information. To produce reliable automatic video summarization algorithms, it is essential to first understand how human beings create video summaries with manual efforts. This paper focuses on a corpus of instructional documentary video, and seeks to improve automatic video summaries by understanding what features in the video catch the eyes and ears of human assessors, and using these findings to inform automatic summarization algorithms. The paper contributes a thorough and valuable methodology for performing automatic video summarization, and the methodology can be extended to inform summarization of other video corpuses. Categories and Subject Descriptors H.3.1 [Information Storage And Retrieval]: Content Analysis and Indexing-abstracting methods, indexing methods.

International Journal of Computer Theory and Engineering, 2014

The growing number of videos on the internet requires effective management strategies. Video summarization is a method for managing video data which provides succinct versions of the videos for efficient browsing and retrieval. The inter-frame disparity based key frame extraction is a popular scheme for summarizing videos. However, the performance of such schemes is limited by the fact that most of these techniques select the first frame in a shot as key frame which may not be representative of the shot. In this paper, we propose a saliency inspired inter-frame difference based summarization scheme which selects the most important frame of the shot based on color contrast saliency. The preliminary experimental results prove the efficacy of the proposed method.