Semantic multimedia analysis

Detecting semantic events in sports video is crucial for video indexing and retrieval. Most existing works have exclusively relied on video content features, namely, directly available and extractable data from the visual and/or aural channels. Sole reliance on such data however, can be problematic due to the high-level semantic nature of video and the difficulty to properly align detected events with their exact time of occurrences. This paper proposes a framework for soccer goal event detection through collaborative analysis of multimodal features. Unlike previous approaches, the visual and aural contents are not directly scrutinized. Instead, an external textual source (i.e., minute-by-minute reports from sports websites) is used to initially localize the event search space. This step is vital as the event search space can significantly be reduced. This also makes further visual and aural analysis more efficient since excessive and unnecessary non-eventful segments are discarded, culminating in the accurate identification of the actual goal event segment. Experiments conducted on thirteen soccer matches are very promising with high accuracy rates being reported.

Movie data has a prominent role in the exponential growth of multimedia data over the Internet, and its analysis has become a hot topic with computer vision. The initial step towards movie analysis is scene segmentation. In this article, we investigated this problem through a novel intelligent Convolutional Neural Network (CNN) based three folded framework. The first fold segments the input movie into shots, the second fold detects objects in the segmented shots and the third fold performs object-based shots matching for detecting scene boundaries. Texture and shape features are fused for shots segmentation, and each shot is represented by a set of detected objects acquired from a light-weight CNN model. Finally, we apply set theory with the sliding window–based approach to integrate the same shots to decide scene boundaries. The experimental evaluation indicates that our proposed approach outran the existing movie scene segmentation approaches.

In this paper we propose the use of enhanced mid-level information, such as information obtained from the application of supervised or unsupervised learning methodologies on low-level characteristics, in order to improve semantic multimedia analysis. High-level, a priori contextual knowledge about the semantic meaning of objects and their low-level visual descriptions are combined in an integrated approach that handles in a uniform way the gap between semantics and low-level features. Prior work on low-level feature extraction is extended and a region thesaurus containing all mid-level features is constructed using a hierarchical clustering method. A model vector that contains the distances from each mid-level element is formed and a neural network-based detector is trained for each semantic concept. Contextual adaptation improves the quality of the produced results, by utilizing fuzzy algebra, fuzzy sets and relations. The novelty of the presented work is the contextdriven mid-level manipulation of region types, utilizing a domain-independent ontology infrastructure to handle the knowledge. Early experimental results are presented using data derived from the beach domain.

TV streams represent a principal source of multimedia information. The goal of the proposed approach is to enable a better exploitation of this source of video by multimedia services (i.e., TV-On-Demand, catch-up TV), social community, and video-sharing platforms (Vimeo, Youtube, Facebook …). In this work, we present an automatic structuring approach of TV news. The originality of the approach is the use of the contextual and operational characteristics as prior knowledge. This knowledge is modeled as video grammar which governs the structuring of TV stream content. This structuring is carried out on two levels. The first level identifies news programs in TV stream. The second level aims to identify the internal structure of the identified news programs. At this level, we opt to treat the case of TV news programs due to the large audience because of pertinent information within. Comparison experiments to similar works have been carried out on the TRECVID 2003 database. We show significant improvements to TV news structuring exceed 90 %.

The amount of digital video data is increasing over the world. It highlights the need for efficient algorithms that can index, retrieve and browse this data by content. This can be achieved by identifying semantic description captured automatically from video structure. Among these descriptions, text within video is considered as rich features that enable a good way for video indexing and browsing. Unlike most video text detection and extraction methods that treat video sequences as collections of still images, we propose in this paper spatiotemporal video-text localization and identification approach which proceeds in two main steps: text region localization and text region classification. In the first step we detect the significant appearance of the new objects in a frame by a split and merge processes applied on binarized edge frame pair differences. Detected objects are, a priori, considered as text. They are then filtered according to both local contrast variation and texture criteria in order to get the effective ones. The resulted text regions are classified based on a visual grammar descriptor containing a set of semantic text class regions characterized by visual features. A visual table of content is then generated based on extracted text regions occurring within video sequence enriched by a semantic identification. The experimentation performed on a variety of video sequences shows the efficiency of our approach.

The trend of learning from videos instead of documents has increased. There could be hundreds and thousands of videos on a single topic, with varying degrees of context, content, and depth of the topic. The literature claims that learners are nowadays less interested in viewing a complete video but prefers the topic of their interests. This develops the need for indexing of video lectures. Manual annotation or topic-wise indexing is not new in the case of videos. However, manual indexing is time-consuming due to the length of a typical video lecture and intricate storylines. Automatic indexing and annotation is, therefore, a better and efficient solution. This research aims to identify the need for automatic video indexing for better information retrieval and ease users navigating topics inside a video. The automatically identified topics are referred to as "Index Points." 137-layer YoloV4 Darknet Neural Network creates a custom object detector model. The model is trained on approximately 6000 video frames and then tested on a suite of 50 videos of around 20 hours of run time. Shot Boundary detection is performed using Structural Similarity fused with a Binary Search Pattern algorithm which outperformed the state-of-the-art SSIM technique by reducing the processing time to approximately 21% and providing around 96% accuracy. Generation of accurate index points in terms of true positives and false negatives is detected through precision, recall, and F1 score, which varies between 60-80% for every video. The results show that the proposed algorithm successfully generates a digital index with reasonable accuracy in topic detection.

Multimedia technology has been applied to many types of applications and the great amount of multimedia data need to be indexed. Especially the usage of digital video data is very popular today. In particular video browsing is a necessary activity in many kinds of knowledge. For effective and interactive exploration of large digital video archives there is a need to index the videos using their visual, audio and textual data. In this paper, we focus on the visual and textual content of video for indexing. In the former approach we use the Virtual Image and in the latter one we use the Dublin Core Metadata, opportunely extended and multilayered for the video browsing and indexing. Before to concentrate our attemption on the visual content we will explain main methods to video segmentation and annotation, in order to introduce the steps for video keyfeature extraction and video description generation.

This paper presents reliable techniques for detecting, tracking, and storing keyframes of people in surveillance video. The first component of our system is a novel face detector algorithm, which is based on first learning local adaptive features for each training image, and then using Adaboost learning to select the most general features for detection. This method provides a powerful mechanism for combining multiple features, allowing faster training time and better detection rates. The second component is a face tracking algorithm that interleaves multiple view-based classifiers along the temporal domain in a video sequence. This interleaving technique, combined with a correlation-based tracker, enables fast and robust face tracking over time. Finally, the third component of our system is a keyframe selection method that combines a person classifier with a face classifier. The basic idea is to generate a person keyframe in case the face is not visible, in order to reduce the number of false negatives. We performed quantitatively evaluation of our techniques on standard datasets and on surveillance videos captured by a camera over several days.

Abstraction is a strategy that gives the essential points of a document in a short period of time. The video abstraction approach proposed in this research is based on multi-modal video data, which comprises both audio and visual data. Segmenting the input video into scenes and obtaining a textual and visual summary for each scene are the major video abstraction procedures to summarize the video events into a static document. To recognize the shot and scene boundary from a video sequence, a hybrid features method was employed, which improves detection shot performance by selecting strong and flexible features. The most informative keyframes from each scene are then incorporated into the visual summary. A hybrid deep learning model was used for abstractive text summarization. The BBC archive provided the testing videos, which comprised BBC Learning English and BBC News. In addition, a news summary dataset was used to train a deep model. The performance of the proposed approaches was assessed using metrics like Rouge for textual summary, which achieved a 40.49% accuracy rate. While precision, recall, and F-score used for visual summary have achieved (94.9%) accuracy, which performed better than the other methods, according to the findings of the experiments.

Text present in a camera captured scene images is semantically rich and can be used for image understanding. Automatic detection, extraction, and recognition of text are crucial in image understanding applications. Text detection from natural scene images is a tedious task due to complex background, uneven light conditions, multi-coloured and multi-sized font. Two techniques, namely ‘edge detection' and ‘colour-based clustering', are combined in this paper to detect text in scene images. Region properties are used for elimination of falsely generated annotations. A dataset of 1250 images is created and used for experimentation. Experimental results show that the combined approach performs better than the individual approaches.

Multi-View Video summarization is a process to ease the storage consumption that facilitates organized storage, and perform other mainline videos analytical task. This in-turn helps quick search or browse and retrieve the video data with minimum time and without losing crucial data. In static video summarization, there is less challenge in time and sequence issues to rearrange the video-synopsis. The low-level features are easy to compute and retrieve. But for high-level features like event detection, emotion detection, object recognition, face detection, gesture detection, and others requires the comprehension of the video content. This research is to propose an approach to overcome the difficulties in handling the high-level features. The distinguishable contents from the videos are identified by object detection and feature-based area strategy. The major aspect of the proposed solution is to retrieve the attributes of a motion source from a video frame. By dividing the details of the object that are available in the video frame wavelet decomposition are achieved. The motion frequency scoring method records the time of motions in the video. The frequency motion feature of video usage is a challenge given the continuous change of objects shape. Therefore, the object position and corner points are spotted using Speeded Up Robust Features (SURF) feature points. Support vector machine clustering extracts keyframes. The memory-based recurrent neural network (RNN) recognizes the object in the video frame and remembers a long sequence. RNN is an artificial neural network where nodes form a temporal relationship. The attention layer in the proposed RNN network extracts the details about the objects in motion. The motion objects identified using the three video clippings is finally summarized using video summarization algorithm. To perform the simulation, MATLAB R 2014b software was used.

Multi-View Video summarization is a process to ease the storage consumption that facilitates organized storage, and perform other mainline videos analytical task. This in-turn helps quick search or browse and retrieve the video data with minimum time and without losing crucial data. In static video summarization, there is less challenge in time and sequence issues to rearrange the video-synopsis. The low-level features are easy to compute and retrieve. But for high-level features like event detection, emotion detection, object recognition, face detection, gesture detection, and others requires the comprehension of the video content. This research is to propose an approach to overcome the difficulties in handling the high-level features. The distinguishable contents from the videos are identified by object detection and feature-based area strategy. The major aspect of the proposed solution is to retrieve the attributes of a motion source from a video frame. By dividing the details of the object that are available in the video frame wavelet decomposition are achieved. The motion frequency scoring method records the time of motions in the video. The frequency motion feature of video usage is a challenge given the continuous change of objects shape. Therefore, the object position and corner points are spotted using Speeded Up Robust Features (SURF) feature points. Support vector machine clustering extracts keyframes. The memory-based recurrent neural network (RNN) recognizes the object in the video frame and remembers a long sequence. RNN is an artificial neural network where nodes form a temporal relationship. The attention layer in the proposed RNN network extracts the details about the objects in motion. The motion objects identified using the three video clippings is finally summarized using video summarization algorithm. To perform the simulation, MATLAB R 2014b software was used.

The last generation video compression techniques such as MPEG-4 and H.263+, allow presence of shot-cut detection methods that are usually built into the encoders. Such methods are used to extract key-frames to be encoded as intra frames. This paper presents automated scene change detection techniques, which mainly work over these key-frames to extract scene frames. Proposed techniques aim at unsupervised or semi-automatic scene detection schemes. Preliminary results show that the proposed framework provides near optimal scene frames suitable for video summarization by choosing semantically important key-frames as the scene frames. 1.

Color moments are measures that can be used to differentiate images based on their color. Once computed, these moments provide a measure for color similarity between images. This color similarity can become a measure of finding difference between consecutive frames in a video. Finding the difference between temporally arranged frames in video is one of the issues in shot boundary detection which will segment the video in to meaningful units called shots. So a Shot boundary detection method based on primitives of color moments is proposed in this paper. The YCbCr color model is used for the experimentation. Each frame of the video is partitioned in to 8 X 8 smaller sub blocks. The first three color moments of every sub blocks of each frame in the video are computed. The feature vector is formed from the moments and compared between successive frames. The difference value higher than a predefined threshold at any pair of consecutive frames indicates the presence of a shot boundary between frames. By using YCbCr color space, the influence of illumination change and shadows is reduced. Experiments have been conducted on the standard soccer video database to evaluate the performance of the proposed method.

As a partner in the Centre for Digital Video Processing, the Visual Media Processing Group at Dublin City University conducts research and development in the area of digital video management. The current stage of development is demonstrated on our Web-based digital video system called Físchlár [1,2], which provides for efficient recording, analyzing, browsing and viewing of digitally captured television programmes. In order to make the browsing of programme material more efficient, users have requested the option of automatically deleting advertisement breaks. Our initial work on this task focused on locating ad-breaks by detecting patterns of silent black frames which separate individual advertisements and/or complete ad-breaks in most commercial TV stations. However, not all TV stations use silent, black frames to flag ad-breaks. We therefore decided to attempt to detect advertisements using the rate of shot cuts in the digitised TV signal. This paper describes the implementation and performance of both methods of ad-break detection.

The amount of digital video data is increasing over the world. It highlights the need for efficient algorithms that can index, retrieve and browse this data by content. This can be achieved by identifying semantic description captured automatically from video structure. Among these descriptions, text within video is considered as rich features that enable a good way for video indexing and browsing. Unlike most video text detection and extraction methods that treat video sequences as collections of still images, we propose in this paper spatiotemporal video-text localization and identification approach which proceeds in two main steps: text region localization and text region classification. In the first step we detect the significant appearance of the new objects in a frame by a split and merge processes applied on binarized edge frame pair differences. Detected objects are, a priori, considered as text. They are then filtered according to both local contrast variation and texture criteria in order to get the effective ones. The resulted text regions are classified based on a visual grammar descriptor containing a set of semantic text class regions characterized by visual features. A visual table of content is then generated based on extracted text regions occurring within video sequence enriched by a semantic identification. The experimentation performed on a variety of video sequences shows the efficiency of our approach.

The amount of digital video data is increasing over the world. It highlights the need for efficient algorithms that can index, retrieve and browse this data by content. This can be achieved by identifying semantic description captured automatically from video structure. Among these descriptions, text within video is considered as rich features that enable a good way for video indexing and browsing. Unlike most video text detection and extraction methods that treat video sequences as collections of still images, we propose in this paper spatiotemporal video-text localization and identification approach which proceeds in two main steps: text region localization and text region classification. In the first step we detect the significant appearance of the new objects in a frame by a split and merge processes applied on binarized edge frame pair differences. Detected objects are, a priori, considered as text. They are then filtered according to both local contrast variation and texture criteria in order to get the effective ones. The resulted text regions are classified based on a visual grammar descriptor containing a set of semantic text class regions characterized by visual features. A visual table of content is then generated based on extracted text regions occurring within video sequence enriched by a semantic identification. The experimentation performed on a variety of video sequences shows the efficiency of our approach.

Scene change detection is a primitive operator for a wide range of video applications. At the same time, wavelet transformation has been validated as a powerful tool for multiresolution digital signal processing. In this paper, a novel scene change detection algorithm based on wavelet analysis is proposed and a browsing system is established that can perform scene change detection via multiresolution wavelet analysis on video streams.

Advances in multimedia compression standards, data storage, digital hardware technology
and network performance have led to a considerable increase in the amount of
digital content being archived and made available online. As a result, data organization,
representation and efficient search and retrieval from digital video repositories
has seen increased interest from the research community in recent years. In order to
facilitate access to desired media segments, many indexing techniques have been employed.
Automatic content structuring is one enabling technology used to aid browse/
retrieval. Scene-level analysis and sports summarization are two examples of active
research in this area. Content structuring can be considered as the task of building
an ’’index” and/or ’’table of contents” for events or objects that occur throughout a
programme.
Our approach to content structuring is to build an index based on the reappearance
of the main characters within the content. For news programmes, this can be used for
temporal segmentation into individual news stories based on the fact that the anchorperson,
the main ’’character” in this scenario signals the beginning of a news item.
For movie content, this could provide enhanced random access browsing functionality
to the end user. In this thesis we propose an approach to news story segmentation
that uses low-level features and three different algorithms for temporal segmentation.
We then extend this system to perform anchor-person detection using automatic face
detection and clustering algorithms. An extensive manually marked up test set has
been used to validate each component of our overall approach. Finally, we discuss
how our approach could be extended to identify the main characters in movie content
using similar classification techniques and directorial conventions.

The amount of digital video data is increasing over the world. It highlights the need for efficient algorithms that can index, retrieve and browse this data by content. This can be achieved by identifying semantic description captured automatically from video structure. Among these descriptions, text within video is considered as rich features that enable a good way for video indexing and browsing. Unlike most video text detection and extraction methods that treat video sequences as collections of still images, we propose in this paper spatiotemporal video-text localization and identification approach which proceeds in two main steps: text region localization and text region classification. In the first step we detect the significant appearance of the new objects in a frame by a split and merge processes applied on binarized edge frame pair differences. Detected objects are, a priori, considered as text. They are then filtered according to both local contrast variation and texture criteria in order to get the effective ones. The resulted text regions are classified based on a visual grammar descriptor containing a set of semantic text class regions characterized by visual features. A visual table of content is then generated based on extracted text regions occurring within video sequence enriched by a semantic identification. The experimentation performed on a variety of video sequences shows the efficiency of our approach.

In this paper, we propose a spatial temporal video-text detection technique which proceed in two principal steps: potential text region detection and a filtering process. In the first step we divide dynamically each pair of consecutive video frames into sub block in order to detect change. A significant difference between homologous blocks implies the appearance of an important object which may be a text region. The temporal redundancy is then used to filter these regions and forms an effective text region. The experimentation driven on a variety of video sequences shows the effectiveness of our approach by obtaining a 89,39% as precision rate and 90,19 as recall.

We introduce a video browsing interface 'mediaWalker' that lets users explore a news video archive based on a time-series semantic structure; the 'topic thread' structure. The interface lets users efficiently track up and down the development of news in an archive with more than 1,000 hours of video.

Recently, there is a strong demand for making use of large amounts of video data efficiently and effectively. When considering broadcast news video, people who appear in it is one of the major interests to a viewer. This is the common motivation of recent works that focus on extracting names of people that appear in news video footages. However, these works suffer a serious problem; a person is often referred to by various names depending on situations and along time. In this paper, we propose and evaluate a method that handles this problem by identifying faces together with names. Faces are extracted by face detection technology and annotated with person name candidates extracted from closedcaption text. Then, all face-name pairs are compared by face identification technology and text matching of names. As a result, different names of a same person are identified.

A news video can be modeled using the stratification approach by identifying, among other entities, human faces appearing in the video stream. To facilitate this, we need to develop techniques to detect and track human faces in video. This paper presents a frontal face detection method that uses the gradient energy representation extracted directly from the MPEG video. The gradient energy representation permits pertinent facial features of high contrast, such as the eyes, nose and mouth, to be highlighted. A rule-based classifier and a neural-network-based classifier are designed to classify a gradient energy pattern as face or non-face. The parameters for the two classifiers are learnt from face and non-face samples. First, we use the gradient energy face model to locate potential face regions at multiple scales and locations. Second, we perform skin-color verification to eliminate falsely detected regions. The main contribution of this work is in developing an efficient scale and position invariant method to detect faces that operates in a transformed gradient energy space in a compressed domain. The system was tested on selected video clips from an MPEG-7 data set and was found to be effective.

Detecting semantic events in sports video is crucial for video indexing and retrieval. Most existing works have exclusively relied on video content features, namely, directly available and extractable data from the visual and/or aural channels. Sole reliance on such data however, can be problematic due to the high-level semantic nature of video and the difficulty to properly align detected events with their exact time of occurrences. This paper proposes a framework for soccer goal event detection through collaborative analysis of multimodal features. Unlike previous approaches, the visual and aural contents are not directly scrutinized. Instead, an external textual source (i.e., minute-by-minute reports from sports websites) is used to initially localize the event search space. This step is vital as the event search space can significantly be reduced. This also makes further visual and aural analysis more efficient since excessive and unnecessary non-eventful segments are discarded, culminating in the accurate identification of the actual goal event segment. Experiments conducted on thirteen soccer matches are very promising with high accuracy rates being reported.

Abstract Instead of clustering video shots into scenes using low level image features, in this paper, we propose a rule-based model to extract simple dialog or action scenes. Through analyzing video editing rules and observing temporal appearance patterns of shots in dialog scenes of movies, we deduce a set of rules to recognize dialog or action scenes. Based on these rides, a finite state machine is designed to extract dialog or action scenes from videos automatically.

Abstract Instead of clustering video shots into scenes using low level image features, in this paper, we propose a rule-based model to extract simple dialog or action scenes. Through analyzing video editing rules and observing temporal appearance patterns of shots in dialog scenes of movies, we deduce a set of rules to recognize dialog or action scenes. Based on these rides, a finite state machine is designed to extract dialog or action scenes from videos automatically.

Abstract This research proposes a novel method to extract image regions of products from an advertisement video, by analyzing features which are completely independent from the target object. Namely, we focus on how each product is emphasized in the video production, and propose the utilization of low-level visual features which leverage the technical know-how of video producers.

Content-based video retrieval is the research that creates indices of videos. Early studies usually extract low-level image features as indices. Shot classification is one of various salient approaches to mine semantic information in video. This paper presents a classification approach that adopts classified results as indices for the retrieval of videos. After video segmentation and key frame extraction, a lot of features are extracted for each shot. These features include color and texture features. A classification framework using backpropagation neural networks as multiple binary classifiers is applied to classify video shots. Movie videos of 1016 shots are used to training the neural networks, and 1097 shots are experimented to testify the feasibility of our method. Video shots are classified into three semantic classes. The best experimental results can achieve 93.798% recognition rate. The high recognition rate indicates that this approach can produce high-level indices with high reliability.