Multimedia Information Indexing and Retrieval

The image retrieval and semantic extraction play an important role in the multimedia systems such as geographic information system, hospital information system, digital library system, etc. Therefore, the research and development of semantic-based image retrieval (SBIR) systems have become extremely important and urgent. Major recent publications are included covering different aspects of the research in this area, including building data models, low-level image feature extraction, and deriving high-level semantic features. However, there is still no general approach for semantic-based image retrieval (SBIR), due to the diversity and complexity of high-level semantics. In order to improve the retrieval accuracy of SBIR systems, our focus research is to build a data structure for finding similar images, from that retrieving its semantic. In this paper, we proposed a data structure which is a self-balanced clustering tree named C-Tree.  Firstly, a method of visual semantic analysis re...

The image retrieval and semantic extraction play an important role in the multimedia systems such as geographic information system, hospital information system, digital library system, etc. Therefore, the research and development of semantic-based image retrieval (SBIR) systems have become extremely important and urgent. Major recent publications are included covering different aspects of the research in this area, including building data models, low-level image feature extraction, and deriving high-level semantic features. However, there is still no general approach for semantic-based image retrieval (SBIR), due to the diversity and complexity of high-level semantics. In order to improve the retrieval accuracy of SBIR systems, our focus research is to build a data structure for finding similar images, from that retrieving its semantic. In this paper, we proposed a data structure which is a self-balanced clustering tree named C-Tree.  Firstly, a method of visual semantic analysis re...

The image retrieval and semantic extraction play an important role in the multimedia systems such as geographic information system, hospital information system, digital library system, etc. Therefore, the research and development of semantic-based image retrieval (SBIR) systems have become extremely important and urgent. Major recent publications are included covering different aspects of the research in this area, including building data models, low-level image feature extraction, and deriving high-level semantic features. However, there is still no general approach for semantic-based image retrieval (SBIR), due to the diversity and complexity of high-level semantics. In order to improve the retrieval accuracy of SBIR systems, our focus research is to build a data structure for finding similar images, from that retrieving its semantic. In this paper, we proposed a data structure which is a self-balanced clustering tree named C-Tree.  Firstly, a method of visual semantic analysis re...

http://www.regim.org Description of Submitted Runs Semantic Indexing Regim_4: The indexing process is based on the visual modality analysis and relationships within LSCOM Ontology to improve the detection of large set of semantic concepts. The visual modality analysis is orientated towards an automatic categorization of video contents to create relevance relationships between low-level descriptions and semantic contents according to a user point of view. Regim_5: This indexing system is based on Multimodal fuzzy fusion using positive rules extracted from LSCOM Ontology. The fusion process employs both a deduction and abduction reasoning engines. Regim_6: This indexing system is based on Multimodal fuzzy fusion using positive and negative rules extracted from LSCOM Ontology. The fusion process employs also both a deduction and abduction reasoning engines.

In this paper, a technique for making more effective the similarity search process of images in a Multimedia Content Management System is proposed. The contentbased retrieval process integrates the search on different multimedia components, linked in XML structures.

In this paper a technique for evaluating the effectiveness of MPEG-7 image features on specific image data sets is proposed. It is based on well defined statistical characteristics. The aim is to improve the effectiveness of the image retrieval process, based on the similarity computed on these features. This technique is validated with extensive experiments with real users.

Most existing content-based video retrieval (CBVR) systems are now amenable to support automatic low-level feature extraction, but they still have limited effectiveness from a user's perspective because of the semantic gap. Automatic video concept detection via semantic classification is one promising solution to bridge the semantic gap. To speed up SVM video classifier training in high-dimensional heterogeneous feature space, a novel multimodal boosting algorithm is proposed by incorporating feature hierarchy and boosting to reduce both the training cost and the size of training samples significantly. To avoid the inter-level error transmission problem, a novel hierarchical boosting scheme is proposed by incorporating concept ontology and multitask learning to boost hierarchical video classifier training through exploiting the strong correlations between the video concepts. To bridge the semantic gap between the available video concepts and the users' real needs, a novel hyperbolic visualization framework is seamlessly incorporated to enable intuitive query specification and evaluation by acquainting the users with a good global view of large-scale video collections. Our experiments in one specific domain of surgery education videos have also provided very convincing results.

In this paper, we describe our participation in the Image-CLEF 2015 Scalable Concept Image Annotation task. In this participation, we display our approach for an automatic image annotation by the use of an ontology-based semantic hierarchy handled at both learning and annotation steps. While recent works focused on the use of semantic hierarchies to improve concept detector accuracy, we are investigating the use of such hierarchies to reduce detector complexity and then, to handle efficiently large-scale image datasets. Our framework is based on two steps: (1) constructing a fuzzy ontology through analyzing learning dataset, and (2) guiding the annotation process through a reasoning engine. The obtained results confirm that this approach is promising for scalable image annotation.

In this paper a technique for evaluating the effectiveness of MPEG-7 image features on specific image data sets is proposed. It is based on well defined statistical characteristics. The aim is to improve the effectiveness of the image retrieval process, based on the similarity computed on these features. This technique is validated with extensive experiments with real users.

In this paper, a technique for making more effective the similarity search process of images in a Multimedia Content Management System is proposed. The content-based retrieval process integrates the search on different multimedia components, linked in XML structures. Depending on the specific characteristics of an image data set, some features can be more effective than others when performing similarity search. Starting from this observation, we propose a technique that predicts the effectiveness of MPEG-7 image features based on a statistical analysis of the specific data sets in the Multimedia Content Management System. This technique is validated through an extensive experimentation with real users.

The ImageCLEF 2013 Scalable Concept Image Annotation Subtask was the second edition of a challenge aimed at developing more scalable image annotation systems. Unlike traditional image annotation challenges, which rely on a set of manually annotated images as training data for each concept, the participants were only allowed to use automatically gathered web data instead. The main objective of the challenge was to focus not only on the image annotation algorithms developed by the participants, where given an input image and a set of concepts they were asked to decide which of them were present in the image and which ones were not, but also on the scalability of their systems, such that the concepts to detect were not exactly the same between the development and test sets. The participants were provided with web data consisting of 250,000 images, which included textual features obtained from the web pages on which the images appeared, as well as various visual features extracted from the images themselves. To evaluate the performance of the submitted systems a development set was provided containing 1,000 images that were manually annotated for 95 concepts and a test set containing 2,000 images that were annotated for 116 concepts. In total 13 teams participated, submitting a total of 58 runs, most of which significantly outperformed the baseline system for both the development and test sets, including for the test concepts not present in the development set and thus clearly demonstrating potential for scalability.

In this paper, a technique for making more effective the similarity search process of images in a Multimedia Content Management System is proposed. The contentbased retrieval process integrates the search on different multimedia components, linked in XML structures.

The amount of audio-visual information has increased dramatically with the advent of High Speed Internet. Furthermore, technological advances in recent years in the field of information technology, have simplified the use of video data in various fields by the general public. This made it possible to store large collections of video documents into computer systems. To enable efficient use of these collections, it is necessary to develop tools to facilitate access to these documents and handling them. In this paper we propose a method for indexing and retrieval of video sequences in a video database of large dimension, based on a weighting technique to calculate the degree of membership of a concept in a video also a structuring of the data of the audio-visual (context / concept / video) and a relevance feedback mechanism.

Semantic multimedia organization is an open challenge. In this chapter, we present an innovative way of automatically organizing multimedia information to facilitate content-based browsing. It is based on self-organizing maps. The visualization capabilities of the self-organizing map provide an intuitive way of representing the distribution of data as well as the object similarities. The main idea is to visualize similar documents spatially close to each other, while the distance between different documents is bigger. We demonstrate this on the particular case of video information. One key concept is the disregard of the temporal aspect during the clustering. We introduce a novel time bar visualization that reprojects the temporal information. The combination of innovative visualization and interaction methods allows efficient exploration of relevant information in multimedia content.

La visita a musei o a luoghi di interesse di città d'ar-te può essere completamente reinventata attraverso modalità di fruizione moderne e dinamiche, basa-te su tecnologie di riconoscimento e localizzazione visuale, ricerca per immagini e visualizzazioni in realtà aumentata. Da anni il gruppo di ricerca AI-MIR porta avanti attività di ricerca su queste temati-che ricoprendo anche ruoli di responsabilità in pro-getti nazionali ed internazionali. Questo contributo riassume alcune delle attività di ricerca svolte e del-le tecnologie utilizzate, nonché la partecipazione a progetti che hanno utilizzato tecnologie di intelli-genza artificiale per la valorizzazione e la fruizione del patrimonio culturale. 1 Introduzione Il gruppo di ricerca Artificial Intelligence for Multimedia Information Retrieval (AIMIR) studia soluzioni di intelligen-za artificiale per l'analisi, ricerca e riconoscimento visuale in database di immagini di grandi dimensioni, tramite disposi-tivi mobili, sistemi informativi e motori di ricerca multime-diali. Negli ultimi anni, ha partecipato a numerosi progetti nazionali ed internazionali in ambito Beni Culturali, svilup-pando sistemi che consentono di riconoscere automaticamen-te, a partire da un'immagine, opere d'arte quali quadri, statue , edifici, iscrizioni antiche, effettuarne ricerche visuale su larga scala e visualizzazioni in realtà aumentata. Si consi-derino, ad esempio, il sistema http://art.isti.cnr.it/ capace di riconoscere e fornire informazioni su più di 100 mila quadri, o http://www.eagle-network.eu/image-search/ capace di rico-noscere visivamente iscrizioni antiche, in un database di più di un milione di immagini, anche da dispositivi mobili. Le tecniche sviluppate tengono in considerazione sia le problematiche di accuratezza che di scalabilità, garantendo lo sviluppo di sistemi con tempi di risposta fluidi e natura-li anche in situazioni e contesti dove la quantità di elementi da riconoscere, localizzare visivamente, e rendere aumentati è enorme, come all'interno di musei, o in zone di interesse di importanti città d'arte (piazze storiche, cattedrali, etc.). 2 Attività Scientifica L'attività scientifica portata avanti dal gruppo AIMIR sfrut-ta una sinergia di tecniche di analisi delle immagini, deep learning, strutture dati ed algoritmi di ricerca per similarità scalabili. I prototipi di ricerca sviluppati sono stati applica-ti con successo nell'ambito dei beni culturali, ad esempio, per riconoscere opere d'arte o edifici storici, per accedere ad informazioni in realtà aumentata, e per generare descri-zione automatiche di materiale digitale non adeguatamente annotato. Nell'ambito del riconoscimento visuale sono stati investi-gati sia approcci basati su aggregazioni (per es. BoW, VLAD, FV) di feature locali di immagini (quali SIFT ed ORB), sia feature estratte da reti neurali convoluzionali (CNN feature), che approcci ibridi (quale la combinazione di FV con CNN feature). Gli approcci ibridi basati sulla combinazione di aggregazioni di feature locali e CNN feature, per esempio, hanno mostrato una elevata efficacia nel riconoscimento di iscrizioni antiche [Amato et al., 2016b]. Approcci basati su "hand-crafted" feature e deep learning sono stati studiati ed utilizzati anche per la classifi-cazione automatica, il retrieval di immagini, la localizza-zione visuale ed applicazioni di realtà aumentata [Amato et al., 2015; Bolettieri et al., 2015; Amato et al., 2017b; Amato et al., 2017a]. Inoltre, per poter effettuare ricer-che visuali anche in datatabase di enormi dimensioni, sono state sviluppati innovativi algoritmi di ricerca per similari-tà approssimata [Amato et al., 2014; Amato et al., 2016a; Amato et al., 2018]. 3 Progetti in Ambito Beni Culturali Negli ultimi anni, il gruppo AIMIR ha partecipato a numerosi progetti nazionali ed internazionali su tematiche relative ai beni culturali e all'analisi del contenuto delle immagini per l'estrazione automatica di informazioni che ne permettano la descrizione automatica, il riconoscimento, la classificazione, la ricerca su larga scala, ed il loro accesso in realtà aumentata. Si citano a titolo d'esempio: VISECH-Visual Engines for Cultural Heritage, progetto regionale che ha lo scopo di avanzare lo stato dell'arte nel-l'ambito dell'analisi automatica delle immagini, sviluppando tecniche di riconoscimento e localizzazione visuale per effet-tuare realtà aumentata, mediante algoritmi altamente scala

In this paper a technique for evaluating the effectiveness of MPEG-7 image features on specific image data sets is proposed. It is based on well defined statistical characteristics. The aim is to improve the effectiveness of the image retrieval process, based on the similarity computed on these features. This technique is validated with extensive experiments with real users.

In this paper, a technique for making more effective the similarity search process of images in a Multimedia Content Management System is proposed. The content-based retrieval process integrates the search on different multimedia components, linked in XML structures. Depending on the specific characteristics of an image data set, some features can be more effective than others when performing similarity search. Starting from this observation, we propose a technique that predicts the effectiveness of MPEG-7 image features based on a statistical analysis of the specific data sets in the Multimedia Content Management System. This technique is validated through an extensive experimentation with real users.

—Audio Event Detection (AED) aims to recognize sounds within audio and video recordings. AED employs machine learning algorithms commonly trained and tested on annotated datasets. However, available datasets are limited in number of samples and hence it is difficult to model acoustic diversity. Therefore, we propose combining labeled audio from a dataset and unlabeled audio from the web to improve the sound models. The audio event detectors are trained on the labeled audio and ran on the unlabeled audio downloaded from YouTube. Whenever the detectors recognized any of the known sounds with high confidence, the unlabeled audio was use to retrain the detectors. The performance of the retrained detectors is compared to the one from the original detectors using the annotated test set. Results showed an improvement of the AED, and uncovered challenges of using web audio from videos.

In this paper, we consider the automatic identification of video shots that are relevant to a given semantic concept from large video databases. We apply a method of representing semantic concepts as class models on a set of parallel Self-Organizing Maps trained with multimodal low-level features. The presented experiments were conducted using a set of 170 hours of video containing recorded television news programs.

In this paper, we propose a semantic indexing system for reducing the semantic gap between the machine and human interpretations on a video document by generating a finer indexing quality. To do so, data fusion of analyzed interpretation (concepts) from different video modalities (sound and images) is viewed as the required choice.

Multimedia indexing systems based on semantic concept detectors are incomplete in the semantic sense. We can improve the effectiveness of these systems by using knowledge-based approaches which utilize semantic knowledge. In this paper, we propose a novel and efficient approach to enhance semantic concept detection in multimedia content, by exploiting contextual information about concepts from visual modality. First, a semantic knowledge is extracted via a contextual annotation framework. Second, a Fuzzy ontology is proposed to represent the fuzzy relationships (roles and rules) among every context and its semantic concepts. We use an abduction engine based on βeta function as a membership function for fuzzy rules. Third, a deduction engine is used to handle richer results in our video indexing system by running the proposed fuzzy ontology. Experiments on TRECVID 2010 benchmark have been performed to evaluate the performance of this approach. The obtained results show consistent improvement in semantic concepts detection, when a context space is used, and a good degree of indexing effectiveness as compared to existing approaches.

A video retrieval system user hopes to find relevant information when the pro-
posed queries are ambiguous. The retrieval process based on detecting concepts remains
ineffective in such a situation. Potential relationships between concepts have been shown as
a valuable knowledge resource that can enhance the retrieval effectiveness, even for ambigu-
ous queries. Recent researches in multimedia retrieval have focused on ontology modeling
as a common framework to manage knowledge. Handling these ontologies has to cope with
issues related to generic knowledge management and processing scalability. Considering
these issues, we suggest a context-based fuzzy ontology framework for video content anal-
ysis and indexing. In this paper, we focused on the way in which we modeled our fuzzy
ontology: First, we populate automatically the generated ontology by gathering various
available video annotation datasets. Then, the ontology content was used to infer enhanced
video semantic interpretation. Finally, considering user feedback, the content of the ontol-
ogy was improved. Experimental results showed that our approach achieves the goal of
scalability while at the same time allowing better video content semantic interpretation.

In this paper, we describe our participation in the Image-
CLEF 2015 Scalable Concept Image Annotation task. In this participa-
tion, we display our approach for an automatic image annotation by the
use of an ontology-based semantic hierarchy handled at both learning
and annotation steps. While recent works focused on the use of semantic
hierarchies to improve concept detector accuracy, we are investigating
the use of such hierarchies to reduce detector complexity and then, to
handle efficiently large-scale image datasets. Our framework is based on
two steps: (1) constructing a fuzzy ontology through analyzing learning
dataset, and (2) guiding the annotation process through a reasoning en-
gine. The obtained results confirm that this approach is promising for
scalable image annotation.

This paper introduces a novel approach of video annota-
tion by the use of context-based assistance for the annotator.
The notion of context plays, actually, a significant role in the
multimedia content search and retrieval systems. In fact, a
semantic interpretation (or concept) separated from its con-
text produces incomplete information. Moreover, each con-
cept interpretation varies according to different contexts.
The assistance that we introduce uses intelligent structures
such as context ontology and previous annotation. The eval-
uation of the proposed assisted annotation prototype has led
to promising results by the use of context in the annotation
process.

Multimedia indexing systems based on semantic concept detectors
are incomplete in the semantic sense. We can improve the
effectiveness of these systems by using knowledge-based
approaches which utilize semantic knowledge. In this paper, we
propose a novel and efficient approach to enhance semantic
concept detection in multimedia content, by exploiting contextual
information about concepts from visual modality. First, a semantic
knowledge is extracted via a contextual annotation framework.
Second, a Fuzzy ontology is proposed to represent the fuzzy
relationships (roles and rules) among every context and its
semantic concepts. We use an abduction engine based on βeta
function as a membership function for fuzzy rules. Third, a
deduction engine is used to handle richer results in our video
indexing system by running the proposed fuzzy ontology.
Experiments on TRECVID 2010 benchmark have been performed
to evaluate the performance of this approach. The obtained results
show consistent improvement in semantic concepts detection,
when a context space is used, and a good degree of indexing
effectiveness as compared to existing approaches.

In this paper, we propose a semantic indexing system
for reducing the semantic gap between the machine and human
interpretations on a video document by generating a finer
indexing quality. To do so, data fusion of analyzed interpretation
(concepts) from different video modalities (sound and images) is
viewed as the required choice.
Considering problems that can emanate from employing data
fusion, our approach is based on some intelligent techniques
aiming to improve semantic indexing quality of the R EGIM V ID
framework. At first, we use fuzzy logic to deal with uncertain
and conflicting situations. An ontology is exploited for laying
out relationships between different concepts. Finally, a deductive
and inductive inference engines are used to discover respectively
further concepts and further relationships between concepts.
We developed a fusion system that responds, in its first version,
to such point treated in our approach. Our participation in the
semantic indexing task of the TRECVID2010 competition has led
to promising results.

In this paper, we describe an overview of a software platform that has been developed within REGIMVid project for TRECVID 2010 video retrieval experiments. The REGIMVID team participated in Semantic Indexing task. In
TRECVID 2010, we explore several novel techniques to perform the detection of semantic concepts, including multi
classifiers with supervised learning process, discriminative feature representation based on local keypoints, visual
concept fusion using LSCOM rules, and also multimodal concept fusion using LSCOM Ontology.

This paper proposes a learning method for Beta fuzzy systems (BFS) based on a multiagent genetic algo-rithm. This method, called Multi-Agent Genetic Algorithm for the Design of BFS has two advantages. First, thanks to genetic algorithms (GA) efficiency, it allows to design ...

Abstract Traditional Content Based Multimedia Retrieval (CBMR) systems measure the relevance of visual samples using a binary scale (Relevant/Non Relevant). However, a picture can be relevant to a semantic category with different degrees, depending on the way such concept is represented in the image. In this paper, we build a CBMR framework that supports graded relevance judgments.

Abstract Content-based classification of audio data is an important problem for various applications such as overall analysis of audio-visual streams, boundary detection of video story segment, extraction of speech segments from video, and content-based video retrieval. Though the classification of audio into single type such as music, speech, environmental sound and silence is well studied, classification of mixed type audio data, such as clips having speech with music as background, is still considered a difficult problem.

Abstract—In this paper, we propose to organize web images by actively creating visual clusters via crowdsourcing. We develop a two-phase framework to efficiently and effectively combine computers and a large number of human workers to build high quality visual clusters. The first phase partitions an image collection into multiple clusters; the second phase refines each generated cluster independently. In both phases, informative images are selected by computers and manually labeled by the crowds to learn improved models.