A visual descriptor ontology for multimedia reasoning

Proc. of the European …, 2004

2005

In this paper, an ontology infrastucture for multimedia reasoning is presented, making it possible to combine low-level visual descriptors with domain specific knowledge and subsequently analyze multimedia content with a generic algorithm that makes use of this knowledge. More specifically, the ontology infrastructure consists of a domain-specific ontology, a visual descriptor ontology (VDO) and an upper ontology. In order to interpret a scene, a set of atom regions is generated by an initial segmentation and their descriptors are extracted. Considering all descriptors in association with the related prototype instances and relations, a genetic algorithm labels the atom regions. Finally, a constraint reasoning engine enables the final region merging and labelling into meaningful objects.

Multimedia reasoning, which is suitable for, among others, multimedia content analysis and high-level video scene interpretation, relies on the formal and comprehensive conceptualization of the represented knowledge domain. However, most multimedia ontologies are not exhaustive in terms of role definitions, and do not incorporate complex role inclusions and role interdependencies. In fact, most multimedia ontologies do not have a role box at all, and implement only a basic subset of the available logical constructors. Consequently, their application in multimedia reasoning is limited. To address the above issues, VidOnt, the very first multimedia ontology with SROIQ(D) expressivity and a DL-safe ruleset has been introduced for next-generation multimedia reasoning. In contrast to the common practice, the formal grounding has been set in one of the most expressive description logics, and the ontology validated with industry-leading reasoners, namely HermiT and FaCT++. This paper also presents best practices for developing multimedia ontologies, based on my ontology engineering approach.

Knowledge representation and annotation of multimedia documents typically have been pursued in two different directions. Previous approaches have focused either on low level descriptors, such as dominant color, or on the content dimension and corresponding manual annotations, such as person or vehicle. In this paper, we present a knowledge infrastructure to bridge the gap between the two directions. Ontologies are being extended and enriched to include low-level audiovisual features and descriptors. Additionally, a tool for linking low-level MPEG-7 visual descriptions to ontologies and annotations has been developed. In this way, we construct ontologies that include prototypical instances of domain concepts together with a formal specification of the corresponding visual descriptors. Thus, we combine high-level domain concepts and low-level multimedia descriptions, enabling for new media content analysis.

International Semantic Web Working Symposium (SWWS), 2001

For the past two years the Moving Pictures Expert Group (MPEG), a working group of ISO/IEC, have been developing , the "Multimedia Content Description Interface", a standard for describing multimedia content. The goal of this standard is to develop a rich set of standardized tools to enable both humans and machines to generate and understand audiovisual descriptions which can be used to enable fast efficient retrieval from digital archives (pull applications) as well as filtering of streamed audiovisual broadcasts on the Internet (push applications). MPEG-7 is intended to describe audiovisual information regardless of storage, coding, display, transmission, medium, or technology. It will address a wide variety of media types including: still pictures, graphics, 3D models, audio, speech, video, and combinations of these (e.g., multimedia presentations). MPEG-7 is due for completion in October 2001. At this stage MPEG-7 definitions (description schemes and descriptors) are expressed solely in XML Schema . XML Schema has been ideal for expressing the syntax, structural, cardinality and datatyping constraints required by MPEG-7. However it has become increasingly clear that in order to make MPEG-7 accessible, re-usable and interoperable with other domains then the semantics of the MPEG-7 metadata terms also need to be expressed in an ontology using a machine-understandable language. This paper describes the trials and tribulations of building such an ontology represented in RDF Schema and demonstrates how this ontology can be exploited and reused by other communities on the semantic web (such as TV-Anytime [6], , NewsML [8], museum, educational and geospatial domains) to enable the inclusion and exchange of multimedia content through a common understanding of the associated MPEG-7 multimedia content descriptions. • A Description Definition Language (DDL) to allow the creation of new Description Schemes and, possibly, Descriptors and to allows the extension and modification of existing Description Schemes; • System tools, to support multiplexing of descriptions, synchronization of descriptions with content, transmission mechanisms and coded representations (both textual and binary formats) for efficient storage and transmission, management and protection of intellectual property in MPEG-7 descriptions. XML Schema language has been chosen as the DDL [9] for specifying MPEG-7 descriptors and description schemes because of its ability to express the syntactic, structural, cardinality and datatyping constraints required by MPEG-7 and because it also provides the necessary mechanisms for extending and refining existing DSs and Ds. However it has recently become increasingly clear that there is also a need for a machine-understandable representation of the semantics associated with MPEG-7 DSs and Ds to enable the interoperability and integration of MPEG-7 with metadata descriptions from other domains. New metadata initiatives such as TV-Anytime [6], MPEG-21 [7], NewsML , and communities such as the museum, educational, medical and geospatial communities, want to combine MPEG-7 multimedia descriptions with new and existing metadata standards for simple resource discovery (Dublin Core [10]), rights management (INDECS [11]), geospatial (FGDC [12]), educational (GEM [13], IEEE LOM ) and museum (CIDOC CRM ) content, to satisfy their domain-specific requirements. In order to do this, there needs to be a common understanding of the semantic relationships between metadata terms from different domains. XML Schema provides little support for expressing semantic knowledge. RDF Schema provides us with a way to do this. The Resource Description Framework (RDF) is the accepted language of the semantic web due to its ability to express semantics and semantic relationships through class and property hierarchies. In this paper, we investigate the feasibility of expressing the semantics of MPEG-7 Descriptors (Ds) and Description Schemes (DSs) in an RDF Schema ontology. An earlier paper evaluated RDF Schema for video metadata representation (prior to the development of MPEG-7) and determined a number of limitations . In this paper we hope to ascertain whether those limitations still exist when representing the semantics of MPEG-7 DSs and Ds or whether they can be overcome -either by using the extra constraints provided by DAML+OIL [17] or through combining RDF Schema semantics with XML Schema encoding specifications in a complementary manner. Whilst manually building the RDF Schema for a core subset of MPEG-7, we also hope to be able to recognize patterns and hence determine automatic mechanisms for generating compatible RDF Schema definitions corresponding to the complete set of MPEG-7 XML Schema definitions. In Section 2 we describe the methodology, problems encountered and results of building an RDF Schema ontology for MPEG-7. In Section 3 we describe how the RDF Schema semantic definitions for MPEG-7 can be linked to their corresponding pre-existing XML Schema definitions (or recommended encodings). In Section 4 we describe how the MPEG-7 RDF Schema can be merged with RDF schemas from other domains to generate a single "super-ontology" called MetaNet. Expressed in DAML+OIL , MetaNet can be used to provide common semantic understanding between domains. Finally we illustrate how this super-ontology can be used to enable the co-existence of interoperability, extensibility and diversity within metadata descriptions generated by integrating metadata terms from different domains.