Multimedia information seeking through search and hyperlinking

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, 2017

In this paper, we survey different state-of-the-art visual processing methods and utilize them in hyperlinking. Visual information, calculated using Features Signatures, SIMILE descriptors and convolutional neural networks (CNN), is utilized as similarity between video frames and used to find similar faces, objects and setting. Visual concepts in frames are also automatically recognized and textual output of the recognition is combined with search based on subtitles and transcripts. All presented experiments were performed in the Search and Hyperlinking 2014 MediaEval task and Video Hyperlinking 2015 TRECVid task.

Lecture Notes in Computer Science, 2012

In this article, a unified framework for multimodal search and retrieval is introduced. The framework is an outcome of the research that took place within the I-SEARCH European Project. The proposed system covers all aspects of a search and retrieval process, namely low-level descriptor extraction, indexing, query formulation, retrieval and visualisation of the search results. All I-SEARCH components advance the state of the art in the corresponding scientific fields. The I-SEARCH multimodal search engine is dynamically adapted to end-user's devices, which can vary from a simple mobile phone to a high-performance PC.

Finding useful information from large multimodal document collections such as the WWW is one of the major challenges of Information Retrieval (IR). The many sources of information now available -text, images, audio, video and more -increases the need for multimodal search. Particularly important is also the recognition, that each information item is inherently multimodal (i.e. has aspects in its information character that stem from different modalities) and forms part of a networked set of related information items. In this paper we propose a graph-based model for multimodal information retrieval based on a faceted view of information objects. For retrieval purposes, we consider both relatedness and similarity relations between objects.

2010

There is a trend towards construction of multimedia digital information resources which may hold diverse data types in the form of image, graphics, audio, video, and text based retrieval artifacts or objects. WWW is a huge multimedia information resource. Existing search mechanisms available on WWW are mostly mono-modal. Multimedia information needs are partially satisfied by using mono-modal search mechanisms. Multiple modalities of information are associated with multimedia retrieval object so multimodal search mechanisms are required for searching multimedia information resources. Existing search mechanisms available on WWW whether they are general purpose, domain specific or media specific partially fulfil multimedia information needs because they are mostly mono-modal and able to perform search with in one media type at a time. We explore with the help of multimedia object analysis that search with in all the subsets of multimedia object types is possible by using only four index types, two models of interactions and fifteen possible modes at most. A framework based on an architectural approach is proposed for multimedia information retrieval. Proposed retrieval framework is implemented. Present implementation gives query formulation and information presentation for all modes and search functionality with in few modes of multimedia search framework.

Journal of the American Society for Information Science and Technology, 2009

Searching for multimedia is an important activity for users of Web search engines. Studying user's interactions with Web search engine multimedia buttons, including image, audio, and video, is important for the development of multimedia Web search systems. This paper provides results from a Weblog analysis study of multimedia Web searching by Dogpile users in 2006.

2018 IEEE International Conference on Multimedia and Expo (ICME), 2018

Video hyperlinking offers a way to explore a video collection, making use of links that connect segments having related content. Hyperlinking systems thus seek to automatically create links by connecting given anchor segments to relevant targets within the collection. In this paper, we further investigate multimodal representations of video segments in a hyperlinking system based on bidirectional deep neural networks, which achieved state-of-the-art results in the TRECVid 2016 evaluation. A systematic study of different input representations is done with a focus on the aggregation of the representation of multiple keyframes. This includes, in particular, the use of memory vectors as a novel aggregation technique, which provides a significant improvement over other aggregation methods on the final hyperlinking task. Additionally, the use of metadata is investigated leading to increased performance and lower computational requirements for the system.

Journal of the American Society for Information Science and Technology, 2009

Multimodal information retrieval is a research problem of great interest in all domains, due to the huge collections of multimedia data available in different contexts like text, image, audio and video. Researchers are trying to incorporate multimodal information retrieval using machine learning, support vector machines, neural network and neuroscience etc. to provide an efficient retrieval system that fulfills user need. This paper is an overview of multimodal information retrieval, challenges in the progress of multimodal information retrieval.

World Wide Web, 2000

The development of digital libraries has enhanced the integration of textual and multimedia information in many document collections. The World Wide Web provides the connectivity for many digital library users. Studies exploring the searching characteristics of Web users are an important and a growing area of research. Most Web user studies have focused on general Web searching, regardless of subject matter or format. Little research has examined how Web users search for multimedia information. Our study examines users' multimedia searching on a major Web search service. The data set examined consisted of 1,025,908 queries from 211,058 users of Excite , a major Web search service. From this data set, we identified and analyzed queries for audio, image, and video queries. Our findings were compared to results from general Web searching studies. Implications for the design of Web searching services and interfaces are discussed.

Signal Processing: Image Communication, 2013

In this paper, a complete solution for search and retrieval of rich multimedia content over modern databases is presented. The framework proposed in this paper combines the advantages of multimodal search with those of annotation propagation into a unified system. Moreover, an effective technique, which is appropriate for large-scale indexing, is adopted, extended and integrated to the proposed framework so as to achieve optimized search and retrieval of rich media content even from large-scale databases.