580 California St., Suite 400
San Francisco, CA, 94104
Key research themes
1. How can multimodal data fusion techniques be effectively designed and selected for diverse multimedia analysis and classification tasks?
This research area focuses on understanding the various methodologies and strategies for combining multiple data modalities (e.g., audio, video, textual, sensor) to enhance accuracy and robustness in multimedia analysis and classification. It emphasizes the comparison and selection of data fusion techniques (early fusion, late fusion, hybrid methods) depending on task characteristics, modalities involved, and computational constraints. Effective multimodal fusion improves decision making in applications like event detection, image annotation, and sensor data integration.
Key finding: The survey systematically categorizes multimodal fusion strategies into feature-level (early), decision-level (late), and hybrid fusion, noting that each approach has trade-offs in handling asynchronous data, heterogeneous... Read more
Key finding: This paper empirically compares three predominant fusion techniques — late fusion, early fusion, and sketch-based fusion — across multiple datasets combining diverse modalities including text, visual, and graph data. It... Read more
Key finding: This foundational work proposes a comprehensive multimedia information system architecture that integrates multiple media types (audio, video, image, text) and incorporates fusion at multiple levels (multimedia authoring,... Read more
2. What architectures and methods enable efficient and adaptive image and video fusion for enhanced visual information representation?
This theme investigates signal-level fusion methodologies and system architectures designed to precisely merge visual data from multiple sources—such as different sensors or image modalities—to form composite representations that preserve critical information while minimizing distortion. Special attention is given to multiresolution approaches, adaptive fusion within resource constraints, and implementation on specialized hardware platforms to support real-time and dynamic environmental conditions.
Key finding: Presents a novel 'fuse-then-decompose' multiresolution gradient map fusion approach that significantly reduces distortion artifacts and contrast loss in fused images compared to conventional methods. Gradient domain fusion... Read more
Key finding: Demonstrates the feasibility of implementing adaptive feature-based image fusion entirely on dynamically reconfigurable FPGA hardware onboard UAVs, enabling on-the-fly algorithm changes dependent on operational context. The... Read more
Key finding: Extends conventional web architectures to support real-time video and audio streaming integrated into hypertext documents using a specialized protocol (VDP) instead of TCP, drastically improving frame rates (from 0.2 to 9... Read more
3. How does spatial and contextual key integration facilitate multimedia content fusion for enhanced retrieval and situational awareness?
This research area explores leveraging spatial data and contextual metadata as unifying keys to fuse heterogeneous multimedia content—text, images, video—linking them through geospatial coordinates or other physical references. By contextualizing multimedia data within spatial frameworks, such as geographic information systems or location-based metadata, these approaches aim to improve multimedia content indexing, retrieval, and situational inference, applicable in environments ranging from digital libraries to crisis monitoring.
Key finding: Develops a framework utilizing spatial keys, including geographic coordinates and referenced data like addresses, to integrate disparate multimedia data (texts, images, videos) on the Web. Through automated extraction,... Read more
Key finding: Presents a geospatially grounded fusion architecture combining multisensor imagery (visible, hyperspectral) and signals (GMTI, ELINT) with 3D site modeling for urban crisis monitoring. Neural network search agents are trained... Read more
Key finding: Proposes a situation-aware multimedia composition framework for avionics, dynamically fusing diverse multimedia contents (text, image, audio, video) based on contextual parameters and real-time conditions. The system employs... Read more