Papers by Klara Nahrstedt
CZLoD
ACM Transactions on Multimedia Computing, Communications, and Applications, Sep 1, 2012
ABSTRACT This article presents a psychophysical study that measures the perceptual thresholds of ... more ABSTRACT This article presents a psychophysical study that measures the perceptual thresholds of a new factor called Color-plus-Depth Level-of-Details (CZLoD) peculiar to polygon-based 3D tele-immersive video. The results demonstrate the existence of Just Noticeable Degradation and Just Unacceptable Degradation thresholds on the factor. In light of the results, we design and implement a real-time perception-based quality adaptor for 3D tele-immersive video. Our experimental results show that the adaptation scheme can reduce resource usage (e.g., CPU cycles) while considerably enhancing the overall perceived visual quality. Our analysis confirms the potential temporal and spatial performance benefits achievable with CZLoD adaptation.
2015 IEEE International Symposium on Multimedia (ISM), 2015
3D video is a new form of video featuring omniview and immersive properties that are beneficial t... more 3D video is a new form of video featuring omniview and immersive properties that are beneficial to many applications like telediagnosis, remote physiotherapy and elearning. In these applications, the data are always multi-modal and highly synchronized. For 2D video streaming, the MPEG group standardized Dynamic Adaptive Streaming over HTTP (DASH) with major transport formats, MPEG-2 TS and ISO Base Media File, including video, audio and subtitle formats. This paper presents OmniViewer, a multi-modal 3D streaming system based DASH, enhancing the versatility of the existing DASH standard and delivery infrastructure such as HTTP servers. Besides 3D video, OmniViewer supports multi-modal media in the DASH content for real-time media-synchronized interactive services. OmniViewer provides a complete multi-modal 3D video streaming solution from recording to rendering through DASH, which fills the gap between monoscopic 3D systems and 3D streaming systems.
IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications, 2016
Free-viewpoint television (FTV) is a visionary application that provides immersive experience to ... more Free-viewpoint television (FTV) is a visionary application that provides immersive experience to the audience with the freedom of changing viewpoint during the video playout. However, live broadcasting and user interaction do not coexist in existing FTV systems. In this paper, we propose F.Live, a framework of FTV content dissemination that supports userinitiated viewpoint changing for live broadcasting. Simulation result of a large-scale experiment, based on camera array settings of existing Nagoya systems and EyeVision System, shows that F.Live is capable of supporting 100,000 concurrent audiences with free-viewpoint low user interaction latency and feasible bandwidth requirements.
Proceedings of the 21st ACM international conference on Multimedia - MM '13, 2013
We present TEEVE Endpoint, which is a runtime engine to handle the creation, transmission and ren... more We present TEEVE Endpoint, which is a runtime engine to handle the creation, transmission and rendering of 3D Tele-immersive (3DTI) data and provides application programming interfaces (APIs) to developers to easily create 3DTI applications.
Proceedings of the 21st ACM international conference on Multimedia, 2013
In view of the different demands on quality of service of different user activities in the 3D Tel... more In view of the different demands on quality of service of different user activities in the 3D Tele-immersive (3DTI) environment, we combine activity recognition and real-time morphing-based compression and present the Activity-Aware Adaptive Compression. We implement this scheme on our 3DTI platform: the TEEVE Endpoint, which is a runtime engine to handle the creation, transmission and rendering of 3DTI data. User study as well as objective evaluation of the scheme show that it can achieve 25% more bandwidth saving compared to conventional 3D data compression as zlib without perceptible degradation in the user experience.
Proceedings of the 14th ACM international conference on Multimedia, 2006
We present a study of collaborative dancing between remote dancers in a tele-immersive environmen... more We present a study of collaborative dancing between remote dancers in a tele-immersive environment which features 3D full and real body capturing, wide field of view, multi-display 3D rendering, and attachment free participant. We invite two professional dancers to perform collaborative dancing in the environment. The coordination requires one dancer to take the lead while the other follows by appropriate movement. Throughout the experiment, the dancers are dancing at various motion rates to evaluate how well the collaborative dancing is supported with the current technical boundary. Our important findings indicate that 1) tele-immersive environments have strong potential impact on the concept of choreography and communication of live dance performance, 2) the presence of multi-view display, real body 3D rendering, audio channel, and less intrusiveness greatly enhances the immersive and dancing experience, and 3) the level of synchronization achieved by the dancers is higher than that expected from the video rate.
Proceedings of the 19th ACM international conference on Multimedia, 2011
In this demonstration, we present two 3D tele-immersive games: light-saber dual and block fencing... more In this demonstration, we present two 3D tele-immersive games: light-saber dual and block fencing that merge 3D video representations of participants in real-time to enable remote interactions in a virtual world. The light-saber dual arranges participants in a symmetric setup where both participants interact with each other in a virtual world with similar goals. On the other hand, the block fencing creates an asymmetric setup where participants interact with virtual objects having different goals. Using these two setups, we address the challenges and novelty of our solutions in portable environment setup, data acquisition, multi-stream synchronization, multi-stream session management, mobile device rendering, and overlay communication in the design and implementation of advanced 3D tele-immersive systems.
Active 4d tele-vision
Proceedings of the 20th international workshop on Network and operating systems support for digital audio and video - NOSSDAV '10, 2010
Abstract 3D television, 3D movie experiences, 3D tele-immersive and tele-presence environments, a... more Abstract 3D television, 3D movie experiences, 3D tele-immersive and tele-presence environments, and extensive 3D gaming environments from Nintendo and Microsoft are becoming integral parts of our lives. Further advances in stereo and multi-view video, 3D graphics, spatial audio, Wii-like interface, 3D displays, and other multi-modal sensory devices, all contributing to acquisition, generation, display and control of new 4D media, already have and will have major impact on the next generation of multimedia systems ...
Proceedings of the ACM International Conference on Multimedia - MM '14, 2014
In this paper we verify the insufficiency of Stevens' power law to describe the relationship betw... more In this paper we verify the insufficiency of Stevens' power law to describe the relationship between QoS and QoE factors. User studies that target different types of application scenarios of 3D Tele-immersion (3DTI) are conducted and the results show no significant power trend in the relationship between packet loss and perceptual quality metrics. We further verify that activity characteristics, activity objectives, and users' roles in the 3DTI session also have profound effects on the service quality aside to the QoS level. Thus, simple one-factor psychophysical laws are inadequate of serving as a QoS-QoE mapping model.
Proceedings of the 16th ACM international conference on Multimedia, 2008
The Tele-immersive Dance Environment (TED) is a geographically distributed, real-time 3-D virtual... more The Tele-immersive Dance Environment (TED) is a geographically distributed, real-time 3-D virtual room where multiple participants interact independent of physical distance. TED, a highly interactive collaborative environment, offers digital options with multiple viewpoints, enhancing the creative movement composition involved with dance choreography. A symbiotic relationship for creativity and design exists between dance artists and computer scientists as the tele-immersive environment is analyzed as a creativity and learning tool. We introduce the advancements of the interactive digital options, new interface developments, user study results, and the possibility of a computational model for human creativity through Laban Movement Analysis.
2008 IEEE International Conference on Multimedia and Expo, 2008
The 3D tele-immersive (3DTI) environments are emerging as the next generation technique for tele-... more The 3D tele-immersive (3DTI) environments are emerging as the next generation technique for tele-communication. In this paper, we present a novel and generic interface for view control in 3D environments. The interface uses Wii Remote, a wireless controller for a video game console. It allows the user to manipulate the virtual camera and 3D objects intuitively with buttons and through motions (pitch and roll). We conducted user studies to evaluate the interface with a professional dancer and average audiences. The results demonstrate that the Wii Remote interface is much better suited to view control in 3DTI environments since it is much easier to use, more effective, and more accurate than the conventional mouse-based interfaces.
2009 29th IEEE International Conference on Distributed Computing Systems, 2009
Users and administrators of large distributed systems are frequently in need of monitoring and ma... more Users and administrators of large distributed systems are frequently in need of monitoring and management of its various components, data items and resources. Though there exist several distributed query and aggregation systems, the clustered structure of tele-immersive interactive frameworks and their time-sensitive nature and application requirements represent a new class of systems which poses different challenges on this distributed search. Multi-attribute composite range queries are one of the key features in this class. Queries are given in high level descriptions and then transformed into multi-attribute composite range queries. Designing such a query engine with minimum traffic overhead, low service latency, and with static and dynamic nature of large datasets, is a challenging task. In this paper, we propose a general multi-attribute based range query framework, Q-Tree, that provides efficient support for this class of systems. In order to serve efficient queries, Q-Tree builds a single topology-aware tree overlay by connecting the participating nodes in a bottom-up approach, and assigns range intervals on each node in a hierarchical manner. We show the relative strength of Q-Tree by analytically comparing it against P-Tree, P-Ring, Skip-Graph and Chord. With fine-grained load balancing and overlay maintenance, our simulations with PlanetLab traces show that our approach can answer complex queries within a fraction of a second.
Proceedings of the 17th ACM international conference on Multimedia, 2009
We present MobileTI, a portable tele-immersive system that merges 3D video representations of use... more We present MobileTI, a portable tele-immersive system that merges 3D video representations of users in real time to enable remote collaboration across geographical distances. With portability as a main goal, we address the challenges in the camera setup, time synchronization, video acquisition, and networking in the design and implementation of the system. Having been deployed in public performances, MobileTI proves to be effective, efficient, and user-friendly. Our experimental findings in terms of technical performance and user feedback are presented.
Proceedings of 2nd Workshop on General Purpose Processing on Graphics Processing Units, 2009
As computational power increases, tele-immersive applications are an emerging trend. These applic... more As computational power increases, tele-immersive applications are an emerging trend. These applications make extensive demands on computational resources through their heavy use of real-time 3D reconstruction algorithms. Since computer vision developers do not necessarily have parallel programming expertise, it is important to give them the tools and capabilities to naturally express computer vision algorithms, yet retain high efficiency by exploiting modern GPU and large-scale multi-core platforms. In this paper, we describe our optimization efforts for a tele-immersion application by tuning it for GPU and multicore platforms. Additionally, we introduce a method to obtain portability, high performance, and increase programmer productivity.
Proceedings of the 18th ACM international conference on Multimedia, 2010
As an emerging technology, 3D video shows a great potential to become the next generation media f... more As an emerging technology, 3D video shows a great potential to become the next generation media for tele-immersion. However, streaming and rendering this dynamic 3D data in real-time requires tremendous network bandwidth and computing resources. In this paper, we build a remote rendering model to better study different remote rendering designs and define 3D video rendering as an optimization problem. Moreover, we design a 3D video remote rendering system that significantly reduces the delay while maintaining high rendering quality. We also propose a reference viewpoint prediction algorithm with super sampling support that requires much less computation resources but provides better performance than the search-based algorithms proposed in the related work.
Proceedings of the 16th ACM international conference on Multimedia, 2008
3D video is an emerging technology that promises immersive experiences in a truly seamless enviro... more 3D video is an emerging technology that promises immersive experiences in a truly seamless environment. Currently, 3D video systems still require excessive bandwidth and computation power provided by gigabit switches and multi-core workstations machines. In order to extend the experience to mobile devices, we present a view-dependent compression methodology that shows great promise in making 3D video a reality on resource-constrained mobile devices. Using our technology, we are able to achieve a software-only rendering on a Nokia N800 PDA with only wireless network transmission. We believe that with the use of newer handhelds and improvements to our compression techniques, we will be able to deliver full-motion 3D video soon.
2010 IEEE International Symposium on Multimedia, 2010
Distributed Interactive Multimedia Environments (DIMEs) show important dependency constraints bet... more Distributed Interactive Multimedia Environments (DIMEs) show important dependency constraints between application and underlying system components over time. For example, the video frame rate and the underlying bandwidth usage have a strong performance dependency. Performance dependencies must also be considered among distributed components. These dependencies over a time-span form correlation relationships. Violations of such correlation relationships represent collective anomalies. Users and most specifically DIME application developers face problems of finding (detecting), localizing such anomalies, and adapting against them in real-time. Current practices are to collect joint application-system metadata characterizing behaviors of application and system components while a DIME session is running, and then analyze them offline. Our goal is to provide a framework, called DIAMOND, that allows for real-time and unobtrusive collection and organization of joint applicationsystem metadata in order to assist in finding such correlation violations in the system. DIAMOND works in four steps: (a) real-time metadata collection, (b) metadata processing to allow efficient computation of correlation constraints, (c) metadata distribution for efficient clustering of distributed metadata, and (d) anomaly detection, localization, and evolution monitoring based on violations of correlation relationships. Our results from real implementations and simulations with PlanetLab traces show the effectiveness of DIAMOND in terms of network overhead and anomaly detection time.
Proceedings of the 12th annual ACM international conference on Multimedia, 2004
The realization of multimedia systems still heavily relies on building monolithic systems that ne... more The realization of multimedia systems still heavily relies on building monolithic systems that need to be reengineered for every change in the application and little of which can be reused in subsequent developments even for similar applications. Hence, building complex large scale multimedia systems is still a difficult and challenging problem. Service-based architectures, like researched in the Web community, form a possible solution to this problem: The service-based paradigm decomposes complex tasks into smaller independent entities (e.g. Web services), and then supports a flexible service composition in a variety of ways. However, due to the characteristics of multimedia applications and rich semantic structure of multimedia data and workflows, a direct application of Web-based research results is still difficult. The reason is that Web service frameworks cannot yet cope with the complexity of multimedia applications and their metadata. In this paper, we describe a basic taxonomy for the composition of services to support complex multimedia workflows. We will investigate in detail the necessary steps and methodology for multimedia service compositions and apply our taxonomy to different service composition instances. We will illustrate all composition instances within our taxonomy with case studies and point to possible techniques for the composition problem.
2012 Proceedings IEEE INFOCOM, 2012
Existing media playout scheduling (MPS) schemes usually focus on selecting and scheduling packets... more Existing media playout scheduling (MPS) schemes usually focus on selecting and scheduling packets according to optimized Internet media metrics, which are only partially relevant to the subjective human perception in the interactive system. The MPS design challenges are twofold. First, human preferences are concurrently dominated by multiple quality attributes of the streaming media whose perceptual tradeoffs were not well understood, so they were not used as an integral part of an efficient MPS design. Second, people's perceptions can be impacted by the flicker effect caused by Internet dynamics and the resulting MPS adaptations. In this paper, we propose a new and adaptive perception-based MPS scheme to deliver highquality real-time interactive multimedia. We first investigate the perceptual tradeoffs among the multi-modal bundle streaming qualities in a real Internet environment. We then present our MPS design that finds the bundle quality tradeoffs, while minimizing flicker degradations. Evaluation results show the performance of our MPS scheme.
2009 IEEE International Conference on Multimedia and Expo, 2009
The next-generation social communication medium, teleimmersion, is receiving increasing attention... more The next-generation social communication medium, teleimmersion, is receiving increasing attention from both the research and industrial communities. It enables interaction between geographically distributed sites through realistic reconstruction of physical scenes in immersive spaces. While the previous research focused on real-world measurements of implemented tele-immersive systems, we take a model-based approach to study the system more generally and thoroughly. With a model validated by actual measurements, we evaluate the performance of the system in a larger scale, and understand the quantitative effect of the algorithmic enhancements that were proposed in the literature. We also predict futuristic performance characteristic and show that high interactivity can be foreseen with hardware advances in the near future.
Uploads
Papers by Klara Nahrstedt