A high-quality low-delay remote rendering system for 3D video

EURASIP Journal on Image and Video Processing, 2008

Interest in 3D video applications and systems is growing rapidly and technology is maturating. It is expected that multiview autostereoscopic displays will play an important role in home user environments, since they support multiuser 3D sensation and motion parallax impression. The tremendous data rate cannot be handled efficiently by representation and coding formats such as MVC or MPEG-C Part 3. Multiview video plus depth (MVD) is a new format that efficiently supports such advanced 3DV systems, but this requires high-quality intermediate view synthesis. For this, a new approach is presented that separates unreliable image regions along depth discontinuities from reliable image regions, which are treated separately and fused to the final interpolated view. In contrast to previous layered approaches, our algorithm uses two boundary layers and one reliable layer, performs image-based 3D warping only, and was generically implemented, that is, does not necessarily rely on 3D graphics support. Furthermore, different hole-filling and filtering methods are added to provide high-quality intermediate views. As a result, highquality intermediate views for an existing 9-view auto-stereoscopic display as well as other stereo-and multiscopic displays are presented, which prove the suitability of our approach for advanced 3DV systems.

IEEE Transactions on Multimedia, 2007

Three-dimensional (3-D) graphic scenes require considerable network bandwidth to be transmitted and computing power to be rendered on a user's terminal. Toward high-quality display in real time, we propose a sender-driven mechanism for streaming 3-D scenes in a resource-constrained environment. In doing so, objects are encoded into multiresolutions to provide transmission and rendering scalability, and a weighted distortion metric is developed to measure the quality of a scene rendered with multiresolution objects, modeling objects' unequal importance regarding display. To preserve the manipulation independency of multiple objects in data delivery while provide preferential treatment for different objects as well as different layers of each object, transmission of the objects is performed over multiple streams in a partially sequenced and partially reliable fashion. A rate-distortion optimization framework is developed, which determines an optimal level of reliability for every chunk of data in each stream, taking into account the rendering importance of the object, the distortion-rate performance of the data chunks, and the statistics of the network link. Compared with heuristical methods, simulation results show that the proposed framework maximizes the display quality of the scene while minimizing the amount of data that needs to be processed by the client's rendering engine.

Proceedings of the 21st ACM international conference on Multimedia, 2013

3D collaborative Tele-Immersive environments allow reconstruction of real world 3D scenes in the virtual world across multiple physical locations. This kind of reconstruction results in a lot of 3D data being transmitted over the internet in real time. The current systems allow for transmission at low frame rates due to the large volume of data and network bandwidth restrictions. In this paper we propose a prediction based approach that generates future frames by animating the live model based on few skeleton points. By doing so the magnitude of data transmitted is reduced to few hundred bytes. The prediction errors are corrected when an entire frame is received. This approach allows minimal amounts (few bytes) of data to be transmitted per frame, thus allowing for high frame rates and still maintain an acceptable visual quality of reconstruction at the receiver side.

Journal of Zhejiang University Science, 2006

The authors propose a novel method for transporting multi-view videos that aims to keep the bandwidth requirements on both end-users and servers as low as possible. The method is based on application layer multicast, where each end point receives only a selected number of views required for rendering video from its current viewpoint at any given time. The set of selected videos changes in real time as the user’s viewpoint changes because of head or eye movements. Techniques for reducing the black-outs during fast viewpoint changes were investigated. The performance of the approach was studied through network experiments.

Lecture Notes in Computer Science, 2014

We present an approach for high quality rendering of the 3D representation of a remote collaboration scene, along with real-time rendering speed, by expanding the unstructured lumigraph rendering (ULR) method. ULR uses a 3D proxy which is in the simplest case a 2D plane. We develop dynamic proxy for ULR, to get a better and more detailed 3D proxy in real-time; which leads to the rendering of highquality and accurate 3D scenes with motion parallax support. The novel contribution of this work is the development of a dynamic proxy in realtime. The dynamic proxy is generated based on depth images instead of color images as in the Lumigraph approach.

3D Research, 2011

Autostereoscopic 3DTV is becoming an exciting media that enable us to view a 3D scene from more than one viewpoint. Meanwhile, considered as the ultimate autostereoscopic 3DTV, Free-viewpoint TV (FTV) can provide arbitrary views by freely synthesizing and changing viewpoints. Essentially, either 3DTV or FTV is based on virtual view synthesis using captured views along with corresponding depth information. In this paper, we study how virtual views can be reliably generated from multiple captured videos for 3D display. One key challenge is that the required depth information may contain depth errors, leading to uncomfortable artifacts in the synthesized view. We review the recent progress in virtual view synthesis methods where depth reliability is considered to handle synthesis artifacts and improve the quality of the virtual view. Not only for intermediate virtual view, have we also presented high-quality close-up view synthesis methods for wider navigation in 3DTV and FTV.

A 3D video stream is typically obtained from a set of synchronized cameras, which are simultaneously capturing the same scene (multiview video). This technology enables applications such as free-viewpoint video which allows the viewer to select his preferred viewpoint, or 3D TV where the depth of the scene can be perceived using a special display. Because the user-selected view does not always correspond to a camera position, it may be necessary to synthesize a virtual camera view. To synthesize such a virtual view, we have adopted a depthimage-based rendering technique that employs one depth map for each camera. Consequently, a remote rendering of the 3D video requires a compression technique for texture and depth data. This paper presents a predictivecoding algorithm for the compression of depth images across multiple views. The presented algorithm provides (a) an improved coding efficiency for depth images over block-based motion-compensation encoders (H.264), and (b), a random access to different views for fast rendering. The proposed depth-prediction technique works by synthesizing/computing the depth of 3D points based on the reference depth image. The attractiveness of the depth-prediction algorithm is that the prediction of depth data avoids an independent transmission of depth for each view, while simplifying the view interpolation by synthesizing depth images for arbitrary view points. We present experimental results for several multiview depth sequences, that result in a quality improvement of up to 1.8 dB as compared to H.264 compression.

2007

Multi-view 3D displays are preferable to other stereoscopic display technologies because they provide autostereoscopic viewing from any viewpoint without special glasses. However, they require a large number of pixels to achieve high image quality. Therefore, data compression is a major issue for this approach. In this paper, we describe a framework for efficient compression of multi-view video streams for multi-view 3D displays. We enhance conventional multi-view compression pipelines with an additional pre-filtering step that bandlimits the multi-view signals to the display bandwidth. We show that this prefiltering step leads to increased image quality compared to state-of-the-art multi-view coding at equal bitrate. We present results of an extensive user study that corroborate the benefits of our approach. Our work suggests that any multi-view compression scheme will benefit from our pre-filtering techniques.

2006 IEEE International Conference on Multimedia and Expo, 2006

This paper presents a layered framework for 3D-TV applications, combining multiview and depth-image based approaches in a scalable fashion. To solve the problem of missing data due to disocclusions, we add specific layers for coded occlusion data and the edge-mask information for high-quality 3D rendering of key objects in the scene. We show how the same framework can be extended towards FTV applications by jointly addressing simulcast and multicast transmission. By adopting a distributed delivery architecture, new interesting properties can be realized such as shared processing for the creation and streaming of virtual viewpoints.

2005

This paper presents a method to transmit liv e 3D v ideos for a soccer stadium and display them at remote PCs for users who watch soccer games. All processes are done in real time, including soccer scene capture with multiple cameras, image analy sis, data transmission, and 3D v ideo display . To generate and transmit 3D v ideo v ia network in real time, each play er is represented by "play er-billboard" which is composed of one rectangle and associated liv e tex ture. Because a play er-billboard does not require precise 3D shape reconstruction of each play er, computation cost and data siz e required to generate liv e 3D v ideos are v ery low.