ROI based video streaming for 3D remote rendering

2009 16th IEEE International Conference on Image Processing (ICIP), 2009

In this paper, we present a method to speed up video encoding of GPU rendered scenes. Modern video codecs, like H.264/AVC, are based on motion compensation and support partitioning of macroblocks, e.g. 16x16, 16x8, 8x8, 8x4 etc. In general, encoders use expensive search methods to determine suitable motion vectors and compare the rate-distortion score for possible macroblock partitionings, which results in high computational encoder load. We present a method to accelerate this process for the case of streaming graphical output of unmodified commercially available 3D games which use a Skybox or Skydome rendering technique. For rendered images, usually additional information from the render context of OpenGL resp. DirectX is available which helps in the encoding process. By incorporating the depth map from the graphics board, such regions can be uniquely identified. By adapting the macroblock partitioning accordingly, the computationally expensive search methods can often be avoided. Further reduction of encoding load is achieved by additionally capturing the projection matrices during the Skybox rending and using them to directly calculate a motion vector which is usually the result of expensive search methods. In experiential results, we demonstrate the reduced computational encoder load.

Pattern Recognition Letters, 2004

Realistic and interactive telepresence has been a hot research topic in recent years. Enabling telepresence using depth-based new view rendering requires the compression and transmission of video as well as dynamic depth maps from multiple cameras. The telepresence application places additional requirements on the compressed representation of depth maps, such as preservation of depth discontinuities, low complexity decoding, and amenability to real-time rendering using graphics cards. We propose an adaptation of an existing triangular mesh generation method for depth representation that can be encoded efficiently. The mesh geometry is encoded using a binary tree structure where single bit enabled flags that mark the split of triangles and the depth values at the tree nodes are differentially coded. By matching the tree traversal to the mesh rendering order, both depth map decoding and triangle strip generation for efficient rendering are achieved simultaneously. The proposed scheme also naturally lends itself to coding segmented foreground layers and providing error resilience. At similar compression ratio, new view generation using the proposed method provided similar quality as depth compression using JPEG2000. However, the new mesh based depth map representation and compression method showed a significant improvement in rendering speed when compared to using separate compression and rendering processes.

The Visual Computer, 2008

Most current multiplayer 3D games can only be played on dedicated platforms, requiring specifically designed content and communication over a predefined network. To overcome these limitations, the OLGA (On-Line GAming) consortium has devised a framework to develop distributive, multiplayer 3D games. Scalability at the level of content, platforms and networks is exploited to achieve the best trade-offs between complexity and quality.

2010

In this paper, we present a method to speed up video encoding of GPU rendered 3D scenes, which is particularly suited for the efficient and low-delay encoding of 3D game output as a video stream. The main idea of our approach is to calculate motion vectors directly from the 3D scene information used during rendering of the scene. This allows the omission of the computationally expensive motion estimation search algorithms found in most of today's video encoders. The presented method intercepts the graphics commands during runtime of 3D computer games to capture the required projection information without requiring any modification of the game executable. We demonstrate that this approach is applicable to games based on Linux/OpenGL as well as Windows/DirectX. In experimental results we show an acceleration of video encoding performance of approximately 25% with almost no degradation in image quality.

Concurrency and Computation: Practice and Experience, 2012

Interactive distributed visualization is an emerging technology with numerous applications. However, many of the present approaches to interactive distributed visualization have limited performance since they are based on the traditional polygonal processing graphics pipeline. In contrast, image-based rendering uses multiple images of the scene instead of a 3D geometrical representation, and so has the key advantage that the final output is independent of the scene complexity and depends on the desired final image resolution. Furthermore, the discrete nature of the Light Field dataset maps well to a hybrid solution which can overcome the identified drawbacks. In this paper we propose an on-demand solution for efficiently transmitting visualization data to remote users/clients. This is achieved through sending selected parts of the dataset based on the current client viewpoint, and is done instead of downloading a complete replica of the Light Field dataset to each client, or remotely sending a single rendered view back from a central server to the user each time the user updates their viewing parameters. The on-demand approach shows stable performance as the number of clients increases because the load on the server and the network traffic are reduced. Furthermore, detailed performance studies show that the proposed scheme outperforms the current solution in terms of interactivity measured in frames per second. of delivering between 10-15 frames/second to each user connected to the system [3]. Such a high interactive frame rate is generally difficult to achieve for systems with low to mid range rendering hardware. The second issue is the limitation of the network resources, such as low bandwidth and high latency. In order for distributed visualization systems to work efficiently under conditions of high user interactivity, which places a heavy load on the communication infrastructure, careful performance characterization and tuning are required. Current visualization systems require a traditional graphics pipeline, which uses 3D geometrical rendering to produce a 2D frame buffer, which is then sent over the network. Once datasets become sufficiently complex the number of polygons is too large to fit into the memory of a Graphics Processing Unit (GPU), and must be paged in from the main system memory. Once the dataset is larger than the local system memory, then these data must in turn be paged in from local disk. Furthermore, mobile devices, such as smart phones, are becoming more widely used. These devices have less memory and weaker processing capabilities than desktop and laptop computers, and cannot accommodate modern GPUs because of chip size, cost, and power consumption considerations. There is, therefore, a need to investigate ways to provide interactive remote visualization solutions without requiring powerful processing hardware, and this is addressed by the on-demand approach proposed in this paper.

IEEE Transactions on Circuits and Systems for Video Technology, 2004

New kinds of media are emerging that extend the functionality of available technology. The growth of immersive recording technologies has led to video-based rendering systems for photographing and reproducing environments in motion. This lends itself to new forms of interactivity for the viewer, including the ability to explore a photographic scene and interact with its features. The three-dimensional (3-D) qualities of objects in the scene can be extracted by analysis techniques and displayed by the use of stereo vision. The data types and image bandwidth needed for this type of media experience may require especially efficient formats for representation, coding, and transmission. An overview is presented here of the MPEG activity exploring the need for standardization in this area to support these new applications, under the name of 3DAV (for 3-D audio-visual). As an example, a detailed solution for omnidirectional video is presented as one of the application scenarios in 3DAV.

This paper proposes two H.264/AVC compliant methods for encoding Regions-of-Interest (ROI) with spatial scalability and evaluates their respective rate-distortion-complexity performance. The base layer is kept unchanged and provides lower resolution images with roughly constant quality, without identification of the ROI. In the proposed methods there is no need to encode contour information because the ROI is implicitly defined in the upper layer of the spatial resolution in a transparent way by using different encoding parameters for the ROI and its complementary region. It is shown, that spatial scalability in ROI can be efficiently used to enhance specific regions of an image sequence in both spatial resolution and quality with low coding complexity. The proposed encoding scheme is suitable for remote surveillance, medical applications and entertainment, where higher resolution and higher quality ROI is a useful functionality for object/face recognition, selective encryption, detail analysis, etc.

ACM Transactions on Multimedia Computing, Communications, and Applications, 2012

Mobile devices are gradually changing people's computing behaviors. However, due to the limitations of physical size and power consumption, they are not capable of delivering a 3D graphics rendering experience comparable to desktops. Many applications with intensive graphics rendering workloads are unable to run on mobile platforms directly. This issue can be addressed with the idea of remote rendering: the heavy 3D graphics rendering computation runs on a powerful server and the rendering results are transmitted to the mobile client for display. However, the simple remote rendering solution inevitably suffers from the large interaction latency caused by wireless networks, and is not acceptable for many applications that have very strict latency requirements. In this article, we present an advanced low-latency remote rendering system that assists mobile devices to render interactive 3D graphics in real-time. Our design takes advantage of an image based rendering technique: 3D im...

2009

Abstract Depth-Image-Based Rendering (DIBR) is widely used for view synthesis in 3D video applications. Compared with traditional 2D video applications, both the texture video and its associated depth map are required for transmission in a communication system that supports DIBR. To efficiently utilize limited bandwidth, coding algorithms, eg the Advanced Video Coding (H. 264/AVC) standard, can be adopted to compress the depth map using the 4: 0: 0 chroma sampling format.

Lecture Notes in Computer Science, 2006

In many application scenarios, the use of Regions of Interest (ROIs) within video sequences is a useful concept. It is shown in this paper how Flexible Macroblock Ordering (FMO), defined in H.264/AVC as an error resilience tool, can be used for the coding arbitrary-shaped ROIs. In order to exploit the coding of ROIs in an H.264/AVC bitstream, a description-driven content adaptation framework is introduced that is able to extract the ROIs of a given bitstream. The results of a series of tests indicate that the ROI extraction process significantly reduces the bit rate of the bitstreams and increases the decoding speed. In case of a fixed camera and a static background, the impact of this reduction on the visual quality of the video sequence is negligible. Regarding the adaptation framework itself, it is shown that in all cases, the framework operates in real time and that it is suited for streaming scenarios by design.