A tele-immersive system based on binocular view interpolation

2000

Today's technology and advances in networking and multimedia systems stimulate a change in the way business is carried out, making it a globally distributed process, in which communication and collaboration of geographically dispersed group is of vital importance. Teleconferencing and collaborative telepresence systems that provide high-degree of copresence give enough evidences that projective VR systems when combined with multimedia facilities, such as real-time video and audio, can greatly facilitate the communication and collaboration over distance in a variety of application areas. The approach presented in this paper, creates an environment where remote participants not only meet as if face to face, but also share the same virtual space and perform common tasks. Multimedia datastreams, such as live stereo-video and audio, from a projective VR system are transmitted and integrated into the virtual space of another participant at a distant VR system, allowing geographically separated groups to meet in a common virtual space, while maintaining eyecontact, gaze awareness and body language.

1994

A new approach to telepresence is presented in which a m ultitude of stationary cameras are used to acquire both photometric and depth information. A virtual environment is constructed by displaying the acquired data from the remote site in accordance with the head position and orientation of a local participant. Shown are preliminary results of a depth image of a human subject calculated from 11 closely spaced video camera positions. A user wearing a head-mounted display walks around this 3D data that has been inserted into a 3D model of a simple room. Future systems based on this approach m a y exhibit more natural and intuitive i n teraction among participants than current 2 D teleconferencing systems.

International Symposium on VIPromCom Video/Image Processing and Multimedia Communications, 2002

An important aim of immersive teleconferencing systems is to create realistic 3D virtual views of remote conferees. Hence, systems should be able to deal with hand gestures as well as occluded areas in reference images required in derived views. The quality of such derived views is dependent not only on the analysis and synthesis process but also the multiview camera setup. Often the popular convergent wide-baseline stereo approach aspires to achieve too much through a single camera pair: maximum information and reliable disparity maps. We identify how this dichotomy leads to problems in the analysis and synthesis process, often leading to a restrictive system specific solution. We then define a new approach, a multiple narrow-baseline setup , designed to overcome the limitations of the wide-baseline setup , being modular, both in terms of system requirements as well as algorithmically, and scalable, with respect to the number of conferees.

In this paper, a next generation 3-D video conferencing system is presented that provides immersive telepresence and natural representation of all participants in a shared virtual meeting space. The system is based on the principle of a shared virtual table environment which guarantees correct eye contact and gesture reproduction and enhances the quality of human-centered communication. The virtual environment is modeled in MPEG-4 which also allows the seamless integration of explicit 3-D head models for a low-bandwidth connection to mobile users. In this case, facial expression and motion information is transmitted instead of video streams resulting in bit-rates of a few kbit/s per participant. Beside low bit-rates, the model-based approach enables new possibilities for image enhancements like digital make-up, digital dressing, or modification of scene lighting.

This paper details a new modular approach to virtual view creation that is designed to work in conjunction with a proposed scalable teleconferencing configuration. This scalable system is configured using building blocks defined as Stereo Camera Analysis Blocks (SCAB). SCABs consist of axe-parallel narrow baseline stereo cameras. These provide design flexibility and improve the image analysis process. Virtual view creation is modular in such that we can add or remove SCABs based on our particular requirements without having to modify the view synthesis algorithm. A new approach to virtual view creation from multiple images is defined. This contains two separate processes: surface segmentation, which identifies surfaces, and surface selection and merging, which selects and integrates the best view of a required surface into the virtual view. The surfaces are identified via a sampling density function while the surface selection procedure is based on a weighting scheme.

Stereoscopic Displays and Virtual Reality Systems XIV, 2007

We present a point based reconstruction and transmission pipeline for a collaborative tele-immersion system. Two or more users in different locations collaborate with each other in a shared, simulated environment as if they were in the same physical room. Each user perceives point-based models of distant users along with collaborative data like molecule models. Disparity maps, computed by a commercial stereo solution, are filtered and transformed into clouds of 3D points. The clouds are compressed and transmitted over the network to distant users. At the other side the clouds are decompressed and incorporated into the 3D scene. The viewpoint used to display the 3D scene is dependent on the position of the head of the user. Collaborative data is manipulated through natural hand gestures. We analyse the performance of the system in terms of computation time, latency and photo realistic quality of the reconstructed models.

We present an image-based rendering system to viewpoint-navigate through space and time of complex real-world, dynamic scenes. Our approach accepts unsynchronized, uncalibrated multi-video footage as input. Inexpensive, consumer-grade camcorders suffice to acquire arbitrary scenes, e.g., in the outdoors, without elaborate recording setup procedures, allowing also for hand-held recordings. Instead of scene depth estimation, layer segmentation, or 3D reconstruction, our approach is based on dense image correspondences, treating view interpolation uniformly in space and time: spatial viewpoint navigation, slow motion, or freeze-and-rotate effects can all be created in the same way. Acquisition simplification, integration of moving cameras, generalization to difficult scenes, and space-time symmetric interpolation amount to a widely applicable Virtual Video Camera system.

IEEE Transactions on Circuits and Systems for Video Technology, 2004

The processing power and network bandwidth required for true immersive telepresence applications are only now beginning to be available. We draw from our experience developing stereo based tele-immersion prototypes to present the main issues arising when building these systems. Tele-immersion is a new medium that enables a user to share a virtual space with remote participants. The user is immersed in a rendered three-dimensional (3-D) world that is transmitted from a remote site. To acquire this 3-D description, we apply binocular and trinocular stereo techniques which provide a view-independent scene description. Slow processing cycles or long network latencies interfere with the users' ability to communicate, so the dense stereo range data must be computed and transmitted at high frame rates. Moreover, reconstructed 3-D views of the remote scene must be as accurate as possible to achieve a sense of presence. We address both issues of speed and accuracy using a variety of techniques including the power of supercomputing clusters and a method for combining motion and stereo in order to increase speed and robustness. We present the latest prototype acquiring a room-size environment in real time using a supercomputing cluster, and we discuss its strengths and current weaknesses.

Immersive telecommunication is a new challenging field that enables a user to share a virtual space with remote participants providing users with a sense of telepresence and interaction. This paper presents an implementation of real-time dynamic 3D avatar from multi-view cameras for the immersive telecommunication. The main objective is to offer rich communication modalities, as similar as those used in the face-to-face meetings like gestures, gaze awareness, realistic images, and correct sound direction. The proposed algorithm has been integrated to the framework of distributed VR, called NAVER, and an example of application, Heritage Alive!, has been implemented to demonstrate the concept. As a result, the user can be immersed and has natural interaction with remote participant. This would overcome the limitations both of the conventional videobased telecommunication and also the VR-based collaborative virtual environment approaches.

Stereoscopic Displays and Virtual Reality Systems XIV, 2007

We present a point based reconstruction and transmission pipeline for a collaborative tele-immersion system. Two or more users in different locations collaborate with each other in a shared, simulated environment as if they were in the same physical room. Each user perceives point-based models of distant users along with collaborative data like molecule models. Disparity maps, computed by a commercial stereo solution, are filtered and transformed into clouds of 3D points. The clouds are compressed and transmitted over the network to distant users. At the other side the clouds are decompressed and incorporated into the 3D scene. The viewpoint used to display the 3D scene is dependent on the position of the head of the user. Collaborative data is manipulated through natural hand gestures. We analyse the performance of the system in terms of computation time, latency and photo realistic quality of the reconstructed models.