Exploring augmented live video streams for remote participation

1996

In the future, the computer will be thought of more as an assistant than as a tool, and users will increasingly expect machines to make decisions on their behalf. As with a human assistant, a machine’s ability to make informed choices will often depend on the extent of its knowledge of activities in the world around it. Equipping personal computers with a large number of sensors for monitoring their environment is, however, expensive and inconvenient, and a preferable solution would involve a small number of input devices with a broad scope of application. Video cameras are ideally suited to many real- world monitoring applications for this reason. In addition, recent reductions in the manufacturing costs of simple cameras will soon make their widespread deployment in the home and office economically viable. The use of video as an input device also allows the creation of new types of user-interface, more suitable in some circumstances than those afforded by the conventional keyboard and mouse. This thesis examines some examples of these ‘Video-Augmented Environments’ and related work, and then describes two applications in detail. The first, a ‘software cameraman’, uses the analysis of one video stream to control the display of another. The second, ‘BrightBoard’, allows a user to control a computer by making marks on a conventional whiteboard, thus ‘augmenting’ the board with many of the facilities common to electronic documents, including the ability to fax, save, print and email the image of the board. The techniques which were found to be useful in the construction of these applications are common to many systems which monitor real-world video, and so they were combined in a toolkit called ‘Vicar’. This provides an architecture for ‘video plumbing’, which allows standard video- processing components to be connected together under the control of a scripting language. It is a single application which can be programmed to create a variety of simple Video-Augmented Environments, such as those described above, without the need for any recompilation, and so should simplify the construction of such applications in the future. Finally, opportunities for further exploration on this theme are discussed.

This is a paper accompanying (and describing) a demonstration. The intention of this demonstration is to present a novel immersive telepresence system that enables remote students to participate in seminars and lectures using online streaming video and audio connections. In this system, a virtualized video view is created using a 360° panoramic video projected onto a 180° curved projected screen (immersive shell). This recreates a more natural human-like perception of real environments and thereby stimulating the learning process;. 3D audio is also collected and reproduced at the remote location adding to the realism. To accomplish this we use a 360° mirror situated in the classroom which we use with a camera to transmit a panoramic image to the remote users where they reconstruct the original image from spherical to Cartesian. To process the audio we use a small array of microphones at the classroom end. In addition, we provide various tools to allow the participants to control their position within the virtualized views, thereby creating an innovative technology and user experience. We will be demonstrating this system at the conference.

In this paper, we describe the overall design of the remote collaboration apparatus t-Room and present three applications: playback of a recorded scene using a hand controller, an elevator effect at scene change, and remote golf lessons. These applications are realized by the high controllability and flexibility of the t-Room system, and they can provide the user with a novel type of spatiotemporal experience.

2008

This paper presents a novel tangible user interface for the dynamic control and manipulation of multiple video streams. Streams can be activated, triggered and managed based on the physical use of markers attached to cubes on a table top, whose movement is recognized by a camera placed on top of the working area. Markers can be associated to a single video stream, as well as a group of streams. Videostreams can be played out, paused, stopped, put in foreground, maximized or minimized, thanks to the use of our tangible interface; audio volumes of different streams can be controlled. Needless to say, streams can be distributed at different servers, or dynamically generated at distributed nodes in real-time. We claim that such an interface results as quite intuitive and easy-to-use, and can be employed in several videostreaming-based application domains, such as tele-conferencing, video-on-demand, video mixing. Results coming from an experimental assessment confirm the viability of our approach.

Proceedings of the …, 2011

The increasing number of media facades in urban spaces offers great potential for new forms of interaction -especially for collaborative multi-user scenarios. In this paper, we present a way to directly interact with them through live video on mobile devices. We extend the Touch Projector interface to accommodate multiple users by showing individual content on the mobile display that would otherwise clutter the facade's canvas or distract other users. To demonstrate our concept, we built two collaborative multi-user applications: (1) painting on the facade and (2) solving a 15-puzzle. We gathered informal feedback during the ARS Electronica Festival in Linz, Austria and found that our interaction technique is (1) considered easy-to-learn, but (2) may leave users unaware of the actions of others.

2003

We present the architecture, technology and experimental applications of a real-time, multi-site, interactive and collaborative environment called Distributed Immersive Performance (DIP). The objective of DIP is to develop the technology for live, interactive musical performances in which the participants -subsets of musicians, the conductor and the audience -are in different physical locations and are interconnected by very high fidelity multichannel audio and video links. DIP is a specific realization of broader immersive technology -the creation of the complete aural and visual ambience that places a person or a group of people in a virtual space where they can experience events occurring at a remote site or communicate naturally regardless of their location. The DIP experimental system has interaction sites and servers in different locations on the USC campus and at several partners, including the New World Symphony of Miami Beach, FL. The sites have different types of equipment to test the effects of video and audio fidelity on the ease of use and functionality for different applications. Many sites have high-definition (HD) video or digital video (DV) quality images projected onto wide screen wall displays completely integrated with an immersive audio reproduction system for a seamless, fully three-dimensional aural environment with the correct spatial sound localization for participants. The system is capable of storage and playback of the many streams of synchronized audio and video data (immersidata), and utilizes novel protocols for the low-latency, seamless, synchronized realtime delivery of immersidata over local area networks and widearea networks such as Internet2. We discuss several recent interactive experiments using the system and many technical challenges common to the DIP scenario and a broader range of applications.

Proceedings of the eleventh ACM international conference on Multimedia - MULTIMEDIA '03, 2003

We present a system that allows remote and local participants to control devices in a meeting environment using mouse or pen based gestures "through" video windows. Unlike state-of-theart device control interfaces that require interaction with text commands, buttons, or other artificial symbols, our approach allows users to interact with devices through live video of the environment. This naturally extends our video supported pan/tilt/zoom (PTZ) camera control system, by allowing gestures in video windows to control not only PTZ cameras, but also other devices visible in video images. For example, an authorized meeting participant can show a presentation on a screen by dragging the file on a personal laptop and dropping it on the video image of the presentation screen. This paper presents the system architecture, implementation tradeoffs, and various meeting control scenarios.

Abstract Presented at ELFI 2019, European Light Field Imaging Workshop , Borovets, Bulgaria, 2019

Communication is the most useful tool to impart knowledge, understand ideas, clarify thoughts and expressions, organize plan and manage every single day-to-day activity. Although there are different modes of communication, physical barrier always affects the clarity of the message due to the absence of body language and facial expressions. These barriers are overcome by video calling, which is technically the most advance mode of communication at present. The proposed work concentrates around the concept of video calling in a more natural and seamless way using Augmented Reality (AR). AR can be helpful in giving the users an experience of physical presence in each other’s environment. Our work provides an entirely new platform for video calling, wherein the users can enjoy the privilege of their own virtual space to interact with the individual’s environment. Moreover, there is no limitation of sharing the same screen space. Any number of participants can be accommodated over a single conference without having to compromise the screen size.

2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2013

This paper describes an AR system for remote collaboration using a captured 3D model of the local user's scene. In the system a remote user can manipulate the scene independently of the view of the local user and add AR annotations that appear projected into the real world. Results from a pilot study and the design of a further full study are presented.

ACM International Conference on Interactive Media Experiences

Figure 1: TeleFest, our system for livestreaming events using tailored 360° content edited in real time by a producer. (Left): A real live performance streamed live to almost 2,000 people using TeleFest. Three 360° cameras were placed among the stage and crowd, and were livestreamed to YouTube with augmented 3D virtual content to enhance the remote viewing experience. (Right): The resulting livestream.