Gestural communication over video stream

The 3D structure of people's body movements and gestures become distorted when a sequence of those movements and gestures is captured by multiple cameras and displayed on multiple 2D screens at a remote site. This increases the chances of remote people losing sight of those gestures, possibly making it difficult to understand the meaning intended by the gestures. To alleviate such problems, we propose a visual augmentation technique, called "remote lag," that applies the concept of telepointer traces to the bodily gestures performed in a videoconferencing system. Remote lag is a visual effect that overlays a user's past motion image onto his/her present image. In our experiment, we compared several types of remote lag and examined their effects on how accurately a user can perceive a remote user's pointing location. Our overall results show that when a user misses gestures, remote lag contributes to the recovery of a gesture context. In particular, when a remote user stands right beside a user, a particular type of remote lag is helpful.

Advances in Human-Computer Interaction, 2011

This paper presents the results of an experimental investigation of two gesture representations (overlaying hands and cursor pointer) in a video-mediated scenario—remote collaboration on physical task. Our study assessed the relative value of the two gesture representations with respect to their effectiveness in task performance, user's satisfaction, and user's perceived quality of collaboration in terms of the coordination and interaction with the remote partner. Our results show no clear difference between these two gesture representations in the effectiveness and user satisfaction. However, when considering the perceived quality of collaboration, the overlaying hands condition was statistically significantly higher than the pointer cursor condition. Our results seem to suggest that the value of a more expressive gesture representation is not so much a gain in performance but rather a gain in user's experience, more specifically in user's perceived quality of colla...

Visual Information Processing and Communication III, 2012

In the context of immersive communications, we propose a method enabling natural video interactions through hand gesture recognition between users and a video meeting system. The interaction can be performed either by the mean of hand posture recognition or by the dynamic hand gesture recognition according to user's preference. The statistical approach adopted in our work to recognize hand posture has shown accurate results for both performance evaluation and user test. Besides, the combination of data-mining fields and signal processing for dynamic gestures recognition allows us to define the appropriate rules and to reduce the confusion between gestures. Furthermore, the hand region extraction is based on both skin color and background subtraction to avoid the detection of static objects that have a similar skin color. Finally, the collected user's feedback allows as to evaluate our approach from the user's point of view and to define the limitations that will be discussed in our perspectives in order to improve the results.

2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2013

In this research, we explore using pointing and drawing in a remote collaboration system. Our application allows a local user with a tablet to communicate with a remote expert on a desktop computer. We compared performance in four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. We found that using drawing annotations would require fewer inputs on an expert side, and would require less cognitive load on the local worker side. In a follow-on study we compared the conditions (2) and (4) using a more complicated task. We found that pointing input requires good verbal communication to be effective and that drawing annotations need to be erased after completing each step of a task.

Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '17, 2017

Personal assistants such as Siri have changed the way people interact with computers by introducing virtual assistants that collaborate with humans through natural speech-based interfaces. However, relying on speech alone as the medium of communication can be a limitation; non-verbal aspects of communication also play a vital role in natural human discourse. Thus, it is necessary to identify the use of gesture and other nonverbal aspects in order to apply them towards the development of computer systems. We conducted an exploratory study to identify how humans use gesture and speech to communicate when solving collaborative tasks. We highlight differences in gesturing strategies in the presence/absence of speech and also show that the inclusion of gesture with speech resulted in faster task completion times than with speech alone. Based on these results, we present implications for the design of gestural and multimodal interactions.

2007 IEEE Conference on Computer …, 2007

We present DIGITABLE, an experimental platform we hope lessen the gap between co-present and distant inter-action. DIGITABLE is combining a multiuser tactile in-teractive tabletop, a video-communication system enabling eye-contact with real size distant user ...

Tsinghua Science & Technology

In this paper, a real-time system that utilizes hand gestures to interactively control the presentation is proposed. The system employs a thermal camera for robust human body segmentation to handle the complex background and varying illumination posed by the projector. A fast and robust hand localization algorithm is proposed, with which the head, torso, and arm are sequentially localized. Hand trajectories are segmented and recognized as gestures for interactions. A dual-step calibration algorithm is utilized to map the interaction regions between the thermal camera and the projected contents by integrating a Web camera. Experiments show that the system has a high recognition rate for hand gestures, and corresponding interactions can be performed correctly.

2006

We present a Computer Vision based system that enables multiple people to interact naturally with a large display table using their own bare-hand gestures. The display presents and supports a particular multimedia application that can be used at the same time even by remote users. Finally we describe two different applications designed for didactic and entertainment scenarios.

2017

In recent years, motion capture technology to measure the movement of the body has been used in many fields. Moreover, motion capture targeting multiple people is becoming necessary in multi-user virtual reality (VR) and augmented reality (AR) environments. It is desirable that motion capture requires no wearable devices to capture natural motion easily. Some systems require no wearable devices using an RGB-D camera fixed in the environment, but the user has to stay in front of the fixed the RGB-D camera. Therefore, in this research, proposed is a motion capture technique for a multi-user VR / AR environment using head mounted displays (HMDs), that does not limit the working range of the user nor require any wearable devices. In the proposed technique, an RGB-D camera is attached to each HMD and motion capture is carried out mutually. The motion capture accuracy is improved by modifying the depth image. A prototype system has been implemented to evaluate the effectiveness of the pro...

pedagogie.ec-nantes.fr

This paper proposes using virtual reality to enhance the perception of actions by distant users on a shared application. Here, distance may refer either to space (e.g. in a remote synchronous collaboration) or time (e.g. during playback of recorded actions). Our approach consists in immersing the application in a virtual inhabited 3D space and mimicking user actions by animating avatars whose motion can be either generated from user actions on the shared application or from motion capture by a computer vision system we briefly describe. We illustrate this approach with two applications, one for remote collaboration on a shared application and the other to playback recorded sequences of user actions. We suggest this could be a low cost enhancement for telepresence.