Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Suggestions for Systems Design
Conclusions
References

Gesture recognition

Profile image of Matthew TurkMatthew Turk

2001

Abstract

1. Introduction A primary goal of virtual environments (VE) is to provide natural, efficient, powerful, and flexible interaction. Gesture as an input modality can help meet these requirements. Human gestures are certainly natural and flexible, and may often be efficient and powerful, especially as compared with alternative interaction modes. This chapter will cover automatic gesture recognition, particularly computer vision based techniques that do not require the user to wear extra sensors, clothing or equipment.

Related papers

Gesture Interaction in Virtual Reality
Simon Schubiger

Human-Computer Interaction – INTERACT 2021, 2021

We explore gestures as interaction methods in virtual reality (VR). We detect hand and body gestures using human pose estimation based on offthe-shelf optical camera images using machine learning, and obtain reliable gesture recognition without additional sensors. We then employ an avatar to prompt users to learn and use gestures to communicate. Finally, to understand how well gestures serve as interaction methods, we compare the studied gesture-based interaction methods with baseline common interaction modalities in VR (controllers, gaze interaction) in a pilot study including usability testing.

View PDFchevron_right
On the usability of gesture interfaces in virtual reality environments
Márcio Cabral

2005

This paper discusses several usability issues related to the use of gestures as an input mode in multimodal interfaces. The use of gestures has been suggested before as a natural solution for applications that require hands-free and notouch interaction with computers, such as in virtual reality (VR) environments. We introduce a simple but robust 2D computer vision based gesture recognition system that was successfully used for interaction in VR environments such as CAVEs and Powerwalls. This interface was tested under 3 different scenarios, as a regular pointing device in a GUI interface, as a navigation tool, and as a visualization tool. Our experiments show that the time to completion of simple pointing tasks is considerably slower when compared to a mouse and that its use during even short periods of time causes fatigue. Despite these drawbacks, the use of gestures as an alternative mode in multimodal interfaces offers several advantages, such as quick access to computing resources that might be embedded in the environment, using a natural and intuitive way, and that scales nicely to group and collaborative applications, where gestures can be used sporadically.

View PDFchevron_right
VirtualBoard: Real-Time Visual Gesture Recognition for Natural Human-Computer Interaction
Giancarlo Iannizzotto

2003

The paper proposes a simple, economical and efficient real-time, visual hand gesture recognition system for human-computer interaction. The system is designed to work in critical lighting conditions, for example when an image is projected onto a wall (conferences, offices, etc). Given the low level of computational complexity, the system can be used instead of a mouse. The extreme naturalness of the interaction makes previous training unnecessary and the robustness of the algorithms used guarantees a high level of reliability. The architecture of the system is described, and experimental results obtained in a large number of tests with different, untrained users are presented and commented.

View PDFchevron_right
Gesture Based Interaction in Immersive Virtual Reality
Engineering and Scientific International Journal

Engineering and Scientific International Journal - Divya Udayan J , 2020

Recent development in virtual reality (VR) interaction with 3D camera and sensors like kinect, range camera, leap motion controller etc., has enabled opportunity in development of human computer interaction (HCI) application. Hand gesture is one of the popular ways that people use to interact with the computer. Even automatic hand gesture recognition appears as a suitable means for interacting with virtual reality systems. This paper focuses on the study and analysis of the application based on gesture interaction technology in virtual reality. Customizing gestures for pointing, grabbing, zoom in/out, swap were defined and implemented in unity 3D with leap motion SDK. The effectiveness of the hand gesture was analyzed through recording user experience and questionnaire.

View PDFchevron_right
GIFT: Gesture-Based Interaction by Fingers Tracking, an Interaction Technique for Virtual Environment
Muhammad Raees

International Journal of Interactive Multimedia and Artificial Intelligence, 2019

Three Dimensional (3D) interaction is the plausible human interaction inside a Virtual Environment (VE). The rise of the Virtual Reality (VR) applications in various domains demands for a feasible 3D interface. Ensuring immersivity in a virtual space, this paper presents an interaction technique where manipulation is performed by the perceptive gestures of the two dominant fingers; thumb and index. The two fingertip-thimbles made of paper are used to trace states and positions of the fingers by an ordinary camera. Based on the positions of the fingers, the basic interaction tasks; selection, scaling, rotation, translation and navigation are performed by intuitive gestures of the fingers. Without keeping a gestural database, the features-free detection of the fingers guarantees speedier interactions. Moreover, the system is user-independent and depends neither on the size nor on the color of the users' hand. With a case-study project; Interactions by the Gestures of Fingers (IGF) the technique is implemented for evaluation. The IGF application traces gestures of the fingers using the libraries of OpenCV at the back-end. At the front-end, the objects of the VE are rendered accordingly using the Open Graphics Library; OpenGL. The system is assessed in a moderate lighting condition by a group of 15 users. Furthermore, usability of the technique is investigated in games. Outcomes of the evaluations revealed that the approach is suitable for VR applications both in terms of cost and accuracy.

View PDFchevron_right
Untethered gesture acquisition and recognition for virtual world manipulation
David Demirdjian

Virtual Reality, 2005

Humans use a combination of gesture and speech to interact with objects and usually do so more naturally without holding a device or pointer. We present a system that incorporates user body-pose estimation, gesture recognition and speech recognition for interaction in virtual reality environments. We describe a vision-based method for tracking the pose of a user in real time and introduce a technique that provides parameterized gesture recognition. More precisely, we train a support vector classifier to model the boundary of the space of possible gestures, and train Hidden Markov Models on specific gestures. Given a sequence, we can find the start and end of various gestures using a support vector classifier, and find gesture likelihoods and parameters with a HMM. A multimodal recognition process is performed using rank-order fusion to merge speech and vision hypotheses. Finally we describe the use of our multimodal framework in a virtual world application that allows users to interact using gestures and speech.

View PDFchevron_right
GIST: Gesture-free Interaction by the Status of Thumb; an interaction technique for Virtual Environments
Sehat Ullah

Journal of Artificial Intelligence and Systems

User interface has special importance in immersive virtual environments. Interactions based on the simple and conceivable gestures of a hand may enhance immersivity of a Virtual Environment (VE). However, due to the structural issues like small size and complex shape of human hand, recognition of hand gestures are more challenging. This work introduces a novel interaction technique to perform the basic interaction tasks by the simple movement of hand instead of distinct gestures. With an ordinary camera, the fist posture of hand is segmented out from the image stream using the optimal segmentation model. Like pressing a button with a thumb, the status of thumb is traced for the activation or deactivation of the interactions. After the activation of interaction, the trajectory of hand is followed to manipulate a virtual object about an arbitrary axis. Without training and comparison of gestures, the basic interactions required in a VE are performed by the perceptive movement of a hand. By incorporating image processing in the realm of VE, the technique is implemented in a case-study project; FIRST (Feasible Interaction by Recognizing the Status of Thumb). A group of 12 users evaluated the system in a moderate lighting condition. Outcomes of the evaluation revealed that the technique is suitable for Virtual Reality (VR) applications.

View PDFchevron_right
Flexible gesture recognition for immersive virtual environments
Achim Ebert

… Visualization, 2006. IV …, 2006

With powerful graphics hardware becoming affordable for everyone, there is an increasing tendency towards a new generation of user interfaces, with the focus shifting from traditional two-dimensional desktops to three-dimensional virtual environments. Therefore ...

View PDFchevron_right
A Natural Interface to a Virtual Environment through Computer Vision-Estimated Pointing Gestures
Erik Granum

Lecture Notes in Computer Science, 2002

This paper describes the development of a natural interface to a virtual environment. The interface is through a natural pointing gesture and replaces pointing devices which are normally used to interact with virtual environments. The pointing gesture is estimated in 3D using kinematic knowledge of the arm during pointing and monocular computer vision. The latter is used to extract the 2D position of the user's hand and map it into 3D. Off-line tests of the system show promising results with an average errors of 76mm when pointing at a screen 2m away. The implementation of a real time system is currently in progress and is expected to run with 25Hz.

View PDFchevron_right
VISUAL RECOGNITION OF HAND POSTURES FOR INTERACTING WITH VIRTUAL ENVIRONMENTS
samuel degrande

The paper addresses the problem of visual recognition of several hand postures corresponding to a few operations commonly performed in virtual environments, such as: object selection, translation, rotation and resizing. Processing is performed in a top-view scenario with a top-mounted camera that monitors the user's hands on the working desktop. By careful choosing and controlling of the scene and lighting conditions, hands segmentation is fast and robust which increases the performances of the hand posture classifier. The chosen classifier was a multilayered perceptron with three layers. By keeping all the processing at a low level of complexity and by considering an appropriate control of the environment, we obtain a real time 25 fps functional system with high detection and recognition accuracy results.

View PDFchevron_right

References (52)

  1. References Anonymous. (1987). Visual Signals. U.S. Army Field Manual FM-2160. Available: http://155.217.58.58/atdls.html.
  2. Assan, M. and Grobel, K. (1997). Video-based sign language recognition using hidden Markov models. In I. Wachsmuth and M. Fröhlich (Eds.), Gesture and Sign Language in Human-Computer Interaction. Proc. International Gesture Workshop, Bielefeld, Germany.
  3. Baudel, T. and Beaudouin-Lafon, M. (1993). CHARADE: remote control of objects using free-hand gestures. Communications of the ACM (pp. 28-35), Vol. 36, No. 7.
  4. Black, M. and Yacoob, Y. (1995). Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. Proc. ICCV (pp. 374-381). Cambridge, MA.
  5. Bobick, A. (1997). Movement, activity, and action: the role of knowledge in the perception of motion. Royal Society Workshop on Knowledge-based Vision in Man and Machine. London, England.
  6. Böhm, K., Broll, W., and Solokewicz, M. (1994). Dynamic gesture recognition using neural networks; a fundament for advanced interaction construction. In S. Fisher, J. Merrit, and M. Bolan (Eds.), Stereoscopic Displays and Virtual Reality Systems . SPIE Conference on Electronic Imaging Science and Technology, Vol. 2177, San Jose, CA.
  7. Bolt, R. A. (1980). Put-That-There: Voice and gesture at the graphics interface. Computer Graphics, 14.3 (pp. 262-270).
  8. Boyd, J. and Little, J. (1998). Shape of Motion and the Perception of Human Gaits. IEEE Workshop on Empirical Evaluation Methods in Computer Vision. CVPR 98, Santa Barbara, CA.
  9. Cadoz, C. (1994). Les réalités virtuelles. Dominos, Flammarion, 1994.
  10. Cassell, J., Steedman, M., Badler, N., Pelachaud, C., Stone, M., Douville, B., Prevost, S., and Achorn, B. (1994). Modeling the interaction between speech and gesture. Proceedings of the Sixteenth Conference of the Cognitive Science Society.
  11. Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. (1997). QuickSet: Multimodal interaction for distributed applications, Proceedings of the Fifth Annual International Multimodal Conference (pp 31-40), Seattle, WA, ACM Press.
  12. Cutler, R. and Turk, M. (1998). View-based interpretation of real-time optical flow for gesture recognition. Proc. Third International Conference on Automatic Face and Gesture Recognition. Nara, Japan.
  13. Davis, J. and Bobick, A. (1997). The representation and recognition of human movement using temporal trajectories. Proc. IEEE Conference on Computer Vision and Pattern Recognition. Puerto Rico.
  14. Ekman, P. and Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, Calif.: Consulting Psychologists Press.
  15. Essa, I. and Pentland, A. (1997). Coding, Analysis, Interpretation and Recognition of Facial Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence,Vol. 19 (7), IEEE Computer Society Press.
  16. Fels, S. and Hinton, G. (1995). Glove-TalkII: An Adaptive Gesture-to-Formant Interface. CHI'95. Denver, CO.
  17. Freeman, W., Tanaka, K., Ohta, J., and Kyuma, K. (1996). Computer vision for computer games. Proc. Second International Conference on Automatic Face and Gesture Recognition. Killington, VT.
  18. Haritaoglu, I., Harwood, D., and Davis, L. (1998). W4: Who? When? Where? What? A real time system for detecting and tracking people. Proc. Third International Conference on Automatic Face and Gesture Recognition. Nara, Japan.
  19. Huang, T. and Pavlovic, V. (1995). Hand gesture modeling, analysis, and synthesis. Proc. International Workshop on Automatic Face-and Gesture-Recognition. Zurich.
  20. Hummels, C. and Stappers, P (1998). Meaningful gestures for human computer interaction: beyond hand gestures. Proc. Third International Conference on Automatic Face and Gesture Recognition. Nara, Japan.
  21. Johnson, T. (1963). Sketchpad III: Three Dimensional Graphical Communication with a Digital Computer. AFIPS Spring Joint Computer Conference, 23. (pp. 347-353).
  22. Kadous, W. (1996). Computer recognition of Auslan signs with PowerGloves. Proc. Workshop on the Integration of Gesture in Language and Speech, Wilmington, DE.
  23. Kendon, A. (1972). Some relationships between body motion and speech. In A. W. Siegman and B. Pope (Eds.), Studies in Dyadic Communication, New York, Pergamon Press.
  24. La Cascia, M., Isidoro, J., and Sclaroff, S. (1998). Head tracking via robust registration in texture map images. Proc. IEEE Conference on Computer Vision and Pattern Recognition. Santa Barbara, CA.
  25. Landay, J. A. and Myers, B. A. (1995). Interactive sketching for the early stages of user interface design. Proceedings of CHI'95 (pp. 43-50).
  26. Latoschik, M. and Wachsmuth, I. (1997). Exploiting distant pointing gestures for object selection in a virtual environment. In I. Wachsmuth, and M. Fröhlich (Eds.), Gesture and Sign Language in Human-Computer Interaction. International Gesture Workshop, Bielefeld, Germany.
  27. Lee, C. and Xu, Y. (1996). Online, Interactive Learning of Gestures for Human/Robot Interfaces. 1996 IEEE International Conference on Robotics and Automation, Vol. 4 (pp. 2982-2987). Minneapolis, MN.
  28. Long, A., Landay, J., and Rowe, L. (1998). PDA and Gesture Uses in Practice: Insights for Designers of Pen- Based User Interfaces. Report #CSD-97-976, CS Division, EECS Department. UC Berkeley. Berkeley, CA.
  29. Lucente, M., Zwart, G., and George, A. (1998). Visualization Space: a testbed for deviceless multimodal user interface. Intelligent Environments Symposium. AAAI Spring Symposium Series. Stanford, CA.
  30. Maggioni, C. and Kämmerer, B. (1998). GestureComputer -history, design and applications. In R. Cipolla and A. Pentland (Eds.), Computer Vision for Human-Machine Interaction. Cambridge University Press.
  31. Marrin, T. and Picard, R. (1998). The Conductor's Jacket: a Testbed for Research on Gestural and Affective Expression. XII Colloquium for Musical Informatics, Gorizia, Italy.
  32. McNeill. D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press.
  33. Moses, Y., Reynard, D., and Blake, A. (1995). Determining facial expressions in real time. Proc. Fifth International Conference on Computer Vision. Cambridge, MA.
  34. Mulder, A. (1996). Hand gestures for HCI. Technical Report 96-1, School of Kinesiology. Simon Fraser University.
  35. Oliver, N., Pentland, A., and Bérard, F. (1997). LAFTER: Lips and face real time tracker. Proc. IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico.
  36. Otsuka, T. and Ohya, J. (1998). Recognizing abruptly changing facial expressions from time-sequential face images. Proc. IEEE Conference on Computer Vision and Pattern Recognition. Santa Barbara, CA.
  37. Oviatt, S. L. (1996). Multimodal interfaces for dynamic interactive maps. Proceedings of CHI'96 Human Factors in Computing Systems (pp. 95-102). ACM Press, NY.
  38. Paradiso, J. and Sparacino, F. (1997). Optical Tracking for Music and Dance Performance. Fourth Conference on Optical 3-D Measurement Techniques. Zurich, Switzerland.
  39. Pavlovic.
  40. V, Sharma, R., and Huang, T. (1996). Gestural interface to a visual computing environment for molecular biologists. Proc. Second International Conference on Automatic Face and Gesture Recognition. Killington, VT.
  41. Stark, M. and Kohler, M. (1995). Video based gesture recognition for human computer interaction. In W. D. Fellner (Ed.), Modeling -Virtual Worlds -Distributed Graphics.
  42. Starner, T. and Pentland, A. (1995). Visual recognition of American Sign Language using hidden Markov models. Proc. International Workshop on Automatic Face-and Gesture-Recognition. Zurich.
  43. Sturman, J. (1992). Whole-hand Input. Ph.D. Thesis, MIT Media Laboratory. Cambridge, MA.
  44. Takahashi, T. and Kishino, F. (1991). Gesture coding based in experiments with a hand gesture interface device. SIGCHI Bulletin, 23(2), (pp. 67-73).
  45. Triesch, J. and von der Malsburg, C. (1996). Robust classification of hand postures against complex backgrounds. Proc. Second International Conference on Automatic Face and Gesture Recognition. Killington, VT.
  46. Väänänen, K. and Böhm, K. (1992). Gesture driven interaction as a human factor in virtual environments -an approach with neural networks. Proc. Virtual Reality Systems . British Computer Society, Academic Press.
  47. Weng, J. and Cui, Y. (1998). Recognition of hand signs from complex backgrounds. In R. Cipolla and A. Pentland (Eds.), Computer Vision for Human-Machine Interaction. Cambridge University Press.
  48. Wexelblat, A. (1994). A Feature-Based Approach to Continuous-Gesture Analysis. M.S. Thesis, MIT Media Laboratory, Cambridge, MA.
  49. Wren, C., Azarbayejani, A., Darrell, T., and Pentland, A. (1996). Pfinder: real-time tracking of the human body. Proc. Second International Conference on Automatic Face and Gesture Recognition. Killington, VT.
  50. Yamamoto, J., Kondo, T., Yamagiwa, T., and Yamanaka, K. (1998). Skill recognition. Proc. Third International Conference on Automatic Face and Gesture Recognition. Nara, Japan.
  51. Zeleznik, R.C., Herndon, K.P., and Hughes J.F. (1996). Sketch: An interface for sketching 3D scenes. Computer Graphics (Proceedings of SIGGRAPH '96).
  52. Zelinsky, A. and Heinzmann, J. (1996). Real-time visual recognition of facial gestures for human-computer interaction. Proc. Second International Conference on Automatic Face and Gesture Recognition. Killington, VT.

Related papers

Interacting with Gestures: An Intelligent Virtual Environment
Stefan-Gheorghe Pentiuc

2006

The paper addresses the opportunity of using gestures in virtual environments by taking into account current technologies for capturing human gestures, existing interaction techniques and various aspects dealing with gesture and gesticulation functionalities. Virtual environments have proven to be of an effective usefulness for a variety of applications such as medical training and exploration, military simulations, architecture, phobia therapy, etc. A great variety of devices have been developed for effectively interacting with such environments and many interaction techniques have been introduced. Many of them prove to be cumbersome and more or less obstructing the user and hence affecting his/her virtual experience. Natural gestures as a mean of human computer interaction have the great advantage of being the ideal interface as natural, efficient and intuitive means of communication (people use gestures everyday in a natural manner). Discussion is equally conducted on gestures being used for the three main tasks commonly encountered in virtual environments: selection, manipulation and travel. References to gesture definition and classification according to different criteria and their relevance for interacting in virtual environments are equally taken into account.

View PDFchevron_right
Gesture recognition in virtual reality environments
Jarosław Andrzejczak

Computer Game Innovations, 2017

In recent years there has been great progress in virtual reality systems utilizing 6D motion controllers, but there has been relatively little progress in user interfaces taking advantage of this technology. We present a motion gesture recognizer based on an artificial neural network. By utilizing both convolutional and recurrent layers we have developed a system which needs relatively few training samples to label a wide variety of classes. With reduced input feature count our solution provides comparable success rate to traditional hidden Markov model recognizer. This allows to use our system with motion controllers based on internal measurement units.

View PDFchevron_right
An approach to natural gesture in virtual environments
Alan Wexelblat

ACM Transactions on Computer-Human Interaction, 1995

This article presents research-an experiment and the resulting prototype-on a method for treating gestural input so that it can be used for multimodal applications, such as interacting with virtual environments.

View PDFchevron_right
A bare-hand gesture interaction system for virtual environments
Alex Flores

2014 International Conference on Computer Graphics Theory and Applications (GRAPP), 2014

Hand-based gestures provide direct mappings of user actions to 3D UI tasks; they are becoming a more attractive interaction alternative than keyboards, mice, controllers, among others. In this paper, we present a fast algorithm for hand gesture recognition for interaction in virtual environments. The method applies mathematical morphology operators (binarization and dilation) to acquire a clean segmented hand image from a depth data stream on which a curvature-metric and K-means algorithm is applied to detect the fingertips, then using fingertip and palm positions together with anthropomorphic metrics and a rule based system we perform gesture recognition. In addition, the intermittence gesture spotting problem is reduced using a digital integrator. Finally, a set of virtual environments were designed to demonstrate the performance, reliability and feasibility of our method.

View PDFchevron_right
Hand gesture interaction with a 3D virtual environment
Mohammad Alhaj

2011

T h e R e s e a r c h B u l l e t i n o f J o r d a n A C M , I S S N : 2 0 7 8 -7 9 5 2 , V o l u m e I I ( I I I ) P a g e | 86 ABSTRACT Virtual Environment (VE) system offers a natural and intelligent user interface. Hand gesture recognition is more efficient and easier interaction in VE than human-computer interface (HCI) devices like keyboards and mouses. We propose a hand gesture recognition interface that generates commands to control objects directly in a game. Our novel hand gesture recognition system utilizes both Bag-offeatures and Support Vector Machine (SVM) to realize user-friendly interaction between human and computers. The HCI based on hand gesture recognition interacts with objects in a 3D virtual environment. With this interface, the user can control and direct a helicopter by a set of hand gesture commands controlling the movements of the helicopter. Our system shows the hand gesture recognition interface can attain an enhanced and more intuitive and flexible interaction for the user than other HCI devices.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved